X
Jamaley Hussain

NLP – Tokenization

Tokenization

It’s the process of breaking down the text into sentences and phrases. The work entails breaking down a text into smaller chunks (known as tokens) while discarding some characters, such as punctuation.

Consider the following example:

Text input: Potter walked to school yesterday.

Potter went to school yesterday, according to the text output.

Related posts:

NLP – Natural language processing [Intro] Dynamic Dependent Multiple Relational Dropdown using (Jquery, AJAX, PHP, MySQL) NLP – Corpus
Categories: Short Notes
Tags: Data ScienceNatural language processingNLP
Jamaley Hussain: Hello, I am Jamaley. I graduated from Staffordshire University and have always been passionate about Computers, Technology, and Generative AI. Currently, I work as a Senior Data Scientist (AI/ML) and I’m also the founder of TechJunkGigs, a platform dedicated to helping programming students with tutorials on Machine Learning, Data Science, Python, LLM, RAG, Generative AI, and NLP. What started as a blog has now evolved into a valuable resource for students, and I'm committed to sharing knowledge to help them stay updated with industry trends
Related Post
  1. ML – Customer Segmentation

    Dividing customers into groups based on similar functionality or customer segmentation is based on the…

  2. NLP – Bag of Words

    Bag of Words A bag of words is a particular representation model used to simplify…

  3. NLP – Word segmentation

    Word segmentation  This is the act of taking a string of text and deriving word…