Back to Artificial-intelligence
Related Knowledge Hub
Knowledge Hub Artificial-intelligence
Published by Contributor
Tokenization and Vectorization Example 01
Accepted Answer
Tokenization is when we break down a sentence into individual words or smaller parts, called tokens. Think of it as cutting a sentence into smaller pieces, like separating the words to understand each one.
Vectorization is when we turn those words into numbers, so a computer can work with them. Each word gets its own number or a list of numbers (called a vector) that helps the computer understand the meaning of the word.
Example:
Sentence:
- "I love apples."
Tokenization:
- After tokenization, the sentence becomes:
["I", "love", "apples"]. - Now, each word is a separate piece the computer can look at.
- After tokenization, the sentence becomes:
Vectorization:
- The computer converts each word into numbers. For example:
- "I" →
[0.1, 0.2, 0.3] - "love" →
[0.4, 0.5, 0.6] - "apples" →
[0.7, 0.8, 0.9]
- "I" →
- These numbers help the computer understand that "love" and "apples" are related in some way.
- The computer converts each word into numbers. For example:
Using Search:
- If someone searches for "I like apples," the system will:
- Tokenize and vectorize "I like apples."
- Compare it to the vectors of "I love apples" (from above).
- Since "like" and "love" are similar, and "apples" matches exactly, the system will see that these two sentences are close in meaning and return it as a result.
- If someone searches for "I like apples," the system will:
This is how tokenization and vectorization help computers understand text and match similar content!
Want to report this post?
Please contact the ChemistAi team.