How is AI made?
You don't need to know . Feed in data and watch being built through → vectorization → → generation. If you want, you can also build it with real code.
A toy : it looks only at the previous word. It does not learn , , or facts. The word splitter is a teaching example, not an .
- Data prep
- ① Tokenization
- ② IDs and one-hot vectors
- ③ Training
- ④ Generation
Data prep
Collect a few short example sentences. Repetition changes the counts; more text is useful only if it fits your purpose. Each sentence ends separately.
① Tokenization
This demo splits sentences into word-like pieces. Real LLM tokenizers can use words, word pieces, characters, and . Tokenization is about representing text, not about text being too long to understand.
② IDs and one-hot vectors
An ID is a number used as a label. A marks that ID's position with 1. Neither tells us which words have similar meanings. A learned embedding is a different kind of ; this demo does not train one.
③ Training
Count each transition, including sentence endings. A candidate's is its count divided by the total count after the selected word. Neural learn instead of just storing this small table.
④ Generation
Choose the most likely candidate (), or draw according to probabilities (). Sentence-end stops generation. Sampling can produce the same result again. Generated text is not checked for truth.
| Word | Number | Number vector (one-hot) |
|---|---|---|
Pick a word to see what the learned about which word is likely to come next:
Percentages are rounded; their displayed total may differ slightly from 100%.
This word only appears at the end of a sentence, so there's no next word. Try another word.
The table shows observed frequencies at 1. Temperature changes sampling probabilities, not the learned counts or factual accuracy.
Preparing the Python runtime…
My Code
Output
Press [Run] and the result will appear here.