Understanding LLMs from Scratch with School Mathematics

1. The whole pipeline in one table

2. Tokenization: text becomes pieces

A model does not multiply letters directly. A tokenizer converts text into a sequence of token IDs. A token may be a whole word, punctuation mark or reusable piece of a longer word depending on the tokenizer.

"students learn fast"
→ ["students", " learn", " fast"]
→ [8142, 2231, 991]

3. Embeddings: IDs become coordinates

Token ID 8142 is only a label. An embedding table maps it to a vector. Imagine a tiny three-number vector even though real models use many more dimensions.

student → [0.8, 0.2, -0.4]
teacher → [0.7, 0.3, -0.2]
banana  → [-0.1, 0.9, 0.5]

4. Dot products are multiplication plus addition

Attention compares learned Query and Key vectors. A dot product multiplies matching positions and adds them. For Q=[1,2] and K=[3,4], the dot product is 1×3 + 2×4 = 11. Higher compatibility can become a higher attention score.

function dot(a, b) {
  return a.reduce((sum, value, index) => sum + value * b[index], 0);
}

console.log(dot([1, 2], [3, 4])); // 11

5. Softmax turns arbitrary scores into percentages

function softmax(values) {
  const max = Math.max(...values);
  const exp = values.map((value) => Math.exp(value - max));
  const total = exp.reduce((sum, value) => sum + value, 0);
  return exp.map((value) => value / total);
}

console.log(softmax([2, 1, 0]));

Subtracting the maximum value before exponentiation is a numerical-stability trick. The final values are positive and sum to approximately 1, so they can be interpreted as shares.

6. Query, Key and Value

The model learns matrices that transform each token representation into these three views. Attention scores come from Query–Key compatibility; the resulting weights mix the Values.

7. Why multiple attention heads?

One head gives one learned projection of relationships. Multiple heads let the layer model several kinds of relationships in parallel before combining them. Do not assume each head always has one clean human-readable purpose; the useful idea is parallel learned views.

8. Position still matters

The same words in a different order can mean something different. Transformer models therefore encode position, often using learned or rotary-style methods, so the representation includes order information.

9. The transformer block

1. 1
   Attention

   Positions exchange information.

2. 2
   Residual path

   Earlier representation is added back.

3. 3
   Normalization

   Keeps activations in a useful numerical regime.

4. 4
   Feed-forward network

   Transforms each position independently after attention.

5. 5
   Repeat

   Many blocks deepen the representation.

10. Causal masking: GPT cannot look into the future

During next-token training, the representation for position i is prevented from attending to later positions. This causal mask makes the task match generation: predict the next token using only earlier context.

11. The final logits and next-token probabilities

The final representation is projected to one score for every token in the vocabulary. Softmax converts those logits into a probability distribution. Decoding chooses a next token, appends it, and repeats.

12. Training

1. 1
   Input

   Feed token sequences.

2. 2
   Predict

   Produce next-token probabilities.

3. 3
   Compare

   Measure error against the actual next token.

4. 4
   Backpropagate

   Compute how parameters influenced the error.

5. 5
   Update

   Adjust parameters a small amount.

6. 6
   Repeat

   Do this across huge numbers of examples.

13. What this simplified lesson leaves out

14. A good mental model

An LLM is a learned numerical system that repeatedly transforms token representations and estimates which token is likely to come next. The scale is huge; the building blocks are understandable.

KEEP READING

You can also read

Suggestions use topic similarity, and — when you enable it — your reading history stored only on this device.

GCE Study GuideCameroon GCE Past Papers: A Complete O-Level and A-Level Study Guide15 min read →Clarvo & PrivacyHow to Delete Your Clarvo Account — What Happens, What You Lose, and What to Check First11 min read →Digital PrivacyDeleting an Online Account Safely: Sign Out vs Unlink vs Delete12 min read →Web DevelopmentGetting Started with Next.js: A Beginner-Friendly App Router Guide14 min read →Web DevelopmentMaster React Step by Step: From Components to Hooks and Real Projects13 min read →