#AI With things heating up like this, I bet many folks still don't get how it works behind the scenes.
Why when we type a piece of text does AI generate corresponding results?
Let’s break down the complete flow from input to output, which is what we call the inference stage, because that's what's really happening when we use AI daily. Here’s an overview of the whole process (7 major steps).
You input a question (Prompt)
Tokenization
Embedding
Transformer layer processing (core computation)
Output logits → sample the next word
Repeat step 5 until generation completes
Decoding (Detokenization) → outputs text for you
Let’s explain in detail step by step, including the principles of each step and what it’s doing.
1. You input your question (Prompt)
You type: “How does the whole AI process work?”
The system combines your input + past dialogue + system prompts into a complete prompt sequence. This step is just text preparation; computation hasn’t started yet.
2. Tokenization
AI doesn’t understand Chinese characters or English words—it only understands numeric IDs (called Token IDs).
A tokenizer splits the text into tokens.
Example: “the whole process of AI” → might be split into ["AI", "the", "whole", "process", "is", "how", "it" ...] (in practice, it’s more fine-grained).
Common English words like “beautiful” may be split into “beauti” + “ful.”
Each Token corresponds to a numeric ID (e.g., “AI” = 12345).
The principle: based on algorithms like BPE (Byte Pair Encoding) or WordPiece, balancing vocabulary size and coverage.
Result: your input becomes a sequence of numbers, like [12345, 6789, 23456, ...]
3. Embedding
Map each Token ID to a high-dimensional vector (usually 768–2048 dimensions).
For example, the Token “AI” becomes a 768-dimensional floating-point vector, encoding semantic information about “AI.”
The principle: the embedding matrix learned during training (a huge lookup table) makes vectors of words with similar semantics be closer in distance.
Also add positional encoding (Positional Encoding): since Transformers don’t inherently understand order, they add a unique vector to each position using sine/cosine functions, so the model knows “the 1st word, the 2nd word.”
4. Transformer processes layer by layer (core computation). This is the main body of AI’s “brain,” typically with dozens to over a hundred layers (Layers). Each layer mainly does two things:
Self-Attention mechanism
The principle: make every word in the sequence “pay attention” to all the other words, computing the correlation weights between them.
For example, when you ask “Is an apple tasty?” the model needs to know that “apple” refers to the fruit, not a company, so it uses attention to focus on the surrounding context.
Multi-Head Attention: view relationships from different angles (syntax, semantics, logic, etc.).
Feed-Forward Network
At each layer, apply a nonlinear transformation to each position separately to increase the model’s expressiveness.
Each layer passes its output to the next layer, refining semantics layer by layer.
This step accounts for more than 90% of the computation during inference, mostly run in parallel on the GPU.
5. Output logits → sample the next word
After the final layer output, attach a huge linear layer (vocabulary size, usually 50k–100k Tokens).
Map the vector at the last position of the current sequence into a “probability distribution” (logits).
For example, the model thinks the probability of the next word is 30% for “is,” 20% for “how,” 15% for “what,” ...
Sampling strategy:
Greedy: directly pick the word with the highest probability (tends to repeat).
Temperature sampling: higher temperature is more random; lower temperature is more deterministic.
Top-k / Top-p (Nucleus): choose from only the words within the top k or the cumulative probability p (the mainstream approach now).
Select a new Token (e.g., “is”), and append it to the end of the sequence.
6. Repeat steps 4–5: autoregressive generation
Feed the newly generated Token back into the model and recompute starting from step 3 (but optimized with KV caching, so it only computes the part for the new Token).
Keep generating the next word until:
Encounter the end marker (<|end|>).
Reaching the maximum length limit.
Or the model decides to stop on its own (modern models generate an end signal).
That’s why generating text is like “jumping out one word at a time.”
7. Decoding (Detokenization)
Convert the generated Token ID sequence back into text.
Merge subwords (e.g., “beauti” + “ful” → “beautiful”).
Finally, it shows you the output: “The whole AI process mainly includes…”
How does the hardware work together to run it?
GPU/TPU workhorse: Transformer matrix multiplications and attention computations are highly parallel. Tens of thousands of GPU cores are perfectly suited for this.
CPU helps: scheduling tasks, moving data, and doing preprocessing/postprocessing.
VRAM is key: large model parameters are tens of GB to TB, so they must be stored in GPU memory; KV cache also consumes memory, so the bigger the model, the more it uses the GPU.
A simple summary metaphor for the whole process is:
You give the AI a story beginning (Prompt).
The AI turns every character into “semantic building blocks.”
Through hundreds of layers of “attention sieves,” continually predicting what the “next most reasonable building block” is.
Put them down one by one until the story is finished.