A developer on my team once spent most of a day debugging why a summarisation endpoint kept dropping the end of long documents. The code was fine. The document was 60,000 words, the model's context window was smaller than that, and the library was silently truncating the input from the top. There was no error. The output just quietly described only part of the document.
That is the kind of bug you get when you use a language model as a black box. You do not need to understand transformer internals to build with these things, but there are about five concepts that, once they click, turn most mysterious behaviour into obvious behaviour.
Here they are, from the perspective of someone who integrates these models rather than trains them.
A Model Predicts the Next Token, and That Is Genuinely All
The core loop is unglamorous. Given a sequence of tokens, the model produces a probability distribution over what token comes next. One is picked. It gets appended to the sequence. The whole thing runs again.
Everything else — the apparent reasoning, the code generation, the personality — emerges from that loop running at enormous scale over a model trained on an enormous amount of text. There is no lookup, no database of facts, no plan being followed. When a model states a wrong version number with total confidence, it is not lying. It generated the sequence of tokens that was most plausible given everything it had seen, and plausible is not the same as true.
Understanding this makes the failure modes predictable rather than spooky. Models are unreliable about specific facts they saw rarely, excellent at patterns they saw constantly, and completely incapable of knowing what they do not know.
Tokens: The Unit That Actually Bills You
Models do not read characters or words. They read tokens — chunks of text, typically three to four characters in English. "Understanding" might be two tokens; a rare technical term might be five; a Urdu or Arabic sentence will use noticeably more tokens per word than the English equivalent, because the tokenizer was mostly fitted to English.
This matters commercially, because you are billed per token in both directions, and it matters practically, because it explains a class of odd behaviour. Ask a model to count the letters in a word and it often gets it wrong — it never saw the letters, it saw two or three tokens. Ask it to reverse a string and you get the same problem.
Rough rule for planning: 750 English words is about 1,000 tokens. Count before you send, not after the API rejects you.
The Context Window Is Working Memory, Not Storage
The context window is how many tokens the model can attend to in one request — system prompt, conversation history, retrieved documents, your question and the answer being generated, all sharing the same budget.
Two misconceptions cause most of the trouble. The first is that a conversation has memory. It does not. Every turn resends the entire history, which is why a long chat gets slower and more expensive as it goes, and why the fifteenth message costs several times what the first one did. The second is that a large advertised window means uniform attention across it. Models reliably attend better to the beginning and end of their context than the middle. Bury the critical instruction in the centre of a 100,000-token prompt and you may find it ignored.
Practical consequences: put instructions at the start or the very end, not the middle. Trim conversation history deliberately rather than letting it grow. And never let a library silently truncate for you — check the token count and decide what to drop yourself.
Temperature and the Rest of the Dials
Temperature controls how the next token is picked from the probability distribution. Low values make the model consistently choose the highest-probability option; higher values let it sample more freely.
The practical guidance is simpler than the theory. If there is a correct answer — extraction, classification, code, anything with a schema — use a very low temperature. If you want variety — copy variations, brainstorming, creative text — raise it. Most production endpoints I build sit near zero, because most production tasks have a right answer and you want yesterday's input to produce today's output.
Worth knowing: low temperature is not the same as deterministic. Even at zero, most hosted models will occasionally produce different output for identical input, because of batching and floating-point non-determinism on the serving side. Do not build a system that assumes byte-identical responses.
Fine-Tuning: Mostly Not the Answer
Fine-tuning is the request I push back on most often, because it is usually being proposed to solve a problem it does not solve.
Here is the distinction that matters. Fine-tuning teaches a model a behaviour or a style. It does not reliably teach it facts. If you want a model that always replies in your company's tone, always returns a particular JSON shape, or handles a specialised classification task with fewer tokens per call, fine-tuning is a good fit. If you want the model to know your product documentation, fine-tuning is the wrong tool and retrieval is the right one — and the failure is ugly, because a fine-tuned model produces confident, fluent, invented answers in exactly the right style.
The order I recommend: get the prompt right, add examples to the prompt, add retrieval if the problem is knowledge, and only then consider fine-tuning if you have a genuine behavioural requirement and a few hundred high-quality examples. Each step is cheaper and more reversible than the next.
And remember the maintenance cost. A fine-tuned model is a frozen artefact. When the base model improves in six months, your fine-tune does not come along for free — you re-run the whole process. That is a real ongoing commitment, not a one-off.
Open-Source Models: When Self-Hosting Makes Sense
The open-weight ecosystem — Llama, Mistral, Qwen, Gemma and the rest — has become genuinely capable. Not equal to the top proprietary models at the hardest reasoning tasks, but well past the threshold for a large amount of practical work, and improving on a schedule that keeps surprising people.
I recommend self-hosting when at least one of these is true:
The data cannot leave. Regulated industries, government work, or a client contract that forbids sending content to a third party. This is the most common genuine reason.
The volume is large and the task is narrow. Classification or extraction running millions of times a month, where a small tuned model on your own hardware beats per-token pricing by a wide margin.
You need stability. Hosted models get deprecated and updated underneath you. A model you host does not change until you change it, which matters if you have validated behaviour for compliance reasons.
Latency and locality. Running close to the user or on-premise, where a round trip to a provider's region is too slow.
What people underestimate is the operational weight. You are now responsible for GPU capacity, batching, queueing, an inference server, model updates and someone who understands all of it at 2am. For a small team with moderate volume, the hosted API is usually cheaper once you count the engineering time honestly. I have talked more clients out of self-hosting than into it.
The Mental Model That Helps Most
Treat the model as a very well-read contractor with no memory and no access to your systems. Extremely capable at anything that is essentially a language task. Completely dependent on what you put in front of it. Confident even when wrong, so the checking is your job.
Every design decision follows from that. Retrieval exists because the contractor has not read your documents. Tool calling exists because they cannot reach your database. Structured output and validation exist because they will occasionally hand you something in the wrong format. Evals exist because you cannot tell whether their work got worse without checking.
None of this requires understanding attention heads. It requires remembering that behind the conversational surface is a function that predicts tokens, and building accordingly.
