Englishবাংলা
SubscribeWrite
Latest
Binary search: patterns, templates and common pitfallsDocker's $250 Cloud Sandboxes credit: what it covers and what it does notTwo pointers and sliding window: patterns with examplesArrays and hash maps: the interview patterns you must knowBig-O notation explained: time and space complexity for interviewsSoftware testing basics: the test pyramid and what to testClean code and code review habits that make you a better developerREST API design best practices: URLs, status codes and errors

How large language models work: tokens, context and sampling

LLMs predict the next piece of text, and that simple idea explains both their power and their mistakes. Learn the concepts every builder needs.

••3 min read
Text split into tokens flowing through a language model

A large language model (LLM) is a program trained on huge amounts of text to predict what text comes next. From that single skill come chat, translation, summaries and code. Knowing the basics helps you use them well and avoid surprises.

Tokens

Models do not read letters or words. They read tokens: pieces of text, often a word or part of a word.

  • "unbelievable" might be split into several tokens; a common word like "the" is usually one.
  • Roughly, in English, a token is about three quarters of a word. Other languages, including Bengali, often need more tokens per word, so the same meaning can cost more and fit less in the context.
  • You are billed by tokens, and limits are in tokens. Different models use different tokenizers.

Next-token prediction

Given the text so far, the model produces a probability for every possible next token, picks one, adds it to the text and repeats. A reply is built one token at a time.

It does not "look up" facts in a database. It generates text that is likely given what it learned and what you gave it. This is why it can be fluent and wrong at the same time.

How they are trained, in short

  1. Pre-training: learn language patterns from a very large body of text.
  2. Fine-tuning and instruction tuning: train on examples of following instructions and answering helpfully.
  3. Preference tuning (such as RLHF): use human or AI feedback to prefer better, safer answers.

The model's knowledge stops at its training cutoff date. It does not know later events unless you give it tools or documents.

The context window

The context window is how much text the model can consider at once: your instructions, the conversation, any documents you include and its own reply, all counted in tokens.

  • It has no memory between separate conversations unless the application stores and re-sends information.
  • When the window is full, older content must be dropped or summarised.
  • Very long contexts can still be used unevenly. Important details buried in the middle can be missed, so put key instructions clearly and keep prompts focused.
  • Longer prompts cost more and respond slower.

Sampling settings

  • Temperature: low values make output more predictable and focused, high values more varied and creative. Use low for extraction and code, higher for brainstorming.
  • Top-p and top-k: limit the choice to the most likely tokens.
  • Max tokens: the cap on reply length.
  • Stop sequences: text at which the model stops.

Even at low temperature, results are not guaranteed to be identical every time.

Hallucinations

A hallucination is a confident answer that is false: invented facts, fake quotes, non-existent functions or made-up links. It happens because the model optimises for plausible text.

Reduce it by:

  • Giving the model the source material (see retrieval-augmented generation in this series).
  • Asking it to say when it does not know, and to quote the supporting text.
  • Using tools for facts that must be exact: calculators, databases, search.
  • Checking important outputs yourself or with automatic tests.

What LLMs are good and bad at

Good: summarising, rewriting, translating, classifying, extracting structured data, explaining, drafting and writing code.

Weaker: exact arithmetic, precise counting, up-to-date facts, long multi-step logic without care, and knowing when they are wrong. Treat the output as a draft to verify, especially for medicine, law, finance and security.

Models, tools and agents

Modern systems add to the model:

  • Tool use (function calling): the model asks the app to run a function (search, database query, calculator) and uses the result.
  • Multimodal input: images, audio or documents as well as text.
  • Agents: loops where the model plans, uses tools and checks results. Powerful, but they need guardrails.

Practical advice

  • Be specific in your instructions and give examples.
  • Keep prompts as short as the task allows to save cost and time.
  • Test on real examples from your own use case, not just demos.
  • Never send secrets or private customer data to a service you have not checked for privacy and retention terms.

Takeaway: an LLM predicts the next token from its context. That makes it very flexible but not a source of truth, so give it good context, use tools for exact facts and verify what matters.

AI & LLM series, Part 1 (building with language models): 1 of 5.

Keep reading