Free LLMs & agents lessons
Tokens, embeddings, attention, and residual computation
A tokenizer maps text fragments to discrete IDs. Embeddings map IDs and positions to vectors. Self-attention lets each position form a weighted mixture of information from permitted earlier positions; feed-forward layers transform each position; residual connections carry and refine representations through many blocks. A final projection produces next-token probabilities.
Each position produces a query; earlier positions produce keys and values. Query–key similarity determines normalized weights, and the weighted values deliver context. Multiple heads can learn different interaction patterns. Attention does not by itself guarantee human-readable reasoning or faithful attribution.
Pretraining, adaptation, inference, and limits
Pretraining compresses patterns from large corpora into parameters by reducing prediction loss. Instruction tuning and preference-based methods shape responses toward desired interaction. At inference, decoding policy controls which likely continuation is produced. None of these steps automatically verifies claims against the current world or exposes a calibrated confidence.
Logits rank candidate tokens; temperature rescales how concentrated the sampling distribution is. Lower values usually make output more repeatable; higher values increase diversity. A low temperature does not turn unsupported content into verified fact, and deterministic decoding can consistently return the same mistake.
Retrieval, tool contracts, and untrusted content
Retrieval finds candidate evidence; ranking and filtering decide which context enters the prompt; generation synthesizes an answer. Tool use lets the model propose structured calls to systems that can calculate, search, or change state. The application must validate arguments, authorize the user, separate instructions from untrusted content, and render provenance.
Did the retriever return evidence that contains the answer? Did the generator use that evidence faithfully? Recall-oriented retrieval, reranking, chunk boundaries, metadata filters, and freshness affect the first. Citation entailment, abstention, and answer constraints affect the second. One end score can hide which half failed.
State, planning, permissions, evaluation, and stop conditions
The system observes state, selects or proposes an action, executes through bounded tools, records the result, and decides whether to continue. Long tasks need explicit state rather than relying on a prompt transcript. Budgets, allowlists, approvals, sandboxing, idempotency, and stop conditions contain mistakes. Evaluation must score final outcomes and the path taken.
A support agent may reach the right final answer after leaking data, calling unnecessary tools, or repeatedly failing. Tests should include task completion, groundedness, authorization, action count, cost, latency, recovery, adversarial inputs, and correct refusal. Production traces then reveal new failure clusters for the suite.