Skip to content
KR7Tech

Learning guide preview

How AI, RAG, and agents work

Build understanding of LLM fundamentals, retrieval-augmented generation, agent workflows, and responsible implementation.

Sign in and explore Pro Back to KR7Tech Student Plan

Enter the PIN separately for each Pro question or lesson. Opening one item does not unlock other items or topics.

How AI, RAG, and agents work

10 free of 92 questions / lessons

Example request: “What failed in last night’s pipeline?”

The user asks one sentence, but the system performs several hidden steps.
User → Copilot/App → Instructions → Context/Search → GPT or Claude → Response
Step 1: Receive the prompt. The application collects the user’s question, conversation context, identity, and system instructions.
Step 2: Decide what is needed. If the question can be answered from supplied context, the model generates directly. If current evidence is required, the application searches or uses a tool.
Important: GPT and Claude are model families. Copilot is an assistant or product experience that can use models together with work data, search, tools, and safety controls.

How the model creates the text

A model does not write the full answer in one operation.
Prompt → Tokens → Attention → Next-token probabilities → Token-by-token answer
Example: After reading “The pipeline failed because…”, the model estimates which next token is most suitable, selects it, and repeats until the response is complete.
Result: The response can sound natural because the model learned language patterns, but fluency alone does not prove correctness. Grounding and validation provide evidence.

GPT vs Claude vs Copilot

People often compare items from different layers.
Model: GPT / Claude → Application: Copilot / Agent → Data and Tools
GPT: A model family from OpenAI used for understanding and generation.
Claude: A model family from Anthropic used for understanding and generation.
Copilot: A Microsoft assistant experience that combines a model with prompts, grounding, search, applications, and enterprise controls.
Simple memory: Model = engine. Copilot = complete car. Agent = a car that can also choose a route and perform approved work.

1 · The application builds the actual prompt

The visible text box is often only one piece of model input.
System/developer instructions + conversation + retrieved evidence + user message + tool definitions → serialized model input

A chat application may add system or developer instructions, recent conversation turns, summaries of older turns, retrieved passages, output-format rules, and tool schemas. It serializes these messages into the model's expected format. Some APIs also include image, audio, or other non-text inputs. The model only receives what the application sends within the request; an application may keep other data outside the model.

The context window is a finite token budget shared by the input and, depending on the service, generated output. If history grows too large, the product may truncate it, summarize it, or reject the request. That is why a long conversation is not guaranteed to be perfectly remembered.

Debugging question: What exact messages, retrieved evidence, tool definitions, and token limits reached the model? Log a privacy-safe request shape and token counts—not secrets or sensitive prompt contents.

2 · Tokenization and embeddings

The network processes numerical representations, not words as humans see them.
Text → tokenizer pieces → token IDs → learned embedding vectors

A tokenizer splits text into model-specific pieces. A token can be a whole common word, part of a word, punctuation, or whitespace; token count is not the same as word count. Each token ID maps to a learned vector. Position information is also represented so the model can distinguish order and relative location in the sequence.

Different tokenizers can encode the same text into different numbers of tokens. Code, uncommon names, and some languages can use more tokens than expected. More input tokens generally mean more prefill work, more context memory, and higher usage cost for metered APIs.

Practical impact: Keep instructions clear and context relevant. Repeating large policy text in every agent round wastes input budget unless the serving system can safely reuse cached prefixes.

3 · What attention does inside a Transformer

Each position builds a context-aware representation using other allowed positions.
Hidden vectors → Query / Key / Value projections → attention weights → mixed context

In a typical self-attention layer, learned projections produce query, key, and value vectors. A position compares its query with keys at permitted positions to calculate relevance weights, then combines their values. Multiple attention heads learn different relationships. A causal mask prevents a next-token model from using future answer tokens that it has not generated yet.

Transformer blocks also use feed-forward (MLP) layers, residual paths, normalization, and learned weights. Repeating these blocks transforms token representations into richer contextual features. Attention is not a literal human thought process or a guarantee that the model understands facts; it is a mathematical operation inside a trained network.

Memory: Attention mixes context; MLP layers transform each position's features; residual and normalization operations help deep networks train and operate.

4 · From hidden state to the next-token choice

Generation repeatedly ranks possible next tokens, then applies a decoding policy.
Final hidden state → vocabulary logits → sampling/selection → token appended

The output head maps the current representation to a score (logit) for each token in the model's vocabulary. A decoding method converts those scores into a next-token choice. Greedy decoding takes the highest-scoring option; sampling can use temperature, top-p, or related controls to vary the choice. These controls affect output variability; they do not add knowledge or verify correctness.

The chosen token is appended to the context and the model predicts again. Generation ends when it emits a stop token, reaches a configured limit, or the service stops the request. The result is decoded back into text. A tool-call response is still model output: the application must validate the requested tool, arguments, permissions, and result before continuing.

Conceptual pseudocode · not a provider SDK
tokens = tokenize(messages)
state = transformer_prefill(tokens)

while not stopped:
    logits = output_head(state.last_position)
    next_token = decode(logits, policy)
    if is_stop(next_token):
        break
    tokens.append(next_token)
    state = transformer_decode_next(tokens, cached_state)
Important: A fluent sentence is not proof of truth. The next-token objective rewards plausible continuation, not a built-in fact-check. Use trusted retrieval, tool verification, structured constraints, and evaluation when correctness matters.

5 · Why the first token and the rest have different latency

Prompt processing is parallelizable; autoregressive output has a sequential dependency.
Prefill input context → first-token latency → decode token 1 → decode token 2 → …

During prefill, the model processes the supplied prompt to build representations and attention state. Hardware can parallelize much of this work across input positions, although a longer context still takes more compute and memory. The service then produces the first output token; user-perceived time-to-first-token (TTFT) also includes network, queueing, routing, and service overhead.

During decode, each new token depends on previous generated tokens, so output proceeds step by step. A KV cache stores reusable keys and values from earlier positions so the model need not recompute the full prefix each time. It saves repeated work, but the cache consumes memory and each token still requires model computation. Long input and long output therefore affect different portions of total latency.

Measure separately: time-to-first-token, output tokens per second, total latency, queue time, input/output token counts, and retry/tool time. “Fast” can mean quick first response, high generation throughput, or low end-to-end time; these are not interchangeable.

6 · Why one model feels fast and another reasons more deeply

Speed and reasoning are product/model trade-offs—not a single “smartness” switch.
Latency ≈ queue + prefill + decode + network + tools/retries

A model can feel faster because it is smaller, uses fewer or more efficient active computations, runs on faster or closer hardware, has an optimized serving stack, receives a shorter prompt, emits fewer tokens, or is less busy. Quantization, batching, speculative decoding, caching, routing, and streaming can also change the measured experience. Providers do not always publish enough details to identify which factor dominates.

A fast model may show less reasoning depth if it uses a smaller capacity, has been tuned for concise answers, receives less inference-time thinking budget, or simply answers before exploring enough alternatives. A reasoning-focused model may take longer because it runs additional internal computation, generates more tokens, uses more complex decoding, or calls tools. More reasoning is not automatically more correct; extra steps can still make mistakes.

Model-name caution: “Astra” is not enough to identify a specific model and deployment. Exact speed and reasoning behavior depend on its provider, version, settings, region, prompt, load, and benchmark. Compare the exact model IDs under the same task and conditions rather than inferring internals from a name.
Choose with evaluations: Test representative easy, multi-step, coding, grounded, and no-answer tasks. Compare correctness, latency percentiles, token use, tool success, and cost—not one demo answer.

7 · Why “reasoning level low” can happen

A model cannot reason over evidence, steps, or budget it never received.
Ambiguous task + missing context + small budget + weak evaluation → shallow/incorrect answer

Common causes include an underspecified question, missing constraints, irrelevant or incomplete context, a small output limit, a model optimized for quick responses, low reasoning effort settings, or a request that does not ask for a structured explanation. In a multi-step task, the model may also need retrieval, calculation, code execution, or another tool rather than more prompt wording.

Improve the task contract: state the goal, assumptions, constraints, evidence sources, expected format, and what to do when information is missing. For critical calculations, use deterministic code; for current facts, retrieve trusted sources; for consequential actions, validate and require approval. Increasing the token budget can allow a longer response but does not guarantee better reasoning.

Prompt structure · guide the work, do not demand hidden chain-of-thought
Goal: Compare these two API designs for our stated workload.
Context: [trusted requirements and measured traffic]
Constraints: p95 under 200 ms; preserve tenant isolation.
Task: Identify assumptions, compare failure modes, and recommend one option.
Evidence: Cite the supplied requirement IDs; say when evidence is missing.
Output: Decision, trade-offs, risks, and validation plan.
Do not invent measurements or claim a test was run.
Evaluation over vibes: Score answer correctness, evidence use, coverage of constraints, calibration, latency, and cost on a repeatable test set. Do not rely on the model exposing private reasoning traces as proof of correctness.
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only
Pro only