RAG Interview Questions and Answers

RAG finds useful information in an outside knowledge source and adds it to the prompt. The LLM then answers using that information.

Flow has two parts.

• Index: load documents, split them into chunks, embed the chunks, and store them in a vector index.
• Query: embed the question and get the top chunks. Build a prompt with the chunks. Generate the answer, and optionally cite sources.

Why use it: the model gets current and private knowledge without retraining. Hallucinations (made-up answers) drop because answers can point to sources. You can apply access control at retrieval time. To update knowledge, you only re-index.

Limits: answer quality depends on retrieval quality. Context length and cost still apply.

What they are checking: Whether you understand the full flow and the reasons to choose RAG

1. Ingestion: load documents (PDF, HTML, databases), clean them and extract the text. Keep metadata (source, date, permissions).
2. Chunking: split the text into passages of a good size.
3. Embedding and indexing: turn chunks into vectors and store them in a vector database. Add a keyword index if you use hybrid search.
4. Retrieval: embed the query and fetch the top-k candidates. You can also filter by metadata.
5. Reranking: reorder the candidates with a stronger model.
6. Augmentation: build the prompt with instructions and the best chunks.
7. Generation: the LLM answers and cites sources.
8. Evaluation and monitoring: track retrieval quality and answer quality.

Most RAG failures come from steps 1 to 5, not from the LLM.

What they are checking: Whether you can describe the whole pipeline and say where failures start

Chunking splits documents into small passages. Each passage is embedded and retrieved on its own.

• Too small: a chunk loses its context. Retrieval returns fragments that cannot answer the question.
• Too large: the embedding becomes blurry and noisy. You waste context tokens, and the useful details get buried.

A common start is a few hundred tokens with 10 to 20 percent overlap. Then tune it on your evaluation set. Overlap keeps ideas that sit across a boundary. Split on natural structure (headings, paragraphs, sentences) instead of fixed character counts. Store metadata such as title and section with each chunk.

What they are checking: Whether you know the trade-offs and can tune using retrieval metrics

A simple LLM call maps a prompt to a response. A chain runs a fixed list of steps (for example retrieve, then summarize, then format). An agent lets the model decide what to do next in a loop. It can choose tools, look at the results, update its plan and continue until the task is done or a limit is reached.

Agents give more flexibility for open tasks (research, debugging, multi-step work). But they also add cost, delay, unpredictable results and risk. Prefer a simple workflow when the steps are known in advance. Use an agent only when the path really depends on earlier results.

What they are checking: Whether you can judge when an agent is needed

Keyword search (BM25, SQL LIKE, full-text) matches the actual words in the query. It is exact for specific terms but misses synonyms and paraphrases. Semantic search embeds the query and the documents and finds the closest meaning. So 'cheap flights' can match 'low-cost airfare'. But it can miss exact identifiers and rare terms.

• Cosine similarity: compares direction and ignores length. It is the common default for text embeddings.
• Dot product: the same as cosine if the vectors are normalized (scaled to length 1), and it is faster. Use it if the model was trained for it.
• Euclidean (L2) distance: straight-line distance.

MCP is an open protocol. It sets one standard way for AI applications to connect to outside tools and data. An MCP server offers tools, resources (data) and prompts. An MCP client lives inside the host application (an assistant, IDE or agent). The client finds the server's tools and calls them over a transport such as stdio or HTTP.

Both are frameworks for building LLM applications.

A small example with plain NumPy shows the idea. embed is any function that returns a vector from your embedding model.

Top-k is the number of most similar chunks that the index returns for a query. A small k may miss the needed evidence (low recall). A large k adds noise, cost and delay, and can trigger the lost-in-the-middle effect.

• Fixed-size with overlap: the simple baseline.
• Recursive character or token splitting: tries paragraph, then sentence, then word boundaries, so units stay whole.
• Structure-aware: split by headings, markdown sections, HTML tags, code functions or table rows.
• Semantic chunking: split where the embedding similarity between neighbouring sentences drops.
• Parent-child (small-to-big): embed small chunks for exact matching, but give the larger parent section to the LLM.
• Contextual chunks: add a short summary or the document title to each chunk before embedding.

Choose by document type (code, legal, tables, chat logs) and measure. Compare the strategies on a labelled question set using recall@k and answer quality. Tables and PDFs often need special parsing before any chunking.

What they are checking: Whether you can match the strategy to the document type and check it with metrics

Hybrid search combines keyword search (BM25) with semantic vector search. It then merges the two result lists, often with Reciprocal Rank Fusion.

Why: embeddings are good at paraphrases and intent. But they can miss exact tokens such as product codes, error messages, names and rare terms. BM25 is very good at exact matches but misses synonyms. Together they find more of the right results across different query types.

Reciprocal Rank Fusion score: sum(1 / (k + rank_i)) across the result lists, with k often 60. It needs no score normalization. Many engines support it (Elasticsearch, OpenSearch, Weaviate, Qdrant, PostgreSQL with full text plus pgvector). Add a reranker after it for better precision.

What they are checking: Whether you know why keyword and semantic search help each other, and how to merge them

A reranker takes the top candidates from the first retrieval step (for example the top 50). It scores each query and passage pair again with a more accurate model. This is usually a cross-encoder, which reads the query and the passage together. Then it keeps only the top few (for example 5) for the prompt.

The first step uses bi-encoders. It is fast but rough, because the query and the document are embedded separately. Rerankers are slower, so they run only on a small set of candidates.

Add one when the right chunk is in the top-k but not at the top. Also add one when the context budget is tight, or when quality matters more than a few hundred milliseconds. Measure the gain in recall@k, MRR and final answer quality against the extra latency and cost.

What they are checking: Whether you understand two-stage retrieval and its trade-off

Build a set of questions with known relevant documents or chunks. Then measure:

• Recall@k / hit rate: is a relevant chunk in the top k?
• Precision@k: what share of the retrieved chunks are relevant?
• MRR: the reciprocal rank of the first relevant result.
• nDCG: gives a higher score when relevant items rank higher.
• Context relevance: is each retrieved chunk useful? This is often judged by an LLM.

To build the set, label real queries by hand. Or generate synthetic questions from chunks and review them. Evaluate retrieval separately from generation, so you know which part to fix. Run the tests again after every change to chunking, embeddings or ranking.

What they are checking: Whether you know the retrieval metrics and how to build the test set

Common things to check:

• Faithfulness (groundedness): is every claim supported by the retrieved context?
• Answer relevance: does the answer address the question?
• Completeness and correctness: compare with a reference answer when one exists.
• Citation accuracy: do the cited sources really support the statement?
• Refusal behaviour: does the system say it does not know when a question has no answer?

Use an LLM as a judge with clear rubrics (tools like Ragas, TruLens or your own scripts). Check the judge against human review on a sample. Include hard cases, such as questions with no answer in the corpus. Track results over time as regression tests. Watch cost and latency too.

What they are checking: Whether you can tell faithfulness from relevance and handle questions with no answer

• Relevant document not retrieved: causes are poor chunking, weak embeddings, no hybrid search or a vague query. Fix it with better chunks, hybrid retrieval, query rewriting and a tuned k.
• Relevant chunk retrieved but ranked low: add reranking.
• Context too long or noisy: use fewer, better chunks. Compress the context. Reorder so key chunks are not lost in the middle.
• Model ignores or contradicts the context: use stronger grounding instructions, quote-then-answer prompts or a verification step.
• Outdated or conflicting documents: add version and date metadata, remove duplicates and use freshness filters.
• Bad parsing of tables, PDFs and images: use dedicated extraction.
• Unanswerable questions answered anyway: add a refusal instruction and test it.

Find the cause with logs of the question, the retrieved chunks and the answer.

What they are checking: Whether you can separate retrieval problems from generation problems

ReAct (Reason + Act) mixes reasoning with tool use. The loop is:

1. Thought: the model reasons about what it needs.
2. Action: it calls a tool with arguments.
3. Observation: the tool result is sent back to the model.
4. Repeat until the model gives a final answer.

You give the model a list of tools. Each tool has a name, a description and JSON-schema parameters. The model returns a structured call. Your code checks the arguments, runs the function and sends the result back. The loop then continues.

Add hard limits in code, not only in the prompt:

A vector database indexes embeddings so it can find the nearest neighbours fast. Without an index, it would compare the query to every vector (exact search is O(n)).

User questions are often vague, chatty, or depend on earlier turns ('what about the second one?'). Query transformation improves retrieval:

Store metadata (source, date, department, tenant, permissions) with each chunk. Apply filters at query time inside the vector search, so the engine only looks at allowed or relevant chunks. Examples: limit results to the current customer's tenant, a document version, or a date range.

A multi-agent system uses several LLM agents. Each agent has its own role, prompt or tools, and they work together. For example: a planner, a researcher, a coder and a reviewer. An orchestrator (a controller agent) or handoffs coordinate them.

• Short-term memory: the conversation and tool history inside the context window. Control its growth by trimming or summarizing.
• Long-term memory: facts and preferences saved outside the model (in a database or vector store) and retrieved when needed.
• Working state or scratchpad: a plan or notes that the agent keeps between steps.
• Episodic memory: records of past tasks and results, kept for reuse.

Evaluate the results and the process.

Track document versions and content hashes. When a document changes, re-ingest only the changed documents. Delete or replace their old chunks. Use a stable document id in the chunk metadata for this. Also handle deletions, so removed content is not served.

Models tend to use information at the start and end of a long context better than information in the middle. In RAG, a relevant chunk in the middle of many passages may be used less.

A workflow follows paths set in code (nodes and edges). The LLM fills in steps such as classification or generation. An agent lets the LLM choose the path and tools while it runs.

• Sparse (BM25, SPLADE): term-based vectors. Strong at exact words and rare terms. Easy to explain.
• Dense (bi-encoder embeddings): one vector per passage that captures meaning. Good at paraphrases.
• Late interaction (ColBERT style): keeps one vector per token. It scores query tokens against document tokens. This matches more finely than a single vector, but needs more storage.

GraphRAG builds a graph of entities and relationships from the corpus, often using an LLM. It uses the graph to retrieve connected information or community summaries.

Ingestion: use connectors for the wiki, drive and PDFs. Parse with layout-aware extraction so tables stay correct. Clean the text. Chunk by headings with overlap. Attach metadata (source URL, owner, updated date, access groups). Embed the chunks and store them in a vector index, with a keyword index for hybrid search. Sync when documents change, including deletions.

Check each stage, using logs for the failing questions:

Use a limited agent with clear tools.

Tools: get_order(order_id), check_refund_eligibility(order_id) (the policy lives in code), issue_refund(order_id, amount), search_help_articles(query) (RAG), handoff_to_human(summary).

Pure embeddings often blur exact identifiers. Add hybrid search with BM25 so exact tokens match. Check that the analyzer or tokenizer keeps codes whole (no splitting on hyphens).

Read the trace to see why. The tool may return an error or an empty result that the model misreads. The tool description may be unclear. Or the goal may be impossible.

• Cache answers for frequent questions (exact match and semantic match). Cache embeddings too.
• Retrieve fewer, better chunks with reranking, to make prompts shorter.
• Send easy queries to a small model and hard ones to a larger model.
• Use prompt caching for the fixed instructions.
• Choose cheaper embedding...

The raw follow-up does not have enough context to retrieve well. Add a question condensation step. Use the chat history to rewrite it as a standalone query ('What are the features of the premium plan?'). Then retrieve with the rewritten query.

Assume that any document may contain hostile instructions. Controls:

• Separate instructions from retrieved content with clear delimiters. Say that the retrieved text is data.
• Strip or flag text that looks like instructions.
• Limit what the model can do. In the RAG path, give it no tools, or only read-only ones.
• Prevent data theft. Do not render arbitrary external URLs or images.
• Apply output filters.
• Restrict who can add documents to the index.

I would write the prompt first, with a clear role, the allowed labels, an output format and one example. Then I would wrap the model call in a small function, so the provider can be swapped.

Make each step a small function with one prompt and one clear output. Then join the steps in order. Frameworks such as LangChain express this as a chain. In current LangChain, steps are joined with the pipe operator: prompt | model | parser. But the structure is just function composition, so I would first show it in plain Python.

Plan, gather, then write.

Invest in parsing before chunking.