Scenario based AI Engineer interview questions

I would look for a mismatch between offline and live conditions.

1. Define the problem: what counts as churn, how far ahead you predict, and what action the business takes. This sets the cost of false positives (a wasted offer) and false negatives (a lost customer).
2. Split by time, not randomly. Build features only from data before the prediction date. This prevents...

I would explain the trade-off. Pushing recall toward 100% means flagging many more good transactions. This hurts customers and costs reviewer time. Instead, agree on a target based on cost.

This is overfitting from epoch 10 onward. What I would do:

Watch the inputs, the predictions and the outcomes.

Loan decisions are regulated and affect people. So interpretability and fairness matter as much as accuracy.

With far more features than samples, overfitting is the main risk.

Suppose the same user (or patient, or device) appears in both train and test rows. The model can memorize patterns of that user. Then the score is too high for new users. Use group-aware splits (GroupKFold) when you must predict for new entities. Use time-based splits when you must predict future behavior. For cold-start problems, hold out whole users or items in validation. Make the split match how the model will be used in production. A random split answers a different question than the real one.

1. Reproduce and measure: collect the failed conversations into an evaluation set with the correct policy answers.
2. Ground it: retrieve the current policy text. Tell the model to answer only from it and to quote the clause. It should say it does not know when the text does not cover the case.
3. **Check...

First measure. Use token logs to break cost down by feature, prompt, user and model. Find the biggest spenders and unusual cases, such as very long inputs, retry loops and agent runs.

First separate random variation from real ambiguity.

Treat model changes like dependency upgrades.

Use hierarchical (map-reduce) summarization.

Give the model the schema (tables, columns, types, short descriptions and a few example queries), not the data. For large schemas, retrieve only the relevant tables.

First find where the time goes. Log retrieval time, prompt size, time to first token and total generation time.

Use layers.

• Add moderation classifiers for input and output (provider moderation APIs or your own classifier). Block or rewrite flagged content with a safe fallback message.
• Set clear policies in the system prompt, but enforce them in code.
• Log flagged cases for review and add them to the evaluation set.
• Test...

Ingestion: use connectors for the wiki, drive and PDFs. Parse with layout-aware extraction so tables stay correct. Clean the text. Chunk by headings with overlap. Attach metadata (source URL, owner, updated date, access groups). Embed the chunks and store them in a vector index, with a keyword index for hybrid search. Sync when documents change, including deletions.

Check each stage, using logs for the failing questions:

Use a limited agent with clear tools.

Tools: get_order(order_id), check_refund_eligibility(order_id) (the policy lives in code), issue_refund(order_id, amount), search_help_articles(query) (RAG), handoff_to_human(summary).

Pure embeddings often blur exact identifiers. Add hybrid search with BM25 so exact tokens match. Check that the analyzer or tokenizer keeps codes whole (no splitting on hyphens).

Read the trace to see why. The tool may return an error or an empty result that the model misreads. The tool description may be unclear. Or the goal may be impossible.

• Cache answers for frequent questions (exact match and semantic match). Cache embeddings too.
• Retrieve fewer, better chunks with reranking, to make prompts shorter.
• Send easy queries to a small model and hard ones to a larger model.
• Use prompt caching for the fixed instructions.
• Choose cheaper embedding...

The raw follow-up does not have enough context to retrieve well. Add a question condensation step. Use the chat history to rewrite it as a standalone query ('What are the features of the premium plan?'). Then retrieve with the rewritten query.

Assume that any document may contain hostile instructions. Controls:

• Separate instructions from retrieved content with clear delimiters. Say that the retrieved text is data.
• Strip or flag text that looks like instructions.
• Limit what the model can do. In the RAG path, give it no tools, or only read-only ones.
• Prevent data theft. Do not render arbitrary external URLs or images.
• Apply output filters.
• Restrict who can add documents to the index.

I would write the prompt first, with a clear role, the allowed labels, an output format and one example. Then I would wrap the model call in a small function, so the provider can be swapped.

Make each step a small function with one prompt and one clear output. Then join the steps in order. Frameworks such as LangChain express this as a chain. In current LangChain, steps are joined with the pipe operator: prompt | model | parser. But the structure is just function composition, so I would first show it in plain Python.

Plan, gather, then write.

Invest in parsing before chunking.