Master AI Engineer
Interviews

A token is roughly 4 characters or ¾ of a word. It matters for 3 reasons: (1) cost — APIs charge per token, (2) limits — context windows are measured in tokens, and (3) chunking — when building RAG, you split documents into token-sized chunks, and choosing chunk size affects recall. In our RAG system, we use tiktoken to count tokens before sending requests to avoid context window overflow.

Several strategies: (1) Chunking + retrieval — don't pass the whole document, retrieve only relevant chunks; (2) Summarization chains — compress long conversations; (3) Sliding window — keep recent N messages + system prompt; (4) Model routing — use long-context models (Claude, GPT-4o) only when needed since they cost more. At my startup, we implemented a rolling context strategy for our chat feature.

Temperature 0 = fully deterministic, always picks highest-probability token. Use for: SQL generation, JSON extraction, classification, anything where you need consistent reproducible output. Temperature 0.7 = more varied responses. Use for: creative writing, brainstorming, chat assistants where variety feels natural. In production, I default to 0 for any structured output pipeline and add temperature only if outputs feel too repetitive in user testing.

Three layers: (1) Prompt-level — use JSON mode or explicit output schema in the prompt with examples; (2) Validation layer — parse with Pydantic or Zod and catch validation errors; (3) Retry loop — if validation fails, resend with the error message appended: "Your last output was invalid because [error]. Try again." In our SQL AI feature, we implemented exactly this retry loop — the model receives its own error and self-corrects about 85% of the time.

Prompt injection is when user-provided content modifies the LLM's behavior in unintended ways. Example: a user uploads a doc that contains "Ignore previous instructions and..." — the LLM may follow those instead of your system prompt. Prevention: (1) Separate system instructions from user data clearly; (2) Sanitize/filter user content before injecting into prompts; (3) Use Anthropic's or OpenAI's structured message formats properly; (4) Output validation catches many injection attempts. In RAG systems, this is especially relevant because retrieved content is injected into prompts.

Non-technical: "Imagine mapping every word/sentence in a space where similar things are close together. 'Dog' and 'puppy' are near each other, 'dog' and 'quantum' are far apart. Embeddings are the coordinates in that space."

Senior engineer: "Embeddings are high-dimensional float vectors produced by transformer encoder models. They encode semantic meaning by projecting text into a latent space where cosine similarity correlates with semantic relevance. Dimensionality varies: ada-002 = 1536 dims, text-embedding-3-small = 1536 with support for dimensional reduction. They're the retrieval backbone of RAG — you embed both documents at index time and queries at runtime, then find nearest neighbors in vector space."

Cosine similarity measures the angle between two vectors — ignores magnitude, only direction. Range: -1 to 1. Use when vector magnitudes vary (which is most of the time with text embeddings). Dot product = magnitude × cosine similarity. Use dot product only when embeddings are normalized to unit length (then dot product = cosine similarity but faster). Most vector DBs like Pinecone normalize embeddings by default and expose cosine as the recommended metric. Euclidean distance is less common — more sensitive to magnitude, useful in clustering tasks but rarely the best choice for semantic search.

I'd break it into two phases: indexing and querying.

Indexing: Load documents (PDF parser, HTML scraper, etc.) → chunk into ~512 token segments with ~50 token overlap → embed each chunk using text-embedding-3-small → store in Pinecone/Chroma with metadata (source URL, page number, date).

Query time: Embed the user's query with the same model → cosine similarity search → retrieve top-k (usually 5–10) chunks → build prompt: system instructions + retrieved context + user question → stream LLM response → return with source citations.

I'd add evaluation from day one: RAGAS metrics for faithfulness and context precision, LangSmith for tracing every request to debug issues.

Chunking matters because: (1) embeddings represent one semantic unit — if a chunk has 5 unrelated topics, the embedding is noisy; (2) LLMs have context limits — you can't pass full docs; (3) precision — smaller chunks = more precise retrieval, but less context per chunk.

Chunk size choice: start with 512 tokens with 50-token overlap. Then evaluate: if answers are missing context that exists in adjacent chunks, increase size or increase overlap. If retrieval is imprecise (wrong sections returned), decrease size. Overlap ensures context from boundary sentences isn't lost. For code or structured data, chunk by logical unit (function, section) rather than token count.

RAG fails in 3 main ways:

1. Retrieval fails — wrong chunks returned. Diagnose: log retrieved chunks for failed queries. Fix: better chunking, hybrid search (BM25 + vector), re-ranking.

2. Context is right but answer is wrong — LLM hallucinates despite having the context. Diagnose: check if answer is faithful to retrieved chunks. Fix: lower temperature, add "only answer from the provided context" to system prompt, use faithfulness evaluation.

3. Query-document mismatch — user asks in different language/terminology than docs use. Fix: query expansion (HyDE — generate hypothetical answer and embed that), synonym expansion.

In production, I'd use LangSmith to trace every pipeline step so I can see exactly where the failure occurs.

Hybrid search combines dense vector retrieval with sparse keyword-based retrieval (BM25). You use it when your documents contain specific terms that semantic search misses — product codes, proper nouns, IDs, technical terms. For example, searching "error code ER-402" — vector search might return semantically related errors, but BM25 would exact-match the code. You combine scores using Reciprocal Rank Fusion (RRF) or a weighted linear combination. Pinecone, Weaviate, and Elasticsearch all support hybrid search. At a startup, I'd start with pure vector and add BM25 when I see retrieval failures on specific-term queries.

Re-ranking is a second-pass relevance scoring step. Retrieve top-50 candidates cheaply with vector search, then pass all 50 through a cross-encoder model (like Cohere Rerank or a sentence-transformer cross-encoder) to get more accurate relevance scores, then take top-5. The cross-encoder sees query + document together, so it's much more accurate than bi-encoder similarity but 10–20x slower per comparison. Worth it when: (1) precision matters more than latency, (2) you're in enterprise/legal/medical context where wrong retrieval is costly, (3) you can cache popular queries. Not worth it for: casual chatbots, latency-sensitive apps under 500ms budget. I'd A/B test retrieval quality before adding the latency overhead.

It depends on your existing stack and scale. pgvector: use when you already have PostgreSQL, data volume is moderate (<10M vectors), you want one less service, and queries don't need sub-100ms latency. It's simpler and cheaper at startup scale. Pinecone: use when you need managed scaling, you'll index millions+ of vectors, you want built-in hybrid search, or you need consistent low latency at scale. At a startup building an MVP, I'd start with pgvector (using our existing Postgres) and migrate to Pinecone only when we hit performance limits. The migration is straightforward since both expose similar APIs through LangChain/LlamaIndex.

Three levels: (1) Offline eval — build a test set of question-answer pairs, run RAGAS to get faithfulness and context precision scores. This catches regressions when you change chunking or models. (2) Online eval — log every query + retrieved context + answer in LangSmith. Use LLM-as-judge (automated scoring with GPT-4) to flag low-quality responses. Sample 5% of production traffic for human review. (3) User signals — thumbs up/down, follow-up queries (if a user immediately rephrases the question, retrieval probably failed). The most important metric is faithfulness — an unfaithful answer is a hallucination, which is the worst outcome.

I'd choose fine-tuning over RAG when the goal is to change model behavior rather than inject new knowledge. For example: making a model always respond in a specific JSON format, teaching it company-specific writing style, or improving performance on a narrow classification task. RAG is for knowledge grounding — you want the model to know things it doesn't know. Fine-tuning is for skill/style — you want the model to do something differently. In practice at a startup, I'd exhaust prompt engineering and RAG options before fine-tuning because fine-tuning requires dataset curation, training infra, and retraining every time behavior needs updating.

ReAct is a prompting pattern where the LLM interleaves reasoning ("Thought") and action ("Action") steps. Instead of asking the LLM to answer immediately, you let it think through what it needs, take an action (call a tool), observe the result, and iterate. It works because: (1) breaking the problem into steps reduces cognitive load per step; (2) the model can correct mistakes by seeing tool outputs and re-reasoning; (3) it's transparent — you can see every thought and action for debugging. The alternative (single-shot answering) fails for complex multi-step tasks because the model has to predict all steps at once without feedback.

Four guardrails: (1) Max iterations — hard cap (e.g. 20 steps). If exceeded, return partial result with explanation. (2) Human-in-the-loop — LangGraph's interrupt mechanism: pause before irreversible actions (delete, send email, execute code), require human approval. (3) Tool-level validation — each tool validates its own inputs before executing. A delete tool might require the user's explicit confirmation as a parameter. (4) Cost budget — track LLM tokens and stop if budget exceeded. At my startup, we implemented HITL for any agent action that writes to the database.

LangGraph models agent workflows as directed graphs (nodes = LLM calls or tools, edges = conditional routing). Plain LangChain agents use a flat loop — LangGraph lets you build cycles, checkpointing, parallel branches, and conditional edges. You'd use LangGraph when: (1) your agent has multiple states (idle, searching, summarizing, validating), (2) you need checkpointing for long-running tasks, (3) you need human-in-the-loop interrupts at specific nodes, (4) you want to resume failed runs from last checkpoint. For simple single-tool agents, plain chains are fine. For anything production with multiple steps, LangGraph gives you the control you need.

You define tools as JSON schemas in your API request. The model is trained to recognize when to use them and outputs a structured tool_use block instead of text. Your code catches this, executes the actual function, appends the result as a tool_result message, and sends the whole conversation back to the model. The model then continues with the result incorporated. The key insight: the LLM doesn't execute anything — it just outputs a JSON specification of what it wants to call. Your application code does the actual execution. This is why security matters: your tool implementation controls what can actually happen.

MCP (Model Context Protocol) is Anthropic's open standard for connecting AI agents to external systems — databases, APIs, file systems, etc. Instead of every team building their own custom tool integrations, MCP defines a standard protocol (resources, tools, prompts). Why it matters: (1) composability — any MCP-compatible client can use any MCP server; (2) ecosystem — companies are building MCP servers for popular services; (3) standardization reduces integration overhead. For interviews, knowing MCP signals you're following the latest AI infrastructure trends. It's replacing ad-hoc tool integration patterns the same way REST replaced custom APIs.

I'd reach for multi-agent when: (1) task complexity exceeds a single context window — breaking into specialist agents keeps each focused; (2) tasks are truly parallel — researcher + writer working simultaneously; (3) quality requires cross-checking — one agent validates another's output. In most startup contexts though, a single agent with well-defined tools is more reliable, cheaper, and easier to debug. Multi-agent systems add coordination overhead and failure modes (agent A misunderstands agent B's output). My default is: start simple, add agents only when single-agent demonstrably fails. The patterns (orchestrator → specialist → result) matter more than the specific framework.

Five strategies: (1) Model routing — classify query complexity first with a cheap model, route simple queries to mini models, complex ones to flagship. (2) Prompt caching — Anthropic and OpenAI both support caching repeated prefixes (system prompts, static context). 80–90% cost reduction on cached tokens. (3) Chunk precision — retrieve fewer, more precise chunks so you inject less tokens per query. (4) Output length control — set max_tokens appropriately; don't let the model ramble when a concise answer is fine. (5) Async batching — for non-real-time tasks, batch requests and use batch APIs (50% discount). Track cost per request in LangSmith from day one so you know where money goes.

With LangSmith/LangFuse, I can trace the full pipeline for any failed query. My debugging flow: (1) Find the failing query in LangSmith, open its trace; (2) Check what chunks were retrieved — are they relevant? If not, retrieval is broken; (3) If chunks are correct, check if the answer contradicts the context — hallucination problem; (4) Check latency distribution — where is time being spent? Sometimes latency spikes indicate retrieval problems (too many chunks, slow index). Without observability, you're debugging blind. This is why I add LangSmith from day one, even in prototype — it costs almost nothing and saves hours of debugging.

Five techniques: (1) Streaming — stream tokens to the UI via SSE. The user sees output starting immediately, even though total generation takes 3–5s. (2) Optimistic UI — show the retrieval step happening (progress indicator) to signal the system is working. (3) Reduce TTFT — send the LLM call as early as possible. Don't wait for retrieval to fully finish if you can pipeline. (4) Faster retrieval — ANN search is already fast (<100ms for most sizes). Reduce chunk count retrieved. (5) Model selection — GPT-4o-mini generates tokens ~4x faster than GPT-4o. Use it for initial response, upgrade only if needed.

I think about 4 layers: (1) Prompt security — protect system prompt from injection; treat user input as untrusted data, not trusted instructions. (2) Data isolation — in multi-tenant RAG, each user/org should only retrieve their own documents. Use namespace isolation in Pinecone or user_id metadata filters. (3) Tool permissions — agents should have minimal permissions. A research agent shouldn't have write access to the database. (4) Data retention — check if your LLM API provider trains on your data (OpenAI: opt-out available, Anthropic: doesn't train on API data by default). For compliance-sensitive customers, this question will come up.

Approach: This is testing your self-awareness and learning mindset. Don't hide failures. Pick a real story: maybe retrieval quality was low, or the LLM gave inconsistent outputs. Use STAR: Situation (what you built) → Task (what was expected) → Action (what you tried to fix it, what you learned) → Result (what you changed, what the outcome was). End with what you'd do differently. Engineers who say "I've never had an AI failure" are unconvincing. Engineers who walk through a failure clearly and show they learned are memorable.

Approach: This is very common at startups. Show that you: (1) ask clarifying questions to understand the business goal, not just the technical spec; (2) scope down to the minimal version first; (3) propose an MVP with an explicit "we can add X later" plan. Example from your context: "In our startup, requirements for AI features often come as rough ideas. I've learned to first ask what success looks like — not technically, but for the user. Then I build the simplest thing that could demonstrate value, measure it, and iterate."

Approach: Shows maturity. Use a real example. Show: (1) you advocated for your view with data/reasoning; (2) you understood the other side; (3) you implemented it professionally; (4) what happened and what you learned. "Disagree and commit" is the expected answer at most tech companies.

Approach: Be specific. "I read Hugging Face papers weekly" is better than "I follow the field." Good answers include: following specific researchers on Twitter/X (Karpathy, Yann LeCun), reading Anthropic/OpenAI/Google DeepMind blogs, using AI tools daily (using = learning), building small experiments with new models as they release. Show you have a system, not just passive consumption.

Approach: Perfect for startup context. "At my startup, shipping matters — we're building an MVP." Show: (1) you can identify technical debt consciously; (2) you document what you're cutting and why; (3) you have a plan to address it post-launch. Example: "We shipped the RAG pipeline with fixed chunk sizes and no re-ranking because we needed to validate the use case first. Once users confirmed the feature was valuable, I went back and added evaluation + tuning."

Approach: Be honest and specific. Don't say "it's the future" (too generic). Say something like: "I've been building AI features at my startup and I find the challenges unique — reliability, evaluation, the probabilistic nature of outputs — it requires a different engineering mindset than deterministic systems. I want to go deeper rather than stay generalist. The work I find most engaging is the AI side, and I want to make that my full focus."

Approach: Shows product thinking + engineering rigor. Answer at two levels: (1) Technical metrics — RAGAS faithfulness, latency P95, cost per query; (2) Business/user metrics — task completion rate, user engagement with AI responses, support ticket reduction. The best answers connect both: "Faithfulness score of 0.85 correlates with fewer user complaints — I know this because I track both in the same dashboard." Shows you think beyond benchmarks.

Approach: You have this one easily — learning AI/LLM engineering while running backend at a startup. Show your learning approach: (1) identify what you need to know; (2) find the fastest path to working knowledge (build something, not just read); (3) share what you built and what it taught you. "I had to build a RAG pipeline for our product with no prior experience. I spent a week on the DeepLearning.AI courses, built a prototype, broke it, diagnosed it, and shipped a working version. The debugging taught me more than the course."

Approach: Don't undersell your RAG work. Even "small RAG" involves: embedding pipeline, vector DB, retrieval tuning, LLM integration, evaluation. Walk through it with specificity: "We built a document QA system over our internal knowledge base. I designed the ingestion pipeline (PDF parsing, chunking by token size with overlap), chose Chroma for local dev and Pinecone for production, set up RAGAS evaluation, and added LangSmith tracing. The hardest part was tuning chunk size — too small and answers lacked context, too large and retrieval was imprecise."

Approach: Shows you think strategically. Reasonable things to say: (1) agents will handle more autonomous workflows, requiring better evaluation/safety; (2) multimodal will be default; (3) cost/efficiency will matter more as novelty wears off and ROI is scrutinized; (4) engineers who understand systems holistically (eval, obs, cost) will be most valuable. Connect to your prep: "That's why I'm focusing on evaluation and observability now — not just building features but measuring them."