Advanced LLM Application Engineering: Hybrid RAG, Tools, Memory & Evals
Build an agent with hybrid RAG, tool calling, persistent memory, evaluation, observability, and guardrails.
Extend a working AI agent with hybrid retrieval, tools, persistent memory, evaluation, and guardrails in a three-hour coding workshop.
Workshop description
Modify and test a prepared agent using Google ADK and Gemini. Connect document retrieval with analytical tools and simulated API actions, then investigate retrieval errors, stale memories, prompt injection, and tool failures.
Inspect traces and compare answer quality, task completion, latency, and model usage before and after your fixes.
Who this workshop is for
- Data scientists and ML practitioners developing agent-based applications.
- AI and software engineers working with retrieval, tools, and memory.
- Technical practitioners seeking practical evaluation and reliability controls.
By the end, you will be able to build
- Hybrid retrieval combining BM25, Chroma vector search, result fusion, filtering, and reranking.
- Analytical tools using pandas, SQLite, and simulated APIs.
- Persistent memory with scoped retrieval and conversation state.
- Tool controls with argument validation, approval checks, and bounded retries.
- Evaluation and debugging workflows using traces, structured logs, and pytest assertions.
Prerequisites
- Comfortable writing Python and working with APIs or data workflows.
- Familiarity with LLM applications is helpful. Google ADK experience is optional.
- Complete the provided environment and model-access checks beforehand. Setup instructions will explain access requirements and usage costs.
Key takeaways
- GitHub project with starter code, exercise checkpoints, and a completed reference implementation.
- Reusable retrieval, memory, tool, and guardrail components.
- Evaluation datasets, failure scenarios, scoring scripts, and sample traces.
- Architecture diagram, design cheatsheet, evaluation checklist, and setup guide.
- Full HD recording, presentation slides, and Certificate of Completion.
About the instructor
Brij Kishore Pandey is a Principal Engineer and AI Architect at Wells Fargo with more than 16 years of experience across AI, data engineering, cloud infrastructure, and enterprise software. His teaching focuses on building systems that can be evaluated, observed, and controlled under realistic conditions.
Lineup
Brij Kishore Pandey
Good to know
Highlights
- 3 hours
- Online
Refund Policy
Location
Online event
Agenda
From Generative Orchestration to System 1 / System 2 Architecture
Why using autoregressive LLMs for classification and control flow is an engineering anti-pattern, and how to separate vector scoring from state execution.
Hybrid RAG and evidence retrieval 40 minutes | 0:00–0:40
Combine keyword and vector search, merge results, and apply reranking. Compare retrieval approaches on questions involving exact identifiers and semantic meaning. Examine metadata filtering, context selection, and how irrelevant retrieved content affects the final answer. Practical outcome: A hybrid retrieval pipeline and a comparison of results on the same query set.
Tool calling and memory 40 minutes | 0:40–1:20
Connect retrieval, SQL, and API tools. Inspect how the agent chooses between searching documents and querying live data. Implement validated tool inputs, short-term conversation state, and persistent memory. Explore failures caused by incorrect tool selection and stale memories. Practical outcome: An agent that selects tools and retrieves relevant memory across conversations.