This is a production-grade multi-agent LLM system designed to tackle complex, multi-step user queries by coordinating specialized sub-agents. It breaks down tasks, retrieves necessary context, drafts responses, and critiques its own work before presenting the final answer. The system is designed to be provider-agnostic but primarily uses DeepSeek as the default LLM provider, while also supporting Groq, Gemini, and a Mock provider for testing. The goal is to create a robust, observable, and self-improving cognitive architecture.
Note: Extensive architectural documentation and interview-prep materials are available as LaTeX PDFs inside the Assignment report for Interviewer-Reviewer/ folder.
To run this project locally, you will need Docker and Docker Compose installed.
-
Clone the repository:
git clone https://github.com/vaibhav3000/multi-agent-system.git cd multi-agent-system -
Configure environment variables: Copy
.env.exampleto.envand fill in your details:cp .env.example .env
Minimum Required Configuration:
LLM_PROVIDER: Set todeepseek,groq,gemini, ormock.- Based on your choice, set the corresponding API key:
DEEPSEEK_API_KEY=your_key_hereGROQ_API_KEY=your_key_hereGEMINI_API_KEY=your_key_here
- Database, Redis, and Chroma configuration are already provided in the
.env.exampleand are ready for the docker stack.
-
Start the system:
docker compose up --build -d
- The API will be available on port 8000 (
http://localhost:8000). - The Log Viewer UI will be available on port 8001 (
http://localhost:8001).
- The API will be available on port 8000 (
User Query
|
[ FastAPI (8000) ]
/ \
SSE Stream Celery Worker (async)
\ /
[ Orchestrator ]
/ | \ \
Decomp. Retrieve Synth. Critique
\ | | /
[ Shared Context / DB ]
|
[ PostgreSQL ] --- [ Meta Agent (Self-Improving Loop) ]
|
[ Tools ] (Web Search, Code Executor, Data Lookup, Reflection)
|
[ LLM Provider ] (Deepseek, Groq, Gemini, Mock)
- Orchestrator: The primary entry point for a user query. It decides which sub-agent should run next based on the current context state and the user's initial request. It operates with a moderate token budget and if it exceeds its budget, it is forced to synthesize whatever it has gathered to avoid looping infinitely.
- Decomposition: Takes a complex query and breaks it down into a list of simpler, actionable sub-tasks. Its decision boundary is strictly limited to task planning and does not execute the tasks itself. It has a strict token budget and will truncate context if exceeded.
- Retrieval: Responsible for gathering external information required to solve the sub-tasks by calling tools. It makes decisions purely on what information is missing. If it runs out of budget, it halts retrieval and passes the incomplete information back to the orchestrator.
- Critique: Reviews the drafted response against the original query to ensure all constraints and requirements are met. It decides whether the draft is sufficient or needs revision. If it exceeds its budget, the current draft is considered final by default to prevent analysis paralysis.
- Synthesis: Aggregates all gathered information and draft critiques into a final, coherent answer. Its boundary is strictly generating the final response. Exceeding its token budget results in aggressive summarization of the context.
- Compression: Analyzes the shared context when it grows too large and summarizes older or less relevant information. It runs continuously to keep the context window manageable for other agents.
- Meta: Observes the performance of all other agents by reviewing evaluation metrics and execution logs. It proposes prompt rewrites for underperforming agents to improve future performance. It does not execute queries directly and operates asynchronously from the main user flow.
Each tool adheres to strict failure contracts to ensure the orchestrator can reliably parse and handle errors without crashing:
| Tool | Timeout Error | Empty/No Results Error | Malformed Input Error |
|---|---|---|---|
| web_search | {"error": "timeout", "code": "TOOL_TIMEOUT"} |
{"error": "no_results", "code": "TOOL_EMPTY"} |
{"error": "invalid_input", "code": "TOOL_MALFORMED"} |
| code_executor | {"error": "timeout", "code": "TOOL_TIMEOUT"} |
{"error": "no_output", "code": "TOOL_EMPTY"} |
{"error": "syntax_error", "code": "TOOL_MALFORMED"} |
| data_lookup | {"error": "timeout", "code": "TOOL_TIMEOUT"} |
{"error": "not_found", "code": "TOOL_EMPTY"} |
{"error": "sql_error", "code": "TOOL_SQL_ERROR"} |
| self_reflection | {"error": "timeout", "code": "TOOL_TIMEOUT"} |
{"error": "no_reflection", "code": "TOOL_EMPTY"} |
{"error": "invalid_format", "code": "TOOL_MALFORMED"} |
- POST /query
- Body:
{"query": "string", "use_celery": boolean} - Success: SSE Event stream yielding
{"event": "...", "payload": {"job_id": "...", ...}} - Error:
{"error_code": "...", "message": "...", "job_id": "..."}
- Body:
- GET /jobs/{job_id}/trace
- Body: None
- Success:
{"job_id": "...", "query": "...", "status": "...", "created_at": "...", "completed_at": "...", "context": {...}} - Error:
{"error_code": "JOB_NOT_FOUND", "message": "Job not found", "job_id": "..."}
- GET /evals/latest
- Body: None
- Success:
{"total_runs": int, "average_scores": {...}, ...} - Error:
{"error_code": "...", "message": "..."}
- POST /rewrites/{rewrite_id}
- Body:
{"action": "approve" | "reject"} - Success:
{"status": "success", "message": "..."} - Error:
{"error_code": "INVALID_REWRITE_ACTION", "message": "..."}
- Body:
- POST /evals/rerun
- Body: None
- Success:
{"status": "eval_suite_completed", "total_runs": int, "average_scores": {...}} - Error:
{"error_code": "...", "message": "..."}
- GET /health
- Body: None
- Success:
{"status": "ok"} - Error: N/A
The system includes 15 curated test cases:
- 5 Baseline: Straightforward queries with clear, expected answers.
- 5 Ambiguous: Queries lacking context, requiring the system to make reasonable assumptions or ask for clarification.
- 5 Adversarial: Queries designed to trick the system, prompt-inject, or request impossible guarantees.
These are scored across 6 dimensions:
- Answer Correctness
- Citation Accuracy
- Contradiction Resolution
- Tool Efficiency
- Budget Compliance
- Critique Agreement
Eval runs are persisted in PostgreSQL, allowing for historical comparisons. Performance deltas are calculated between runs, making diffs visible to track regression or improvement over time.
Running the eval harness with LLM_PROVIDER=mock verifies that the pipeline runs end-to-end. Here is a sample eval score output:
| Test ID | Category | Correctness | Citation | Budget | Overall |
|---|---|---|---|---|---|
| tc_01 | baseline | 0.80 | 0.70 | 1.00 | 0.83 |
| tc_02 | ambiguous | 0.90 | 0.85 | 1.00 | 0.91 |
| tc_03 | adversarial | 0.40 | 0.50 | 0.80 | 0.56 |
The Meta-Agent runs periodically to analyze system performance.
- It queries the
eval_runstable to find the worst performing evaluation case. - It identifies the worst-scoring dimension for that case.
- It maps the poor dimension to the responsible agent (e.g., poor Tool Efficiency maps to Retrieval).
- It proposes a prompt rewrite for that agent to address the deficiency.
- The rewrite is stored in the
prompt_rewritestable aspending. - A human must manually approve the rewrite via the API.
- Upon approval, the rewrite is applied to the
agent_promptstable, and agents dynamically load it at runtime viaget_active_prompt. - The system reruns the failed test cases to measure improvement.
- A performance delta is logged.
What it does not do: It does NOT auto-apply rewrites, it does NOT retrain the base LLM model, and it does NOT continuously loop and mutate prompts without human involvement.
- The system uses a stub search implementation unless
SEARCH_API_URLis configured. - The SSE endpoint uses event-level streaming, not true granular token streaming.
- The
code_executortool relies on python'seval/execlocally and is not a production-safe secure sandbox. - There is currently no authentication or authorization on any endpoint.
- Execution logs are written synchronously to PostgreSQL, which could bottleneck response times under high concurrency loads.
- Secure Sandbox: Implement a Firecracker microVM or gVisor for the code executor tool.
- True Token Streaming: Refactor the agent loops to yield token-by-token for a better UX.
- Authentication: Add JWT or API Key based auth middleware to secure all endpoints.
- Live Web Search: Integrate Tavily or Serper API for real-time, accurate web search.
- Asynchronous Logging: Move
execution_logswrites to a background queue or Kafka to improve API latency. - Advanced Meta-Agent Analysis: Implement clustering on user queries to find patterns in edge-case failures.
- Frontend Dashboard: Build a React/Next.js dashboard to manage rewrites and view traces visually.
AI assistance was kept to be minimum throughout this project. AI tools were strictly used for debugging complex issues and resolving runtime errors, while all core logic, orchestration, and architectural designs were completed manually.