User opens Support Bot chat projects/automations/app/chat/[agent]/page.tsx:1 validates /support from projects/automations/lib/automations.ts:29 and renders ChatView. 2. Frontend sends chat to /api/chat/support projects/automations/components/ChatView.tsx:421 uses DefaultChatTransport({ api: “/api/chat/support” }). For support messages, it also renders benchmark result cards when the agent uses search_llm_benchmarks. 3. API preloads benchmark context projects/automations/app/api/chat/support/route.ts:1 checks the latest user message for benchmark/model keywords like model, rank, score, compare, MMLU, GPQA, open-source, local, etc.
If matched, it calls searchLlmBenchmarks({ query, topK: 6 }) before the agent runs, formats those rows, and injects
them into the agent instructions as retrieved context.
4. Support agent is created locally projects/automations/agents/support/agent.ts:1 creates a ToolLoopAgent named aura_support using the shared OpenAI model builder from projects/automations/agents/_shared/model.ts:1. Unlike scrape, this agent does not use an external MCP server. 5. Agent has one local tool The tool is search_llm_benchmarks, defined inline in projects/automations/agents/support/agent.ts:21. It accepts: query, topK, minScore, provider, model, benchmark, openSourceOnly, localOnly, excludeProprietary, and benchmarkPreference. 6. Instructions control factual behavior projects/automations/agents/support/instruction.ts:1 tells the agent to call search_llm_benchmarks for rankings, benchmark scores, comparisons, or “best model” questions. It must base benchmark claims only on retrieved rows and avoid generic fallback examples. 7. RAG search runs against local Postgres + pgvector projects/automations/lib/rag/llmBenchmarks.ts:1 embeds the query with OpenAI embeddings, then searches LlmBenchmarkChunk.embedding using cosine similarity. For ranking-style questions, it often skips vector search first and does a direct ordered score query for benchmarks like LMArena, GPQA, MMLU-Pro, IFEval, BBH, MATH, or Open LLM Leaderboard Average. 8. Benchmark data comes from import + seed scripts - projects/automations/scripts/import-llm-benchmarks.mjs:1 pulls leaderboard data from Hugging Face datasets into JSON files under data/llm-benchmarks. - projects/automations/scripts/seed-llm-benchmark-rag.mjs:1 embeds those rows and stores them in Postgres. 9. Database tables The support RAG tables are LlmBenchmarkSource and LlmBenchmarkChunk in projects/automations/prisma/ schema.prisma:35. The migration enables vector and creates an ivfflat cosine index in projects/automations/prisma/ migrations/20260429073000_add_llm_benchmark_rag/migration.sql:1.