Airbnb Data Analyst Agent

Five specialized agents that plan, write, validate, chart, and narrate SQL analytics — every number cited back to source rows.

Org: Columbia · Agentic AI for Analytics
Role: Designed & built solo
Period: 2026
Status: live
Stack: FastAPI · LangChain · DuckDB / Postgres / Snowflake · OpenAI function calling · matplotlib · pytest
Links: Live demo ↗GitHub ↗

5^*1

agents on a typed message bus

planner · SQL · validator · chart · narrator — every step inspectable and replayable

×3^*2

retry budget, exponential backoff

validator-triggered on tool errors and intent mismatch

100%^*3

numbers cited to source rows

narrator contract — uncited claims don't ship

01/ CONTEXT

The problem

Analysts spend hours translating business questions into SQL, pulling data, checking it, and reformatting the answer into something a stakeholder can consume. Most LLM "text-to-SQL" demos hallucinate schema, fail silently on bad queries, or skip the last-mile steps that actually matter: a chart, a citation, a sanity check.

The goal: a system that behaves like a junior analyst with guardrails — it decomposes the question, writes SQL that actually runs, validates the result before showing it, plots the data, and cites the rows it used.

02/ CONSTRAINT

The constraints

Text-to-SQL models hallucinate column names, misuse joins, and produce queries that run but return the wrong answer
Single-prompt approaches have no way to recover from tool errors or mid-query course corrections
Charts need semantic intent ("show trend over time"), not just "make a chart of this dataframe"
Auditability is table stakes in any real analytics context — every number shown must trace back to source rows
Latency and cost must stay bounded even as the agent retries and self-critiques

03/ APPROACH

The loop

Instead of one prompt doing everything badly, five specialized agents each own one contract:

Planner — decomposes the natural-language question into a plan of sub-queries and tool calls
SQL Agent — writes SQL against the live warehouse schema, with access to db.schema() and db.query(sql)
Validator — dry-runs the SQL first, audits row counts, nulls, and types; self-critiques when the result doesn't match the planner's stated intent; retries on tool error (×3, exponential backoff)
Chart Agent — calls plot.auto(df, intent) to render the right visualization for the question, not just a visualization
Narrator — composes the final answer, citing every number back to specific source rows

All five communicate over a typed message bus, so every intermediate step is inspectable and replayable — the trace below is rendered from exactly that bus.

The five-agent pipeline. Each box is a separate agent with its own contract; the message bus carries {plan, sql, df, chart, answer, error}.

04/ SYSTEM

The build

Every capability is a typed tool with an explicit contract — the agents can only act through these:

Tool	Signature
`db.schema()`	`→ Table[]`
`db.query(sql)`	`→ DataFrame`
`df.describe(df)`	`→ Stats`
`plot.auto(df, intent)`	`→ PNG`
`web.search(q)`	`→ Link[]`

The warehouse layer is pluggable: DuckDB for the live demo (NYC Airbnb data), with Postgres and Snowflake adapters behind the same interface. FastAPI serves the API; the whole system is exercised by a pytest eval harness on every commit.

05/ EVIDENCE

Evidence

Guardrails are only real if you can watch them fire. Below are recorded runs from the system — step through them. The first run includes a hallucinated column name, the database error it caused, and the validator catching and fixing it.

What keeps the system honest in production:

SQL dry-run + row-count sanity check before any query result is trusted
Null / type audit before any chart is rendered
Self-critique when the SQL result doesn't match the planner's stated intent
Golden Q/A regression suite — every commit runs canonical questions and checks the answers
Per-query latency and cost breakdown, so behavior stays measurable as prompts evolve

agent.trace · recorded run

❯ Do superhosts get better review scores than other hosts?

planner820ms

Compare avg review_scores_rating for superhosts vs non-superhosts; check statistical significance before claiming a difference.

sql[fail]1.1s

SELECT superhost_flag, AVG(review_scores_rating) AS avg_rating,
       COUNT(*) AS n FROM listings
WHERE review_scores_rating IS NOT NULL
GROUP BY superhost_flag;

db[fail]90ms

ERROR: column "superhost_flag" does not exist

validator[retry]940ms

Tool error caught. Re-read schema via db.schema() — column is host_is_superhost. Retrying (attempt 1/3, backoff 0.5s).

sql1.4s

SELECT host_is_superhost, AVG(review_scores_rating) AS avg_rating,
       COUNT(*) AS n FROM listings
WHERE review_scores_rating IS NOT NULL
GROUP BY host_is_superhost;

db160ms

returned 2 rows · n(true)=18,402 · n(false)=26,117

validator2.1s

Null audit OK · sample sizes healthy · running Welch's t-test on the two groups.

chart1.7s

Rendered bar chart — superhost 4.89 vs other 4.61 (0–5 scale).

narrator3.1s

Yes. Superhosts average 4.89 vs 4.61 (Δ=0.28, p<0.001, Welch's t-test). Source: listings.review_scores_rating × listings.host_is_superhost, n=44,519.

run complete · 11.4s end-to-end · $0.014 · 9 bus messages

step 9/9

Recorded runs from the live system, replayed from the message bus. Use the controls to step through — including the failed query and its retry.

06/ IMPACT

Impact

Handles multi-step analytics questions end-to-end with auditable traces
Every answer is grounded — users can drill into the exact rows the agent cited
Regression evals plus per-query latency/cost tracking keep production behavior measurable
Designed to plug into any warehouse with a SQL-compatible adapter

Questions about this project? Press ⌘K — the chat answers from these case studies and cites them.