Why “chat with your database” gives the wrong revenue number, and how to fix the architecture.

The enterprise accuracy cliff

Almost every leadership team has been shown the same demo: an executive types “What was our net revenue in Germany last quarter?” into a chat box, a language model writes a SQL query, and a bar chart appears in three seconds.

Peer-reviewed research shows why that demo breaks in production. On the classic academic Spider 1.0 benchmark—built on small, clean schemas with a handful of tables—frontier models exceed 85% execution accuracy. When researchers evaluated those same frontier model frameworks on Spider 2.0 (Lei et al., 2024/2025), comprising 632 real-world enterprise tasks across production Snowflake and BigQuery schemas averaging roughly 810 columns per database, baseline execution accuracy dropped to 17.0%–21.3%. Similarly, the BIRD-SQL benchmark (Li et al., NeurIPS 2023) across 12,751 queries on 95 domain-heavy databases demonstrated that models routinely fail when queries depend on unwritten domain rules—such as how credit notes, cancelled orders or fiscal calendars are encoded.

Worked example: a +44.4% silent error

A syntax error is harmless because the database rejects the query. The real danger in financial text-to-SQL is a query that executes cleanly and returns a plausible wrong number. Consider two billing documents in a standard ERP header-and-line schema:

Synthetic two-document ERP ledger extract. True net product revenue = €1,000.00 − €150.00 = €850.00; true total billed (with header freight) = €900.00.

When a user asks “What is our total net billed revenue including freight?”, a raw text-to-SQL agent inspecting the column names joins erp_doc_headers to erp_doc_lines on doc_id and sums l.line_eur + h.freight_eur. Look at what that query computes on the four rows above:

Arithmetic check: (€1,300.00 − €900.00) ÷ €900.00 = +44.44% overstatement caused by a 1:N header-to-line chasm trap (+€100) and unsigned credit memo (+€300).

Why syntactically valid SQL is dangerous

The example above is not an edge case; it is how almost every ERP (SAP, NetSuite, Microsoft Dynamics, Odoo) stores transactions. Three structural features trip up unconstrained SQL generation:

1. One-to-many join fan-out (the chasm trap). Joining order headers, invoice headers, or shipment headers to line items duplicates every header-level measure (freight, header discount, order tax) by the number of lines on the document.
2. Unsigned transaction types. Many ERPs store all quantities and amounts as positive numbers and rely on a doc_type, debit_credit_indicator (S/H in SAP) or transaction_class column to determine whether a row adds to or subtracts from revenue.
3. Status and intercompany filters. A raw table contains pro-forma quotes, cancelled invoices, intercompany recharges and test postings alongside statutory revenue. Omitting a single WHERE predicate pollutes the total.

Constrain the model with a semantic layer

The fix is architectural: never allow a language model to write arbitrary JOINs against raw staging or ERP tables. Instead, expose a governed semantic layer—a small set of tested, pre-joined analytical views and explicit metric functions where grain, sign conventions and status filters are locked in code:

CREATE OR REPLACE VIEW sem_verified_billing_lines AS
WITH line_weights AS (
  SELECT
    h.doc_id,
    h.doc_date,
    h.customer_id,
    h.doc_type,
    l.line_no,
    l.sku_id,
    CASE WHEN h.doc_type = 'CREDIT_MEMO' THEN -1.0 ELSE 1.0 END              AS sign_mult,
    l.line_eur,
    h.freight_eur,
    l.line_eur / NULLIF(SUM(l.line_eur) OVER (PARTITION BY h.doc_id), 0)     AS line_share
  FROM erp_doc_headers h
  JOIN erp_doc_lines l USING (doc_id)
  WHERE h.status = 'POSTED'
    AND h.is_intercompany = FALSE
)
SELECT
  doc_id,
  doc_date,
  customer_id,
  sku_id,
  ROUND(sign_mult * line_eur, 2)                                             AS net_product_revenue_eur,
  ROUND(sign_mult * freight_eur * COALESCE(line_share, 0), 2)                AS allocated_freight_eur,
  ROUND(sign_mult * (line_eur + freight_eur * COALESCE(line_share, 0)), 2)   AS total_billed_revenue_eur
FROM line_weights;

When an AI assistant or BI tool is restricted to querying sem_verified_billing_lines, summing net_product_revenue_eur returns €850.00 and summing total_billed_revenue_eur returns €900.00—whether grouped by customer, SKU, month or company total.

Verify every generated number twice

Even on top of a semantic layer, any AI-generated narrative or executive summary should obey a strict publishing rule—the same rule enforced inside 16-62 Labs: the model may only cite figures returned by a deterministic SQL query against verified views, every query is stored alongside the figure it produced, and any number in a drafted summary that does not match the query result is flagged before anyone reads it.

Sources & research references

• Spider 2.0 Benchmark (ICLR 2025): Fangyu Lei et al., Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (arXiv:2411.07763) and spider2-sql.github.io — evaluation of language model agents on 632 enterprise cloud warehouse workflows averaging ~810 columns.
• BIRD-SQL Benchmark (NeurIPS 2023): Jinyang Li et al., Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (arXiv:2305.03111) and bird-bench.github.io — 12,751 text-to-SQL pairs across 95 dirty, domain-specific databases demonstrating the necessity of explicit domain business rules.