Skip to content

Natural-Language Query Evidence

This page is the evidence summary for the natural-language query feature: what was built across issues #811–#816, #851 and #868, how it was evaluated, and what each evaluation run caught. The behaviour itself is documented on the Analytics Query page, and the provider decision in ADR-017.

Demonstration feedback, 6 October 2026

The feature was demonstrated on 6 October 2026, after issues #815, #816 and #851 had delivered the endpoint, the home-page widget and the suggestions. Three observations were recorded:

  • Answers were hard to get with casual phrasing. A question had to be worded close to the shape of a published metric to be answered. Ordinary wording for the same figure — "who smashes the most sixes", "who is the most economical bowler" — did not reliably reach the metric it names.
  • Follow-up questions were not understood. Having asked about a player, asking a second question about the same player did not work: each question was translated with no knowledge of the one before it.
  • The assistant should be reachable from every page. Testers asked for a floating chat bubble fixed to the bottom-right corner of every page, replacing the home-page widget and supporting follow-ups within the conversation it holds. This one is not yet built: it is the floating chat assistant entry on issue #924.

Issue #868 was the response to the first two, and it is what the final behaviour on this page describes. It added:

  • a casual-wording map generated from the metric enum itself, so wording for a published measure reaches that measure while a superlative naming no published measure stays a refusal with suggestions;
  • conversation support — up to five earlier turns sent as delimited prior context, so a follow-up resolves against what an earlier question named;
  • scorecard-name handling with the surname fallback, so a spoken name like "Virat Kohli" reaches the scorecard's V Kohli and a disagreeing initial asks rather than answers as somebody else; and
  • the configured default competition, reported in assumptions so an assumed answer is never shown as an exact one.

The three #868 evaluation runs below are the verification of that response, and item three remains outstanding. The meeting record, with the disposition of each feedback item, is retained at evidence/stakeholder-meetings/2026-10-06-natural-language-query-demonstration.md. The feedback was behavioural rather than task-scored, so it is recorded there as demonstration feedback and is not counted as a formal user-testing task attempt under ADR-013.

How the feature is evaluated

Translation is not deterministic, so it cannot be verified by a unit test asserting one output. It is evaluated instead by a fixed set of questions run against the configured provider, each case accepting one or more defensible readings:

npm run evaluate:natural-language-queries

scripts/evaluate-natural-language-queries.mjs writes a dated record under evidence/validation/. It is run by hand, never in CI, because every run spends money against the $10 workspace limit recorded in ADR-017. The case definitions live in scripts/natural-language-query-cases.mjs and scripts/natural-language-conversational-cases.mjs, and npm run test:scripts guards the runner and the case set without calling the provider.

A case passes only when the returned definition matches an accepted reading. Cases that state expectSuggestions must also offer something answerable, and cases that state expectAssumptions must report exactly those assumptions — a question answered against the default competition without saying so is a failure, because the reader would have been shown an answer to a narrower question than they asked.

Evaluation runs

Run Date Cases Passed Failed
#815 first 2026-10-01 27 26 1
#815 second 2026-10-05 36 35 1
#868 run 1 2026-10-07 63 55 8
#868 run 2 2026-10-07 64 62 2
#868 run 3 2026-10-07 64 62 2

The three #868 runs are the ones that exercise the final behaviour: 55/63 → 62/64 → 62/64.

What each run caught

#815 first run, 26/27. Established the baseline across every supported kind, every unpublished dimension and four prompt-injection attempts. Its one failure was the delimiter-escape injection case, refused correctly but labelled ambiguous rather than outside_cricket_statistics or other. That failure has recurred in every run since and is the one open item below.

#815 second run, 35/36. Added the suggestion cases from #851, the home-page widget's own example questions and the scoped-answer cases. It confirmed that refusals come back with something to ask instead, and that the widget's examples translate to the kinds they claim. Same single failure.

#868 run 1, 55/63. The first run against conversations, casual phrasing, scorecard names and the default competition. Eight failures, which separated into four different problems:

  • Three regressions in the prompt changes. participant-season returned a definition the contract rejected outright; the default-competition rule was filling a competition into season-scoped questions, so default-no-season-assumed and default-no-season-assumed-this-year reported an assumption where none was permitted.
  • One transient provider failure. unsupported-match-phase timed out. It had passed in both earlier runs and passed in both later ones. The larger prompt makes a cold request slower, which is what LLM_TIMEOUT_MS and the single retry exist for.
  • Three cases whose expectations were wrong, not the behaviour. participant-runs-only and participant-apostrophe-name reported expected … got … with identical text on both sides — the failure description was comparing more than it printed, so a real mismatch was indistinguishable from a reporting defect. casual-biggest-hitter expected most_sixes for "biggest hitter", which names no published measure; the wording was removed from the casual map rather than the refusal being treated as wrong.
  • The standing delimiter-escape label.

This run is the reason the failure description now prints participant names, competitions and seasons: a mismatch has to be diagnosable without paying for a second run.

#868 run 2, 62/64. All three regressions and the two wrong name expectations resolved. It caught one new thing — default-abbreviation-is-not-an-assumption, added for this run, showed that a question naming "the IPL" was still being reported as having assumed its competition. A reader who names the competition in the short form everyone uses has assumed nothing. The fix was made deterministic in the adapter rather than by adding more prompt text.

#868 run 3, 62/64. The abbreviation case passed. The same two-failure count is not the same two failures: default-no-season-assumed-this-year refused correctly but offered no suggestion, where twenty-one minutes earlier it had offered one. Comparing the two runs shows the same movement in both directions — unsupported-match-phase went from no suggestions to two, and ambiguous-name from none to two.

That is the clearest evidence for the limitation that suggestion generation varies between runs, and it is why a client must render a refusal that carries nothing to ask instead.

Live verification on the deployed site

The runs above are the repeatable evidence for the translation step. Separately, the issue #940 behaviour was checked by hand against the deployed site through the chat assistant on 10 October 2026 and recorded in evidence/validation/issue-940-live-verification-2026-10-10.md. Three checks, all passing: an unfamiliar named competition answered rather than refused or answered against the default, a match-result question refused with wording that points at the fixtures pages, and a follow-up resolving its competition and season from the earlier turn.

That record is a single hand-performed pass against a paid provider, so it confirms the behaviour reaches a reader rather than guaranteeing every phrasing behaves the same way.

Standing failure

The delimiter-escape injection case has failed in all five runs. The attempt is refused in every run — nothing is executed, no prompt content and no data is disclosed — but the refusal carries the reason ambiguous instead of outside_cricket_statistics or other. It is a reporting inaccuracy in a refusal rather than a security finding, and it is tracked as the injection reason-label entry on issue #924 rather than fixed under #868 or #817.

Automated evidence beside the evaluation

The evaluation covers the translation step only. Everything downstream of it is covered by the ordinary suites, which do not call the provider:

  • the adapter's request shape, retries, typed errors and the assertion that no database content is sent, with a stubbed fetch;
  • the query-definition contract and its JSON Schema projection, checked against each other by Ajv;
  • the endpoint's outcomes, limits, headers and logging;
  • name resolution including the surname fallback and the single-candidate confirmation;
  • the deterministic default-competition rule, including the acronym match and its word boundaries; and
  • the widget's rendering of answers, assumptions, suggestions and refusals.

No test anywhere calls the provider, and no API key exists in the repository.

Deferred work and why

Four pieces of work were identified while building this feature and deliberately left out of it. They are recorded here so the reason survives independently of whoever remembers the conversation. The first has since been done; three remain.

All of this is tracked by issue #924, "Natural-language query follow-up work", which holds the feature's remaining work as a single checklist rather than as scattered issues. One issue keeps the items that depend on each other together: the floating chat assistant the 6 October demonstration asked for needs follow-up resolution after a leaderboard and the empty-suggestions fallback to be useful, and those are separate entries on the same list.

Rate limiting for the free definition-evaluation path — done

POST /api/v1/query-definitions/evaluate had no limiter at all. It was not one of the canonical-read paths carrying the anonymous HMAC-pseudonymised bounds, and the natural-language limiter does not cover it because it makes no provider call.

Why it was deferred. The limits built for this feature were sized to protect a paid provider, and this path spends nothing there. That turned out to be the wrong test: it is free of provider cost, not free. It resolves names and reads published statistics on every anonymous request, and the documentation was actively directing traffic to it.

Done under issue #924. It now goes through the same anonymous canonical-read middleware as the reads it is built on: 30 requests per minute per source and 600 per minute across all sources, keyed on an HMAC digest of the address rather than the address, counted per attempt and failing closed. No new limiter, migration or configuration was needed.

A lint rule for SQL interpolation

The platform's central security property is that a value derived from a question reaches the database only as a bound parameter of an existing parameterised read. Nothing enforces that automatically: there is no ESLint rule forbidding an interpolated SQL template literal.

Why it was deferred. The property is currently held by code review, by the architecture's own shape — the adapter never produces a statement, only a definition — and by the dependency-cruiser boundary checks. A lint rule is defence in depth rather than the primary control. Why it still matters: every other security property in this feature has an automated test, and this one has a convention. The rule would catch the regression that review is most likely to miss.

Token metering against the spending limit

Token counts are written to the request log, one line per request. Nothing aggregates them, compares them to the $10 workspace limit, or raises anything as the limit is approached.

Why it was deferred. The workspace spend limit is a hard provider-side ceiling, so the failure mode is a refused request rather than an unbounded bill — the control works without the metering. Why it still matters: the first symptom of approaching the limit is the feature failing, with no warning beforehand, and ADR-017 records that the margin narrowed from about 3,000 questions a month to about 1,670 after #868. The cheap version is a periodic sum over the existing log lines; the numbers are already being recorded.

A cache for repeated questions

An identical question asked twice costs two provider calls. There is no cache keyed on the question text.

Why it was deferred. definitionVersion already makes the answer reproducible, and prompt caching was separately considered and rejected in ADR-017 — the stable prefix only just cleared the model's minimum cacheable length and requests are sporadic against a five-minute TTL. A question cache is a different mechanism from prompt caching and was not evaluated. Why it still matters: a demonstration asks the same handful of questions repeatedly, and the home-page widget offers four example questions that every visitor is likely to click. Those are exactly the repeated questions a cache would absorb. Translation is also non-deterministic, so a cache would additionally make a demonstrated question answer the same way twice — which is a correctness benefit, not only a cost one.

AI Declaration

The preceding page was drafted with the assistance of Claude-Code[Claude Opus 5 (1M context)] under issue #817. The run figures and per-run findings were read from the five evaluation records in evidence/validation/ rather than restated from memory, and the run 2 against run 3 suggestion comparison was taken by diffing those two records. The evaluation runs themselves were performed by Ben Swartz, and the 6 October demonstration feedback is his own record of that session rather than anything the tool observed. The issue #940 live verification linked above was performed by him against the deployed site and written up under issue #886. The deferred-work reasons were recorded with the same assistance under issue #817; each was checked against the repository rather than taken on trust, which is how the unlimited definition-evaluation path was confirmed. That entry was updated to record the limit as delivered with the assistance of Claude-Code[Claude Opus 5 (1M context)] under issue #924.