Turn fallbacks into intents

When no intent fits a question, or its plan fails validation twice, the search page falls back to an ordinary keyword search. Users still get results, but not the precise search they asked for. The audit log records every fallback, which makes it the best list you have of what to improve.

You’ll use: the NL Search Service’s audit log, the Intent Store, the test set, and the Search Framework’s administration pages.

Find the fallbacks

The audit log is one JSON object per line. List the questions that fell back in the last week:

jq -r 'select(.fallback and .time >= "2026-10-12")
  | .question' ~/nlsearch-audit/nlsearch-audit.jsonl

bills from Acme over $500
invoices on hold for Initech
Acme bills this month
which Globex invoices are on hold
who is my HR contact

Each entry says why the question fell back:

{
  "time": "2026-10-19T09:12:44Z",
  "requestId": "nls-5b0e17",
  "question": "invoices on hold for Initech",
  "versions": {
    "intents": "2026.10.1",
    "prompt": "plan-writer.v9"
  },
  "candidates": ["find_vouchers", "find_requisitions"],
  "intent": null,
  "validation": "passed",
  "fallback": true
}

Sort them

Wording. “Bills from Acme over $500” and “Acme bills this month”: find_vouchers was a candidate, and it has every field these questions need, but nothing in it says that users call vouchers bills. The fix is in the intent’s examples and description.
A missing field. “Invoices on hold for Initech”: find_vouchers was a candidate, but it has no way to say on hold, so the LLM rightly chose no intent. The fix is a new field, which starts in PeopleSoft.
Outside NL Search. “Who is my HR contact” isn’t a search for documents, and a keyword search is a fine answer. Leave it.

With many intents, also check the candidates. If the right intent isn’t among them, its examples don’t look enough like the question, and more varied examples are the fix.

Fix the wording

Add the phrasings to find_vouchers, in the description and as examples:

description: >-
  Find accounts payable vouchers, which users also call
  invoices or bills, by supplier, amount, entry date,
  status, and payment hold.
examples:
  # ... the existing examples, and:
  - Bills from Acme over $500
  - Acme bills this month

Add the missing field

Add it to the index. In the view behind the voucher search, ZZ_VCHR_SRCH_VW, add a column, such as ZZ_PYMNT_HOLD, that’s Y when any of the voucher’s scheduled payments is on hold, and N otherwise. Map it as an attribute of ZZ_VOUCHERS, redeploy the search definition, and run a full build.

Add it to the intent:

fields:
  # ... the existing fields, and:
  hold:
    attribute: ZZ_PYMNT_HOLD
    type: code
    operators: [equals]
    values:
      on hold: Y
      not on hold: N

Add test questions for both fixes:

- question: Invoices on hold for Initech
  today: 2026-10-20
  expect:
    intent: find_vouchers
    filters:
      - { field: supplier, op: contains, value: Initech }
      - { field: hold, op: equals, value: on hold }

- question: Bills from Acme over $500
  today: 2026-10-20
  expect:
    intent: find_vouchers
    filters:
      - { field: supplier, op: contains, value: Acme }
      - { field: amount, op: greater_than, value: 500 }

Test and release the Intent Store as version 2026.10.2, as in Add an intent for requisitions.

Measure it

Count the fallbacks each week, and compare the weeks before and after the release. Each entry’s versions.intents shows which release the question met, so you can tell the two apart even on the day you release. The questions that still fall back are next week’s list.