Fix a misread phrase

Your finance team counts quarters from its fiscal year, which starts on 1 July, so “vouchers entered in Q1” means July to September. The active prompt, plan-writer.v7, reads Q1 as January to March. This guide follows the fix, from a draft prompt to a release, including a mistake the test set catches on the way.

You’ll use: the Prompt Store, the test set, and the service’s plan and test commands.

Draft the change

In a working copy of the service’s files, such as ~/nlsearch-next, copy plan-writer.v7.txt to plan-writer.v8.txt, and rewrite its date rule for fiscal quarters. Never edit a version that’s been released: the audit log records which version made each plan, so a version must always mean the same text.

plan-writer.v8.txt (the changed rule)

- Resolve relative dates from today's date. Quarters are
  fiscal quarters, and the fiscal year starts on 1 July.

Then add a test for the change to tests/find_vouchers.yaml, so the test set checks it from now on:

tests/find_vouchers.yaml (the new test)

- question: Vouchers entered in Q1
  today: 2026-10-01
  expect:
    intent: find_vouchers
    filters:
      - field: entered
        op: between
        value: [2026-07-01, 2026-09-30]

Test the draft

Run the test set against the draft, with test --prompt-version 8:

Output of test --prompt-version 8

Prompt plan-writer v8, Intent Store 2026.10.1
51 of 53 questions produced the expected plan

tests/find_requisitions.yaml, "Laptops ordered since June"
  expected  ordered after 2026-05-31
  got       ordered after 2026-06-01

tests/find_vouchers.yaml, "Vouchers entered since Monday"
  expected  entered after 2026-09-27
  got       entered after 2026-09-28

The new test passes, but two old ones fail, in the same way: after doesn’t include the date itself, so since June has to be after 2026-05-31. The draft’s plans would leave out anything ordered on 1 June, or entered on Monday.

Reproduce it

Ask for one of the plans with the service’s plan command, which prints the plan and the summary that users would see:

podman run --rm \
  -v ~/nlsearch-next:/etc/aipeoplesoft:ro,Z \
  --secret aips_llm_api_key,type=env,target=AIPS_LLM_API_KEY \
  aipeoplesoft-nlsearch:1.0 plan --prompt-version 8 \
  --today 2026-10-01 "Laptops ordered since June"

Output of plan

Intent    find_requisitions
Text      notebook
Filters   ordered after 2026-06-01
Summary   Requisitions · “notebook” · Ordered: after 1 June 2026

Version 7 read since correctly without being told how; the rewritten date rule changed how the LLM reads it. A user would have seen after 1 June 2026 in the summary, but few would notice that it leaves out 1 June.

Fix it with a rule

Say how to read since, rather than relying on the LLM to guess. Save the draft as a new version, plan-writer.v9.txt, with one more rule:

plan-writer.v9.txt (the changed rules)

- Resolve relative dates from today's date. Quarters are
  fiscal quarters, and the fiscal year starts on 1 July.
- "Since" or "from" a date includes that date, so use after
  with the day before it: since June is after May 31.

Compare it with the version in use

Run the test set against the new version, and against the active one:

Output of test --prompt-version 9

Prompt plan-writer v9, Intent Store 2026.10.1
53 of 53 questions produced the expected plan

Output of test --prompt-version 7

Prompt plan-writer v7, Intent Store 2026.10.1
52 of 53 questions produced the expected plan

tests/find_vouchers.yaml, "Vouchers entered in Q1"
  expected  entered between 2026-07-01 and 2026-09-30
  got       entered between 2026-01-01 and 2026-03-31

Version 9 does everything version 7 does, and reads quarters the way your finance team does.

Release it

1. Copy the prompt and the tests from the working copy to ~/nlsearch. Leave plan-writer.v8.txt out, or keep it for the record: nothing uses a version that isn’t active.

2. Make version 9 active. In nlsearch.yaml, set plan-writer under prompts.active to 9, and restart the service with systemctl --user restart aipeoplesoft-nlsearch.

3. Watch the audit log. New entries show plan-writer.v9 as their prompt version. Check the questions with quarters and since in them for the next few days.

4. Roll back if you need to by setting plan-writer back to 7, and restarting the service.

Fix the rule, not the question

It’s tempting to add the failing question to the prompt as an example. A rule, such as how to read since, fixes every question like it, and keeps the prompt short. Keep the question in the test set instead, where it guards the rule.