Skip to content
SparkyaiPeopleSoft

Trace a wrong answer

A user reports: “The Assistant said I have no sick leave, but my balances page shows 30 hours.” The audit log records every turn: the question, each tool call with its arguments and PeopleSoft’s response status, and how long each step took. It doesn’t keep the data the tools returned, so it shows what the Assistant asked for, and you reproduce the rest.

You’ll use: the service’s audit log, and the application service behind the tool.

Find the turn

Section titled “Find the turn”

Search the audit log for the user, around the time they asked:

The turn in the audit log
{
  "time": "2026-09-24T14:05:12Z",
  "user": "[email protected]",
  "conversation": "c-7f3a9b",
  "question": "How many sick hours do I have left?",
  "tools": [
    {
      "name": "get_absence_balances",
      "arguments": { "plan": "Sick Leave" },
      "status": 200,
      "ms": 310
    }
  ],
  "ms": 1840
}

Read it

Section titled “Read it”
  • The right tool ran, get_absence_balances, so the LLM understood the question.
  • PeopleSoft answered with status 200, so the user had access, and the call worked.
  • The argument is the clue. The LLM asked for the plan Sick Leave, but the plan is called Sick. The API found no plan by that name, returned no balances, and the Assistant concluded there were none.

Reproduce it

Section titled “Reproduce it”

Call the operation as a test user, first with the same argument, then without it:

Terminal window
curl -s -H "Authorization: Bearer $TOKEN" \
  "$BALANCES_URL?plan=Sick%20Leave" | jq .
curl -s -H "Authorization: Bearer $TOKEN" "$BALANCES_URL" | jq .

The first call returns an empty list of balances. The second returns every plan, including { "plan": "Sick", "hours": 30 }.

Fix it

Section titled “Fix it”

  1. Make the filter forgiving. Match plan names without regard to case, and by the start of the name, so sick and Sick Leave both find Sick.

  2. Say what exists when nothing matches. Instead of an empty list, return a message such as “No plan called Sick Leave. This user’s plans are Vacation and Sick.” With it, the LLM corrects itself in the same turn, as good tools do with clear errors.

  3. Update the catalog. If the operation’s description or parameters changed, download its OpenAPI description into specs/ again, and restart the service.

  4. Ask the question again, as the test user, and check the audit log for the new call.

Keep it fixed

Section titled “Keep it fixed”

Add the question, with the answer you expect, to the questions you ask test users before each release, as in Test and go live. A wrong answer that a user reported once is the most likely one to come back.

Other patterns in the audit log

Section titled “Other patterns in the audit log”
  • No tool call. The Assistant answered without calling any tool, because none covers the question. It belongs on your list of tools to build.
  • Status 403. The user’s roles don’t include the application service’s permission list. Either grant it, or it’s working as intended, and the answer should have said so.
  • An empty result for an ID. The user may have mistyped it, or may not be allowed to see the record, as in Look up a record by its ID.
  • The wrong tool. Two tools’ descriptions overlap. Make each one say when to use it, rather than the other.
  • The same tool, again and again. The LLM is retrying, often after an error that doesn’t say what to do next. A turn that reaches the limit of 8 calls ends with an apology.
  • A slow turn. Compare each call’s ms with the turn’s. A slow call is PeopleSoft’s to tune; a slow turn with fast calls is the LLM.