A Q&A bot that cites its page, and admits what it does not know.

The problem

Staff at a small contractor ask questions like "how long is probation?" or "can I accept this gift?", and the answer is somewhere in a city employee handbook. The bot has three jobs:

1. Answer from the handbooks only.
2. Cite the page, so the person can check.
3. Say "I can't find that in the handbooks" when the answer is not there, instead of guessing.

The third matters most. A confident wrong answer about leave or gifts is worse than no answer.

The client is invented and no private data is used. The documents are three real, public city employee handbooks, 193 pages in total.

How it works

• It runs on one desktop PC. A small local AI model on an 8 GB graphics card answers in about 5 seconds. No handbook text leaves the machine.
• The code decides what counts as a citation, not the model. The model may only point at passages it was shown. A "found" reply with no usable source is recorded as an error, not passed to the user.
• Keyword search, not anything exotic. Simple, fast, and easy to explain and debug. Its limits show up in the results.
• The model is swappable by changing one settings file, which made the comparison below possible.

How it was tested

Thirty test questions were written before the bot was built, each with its expected answer and page. They were split in two:

• 20 for building, used freely while tuning.
• 10 held back, never used for tuning, and run exactly once at the end. This is the honest score, because a bot can be adjusted until it passes any questions it has already seen.

Scoring was mechanical, the rules were fixed before any score counted, and a person read every failed answer.

Results

Six answerable questions is a small sample. The fair statement is "4 of 6 on unseen questions, and it never invented an answer", not a percentage.

Both failures were search failures. In one, the right page was found but the passage holding the answer was not among those shown to the model, so it refused. In the other, it gave half the answer correctly and was never shown the page with the other half. When search failed, the bot refused or answered partly. It did not fill the gap with a guess. That behaviour is what the design was for, and it held on questions it had never seen.

What went wrong

• The test set had errors of its own. Four of the twenty building questions were wrong, not the bot. One "not in the documents" question had an answer at the back of a handbook, which the bot found and I had missed.
• Mechanical scoring missed correct answers worded in a way no approved phrase matched. Reading every failure is not optional.
• Tuning on a small set overfits. The search settings were adjusted until the building questions passed. The held-back run showed the fix was narrower than it looked.
• Handing the coding to a small local model cost more than it saved. Two three-hour attempts at one step failed review on simple plumbing, so the work moved to stronger tools.

A bigger model was not better

In a second round a large hosted model was swapped in, with the search left alone. On the same 30 questions it scored 18 of 21 against the small local model's 19 of 21, at a measured cost of about 0.44 cents per question. Both failed the same two questions for the same reason: search never showed them the answer. With this search, the search was the limit, not the model. And with a hosted model, the question and the retrieved text leave the building; with the local one, nothing does.

The search was then changed to keep rules whole across passage and page breaks. That fixed both known failures on the old questions. It is not yet confirmed: those questions are now tuning data, and the change will only count once it is run a single time on new ones.

How it was built

I designed the project, wrote the test set, set the targets, and made every decision about what counted as correct. The code was written by AI coding tools working from written specifications. Each change was reviewed, covered by automatic tests that run on Windows and Linux, and recorded in a dated log. Round 1 was finished about 20 hours after the first commit.

Limits

• Six answerable held-back questions cannot support a precise accuracy figure.
• One kind of document, in English, from three sources.
• Keyword search misses passages that answer a question in different words from the question.
• The held-back questions are now spent. Any improvement tested against them would be tuned to them.