When more instructions made the results worse

An illustrative order workflow; not a report of a deployed client system.

The first surprise in my GenomeLens experiments came from a newer Qwen model. I expected it to handle a simple prompt at least as well as the older models I had been testing. In the first comparison, it passed fewer cases. My next instinct was to improve the instructions. That seemed reasonable too: give the agent more context and guidance, and it should have a better chance of doing the job. Later tests would show that adding guidance could also undo an improvement.

GenomeLens is a personal project for exploring genome variants and learning how to build agents around data tools. The underlying software retrieves the records; the agent chooses a tool and explains what it returns. Working with a coding assistant, I could test that behaviour against a small synthetic dataset, with specific expectations about tool use, factual accuracy and the limits of an answer.

In the seven-case comparison, the newly added Qwen model passed five cases, while two older models passed six. Its failures were quite specific. It made an unnecessary tool call in one case and altered notation it was supposed to preserve in another. One failed check was enough to fail a case, so the score required interpretation. An extra call and an incorrectly copied value have different consequences, even when both count as failures.

The result challenged my expectation, but it gave me concrete failures to investigate. A focused prompt revision asked the models to preserve values exactly and avoid inventing where information came from. Keeping the tasks, tools and settings fixed, the aggregate result across five models improved from 22 to 26 passing cases out of 35. The newer Qwen model reached seven out of seven in that run. Clearer instructions had helped.

The second surprise came later, when the project gained an optional public-evidence lookup. That capability needed further instructions: when to use it, which information could be sent, and how to explain the evidence returned. The initial implementation included those instructions in every run, including runs where the lookup tool was disabled.

On the next run of the existing local tests, the aggregate result fell from 26 passing cases to 20 out of 35. These were the same seven local tasks across the same five models. The newer Qwen model still passed all seven; the regressions were in other models.

The correction was to select the instructions alongside the available tools. Local runs kept the previously accepted prompt. Runs with the evidence capability enabled received the additional rules. The local results returned to 26 out of 35. The new capability still had its instructions, and the existing workflow recovered its earlier results.

That sequence made the idea of a “good prompt” more concrete for me. One addition had improved the results; another had made them worse. The useful distinction was whether the instructions belonged to the work the agent could actually perform in that run. A rule can be sensible for one capability and still be unhelpful in a different operating context.

The evaluation also needed scrutiny. During the comparisons, reviewing captured answers exposed gaps in the automatic checks, including cases where a negated claim was treated as a positive one. Those checks were corrected and the affected results reassessed. The scores reflect specific questions and grading rules. Passing the checks does not establish the quality of every possible answer, and the later compact report did not retain the prose for full review.

For a small experiment, these results were enough to change the next implementation decision. They were too limited to rank whole model families or establish a general rule about prompt length. The newer model improved after a targeted change, and the instruction expansion affected models differently. Both details matter to what I take from the experiment.

What I want to carry into other projects is the habit of keeping working examples in the evaluation as the system grows. A new capability should earn its place while the existing work continues to behave as expected. A model upgrade or a prompt revision gives me a reason to compare the results again.

I still expect progress from better models and clearer instructions. GenomeLens has made me more interested in the evidence behind those expectations. Before accepting the next improvement, I want to see which failures it fixes, which behaviours it changes, and whether the work that already passed still does.

← Back to thinking