Controlled Research at a High-Street Practice.

3The ledger

Every correction made between the first draft and publication, with its severity and the control that caught it. Severity is judged by what would have reached print: Headline means a headline figure or direction in a published paper would have been wrong; Count means a count, label or table; Claim means a sentence that said more than the evidence allowed.

Table 3. The corrections log, classified. “Development checks” means the error was found and fixed while the work was under way, without a record of which check found it; we do not claim more precision about those than the record holds.

4Four errors in detail

Four of the nine headline errors deserve full treatment, because each was invisible on the surface of the work and each would have reached print with confident prose around it.

The unreachable branch

The classifier that sorted company exits tested for insolvency before testing for a solvent members’ voluntary liquidation — and every such filing carries the Companies House category “insolvency”. The solvent branch could never be reached. One hundred and thirty-four solvent wind-ups, typically retirements, were counted as failures, and 121 of them sat on the service side of the study’s central comparison: the error biased the very contrast the study existed to measure. The code ran without complaint and its output looked plausible. It was caught only when an audit recomputed the classification against the cached filing histories and found that the database contained no members’ voluntary liquidations at all among sixty thousand exits — a number that cannot be right.

Two observation dates

Surviving companies were censored at the date of one data source, in March, while deaths were counted from another, in August. Deaths therefore occurred at durations where no survivor remained in the risk set, and every published survival level was out by four to eight points. The direction of the result survived; the levels did not. It was caught by recomputing the headline from source rather than trusting the pipeline’s own output.

The duplicated table

The food-premises data was stored one row per candidate address rather than one row per establishment, and the linkage counted rows. A population of 231,089 was really 70,470. An external reviewer caught it, not the drafting process and not us — which is exactly what external reviewers are for, and is recorded here for that reason.

The table that did not add up

The last two corrections were made a month after the research was declared closed. Building the final versions of these papers, a route-by-route table of exits summed to 1,161 fewer than its own stated total. The cause was a fix made weeks earlier: two postcode districts that had never been collected were added to the cohort, but the filing histories of their 1,452 exits were never retrieved, so those exits carried no classification. Every downstream figure that depended on exit routes was quietly computed without them. The repair took a day, most of it machine time. The lesson is older than any software: a table must sum to its total, and when it does not, the discrepancy is a finding about the method.

5The controls that caught them

None of the controls is novel. What matters is that each was applied deliberately, and that the study was designed so that its output could be checked by them.

• A frozen observation. Every response cached to disk. Without it, no recomputation is possible, because the public register changes daily.
• Recomputation from source. Headline figures re-derived from the database by an independent route, not read from the pipeline’s own logs. This caught three of the nine headline errors.
• Written external review, adopted in writing. Three reviews, one of them referee-grade, each answered point by point in the record. This caught eight errors, more than any other control.
• A second classifier. Two hundred exits re-derived by an independently constructed method; agreement 96.5 per cent at route level, with every disagreement adjudicated. It found one genuine gap.
• Reconciliation. Every population count required to add up to its total. This caught a mislabelled status in August, and in October both the missing histories and a borough figure that had survived from an early draft.
• A published corrections log. Not a control in itself, but the condition that makes the others honest: an error that will be published is an error worth finding.

Together these controls meant that no headline figure reached a final paper on the strength of the pipeline’s own output alone: each was recomputed, reviewed or reconciled by a route independent of the one that produced it.

6How the machine went wrong

The arithmetic was rarely the problem. Where the machine was given a precise instruction and a checkable output, it was fast and, after checking, reliable. Its characteristic failures were of two other kinds, and both are worth knowing because both are persuasive.

Narrative tidying

A ranking of food categories by how much VAT advantage each enjoys was constructed during the analysis, after the growth figures had been seen. In the draft it became an ordering “predicted from the tax rules before it was measured”. Nobody instructed that sentence. It is what a fluent writer produces when asked to explain a pattern, and it converted an exploratory observation into an apparent confirmation. External review caught it.

Prose outrunning evidence

The drafts produced memorable lines faster than the evidence could support them. “The counter is where people try, the restaurant is where people lose” rested on a difference in formal insolvency of a fraction of a percentage point. A ratio was described as “a factor of about sixty” where the honest presentation was two small numbers side by side. A category of companies that left the register before their first accounts were due was at first described as having “never traded” — something the public record cannot show either way. Each line was withdrawn. Each would have been quoted.

The practical consequence is a reviewing habit. A figure produced by the pipeline should be checked by recomputation. A sentence produced by the drafting should be checked by asking what exactly would have to be true for it to be accurate, and whether the data shows that.

7What stayed human

• The question. Why a high-street practice would study the high street at all, and which question was worth asking of the register.
• The tax judgement. The VAT treatment in Paper 2 is reviewed against primary sources by the practice, because responsibility for a statement of tax law cannot be delegated to software.
• The confidentiality line. The decision that no client data of any kind would enter the study, not even in aggregate, was made before the analysis and never revisited.
• The scope rules. No suggestion of non-compliance where the data cannot show it; no call for new filing burdens on small businesses; neutral language about registered-office services, which the practice itself provides.
• The decision to publish against our own thesis. The study set out to show that the VAT boundary had reshaped London’s food trade. It did not, and the papers say so. A machine can report a rejected hypothesis; only the people whose name is on the work can decide to publish it.
• Responsibility. An AI cannot be an author, because it cannot answer for the work. Roseworth can, and does.

8A checklist for another practice

The part of this paper with lasting value. Each item is cheap; together they are the difference between a study and a press release.

1. Write the hypotheses down before the data arrives, together with a commitment to publish whatever it shows.
2. Freeze the observation. Cache every response; record the date; never re-fetch silently.
3. Keep a corrections log from the first day, and publish it with the work.
4. Recompute every headline by an independent route before it is printed, from the stored data rather than the pipeline’s output.
5. Make every table sum to its total. Treat a discrepancy as a finding until it is explained.
6. Commission hostile external review, and answer it point by point in writing.
7. Label every number as measured, derived or modelled, and say which on the page.
8. Read every sentence for what it claims, separately from the figure beside it.
9. Keep client data out of the repository entirely unless there is a written basis for using it.
10. Disclose AI use specifically — what it did, what it did not do, and who is responsible.
11. Name the controls you did not apply, so that a reader knows where any remaining risk sits.

Written this way, a study of this kind is within reach of any practice with the patience to check its own work. The machine changes the cost of doing the analysis. It does not change the cost of being wrong in public, and that remains the practice’s to bear.