Model validation

How reliable is the AI analysis?

A fluent AI response can still be wrong. We check separately whether the output is technically valid, supporting evidence can be found in the source release and the interpretation agrees with human assessment.

Diagnostic benchmark 6 / 6

Prelabelled test cases

All six compact, prelabelled test cases were completed correctly within this limited benchmark. This does not establish an error rate for the full corpus.

Evidence check 100%

Verbatim traceability

In this small benchmark, quotations could be found verbatim in the source. Traceability does not prove that a model score is substantively correct.

Important limitation Small

Not a full validation

Six test cases provide a technical check, not evidence of reliability across thousands of actual releases.

Validation protocol

Three layers before a substantive conclusion.

01 · Schema

Technical validity

All required fields, score limits and data types are checked automatically. Truncated JSON is regenerated.

02 · Evidence

Traceability

Exact quotations are located in the model input after whitespace normalisation. Evidence that cannot be traced back lowers the quality score.

03 · Human review

Blinded sample

A sample drawn in advance will be manually labelled without revealing subsequent price outcomes. This comparison follows after processing is complete.

Current processing

Validation develops alongside the corpus.

Full quality figures become final only once both datasets have been processed. In the meantime, we publish only progress and the protocol.

Loading status

Processed —
Phase —
Updated —

Quality of the ongoing extraction

Four steps, four different meanings.

1 · Technically saved —

Releases with valid, saved output.

2 · Automated checks passed —

Passed schema and internal quality rules; not human approval.

3 · Traceable quotations —

All supplied supporting excerpts were found verbatim in the input. This does not validate the interpretation.

4 · Second-model check —

Number of records reanalysed by a second model. This is not a human assessment.

These interim counts come from saved Qwen3-4B records. The evidence ratio measures only whether supplied quotations can be found verbatim. A blinded human sample and final predictive test are still to follow; these figures are not accuracy percentages.

What remains to be done

No final assessment before the checks are complete.

Human–model agreement

To be calculated after the blinded sample has been assessed.

In progress

Stability by company and year

To be assessed on both the original and replication datasets.

In progress

Additional predictive value

Available only after the untouched final test against returns.

In progress