Model validation
How reliable is the AI analysis?
A fluent AI response can still be wrong. We check separately whether the output is technically valid, supporting evidence can be found in the source release and the interpretation agrees with human assessment.
Prelabelled test cases
All six compact, prelabelled test cases were completed correctly within this limited benchmark. This does not establish an error rate for the full corpus.
Verbatim traceability
In this small benchmark, quotations could be found verbatim in the source. Traceability does not prove that a model score is substantively correct.
Not a full validation
Six test cases provide a technical check, not evidence of reliability across thousands of actual releases.
Validation protocol
Three layers before a substantive conclusion.
Technical validity
All required fields, score limits and data types are checked automatically. Truncated JSON is regenerated.
Traceability
Exact quotations are located in the model input after whitespace normalisation. Evidence that cannot be traced back lowers the quality score.
Blinded sample
A sample drawn in advance will be manually labelled without revealing subsequent price outcomes. This comparison follows after processing is complete.
Current processing
Validation develops alongside the corpus.
Full quality figures become final only once both datasets have been processed. In the meantime, we publish only progress and the protocol.
Loading status
Quality of the ongoing extraction
Four steps, four different meanings.
Releases with valid, saved output.
Passed schema and internal quality rules; not human approval.
All supplied supporting excerpts were found verbatim in the input. This does not validate the interpretation.
Number of records reanalysed by a second model. This is not a human assessment.
These interim counts come from saved Qwen3-4B records. The evidence ratio measures only whether supplied quotations can be found verbatim. A blinded human sample and final predictive test are still to follow; these figures are not accuracy percentages.
What remains to be done
No final assessment before the checks are complete.
Human–model agreement
To be calculated after the blinded sample has been assessed.
In progressStability by company and year
To be assessed on both the original and replication datasets.
In progressAdditional predictive value
Available only after the untouched final test against returns.
In progress