Methodology
How we test our models.
AI can analyse information, but convincing model output is not yet a reliable prediction. We evaluate language models, machine learning and statistical analysis against simple baselines and an independent final test.
From hypothesis to conclusion
A consistent research process.
The details vary by data source, but the sequence stays the same. The final test is examined only after all choices have been made.
-
Define the question and horizon
We specify what information is available, what is being predicted and when a decision could realistically be executed.
-
Order data chronologically
Training, validation and holdout periods follow in chronological order, preventing future information from leaking back.
-
Compare the model with a baseline
A complex model must demonstrably add value beyond a simple rule or constant prediction.
-
Lock down the choices
Features, settings and signal thresholds are fixed before the final test.
-
Account for costs and uncertainty
We report intervals, sample sizes, transaction costs and sensitivity to assumptions.
-
Replicate where possible
An interesting result is tested again on a later period or new companies.
Timeline for each earnings release
Public information first, a possible decision afterwards.
-
01
Publication
The SEC filing receives a public timestamp.
-
02
Available for analysis
Only text that could have been known at that time is processed.
-
03
Executable entry
In the earnings-release studies, no earlier than the close of the first subsequent trading day.
-
04
Measure the outcome
Return over 1, 5 or 20 trading days, adjusted for market and sector.
This timeline describes the earnings-release studies. Other studies have their own execution rules.
Quality controls
Four risks that can distort any conclusion.
Data leakage
Features and targets are linked using explicit timestamps. A release is included only after it became publicly available.
Model selection
Model choices are determined by the validation set. The holdout remains untouched until the prespecified final evaluation.
Multiple testing
Testing many variants increases the chance of an attractive result occurring by chance. We therefore report the search process and apply corrections where appropriate.
Economic feasibility
Predicting direction correctly is not enough. Signals are also assessed after costs and with a realistic entry delay.
Reporting standard
A null result is not failed research. It reduces the scope for an incorrect conclusion.
Original findings, independent replications, limitations and example predictions therefore remain available side by side.