A practical guide
How do you read a research result?
Do not start with the highest percentage. A credible conclusion requires a clearly defined question, a final test, an appropriate baseline, an assessment of uncertainty and realistic costs.
The right order
Six questions to ask before accepting a conclusion.
Following this order helps prevent an impressive number from carrying more weight than the research supports.
-
01
What question was specified in advance?
A good question specifies the information used, the outcome to be predicted and the time horizon. Changing these after seeing the results increases the risk of chance findings.
-
02
Was there a genuine holdout?
Development data helps a model learn. A holdout is a later, reserved period that cannot be used to adjust any modelling choices. Only this final test provides a fair assessment.
-
03
Is the sample large and representative enough?
More observations can reduce random noise, but a thousand nearly identical minutes are not equivalent to a thousand independent events. Consider the companies, time periods and market regimes represented as well.
-
04
Does the model beat a simple baseline?
Directional accuracy above 50% can still be weak if always predicting an upward move achieves 52.9%. The relevant comparison is the best simple alternative, not just chance.
-
05
How wide is the uncertainty interval?
A point estimate is a single number; an interval shows the range of values compatible with the data. If the interval for a difference includes zero, the apparent advantage may be due to chance.
-
06
Does the result survive costs and replication?
A statistical pattern must remain economically useful after transaction costs. Repeating the test on different data then provides a stronger check against a chance initial finding.
An actual example from this archive
49.88% is not ‘almost good enough’.
In the independent replication, the text model predicted the direction correctly in 49.88% of cases. The simple baseline achieved 52.90%, while the model’s 95% interval was 47.20–52.51%.
The appropriate conclusion is not that the model narrowly missed the mark, but that this final test found no evidence of additional predictive value.
Read the full replication studyStatus colours
At a glance, always backed by evidence.
Data processing or the final test is still under way; there is no final conclusion yet.
An initial effect that warrants further research, but still lacks an uncertainty assessment or independent replication.
The final test found no convincing improvement over the reference model.
A gross pattern did not remain economically viable after realistic costs.
Only the study design has been prepared. The research has not started and no results are available.