What you will do
You will know which report questions to ask, what replay establishes, and why completion, uncertainty, and validity are separate.
Before you start
- A completed report or a prepared sample result visible to your session.
- Access to the inputs, selected model definitions, and retained source evidence for any private workflow.
- The decision question and baseline against which the reported effects were calculated.
Step / 01
Read the question and inputs first
Check the intervention, baseline, population, geography, period, currency, and selected mechanisms. Identify which values were observed, generated, assumed, or unavailable. A place name does not establish representativeness, and a source link does not prove the model is appropriate for that source.
Population marginal fit, input readiness, evidence integrity, and empirical acceptance mean different things. A report can be technically complete and replayable while its mechanism remains unsuitable for your real-world decision.
Step / 02
Compare pairs and distributions
Baseline and intervention share a stochastic realization within each world pair. Independent replicas give a distribution under the same selected model. Read paired differences on matching metrics and periods, not just the largest displayed number from one branch.
Legacy analytics reports P10, P25, P50, P75, and P90 across worlds. Its directional probability is the fraction of pairs moving a metric in its declared worse direction: lower revenue or real income, for example. Ties do not count as worsening. Household and modular reports use their domain-specific paired measures and applicability rules.
These frequencies are conditional simulation results. They are not probabilities inferred from observed human behavior. Comparisons between separately configured runs may also change populations, parameters, or models, so their difference does not isolate a causal effect.
Step / 03
Ask which uncertainty was sampled
Replica spread and Monte Carlo standard errors describe stochastic variation under the declared model. Parameter sampling or calibration bootstrap describes a separate declared uncertainty treatment. Structural alternatives are explicit sensitivity cases, not automatically part of one predictive interval.
More replicas can stabilize a Monte Carlo estimate. They cannot repair incorrect mechanisms, missing historical observations, selection bias in household donors, or unknown real-world behavior. Check convergence against the stated per-metric criteria, and keep omitted uncertainty visible.
Step / 04
Inspect mechanism evidence and explanations
The legacy product-price detector looks for switching clusters over its declared social graph. It is not applicable to FX runs. Its synthetic tests check detector mechanics and evidence references; they do not establish real-world detection precision or recall.
Household and modular evidence can expose prices, resource/account changes, synthetic ledgers, event timing, and causal references. Read group coverage and any minimum-count exclusions. Conservation checks establish the modeled accounting; cash totals alone do not measure welfare or validate behavior.
Optional language-model explanations interpret structured numerical evidence after analysis. Invalid responses are rejected, and explanation failure is separate from numerical completion. Generated text does not change the simulation, and remote model revisions may not replay exactly.
Step / 05
Verify what can be reproduced
Managed runs record exact input hashes, canonical content, model/source and dependency identity, effective settings, and retained artifacts. A seed alone does not reproduce a run with changed inputs or runtime. Verification checks published bytes; exact replay also recomputes the compatible experiment.
Numerical completion, durable artifact publication, report readiness, explanation status, and calibration acceptance are independent questions. If storage failed after analysis, the finished calculation does not establish a verified durable report. Historical evidence without newer identity fields keeps that absence explicit.
Retain sources and compatible runtime for the chosen evidence period. Expiry, corruption, or erasure can make a previously completed report unavailable. Metadata and hashes identify what is missing; they cannot recreate deleted bytes.
- What exact baseline and intervention were compared?
- Which inputs and mechanisms support the reported metric?
- Which uncertainty was sampled, and which remains outside scope?
- Are source evidence and artifacts retained and verified?
- Did an observed-data assessment meet its predeclared criteria in the intended scope?