Freeze before results
Mission sets, policy rules and decision thresholds should be fixed before the result is known whenever the experiment is intended to support a prospective claim.
methodology / evidence boundaries
How AOPX measures provider performance.
AOPX does not rank providers on opinion. Benchmarks are built around frozen protocols, paired missions, explicit decision rules and reproducible artifacts. Production outcomes are tracked separately and are described as reported outcomes, not independently verified outcomes.
01 · Principles
The benchmark is designed to constrain hindsight.
Compare on the same missions
Providers are compared on paired tasks so differences are attributable to provider behaviour and path logic rather than different mission sets.
Keep evidence classes separate
LAB benchmarks, development experiments, shadow runs and replays do not become reported production outcomes.
02 · Protocol
The evaluation sequence.
01
Define the question
Specify the category, provider set, operating mode, metrics and decision rule before treating a benchmark as evidence.
02
Freeze the mission set
The missions used for the benchmark are fixed so the evaluated providers face the same task distribution.
03
Freeze the policy
The routing or provider-selection policy is fixed for the run. Benchmark conclusions are not retroactively rewritten to make the policy look stronger after results are visible.
04
Run paired provider calls
Each provider is tested against the same missions. This allows fixed-provider baselines, fallback paths and policy results to be compared on shared tasks.
05
Apply the registered rule
GO, PIVOT or STOP is determined using the rule defined for that benchmark rather than by changing the success bar after the result is known.
06
Freeze the artifacts
Mission, results, report and matrix files receive stable SHA-256 hashes so a published number can point to a specific artifact set.
07
Interpret conservatively
Post-hoc analyses are allowed for diagnosis, but they are labelled as post-hoc and do not replace the prospective evidence from the frozen protocol.
03 · LAB-003.5
The current frozen SEARCH benchmark.
LAB-003.5 used 150 fresh SEARCH missions, split across 50 ECONOMY, 50 BALANCED and 50 CRITICAL missions. Each mission produced paired provider calls, for 300 provider calls in total.
For CRITICAL, the measured AOPX policy reached 40/50 final successes, while Tavily fixed reached 37/50.
A later post-hoc Brave → Tavily cascade also reached 40/50. This is important because it limits the causal claim that can be made from the benchmark.
| Path | Successes | Rate | Evidence status |
|---|---|---|---|
| AOPX CRITICAL policy | 40 / 50 | 80% | Measured benchmark result |
| Tavily fixed | 37 / 50 | 74% | Fixed-provider baseline |
| Brave → Tavily | 40 / 50 | 80% | Post-hoc diagnostic cascade |
Current interpretation: LAB-003.5 supports fallback and provider redundancy more strongly than a claim that contextual provider selection alone caused the six-point gap versus Tavily fixed.
04 · Evidence classes
Benchmark evidence is not production evidence.
Benchmark evidence
Generated under a defined LAB protocol. Useful for comparing provider behaviour, fallback logic and policy hypotheses under controlled conditions.
Reported production outcomes
Returned after a real production recommendation and real external provider execution. These records are reported by the caller and are not labelled independently verified.
LAB, test, shadow and replay traffic is excluded from public production outcome totals. Production Vendor Memory is also kept separate from LAB data.
05 · Holdouts
Freshness matters, and so does whether the data stayed untouched.
A prospective holdout is strongest when the missions were not used to design, tune or inspect the policy before evaluation.
LAB-004A explored query-aware routing using a frozen 100-mission paired dataset, but that dataset had already been inspected. For that reason, LAB-004A is treated as development evidence only and did not trigger a production policy change.
Future claims about query-aware routing require a new untouched evaluation after the policy is frozen.
06 · Reproducibility
Every frozen number should point to a frozen artifact.
RUN LAB0035-E33AC6B7B4
MISSION c54798acbdf17ae99a911436c54755f6b02e905c9086ead9d835a36c2693b5cd
RESULTS 550096674fe5045bcbaa06eb1ed7dff298745380730a49b5262740f4d2cac075
REPORT 978ac6f13de0fb26890277070acfb7e45c98b972f3884fd72600c21ab72aa15a
MATRIX 1846d8361c71d16b2f0ad65f09d81625acee66fc2999ebb3a476a8496988d218
Hashes identify the frozen artifact set. They do not expose the private decision engine, policy internals or production Vendor Memory.
07 · Production
The live system has stricter evidence boundaries.
Known recommendation required
A reported production outcome must reference a recommendation that AOPX actually issued.
One outcome per recommendation
The same recommendation cannot be used to create multiple production outcomes.
Attempt path validation
Reported attempts are checked against the provider path returned by AOPX before the outcome is accepted.
08 · Limits
What this methodology does not prove.
- A benchmark result does not prove that one provider is universally superior.
- A post-hoc analysis is not equivalent to a prospective test.
- A reported production outcome is not independently verified merely because it appears in the public ledger.
- LAB-003.5 does not isolate contextual selection as the sole cause of the measured CRITICAL result.
- LAB-004A does not establish query-aware routing superiority and did not change production.
- Current live production scope is SEARCH with Brave Search and Tavily Search.
LAB-003.5 · SEARCH
Open public evidence →
150 fresh missions · 300 paired calls · frozen artifacts · public evidence trail.