# Benchmark methodology Status: reproducible pre-optimization baseline, post-optimization candidate, core-hardening candidate, and stable-instruction-configuration full JIT/AOT candidate captured with the locked dependency set. ## Purpose and non-goals This suite is limited to operations that have a close native equivalent in `usecase_forge`, `bloc`, `riverpod`, `mobx`, and `redux`. It does not assign an overall winner. Architecture, guarantees, ergonomics, ecosystem maturity, and raw operation cost are separate questions. The suite is pure Dart. Flutter widget rebuilds, Flutter bindings, DevTools, networking, storage, serialization, and application business logic are outside this executable suite. Provider is not included: the suite is pure Dart, and an architecture-only row would not produce a comparable measurement. ## Locked inputs The benchmark package uses exact constraints and commits `pubspec.lock`: | Package | Role | Version in candidate lock | |---|---|---:| | `usecase_forge` | runtime under test | `0.1.0-dev.1`, temporary `path: ../..` | | `benchmark_harness` | process runner/warm-up helper | `2.4.0` | | `bloc` | compared runtime | `9.2.1` | | `riverpod` | compared runtime | `3.4.2` | | `mobx` | compared runtime | `2.6.0` | | `redux` | compared runtime | `5.0.0` | | `lints` | dev only | `6.1.0` | | `test` | dev only | `1.31.2` | The current stable versions are rechecked immediately before the final run. No arbitrary downgrade is allowed to make `usecase_forge` look better. After `usecase_forge` is published, the temporary path dependency must be replaced with the hosted pub.dev constraint for the exact release line under test, followed by `dart pub get` and a committed lockfile update. ## Native primitives used - `usecase_forge`: typed `UseCaseCommand`, registered handler, `UseCaseExecutionContext.publish`, snapshot stream, and bounded history. - `bloc`: event-based `Bloc` and `emit`; the common scenario uses command/action ingestion, so replacing Bloc with Cubit would change it. - `riverpod`: `NotifierProvider` hosted by a `ProviderContainer`. - `mobx`: direct `Observable` and `Action`; code generation is not needed for this minimal primitive. - `redux`: `Store`, reducer, and middleware for controlled async work. Each adapter keeps the package's native scheduling behavior. Redux's async `onChange` delivery is not changed to synchronous delivery. Bloc uses its default concurrent event handling. The adapter does not add a replay cache to Bloc or Redux. All long-lived adapters keep one internal listener used only to establish a deterministic completion barrier. Listener benchmarks therefore add the stated 1, 10, or 100 measured callbacks on top of one uniform tracking listener. This is disclosed because it is part of the measured runtime work. ## Scenario contract The exact mapping is in `scenario_matrix.md`. Important boundaries: - `sync_command_one_state` ends after the published state is observed. For `usecase_forge`, it is not presented as handler-only cost: admission, execution lifecycle, snapshots, and terminal processing that occur before the barrier remain real work. - `async_command_controlled_future` uses a `Completer`, with no timer or I/O. - `sequential_updates_10000` and `_100000` submit and await commands/actions in call order. They are not a burst-queue benchmark. - `concurrent_operations_10` and `_100` start all independent controlled operations before releasing their gates. The reported per-operation value is batch latency divided by operation count; raw batch ticks remain in JSON. - `subscribe_current` is measured only where the native API can immediately deliver the current value. A current-state getter plus a non-replaying stream is reported as `N/A`, not wrapped in a custom cache. - lifecycle is `N/A` for the measured MobX primitive because an Observable and Action have no equivalent runtime-wide close operation. - UseCase Forge overhead cases separately measure admission/pending queues, sequential blocking, rate-limit admission, and history reads/writes. Fixed 100/1,000/10,000-command bursts form a scaling study and retain total batch latency as well as normalized ns/operation. - Decomposed UseCase Forge cases separately measure the frozen-default and defaults-plus-registration instruction-resolution paths and the snapshot publisher with 1, 10, and 100 listeners. Publisher operations are actual timed callbacks, not commands. - High-cardinality input-conflict cases submit 100/1,000/10,000 unrelated `rejectNew` groups. A separate `replace` case keeps a large unrelated backlog while repeatedly replacing one group. These rows expose accidental whole- queue scans without changing the policy contract. - The busy-lane scheduler case holds 5,000 distinct sequential lane heads busy, then asks for work repeatedly. It measures whether already blocked heads are rescanned. The timed result retains total batch latency and normalized ns/operation. - The cancellation-storm case starts and cancels 5,000 executions and waits for terminal completion. It exercises cancellation bookkeeping as part of the real lifecycle; it is not a direct map microbenchmark. - The zero-listener publisher case measures accepted snapshot updates without callbacks. Together with the 1/10/100 callback rows it separates fixed publication cost from listener-delivery cost. `N/A` means “no sufficiently close operation in the measured native API”, not “the package cannot solve the broader application requirement”. ## Profiles The profiles tune harness cost, not semantics: | Profile | Warm-up per case | Independent samples | Typical use | |---|---:|---:|---| | `smoke` | 25 ms | 2 | harness correctness only | | `standard` | 250 ms | 5 | reviewed baseline | | `full` | 750 ms | 10 | higher-confidence publication run | The fixed 10,000/100,000 sequential-update scenarios stay fixed across profiles. Smaller setup-oriented cases use profile-specific iteration counts. The full profile repeats short command, listener, subscription, and lifecycle operations enough times to keep sub-millisecond samples from being treated as a stable release baseline. The 10/100-operation concurrency rows remain batch latency measurements and may naturally have higher dispersion. ## Timing and statistics 1. `benchmark_harness:bench` starts a dedicated JIT or compiled AOT process. 2. Each available case is warmed up using `AsyncBenchmarkBase.measureFor`. 3. Every sample constructs a fresh runtime and performs untimed preparation. 4. A `Stopwatch` measures only the workload; disposal and file output are outside the timed region. 5. Results are consumed by a process-level blackhole. 6. Current-state reads pass through the same `vm:never-inline` barrier for all adapters so the VM cannot hoist an invariant getter out of the timed loop. The UseCase Forge history getter is protected by the same kind of barrier. 7. JSON records every elapsed tick, total nanoseconds, and normalized ns/operation. 8. Summaries for both total time and normalized time include median, interpolated p95, min, max, mean, population standard deviation, and coefficient of variation. The Stopwatch suite does not claim allocation counts or GC pause attribution. The Dart standard library does not expose a stable allocation counter for this purpose. A separate VM-service trace may be added without changing these throughput rows; until then, allocation/GC explanations are hypotheses rather than measured causes. Five samples are the minimum baseline, not a claim of statistical certainty. For a public comparison, prefer the `full` profile, inspect raw distributions, repeat the entire matrix, and explain high dispersion rather than selecting a favorable run. The current hardening suite contains 82 rows: 79 measured and three explicit `N/A` rows in each runtime mode. ## Controlled environment Final JIT and AOT runs must share: - the same machine and CPU architecture; - the same Dart SDK and lockfile; - a declared power mode and stable thermal conditions; - no competing build, index, backup, or benchmark workloads; - the same benchmark commit/snapshot. Metadata records OS, CPU, processor count, Dart version, executable, Stopwatch frequency, package versions, repository commit when available, and the supplied power-mode label. An unavailable commit is reported as unavailable, never invented. ## Interpretation rules - Never write “package A is faster than package B” without naming the scenario, package versions, JIT/AOT mode, statistic, and machine context. - Do not extrapolate integer-counter throughput to real application latency. - A package performing additional lifecycle, cancellation, queue, or history work is not semantically equivalent to a raw state assignment. The common scenarios show observed cost; the feature matrix explains the extra work. - A low number does not establish correctness, maintainability, or suitability. - A high number does not justify weakening a contract. - Smoke output verifies the harness only and must not appear as a release claim. ## Reproduction See the package README for exact commands. Reviewers should retain both JSON and CSV, the lockfile, and the package source used for the run. Benchmarks are informational in the first release and are not CI pass/fail gates.