Completion Benchmark Guide¶

Use Tua's benchmark targets to answer three different questions:

1. Did completion ordering or semantic correctness change?
2. Did ordinary completion become slower?
3. Does completion stay fresh while project analysis is busy?

Choose A Benchmark¶

For a completion implementation change, start with:

make completion-ranking-smoke
make profile-completion

Run make representative-benchmark-smoke when the change affects snapshots, caching, scheduling, project indexing, cancellation, or watched files.

Prerequisites¶

Benchmark targets build the required Tua binary automatically. Local runs need:

• Rust and Cargo;
• Node.js;
• enough memory for generated 1,000-file fixtures when running the full representative profile.

Generated fixtures and JSON reports live under target/ and do not modify checked-in benchmark projects.

Profile Ordinary Completion¶

Run:

make profile-completion

The target builds the debug tua binary and profiles benchmarks/lsp/medium. It performs unmeasured warmup requests, then repeated measured requests in one LSP session. The default report is:

target/tua-speed/completion-profile.json

Read The Latency Numbers¶

Each request label reports:

completion is wall-clock time for textDocument/completion. completionResolve measures documentation/detail resolution for a selected completion item. End-to-end completion includes protocol, snapshot, semantic, candidate, ranking, and response-conversion work.

Separate Ranking From Total Time¶

analysis.completion.ranking is an internal profile span inside the full completion request:

• low ranking p95 with high completion p95 points outside ranking;
• high ranking p95 points to candidate scoring, project usage statistics, adaptive data, or sorting;
• no ranking span means internal profiling was disabled.

The printed medium-fixture reference targets are:

• completion p95 at or below 50ms;
• ranking p95 at or below 5ms.

These values are trend targets, not automatic failures for make profile-completion. The command fails on harness errors or when an explicit baseline comparison exceeds its tolerance.

Read Cache Statistics¶

The report includes raw tua/cacheStats counters.

A low hit rate is not automatically a bug on the first request. It becomes suspicious when repeated identical or prefix-compatible requests continue to miss without source or configuration changes.

The report's modelLane object is currently fixed to disabled with zero requests and fallbacks. It is reserved telemetry, not evidence that a network/model provider ran.

Compare Against A Baseline¶

Use a baseline from the same machine and keep baseline/current outputs separate:

TUA_COMPLETION_PROFILE_OUT=target/tua-speed/completion-baseline.json make profile-completion

TUA_COMPLETION_BASELINE=target/tua-speed/completion-baseline.json \
TUA_COMPLETION_PROFILE_OUT=target/tua-speed/completion-current.json \
make profile-completion

The comparison uses completionLatency.completion.p95Ms. By default, current p95 may be at most 5% slower than baseline, with at least 1ms of slack for very short runs.

Set TUA_COMPLETION_BASELINE_TOLERANCE to a decimal ratio such as 0.10 for 10%. Prefer increasing repeats before increasing tolerance.

Check Ranking Correctness¶

Run:

make completion-ranking-smoke

This is a deterministic correctness test, not a timing benchmark. It uses the checked-in medium fixture and fails when representative module-field or typed-local completion items move out of their required order.

Use it whenever candidate sources, scoring, prefix matching, usage statistics, or tie-breakers change.

Profile Representative Scale¶

The representative harness generates deterministic fixtures:

• single-1000: one 1,000-line source;
• single-5000: one 5,000-line source;
• project-100: a 100-file project;
• project-1000: a 1,000-file project.

Generate them without profiling:

make generate-lsp-benchmarks

Run the full release-build profile:

make profile-representative

The default report is target/tua-speed/representative-profile.json.

Cold, Warm, And Contended¶

Only main.tua is opened. Other project files must be discovered through the normal project model, so the profile exercises realistic indexing rather than pre-opening every provider.

The harness fails when:

• warm or contended completion p95 exceeds the configured budget;
• any stale result is observed after a document change;
• a semantic result validation fails.

Cancellation rate is informational. A low rate can mean background requests finished before cancellation was necessary; it is not itself a failure.

Smoke Versus Full Profile¶

make representative-benchmark-smoke uses only single-1000 and project-100 with reduced repeats and a 3-second completion p95 budget. The extra shared-runner allowance keeps this a CI reliability gate designed to catch stalls, stale results, and invalid responses without treating its two-sample p95 as a microbenchmark.

make profile-representative runs every fixture with more samples and a broad 10-second default budget. Use its report for scale comparisons and diagnosis, not as proof that a 9-second completion is acceptable. The performance-trend workflow raises only this full-profile ceiling to 30 seconds because shared GitHub runners can take roughly four times longer on the generated 5,000-line fixture. That runner allowance is not a local target or baseline.

Useful Environment Variables¶

Completion Profile¶

Representative Profile¶

Profiling scripts set TUA_COMPLETION_STATS=0 by default so benchmarks do not create or mutate fixture-local .tua/completion-stats.json files. Set it to 1 only when intentionally measuring persisted adaptive completion behavior.

Diagnose A Regression¶

CI And Reports¶

Pull requests run make benchmark-smoke. CI retains the example and representative smoke reports for 14 days.

The performance workflow runs make profile-all on main, weekly, and on demand. It uploads JSON reports under target/tua-speed/ for 90 days. Use those artifacts for trend investigation; use local same-machine baselines for strict before/after comparisons.