Home
Releases
Params
Notes
Code
Download
Notes 2026/07/29: RTS performance in Comet v2026.02.1
See notes/2024100_FI for the original Comet-FI notes and notes/20260610_FI for v2026.02.0 notes on Comet-FI.
These reported search times/speeds are the theoretical maximums. Previously reported throughput times were slightly affected by file I/O spectrum reading, dampening the reported search speeds. Reported times below are based on pre-loading all spectra into memory so that search throughput is not gated by the slower file parsing.
All timings are run on a Dell Windows 11 computer with 64GB RAM and an Intel Core Ultra 7 265K processor.
From an RTS perspective, the v2026.02.1 release of Comet added performance optimizations to the old peptide index format. This includes multithread support (running concurrent searches) and using the fragment-ion index (FI) peptide generation (and modification permutation) code for optimized index file generation. In order to support concurrent PI searches, the peptide index is now loaded into memory. There is a memory cost associated with loading the peptide index in memory but it’s significantly less than the corresponding FI search.
Real-time search (RTS) throughput and latency for Comet’s two index backends: the fragment-ion index (FI) and the peptide index (PI), searched against a human canonical target/decoy FASTA with variable oxidized methionine and phospho-STY, at 1, 2, 4, 8, and 20 search threads.
Comet RTS peptide index vs. fragment ion index performance:
- Comet version: 2026.02 rev. 1
- Query file: 20240924_Hela_01.raw · 89,593 total scans, 63,488 MS2 spectra
- CPU: Intel Core Ultra 7 265K (8 P-cores + 12 E-cores = 20 threads, no hyperthreading)
- Database: human canonical target+decoys (40,908 sequences)
- Modifications: 16M and 80STY, allowing up to 3 modifications per peptide
- Peptide mass range: 700.0 to 5000.0
- Peptide length range: 7 to 50
- AScorePro localization: turned on to localize all modified residues
| Search threads |
Avg. search time (ms) | Avg. search speed (Hz) | Peak memory (GB) | |||
|---|---|---|---|---|---|---|
| PI | FI | PI | FI | PI | FI | |
| 1 | 13.25 | 0.74 | 75 | 1,356 | 11.7 | 20.2 |
| 2 | 6.79 | 0.38 | 147 | 2,651 | 11.7 | 20.2 |
| 4 | 3.40 | 0.19 | 294 | 5,209 | 11.8 | 20.3 |
| 8 | 1.82 | 0.11 | 548 | 8,991 | 11.9 | 20.4 |
| 20 | 0.79 | 0.08 | 1,259 | 12,681 | 12.2 | 20.7 |
Per-run full summary statistics
Scroll horizontally to see all columns →
| Index | Threads | Spectra | Avg ms | Hz | Max ms | %≤1ms | %≤5ms | %>10ms | Mem GB | Init s | Preload s | Search s | Total s |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FI | 1 | 63,488 | 0.74 | 1,356 | 17.0 | 96.67% | 99.96% | 0.02% | 20.2 | 64.4 | 4.02 | 46.8 | 115.5 |
| FI | 2 | 63,488 | 0.38 | 2,651 | 21.0 | 96.37% | 99.93% | 0.03% | 20.2 | 64.6 | 4.06 | 24.0 | 92.9 |
| FI | 4 | 63,488 | 0.19 | 5,209 | 20.0 | 96.20% | 99.91% | 0.03% | 20.3 | 64.4 | 4.20 | 12.2 | 81.1 |
| FI | 8 | 63,488 | 0.11 | 8,991 | 24.0 | 93.35% | 99.79% | 0.09% | 20.4 | 65.4 | 4.09 | 7.1 | 76.9 |
| FI | 20 | 63,488 | 0.08 | 12,681 | 32.0 | 74.42% | 97.88% | 0.35% | 20.7 | 65.0 | 4.14 | 5.0 | 74.5 |
| PI | 1 | 63,488 | 13.25 | 75 | 406.0 | 11.15% | 48.64% | 29.72% | 11.7 | 30.3 | 4.36 | 841.5 | 876.4 |
| PI | 2 | 63,488 | 6.79 | 147 | 420.0 | 10.53% | 47.28% | 30.65% | 11.7 | 30.3 | 3.94 | 431.1 | 465.7 |
| PI | 4 | 63,488 | 3.40 | 294 | 422.0 | 10.32% | 46.73% | 30.93% | 11.8 | 29.6 | 4.27 | 215.8 | 250.0 |
| PI | 8 | 63,488 | 1.82 | 548 | 417.0 | 8.43% | 41.72% | 34.25% | 11.9 | 29.5 | 4.04 | 115.9 | 149.7 |
| PI | 20 | 63,488 | 0.79 | 1,259 | 459.0 | 7.87% | 38.40% | 37.64% | 12.2 | 30.0 | 4.00 | 50.4 | 84.7 |
Comparison of release v2026.02.0 vs v2026.02.1 in both batch mode and RTS
The RTS results above show the peptide-index (PI) and fragment-ion-index (FI) backends within a single v2026.02.1 build. The two comparisons below instead hold the index backend fixed (FI, batch/offline mode, not RTS) and compare the previous release, v2026.02.0, against v2026.02.1, to see what the v2026.02.1 batch-search optimizations were worth in practice. No AScorePro localization is applied in these following v2026.02.0 vs v2026.02.1 comparisons.
Run 1 — wider modification search space, batch search
- Database: human.canonical.target-decoy.fasta (FI index)
- Params:
20260420-human-phosho/comet.params— M/STY variable mods, up to 3 modifications per residue type and 3 total per peptide, peptide mass 700–5000, length 7–50, 8 search threads, explicitfragindex_*tuning - Query file: 20240924_Hela_01.raw · 63,488 MS2 spectra, ~1.2 GB
- Both versions were run concurrently (20 cores available, 8 threads/process, so neither run was competing with the other for CPU)
| Metric | v2026.02.0 | v2026.02.1 |
|---|---|---|
| Index build time | 14s | 15s |
| Index build peak memory | 2.6 GB | 3.2 GB |
| Search total time (incl. FI generation) | 2m40s | 1m25s |
| — FI generation sub-time | 1m0s | 1m2s |
| — actual search phase | 1m35s (661 Hz) | 18s (3,392 Hz) |
| Search peak memory | 19.7 GB | 20.6 GB |
| PSMs — XCorr, 1% FDR | 17,569 | 17,740 |
| PSMs — XCorr, 5% FDR | 23,748 | 23,871 |
| PSMs — E-value, 1% FDR | 20,519 | 20,545 |
| PSMs — E-value, 5% FDR | 25,761 | 25,842 |
Run 2 — narrower modification search space, batch search
- Database: human.canonical.target-decoy.fasta (FI index)
- Params:
data/comet_phospho.params— M/STY variable mods, up to 2 modifications per residue type - Query file: 20250520_Hela_06.raw · 49,844 MS2 spectra
- Both versions were built fresh from their respective release tags via a git worktree + MSBuild Release/x64 build
| Metric | v2026.02.0 | v2026.02.1 |
|---|---|---|
| Index build time | 10s | 9s |
| Index build peak memory | 1.9 GB | 1.8 GB |
| Search time (incl. FI generation from .idx) | 54s | 51s |
| — FI generation sub-time | 41s | 39s |
| — actual search phase | 11s (4,382 Hz) | 10s (4,881 Hz) |
| Search peak memory | 13.0 GB | 14.0 GB |
| PSMs — XCorr, 1% FDR | 16,352 | 16,428 |
| PSMs — XCorr, 5% FDR | 19,789 | 19,874 |
| PSMs — E-value, 1% FDR | 17,681 | 17,757 |
| PSMs — E-value, 5% FDR | 20,628 | 20,625 |
Why v2026.02.1 is so much faster here
Comet PR #115, “Batch search FI optimization,” changed how the FI backend processes a batch search. The old approach (still used in v2026.02.0) ran in three separate sweeps over all spectra: preprocess everything, then search everything, then post-process everything, with each sweep reading its intermediate results back out of main memory. v2026.02.1 instead fuses that into a single per-spectrum pipeline, where each worker thread carries one spectrum all the way through preprocessing, searching, scoring, E-value calculation, delta-Cn, and AScore in one pass, using scratch buffers that stay local to that thread.
Three follow-up commits tightened that same fused pipeline further:
- 59b0f000 replaced the old pattern of allocating and freeing a sparse XCorr matrix for every spectrum with a per-thread bump allocator. The commit’s own benchmark on a 326K-spectrum FI sweep showed 71–214% higher throughput, with the biggest gains on larger batch sizes.
- c639d8be removed a shared-mutex bottleneck where worker threads were contending to hand off results in the fused loader.
- d2c6f54d added asynchronous spectrum readahead, so file I/O no longer stalls a worker thread that’s ready to search.
Why Run 1 shows a much bigger speedup than Run 2: the fused pipeline’s advantage comes from cutting out round-trips through memory and per-spectrum heap allocation — overhead that grows with how much work each spectrum requires. Run 1’s parameters (3 modifications per residue type, 3 total) build a much larger FI index (roughly 3.6–3.7 billion entries, versus about 2.3 billion for Run 2) and do substantially more candidate matching per spectrum, so the old three-sweep approach’s overhead adds up much faster there. Run 2’s narrower search does less work per spectrum, so both versions’ per-spectrum overhead is small enough that the architectural difference barely shows up (4,382 Hz vs. 4,881 Hz).
Bottom line: the roughly 5x FI batch-search speedup seen in Run 1 for v2026.02.1 is expected, and it’s the direct, intended result of the optimization work in PRs #115, #119, and #120 — not a bug or inconsistency in either version.
RTS-mode FI search: v2026.02.0 vs v2026.02.1
The comparison above drives Comet’s batch search engine (comet.exe). This
section repeats the same v2026.02.0-vs-v2026.02.1 comparison, but driving the
real-time-search API instead (RealtimeSearch.exe, the same console harness
used to generate the PI-vs-FI table earlier in this page), against the exact
same two .raw files and the exact same FI indexes as Run 1 and Run 2 above
— this time swept across the same 1/2/4/8/20 search-thread counts used
in that first table. AScorePro localization was switched off in both versions
so that neither run was doing extra scoring work the other wasn’t.
One methodology note: v2026.02.1’s RTS harness measures its own average search speed differently than v2026.02.0’s does internally, so the two versions’ self-reported Hz numbers aren’t directly comparable to each other. To keep the comparison fair, the “search phase” times and speeds below are instead computed the same way for both versions: total wall-clock run time, minus the one-time fragment-ion-index generation step that happens once at startup.
Run 1 — wider modification search space, RTS search
- Query file: 20240924_Hela_01.raw · 63,488 MS2 spectra (same as the PI-vs-FI table above)
- Database/mods: same wide-mod-space FI index as Run 1’s batch comparison (M/STY up to 3 mods/type, 3 total, mass 700–5000, length 7–50)
Scroll horizontally to see all columns →
| Search threads |
FI generation (s) | Search phase (s) | Search speed (Hz) | Peak memory (GB) | PSMs, XCorr 1% FDR | PSMs, E-value 1% FDR | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| v0 | v1 | v0 | v1 | v0 | v1 | v0 | v1 | v0 | v1 | v0 | v1 | |
| 1 | 62.7 | 62.5 | 64.1 | 48.3 | 990 | 1,314 | 21.4 | 20.2 | 17,847 | 17,742 | 20,444 | 20,530 |
| 2 | 62.7 | 64.0 | 38.3 | 26.8 | 1,657 | 2,367 | 21.5 | 20.2 | 17,847 | 17,742 | 20,445 | 20,530 |
| 4 | 64.2 | 62.4 | 24.8 | 16.2 | 2,556 | 3,926 | 21.4 | 20.3 | 17,847 | 17,742 | 20,446 | 20,530 |
| 8 | 63.2 | 64.5 | 13.7 | 10.3 | 4,641 | 6,152 | 22.9 | 20.4 | 17,847 | 17,742 | 20,445 | 20,530 |
| 20 | 63.7 | 62.4 | 12.9 | 9.3 | 4,914 | 6,812 | 23.5 | 20.7 | 17,847 | 17,742 | 20,446 | 20,530 |
Run 2 — narrower modification search space, RTS search
- Query file: 20250520_Hela_06.raw · 49,844 MS2 spectra
- Database/mods: same narrow-mod-space FI index as Run 2’s batch comparison (M/STY up to 2 mods/type, mass 800–5000, length 8–50)
Scroll horizontally to see all columns →
| Search threads |
FI generation (s) | Search phase (s) | Search speed (Hz) | Peak memory (GB) | PSMs, XCorr 1% FDR | PSMs, E-value 1% FDR | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| v0 | v1 | v0 | v1 | v0 | v1 | v0 | v1 | v0 | v1 | v0 | v1 | |
| 1 | 41.1 | 41.7 | 49.9 | 40.4 | 999 | 1,233 | 14.2 | 12.9 | 16,834 | 16,428 | 17,664 | 17,757 |
| 2 | 41.4 | 40.8 | 27.3 | 22.0 | 1,829 | 2,268 | 14.3 | 12.9 | 16,834 | 16,428 | 17,685 | 17,757 |
| 4 | 41.3 | 40.7 | 16.1 | 13.1 | 3,094 | 3,808 | 14.9 | 13.0 | 16,834 | 16,428 | 17,663 | 17,757 |
| 8 | 41.3 | 41.6 | 10.5 | 9.3 | 4,734 | 5,389 | 15.5 | 13.1 | 16,834 | 16,428 | 17,663 | 17,757 |
| 20 | 41.0 | 40.8 | 8.5 | 7.2 | 5,885 | 6,952 | 16.4 | 13.3 | 16,834 | 16,428 | 17,685 | 17,757 |
(v0 = v2026.02.0, v1 = v2026.02.1, colored to match the PI/FI palette used above — that’s purely a color reuse, not a claim about index backend.)
Isolating pure search-call time from spectrum reading
The tables above measure wall-clock time for the whole per-scan loop each
worker thread runs, which is the honest end-to-end throughput of this
benchmark harness — but RTS itself is properly defined as “submit a
spectrum, get results back” (DoSingleSpectrumSearchMultiResults), and that
call’s own runtime shouldn’t depend on how or when the spectrum’s peak
arrays were read from a file. Since this harness reads spectra out of a
static .raw file to simulate real-time arrival (rather than receiving them
from a live instrument feed), it’s worth asking how much of the gap above is
the search engine itself getting faster, versus an artifact of how this
particular tool simulates spectrum delivery.
Both versions’ C# harness already time-boxes exactly the search call, per
scan, with a thread-local Stopwatch that never touches shared state
(RealtimeSearch/SearchMS1MS2.cs) — so summing those durations and
dividing by thread count gives an estimate of pure search-call throughput,
with all spectrum-reading time excluded entirely. v2026.02.0 already computes
and prints exactly this (its self-reported “MS2 average search time”/Hz).
v2026.02.1 had switched to reporting true wall-clock time instead, so it
didn’t have this number — a few lines were added to
SearchMS1MS2.cs to accumulate the same sum-of-calls figure there too, and
both versions were rerun across the full thread sweep with that
instrumentation in place.
Scroll horizontally to see all columns →
| Search threads |
Run 1 (wide mods) full wall-clock Hz |
Run 1 (wide mods) pure search-call Hz |
Run 2 (narrow mods) full wall-clock Hz |
Run 2 (narrow mods) pure search-call Hz |
||||
|---|---|---|---|---|---|---|---|---|
| v0 | v1 | v0 | v1 | v0 | v1 | v0 | v1 | |
| 1 | 990 | 1,314 | 1,363 | 1,412 | 999 | 1,233 | 1,300 | 1,351 |
| 2 | 1,657 | 2,367 | 2,618 | 2,769 | 1,829 | 2,268 | 2,547 | 2,606 |
| 4 | 2,556 | 3,926 | 4,494 | 5,094 | 3,094 | 3,808 | 4,659 | 5,021 |
| 8 | 4,641 | 6,152 | 7,163 | 8,717 | 4,734 | 5,389 | 7,105 | 8,849 |
| 20 | 4,914 | 6,812 | 9,686 | 10,864 | 5,885 | 6,952 | 9,203 | 13,162 |
Turning those into v2026.02.0→v2026.02.1 percentage deltas makes the pattern clearer:
| Search threads | Run 1 wall-clock Δ | Run 1 pure search-call Δ | Run 2 wall-clock Δ | Run 2 pure search-call Δ |
|---|---|---|---|---|
| 1 | +32.7% | +3.6% | +23.4% | +3.9% |
| 2 | +42.8% | +5.8% | +24.0% | +2.3% |
| 4 | +53.6% | +13.4% | +23.1% | +7.8% |
| 8 | +32.6% | +21.7% | +13.8% | +24.5% |
| 20 | +38.6% | +12.2% | +18.1% | +43.0% |
This changes the picture in an important way. At low thread counts, most of the gap disappears once I/O is excluded: Run 1 at 1 thread goes from +32.7% (full wall-clock) down to +3.6% (pure search-call); Run 2 at 2 threads goes from +24.0% down to +2.3%. That confirms the suspicion behind this section — a good chunk of the originally-reported speedup was this benchmark harness’s read strategy, not the search engine getting faster, and in a live deployment (where spectra arrive already-parsed from the instrument’s own acquisition callback, with no shared file reader for threads to contend over) that portion of the gap likely wouldn’t appear at all.
But a real, thread-count-dependent gap remains even with I/O fully excluded, and it grows at higher concurrency: +21.7% (Run 1) / +24.5% (Run 2) at 8 threads, +12.2% / +43.0% at 20 threads — versus single digits at 1 thread. A gap that scales up with thread count, inside a timer that only wraps the search call itself, points at contention inside the search engine’s own threaded code, not at file I/O. That lines up with two fixes already covered in the batch comparison further up this page, both of which touch the same per-spectrum scoring path RTS calls into: c639d8be (removed a shared-mutex bottleneck in result hand-off) and 59b0f000 (replaced per-spectrum heap alloc/free of the sparse XCorr matrix with a per-thread bump allocator). Mutex contention and heap-allocator contention both get worse as more threads compete for them, which is exactly the pattern in this table — so this residual gap looks like a genuine core-engine improvement, not a benchmark artifact.
Revised bottom line: the original “same underlying idea as batch’s I/O decoupling” explanation for RTS’s speedup was only half the story. Read strategy (preload vs. interleaved-with-search) explains most of the gap at low thread counts; a real, separate engine-level fix (shared-mutex and allocator contention, worse at high thread counts) explains the rest, more of it at 8–20 threads. Neither of those was captured by looking at full wall-clock time alone.
Peak memory tells a different story from batch, worth calling out rather than glossing over: batch’s peak memory went up slightly from v2026.02.0 to v2026.02.1 (its fused pipeline holds a thread-local scratch buffer per thread), while RTS’s peak memory went down by 1–3 GB in both runs here, and the gap barely moves across the thread sweep. That’s consistent with v2026.02.1’s RTS harness trading a large “read every scan up front” preload buffer for whatever per-scan allocation pattern v2026.02.0 used, but this hasn’t been traced through the allocator code directly, so treat it as an observation rather than a confirmed root cause.
XCorr-based PSM counts are exactly thread-count-invariant within each version (17,847 for every v2026.02.0 row in Run 1, 17,742 for every v2026.02.1 row, and likewise 16,834 / 16,428 in Run 2) — as they should be, since thread count only changes how fast spectra are processed, never which peptide wins. E-value-based counts are effectively the same story, but with a small (≤0.15%, ~20 PSMs) run-to-run wobble visible only in v2026.02.0 (20,444–20,446 in Run 1, 17,663–17,685 in Run 2, depending on thread count); v2026.02.1 comes out perfectly identical across all five thread counts in both runs. That’s a nice independent confirmation of the RTS e-value determinism fix described elsewhere in this repo’s history: the older version still shows the residual thread-count-linked jitter, the current version doesn’t.
PSM counts also track the batch numbers closely (e.g. Run 1, XCorr at 1% FDR: 17,847/17,742 PSMs here for RTS v2026.02.0/v2026.02.1, versus 17,569/17,740 for batch in the same two versions above), and the small differences between RTS and batch, or between v2026.02.0 and v2026.02.1 within RTS, are all within the same low-single-digit-percent range seen everywhere else in this document — not evidence of a correctness problem in either version.