Validation on synthetic ground truth

Numbers for the default parameters, measured against data where the answer is known. They are published so that you can decide whether this tool is adequate for your recordings, and they include the levels at which it is not.

How the test data is made#

• Templates. Twelve mean waveforms extracted from three real tetrode sessions, four channels each, held fixed and scaled to hit a target amplitude.
• Noise. Real recording segments with all detected events cut out and the gaps filled with time-reversed noise from the same channel. Synthetic Gaussian noise makes every detector look better than it is, because real noise is neither white nor stationary at short scales.
• Signal-to-noise ratio. Defined here as the absolute negative peak of the template divided by σ, the robust noise estimate on that channel. Other papers divide by 2σ or use peak-to-peak; a ratio of 6 here is a ratio of 3 under that convention.
• Spike trains. Six units per repetition drawn from the pool of twelve templates, Poisson with a 2 ms refractory period, rates drawn uniformly between 2 and 18 per second, 600 s per repetition.
• Repetitions. Twenty per noise level, seeds 0 to 19.
• Matching. A detected event matches a ground-truth spike if it falls within ±0.5 ms of it; each truth spike can be matched once.

python -m axonforge.bench --snr 2,3,4,6,8,10 --reps 20 --out bench.csv
# 26 min on 4 cores; writes one row per repetition

Detection#

Read the first row as a warning rather than a result. At a ratio of 2 the detector finds two spikes in five and is wrong about three of every ten it reports; no downstream curation rescues that, and a unit whose amplitude is twice the noise should be reported as undetected rather than as a low-rate cell. The time error at that level, 95 µs, is three samples at 30 kHz, which is enough to blur a cross-correlogram beyond use.

Effect of the threshold multiplier#

F1 is identical at 4.0 and 4.5, so the default was not chosen by maximising it. It was chosen because the error it leaves is the recoverable kind: a missed spike lowers a rate estimate, while a false positive at 4.0 — twice as many as at 4.5 — joins a cluster and corrupts both its waveform and its interval statistics. If your question is about rates rather than about shapes, 4.0 is the better default and you should change it.

Sorting#

A unit counts as recovered when a single cluster contains at least 80 % of its spikes and no more than 20 % contamination from others. The one merge at a ratio of 10 is the pair of templates in the pool whose waveforms differ mainly in the second principal component; with three components they separate in eighteen of twenty repetitions, and with four they separate in all twenty at the cost of a split elsewhere.

Overlapping spikes#

Two units were forced to fire at a controlled separation, 2 000 pairs per separation, at a ratio of 6.

The third column explains the shape of the curve. With a 1.0 ms dead time a second crossing on the same channel cannot be recorded at all, so everything recovered below that separation was detected on a different channel of the group. That is why the curve rises gradually from 0.3 ms instead of stepping at 1.0 ms, and why the recovery depends on how far apart the two cells sit relative to the electrode.

Shortening the dead time does not fix this. At dead_time_ms=0.4 overlap recovery at 0.5 ms rises to 0.74, and precision at a ratio of 6 falls from 0.98 to 0.91, because the trailing edge of large spikes starts producing second events. Resolving genuine overlaps needs template subtraction, which axonforge does not implement.

What these numbers do not cover#

• Templates are static. Real bursting cells lose amplitude within a burst, which smears a cluster along its principal axis. Nothing here measures that.
• No drift. The templates do not move. A session with electrode movement will do worse than any row above, and the published figures should not be quoted as if it would not.
• Linear superposition. Overlapping spikes are added. Real extracellular potentials are close to additive but not exactly so.
• Four channels. Nothing here speaks to probes with tens of closely spaced contacts, where the right algorithm is a different one.
• Shared code between detector and matcher. Both use the same alignment routine. A bug in it could make the matcher forgiving in exactly the way the detector is wrong, and this benchmark would not reveal it. An independent re-implementation of the matcher is the obvious next piece of work and has not been done.