The listening log that mostly listened twice

A believable Spotify history turned out to be 68.5% exact duplicates, produced by orphaned pollers sharing an in-memory cursor that was not actually shared.

Symptom

The play log looked healthy: lots of rows, plausible timestamps, familiar tracks. The first suspicious result only appeared when the data was used for streak and session analysis. Runs were implausibly repetitive.

Wrong assumption

Each polling process kept a cursor in memory, so it appeared that a play could only be recorded once. That was true inside one process and irrelevant once more than one poller existed.

Actual cause

Orphaned concurrent polling processes were writing to the same JSONL log. Each had its own perfectly reasonable, completely independent memory of the last seen play. The resulting file was 68.5% exact duplicates.

Fix

Deduplication moved to file-backed state: the durable log, not a process-local cursor, became the authority. The existing history was cleaned before any derived statistics were trusted.

Prevention

Treat data quality as a first-class observable. A collector now needs invariants—duplicate rate, monotonic timestamps, and single-writer expectations—not merely a green process and a growing file.

Lesson

Plausible data is more dangerous than obviously broken data. If a metric feeds a charming visualization, validate the boring properties before believing the charming result.