Deduplicating the post while keeping both query hits

One post can answer two searches#

Two collected results can have the same post_id and different query IDs. Their text repeats, but the two discoveries still carry different information. footie represents that distinction with raw and normalized rows keyed by post ID, plus query-hit rows keyed by post ID and query ID.

The normalizer checks a batch-local set of post IDs before constructing those rows. Once it has seen a post, it skips the next item altogether, including a new query match. The storage key supports both discoveries, but the earlier filter prevents the second one from reaching it.

Apply each filter to the thing it identifies#

Post deduplication and query-hit deduplication need different keys. The optional text-hash filter makes a third decision: it removes URLs, mentions, hashtags and punctuation before hashing. That can merge texts with different original context, which is why it should be understood as a content filter rather than a security control. The export leaves it off by default.

The writes need checking independently too. Raw posts, normalized posts and hits go through separate upserts. Stable keys help a retry find the same rows, but they don't make partial failure disappear.

I'd supply one post under two query IDs and assert one raw row, one normalized row and two hits. Then I'd interrupt each write and replay the batch. These remain proposed regressions: this note is based on the exported logic, not a workflow run or an injected storage failure.