extindex: support --dedupe[=MSGID]
Sometimes I just want to dedupe a single Message-ID to test
something, and this lets me do it.
This patch appears to do what its supposed to. But it also
appears to be finding duplicates that were previously missed.
That's a good thing, but I wish I understood what seems to be
fixed :x
I'm not sure why the previous ExtSearchIdx.pm (blob
357312b8)
was causing messages to be missed, even, and why this patch
seems to fix it... And it's not infinite looping, either.
Anyways, before this patch, "-extindex --dedupe" was taking ~5
min to no-op every message (after the initial full --dedupe run
which took over a day to run). No-op --dedupes now take just
under 2 hours to scan every single cross-posted message for a
no-op dedupe. The initial dedupe took nearly 44 hours on my
system for <https://yhbt.net/lore/all/> due to SATA-2 TLC SSD
latency on 3 gigantic Xapian shards.
Running --dedupe with this change seems to prevent
/BUG\?.*?not deduplicated properly/ stderr messages from being
triggered by View.pm. Current versions of -extindex do not
seem susceptible to introducing duplicates.