]> Sergey Matveev's repositories - public-inbox.git/commit
v2: improve deduplication checks
authorEric Wong (Contractor, The Linux Foundation) <e@80x24.org>
Wed, 18 Apr 2018 09:13:11 +0000 (09:13 +0000)
committerEric Wong (Contractor, The Linux Foundation) <e@80x24.org>
Wed, 18 Apr 2018 09:14:15 +0000 (09:14 +0000)
commitf0ef0a56a8957d6f3095b1a24798e54b0b815d04
treefcab14a29eaf1ec68564aa2163e31751f7e9936d
parent69329215485cf2ab9d8cd1fa7faf65d8ec42dc0b
v2: improve deduplication checks

First off, decode text portions of messages since some archived
mail I got was converted from quoted-printable or base-64 to
8bit by the original recipient.  Attempting to merge them with
my own archives (which had no conversion done) led to
unnecessary duplicates showing up.

Then, normalize CRLF line endings in text portions to LF.

In the headers, we relax the content_id hashing to ignore quotes
and lower-case domain names in To, Cc, and From headers since
some mail processors will alter them.

Finally, I've discovered Email::MIME->new($mime->as_string)
does not always round-trip reliably, so we calculate the
content_id twice on user-supplied messages.
lib/PublicInbox/ContentId.pm
lib/PublicInbox/V2Writable.pm
t/content_id.t