Eric Wong (Contractor, The Linux Foundation) [Mon, 19 Mar 2018 08:14:42 +0000 (08:14 +0000)]
import: (v2): write deletes to a separate '_' subdirectory
In the future, we may store "purged" content IDs or other
uncommon stuff under "_/" of the git tree. This keeps the
top-level tree small and more amenable to deltafication.
This helps the the common case where "m" is most commonly
changed file at the top level.
Also, use 'D' instead of 'd' since it matches git's '--raw'
output format.
Eric Wong (Contractor, The Linux Foundation) [Mon, 19 Mar 2018 08:14:41 +0000 (08:14 +0000)]
import: (v2) delete writes the blob into history in subdir
This makes it easier to audit deletes with "git log -p" and
prevents an unstable specification of "content_id" from being
stored in history.
This should be cost-free if done in the same partition (and even
cheaper than before as it introduces no new blobs). It does
have a higher cost across partitions, but is probably irrelevant
given the typical ham:spam ratio.
Eric Wong (Contractor, The Linux Foundation) [Mon, 19 Mar 2018 08:14:34 +0000 (08:14 +0000)]
v2writable: support "barrier" operation to avoid reforking
Stopping and starting a bunch of processes to look up duplicates
or removals is inefficient. Take advantage of checkpointing
in "git fast-import" and transactions in Xapian and SQLite.
Eric Wong (Contractor, The Linux Foundation) [Mon, 19 Mar 2018 07:51:09 +0000 (07:51 +0000)]
extmsg: rework partial MID matching to favor current inbox
The current inbox is more important for partial Message-ID
matching, so we try harder on that to fix common errors before
moving onto other inboxes. Then, prevent expensive scanning of
other inboxes by requiring a Message-ID length of at least 16
bytes.
Finally, we limit the overall partial responses to 200 when
scanning other inboxes to avoid excessive memory usage.
Eric Wong (Contractor, The Linux Foundation) [Tue, 6 Mar 2018 07:28:56 +0000 (07:28 +0000)]
v2writable: detect and use previous partition count
We need to detect the number of partitions the repository was
created with to ensure Xapian DBs can work across different
machines (or even CPU affinity changes) without leaving messages
unaffected by search.
Eric Wong (Contractor, The Linux Foundation) [Tue, 6 Mar 2018 04:15:38 +0000 (04:15 +0000)]
favor Received: date over Date: header globally
The first Received: header is believable since it typically
hits the user's mail server and can be treated as relatively
trustworthy. We still show the Date: in per-message (permalink)
views, which may expose users for having incorrect Date:
headers, but all the ISO YYYY-MM-DD dates we display will
match what we see.
Eric Wong (Contractor, The Linux Foundation) [Sat, 3 Mar 2018 20:56:15 +0000 (20:56 +0000)]
v2: avoid redundant/repeated configs for git partition repos
We'll let the config of all.git dictate every other subrepo to
ease maintenance and configuration. The "include" directive has
been supported since git 1.7.10, so it's safe to depend on as v2
requires git 2.6.0+ anyways for "get-mark" in fast-import.
Eric Wong (Contractor, The Linux Foundation) [Sat, 3 Mar 2018 20:21:05 +0000 (20:21 +0000)]
searchidx: store the primary MID in doc data for NNTP
We can't rely on header order for Message-ID after all
since we fall back to existing MIDs if they exist and
are unseen. This lets us use SearchMsg->mid to get the
MID we associated with the NNTP article number to ensure
all NNTP article lookups roundtrip correctly.
Eric Wong (Contractor, The Linux Foundation) [Sat, 3 Mar 2018 18:47:47 +0000 (18:47 +0000)]
nntp: use NNTP article numbers for lookups
Since Message-IDs are no longer unique within Xapian
(but are within the SQLite Msgmap); favor NNTP article
numbers for internal lookups. This will prevent us
from finding the "wrong" internal Message-ID.
Eric Wong (Contractor, The Linux Foundation) [Sat, 3 Mar 2018 17:57:57 +0000 (17:57 +0000)]
mid: truncate excessively long MIDs early
Since we support duplicate MIDs in v2, we can safely truncate
long MID terms in the database and let other normal duplicate
resolution sort it out. It seems only spammers use excessively
long MIDs, and there'll always be abuse/misuse vectors for causing
mis-threaded messages, so it's not worth worrying about
excessively long MIDs.
Eric Wong (Contractor, The Linux Foundation) [Sat, 3 Mar 2018 05:55:26 +0000 (05:55 +0000)]
searchidx: avoid excessive XNQ indexing with diffs
When indexing diffs, we can avoid indexing the diff parts under
XNQ and instead combine the parts in the read-only search
interface. This results in better indexing performance and
10-15% smaller Xapian indices.
Eric Wong (Contractor, The Linux Foundation) [Sat, 3 Mar 2018 05:14:33 +0000 (05:14 +0000)]
mid: be strict with References, but loose on Message-Id
Traditionally we've been more lax on parsing Message-Id
and allow it without the angle brackets. We've always been
strict on References and can't have it be pointlessly
large when some MUA decides to use HTML-escaped angle
brackets ("<", ">").
Eric Wong (Contractor, The Linux Foundation) [Fri, 2 Mar 2018 20:46:55 +0000 (20:46 +0000)]
search: revert to using 'Q' as a uniQue id per-Xapian conventions
'Q' is merely a convention in the Xapian world, and is close
enough to unique for practical purposes, so stop using XMID
and gain a little more term length as a result.
Eric Wong (Contractor, The Linux Foundation) [Fri, 2 Mar 2018 19:32:19 +0000 (19:32 +0000)]
v2writable: inject new Message-IDs on true duplicates
Since we'll need to support multiple Message-IDs anyways,
inject a new one if we hit a duplicate (or don't get one at
all).
Try to use a deterministic Message-Id for consistency, but give
up determinism and use a random Message-Id if an "attacker"
wants to prevent their message from being archived.
Eric Wong (Contractor, The Linux Foundation) [Fri, 2 Mar 2018 18:27:54 +0000 (18:27 +0000)]
content_id: no need to be human-friendly
We merely use this for internal comparisons and do not store
this in Xapian. So using a shorter, non-human readable digest
is enough. Furthermore, introduce "content_digest" which
returns the Digest::SHA object for extra changes.
Eric Wong (Contractor, The Linux Foundation) [Fri, 2 Mar 2018 09:39:31 +0000 (09:39 +0000)]
content_id: use `mids' and `references' for MID extraction
These already take care of deduping internally, so we'll save
ourselves at least some of the trouble while using a more
consistent API. While we're at it, hash the header name as
well, since we need to distinguish which header a certain value
came from.
Eric Wong (Contractor, The Linux Foundation) [Fri, 2 Mar 2018 03:43:30 +0000 (03:43 +0000)]
evcleanup: do not create event loop if nothing was registered
This was creating an unnecessary epoll descriptor via
Danga::Socket when using V2Writable to import a mbox. That
said, there should probably be better way of detecting whether
or not we're inside a Danga::Socket event loop.
Fixes: 427245acacaf04a8
("evcleanup: ensure deferred close from timers are handled ASAP")
Eric Wong (Contractor, The Linux Foundation) [Fri, 2 Mar 2018 03:12:23 +0000 (03:12 +0000)]
evcleanup: disable outside of daemon
We'll be using these in a more OO manner for V2Writable
(which doesn't use Danga::Socket), so lets not unnecessarily
register cleanup handlers intended for network daemons.
Eric Wong (Contractor, The Linux Foundation) [Fri, 2 Mar 2018 03:08:33 +0000 (03:08 +0000)]
content_id: special treatment for Message-Id headers
Some emails in LKML archives are identical with the only
difference being s/References:/In-Reply-To:/ in the headers.
Since this difference doesn't affect how we handle message
threading, we will treat them the same way for the purposes
of deduplication.
There may be more changes to how we do content_id along these
lines (e.g. using msg_iter to walk the message).
Eric Wong (Contractor, The Linux Foundation) [Thu, 1 Mar 2018 08:24:00 +0000 (08:24 +0000)]
v2writable: delete ::Import obj when ->done
As with the ::Import class this wraps, we want this to be
usable as a checkpoint and be able to call ->add afterwards.
We'll be relying on ->done to flush changes through all
partition and skeleton DBs for deduplication checks.
Eric Wong (Contractor, The Linux Foundation) [Tue, 27 Feb 2018 08:29:09 +0000 (08:29 +0000)]
search: query_xover uses skeleton DB iff available
The skeleton DB is where we store all the information needed
for NNTP overviews via XOVER. This seems to be the only change
necessary (besides eventually handling duplicates) necessary
to support our nntpd interface for v2 repositories.
Eric Wong (Contractor, The Linux Foundation) [Tue, 27 Feb 2018 20:25:23 +0000 (20:25 +0000)]
searchidx: do not modify Xapian DB while iterating
Iterating through a list of documents while modifying them does
not seem to be supported in Xapian and it can trigger
DatabaseCorruptError exceptions. This only worked with past
datasets out of dumb luck. With the work-in-progress "v2"
public-inbox layout, this problem might become more visible
as the "thread skeleton" is partitioned out to a separate,
smaller Xapian database.
I've reproduced the problem on both Debian 8.x and 9.x with
Xapian 1.2.19 (chert backend) and 1.4.3 (glass backend)
respectively.
Eric Wong (Contractor, The Linux Foundation) [Tue, 27 Feb 2018 20:29:55 +0000 (20:29 +0000)]
v2writable: commit to skeleton via remote partitions
We need to ensure Xapian transaction commits are made to remote
partitions before associated commits hit the skeleton DB.
This causes unnecessary commits to be made to the skeleton DB;
but they're mostly harmless. Further work will be necessary
to ensure proper ordering and avoidance of unnecessary commits.
Eric Wong (Contractor, The Linux Foundation) [Tue, 27 Feb 2018 00:41:21 +0000 (00:41 +0000)]
rename SearchIdxThread to SearchIdxSkeleton
Interchangably using "all", "skel", "threader", etc. were
confusing. Standardize on the "skeleton" term to describe
this class since it's also used for retrieval of basic headers.
Eric Wong (Contractor, The Linux Foundation) [Thu, 22 Feb 2018 17:55:35 +0000 (17:55 +0000)]
v2writable: round-robin to partitions based on article number
Instead of relying on the git object_id hash to partition,
round-robin to these partitions based on the NNTP article
number. This reduces the partition pipes as a source of
contention when two (or more) sequential messages end up
going to the same partition.
Eric Wong (Contractor, The Linux Foundation) [Thu, 22 Feb 2018 01:49:08 +0000 (01:49 +0000)]
v2: parallelize Xapian indexing
The parallelization requires splitting Msgmap, text+term
indexing, and thread-linking out into separate processes.
git-fast-import is fast, so we don't bother parallelizing it.
Msgmap (SQLite) and thread-linking (Xapian) must be serialized
because they rely on monotonically increasing numbers (NNTP
article number and internal thread_id, respectively).
We handle msgmap in the main process which drives fast-import.
When the article number is retrieved/generated, we write the
entire message to per-partition subprocesses via pipes for
expensive text+term indexing.
When these per-partition subprocesses are done with the
expensive text+term indexing, they write SearchMsg (small data)
to a shared pipe (inherited from the main V2Writable process)
back to the threader, which runs its own subprocess.
The number of text+term Xapian partitions is chosen at import
and can be made equal to the number of cores in a machine.
Eric Wong (Contractor, The Linux Foundation) [Thu, 22 Feb 2018 17:55:36 +0000 (17:55 +0000)]
import_vger_from_mbox: use PublicInbox::MIME and avoid clobbering
It is less confusing without the clobber assignment; and
PublicInbox::MIME exists to workaround bugs in older
Email::MIME (which is in Debian 9 (stretch))
Eric Wong (Contractor, The Linux Foundation) [Mon, 19 Feb 2018 20:07:11 +0000 (20:07 +0000)]
git: reload alternates file on missing blob
Since we'll be adding new repositories to the `alternates' file
in git, we must restart the `git cat-file --batch' process as
git currently does not detect changes to the alternates file
in long-running cat-file processes.
Don't bother with the `--batch-check' process since we won't be
using it with v2.
Eric Wong (Contractor, The Linux Foundation) [Sat, 17 Feb 2018 10:19:47 +0000 (10:19 +0000)]
v2writable: initial cut for repo-rotation
Wrap the old Import package to enable creating new repos based
on size thresholds. This is better than relying on time-based
rotation as LKML traffic seems to be increasing.
Eric Wong (Contractor, The Linux Foundation) [Fri, 16 Feb 2018 02:59:11 +0000 (02:59 +0000)]
search: stop assuming Message-ID is unique
In general, they are, but there's no way for or general purpose
mail server to enforce that. This is a step in allowing us
to handle more corner cases which existing lists throw at us.
Eric Wong (Contractor, The Linux Foundation) [Thu, 15 Feb 2018 01:10:06 +0000 (01:10 +0000)]
import: pass "raw" dates to git-fast-import(1)
For LKML, it appears we need an even more liberal parser than
RFC2822 date parser in git. I have not validated Date::Parse
parses dates correctly, but this at least prevents
git-fast-import(1) from choking.
Eric Wong (Contractor, The Linux Foundation) [Thu, 15 Feb 2018 00:25:53 +0000 (00:25 +0000)]
address: extract more characters from email addresses
There's a lot of weird characters which show up in LKML archives
which we did not support before. Furthermore, allow spaces
before the '>' in the From: line as at least some non-spam
poster used it.
Eric Wong (Contractor, The Linux Foundation) [Tue, 13 Feb 2018 03:02:56 +0000 (03:02 +0000)]
search: free up 'Q' prefix for a real unique identifier
This will allow easier-compatibility with v2 code which will
introduce content_id as the unique identifier.
The old "XMID" becomes "XM" as a free text searchable term.
"Q" becomes "XMID" as a boolean prefix.
There's no user-visible changes in this, but there needs to
be a schema version bump later on...
(more changes planned which can affect v1)
Eric Wong (Contractor, The Linux Foundation) [Tue, 13 Feb 2018 18:39:40 +0000 (18:39 +0000)]
scripts/import_vger_from_mbox: support --dry-run option
This can be useful for getting baseline of performance
of just Email::MIME and Date: header parsing. We'll need
to do some Date: header parsing for LKML since there are
some wonky date formats which causes the git RFC822 parser
to choke.
Eric Wong (Contractor, The Linux Foundation) [Thu, 8 Feb 2018 03:49:51 +0000 (03:49 +0000)]
scripts/import_vger_from_mbox: relax From_ line match slightly
The mboxes I got from cregit have two spaces after the email
address, while the "git format-patch" output I'm used to dealing
with only has one space.
It's still a "strict" match in that it checks for something
resembling a timestamp, but it relaxes the number of spaces
between the email address and date.
Eric Wong [Sat, 3 Feb 2018 02:50:46 +0000 (02:50 +0000)]
view: allow expanding directly to "nested" view
Sometimes, it can be desirable to jump directly to the "nested"
view when viewing a thread skeleton. This makes it possible.
While we're at it, shorten some of the text to ensure it still
fits in 80 columns.