Eric Wong [Sat, 5 Dec 2020 10:11:38 +0000 (10:11 +0000)]
isearch: emulate per-inbox search with ->ALL
Using "eidx_key:" boolean prefix to limit results to a given
inbox, we can use ->ALL to emulate and replace per-Inbox
xap15/[0-9] search indices.
With this change, the presence of "extindex.all.topdir" in the
$PI_CONFIG will cause the WWW code to use that extindex and
ignore per-inbox Xapian DBs in xap15/[0-9].
Unfortunately IMAP search still requires old per-inbox indices,
for now. Mapping extindex Xapian docids to per-Inbox UIDs and
vice-versa is proving tricky. Fortunately, IMAP search is
rarely used and optional. The RFCs don't specify expensive
phrase search, either, so `indexlevel=medium' can be used in
per-inbox Xapian indices to save space.
For primarily WWW (and future JMAP) users; this should result in
significant disk space, FD, and page cache footprint savings for
large instances with many inboxes and many cross-posted
messages.
Eric Wong [Sat, 5 Dec 2020 10:11:37 +0000 (10:11 +0000)]
inbox: simplify ->search and callers
Stop leaking WWW/PSGI-specific logic into classes like
PublicInbox::Inbox, which is used universally.
We'll also decouple $ibx->over from $ibx->search and just deal
with duplicate the code inside ->over to reduce argument
complexity in ->search.
This is also a step in moving away from using {psgi.errors}
to ease code sharing between IMAP, NNTP, and command-line
interfaces. Perl's built-in `warn' and `local $SIG{__WARN__}'
provides all the flexibility we need to control warning output
and should be universally understood by Perl hackers who may
be unfamiliar with PSGI.
Eric Wong [Fri, 4 Dec 2020 22:03:48 +0000 (22:03 +0000)]
newswww: use ->ALL to avoid O(n) inbox scan
We can avoid doing a Message-ID lookup on every single inbox
by using ->ALL to scan its over.sqlite3 DB. This mimics NNTP
behavior and picks the first message indexed, though redirecting
to /all/$MESSAGE_ID/ could be done.
With the current lore.kernel.org set of inboxes (~140), this
provides a 10-40% speedup depending on inbox ordering.
Eric Wong [Fri, 4 Dec 2020 22:03:47 +0000 (22:03 +0000)]
search: remove mdocid export
There's no need to export it, as shown by the change to
SearchView. This should pave the way to making search
more flexible and allow per-Inbox search to reuse ->ALL.
Eric Wong [Fri, 4 Dec 2020 22:03:46 +0000 (22:03 +0000)]
nntp: small speed up for multi-line responses
Using a non-zero-length separator for `join' requires extra work
inside Perl. We can shove the cost of appending "\r\n" into the
`map' loop, instead. This speeds up the `join' operation.
The "deferred" log entry for a "LISTGROUP org.kernel.vger.linux-kernel"
command (with nearly 3.8 million messages) goes from ~3.96s to 3.86s
on my workstation.
Eric Wong [Fri, 4 Dec 2020 22:03:45 +0000 (22:03 +0000)]
nntp: xref_by_tc: simplify slightly
We can invalidate ibx->{newsgroup} at config load-time to avoid
having to check ibx->{newsgroup} validity in To/Cc: matching.
This saves us some hash lookups in all cases.
Eric Wong [Fri, 4 Dec 2020 12:09:29 +0000 (12:09 +0000)]
over: ensure old, merged {tid} is really gone
We must use the result of link_refs() since it can trigger
merge_threads() and invalidate $old_tid. In case
merge_threads() isn't triggered, link_refs() will return
$old_tid anyways.
When rethreading and allocating new {tid}, we also must update
the row where the now-expired {tid} came from to ensure only the
new {tid} is seen when reindexing subsequent messages in
history. Otherwise, every subsequently reindexed+rethreaded
message could end up getting a new {tid}.
Eric Wong [Mon, 30 Nov 2020 23:37:42 +0000 (23:37 +0000)]
nntp: make ->ALL Xref generation more fuzzy
For ->ALL users, this mitigates the regression introduced
by commit 811b8d3cbaa790f59b7b107140b86248da16499b
("nntp: xref: use ->ALL extindex if available"), since
it's common to cross post messages to some mailing
lists with per-list trailers for unsubscribe information.
We won't bother dealing with Bcc-ed messages since those
are nearly all spam when it comes to public mailing lists.
Eric Wong [Sun, 29 Nov 2020 10:52:30 +0000 (10:52 +0000)]
git: ensure subclassed ->fail gets called
Some of these changes may not be strictly necessary, but it
makes code easier to maintain and change. Hackers
using/modifying this code will no longer wonder if a particular
callsite needs to care about subclasses or not.
Eric Wong [Sun, 29 Nov 2020 10:52:29 +0000 (10:52 +0000)]
git: set non-blocking flag in case of other bugs
This makes GitAsyncCat more resilient to bugs in Gcf2 or even
git-cat-file itself. I noticed -imapd stuck on read(2) from
the Gcf2 pipe, so there may be a bug somewhere in Gcf2 or
PublicInbox::Git. This should make us more resilient to them
and hopefully help us notice and fix them.
Eric Wong [Sun, 29 Nov 2020 05:09:35 +0000 (05:09 +0000)]
extindex: support `--gc' to remove dead inboxes
Inboxes may be removed or newsgroups renamed over time.
Introduce a switch to do garbage collection and eliminate stale
search and xref3 results based on inboxes which remain in the
config file.
This may also fixup stale results leftover from any bugs which
may leave stale data around.
This is also useful in case a clumsy BOFH (me :P) is swapping
between several PI_CONFIGs and accidentally indexed a bunch of
inboxes they didn't intend to.
Eric Wong [Sat, 28 Nov 2020 05:09:16 +0000 (05:09 +0000)]
nntpd: remove redundant {groups} shortcut
It's not worth confusing hackers reading the source to have
two ways to access the same (large) hash table. So just
go through PublicInbox::Config objects for now since the
extra hash lookup isn't going to be noticeable.
I've also started favoring "for" instead of "foreach"
since they're the equivalent perlop and less wear on
my fingers + keyboard.
Eric Wong [Sat, 28 Nov 2020 05:09:15 +0000 (05:09 +0000)]
nntp: XPATH uses ->ALL extindex, too
Another 30-40% speedup when testing against a local
lore.kernel.org mirror. In either case, we'll consistently sort
the response for ease-of-testing and client-side
cache-friendliness.
Eric Wong [Sat, 28 Nov 2020 05:09:14 +0000 (05:09 +0000)]
nntp: art_lookup: use mid_lookup and simplify
This lets us take advantage of mid_lookup speedup from the
previous commit.
While we're at it, start moving towards using `$ibx' as the
abbreviation for PublicInbox::Inbox objects even in the NNTP
code, since they've been shared with the WWW code for several
years, now.
Eric Wong [Sat, 28 Nov 2020 05:09:13 +0000 (05:09 +0000)]
nntp: speed up mid_lookup() using ->ALL extindex
We can reuse "xref3" information in extindex to quickly match
messages matching a given Message-ID across hundreds or
thousands of newsgroups with a few SQL statements.
"XHDR Xref $MESSAGE_ID" is around 40% faster, on top of
previous speedups.
Eric Wong [Sat, 28 Nov 2020 05:09:12 +0000 (05:09 +0000)]
nntp: NEWGROUPS uses long_response
We can amortize the cost of NEWGROUPS time filtering using the
long_response API. This lets us handle hundreds/thousands of
inboxes without monopolizing the event loop for this command.
Further speedup is possible using MiscSearch, but that requires
not-yet-done indexing changes to MiscIdx.
Eric Wong [Sat, 28 Nov 2020 08:45:21 +0000 (08:45 +0000)]
extindex: fix delete (`d') handling
We need to completely remove a message from over.sqlite3 and
Xapian when no references remain, otherwise users will still see
the removed messages in NNTP overviews and WWW search
results/summaries.
References to messages are now solely handled by the `xref3'
table of over.sqlite3. We can also trust `xref3' when deciding
whether to remove only the "O$eidx_key" and "G$lid" terms from a
document in Xapian or to remove the entire Xapian document.
Eric Wong [Sat, 28 Nov 2020 07:06:50 +0000 (07:06 +0000)]
searchidxshard: chomp $eidx_key from pipe
We were accidentally adding "\n" to terms (which Xapian happily
accepts), causing incompatibilities when enabling parallel
sharding in some invocations of -extindex but not others.
This is an extindex incompatibility and starting a new extindex
will be required to take advantage of in-development features,
so it's not urgent to start another one, either.
(other incompatible things may happen before a 1.7 release)
Eric Wong [Fri, 27 Nov 2020 21:33:55 +0000 (21:33 +0000)]
*index: more consistent graceful shutdown checks
v1 and v2 inbox indexing now supports graceful shutdown checks
just like ExtSearchIdx. Additionally, we'll consistently
perform quit checks at the top of loops for consistency.
Interaction with the --xapian-only and --sequential-shard
options are a bit lacking, and will warn the user to use
"--reindex --xapian-only" to fix.
Eric Wong [Fri, 27 Nov 2020 09:52:54 +0000 (09:52 +0000)]
nntp: xref: use ->ALL extindex if available
Getting Xref for cross-posted messages is an O(n) operation
where `n' is the number of newsgroups on the server. This works
acceptably when there are dozens of groups, but would be
unnacceptable when there's tens of thousands of newsgroups.
With ~140 newsgroups, a lore.kernel.org mirror already handles
"XHDR Xref $MESSAGE_ID" requests around 30% faster after
creating the xref3.idx_nntp index.
The SQL additions to ExtSearch.pm may be a bit strange and
seem more appropriate for Over.pm; however it currently makes
sense to me since those bits of over.sqlite3 access are
exclusive to ExtSearch and can't be used by traditional
v1/v2 inboxes...
Eric Wong [Fri, 27 Nov 2020 09:52:53 +0000 (09:52 +0000)]
nntp: xref: simplify sub signature
We'll be using the `xref3' table in extindex to speed up xref(),
and that'll require comparisons against $smsg->{blob}. So pass
the entire $smsg through.
Eric Wong [Fri, 27 Nov 2020 09:52:51 +0000 (09:52 +0000)]
t/extsearch: show a more realistic case
Different messages to different public Inboxes are likely to
have different List-IDs, so show that we can deduplicate based
on content (but per-mailing-list trailers need to go through a
PublicInbox::Filter::* or be disabled by mailing list admins).
Eric Wong [Fri, 27 Nov 2020 09:52:50 +0000 (09:52 +0000)]
nntp: move LIST iterators to long_response
Iterating through many newsgroups can hog the event loop if many
random seeks are required. Avoid monopolizing the event loop in
that case by using the long_response API.
For now, we can still rely on grep() since it seems to work
reasonably well with 50K test newsgroup names.
Eric Wong [Fri, 27 Nov 2020 09:52:47 +0000 (09:52 +0000)]
nntp: NEWNEWS: speed up filtering
With 50K newsgroups, the filtering phase goes from ~2000 seconds
to ~90 MILLISECONDS by relying on the grep perlop. This moves
->over checking out of the main dispatch and amortizes the cost
via long_response. (Fairly scheduled) long_response time in
newnews_i now takes ~360 seconds as opposed to ~30 seconds
before this change, however; but the initial filtering speedup
eliminating 2000s is more than worth it.
Eric Wong [Fri, 27 Nov 2020 09:52:44 +0000 (09:52 +0000)]
nntpd: share {groups} hash with {-by_newsgroup} in Config
There's no need to duplicate a potentially large hash,
but we can keep the inexpensive shortcut to it. We may
eventually drop the {groups} shortcut if it's no longer
useful.
Eric Wong [Fri, 27 Nov 2020 09:52:43 +0000 (09:52 +0000)]
nntp: use Inbox->uidvalidity instead of ->mm->created_at
This is memoized, and may allow us some future flexibility w.r.t
PublicInbox::Inbox-like objects. While we're at it, use
defined-or ("//") in case somebody really set a public-inbox
creation time to the Unix epoch.
Eric Wong [Mon, 23 Nov 2020 23:32:29 +0000 (23:32 +0000)]
extsearchidx: deduplicate alternates based on st_dev + st_ino
This allows us to filter out duplicate alternates entries in case
there's symlinks or bind mounts in play, as I (and perhaps some
other users) tend to use symlinks and/or bind mounts heavily.
Eric Wong [Mon, 23 Nov 2020 14:15:35 +0000 (14:15 +0000)]
wwwattach: prevent deep-linking via Referer match
This prevents `<img src=' tags from being used to deep-link
image attachments from HTML outside of the current host and
reduces potential for abuse.
Some browsers (e.g. Firefox) favor content detection and will
display images irrespective of the Content-Type header being
"application/octet-stream", and "Content-Disposition: attachment"
doesn't stop them, either.
Eric Wong [Tue, 24 Nov 2020 00:35:51 +0000 (00:35 +0000)]
gcf2: workaround libgit2 alternates bug for extindex
While libgit2 handles alternates with relative paths properly
for v2 epochs; nesting them another layer with extindex uses
the wrong relative path expansion (and is inconsistent with
git(1) behavior).
Fortunately, it's possible to work around this libgit2 bug
entirely within Gcf2 and avoid further special cases throughout
the rest of our code to support extindex.
Eric Wong [Mon, 23 Nov 2020 07:06:01 +0000 (07:06 +0000)]
manifest: support faster generation via [extindex "all"]
For a mirror of lore.kernel.org with >140 inboxes, this speeds
up manifest.js.gz generation from ~1s to 40ms on my HW. This
is still unacceptable when dealing with thousands of inboxes,
but gets us closer to where we need to be.
Eric Wong [Mon, 23 Nov 2020 07:06:00 +0000 (07:06 +0000)]
extsearchidx: do not short-circuit MiscIdx on no-op v2 prepare
This was intended to make development easier; but also allows us
description, URL, and address changes to be picked up
independently of message history.
Eric Wong [Mon, 23 Nov 2020 07:05:53 +0000 (07:05 +0000)]
git: add manifest_entry method
We'll be using this for MiscIdx and pre-generating the necessary
JSON for manifest.js.gz, so make it easier to share code for
generating per-repo JSON entries for grokmirror.
Eric Wong [Mon, 23 Nov 2020 07:05:51 +0000 (07:05 +0000)]
miscsearch: a new Xapian sub-DB for extindex
This will be used to index and search Inbox objects and perhaps
individual git repositories/epochs for grokmirror manifest.js.gz
generation. There is no sharding planned for this at the moment
since inbox count should remain low (~100K to 1M) compared to
message count.
Folding this into the existing sharded DBs could be possible;
but would likely increase query and maintenance costs, as well
as development complexity. So we'll use a few more inodes and
FDs at runtime, instead.
Eric Wong [Sun, 15 Nov 2020 23:20:17 +0000 (23:20 +0000)]
searchidx: check for graceful shutdown in log2stack
The initial "git log" invocation for a git epoch can be time
consuming, so check for graceful shutdown at each line to ensure
timely shutdowns and avoid SSD/HDD wear.
Eric Wong [Sun, 15 Nov 2020 08:56:09 +0000 (08:56 +0000)]
t/eml.t: workaround newer Email::MIME* behavior
Recent (2020) versions of Email::MIME (and/or dependencies)
have different behavior than historical versions which seem
to be less DWIM and perhaps technically more correct. We'll
retain historical behavior for now, since it doesn't seem to
cause real problems and DWIM-ness is often required to make
sense of historical mail.
Tested on a FreeBSD 11.4 VM with the following packages:
Eric Wong [Fri, 13 Nov 2020 11:11:44 +0000 (11:11 +0000)]
extindex: support graceful shutdown via QUIT/INT/TERM
Just like the daemon processes, -extindex now supports graceful
shutdown via the same signals. This lets users avoid having to
repeat indexing messages when a power outage strikes during a
long (multi-hour/day) indexing run.
Per-inbox (v1/v2) -index graceful shutdowns are not supported,
yet, but is planned for later.
Eric Wong [Fri, 13 Nov 2020 11:11:43 +0000 (11:11 +0000)]
*index: discard sync->{todo} on iteration
There's no need to continuously append to {todo} when indexing
multiple inboxes. They're not redundantly indexed (because the
IdxStack is discarded, making it a noop), but it's still a waste
of memory keeping the $unit hashrefs around.
Eric Wong [Fri, 13 Nov 2020 11:11:42 +0000 (11:11 +0000)]
*index: avoid per-epoch --batch-check processes
Since all.git (v2) and ALL.git (extindex) encompass every single
epoch or indexed inbox; and is_ancestor() only uses hexadecimal
OIDs; there is no good reason to use $unit->{git} for an
epoch-local $git->check.
This prevents dozens/hundreds of --batch-check processes from
being left running after indexing and can improve locality
if size checks are being done (since that uses --batch-check,
too).
Theoretically several epochs may have conflicting OIDs, but
we're screwed in those cases, anyways, so we might as well
detect it earlier (though I'm not sure what the behavior would
be :x).
Eric Wong [Tue, 10 Nov 2020 03:20:29 +0000 (03:20 +0000)]
searchidx: fix fallback on unindex miss
In case of other bugs or intentional corruption of over.sqlite3,
we don't want to attempt dereferencing a non-ref scalar when
calling ->mid_delete in the fallback code path.
Noticed while chasing another bug in extindex development...
Eric Wong [Sat, 7 Nov 2020 10:57:00 +0000 (10:57 +0000)]
extindex: fix --batch-size support
Calling PublicInbox::Admin::index_prepare is required for
--batch-size (k|m|g) modifiiers and indexBatchSize in the config
file. Otherwise, the default 1m batch size stuck and led
to unexpectedly bad performance on a machine which could index
v2 inboxes faster with larger batch sizes.
Eric Wong [Sat, 7 Nov 2020 10:56:56 +0000 (10:56 +0000)]
extsearchidx: quiet warning for unindexed `d' messages
"deleted" messages (via -learn <spam|rm>) in the source inboxes
are likely to already be unindexed, so avoid triggering needless
warnings about the spam message being missing.
Eric Wong [Sat, 7 Nov 2020 10:56:55 +0000 (10:56 +0000)]
v2writable: less expensive checkpoint for extindex
Since extindex holds no locks on parallel inbox writers,
we can simply use "barrier" IPC shard commands to checkpoint
and avoid respawning shard or git processes.
Eric Wong [Sat, 7 Nov 2020 10:56:53 +0000 (10:56 +0000)]
searchidxshard: reduce syscalls when writing ->eidx_key
We use ->autoflush(1) on this pipe to ensure the shard workers
see data immediately on print; so this means we have to do our
own buffering for optional data.
Eric Wong [Sat, 7 Nov 2020 10:56:51 +0000 (10:56 +0000)]
extsearch: rename -eindex to -extindex
Upon "eindex" rhymes with "reindex", which could be confusing;
so name the command and config prefix to use "extindex" which
is hopefully less confusing.
Eric Wong [Tue, 27 Oct 2020 07:54:53 +0000 (07:54 +0000)]
searchidxshard: make warnings with eidx_key less confusing
Seeing "Xorg.foo.bar" can be confusing in warnings if the
eidx_key is only "org.foo.bar" with no relation to "Xorg" at
all. Furthermore, printing "\0" to log or terminal output isn't
very nice and could throw off some users/tools.
Eric Wong [Tue, 27 Oct 2020 07:54:52 +0000 (07:54 +0000)]
extsearchidx: support --batch-size checkpoints
This is needed to limit the RSS of processes and ensure the
stored data in over.sqlite3 and Xapian DBs are consistent if
interrupted. Without checkpoints, indexing lore causes shard
workers to take several GB of memory and thrash/OOM smaller
systems.
Eric Wong [Tue, 27 Oct 2020 07:54:48 +0000 (07:54 +0000)]
extsearchidx: handle edits
We can now handle cases where messages are edited in one inbox
but not another, bifurcating the message.
V2Writable::log_range handles some edge-cases which could happen
in v2-only code paths, as well, but weren't usually triggered
due to default git-gc knobs not pruning immediately
Eric Wong [Tue, 27 Oct 2020 07:54:43 +0000 (07:54 +0000)]
over: store xref3 data in over.sqlite3
We may not end up storing xref3 data in Xapian, actually.
This will make indexlevel=basic possible, and along with
--sequential-shard indexing support for slow storage.
Making oidmap a separate table seems unnecessary, too, so
fold it into the xref3 table since it's unlikely a git blob
will be responsible for multiple xref3 rows.
Eric Wong [Tue, 27 Oct 2020 07:54:29 +0000 (07:54 +0000)]
v2writable: make *last_commits and sync_prepare OO methods
This will allow ExtSearchIdx to override or reuse them more
easily. Unfortunately we lose prototype validation, but that
seems to be discouraged anyways given the 'signatures' feature
in Perl 5.20+.
Eric Wong [Tue, 27 Oct 2020 07:54:23 +0000 (07:54 +0000)]
searchidxshard: special init for eidx
Having a special init path for external indices is probably
easier than further overloading SearchIdx->new initialization
to work without an Inbox object.