Eric Wong [Sun, 15 Nov 2020 23:20:17 +0000 (23:20 +0000)]
searchidx: check for graceful shutdown in log2stack
The initial "git log" invocation for a git epoch can be time
consuming, so check for graceful shutdown at each line to ensure
timely shutdowns and avoid SSD/HDD wear.
Eric Wong [Sun, 15 Nov 2020 08:56:09 +0000 (08:56 +0000)]
t/eml.t: workaround newer Email::MIME* behavior
Recent (2020) versions of Email::MIME (and/or dependencies)
have different behavior than historical versions which seem
to be less DWIM and perhaps technically more correct. We'll
retain historical behavior for now, since it doesn't seem to
cause real problems and DWIM-ness is often required to make
sense of historical mail.
Tested on a FreeBSD 11.4 VM with the following packages:
Eric Wong [Fri, 13 Nov 2020 11:11:44 +0000 (11:11 +0000)]
extindex: support graceful shutdown via QUIT/INT/TERM
Just like the daemon processes, -extindex now supports graceful
shutdown via the same signals. This lets users avoid having to
repeat indexing messages when a power outage strikes during a
long (multi-hour/day) indexing run.
Per-inbox (v1/v2) -index graceful shutdowns are not supported,
yet, but is planned for later.
Eric Wong [Fri, 13 Nov 2020 11:11:43 +0000 (11:11 +0000)]
*index: discard sync->{todo} on iteration
There's no need to continuously append to {todo} when indexing
multiple inboxes. They're not redundantly indexed (because the
IdxStack is discarded, making it a noop), but it's still a waste
of memory keeping the $unit hashrefs around.
Eric Wong [Fri, 13 Nov 2020 11:11:42 +0000 (11:11 +0000)]
*index: avoid per-epoch --batch-check processes
Since all.git (v2) and ALL.git (extindex) encompass every single
epoch or indexed inbox; and is_ancestor() only uses hexadecimal
OIDs; there is no good reason to use $unit->{git} for an
epoch-local $git->check.
This prevents dozens/hundreds of --batch-check processes from
being left running after indexing and can improve locality
if size checks are being done (since that uses --batch-check,
too).
Theoretically several epochs may have conflicting OIDs, but
we're screwed in those cases, anyways, so we might as well
detect it earlier (though I'm not sure what the behavior would
be :x).
Eric Wong [Tue, 10 Nov 2020 03:20:29 +0000 (03:20 +0000)]
searchidx: fix fallback on unindex miss
In case of other bugs or intentional corruption of over.sqlite3,
we don't want to attempt dereferencing a non-ref scalar when
calling ->mid_delete in the fallback code path.
Noticed while chasing another bug in extindex development...
Eric Wong [Sat, 7 Nov 2020 10:57:00 +0000 (10:57 +0000)]
extindex: fix --batch-size support
Calling PublicInbox::Admin::index_prepare is required for
--batch-size (k|m|g) modifiiers and indexBatchSize in the config
file. Otherwise, the default 1m batch size stuck and led
to unexpectedly bad performance on a machine which could index
v2 inboxes faster with larger batch sizes.
Eric Wong [Sat, 7 Nov 2020 10:56:56 +0000 (10:56 +0000)]
extsearchidx: quiet warning for unindexed `d' messages
"deleted" messages (via -learn <spam|rm>) in the source inboxes
are likely to already be unindexed, so avoid triggering needless
warnings about the spam message being missing.
Eric Wong [Sat, 7 Nov 2020 10:56:55 +0000 (10:56 +0000)]
v2writable: less expensive checkpoint for extindex
Since extindex holds no locks on parallel inbox writers,
we can simply use "barrier" IPC shard commands to checkpoint
and avoid respawning shard or git processes.
Eric Wong [Sat, 7 Nov 2020 10:56:53 +0000 (10:56 +0000)]
searchidxshard: reduce syscalls when writing ->eidx_key
We use ->autoflush(1) on this pipe to ensure the shard workers
see data immediately on print; so this means we have to do our
own buffering for optional data.
Eric Wong [Sat, 7 Nov 2020 10:56:51 +0000 (10:56 +0000)]
extsearch: rename -eindex to -extindex
Upon "eindex" rhymes with "reindex", which could be confusing;
so name the command and config prefix to use "extindex" which
is hopefully less confusing.
Eric Wong [Tue, 27 Oct 2020 07:54:53 +0000 (07:54 +0000)]
searchidxshard: make warnings with eidx_key less confusing
Seeing "Xorg.foo.bar" can be confusing in warnings if the
eidx_key is only "org.foo.bar" with no relation to "Xorg" at
all. Furthermore, printing "\0" to log or terminal output isn't
very nice and could throw off some users/tools.
Eric Wong [Tue, 27 Oct 2020 07:54:52 +0000 (07:54 +0000)]
extsearchidx: support --batch-size checkpoints
This is needed to limit the RSS of processes and ensure the
stored data in over.sqlite3 and Xapian DBs are consistent if
interrupted. Without checkpoints, indexing lore causes shard
workers to take several GB of memory and thrash/OOM smaller
systems.
Eric Wong [Tue, 27 Oct 2020 07:54:48 +0000 (07:54 +0000)]
extsearchidx: handle edits
We can now handle cases where messages are edited in one inbox
but not another, bifurcating the message.
V2Writable::log_range handles some edge-cases which could happen
in v2-only code paths, as well, but weren't usually triggered
due to default git-gc knobs not pruning immediately
Eric Wong [Tue, 27 Oct 2020 07:54:43 +0000 (07:54 +0000)]
over: store xref3 data in over.sqlite3
We may not end up storing xref3 data in Xapian, actually.
This will make indexlevel=basic possible, and along with
--sequential-shard indexing support for slow storage.
Making oidmap a separate table seems unnecessary, too, so
fold it into the xref3 table since it's unlikely a git blob
will be responsible for multiple xref3 rows.
Eric Wong [Tue, 27 Oct 2020 07:54:29 +0000 (07:54 +0000)]
v2writable: make *last_commits and sync_prepare OO methods
This will allow ExtSearchIdx to override or reuse them more
easily. Unfortunately we lose prototype validation, but that
seems to be discouraged anyways given the 'signatures' feature
in Perl 5.20+.
Eric Wong [Tue, 27 Oct 2020 07:54:23 +0000 (07:54 +0000)]
searchidxshard: special init for eidx
Having a special init path for external indices is probably
easier than further overloading SearchIdx->new initialization
to work without an Inbox object.
Eric Wong [Tue, 27 Oct 2020 07:54:18 +0000 (07:54 +0000)]
v2writable: rename remaining "remote" terminology
"remote" used to imply "child process on the same machine" which
was somewhat non-sensical, anyways. And OverIdx has been in the
same process since v2 was finalized. So use the suffix "aux"
for "auxiliary" since it can be safely jettisoned without
breaking URLs.
Eric Wong [Tue, 27 Oct 2020 07:54:16 +0000 (07:54 +0000)]
v2: some changes for ExtSearchIdx compatibility
We'll be using per-sync-state {ibx} refs instead, so make parts
of the v2 indexing code less-dependent on $self->{ibx} where
$self is a V2Writable object.
Eric Wong [Tue, 27 Oct 2020 07:54:15 +0000 (07:54 +0000)]
overidx: introduce changes for external index
Since external indices won't have msgmap.sqlite3, we'll need to
store last_commit-* metadata in over.sqlite3 instead. This
has a longer limits to account for path names or newsgroup names
stored in keys.
We'll also rely on built-in counters for Xapian document IDs,
since msgmap.sqlite3 no longer provides an AUTOINCREMENT column.
Eric Wong [Tue, 27 Oct 2020 07:54:12 +0000 (07:54 +0000)]
searchidxshard: allow msgref to be undef
We don't need to keep it in code paths which are guaranteed to
only see PublicInbox::Eml (and not Email::MIME or PublicInbox::MIME
which did not round-trip properly). However, we must set
{raw_bytes} since PublicInbox::Eml may add an extra "\n" for
rare messages with no bodies.
Eric Wong [Tue, 27 Oct 2020 07:54:10 +0000 (07:54 +0000)]
v2writable: prepare initialization for external indices
External indices won't have $self->{ibx} since it needs to
deal with multiple inboxes. We can also hoist out
->parallel_init to make it easier to distinguish the
non-parallel control flow.
Eric Wong [Tue, 27 Oct 2020 07:54:07 +0000 (07:54 +0000)]
v2writable: make OO calls to last_commit-related methods
We'll try to reuse as much V2Writable code as possible for
external indices, but the way "last_commit" info is stored
must be different as external indices will deal with last_commit
info for multiple inboxes.
Eric Wong [Tue, 27 Oct 2020 07:54:02 +0000 (07:54 +0000)]
doc/standards: add RFCs for URL schemes
We linkify these in the WWW UI, and will support them in other
places. These URL schemes may end up being stored in
external/detached indices for indexing non-git-based mail
stores.
Perhaps some NNTP clients would be unhappy with the old value
"y". So use a bit more bandwidth+space to use the server-name
and historical "!not-for-mail" tail-entry to better conform to
a published RFC.
Eric Wong [Fri, 30 Oct 2020 02:13:58 +0000 (02:13 +0000)]
tls: epollbit: account for miscellaneous OpenSSL errors
Apparently they happen (triggered by my -imapd instance), so
bail out by closing the underlying socket rather than stopping
the event loop and daemon process.
Eric Wong [Sat, 17 Oct 2020 08:17:01 +0000 (08:17 +0000)]
xt: remove eml_check_roundtrip
If there's no body ({bdy} field), ->each_part set the {bdy}
field to "\n" and the ->as_string result afterwards is one
extra "\n" byte longer than the original.
It's not worth extra cycles in common ->each_part calls to
ensure 100% round-trip matches of header-only messages (which
are likely spam), especially when the only difference is a
trailing "\n".
Eric Wong [Sat, 17 Oct 2020 08:04:24 +0000 (08:04 +0000)]
git: introduce async_wait_all
->cat_async and ->check_async may trigger each other (in future
callers) while waiting, so we need a unified method to ensure
both complete. This doesn't affect current code, but allows us
to slightly simplify existing callers.
Eric Wong [Fri, 16 Oct 2020 06:59:34 +0000 (06:59 +0000)]
git: async: loop inflight checks for nested callbacks
We need to loop the inflight check for nested callback
invocations to ensure we don't clog the pipe that feeds
`git cat-file'.
This bug was obscured by the fact that we're already
accounting for 64-char git OIDs with SHA-256 in the
pipe space calculation; perhaps we shouldn't do that.
Eric Wong [Fri, 16 Oct 2020 06:59:33 +0000 (06:59 +0000)]
git: *_async: support nested callback invocations
For external indices, we'll need to support nested cat_async
invocations to deduplicate cross-posted messages.
Thus we need to ensure we do not clobber the {inflight*} queues
while stepping through and ensure {cat_rbuf} is stored before
invoking callbacks.
This fixes the ->cat_async-only case, but does not yet
account for the mix of ->check_async interspersed with
->cat_async calls, yet. More work will be needed on that
front at a later date.
It's currently not a problem as ->destroy doesn't
happen for no reason, we'll need to ensure future uses of
->destroy correctly discard the check_async buffer.
Eric Wong [Tue, 13 Oct 2020 01:34:30 +0000 (01:34 +0000)]
admin: preserve config ordering of `--all' switch
When `--all' is passed to -index and similar commands, process
them in the same order as what is given in the config file.
This ensures predictable behavior so admins can ensure certain
inboxes see updated indices before others. For (upcoming)
external indices, this will ensure stable Xref: ordering for
predictable caching/memoization by NNTP clients.
Eric Wong [Sat, 3 Oct 2020 23:12:46 +0000 (23:12 +0000)]
manifest: favor Cpanel::JSON::XS
JSON::MaybeXS already favors Cpanel::JSON::XS (and has for many
years, now). Allow users to skip installing JSON::MaybeXS if
they want an XS-based JSON implementation.
Eric Wong [Sun, 27 Sep 2020 22:12:48 +0000 (22:12 +0000)]
gcf2: improve error handling and do not ->fail on wbuf
For historical reasons, both Danga::Socket::write and
PublicInbox::DS::write will return 0 when data is buffered;
so Gcf2Client must not call ->fail when DS::write returns 0.
We'll also improve robustness by recreating the entire
Gcf2Client object if it does die for other reasons, instead of
risking mismatched fields due to deferred close.
We also need to ensure we only get one EPOLLERR wakeup and
issue EPOLL_CTL_DEL if ->event_step is triggered by a dying
Gcf2 process, so always register the FD with EPOLLONESHOT.
This switch is still undocumented, but we can reduce the scope
of our Xapian docdata dependency by moving its only caller to
SearchIdx. This reduces the amount of code loaded by read-only
code paths.
Eric Wong [Tue, 22 Sep 2020 18:49:50 +0000 (18:49 +0000)]
v2writable: drop outdated {unindex_range} check
{unindex_range} only exists in the $sync state, nowadays, not the
V2Writable ($self) object. $sync->{unindex_range} won't be
populated if $regen_max is zero, either, unless somebody is
injecting importable commits into an epoch history, in which
this change will result in no-op indexing doing no work.
Eric Wong [Sun, 20 Sep 2020 01:43:15 +0000 (01:43 +0000)]
config: warn on multiple values for some fields
Our code doesn't support multi-values for these, and having
unexpected arrays leads to unexpected results (e.g. showing
stuff like "ARRAY(0xDEADBEEFADD12E55)" in user interfaces). So
warn and only use the last value (matching git-config(1)
behavior without `--get-all').
Eric Wong [Sat, 19 Sep 2020 09:37:14 +0000 (09:37 +0000)]
gcf2: wire up read-only daemons and rm -gcf2 script
It seems easiest to have a singleton Gcf2Client client object
per daemon worker for all inboxes to use. This reduces overall
FD usage from pipes.
The `public-inbox-gcf2' command + manpage are gone and a `$^X'
one-liner is used, instead. This saves inodes for internal
commands and hopefully makes it easier to avoid mismatched
PERL5LIB include paths (as noticed during development :x).
We'll also make the existing cat-file process management
infrastructure more resilient to BOFHs on process killing
sprees (or in case our libgit2-based code fails on us).
(Rare) PublicInbox::WWW PSGI users NOT using public-inbox-httpd
won't automatically benefit from this change, and extra
configuration will be required (to be documented later).