Eric Wong [Tue, 27 Oct 2020 07:54:23 +0000 (07:54 +0000)]
searchidxshard: special init for eidx
Having a special init path for external indices is probably
easier than further overloading SearchIdx->new initialization
to work without an Inbox object.
Eric Wong [Tue, 27 Oct 2020 07:54:18 +0000 (07:54 +0000)]
v2writable: rename remaining "remote" terminology
"remote" used to imply "child process on the same machine" which
was somewhat non-sensical, anyways. And OverIdx has been in the
same process since v2 was finalized. So use the suffix "aux"
for "auxiliary" since it can be safely jettisoned without
breaking URLs.
Eric Wong [Tue, 27 Oct 2020 07:54:16 +0000 (07:54 +0000)]
v2: some changes for ExtSearchIdx compatibility
We'll be using per-sync-state {ibx} refs instead, so make parts
of the v2 indexing code less-dependent on $self->{ibx} where
$self is a V2Writable object.
Eric Wong [Tue, 27 Oct 2020 07:54:15 +0000 (07:54 +0000)]
overidx: introduce changes for external index
Since external indices won't have msgmap.sqlite3, we'll need to
store last_commit-* metadata in over.sqlite3 instead. This
has a longer limits to account for path names or newsgroup names
stored in keys.
We'll also rely on built-in counters for Xapian document IDs,
since msgmap.sqlite3 no longer provides an AUTOINCREMENT column.
Eric Wong [Tue, 27 Oct 2020 07:54:12 +0000 (07:54 +0000)]
searchidxshard: allow msgref to be undef
We don't need to keep it in code paths which are guaranteed to
only see PublicInbox::Eml (and not Email::MIME or PublicInbox::MIME
which did not round-trip properly). However, we must set
{raw_bytes} since PublicInbox::Eml may add an extra "\n" for
rare messages with no bodies.
Eric Wong [Tue, 27 Oct 2020 07:54:10 +0000 (07:54 +0000)]
v2writable: prepare initialization for external indices
External indices won't have $self->{ibx} since it needs to
deal with multiple inboxes. We can also hoist out
->parallel_init to make it easier to distinguish the
non-parallel control flow.
Eric Wong [Tue, 27 Oct 2020 07:54:07 +0000 (07:54 +0000)]
v2writable: make OO calls to last_commit-related methods
We'll try to reuse as much V2Writable code as possible for
external indices, but the way "last_commit" info is stored
must be different as external indices will deal with last_commit
info for multiple inboxes.
Eric Wong [Tue, 27 Oct 2020 07:54:02 +0000 (07:54 +0000)]
doc/standards: add RFCs for URL schemes
We linkify these in the WWW UI, and will support them in other
places. These URL schemes may end up being stored in
external/detached indices for indexing non-git-based mail
stores.
Perhaps some NNTP clients would be unhappy with the old value
"y". So use a bit more bandwidth+space to use the server-name
and historical "!not-for-mail" tail-entry to better conform to
a published RFC.
Eric Wong [Fri, 30 Oct 2020 02:13:58 +0000 (02:13 +0000)]
tls: epollbit: account for miscellaneous OpenSSL errors
Apparently they happen (triggered by my -imapd instance), so
bail out by closing the underlying socket rather than stopping
the event loop and daemon process.
Eric Wong [Sat, 17 Oct 2020 08:17:01 +0000 (08:17 +0000)]
xt: remove eml_check_roundtrip
If there's no body ({bdy} field), ->each_part set the {bdy}
field to "\n" and the ->as_string result afterwards is one
extra "\n" byte longer than the original.
It's not worth extra cycles in common ->each_part calls to
ensure 100% round-trip matches of header-only messages (which
are likely spam), especially when the only difference is a
trailing "\n".
Eric Wong [Sat, 17 Oct 2020 08:04:24 +0000 (08:04 +0000)]
git: introduce async_wait_all
->cat_async and ->check_async may trigger each other (in future
callers) while waiting, so we need a unified method to ensure
both complete. This doesn't affect current code, but allows us
to slightly simplify existing callers.
Eric Wong [Fri, 16 Oct 2020 06:59:34 +0000 (06:59 +0000)]
git: async: loop inflight checks for nested callbacks
We need to loop the inflight check for nested callback
invocations to ensure we don't clog the pipe that feeds
`git cat-file'.
This bug was obscured by the fact that we're already
accounting for 64-char git OIDs with SHA-256 in the
pipe space calculation; perhaps we shouldn't do that.
Eric Wong [Fri, 16 Oct 2020 06:59:33 +0000 (06:59 +0000)]
git: *_async: support nested callback invocations
For external indices, we'll need to support nested cat_async
invocations to deduplicate cross-posted messages.
Thus we need to ensure we do not clobber the {inflight*} queues
while stepping through and ensure {cat_rbuf} is stored before
invoking callbacks.
This fixes the ->cat_async-only case, but does not yet
account for the mix of ->check_async interspersed with
->cat_async calls, yet. More work will be needed on that
front at a later date.
It's currently not a problem as ->destroy doesn't
happen for no reason, we'll need to ensure future uses of
->destroy correctly discard the check_async buffer.
Eric Wong [Tue, 13 Oct 2020 01:34:30 +0000 (01:34 +0000)]
admin: preserve config ordering of `--all' switch
When `--all' is passed to -index and similar commands, process
them in the same order as what is given in the config file.
This ensures predictable behavior so admins can ensure certain
inboxes see updated indices before others. For (upcoming)
external indices, this will ensure stable Xref: ordering for
predictable caching/memoization by NNTP clients.
Eric Wong [Sat, 3 Oct 2020 23:12:46 +0000 (23:12 +0000)]
manifest: favor Cpanel::JSON::XS
JSON::MaybeXS already favors Cpanel::JSON::XS (and has for many
years, now). Allow users to skip installing JSON::MaybeXS if
they want an XS-based JSON implementation.
Eric Wong [Sun, 27 Sep 2020 22:12:48 +0000 (22:12 +0000)]
gcf2: improve error handling and do not ->fail on wbuf
For historical reasons, both Danga::Socket::write and
PublicInbox::DS::write will return 0 when data is buffered;
so Gcf2Client must not call ->fail when DS::write returns 0.
We'll also improve robustness by recreating the entire
Gcf2Client object if it does die for other reasons, instead of
risking mismatched fields due to deferred close.
We also need to ensure we only get one EPOLLERR wakeup and
issue EPOLL_CTL_DEL if ->event_step is triggered by a dying
Gcf2 process, so always register the FD with EPOLLONESHOT.
This switch is still undocumented, but we can reduce the scope
of our Xapian docdata dependency by moving its only caller to
SearchIdx. This reduces the amount of code loaded by read-only
code paths.
Eric Wong [Tue, 22 Sep 2020 18:49:50 +0000 (18:49 +0000)]
v2writable: drop outdated {unindex_range} check
{unindex_range} only exists in the $sync state, nowadays, not the
V2Writable ($self) object. $sync->{unindex_range} won't be
populated if $regen_max is zero, either, unless somebody is
injecting importable commits into an epoch history, in which
this change will result in no-op indexing doing no work.
Eric Wong [Sun, 20 Sep 2020 01:43:15 +0000 (01:43 +0000)]
config: warn on multiple values for some fields
Our code doesn't support multi-values for these, and having
unexpected arrays leads to unexpected results (e.g. showing
stuff like "ARRAY(0xDEADBEEFADD12E55)" in user interfaces). So
warn and only use the last value (matching git-config(1)
behavior without `--get-all').
Eric Wong [Sat, 19 Sep 2020 09:37:14 +0000 (09:37 +0000)]
gcf2: wire up read-only daemons and rm -gcf2 script
It seems easiest to have a singleton Gcf2Client client object
per daemon worker for all inboxes to use. This reduces overall
FD usage from pipes.
The `public-inbox-gcf2' command + manpage are gone and a `$^X'
one-liner is used, instead. This saves inodes for internal
commands and hopefully makes it easier to avoid mismatched
PERL5LIB include paths (as noticed during development :x).
We'll also make the existing cat-file process management
infrastructure more resilient to BOFHs on process killing
sprees (or in case our libgit2-based code fails on us).
(Rare) PublicInbox::WWW PSGI users NOT using public-inbox-httpd
won't automatically benefit from this change, and extra
configuration will be required (to be documented later).
Eric Wong [Sat, 19 Sep 2020 09:37:11 +0000 (09:37 +0000)]
gcf2: transparently retry on missing OID
Since we only get OIDs from trusted local data sources
(over.sqlite3), we can safely retry within the -gcf2 process
without worry about clients spamming us with requests for
invalid OIDs and triggering reopens.
Eric Wong [Sat, 19 Sep 2020 09:37:09 +0000 (09:37 +0000)]
t/gcf2: test changes to alternates
Calling ->add_alternate won't pick up new additions to
$OBJDIR/info/alternates, unfornately. Thus v2 inboxes will
need to do something to invalidate Gcf2 objects.
Eric Wong [Sat, 19 Sep 2020 09:37:08 +0000 (09:37 +0000)]
gcf2: libgit2-based git cat-file alternative
Having tens of thousands of inboxes and associated git processes
won't work well, so we'll use libgit2 to access the object DB
directly. We only care about OID lookups and won't need to rely
on per-repo revision names or paths.
The Git::Raw XS package won't be used since its manpages don't
promise a stable API. Since we already use Inline::C and have
experience with I::C when it comes to compatibility, this only
introduces libgit2 itself as a source of new incompatibilities.
This also provides an excuse for me to writev(2) to reduce
syscalls, but liburing is on the horizon for next year.
Eric Wong [Thu, 17 Sep 2020 08:57:51 +0000 (08:57 +0000)]
git_async_cat: inline + drop redundant batch_prepare call
$git->cat_async already calls $git->batch_prepare iff needed, so
we can reduce subroutine calls and inline a one-off subroutine
to save some memory, here.
Eric Wong [Thu, 17 Sep 2020 21:25:22 +0000 (21:25 +0000)]
doc: txt2pre: more manpage URLs
We host our own -imapd manpage, and we started using a few more
git commands (fast-import for ages). We'll also need to link to
manpages.debian.org and live with long URLs for a few
non-standard manpages in software we reference.
Eric Wong [Tue, 15 Sep 2020 06:11:18 +0000 (06:11 +0000)]
imap: quiet uninitialized variable warning on FETCH
This was triggered by blindly trying to FETCH an MSN (not
"UID FETCH") on an empty dummy inbox. It's harmless, and
probably triggered by a wayward client or misbehaving bot.
Eric Wong [Mon, 14 Sep 2020 06:29:26 +0000 (06:29 +0000)]
tests: consistently check for xapian-compact
We may need to test against development versions of Xapian,
which may rely on setting `XAPIAN_COMPACT=xapian-compact-1.5'.
Ensure it's possible to do that.
And add a missing check in t/xcpdb-reshard.t, too.
Eric Wong [Mon, 14 Sep 2020 03:42:30 +0000 (03:42 +0000)]
sigfd: fix typos and scoping on systems w/o epoll+kqueue
Unfortunately, I'm not sure how easy catching these at
compile-time, is. Prototypes do not seem to check these
at compile time when crossing packages (not even with
exported subroutines).
Eric Wong [Fri, 11 Sep 2020 07:32:32 +0000 (07:32 +0000)]
t/nntpd: add test for the XPATH command
It's only in RFC 2980 (not 977 or 3977), but Net::NNTP has
supported it since 2001, at least. We'll be making changes
to avoid pathological behavior, so test it, first.
Eric Wong [Fri, 11 Sep 2020 07:32:31 +0000 (07:32 +0000)]
treewide: avoid `goto &NAME' for tail recursion
While Perl implements tail recursion via `goto' which allows
avoiding warnings on deep recursion. It doesn't (as of 5.28)
optimize the speed of such dispatches, though it may reduce
ephemeral memory usage.
Make the code less alien to hackers coming from other languages
by using normal subroutine dispatch. It's actually slightly
faster in micro benchmarks due to the complexity of `goto &NAME'.
Eric Wong [Thu, 10 Sep 2020 01:51:53 +0000 (01:51 +0000)]
solver: async blob retrieval for diff extraction
Like the rest of the WWW code, public-inbox-httpd now uses
git_async_cat to retrieve blobs without blocking the event loop.
This improves fairness when git blobs are on slow storage and
allows us to take better advantage of SMP systems.
Eric Wong [Wed, 9 Sep 2020 06:26:17 +0000 (06:26 +0000)]
solver: check one git coderepo and inbox at a time
With public-inbox-httpd, this mitigates the effect of slow git
blob storage with multiple coderepos configured for an inbox.
It's still synchronous for now (and may need to remain that way
for ->last_check_err), but no longer monopolizes the event loop
when checking multiple coderepos.
We don't yet support multi-inbox scanning, yet; but this also
prepares us for a future where we do.
We'll also support >=40 char blob OIDs in preparation for future
git SHA-256 support, too.
Eric Wong [Wed, 9 Sep 2020 06:26:13 +0000 (06:26 +0000)]
config: split out iterator into separate object
We will need to allow simultaneous iterators on the same
config object, since we'll need this for ExtMsg, NNTPD,
WwwListing, NewsWWW, and other places.
Eric Wong [Wed, 9 Sep 2020 06:26:12 +0000 (06:26 +0000)]
config: flatten each_inbox and iterate_start args
In Perl, we can simplify callers by passing a single array
all the way down the stack instead of a single array ref which
needs to be expanded every call.
Eric Wong [Wed, 9 Sep 2020 06:26:11 +0000 (06:26 +0000)]
www: manifest.js.gz generation no longer hogs event loop
It's still as slow as before with hundreds/thousands of inboxes,
but at least it's fair. Future changes will allow it to be
cached and memoized with persistent HTTP servers.
Eric Wong [Wed, 9 Sep 2020 06:26:10 +0000 (06:26 +0000)]
use "\&" where possible when referring to subroutines
"*foo" is ambiguous in that it may refer to a bareword file handle;
so we'll use it where we can without triggering warnings.
PublicInbox::TestCommon::run_script_exit required dropping the
prototype, however. We'll also future-proof by dropping "use
warnings" in Cgit.pm and use the less-ambiguous "//=" in Inbox.pm
while we're in the area.
Eric Wong [Wed, 9 Sep 2020 06:26:09 +0000 (06:26 +0000)]
solver: drop warnings, modernize use v5.10.1, use SEEK_SET
With Perl upstream preparing to deprecate things, we'll move
towards only enabling warnings during development via shebang
and stop enabling them via "use".
We'll also favor "use v5.10.1" over the Perl 5.6-compatible "use
5.010_001", since our code base never worked on 5.6.
Finally, were also importing SEEK_SET without using it, just use it
for readability since we can't avoid loading Fcntl in other
places and it'll get constant-folded, anyways.
Eric Wong [Wed, 9 Sep 2020 06:26:08 +0000 (06:26 +0000)]
xt/solver: test with public-inbox-httpd, too
We'll be making changes to solver to make it even fairer
to slow clients on slow storage. Ensure we test with
public-inbox-httpd-specific codepaths, since the generic
PSGI code paths are rare in production use.
Eric Wong [Wed, 9 Sep 2020 21:23:26 +0000 (21:23 +0000)]
wwwtext: config comment improvements
Use the full URL of the inbox being mirrored to reduce ambiguity
(instead of just the inbox name).
Using asymmetric quotes (e.g `foo') improves readability for me
in that it's more obvious when a quote begins and ends. It also
lights up fewer pixels and reduces visual noise compared to
double-quotes.
We'll also reflow the `mainrepo' vs `inboxdir' comment slightly
to emphasize the word `instead'.
Eric Wong [Thu, 10 Sep 2020 09:38:39 +0000 (09:38 +0000)]
nntp: fix cross-newsgroup Message-ID lookups
We cannot blindly use the selected newsgroup for
HEAD/ARTICLE/BODY requests using Message-ID, since
those commands look across all newsgroups; not just
the selected one (if any).
So stuff a reference to the Inbox object into $smsg.
We can reduce args passed into set_nntp_headers() and
msg_hdr_write(), too.
Fixes: 0e6ceff37fc38f28 ("nntp: support slow blob retrievals")
Eric Wong [Wed, 9 Sep 2020 20:24:55 +0000 (20:24 +0000)]
contrib/css: limit <a> coloring to links, only
We don't want <a> tags without href= attributes to be colored,
since the `<a id=mirror>' tag in the HTML footer is intended
as an anchor destination for `<a href=#mirror>' link at the
top.