wwwlisting: fix grep call for match=domain filtering
The grep call in list_match_domain_i returns true for all inboxes,
even ones without a URL that matches the regular expression, because
the qr value passed to grep is not surrounded by slashes. Add them.
Fixes: 1988d730c0088e8b (config: support multi-value inbox.*.*url)
Eric Wong [Wed, 29 Jul 2020 08:43:09 +0000 (08:43 +0000)]
emergency: create full path to PI_EMERGENCY
It's possible for ~/.public-inbox/ to not exist if PI_CONFIG
points to an alternate location. Only noticed from the previous
patch fixing t/init.t behavior.
Eric Wong [Wed, 29 Jul 2020 11:20:00 +0000 (11:20 +0000)]
xapcmd: -xcpdb and -compact disable CoW, too
This gives an opportunity for users already suffering from CoW
fragmentation to at least get the Xapian DBs off CoW. Aside
from over.sqlite3 in v1, the SQLite DBs remain untouched; though
VACUUM support may come in the future.
Eric Wong [Tue, 28 Jul 2020 22:21:58 +0000 (22:21 +0000)]
searchidx: disable CoW for SQLite and Xapian under btrfs
SQLite and Xapian files are written randomly, thus they become
fragmented under btrfs with copy-on-write. This leads to
noticeable performance problems (and probably ENOSPC) as these
files get big.
lore/git (v2, <1GB) indexes around 20% faster with this on an
ancient SSD. lore/lkml seems to be taking forever and I'll
probably cancel it to save wear on my SSD.
Unfortunately, disabling CoW also means disabling checksumming
(and compression), so we'll be careful to only set the No_COW
attribute on regeneratable data. We want to keep CoW (and
checksums+compression) on git storage because current ref
storage is neither checksummed nor compressed, and git streams
pack output.
Eric Wong [Sat, 25 Jul 2020 21:12:21 +0000 (21:12 +0000)]
v2writable: support async git blob retrievals
This seems to speed up --reindex on smallish v2 inboxes by about
30% on both HDD and SSD. lore/git (~1GB) on an SSD even gives a
30% improvement with 3 shards. I'm only seeing a ~4% speedup on
LKML with a SATA SSD (which is difficult to repeat because it
takes around 4 hours).
Testing LKML on an HDD will take much more time...
Eric Wong [Sun, 26 Jul 2020 06:57:30 +0000 (06:57 +0000)]
imap: introduce and use Git->async_prefetch
We can keep the git process more active by sending another
request to it while fetch_run_ops() is running. This
parallelization speeds up mutt's initial FETCH for headers by
around ~35%(!).
Eric Wong [Sun, 26 Jul 2020 09:31:10 +0000 (09:31 +0000)]
index: --compact respects --jobs
And -compact supports --jobs=0 like -index to disable parallel
execution. Running three xapian-compact processes in parallel
on a USB 2.0 HDD is pretty painful.
Eric Wong [Sun, 26 Jul 2020 19:43:06 +0000 (19:43 +0000)]
overidx: fix compatibility with current versions
We still need to use SQL_BLOB to ensure existing versions of
public-inbox can read over.sqlite3 because they're still using
{sqlite_unicode}. This partially reverts commit e9fc1290ead44e06d20ff58e0a6acb5306d4fbe2.
Eric Wong [Fri, 24 Jul 2020 05:56:01 +0000 (05:56 +0000)]
searchidx: make v1 indexing closer to v2
We'll switch to using IdxStack here to ensure we get repeatable
results and ascending THREADIDs according to git chronology.
This means we'll need a two-pass reindex to index existing
messages before indexing new messages.
Since we no longer have a long-lived git-log process, we don't
have to worry about old Xapian referencing the git-log pipe
w/o FD_CLOEXEC, either.
Eric Wong [Fri, 24 Jul 2020 05:56:00 +0000 (05:56 +0000)]
searchidx: rename _xdb_{acquire,release} => idx_
The "xdb" prefix was inaccurate since it's used by
indexlevel=basic, which is Xapian-free. The '_' (underscore)
prefix was also wrong for a method which is called across
package boundaries.
Eric Wong [Fri, 24 Jul 2020 05:55:57 +0000 (05:55 +0000)]
v2writable: get rid of {reindex_pipe} field
Since normal per-epoch indexing no longer holds a "git log"
process open, we don't need to worry about not sharing the
pipe with forked shards when we restart the indexer.
While we're in the area, better describe what `unindex' does,
since it's a rarely-used but necessary code path.
Eric Wong [Fri, 24 Jul 2020 05:55:54 +0000 (05:55 +0000)]
use consistent {ibx} field for writable code paths
This is a step which makes our use of abbreviations more
consistent when referring to PublicInbox::Inbox objects.
We'll also be reducing the number of redundant fields
in SearchIdx and V2Writable code paths to make the
object graph easier-to-follow.
Instead of doing fill_alternates for every epoch we're indexing,
just do it once at the start of index_sync invocation. This
will set us up for using a single "git cat-file" process for
indexing multiple epochs.
Eric Wong [Fri, 24 Jul 2020 05:55:49 +0000 (05:55 +0000)]
v2writable: introduce idx_stack
This avoids pinning a potentially large chunk of memory from
`git-log --reverse' into RAM (or triggering less predictable
swap behavior). Instead it uses a contiguous temporary file
with a fixed-size record for every blob we'll need to index.
Eric Wong [Fri, 24 Jul 2020 05:55:48 +0000 (05:55 +0000)]
v2: index forwards (via `git log --reverse')
Since we'll need to expose THREADID to JMAP and IMAP users,
index all messages in the order they were committed to ensure
our `tid' (thread ID) column ascends in mirrors the same way
they do in the source inbox.
This drastically simplifies our code but increases memory
usage of `git-log'. The next commit will bring memory use
back down at the expense of $TMPDIR usage.
Eric Wong [Fri, 24 Jul 2020 05:55:47 +0000 (05:55 +0000)]
index: support --rethread switch to fix old indices
Older versions of public-inbox < 1.3.0 had subtly
different semantics around threading in some corner
cases. This switch (when combined with --reindex)
allows us to fix them by regenerating associations.
Eric Wong [Fri, 17 Jul 2020 07:25:07 +0000 (07:25 +0000)]
v2writable: git_hash_raw: avoid $TMPDIR write
We can rely on FD_CLOEXEC being set by default (since Perl 5.6+)
on pipes to avoid FS/page-cache traffic, here. We also know
"git hash-object" won't output anything until it's consumed all
of its standard input; so there's no danger of a deadlock even
in the the unlikely case git uses a hash that can't fit into
PIPE_BUF :P
Eric Wong [Fri, 17 Jul 2020 06:31:55 +0000 (06:31 +0000)]
search: simplify unindexing
Since over.sqlite3 seems here to stay, we no longer need to do
Message-ID lookups against Xapian and can simply rely on the
docid <=> NNTP article number equivalancy SCHEMA_VERSION=15
gave us.
This rids us of the closure-using batch_do sub in the v1
code path and vastly simplifies both v1 and v2 unindexing.
Eric Wong [Fri, 17 Jul 2020 06:31:54 +0000 (06:31 +0000)]
searchidx: use v5.10.1, parent.pm, drop warnings
Prefer "parent" to "base" since the former is lighter and part
of Perl 5.10+. We'll also rely on warnings from "-w" globally
(or not) instead of via "use".
Eric Wong [Fri, 17 Jul 2020 06:31:53 +0000 (06:31 +0000)]
overidx: favor non-OO sub dispatch for internal subs
OO method dispatch was 10-15% slower when I was implementing the
NNTP server. It also serves as a helpful reminder to the reader
at the callsite as to whether a sub is likely in the same
package as the caller or not.
Eric Wong [Fri, 17 Jul 2020 06:31:49 +0000 (06:31 +0000)]
drop binmode usage
We only support Unix-like platforms where binmode (":raw") is
the default anyways, and v5.10 semantics means it won't do
unicode_strings (unlike v5.12). So save some lines of code.
Eric Wong [Fri, 17 Jul 2020 06:31:48 +0000 (06:31 +0000)]
v2: use v5.10.1, parent.pm, drop warnings
The "5.010_001" form was for Perl 5.6, which I doubt anybody
would attempt; so favor "v5.10.1" as it is more readable to
humans. Prefer "parent" to "base" since the former is lighter.
We'll also rely on warnings from "-w" globally (or not) instead
of via "use".
We'll also update "use" statements to reflect what's actually
used by V2Writable.
Eric Wong [Thu, 16 Jul 2020 21:54:24 +0000 (21:54 +0000)]
t/import: quiet warning, clobber variable
The eval in key2sub via t/run.perl ("make check-run") won't
trigger the warning, but running "prove -bvw t/import.t"
directly, does. In any case, ensure the contents of this
variable doesn't linger across runs.
Eric Wong [Fri, 17 Jul 2020 03:57:24 +0000 (03:57 +0000)]
doc: add some recommendations around slow HDDs
grok-pull is still painful with serialization on an old USB 2.0
HDD, but at least it can finish with flock(1) and disabling
parallelization. While parallel "git fetch" doesn't seem so
bad, slow seeks are exacerbated by parallel reads in Xapian.
That means some updates can take days instead of hours. The
same updates take only seconds or minutes on an SSD.
Eric Wong [Fri, 17 Jul 2020 06:07:10 +0000 (06:07 +0000)]
config: reject `\n' in `inboxdir'
"\n" and other characters requiring quoting and/or escaping in
in $GIT_DIR/objects/info/alternates was not supported in git 2.11
and earlier; nor does it seem supported at all in libgit2.
This will allow us to support sharing git-cat-file or similar
endpoints across multiple inboxes via alternates.
This breaks an existing use case for anybody wacky
enough to put `\n' in the `inboxdir' pathname; but I doubt
this affects anybody.
Eric Wong [Tue, 14 Jul 2020 10:06:40 +0000 (10:06 +0000)]
doc: release notes and version info updates
Update release notes with some features in the 1.6 timeline.
We'll note the version availability of some command-line
options, it may help users who are reading the latest
documentation online but running older versions.
Eric Wong [Tue, 14 Jul 2020 02:14:31 +0000 (02:14 +0000)]
nntpd+imapd: detect unlinked msgmap
While it's even less common to experience a replaced
msgmap.sqlite3 file, BOFHs may do the darndest things. This is
another step towards reducing the number of needless wakeups
we need to do in long-lived read-only daemons.
Eric Wong [Tue, 14 Jul 2020 02:14:30 +0000 (02:14 +0000)]
over: unset sqlite_unicode attribute
None of the human-readable strings stored in over.sqlite3
require UTF-8. Message-IDs do not, nor do the compressed
Subject IDs (sid) we use for Subject-based threading. And the
`ddd' (doc-data-deflated) column is of course binary data.
This frees us of having to use SQL_BLOB for the `ddd', column,
and will open the door for us to use dbh_new for Msgmap, too.
Eric Wong [Sun, 5 Jul 2020 21:51:31 +0000 (21:51 +0000)]
xapcmd: delay over->check_inodes trigger
We must not trigger wakeups on InboxIdle users until after we've
renamed all files into place. Otherwise, the InboxIdle caller
may just reopen the old (soon-to-be-unlinked) file.
Eric Wong [Mon, 13 Jul 2020 02:23:28 +0000 (02:23 +0000)]
xt/mem-imapd-tls: avoid EMFILE in -imapd process
Test::More dups standard FDs and may create FDs for other
purposes. run_mode => 0 lets us rely on FD_CLOEXEC to ensure
-imapd has enough FDs to accept all incoming connections at
the cost of higher (one-off) startup time.
Eric Wong [Tue, 7 Jul 2020 20:37:38 +0000 (20:37 +0000)]
wwwatomstream: avoid uninitialized warnings for $email
As in Import, we'll fall back to Sender: if From: is missing,
and use the primary_address of the inboxes to indicate the total
absence of those fields.
Eric Wong [Tue, 7 Jul 2020 06:17:24 +0000 (06:17 +0000)]
t/spawn: fix test reliability
Since Perl doesn't internally use a self-pipe for
sleep/select/poll/etc, wake up every 10ms to ensure
it can see the SIGCHLD; since neither signalfd nor EVFILT_SIGNAL
are always available.
Fixes: 761baa2a300e4268 ("spawn: unblock SIGCHLD in subprocess")
Eric Wong [Sun, 5 Jul 2020 22:44:52 +0000 (22:44 +0000)]
doc/technical/whyperl: note Perl 7 announcement
Right now[1] the Perl upstream plan is to maintain 5 compatibility
in Perl 7 for at least 5 years[1], and perhaps drop it when Perl 8
comes along. That said, distros may pick it and maintain 5 on their
own given the vast amounts of perfectly good legacy code out there.
Eric Wong [Sun, 5 Jul 2020 22:44:49 +0000 (22:44 +0000)]
git: use v5.10.1, parent.pm and Time::HiRes::stat
parent.pm is leaner than base.pm, and Time::HiRes::stat is
more accurate, so take advantage of these Perl 5.10+-isms
since it's been over a year since we left 5.8 behind.
Eric Wong [Sun, 5 Jul 2020 23:27:57 +0000 (23:27 +0000)]
gzipfilter: drop HTTP connection on bugs or data corruption
While all the {async_next} callbacks needed eval guards anyways
because of DS->write, {async_eml} callbacks did not.
Ensure any bugs in our code or data corruption result in
termination of the HTTP connection, so as not to leave clients
hanging on a response which never comes or is mangled in some
way.
Eric Wong [Sun, 5 Jul 2020 23:27:56 +0000 (23:27 +0000)]
wwwattach: support async blob retrievals
We can reuse some of the GzipFilter infrastructure used by other
WWW components to handle slow blob retrieval, here. The
difference from previous changes is we don't decide on the 200
status code until we've retrieved the blob and found the
attachment.
While we're at it, ensure we can compress text attachment
responses once again, since all text attachments are served
as text/plain.
Eric Wong [Sun, 5 Jul 2020 23:27:55 +0000 (23:27 +0000)]
wwwtext: simplify gzf_maybe use
gzf_maybe always returns a GzipFilter object, even if it uses
CompressNoop. We can also use ->zflush instead of
->translate(undef) here for the final bit.
Eric Wong [Sun, 5 Jul 2020 23:27:52 +0000 (23:27 +0000)]
wwwstream: eliminate ::response, use html_oneshot
All of our streaming responses use ::aresponse, now, and our
synchronous responses use html_oneshot. So there's no need
for the old WwwStream::response.
Eric Wong [Sun, 5 Jul 2020 23:27:51 +0000 (23:27 +0000)]
view: /$INBOX/$MSGID/t/: avoid extra hash lookup in eml case
We can build and buffer the HTML <head> section once the first
non-ghost message in a thread is loaded, so there's no need to
perform an extra check on $ctx->{nr} once the $eml is ready.
Eric Wong [Sun, 5 Jul 2020 23:27:41 +0000 (23:27 +0000)]
remove unused/redundant zlib-related imports
Z_FINISH is the default for Compress::Raw::Zlib::Deflate->flush,
anyways, so there's no reason to import it. And none of C::R::Z
is needed in WwwText now that gzf_maybe handles it all.
Eric Wong [Sun, 5 Jul 2020 23:27:40 +0000 (23:27 +0000)]
www: start making gzipfilter the parent response class
Virtually all of our responses are going to be gzipped, anyways.
This will allow us to utilize zlib as a buffering layer and
share common code for async blob retrieval responses.
To streamline this and allow GzipFilter to be a parent class,
we'll replace the NoopFilter with a similar CompressNoop class
which emulates the two Compress::Raw::Zlib::Deflate methods we
use.
This drops a bunch of redundant code and will hopefully make
upcoming WwwStream changes easier to reason about.
Eric Wong [Sun, 5 Jul 2020 23:27:39 +0000 (23:27 +0000)]
wwwstream: reduce blob fetch paths for ->getline
This will make it easier to support asynchronous blob
retrievals. The `$ctx->{nr}' counter is no longer implicitly
supplied since many users didn't care for it, so stack overhead
is slightly reduced.
Eric Wong [Sun, 5 Jul 2020 23:27:38 +0000 (23:27 +0000)]
wwwstream: reduce object graph depth
Like with WwwAtomStream and MboxGz, we can bless the existing
$ctx object directly to avoid allocating a new hashref. We'll
also switch from "->" to "::" to reduce stack utilization.
Eric Wong [Sun, 5 Jul 2020 23:27:37 +0000 (23:27 +0000)]
wwwatomstream: support async blob fetch
This allows -httpd to handle other requests while waiting
for git to retrieve and decode blobs. We'll also break
apart t/psgi_v2.t further to ensure tests run against
-httpd in addition to generic PSGI testing.
Using xt/httpd-async-stream.t to test against clones of meta@public-inbox.org
shows a 10-12% performance improvement with the following env:
TEST_JOBS=1000 TEST_CURL_OPT=--compressed TEST_ENDPOINT=new.atom
Eric Wong [Sun, 5 Jul 2020 23:27:32 +0000 (23:27 +0000)]
mbox: async blob fetch for "single message" raw mboxrd
This restores gzip-by-default behavior for /$INBOX/$MSGID/raw
endpoints for all indexed inboxes. Unindexed v1 inboxes will
remain uncompressed, for now.
Eric Wong [Sun, 5 Jul 2020 23:27:30 +0000 (23:27 +0000)]
mboxgz: do asynchronous git blob retrievals
This lets the -httpd worker process make better use of time
instead of waiting for git-cat-file to respond. With 4 jobs in
the new test case against a clone of
<https://public-inbox.org/meta/>, a speedup of 10-12% is shown.
Even a single job shows a 2-5% improvement on an SSD.
Eric Wong [Sun, 5 Jul 2020 23:27:29 +0000 (23:27 +0000)]
stop auto-loading Plack::Middleware::Deflater
Instead of gzipping some (mbox.gz, manifest.js.gz) responses and
leaving P::M::D to do the rest, we gzip everything ourselves,
now, so P::M::D is redundant.
Eric Wong [Sun, 5 Jul 2020 23:27:27 +0000 (23:27 +0000)]
wwwstatic: support gzipped directory listings
This will allow others to mimic our award-winning homepage
design without needing to rely on Plack::Middleware::Deflater
or varnish to compress responses.