Eric Wong [Wed, 13 Oct 2021 07:00:36 +0000 (07:00 +0000)]
treewide: use warn() or carp() instead of env->{psgi.errors}
Large chunks of our codebase and 3rd-party dependencies do not
use ->{psgi.errors}, so trying to standardize on it was a
fruitless endeavor. Since warn() and carp() are standard
mechanism within Perl, just use that instead and simplify a
bunch of existing code.
Eric Wong [Tue, 12 Oct 2021 22:44:56 +0000 (22:44 +0000)]
extindex: flush pending reindex before unref
This prevents unnecessary message renumbering and I/O.
Without this change, there is a small window for long-running
WWW streaming requests to miss a message that was unref-ed
before reindexing. If we expose an "All Mail" mailbox via
IMAP/JMAP, this will save client traffic.
Eric Wong [Tue, 12 Oct 2021 11:47:05 +0000 (11:47 +0000)]
www: _/text/config/raw Last-Modified: is mm->created_at
This allows IMAP mirrors to keep UIDVALIDITY synchronized (and
"LIST ACTIVE.TIMES" in NNTP). "lei add-external --mirror" will
automatically set it, as will the combination of
public-inbox-clone + public-inbox-index.
This avoids the need for extra endpoints or config entries,
at least...
Eric Wong [Tue, 12 Oct 2021 11:47:04 +0000 (11:47 +0000)]
msgmap: ->new_file to supports $ibx arg, drop ->new
The original Msgmap->new API was v1-specific and not necessary.
The ->new_file API now supports an $ibx object being passed to
it, simplify -no_fsync use. It will also make an upcoming
change easier...
Eric Wong [Tue, 12 Oct 2021 11:47:03 +0000 (11:47 +0000)]
daemon: unconditionally close Xapian shards on cleanup
The cost of opening a Xapian DB (even with shards) isn't high,
so save some FDs and just close it. We hit Xapian far less than
over.sqlite3 and we discard the MSet ASAP even when streaming
large responses.
This simplifies our code a bit and hopefully helps reduce
fragmentation by increasing mortality of late allocations.
Eric Wong [Tue, 12 Oct 2021 11:47:01 +0000 (11:47 +0000)]
msgmap: use DBI->prepare_cached
msgmap is not performance-critical enough to justify doing our
own prepared statement caching. Just rely on the functionality
of DBI here so future changes will be easier.
There's also minor style changes to avoid dirtying refcount
cache lines bumping by repeating hash lookups rather than attempting
to store them as locals.
Eric Wong [Tue, 12 Oct 2021 11:46:59 +0000 (11:46 +0000)]
search: delete QueryParser along with DB handle
Xapian::QueryParser is attached to the Xapian::Database,
so holding onto the QueryParser was preventing us from
releasing DB handles if a query was performed.
Eric Wong [Mon, 11 Oct 2021 08:06:20 +0000 (08:06 +0000)]
extindex: avoid invalid blobs after unref
When unref-ing a blob from xref3, make sure the "preferred"
smsg->{blob} doesn't point to the blob we just unrefed. This
is necessary because we periodically checkpoint our extindex
process to allow -watch and -mda processes to run.
This also gets rid of a lot of redundant code for ->remove_xref3,
since it's all handled in ExtSearchIdx, now.
Eric Wong [Mon, 11 Oct 2021 08:06:19 +0000 (08:06 +0000)]
extindex: more consistent doc removal
We need to ensure a message is consistently removed from eidxq,
over and Xapian in all cases. Removing from eidxq saves users
from some noisy error messages.
Eric Wong [Mon, 11 Oct 2021 08:06:18 +0000 (08:06 +0000)]
extindex: share unref logic in more places
We can use the same logic for --gc and --reindex and
'd' log entries
They're similar enough and the actual need to unref should
be fairly rare. We could go a lot faster if we didn't show
progress for --gc and --reindex, actually.
Eric Wong [Mon, 11 Oct 2021 08:06:16 +0000 (08:06 +0000)]
sqlite: PRAGMA optimize on close
As recommended by SQLite documentation[1]:
To achieve the best long-term query performance without the need
to do a detailed engineering analysis of the application schema
and SQL, it is recommended that applications run "PRAGMA optimize"
(with no arguments) just before closing each database connection.
Hopefully that works for our use cases and can make things
faster for us.
Eric Wong [Mon, 11 Oct 2021 08:06:15 +0000 (08:06 +0000)]
extindex: speed up --reindex --fast
This required some tweaking of xref3 indices in over.sqlite3,
but the end result is it brings no-op "--reindex --fast --all"
checks down to roughly 20 minutes (from 30-40 minutes) on
lore/all.
This is faster because a bunch of small SQLite queries are still
slower en-mass than a bunch of perlops. Despite the lack of IPC
overhead, crossing .so boundaries and repeating lookups over
btrees is still slower than doing the same with Perl hash tables.
Eric Wong [Sun, 10 Oct 2021 14:25:17 +0000 (14:25 +0000)]
lei/store: keep ".err-XXXX" in stderr tmpfile
This is slighly more meaningful since the file is already
in ~/.local/share/lei/store, so "lei_store" was redundant
(and the "XXXX" are random characters replaced by File::Temp)
Eric Wong [Sun, 10 Oct 2021 14:25:16 +0000 (14:25 +0000)]
extindex: --gc doesn't touch ghost entries
We were deleting ghost entries, this was usually harmless since
other messages could fill-in-the-blanks, but could cause
misthreading in odd cases where a big chunk of a thread is
missing and the latest messages only referenced ghosts.
We'll also save some cycles when scanning Xapian shards since
docids won't be <= 0.
Eric Wong [Sun, 10 Oct 2021 14:25:15 +0000 (14:25 +0000)]
extindex: minor cost reductions
Don't bother decoding the 20-byte SHA-1 to a 40-byte hex value
since we don't read it, anyways. We can also use the on-stack
ibx->eidx_key value instead of dispatching the method again.
Eric Wong [Sun, 10 Oct 2021 14:25:13 +0000 (14:25 +0000)]
set nodatacow on more SQLite files
We'll set nodatacow when detecting existing but empty
files, and also their directories in more cases (for
auxiliary -wal, -journal, -shm files). Hopefully
this keeps performance reasonable on CoW FSes.
Eric Wong [Sat, 9 Oct 2021 12:03:36 +0000 (12:03 +0000)]
view: save memory by dropping smsg->{from_name} on use
We'll also save a few LoC when generating it. $smsg objects can
linger a while when rendering large threads, so saving a few
bytes here can add up to several hundred KB saved.
I noticed this while chasing the ref cycle leak in commit b28e74c9dc0a (www: fix ref cycle from threading w/ extindex, 2021-10-03).
While there's no longer a leak, releasing memory earlier can
allow it to be reused sooner and reduce both memory traffic and
memory pressure.
Eric Wong [Sat, 9 Oct 2021 12:03:35 +0000 (12:03 +0000)]
http: avoid Perl target cache for psgi.input
By using syswrite to populate env->{psgi.input}. The substr()
call IO::Handle->write will trigger Perl's target/scratchpad and
result in a permanent allocation. Since this is a cold path,
that allocation is pointless, and syswrite() can already write a
substring.
Allowing Perl to cache a large allocation in a cold path only
result in fragmentation and wasted RAM.
write(2) on a regular file won't result in short writes
unless the FS quotas or free space limits are hit, or the buffer
is close to overflowing (e.g. the 0x7ffff000-byte Linux limit).
Since our HTTP server will never buffer that much in RAM,
there's no need to retry syswrite nor rely on the retrying
implicit in IO::Handle->write and the "print" perlop.
Eric Wong [Sat, 9 Oct 2021 12:03:34 +0000 (12:03 +0000)]
view: discard Eml->{bdy} when done using
We can release the raw body buffer once we've obtained a copy of
the decoded buffer. This reduces memory pressure ahead of some
expensive diff processing.
Eric Wong [Fri, 8 Oct 2021 10:20:04 +0000 (10:20 +0000)]
git: fatalize async callback errors by default
This should help us catch BUG: errors (and then some) in
-extindex and other read-write code paths. Only read-only
daemons should warn on async callback failures, since those
aren't capable of causing data loss.
Eric Wong [Wed, 6 Oct 2021 11:50:42 +0000 (11:50 +0000)]
ds: tmpio: avoid Perl target cache
The use of `substr' here an argument to `print' was causing Perl
to internally cache its target buffer. Since `syswrite()'
already offers a buffer offset arg and length limits, just use
`syswrite' directly. We were using autoflush anyways, so the
lack of buffering was of no concern performance-wise.
The target buffer could get to roughly ~10MB under some loads,
but it was usually a cold path and using memory which cannot be
released nor reused in other places.
note: IO::Handle::write uses `substr' internally, too;
so nothing would be gained using IO::Handle:write.
Eric Wong [Wed, 6 Oct 2021 11:19:36 +0000 (11:19 +0000)]
msg_iter: split_quotes adds trailing "\n"
The regexp in split_quotes relies on the presence of a
final "\n", so add it wherever we need to instead of
making it the responsibility of every caller.
This probably doesn't matter in practice since every
email seems to have a "\n" as the final byte (due to
the way SMTP works), but maybe there's some odd ones
that'll get imported via lei.
Eric Wong [Wed, 6 Oct 2021 10:12:21 +0000 (10:12 +0000)]
overidx: subject_path: allow non-ASCII char in subject matches
This should bring us closer to the "Base subject" definition in
IMAP ORDEREDSUBJECT (RFC 5256 2.1). Larger changes may cause
some breakage (until --reindex). But for now, a reindex will
prevents the non-ASCII subjects from being normalized to the
same fuzzy "thread" in the thread view.
Eric Wong [Wed, 6 Oct 2021 09:44:50 +0000 (09:44 +0000)]
extindex: --gc checkpoints
We need to ensure -extindex --gc runs don't prevent other
work from happening in the meantime. I actually caused
my -extindex to OOM due to the lack of checkpoints :x
We'll also hoist out the shard scanning into its own sub
in preparation for lei/store usage.
Eric Wong [Tue, 5 Oct 2021 09:40:17 +0000 (09:40 +0000)]
index: --reindex w/ --{since,until,before,after}
This lets administrators reindex specific time ranges
according to git "approxidate" formats. These arguments
are passed directly to underlying git-log(1) invocations
and may still reach into old epochs.
Since these options rely on git committer dates (which we infer
from the most recent Received: header), they are not guaranteed
to be strictly tied to git history and it's possible to
over/under-reindex some messages. It's probably not a major
problem in practice, though; reindexing a few extra messages
is generally harmless aside from some extra device wear.
Since this currently relies on git-log, these options do not
affect -extindex, yet.
Eric Wong [Mon, 4 Oct 2021 11:11:43 +0000 (11:11 +0000)]
overidx: update comment for new sub name
`shard_remove_eidx_info' was made unnecessary with commit 82b805db3ad9 (searchidxshard: IPC conversion, part 2, 2021-01-03)
and we now call `remove_eidx_info' directly.
Eric Wong [Mon, 4 Oct 2021 08:26:33 +0000 (08:26 +0000)]
{dir,inbox}idle: use level-triggered epoll
Both read(2) on inotify and kevent(2) return a finite amount of
events. Let the kernel notify us again in cases where we'd
need to retry instead of looping ourselves. This can prevent
missed/delayed notifications while still ensuring fairness in
busy event loops.
Making them immortal doesn't seem worth it, since doing immortal
allocations after process startup leads to fragmentation. While
the allocations made by highlight are small, those small
allocations can break up contiguous regions and prevent
consolidation by the malloc implementation.
Since instantiating code generators doesn't seem too expensive,
just use and delete them ASAP.
Eric Wong [Mon, 4 Oct 2021 00:07:17 +0000 (19:07 -0500)]
www: fix ref cycle from threading w/ extindex
Unlike v1 inboxes (which don't accept duplicate Message-IDs at
all), and v2 inboxes (which generate a new Message-ID for
duplicates), extindex must accept duplicate Message-IDs as-is.
This was fine for storage, but prevented the reference-cycle
mechanism of our message threading display algorithm from working
reliably. It could no longer delete the ->{parent} field from
clobbered entries in the %id_table.
So we now take into account reused Message-IDs and never clobber
entries in %id_table. Instead, we mark reused Message-IDs as
"imposters" and special-case them by injecting them as children
after all other threading is complete.
This cycle was noticed using a pre-release of Devel::Mwrap::PSGI:
https://80x24.org/mwrap-perl.git
Eric Wong [Sat, 2 Oct 2021 11:18:34 +0000 (11:18 +0000)]
content_hash: normalize whitespace before hashing addresses
This should prevent some false duplicates. I noticed this
while implementing "lei mail-diff", and only noticed it when
I implemented the ContentDigestDbg wrapper for mail-diff.
Eric Wong [Sat, 2 Oct 2021 11:18:33 +0000 (11:18 +0000)]
lei mail-diff: diagnostic command to diff mail contents
This is useful in finding the cause of deduplication bugs,
and possibly the cause of missing threads reported by
Konstantin in <20211001130527.z7eivotlgqbgetzz@meerkat.local>
usage:
u=https://yhbt.net/lore/all/87czop5j33.fsf@tynnyri.adurom.net/raw
lei mail-diff $u
Eric Wong [Fri, 1 Oct 2021 09:54:44 +0000 (09:54 +0000)]
ds: inline set_cloexec
I'm thinking we can drop support for Linux <2.6.27 soonish and
just use EPOLL_CLOEXEC. Perl without signalfd (or
EVFILT_SIGNAL) is miserable, actually.
Eric Wong [Fri, 1 Oct 2021 09:54:43 +0000 (09:54 +0000)]
inbox: keep DB handles if git processes are live
Having git processes outlive DB handles is likely to hurt
from a fragmentation perspective if the DB handle needs to
be recreated immediately due to a git->cat_async callback.
So only unref DB handles when we're sure there's no live
git users left, otherwise check the inodes.
We'll also avoid needless localization checks in git->cleanup
and make the return value more obvious since the pid fields are
unconditionally deleted nowadays.
Eric Wong [Fri, 1 Oct 2021 09:54:42 +0000 (09:54 +0000)]
inbox: inline and eliminate git_cleanup
It was probably incorrect to use from max_git_epoch, and it's
small enough to inline into do_cleanup. We'll also eliminate
the unnecessary deletion of {-altid_map} while we're in the
area, since we no longer cache/memoize that.
Fixes: 7e5cea05f061e757 ("inbox: rewrite cleanup to be more aggressive")
Eric Wong [Fri, 1 Oct 2021 09:54:41 +0000 (09:54 +0000)]
ds: simplify signalfd use
Since signalfd is often combined with our event loop, give it a
convenient API and reduce the code duplication required to use it.
EventLoop is replaced with ::event_loop to allow consistent
parameter passing and avoid needlessly passing the package name
on stack.
We also avoid exporting SFD_NONBLOCK since it's the only flag we
support. There's no sense in having the memory overhead of a
constant function when it's in cold code.
Eric Wong [Fri, 1 Oct 2021 09:54:40 +0000 (09:54 +0000)]
ipc: run Net::SSLeay::randomize
Currently we don't use OpenSSL from child processes of parents
which use OpenSSL, but we may in the future. So ensure OpenSSL
initializes its PRNG after these forks to avoid one security
pitfall down the line.
Eric Wong [Fri, 1 Oct 2021 09:54:38 +0000 (09:54 +0000)]
listener: switch to level-triggered epoll
On second thought, the ->requeue + accept retry code path isn't
worth the userspace complexity and overhead. Level-triggered
epoll has always annoyed me since it takes an inefficient code
path in the kernel; but taking our less-efficient code path in
Perl seems even worse. We also need to take load distribution
into account for multi-worker systems.
Eric Wong [Fri, 1 Oct 2021 09:54:37 +0000 (09:54 +0000)]
doc: lei-security: some more updates
Virtual users will probably be used for read-write IMAP/JMAP
support. The potential for various kernel/hardware bugs and
attacks also needs to be highlighted.
Eric Wong [Fri, 1 Oct 2021 02:10:27 +0000 (02:10 +0000)]
search_view: various navigation tweaks
This improves the "&x=t" navigation between the thread overview
(skeleton) section at the bottom and jumping back to the top for
the mbox download form. The "--links below ..." text ought to
be helpful for users unfamiliar with the /$MSGID/T/ and /$MSGID/t/
views.
Eric Wong [Wed, 29 Sep 2021 21:25:20 +0000 (21:25 +0000)]
git: shorten --git-dir= in CLI with chdir in spawn
Long pathnames are difficult to read and distinguish in ps(1)
output. Deep paths can also slow down pathname resolution
when dealing with loose objects, so we put "cat-file --batch"
deeper into the directory tree.
Since v2 processes are in the form of $INBOXDIR/all.git, keep
the basename of $INBOXDIR in --git-dir= so it's easy to
distinguish between processes just by looking at ps(1).
While "git -C" also exists, it's only present in git 1.8.5+.
We also need to keep in mind the "directory" pointed to by
--git-dir= need not be a directory (nor a symlink pointing
to one).
This reduces pathname resolution overhead for v1 and v2 inbox
git processes, but unfortunately not for extindex since that
needs to store alternates as absolute paths.
Eric Wong [Wed, 29 Sep 2021 12:40:46 +0000 (07:40 -0500)]
ds: simplify idle time expiry, slightly
While it doesn't look like $EXPMAP can be populated in
non-obvious ways via ->DESTROY, it still makes sense to keep it
close to some of our other code around cleanup to reduce
the likelyhood of subtle bugs in case semantics change..
Eric Wong [Wed, 29 Sep 2021 03:02:54 +0000 (03:02 +0000)]
t/solver_git: fix test to work with git <2.29
'git diff --abbrev=40' did not abbreviate /^index / lines of
diff output with git <2.29, and 40 will be insufficient for
SHA-256. --full-index has been around since 2005, so it's safe
to rely on.
Tested git version 2.20.0 (Debian buster).
Fixes: 751df49e7db8ba77 ("lei rediff: add --drq and --dequote-only")
Eric Wong [Tue, 28 Sep 2021 23:11:06 +0000 (23:11 +0000)]
inbox: drop memoization/preload, cleanup expires caches
cloneurl, description, and base_url are no longer memoized. The
non-$env form of base_url is rare in WWW, and is fast enough to
not require memoization.
cloneurl and description are now expired during cleanup,
allowing admins to change these files without restarting
(or SIGHUP).
-altid_map is no longer cached nor memoized at all, since the
endpoint(s) which hit it seem rarely accessed.
nntp_url and imap_url are now cached (instead of memoized) in
case an inbox is unvisited for a long time. They remain cached
since the truthiness check gets called in every per-inbox HTML
page, which can potentially be expensive.
Eric Wong [Tue, 28 Sep 2021 23:11:05 +0000 (23:11 +0000)]
inbox: rewrite cleanup to be more aggressive
Avoid relying on a giant cleanup hash and instead use the new
DS->add_uniq_timer API to amortize the pause times associated
with having to cleanup many inboxes. We can also use smaller
intervals for this, as well.
We now discard SQLite DB handles at cleanup. Each of these can
use several megabytes of memory, which adds up with
hundreds/thousands of inboxes. Since per-inbox access intervals
are unpredictable and opening an SQLite handle is relatively
inexpensive, release memory more aggressively to avoid the heap
having to hit swap.
Eric Wong [Tue, 28 Sep 2021 23:11:04 +0000 (23:11 +0000)]
www: do not bump {over} refcnt on long responses
SQLite files may be replaced or removed by admins while
generating a large threads or mailbox responses. Ensure we
don't hold onto DBI handles and associated file descriptors
past their cleanup.
Eric Wong [Tue, 28 Sep 2021 07:53:49 +0000 (07:53 +0000)]
www+httpd: lower priority of large mbox downloads
While each git blob request is treated fairly w.r.t other git
blob requests, responses triggering thousands of git blob
requests can still noticeably increase latency for
less-expensive responses.
Move large mbox results and the nasty all.mbox endpoint to
a low priority queue which only fires once per-event loop
iteration. This reduces the response time of short HTTP
responses while many gigantic mboxes are being downloaded
simultaneously, but still maximizes use of available I/O
when there's no inexpensive HTTP responses happening.
This only affects PublicInbox::WWW users who use
public-inbox-httpd, not generic PSGI servers.
Eric Wong [Mon, 27 Sep 2021 21:05:45 +0000 (16:05 -0500)]
lei completion: workaround old Perl bug
While `$argv[-1]' is `undef' on an empty @argv, using `$argv[-1]'
as a subroutine argument would fail incorrectly with:
Modification of non-creatable array value attempted, subscript -1 at ...
...even though we'd never attempt to modify @_ itself in the
subroutines being called. Work around the bug (tested on
5.16.3) by passing `undef' explicitly when `$argv[-1]' is
already `undef'.
Eric Wong [Mon, 27 Sep 2021 07:53:07 +0000 (02:53 -0500)]
config: get_1: use full parameter name
Instead of passing the prefix section and key separately, pass
them together as is commonly done with git-config(1) usage as
well as our ->get_all API. This inconsistency in the get_1 API
is a needless footgun and confused me a bit while working on
"lei up" the other week.
Eric Wong [Mon, 27 Sep 2021 04:59:31 +0000 (04:59 +0000)]
lei rediff: add --drq and --dequote-only
More switches which can be useful for users who pipe from text
editors. --drq can be helpful while writing patch review email
replies, and perhaps --dequote-only, too.
Eric Wong [Sun, 26 Sep 2021 01:42:38 +0000 (01:42 +0000)]
t/run.perl: less confusing error reporting
The $sigchld handler was reporting the last test (successful or
not) for a given PID in case a worker dies prematurely.
Instead, redisplay all failed test in $run_log to ensure the
report only shows failed tests, and not the last started (and
possibly successful) one.