Eric Wong [Sun, 24 Jan 2021 11:46:51 +0000 (04:46 -0700)]
lei q: honor --no-local to force remote searches
This can be useful for testing remote behavior, or for
augmenting local results. It'll also be possible to explicitly
include/exclude externals via CLI switches (once names are
decided).
Eric Wong [Sun, 24 Jan 2021 11:46:49 +0000 (04:46 -0700)]
ipc: get rid of wq_set_recv_modes
Just open every FD as read/write. Perl (or any non-broken
runtime) won't care and won't attempt to use F_SETFL to alter
file description flags; as attempting to change those would
lead to unpleasant side effects if the file description is
shared with another process.
Eric Wong [Sun, 24 Jan 2021 11:46:48 +0000 (04:46 -0700)]
ipc: wq supports arbitrarily large payloads
This should not be needed, but somebody using lei could
theoretically create thousands of external URLs and
only have a handful of workers, which means the per-worker
URI list could be large.
Eric Wong [Sun, 24 Jan 2021 11:46:47 +0000 (04:46 -0700)]
lei q: limit concurrency to 4 remote connections
Unfortunately, this isn't a per-host limit, yet; but
nevertheless reduces load on existing PublicInbox::WWW
instances, since requesting a mboxrd is one of the more
expensive operations.
Eric Wong [Fri, 22 Jan 2021 20:01:19 +0000 (20:01 +0000)]
treewide: reseed RNG in child processes
This prevents name conflicts leading to retries and slowdowns in
temporary file name generation. No actual data corruption
resulted because all temporary files are opened with O_EXCL
anyways.
This may increase security for IMAP, NNTP, and HTTPS sessions
using TLS, but it's all public data anyways.
Eric Wong [Sat, 23 Jan 2021 10:27:54 +0000 (10:27 +0000)]
lei forget-external: don't show redundant "not found"
Pathname/URL canonicalization may not change the result at
all, so there's no point in trying (and failing) the same
form twice if pre and post-canonicalization are identical.
Eric Wong [Sat, 23 Jan 2021 10:27:53 +0000 (10:27 +0000)]
lei q: support a bunch of curl(1) options
Some of these options will make sense when on weird networks
(behind firewalls, etc.) Some of these options may not make
sense at all.
This allows users who prefer to use the SOCKS5 proxy support in
curl rather than torsocks(1), but we'll still support torsocks
by default since some Tor instances aren't on the default
127.0.0.1:9050.
Eric Wong [Sat, 23 Jan 2021 10:27:50 +0000 (10:27 +0000)]
lei: default "-f $mfolder" args for common MUAs
At least mail, mailx, mutt, and neomutt follow this convention.
Heirloom mailx doesn't support Maildir (our default), but GNU
mailutils mail/mailx does.
Eric Wong [Sat, 23 Jan 2021 10:27:47 +0000 (10:27 +0000)]
lei: support remote externals
Via curl(1), since that lets us easily use tor on a
per-connection basis via LD_PRELOAD (torsocks) or proxy.
We'll eventually support more curl options which can allow
users to get past firewalls and deal with other odd network
configurations.
Eric Wong [Thu, 21 Jan 2021 19:46:23 +0000 (19:46 +0000)]
lei: forget-external support with canonicalization
For proper matching, we'll do a better job canonicalizing
URLs and path names for matching. Of course, users may edit
the file outside of lei, so ensure we try both the canonicalized
and as-is form provided by the user.
I also don't think we'll need to store externals info in
MiscIdx; just the config file is fine.
Eric Wong [Thu, 21 Jan 2021 19:46:22 +0000 (19:46 +0000)]
lei: remove @TO_CLOSE_ATFORK_CHILD
..At least limit it to a single file handle. The write end
EOFpipe can be limited in scope and auto-closed when $quit is
clobbered, leaving only the listener. The listener is the only
handle that needs to be closed explicitly due to it being on the
stack in the Listener->event_step => accept_dispatch => lei_$FOO
code path.
Everything else gets clobbered by DS->Reset in children after
forking.
Eric Wong [Thu, 21 Jan 2021 19:46:21 +0000 (19:46 +0000)]
lei_xsearch: reduce reference paths to lxs
Having an extra reference to LeiXSearch from the OpPipe $done_op
map is unnecessary and makes the reference graph more complex
than it needs to be. Just use $lei->{lxs} to simplify and
reduce the likelyhood of bugs.
The signal handlers on the client side were unnecessary,
all we need is to handle socket EOF properly in the daemon
by killing xsearch and l2m workers.
Eric Wong [Thu, 21 Jan 2021 19:46:18 +0000 (19:46 +0000)]
lei_to_mail: avoid segfault on exit
Worker exit causes DESTROY ordering to become unpredictable and
leads to Perl segfaulting. Instead, rely on OnDestroy and
explicit triggering after wq_worker_loop to ensure we finish
all outstanding git requests before worker exit.
Eric Wong [Thu, 21 Jan 2021 19:46:16 +0000 (19:46 +0000)]
lei: show {pct} and {oid} in From_ lines and filenames
From_ lines are shown when mbox* variants are output to stdout,
making {oid} and {pct} information visible without risking being
propagated to other importer processes if they were in
lei-specific X-* headers.
Maildirs already had OIDs in the filename, now they gain Xapian
{pct} in case anybody cares.
Eric Wong [Thu, 21 Jan 2021 19:46:13 +0000 (19:46 +0000)]
lei_overview: rename {relevance} => {pct}
The old name was too long compared to the rest of the field
names. With the Xapian method being named ->get_percent,
"pct" is a well known abbreviation for "percent" and already
used internally by our wrapper.
..And cleanup some excess whitespace while we're in the area.
Eric Wong [Wed, 20 Jan 2021 05:04:43 +0000 (14:04 +0900)]
lei: allow more mbox inode types
We may attempt to write an mbox to any terminal, block, or
character device, not just regular files and FIFOs/pipes.
The only thing that is known to not work is a directory.
Sockets may be possible with some OSes (e.g. Plan 9) or
filesystems. This fixes t/lei.t on FreeBSD 11.x
Eric Wong [Tue, 19 Jan 2021 09:34:33 +0000 (09:34 +0000)]
t/lei: fix double-running of socket test with oneshot
We split out t/lei-oneshot.t and t/lei.t so it's easier
to isolate run-mode specific bugs and behavior and there's
no reason to rerun the socket daemon tests.
Eric Wong [Tue, 19 Jan 2021 09:34:31 +0000 (09:34 +0000)]
lei q: fix augment of compressed mailboxes
We need to delay writing out the mailbox until the compressor
process is up and running, so have startq wait a bit. This
means we must create the pipe early and hand it off to the
workers before augmenting, despite spawning the
gzip/pigz/xz/bzip2 process after augment is complete.
Eric Wong [Tue, 19 Jan 2021 09:34:30 +0000 (09:34 +0000)]
lei: write daemon errors to the sock directory
Most everything should be captured by the __WARN__ handlers and
routed to syslog, but it appears Perl may write to stderr in
some emergency cases, as can libc or other libraries. Just
point it to a small file that's cleared on reboot.
Eric Wong [Tue, 19 Jan 2021 09:34:28 +0000 (09:34 +0000)]
lei q: fix SIGPIPE handling from lei2mail workers
We need to properly propagate SIGPIPE to the top-level
lei-daemon process and avoid relying on auto-close,
since auto-close triggers Perl warnings when explicit
close() does not.
Eric Wong [Tue, 19 Jan 2021 09:34:27 +0000 (09:34 +0000)]
lei q: start ->mset while query_prepare runs
We don't need the result of query_prepare (for augmenting or
mass unlinking) until we're ready to deduplicate and write
results to the filesystem. This ought to let us hide some of
the cost of Xapian searches on multi-device/core systems for
extremely expensive searches.
Eric Wong [Mon, 18 Jan 2021 10:30:32 +0000 (04:30 -0600)]
lei_to_mail: optimize for MUAs
Instead of optimizing our own performance, this optimizes
our data to reduce work done by the MUA consumer.
Maildir and mbox destinations no longer support any notion of
the IMAP \Recent flag. JMAP has no functioning \Recent
equivalent, and neither do we.
In practice, having MUAs (e.g. mutt) clear the \Recent flag when
committing changes to the mbox is expensive: it creates a
rename(2) storm with Maildir and overwrites the entire mbox.
For mboxcl2 (and mboxcl), we'll further optimize mutt behavior
by setting the Lines: header in addition to Content-Length.
With these changes, mutt exits instantaneously on mboxcl2,
mboxcl, and Maildirs generated by "lei q".
Eric Wong [Mon, 18 Jan 2021 10:30:31 +0000 (04:30 -0600)]
lei q: parallelize Maildir and mbox writing
With 4 dedicated workers, this seems to provide a 100-120%
speedup on a 4 core machine when writing thousands of search
results to a Maildir or mbox. This also sets us up for
high-latency IMAP destinations in the future.
This opens the door to more speedup opportunities such
as optimizing dedupe locking and other ways to reduce
contention.
This change is fairly complex and convoluted, unfortunately.
Further work may allow us to simplify it and even improve
performance.
Eric Wong [Sun, 17 Jan 2021 08:52:27 +0000 (20:52 -1200)]
lei q: add --mua-cmd switch
It can be convenient to invoke an MUA as search results
are being written to it, as an eager person may want to
start seeing results ASAP. This lets Maildir users
see results in the MUA as we are writing them. Users
of IMAP will eventually be able to take advantage of
them, too.
Since we don't support mbox locking (yet?), we'll only invoke
the MUA after results are done for mbox formats.
Eric Wong [Sat, 16 Jan 2021 11:36:23 +0000 (23:36 -1200)]
lei: q: results output to Maildir and mbox* working
All the augment and deduplication stuff seems to be working
based on unit tests. OpPipe is a nice general addition that
will probably make future state machines easier.
Eric Wong [Sat, 16 Jan 2021 11:36:22 +0000 (23:36 -1200)]
ipc: children don't kill on DESTROY, reduce FD sharing
Children should not be blindly killing siblings on ->DESTROY
since they're typically shorter-lived than parents. We'll
also be more careful about on-stack variables and now we
can rely exclusively on delete ops to close FDs.
We also need to fix our SIGPIPE handling for the oneshot case
while fixing a typo for delete, so we write "!" to the EOF pipe
to ensure the parent oneshot process exits on the first worker
that hits SIGPIPE, rather than waiting for the last worker to
hit SIGPIPE.
Eric Wong [Sun, 17 Jan 2021 07:09:59 +0000 (07:09 +0000)]
extindex: fix w/ Xapian 1.2.21..1.2.24
Xapian v1.2.21..v1.2.24 failed to set the close-on-exec flag
on the flintlock FD, causing "git cat-file" processes to
hold onto the lock and prevent subsequent Xapian::WritableDatabase
from locking the DB. So cleanup git processes after committing
the miscidx transaction.
Eric Wong [Sun, 17 Jan 2021 07:09:58 +0000 (07:09 +0000)]
t/shared_kv: workaround old File::Spec
The version of File::Spec shipped with Perl 5.16.3 memoizes the
value of File::Spec->tmpdir, causing changes to $ENV{TMPDIR}
after-the-fact to be ignored.
We'll only work around this in the test since it's innocuous and
unlikely to matter in real-world usage (and there's many places
where we'd have to workaround this in non-test code).
Eric Wong [Sun, 17 Jan 2021 07:09:56 +0000 (07:09 +0000)]
initialize scalar for `vec' perlop modification
Older Perls (tested 5.16.3) would warn on uninitialized scalars while
newer (tested 5.28.1) do not. Just initialize it to an empty string
since it'll be filled in by `vec'.
Eric Wong [Thu, 14 Jan 2021 07:06:27 +0000 (19:06 -1200)]
lei: pass FD to CWD via cmsg, use fchdir on server
Perl chdir() automatically does fchdir(2) if given a file
or directory handle since 5.8.8/5.10.0, so we can safely
rely on it given our 5.10.1+ requirement.
This means we no longer have to waste several milliseconds
loading the Cwd.so and making stat() calls to ensure
ENV{PWD} is correct and usable in the server. It also lets
us work in directories that are no longer accessible via
pathname.
Eric Wong [Thu, 14 Jan 2021 07:06:24 +0000 (19:06 -1200)]
lei: q: lock stdout on overview output
Most writes to stdout aren't atomic and we need locking to
prevent workers from interleaving and corrupting JSON output.
The one case stdout won't require locking is if it's pointed
to a regular file with O_APPEND; as POSIX O_APPEND semantics
guarantees atomicity.
Eric Wong [Thu, 14 Jan 2021 07:06:23 +0000 (19:06 -1200)]
lei_overview: rename "references" to "refs"
"references" was too long of a name compared to the other field
names we output in the JSON. While we currently don't have a
"refs:" search prefix for the "References:" header, we may in
the future.
Eric Wong [Thu, 14 Jan 2021 07:06:22 +0000 (19:06 -1200)]
search: rename "ts:" prefix to "rt:"
Meaning "Received time", as it is the best description of the
value we use from the "Received:" header, if present. JMAP
calls it "receivedAt", but "rt:" seems like a better
abbreviation being in line with "dt:" for the "Date" header.
"Timestamp" ("ts") was potentially ambiguous given the presence
of the "Date" header.
Eric Wong [Thu, 14 Jan 2021 07:06:18 +0000 (19:06 -1200)]
lei: reduce live FD references in wq child
We can shrink the @TO_CLOSE_ATFORK_CHILD array by two
elements, at least. I may be possible to eliminate this
array entirely but clobbering $quit doesn't seem to
remove references to $eof_w or the $listener socket.
Eric Wong [Thu, 14 Jan 2021 07:06:17 +0000 (19:06 -1200)]
lei: do not unlink socket path at exit
This matches existing -httpd/-nntpd/-imapd daemon behavior.
From what I can recall, it is less racy for the process doing
bind(2) to unlink it if stale.
Eric Wong [Thu, 14 Jan 2021 07:06:16 +0000 (19:06 -1200)]
daemon+watch: fix localization of %SIG for non-signalfd users
It turns out "local" did not take effect in the way we used it:
http://nntp.perl.org/group/perl.perl5.porters/258784
<CAHhgV8hPbcmkzWizp6Vijw921M5BOXixj4+zTh3nRS9vRBYk8w@mail.gmail.com>
Fortunately, none of the old use cases seem affected, unlike the
previous lei change to ensure consistent SIGPIPE handling.
Eric Wong [Thu, 14 Jan 2021 07:06:15 +0000 (19:06 -1200)]
lei: test SIGPIPE, stop xsearch workers on client abort
The new test ensures consistency between oneshot and
client/daemon users. Cancelling an in-progress result now also
stops xsearch workers to avoid wasted CPU and I/O.
Note the lei->atfork_child_wq usage changes, it is to workaround
a bug in Perl 5: http://nntp.perl.org/group/perl.perl5.porters/258784
<CAHhgV8hPbcmkzWizp6Vijw921M5BOXixj4+zTh3nRS9vRBYk8w@mail.gmail.com>
This switches the internal protocol to use SOCK_SEQPACKET
AF_UNIX sockets to prevent merging messages from the daemon to
client to run pager and kill/exit the client script.
Eric Wong [Sun, 10 Jan 2021 12:15:19 +0000 (12:15 +0000)]
lei: query: restore JSON output overview
This internal API is better suited for fork-friendliness (but
locking + dedupe still needs to be re-added).
Normal "json" is the default, though stream-friendly "concatjson"
and "jsonl" (AKA "ndjson" AKA "ldjson") all seem working
(though tests aren't working, yet).
For normal "json", the biggest downside is the necessity of a
trailing "null" element at the end of the array because of
parallel processes, since (AFAIK) regular JSON doesn't allow
trailing commas, unlike JavaScript.
Eric Wong [Sun, 10 Jan 2021 12:15:18 +0000 (12:15 +0000)]
lei_xsearch: transfer 4 FDs internally, drop IO::FDPass
It's easier to make the code more generic by transferring
all four FDs (std(in|out|err) + socket) instead of omitting
stdin.
We'll be reading from stdin on some imports, and possibly
outputting to stdout, so omitting stdin now would needlessly
complicate things.
The differences with IO::FDPass "1" code paths and the "4"
code paths used by Inline::C and Socket::MsgHdr are far too
much to support and test at the moment.
Eric Wong [Sun, 10 Jan 2021 12:15:17 +0000 (12:15 +0000)]
lei: run pager in client script
While most single keystrokes work fine when the pager is
launched from the background daemon, Ctrl-C and WINCH can cause
strangeness when connected to the wrong terminal.
Eric Wong [Sun, 10 Jan 2021 12:15:16 +0000 (12:15 +0000)]
lei: fork + FD cleanup
Do a better job of closing FDs that we don't want shared with
the work queue workers. We'll also fix naming and use
"atfork_prepare" instead of "atfork_parent" to match
pthread_atfork(3) naming.
Eric Wong [Sun, 10 Jan 2021 12:15:15 +0000 (12:15 +0000)]
lei: get rid of client {pid} field
Using kill(2) is too dangerous since extremely long
queries may mean the original PID of the aborted lei(1)
client process to be recycled by a new process. It would
be bad if the lei_xsearch worker process issued a kill
on the wrong process.
So just rely on sending the exit message via socket.
Eric Wong [Sun, 10 Jan 2021 12:15:14 +0000 (12:15 +0000)]
ipc: drop unused fields, default sighandlers for wq
Relying on signal handlers to kill a particular worker was a
laggy/racy idea and I gave up on the idea of targetting workers
explicitly and instead chose to make wq_worker_decr stop the
next idle worker ->wq_exit.
We will however attempt to support sending signals to
a process group.
Eric Wong [Sun, 10 Jan 2021 12:15:13 +0000 (12:15 +0000)]
ipc: fix IO::FDPass use with a worker limit of 1
IO::FDPass is our last choice for implementing the workqueue
because its lack of atomicity makes it impossible to guarantee
all requests of a single group hit a single worker out of many.
So the only way to use IO::FDPass for workqueues it to only have
a single worker. A single worker still buys us a small amount
of parallelism because of the parent process.
Eric Wong [Sun, 10 Jan 2021 12:15:12 +0000 (12:15 +0000)]
ipc: start supporting sending/receiving more than 3 FDs
Actually, sending 4 FDs will be useful for lei internal xsearch
work once we start accepting input from stdin. It won't be used
with the lightweight lei(1) client, however.
Eric Wong [Sun, 10 Jan 2021 12:15:11 +0000 (12:15 +0000)]
lei: query: ensure pager exit is instantaneous
Improve interactivity and user experience by allowing the user
to return to the terminal immediately when the pager is exited
(e.g. hitting the `q' key in less(1)).
This is a massive change which restructures query handling to
allow parallel search when --thread expansion is in use and
offloading to a separate worker when --thread is not in use.
The Xapian query offload changes allow us to reenter the event
loop right away once the search(es) are shipped off to the work
queue workers.
This means the main lei-daemon process can forget the lei(1)
client socket immediately once it's handed off to worker
processes.
We now unblock SIGPIPE in query workers and send an exit(141)
response to the lei(1) client socket to denote SIGPIPE.
This also allows parallelization for users using "lei q" from
multiple terminals.
JSON output is currently broken and will need to be restructured
for more flexibility and fork-safety.
Eric Wong [Sun, 10 Jan 2021 12:15:10 +0000 (12:15 +0000)]
lei: fix oneshot TTY detection by passing STD*{GLOB}
... instead of STD*{IO}. I'm not sure why *STDOUT{IO} being an
IO::File object disqualifies it from the "-t" perlop check
returning true on TTY, but it does. So use *STDOUT{GLOB} for
now.
Eric Wong [Sun, 10 Jan 2021 12:15:04 +0000 (12:15 +0000)]
ipc: work queue support via SOCK_SEQPACKET
This will allow any number of younger sibling processes to
communicate with older siblings directly without relying on a
mediator process. This is intended to be useful for
distributing search work across multiple workers without caring
which worker hits it (we only care about shard members).
And any request sent with this will be able to hit any worker
without locking on our part.
Unix stream sockets with a listener were also considered;
binding to a file on the FS may confuse users given there's
already a socket path for lei(1). Linux-only Abstract or
autobind sockets are rejected due to lack of portability.
SOCK_SEQPACKET via socketpair(2) was chosen since it's POSIX
2008 and available on FreeBSD 9+ in addition to Linux, and
doesn't require filesystem access.
Eric Wong [Sun, 10 Jan 2021 12:15:03 +0000 (12:15 +0000)]
ipc: avoid excessive evals
We should not need an eval for warning with our code base.
Nowadays, dwaitpid() automatically does the right thing
regardless of whether we're in the event loop, so no eval
is needed there, either.
Eric Wong [Sun, 10 Jan 2021 12:15:02 +0000 (12:15 +0000)]
cmd_ipc: send FDs with buffer payload
For another step in in syscall reduction, we'll support
transferring 3 FDs and a buffer with a single sendmsg/recvmsg
syscall using Socket::MsgHdr if available.
Beyond script/lei itself, this will be used for internal IPC
between search backends (perhaps with SOCK_SEQPACKET). There's
a chance this could make it to the public-facing daemons, too.
This adds an optional dependency on the Socket::MsgHdr package,
available as libsocket-msghdr-perl on Debian-based distros
(but not CentOS 7.x and FreeBSD 11.x, at least).
Our Inline::C version in PublicInbox::Spawn remains the last
choice for script/lei due to the high startup time, and
IO::FDPass remains supported for non-Debian distros.
Since the socket name prefix changes from 3 to 4, we'll also
take this opportunity to make the argv+env buffer transfer less
error-prone by relying on argc instead of designated delimiters.
Eric Wong [Sun, 10 Jan 2021 12:15:01 +0000 (12:15 +0000)]
ipc: add support for asynchronous callbacks
Similar to git->cat_async, this will let us deal with responses
asynchronously, as well as being able to mix synchronous and
asynchronous code transparently (though perhaps not optimally).
Eric Wong [Sun, 10 Jan 2021 12:15:00 +0000 (12:15 +0000)]
ds: block signals when reaping
This lets us call dwaitpid long before a process exits
and not have to wait around for it.
This is advantageous for lei where we can run dwaitpid on the
pager as soon as we spawn it, instead of waiting for a client
socket to go away on DESTROY.
Eric Wong [Sun, 10 Jan 2021 12:14:58 +0000 (12:14 +0000)]
lei query + pagination sorta working
Parallelism and interactivity with pager + SIGPIPE needs work;
but results are shown and phrase search works without shell
users having to apply Xapian quoting rules on top of standard
shell quoting.
Eric Wong [Tue, 5 Jan 2021 01:29:10 +0000 (01:29 +0000)]
v2writable: exact discontiguous history handling
We've always temporarily unindexeded messages before reindexing
them again if there's discontiguous history.
This change improves the mechanism we use to prevent NNTP and
IMAP clients from seeing duplicate messages.
Previously, we relied on mapping Message-IDs to NNTP article
numbers to ensure clients would not see the same message twice.
This worked for most messages, but not for for messages with
reused or duplicate Message-IDs.
Instead of relying on Message-IDs as a key, we now rely on the
git blob object ID for exact content matching. This allows
truly different messages to show up for NNTP|IMAP clients, while
still those clients from seeing the message again.
Eric Wong [Tue, 5 Jan 2021 09:04:37 +0000 (09:04 +0000)]
address: pairs: new helper for JMAP (and maybe lei)
Per JMAP RFC 8621 sec 4.1.2.3, we should be able to
denote the lack of a phrase/comment corresponding to an
email address with a JSON "null" (or Perl `undef').
Eric Wong [Tue, 5 Jan 2021 09:04:36 +0000 (09:04 +0000)]
lei: use client env as-is, drop daemon-env command
There may be subtle misbehaviours when mixing the existing
daemon env and the client-supplied env. Just do the simplest
thing and use the client env as-is.
We'll also start the ->event_step callback since we'll need
to remember some things for long-lived commands.
Eric Wong [Tue, 5 Jan 2021 09:04:34 +0000 (09:04 +0000)]
lei: completion: fix filename completion
"-o default" is what we want from "complete", "-o filename" just
tells readline the result from the "_lei" function might be a
filename and quote appropriately.