Eric Wong [Thu, 18 Feb 2021 20:22:24 +0000 (23:22 +0300)]
lei: consolidate the bulk of the IPC code
The backends for "lei add-external --mirror", "lei convert", and
"lei import" all share a similar pattern for spawning background
workers. Hoist out the common parts to slim down our code base
a bit.
The LeiXSearch and LeiToMail workers for "lei q" remains a the
odd duck due to the deep pipelining and parallelization.
Eric Wong [Thu, 18 Feb 2021 20:22:22 +0000 (23:22 +0300)]
lei convert: mail format conversion sub-command
This will make testing IMAP support for other commands easier, as
it doesn't write to lei/store at all. Like the pager and MUA,
"git credential" is always spawned by script/lei (and not
lei-daemon) so it has a controlling terminal for password
prompts.
v2: fix missing requires, correct test ordering
v3: ensure config exists for IMAP auth
Eric Wong [Wed, 17 Feb 2021 10:07:02 +0000 (09:07 -0100)]
tests: setup_public_inboxes: use IMAP-friendly newsgroups
-imapd won't support newsgroups ending with /\.[0-9]+\z/ since
it reserves those for partitioning inboxes into 50K slices.
So bump the home[0-9]+ version and switch to IMAP-friendly
newsgroup names.
Eric Wong [Thu, 11 Feb 2021 05:57:28 +0000 (12:57 +0700)]
search: query_approxidate: cleanup regexp, more tests
The cleanup doesn't seem to matter, I initially thought I needed
to handle "" (two double quotes) explicitly because that's what
Xapian does to escape a double quote inside a double-quoted
phrase. It turns out we only need to be able to pass phrases
through to Xapian unmodified, and the existing group of
["\x{201c}\x{201d}] is sufficient for our purposes.
Eric Wong [Fri, 12 Feb 2021 07:05:52 +0000 (00:05 -0700)]
mbox_reader: do not chomp non-blank EOL
It's conceivable some cases won't generate an empty line before
an mboxrd or mboxo From_ line. Ensure we can handle that case
and don't leave the Eml->{bdy} without a trailing LF character.
And drop an unnecessary alarm import while we're in the area.
Eric Wong [Fri, 12 Feb 2021 07:05:50 +0000 (00:05 -0700)]
filter/vger: kill trailing newlines aggressively
PublicInbox::MboxReader->(mboxrd|mboxo) only deletes the last
trailing newline, not every single trailing newline like
InboxWritable->import_mbox does.
Testing PublicInbox::MboxReader->mboxrd (next commit) with
scripts/import_vger_from_mbox on the LKML archive I got 2018 for
v2 development; this difference was responsible for a single
spam message(*) from out of 2722831 not being filtered correctly
and returning a different result.
Eric Wong [Wed, 10 Feb 2021 19:57:58 +0000 (18:57 -0100)]
search: use git approxidate in WWW and "lei q --stdin"
This greatly improves the usability of d:, dt:, and rt: search
prefixes for users already familiar git's "approxidate" feature.
That is, users familiar with the --(since|after|until|before)=
options in git-log(1) and similar commands will be able to use
those dates in the WWW UI.
Eric Wong [Wed, 10 Feb 2021 07:07:49 +0000 (07:07 +0000)]
net_reader: new package split from -watch
We'll be using some of this for IMAP and NNTP support in lei,
too. More will need to be done to improve code sharing and
reusability, soon, but this is a start.
Eric Wong [Wed, 10 Feb 2021 07:07:44 +0000 (07:07 +0000)]
lei *external: glob improvements, ls-external filtering
The "ls-external" now accepts the same glob patterns used by
with lei q --{include,only,exclude}. If no glob is detected, it
will be treated as a literal substring match (like "grep -F").
Eric Wong [Tue, 9 Feb 2021 08:09:37 +0000 (07:09 -0100)]
tests|lei: fixes for TEST_RUN_MODE=0 and lei oneshot
DESTROY callbacks can clobber $?, so we must take care to
preserve it when exiting. We'll also try to make an effort to
ensure better DESTROY ordering and delete as much as possible
before x_it finishes.
We also need to load PublicInbox::Config when setting up
public inboxes.
Eric Wong [Tue, 9 Feb 2021 08:09:34 +0000 (07:09 -0100)]
lei q: prefix --alert ops with ':' instead of '-'
Using dashed keywords confuses the option parser without
"=" signs (and bash completion doesn't yet work with "=").
So use ":" instead of "-" as the prefix for internal ops,
since ":" is just as unlikely to be the first character of
an executable file in a user's $PATH.
Eric Wong [Tue, 9 Feb 2021 08:09:33 +0000 (07:09 -0100)]
use MdirReader in -watch and InboxWritable
MdirReader now handles files in "$MAILDIR/new" properly and
is stricter about what it accepts. eml_from_path is also
made robust against FIFOs while eliminating TOCTOU races with
between stat(2) and open(2) calls.
Eric Wong [Tue, 9 Feb 2021 08:09:32 +0000 (07:09 -0100)]
t/run.perl: fix for >128 tests
We need to explicitly close the write-end of the pipe in workers
to ensure they don't prevent each other from seeing EOF.
Also, make a note to keep using the pipe for now since
Linux <3.14 had broken read(2) semantics when file descriptions
are shared across threads/processes.
Eric Wong [Tue, 9 Feb 2021 08:09:31 +0000 (07:09 -0100)]
lei: split out MdirReader package, lazy-require earlier
We'll do more requires in the top-level lei-daemon process to
save work in workers. We can also work towards aborting on
user errors in lei-daemon rather than worker processes.
"lei import -f mbox*" is finally tested inside t/lei_to_mail.t
Eric Wong [Tue, 9 Feb 2021 08:09:30 +0000 (07:09 -0100)]
git: ->qx: respect caller's $/ in array context
This could lead to bad results when doing ls-tree -z
for v2 import in case there's multiple files. In any case,
the `local $/ = "\0"' in Import.pm is also eliminated to
reduce potential confusion and surprises.
Eric Wong [Tue, 9 Feb 2021 08:09:29 +0000 (07:09 -0100)]
t/cgi.t: modernizations and style updates
We prefer BAIL_OUT or fail to die in tests (I didn't know
BAIL_OUT existed when I started the project). We can also
depend on IO::Uncompress::Gunzip being available,
We'll keep the cgi_run wrapper since the .cgi could
use some coverage and remove the FIXME note. run_script
makes tests fast enough.
Eric Wong [Mon, 8 Feb 2021 18:33:39 +0000 (08:33 -1000)]
www: stream mboxrd in descending docid order
Order doesn't matter when users are completely downloading
mboxrds onto the FS and then opening them with an MUA. The
MUA is expected to sort the results in the user's preferred
order.
However, lei can start streaming the results to its destination
Maildir (or eventually IMAP/JMAP mailbox) with an MUA already
open. This will let users see recent results sooner in their
MUA, as those tend to have a higher docid. This matches the
behavior of the HTML results, as well.
As a bonus, this is around ~5% faster in a one-off, informal
test case with 66k results. I expect this to hold true in all
all cases since git has always optimized storage to favor recent
objects.
Eric Wong [Mon, 8 Feb 2021 09:05:19 +0000 (23:05 -1000)]
lei q: use git approxidate with d:, dt: and rt: ranges
Instead of having --(sent|received)-(before|after)=s
command-line switches, we'll just try to make sense of argv so
it's usable within parenthesized statements and such.
Given the negligible performance penalty with Inline::C
process spawning, we'll probably wire this up to the
WWW interface, too.
"d:" is for mairix compatibility. I don't know if "dt:" and
"rt:" will be too useful, but they exist because of IMAP
(and JMAP).
Eric Wong [Mon, 8 Feb 2021 09:05:18 +0000 (23:05 -1000)]
git: implement date_parse method
Users are expected to be familiar with git's "approxidate"
functionality for parsing dates, so we'll expose that
in our UIs. Xapian itself has limited date parsing functionality
and I can't expect users to learn it.
This takes around 4-5ms on my aging workstation, so it'll
probably be made acceptable for the WWW UI, even.
libgit2 has a git__date_parse function which I expect to have
less overhead, but it's only for internal use at the moment.
Eric Wong [Mon, 8 Feb 2021 09:05:16 +0000 (23:05 -1000)]
lei: avoid racing on unlink + bind + listen
When multiple lei(1) processes are starting in parallel without
lei-daemon already running, it's possible for them to trample
each others' socket path trying to start lei-daemon. Lock
errors.log before unlink/bind/listen. We'll add an extra
connect(2) attempt to check if the starter lost the race.
Without this change, a stress script like the following could
easily cause problems:
lei q -o ~/tmp/a foo ... &
lei q -o ~/tmp/b bar ... &
lei q -o ~/tmp/c quux ... &
lei q -o ~/tmp/d baz ... &
Eric Wong [Mon, 8 Feb 2021 09:05:15 +0000 (23:05 -1000)]
lei: start_pager: drop COLUMNS default
It shouldn't be needed since none of our subcommands will care
or attempt to format output. Once "lei show" is implemented,
we'll run "git show" directly on the result.
Eric Wong [Mon, 8 Feb 2021 09:05:13 +0000 (23:05 -1000)]
tests: favor IPv6
IPv4 gets plenty of real-world coverage, and apparently there's
Debian buildd hosts which lack IPv4(*). So ensure everything
can work on IPv6 and not cause problems for odd setups.
Eric Wong [Mon, 8 Feb 2021 09:05:12 +0000 (23:05 -1000)]
lei q: support --alert=CMD for early MUA users
For --mua users writing to lock-free -o MFOLDER destinations;
we'll keep -WINCH and send an ASCII terminal bell when results
are complete. This is intended to let early MUA spawners know
when lei2mail is done writing results.
We'll also support running arbitrary commands. It may be used
to run play(1) (from SoX), handle pipelines+redirects
(e.g. "/bin/sh -c 'echo search done | wall'") or other commands.
Eric Wong [Mon, 8 Feb 2021 09:05:09 +0000 (23:05 -1000)]
lei q: improve remote mboxrd UX + MUA
For early MUA spawners using lock-free outputs, we we need to
on the startq pipe to silence progress reporting. For
--augment users, we can start the MUA even earlier by
creating Maildirs in the pre-augment phase.
To improve progress reporting for non-MUA (or late-MUA)
spawners, we'll no longer blindly append "--compressed" to the
curl(1) command when POST-ing for the gzipped mboxrd.
Furthermore, we'll overload stringify ('""') in LeiCurl to
ensure the empty -d '' string shows up properly.
v2: fix startq waiting with --threads
mset_progress is never shown with early MUA spawning,
The plan is to still show progress when augmenting and
deduping. This fixes all local search cases.
A leftover debug bit is dropped, too
Eric Wong [Mon, 8 Feb 2021 09:11:03 +0000 (09:11 +0000)]
INSTALL: depend on Text::ParseWords
It's been distributed with Perl since 1994, and we use it for
both -imapd and lei. It's split out as a separate package in
CentOS 7.x, so we'll depend on it to avoid surprising users
of RPM-based distros.
Eric Wong [Sun, 7 Feb 2021 08:52:00 +0000 (08:52 +0000)]
imap: avoid unnecessary on-stack delete
None of the Content-Type attributes are long-lived
(and unlikely to be memory intensive). While these
callsites won't trigger $DB::args segfaults via
confess or longmess, it'll make future code audits
easier.
cf. commit 0795b0906cc81f40
("ds: guard against stack-not-refcounted quirk of Perl 5")
Eric Wong [Sun, 7 Feb 2021 08:51:56 +0000 (08:51 +0000)]
lei: replace --thread with --threads
Nobody is expected to use long options, but for consistency
with mairix(1), we'll use the pluralized option throughout
(including existing PublicInbox::{Search,SearchView}).
Eric Wong [Sun, 7 Feb 2021 08:51:55 +0000 (08:51 +0000)]
lei: remove --mua-cmd alias for --mua
While "mua-cmd" may be more accurate, nobody is expected
to type 4 extra characters. It's a needless ambiguity
with no precedence or prior art to follow.
Eric Wong [Sun, 7 Feb 2021 08:51:47 +0000 (08:51 +0000)]
ipc: do not die inside wq_worker child process
die() in a child zips up the stack into the parent, which is
undesirable behavior. We're going to exit anyways, just warn
and let exit(1) happen due to $@ being set.
Eric Wong [Sun, 7 Feb 2021 08:51:46 +0000 (08:51 +0000)]
spawn_pp: die more consistently in child
The default $SIG{__DIE__} inside a forked child doesn't actually
do what we want it to do. We don't want it to zip up the stack
the parent used, but instead want to exit the child process
after warning.
Eric Wong [Sun, 7 Feb 2021 08:51:44 +0000 (08:51 +0000)]
spawn: pi_fork_exec: support "pgid"
We'll be using this to allow the "git clone" process hierarchy
to be killed via Ctrl-C. This also fixes a long-standing bug
in error reporting for the Inline::C version, because we're
actually testing for errors, now!
n.b. strlen(3) is officially async-signal-safe as of
POSIX.1-2016, but I can't think of a reason any previous
implementation prior to that wouldn't be.
Eric Wong [Sun, 7 Feb 2021 08:51:43 +0000 (08:51 +0000)]
spawn: pi_fork_exec: restore parent sigmask in child
We continue to unblock SIGCHLD unconditionally, but also
any signals not blocked by the parent (wq_worker).
This will allow Ctrl-C (SIGINT) to stop "git clone" and allow
git-clone cleanup to be performed and other long-running
processes when pi_fork_exec supports setpgid(2). This won't
affect existing daemons on systems with signalfd(2) or
EVFILT_SIGNAL at all, since those run with signals blocked
anyways.
Eric Wong [Sat, 6 Feb 2021 12:18:43 +0000 (12:18 +0000)]
lei_curl: replace -K/--config with --curl-config
Seeing --config in the command-line for lei may mislead users
into thinking we support config file overrides that way. Rename
the option to --curl-config and drop the short switch for now.
Eric Wong [Sat, 6 Feb 2021 12:18:41 +0000 (12:18 +0000)]
lei help: split out into separate file
We'll reword and improve formatting with non-breaking spaces
("\xa0") which is only replaced with SP after wrapping.
Some terminology is shortened (e.g. "URL_OR_PATHNAME" => "LOCATION")
to improve formatting.
This also enables completion for -h/--help and lets us
prioritize favored switch names while attempting to
satisfy users relying on muscle memory from other tools.
Eric Wong [Sat, 6 Feb 2021 12:18:40 +0000 (12:18 +0000)]
lei: add-external --mirror support
This can be useful for users who want to clone and
mirror an existing public-inbox. This doesn't have
update support, yet, so users will need to run
"git fetch && public-inbox-index" for now.
Eric Wong [Sat, 6 Feb 2021 12:18:39 +0000 (12:18 +0000)]
script/lei: avoid waitpid(-1, ...) to keep tests fast
We only spawn one process to be reaped at the moment. tests
will run the contents of script/* in the same process if
possible, so any test scripts which spawn -httpd or other
read-only can cause us to stall with waitpid(-1, ...)
Eric Wong [Sat, 6 Feb 2021 12:18:38 +0000 (12:18 +0000)]
treewide: replace confess with croak
The PublicInbox::Eml (and previously Email::MIME) use of confess
was the primary (or only) culprit behind the lei2mail segfaults
fixed by commit 0795b0906cc81f40.
("ds: guard against stack-not-refcounted quirk of Perl 5").
We never care about a backtrace when dealing with Eml objects
anyways, so it was just a worthless waste of CPU cycles.
We can also drop confess in a few other places. Since we only
use Perl and Inline::C, users will never be without source
and can replace s/croak/Carp::confess/ on a per-callsite basis
to help report problems.
It's also possible to use PERL5OPT=-MCarp=verbose in the
environment though still potentially risky.
Eric Wong [Sat, 6 Feb 2021 12:18:34 +0000 (12:18 +0000)]
tests: add test_lei wrapper, split out t/lei-import.t
This will make it easier to maintain and test lei going forward,
we need to be testing against existing read-only daemons. We'll
also save ourselves some boilerplate by exporting all the
Test::More methods directly in TestCommon
We'll start using this by splitting out the latest "lei import"
tests into its own file.
Eric Wong [Sat, 6 Feb 2021 12:18:33 +0000 (12:18 +0000)]
lei_query: trim curl options
Get rid of short options which will or may conflict with
some of our own. We may switch over to "git -c http.*"
options since we need to run "git clone" and "git fetch"
anyways.
Since "keywords" is a bit long, support "kw" as a shortcut since
there's no conflict and "kw:" will be our search prefix for
looking up messages by keyword.
Eric Wong [Thu, 4 Feb 2021 09:59:29 +0000 (00:59 -0900)]
lei_xsearch: drop unused imports
Reaping is handled by the parent PublicInbox::IPC, and we
have no business using PublicInbox::Import since LeiXSearch
won't write to git directly (it will write via LeiStore).
Eric Wong [Thu, 4 Feb 2021 09:59:27 +0000 (00:59 -0900)]
lei q: eliminate $not_done temporary git dir hack
Another step towards simplifying lei internals.
None of our current uses of ->wq_do involve FD passing, and the
plan is only rely on FD passing between lei-daemon and lei(1).
Internally, it ought to be possible for lei-daemon internal bits
to be ordered properly to not need FD passing.