Eric Wong [Sat, 23 Oct 2021 21:53:46 +0000 (21:53 +0000)]
cmd_ipc4: retry sendmsg on ENOBUFS/ENOMEM/ETOOMANYREFS
I'm seeing ENOBUFS on a RAM-starved system, and slowing the
sender down enough for the receiver to drain the buffers seems
to work. ENOMEM and ETOOMANYREFS could be in the same boat
as ENOBUFS.
Watching for POLLOUT events via select/poll/epoll_wait doesn't
seem to work, since the kernel can already sleep (or return
EAGAIN) for cases where POLLOUT would work.
Eric Wong [Fri, 22 Oct 2021 08:22:45 +0000 (08:22 +0000)]
lei export-kw: don't recreate deleted IMAP folders
In case an IMAP folder is deleted, just set an error and
ignore it rather than creating an empty folder which we
attempt to export keywords to for non-existent messages.
Kyle Meyer [Fri, 22 Oct 2021 04:49:35 +0000 (00:49 -0400)]
wwwatomstream: call gmtime with scalar
When the gmtime() calls were moved from feed_entry() and atom_header()
into feed_updated() in c447bbbd, @_ rather than a scalar was passed to
gmtime(). As a result, feed <updated> values end up as
"1970-01-01T00:00:00Z".
Switch back to using a scalar argument to restore the correct
timestamps.
Eric Wong [Thu, 21 Oct 2021 21:10:32 +0000 (21:10 +0000)]
lei: use RENAME_NOREPLACE on Linux 3.15+
One syscall is better than two for atomicity in Maildirs. This
means there's no window where another process can see both the
old and new file at the same time (link && unlink), nor a window
where we might inadvertantly clobber an existing file if we were
to do `stat && rename'.
Eric Wong [Thu, 21 Oct 2021 21:10:25 +0000 (21:10 +0000)]
watch: check for {quit} before IDLE
This may make it less likely for watch-dependent tests to get
stuck. Unfortunately, due to the synchronous API of
Mail::IMAPClient, ->idle is still susceptible to missing
signals.
Eric Wong [Thu, 21 Oct 2021 21:10:18 +0000 (21:10 +0000)]
t/lei-{auto-watch,export-kw}: extra diagnostics on failure
Maybe these will help track down some failures and make
diagnosing bugs easier. "lei export-kw" should also become
optional, even, so allow disabling it easily in the test.
Eric Wong [Tue, 19 Oct 2021 21:26:15 +0000 (21:26 +0000)]
httpd: reject requests with spaces in header names
Malicious clients may attempt HTTP request smuggling this way.
This doesn't affect our current code as we only look for exact
matches, but it could affect other servers behind a
to-be-implemented reverse proxy built around our -httpd.
This doesn't affect users behind varnish at all, nor the
HTTPS/HTTP reverse proxy I use (I don't know about nginx), but
could be passed through by other reverse proxies.
This change is only needed for HTTP::Parser::XS which most users
probably use. Users of the pure Perl parser (via
PLACK_HTTP_PARSER_PP=1) already hit 400 errors in this case,
so this makes the common XS case consistent with the pure Perl
case.
Eric Wong [Tue, 19 Oct 2021 09:33:45 +0000 (09:33 +0000)]
lei inspect: show ISO8601 {rt} and {dt}, too
While inspect is intended for debugging, the Unix epoch in
seconds requires extra steps for human consumption; just
steal what we used for "lei q -f json" output.
Eric Wong [Tue, 19 Oct 2021 09:33:40 +0000 (09:33 +0000)]
lei: conditionally add "\n" to error messages
Some error messages already include "\n" (w/ file+line info),
so don't add another one. (`warn' will automatically add its
caller location unless there's a final "\n").
Makefile.PL: drop generated lib/PublicInbox.pm in blib/
Running "make test" on this project doesn't pass unless you've got an
existing PublicInbox.pm in your @INC, presumably nobody's set this up
on a fresh machine in a while.
This Makefile.PL trickery seems to do it, I've validated this with
this ad-hoc test of committing blib/ and Makefile to the repository:
git clean -dxf; perl Makefile.PL && make -j8 all && git add -f blib Makefile.PL Makefile && git commit -m"now"
Running that in interactive rebase before/after shows that only the
PublicInbox.pm file was added to blib/lib/. We use $(INST_LIB) instead
of a hardcoded 'blib/lib' now, but it's what ExtUtils::MakeMaker
recommends, so it's probably for the better.
As far as I can tell this broke with 1fae720d (build: generate
PublicInbox.pm with $VERSION, 2021-04-01), but I have not tested
that. See also 1fae720d (build: generate PublicInbox.pm with $VERSION,
2021-04-01) which made the PublicInbox.pm a generated file.
For odd messages with reused Message-IDs, the second message
showing up in a mirror (via git-fetch + -index) should never
clobber an entry with a different blob in over.
This is noticeable only if the messages arrive in-between
indexing runs.
Fixes: 4441a38481ed ("v2: index forwards (via `git log --reverse')")
Eric Wong [Sat, 16 Oct 2021 19:11:33 +0000 (19:11 +0000)]
msgmap: do not cache num_highwater
Caching the value doesn't seem necessary from a performance
perspective, and it adds a caveat for read-only users which
may lead to bugs in future code.
Eric Wong [Sat, 16 Oct 2021 23:23:01 +0000 (23:23 +0000)]
eml: fix leak workaround
Our previous workaround didn't actually work around the leak in
<https://rt.cpan.org/Public/Bug/Display.html?id=139622> since
croak()-via-Perl was still invoked before the SV reference
count could be decremented.
Put in a proper workaround which saves warnings onto a temporary
variable and only croak after ->decode or ->encode returns; not
inside those methods.
Eric Wong [Sat, 16 Oct 2021 09:29:52 +0000 (09:29 +0000)]
input_pipe: do not loop in ->event_step for fairness
Sigfd->event_step needs priority over InputPipe (and everything
else). We keep Edge Triggering here but use ->requeue instead
of looping inside event_step. This was necessary because
InputPipe can be used with regular files which can't be
monitored with epoll.
We'll also rid of the vestigial lei-oneshot support while we're
at it.
Eric Wong [Sat, 16 Oct 2021 09:29:51 +0000 (09:29 +0000)]
pkt_op: favor level-triggered epoll for fairness
Sigfd->event_step needs priority over PktOp (and everything else).
We'll also add ECONNRESET checking, here, since it could see
bidirectional use in the future.
This is unlikely to have any sort of performance difference
since this is only for small, occasional packets, but the code
reduction is nice.
Eric Wong [Sat, 16 Oct 2021 09:29:50 +0000 (09:29 +0000)]
wqworker: favor level-triggered epoll for fairness
Sigfd->event_step needs priority over WQWorkers (and everything
else). Do that by running once per event_loop iteration rather
than looping inside event_step. This lowers throughput since it
requires more syscalls, but that's the price of fairness.
Eric Wong [Sat, 16 Oct 2021 01:41:34 +0000 (01:41 +0000)]
extindex: avoid triggering a buggy unref
We can't attempt to unref messages beyond the highwater mark of
an inbox. This bugfix was found by commit c485036d0b1ce7ed
(extindex: guard against buggy unrefs, 2021-10-14), which
actually did its intended job and guarded against a buggy unref.
Eric Wong [Sat, 16 Oct 2021 01:00:57 +0000 (01:00 +0000)]
lei: always keep cwd fd {3} for ->fchdir
The extra FD shouldn't cause noticeable overhead in short-lived
workers, and it lets us simplify lei->rel2abs. Get rid of a
2-argument form of open() while we're at it, since it's been
considered for warning+deprecation by Perl for safety reasons.
Eric Wong [Sat, 16 Oct 2021 01:00:55 +0000 (01:00 +0000)]
httpd: move pipeline logic into event_step
Most of the HTTP server code was written for Danga::Socket and
not fully-transitioned to take advantage of PublicInbox::DS.
This change brings it up-to-date with the style of pipeline
handling used for -imapd and -nntpd.
Eric Wong [Sat, 16 Oct 2021 01:00:54 +0000 (01:00 +0000)]
imapd+nntpd: drop timer-based expiration
It's needlessly complex and O(n), so it doesn't scale well to a
high number of clients nor is it easy-to-scale with the data
structures available to us in pure Perl.
In any case, I see no evidence of either -imapd nor -nntpd
experiencing high connection loads on public-facing sites.
-httpd has never had its own timer-based expiration, either.
Fwiw, public-inbox.org itself has been running a public-facing
HTTP/HTTPS server with no userspace idle client expiration for
the past 8 years or with no ill effect. Clients can come and go
as they wish, and SO_KEEPALIVE takes care of truly broken
connections if they're gone for ~2 hours.
Internet connections drop all time, so it should be harmless to
drop connections w/o warning since both NNTP and IMAP protocols
have well-defined semantics for determining if a message was
truncated (as does HTTP/1.1+).
Eric Wong [Fri, 15 Oct 2021 14:02:15 +0000 (14:02 +0000)]
lei forget-search: support multiple args
I've been testing a lot of searches which I don't want to keep
around, so make it easy to remove a bunch at once. We'll behave
like rm(1) and keep going in the face of failure.
Eric Wong [Fri, 15 Oct 2021 13:30:55 +0000 (13:30 +0000)]
lei + ipc: simplify process reaping
Simplify our APIs and force dwaitpid() to work in async mode for
all lei workers. This avoids having lingering zombies for
parallel searches if one worker finishes soon before another.
The old distinction between "old" and "new" workers was
needlessly complex, error-prone, and embarrasingly bad.
We also never handled v2:// writers properly before on
Ctrl-C/Ctrl-Z (SIGINT/SIGTSTP), so add them to @WQ_KEYS
to ensure they get handled by $lei when appropropriate.
Eric Wong [Fri, 15 Oct 2021 09:52:53 +0000 (09:52 +0000)]
lei q: avoid kw lookup failure on remote mboxrd
When importing several sources in parallel via http(s) mboxrd,
we need to be able to get keywords of uncommitted documents
directly from shard workers. Otherwise, Xapian DocNotFound
errors happen because the read-only LeiSearch won't see
documents from uncomitted transactions. Keep in mind that it's
possible the keywords can be changed on-the-fly even for
uncommitted documents because of inotify watches from LeiNoteEvent.
Eric Wong [Fri, 15 Oct 2021 07:30:01 +0000 (07:30 +0000)]
www: various help text updates
`dt:' documentation is redundant with `d:' approxidate support;
so drop `dt:' since mairix uses `d:'. We'll also document
`rt:' since there are legit messages from senders with broken
clocks.
Reduce indentation level of help texts to be in 2-space
increments to using too much horizontal space.
We'll always place IMAP ahead of NNTP since it's alphabetical
and there's likely more IMAP clients out there.
Add "--ng NEWSGROUP" to -init instructions if configured.
There's also some minor wording changes throughout.
Eric Wong [Thu, 14 Oct 2021 13:16:09 +0000 (13:16 +0000)]
lei up --all: send signals to workers, receive errors
The redispatch mechanism wasn't routing signals and messages
between redispatched workers and script/lei properly. We now
rely on PktOp to do bidirectional message forwarding and
carefully avoiding circular references by using PktOp.
Eric Wong [Thu, 14 Oct 2021 13:16:07 +0000 (13:16 +0000)]
lei: TSTP affects all curl and related subprocesses
By relying more on pgroups for remaining remaining processes,
this lets us pause all curl+tail subprocesses with a single
kill(2) to avoid cluttering stderr.
We won't bother pausing the pigz/gzip/bzip2/xz compressor
process not cat-file processes, though, since those don't write
to the terminal and they idle soon after the workers react to
SIGSTOP.
AutoReap is hoisted out from TestCommon.pm. CLONE_SKIP
is gone since we won't be using Perl threads any time
soon (they're discouraged by the maintainers of Perl).
Eric Wong [Thu, 14 Oct 2021 13:16:06 +0000 (13:16 +0000)]
git: cat-file --batch are their own pgrp
We want these long-lived processes to die naturally when their
parent dies. Hopefully this improves graceful shutdown for
-extindex because I'm interrupting a lot of reindexing...
Eric Wong [Thu, 14 Oct 2021 13:16:05 +0000 (13:16 +0000)]
git: ->fail invokes current callback
While we try to invoke all pending callbacks to force error
handling, the current callback wasn't getting invoked on
invoked on async_abort if my_read/my_readline failed.
Eric Wong [Thu, 14 Oct 2021 04:32:53 +0000 (04:32 +0000)]
clone+fetch: respect umask for all downloaded files
Since public inboxes are usually intended to be public,
the File::Temp default permission of 0600 is wrong.
Just respect the user's umask in this case as git-clone
does.
This doesn't work for "lei add-external --mirror", yet;
but it will...
Eric Wong [Thu, 14 Oct 2021 03:12:25 +0000 (03:12 +0000)]
lei inspect: account for non-extindex inboxes
Inbox->xdb does not exist, but this code path was apparently
never tested :x I noticed this on basic v2 inbox, but it could
happen with any v1/v2 inbox. Move ->num2docid into Search
so it's less awkward to use.
Eric Wong [Thu, 14 Oct 2021 06:06:29 +0000 (06:06 +0000)]
extindex: guard against buggy unrefs
I noticed some unref messages which shouldn't have been
happening, but they were. Which is troubling. So add
a guard around an unref path until we can get to the bottom
of this.
Eric Wong [Wed, 13 Oct 2021 10:16:08 +0000 (10:16 +0000)]
eml: avoid Encode 2.87..3.12 leak
Encode::FB_CROAK leaks memory in old versions of Encode:
<https://rt.cpan.org/Public/Bug/Display.html?id=139622>
Since I expect there's still many users on old systems and old
Perls, we can use "$SIG{__WARN__} = \&croak" here with
Encode::FB_WARN to emulate Encode::FB_CROAK behavior.
Eric Wong [Wed, 13 Oct 2021 10:16:07 +0000 (10:16 +0000)]
t/www_listing: require opt-in for grokmirror tests
grokmirror 2.x seems to idle in several places for 5s at-a-time,
causing t/www_listing.t to take longer than "make check-run" on
a 4-core system when run without grokmirror. So make it
optional but add some test knobs to allow tailing the log
output so I can see what's going on.
Eric Wong [Wed, 13 Oct 2021 07:00:36 +0000 (07:00 +0000)]
treewide: use warn() or carp() instead of env->{psgi.errors}
Large chunks of our codebase and 3rd-party dependencies do not
use ->{psgi.errors}, so trying to standardize on it was a
fruitless endeavor. Since warn() and carp() are standard
mechanism within Perl, just use that instead and simplify a
bunch of existing code.