Eric Wong [Thu, 4 Aug 2022 08:17:02 +0000 (08:17 +0000)]
feed: avoid unnecessary map loop in non-over path
We can bless objects while doing the initial insertion to avoid
extra the extra map iteration and temporary array(s). Fewer ops
means memory savings for the likely case of ->over users, too.
Eric Wong [Thu, 4 Aug 2022 08:17:01 +0000 (08:17 +0000)]
imap: ensure_slices_exist: drop needless map and array
We can reduce ops and temporary objects here by folding the
stringification into the `for' loop and push directly into the
{mailboxlist} array; relying on autovivification to turn it into
a noop for the initial population.
Eric Wong [Thu, 4 Aug 2022 07:23:49 +0000 (07:23 +0000)]
TODO: remove done items, adjust/add/abandon some
public-inbox-pop3d (and -netd) gives us POP3 support, and
it seems to work. Proxy support can come independently,
probably after JMAP.
public-inbox-netd provides the multi-protocol "super server"
which allows code memory savings. Work is ongoing to further
reduce memory use...
Automatically updating on TLS cert and key changes on
inotify/EVFILT_VNODE won't be done, since (IMHO) there's too
much risk of inadvertent updates on incomplete changes.
My same train-of-thought applies to auto-reloading on config
file changes: an admin may save a file halfway through a
multi-step change and auto-reloading can be too surprising and
break things.
I don't think lei+FUSE will be as portable or useful as a
local IMAP server (and maybe JMAP, eventually); but r/w IMAP
support would be nice..
Finally, git SHA-256 repo support will need to be taken into
account.
Eric Wong [Thu, 4 Aug 2022 06:27:39 +0000 (06:27 +0000)]
daemon: handle per-listener options on inherited, well-known ports
We must not clobber already-parsed per-listener options when
handling inherited sockets which are well-known. Unfortunately,
this isn't easy to test in a non-intrusive way for regular
users.
Eric Wong [Wed, 3 Aug 2022 20:03:56 +0000 (20:03 +0000)]
nntp: speed up group listings via ->ALL->misc
By taking advantage of the new ART_MIN/ART_MAX value in MiscIdx,
we can avoid the overhead of opening per-inbox msgmap DB files.
The result gives us a ~40 speedup with 50K newgroups.
Eric Wong [Wed, 3 Aug 2022 08:06:03 +0000 (08:06 +0000)]
daemon: reload TLS certs and keys on SIGHUP
This allows new TLS certificates to be loaded for new clients
without having to timeout nor drop existing clients with
established connections made with the old certs. This should
benefit users with admins who expire certificates frequently (as
encouraged by Let's Encrypt).
Socket ->write failures are expected and common for TCP traffic,
especially if it's facing unreliable remote connections. So
just bail out silently if our {gz} field was already clobbered
during the small bit of recursion we hit on ->write failures
from async responses.
This ought to fix some GzipFilter::zflush errors (via $forward
->close from PublicInbox::HTTP) I've been noticing on
deployments running -netd. I'm still unsure as to why I hadn't
seen them before, but it might've only been ignorance on my
part...
Eric Wong [Mon, 1 Aug 2022 21:24:47 +0000 (21:24 +0000)]
daemon: share FDs for identical log paths
We rely on the %logs hash for SIGUSR1 log reopening. Without this sharing,
some FDs would be hidden inside its respective {HTTP,IMAP,POP3}D
object and not reopened on USR2
Eric Wong [Mon, 1 Aug 2022 21:24:42 +0000 (21:24 +0000)]
httpd: make internals slightly more generic
This brings the HTTP server closer to the IMAP/NNTP/POP3
implementations and eliminates package-wide globals in
PublicInbox::HTTPD. The end goal is to be able to host
completely different PSGI applications on different listen
ports.
Eric Wong [Sat, 30 Jul 2022 09:38:24 +0000 (09:38 +0000)]
solver: avoid deprecation warnings in git 2.36.0+
git deprecated core.fsyncObjectFiles in favor of core.fsync
with 2.36.0+, while GIT_TEST_FSYNC was added in 2.35.0. So
use the environment variable since it's been supported slightly
longer than the new configuration knob.
Eric Wong [Fri, 22 Jul 2022 20:18:09 +0000 (20:18 +0000)]
www: drop --subject from "git send-email" instructions
Apparently, --subject doesn't work[1] with "git send-email" in
this context. So drop the CLI arg and add a note to tell the
user to set a "Subject:" line in their response body, instead.
[1] I'm not sure if --subject ever worked as I thought it would,
or if it's a regression. In either case, there are current
versions of git where it doesn't, so just tell users to use
the currently supported method.
Eric Wong [Sat, 23 Jul 2022 15:52:09 +0000 (15:52 +0000)]
add xt/mem-nntpd-tls maintainer test
This ensures memory usage is reasonable when DEFLATE and TLS are
enabled. It's also our only coverage for NNTP COMPRESS since
Net::NNTP has yet to implement compression support:
Eric Wong [Sat, 23 Jul 2022 06:12:16 +0000 (06:12 +0000)]
pop3: reduce memory use while generating the mailbox cache
While the cache itself is relatively compact for 50K messages,
generating it was inefficient due to our schema and Over.pm APIs
being designed for NNTP. While we won't change our schema for
now, we can choose better DBI APIs to use and limit our ephemeral
memory use.
This amounts to a 60% reduction in memory usage and a 5-10%
speedup against org.kernel.vger.git.0:
Eric Wong [Sat, 23 Jul 2022 04:41:54 +0000 (04:41 +0000)]
nntp: resolve inboxes immediately on group listings
This prevents potential races between SIGHUP config reloads
while gigantic group listings are streaming, allowing us to
avoid many invalidation checks.
This also reduces send(2) syscalls and avoid Perl internal pad
allocations in a few places where it's not beneficial. There
might be a slight (0.5%) speedup, but I'm not sure if that's
down to system noise, power/thermal management, or other users
on my VM.
Eric Wong [Sat, 23 Jul 2022 04:41:51 +0000 (04:41 +0000)]
nntp: listgroup_range_i: remove useless `map' op
No need to iterate through the array twice; and this even seems
a hair faster than what I got with commit 726d6e71aee5d974
(nntp: small speed up for multi-line responses, 2020-12-04)
Eric Wong [Thu, 21 Jul 2022 05:36:12 +0000 (05:36 +0000)]
pop3: drop File::FcntlLock requirement for FreeBSD and Linux
I know Linux has a stable ABI for this, and FreeBSD seems to,
too (*BSDs don't have stable syscall numbers, though).
I suspect this is safe enough for all *BSDs.
This is stricter than the MboxLock one since we use exact byte
ranges with these locks.
Eric Wong [Wed, 20 Jul 2022 22:57:07 +0000 (22:57 +0000)]
www: note "x=m" and "t=1" (mis)use for GET requests
We require "x=m" (requests for mboxes) to be POST requests to
avoid unnecessary traffic from crawlers. "t=1" only collapses
threads in the summary view, which isn't normally accessible
from <form> elements.
This also fixes the missing "[summary|nested]" element when
"x=m" is used.
Eric Wong [Wed, 20 Jul 2022 09:24:11 +0000 (09:24 +0000)]
pop3: TOP requests do not expire messages
RFC 2449 only documents "EXPIRE 0" behavior for RETR requests
which fetch the whole message. TOP requests only fetch
the headers and top $N lines of the body, so it's probably
harmful for deletions to be triggered in those cases.
Eric Wong [Wed, 20 Jul 2022 09:24:09 +0000 (09:24 +0000)]
public-inbox-pop3d - a mostly read-only POP3 server
Old account expiry has not been implemented, but it seems to
work well with both mpop(1) and getmail(1). The strictness of
mpop was particularly helpful in ironing out bugs in our
implementation of (dreaded) message sequence numbers.
"EXPIRE 0" (RFC 2449) can theoretically save numerous "DELE"
commands, but that's untested by real-world clients. mpop
supports PIPELINING which is effective in hiding latency,
and the core networking functionality is already well-tested
from our NNTP and IMAP implementations.
Configuration requires "publicinbox.pop3state" to point to
a directory writable by the otherwise read-only daemon.
See public-inbox-pop3d(1) manpage for more usage details.
Eric Wong [Wed, 20 Jul 2022 01:22:04 +0000 (01:22 +0000)]
netd: load modules for well-known ports
When inheriting well-known ports from systemd (or similar),
we can auto-load the proper *D.pm file based on the port number
without requiring command-line args.
load_mod also gets fixed to use its argument, instead of implicit
$1 since that won't work for our well-known.
Eric Wong [Tue, 19 Jul 2022 22:42:53 +0000 (22:42 +0000)]
lei note-event: inline note_event_arm_done
This was a single-caller sub since 47d4e53734820b4e
(lei_mail_sync: rely on flock(2), avoid IPC, 2021-09-18)
and unlikely to be used further, so inline it and save
a few KB of memory.
Eric Wong [Tue, 19 Jul 2022 22:42:52 +0000 (22:42 +0000)]
lei: avoid deadlock on inotify/EVFILT_VNODE wakeups
Enqueuing "note-event" requests from the DS event loop must
not wait on workers being able to drain the queue quickly
enough. Thus we make the SOCK_SEQPACKET writes nonblocking
and rely on the lei-daemon event loop to enqueue writes.
This is a unique problem for "note-event" since it reuses
workers in between commands, while most lei commands currently
fork off new workers.
Eric Wong [Mon, 20 Jun 2022 19:27:30 +0000 (19:27 +0000)]
search: do not index base-85 binary patches
Base-85 binary patches generated by git lead to many false
positives, so skip over gibberish words which may occur in them.
To avoid regressions in search results, continue to allow
searching for exact size matches (via "literal $SIZE") and the
phrase "GIT binary patch" for the mere presence of a binary
patch.
Eric Wong [Mon, 20 Jun 2022 19:27:29 +0000 (19:27 +0000)]
search: support "patchid:" prefix (git patch-id --stable)
This allows easy searching via patch-id from a git commit.
Currently, abbreviations are not supported, and it seems
needless to support them since AFAIK (git) doesn't generate
nor resolve abbreviated patch-ids anywhere.
t/spawn: Find invalid PID to try to join its process group
In the container used to build packages of the GNU Guix distribution, PID 1
runs as the same user as the test so this spawn that should fail actually
succeeds.
Fix the problem by going through different PIDs and picking one that
either doesn't exist or we aren't allowed to signal.
Eric Wong [Thu, 5 May 2022 10:52:15 +0000 (10:52 +0000)]
public-inbox-netd: a multi-protocol superserver
Since we'll be adding POP3 support as our 4th network protocol;
asking admins to run yet another daemon on top of existing
-httpd, -nntpd, -imapd is a maintenance burden and a waste of
memory.
The goal of public-inbox-netd is to be able to replace all
existing read-only daemons with a single process to save memory
and reduce administrative overhead; hopefully encouraging more
users to self-host their own mirrors.
It's barely-tested at the moment. Eventually, multiple
PI_CONFIG and HOME directories will be supported, as are
per-listener .psgi config files.
Eric Wong [Mon, 2 May 2022 18:10:07 +0000 (18:10 +0000)]
lei import: add label completions (+L:$LABEL)
This can probably be added for "lei q", too, but we typically
import first. Labels can probably be made persistent on a
per-folder basis in the future.
Eric Wong [Sat, 30 Apr 2022 21:29:30 +0000 (21:29 +0000)]
lei refresh-mail-sync: filter NNTP(S) from --all
We currently do not support refresh from NNTP since deletes are
rare with public-inbox NNTP servers; but traditional Usenet
servers do delete/expire messages and we should probably support
that at some point.
Eric Wong [Thu, 21 Apr 2022 11:59:06 +0000 (11:59 +0000)]
lei: commit store on interrupted partial imports
This change prevents lingering shard and git-fast-import
processes from remaining after interrupted "lei import" (and
similar). It also reduces the likelyhood of data-loss in case
of subsequent abnormal termination of the daemon.
I think this is the least surprising way to handle users
prematurely aborting imports or other similar operations which
write to lei/store and will result in reduced bandwidth waste
for users with intermittent connections. This is because the
lei/store processes may be shared by parallel "lei import"
callers, and commits done by any "lei import" caller will
inevitably trigger writes for all of them.
Eric Wong [Mon, 18 Apr 2022 09:50:04 +0000 (09:50 +0000)]
syscall: golf + more idiomatic buffer initialization
While `vec' is useful for user-supplied buffers to avoid excess
memory traffic, but provides no benefit when we need to allocate
our own buffers as we do in nodatacow_fh, since Perl can't elide
memset(ptr, 0, len). So just use the idiomatic `"\0" x $LEN' here.
Eric Wong [Mon, 18 Apr 2022 09:50:03 +0000 (09:50 +0000)]
lei: wire up pure Perl sendmsg/recvmsg for Linux users
This enables lei-daemon to work without Inline::C nor
Socket::MsgHdr installed. Prior to this, only the `lei' client
was using the pure Perl implementation. Either C implementation
is still marginally faster, however.
Eric Wong [Mon, 18 Apr 2022 09:50:02 +0000 (09:50 +0000)]
syscall: more idiomatic cmsghdr space allocation
Since we know the space required under Linux, we can use the
same initialization as the Inline::C version instead of
hard-coding 256 as we do for Socket::MsgHdr.
Eric Wong [Mon, 18 Apr 2022 09:44:01 +0000 (09:44 +0000)]
lei_mail_sync: explicit bind for old SQL_VARCHAR compat
This avoids repeated work for incremental "lei import" runs when
users upgrade from 1.7 to current public-inbox.git (and eventually
1.8).
We need the explicit bind_param for fallback calls because
previous bind_param calls are "sticky" for a given statement
handle. The DBI(3pm) manpage states:
The data type is 'sticky' in that bind values passed to execute()
are bound with the data type specified by earlier bind_param()
calls, if any. Portable applications should not rely on being
able to change the data type after the first "bind_param" call.
Eric Wong [Tue, 5 Apr 2022 08:18:24 +0000 (08:18 +0000)]
lei: always open mail_sync.sqlite3 R/W
This will make transparently upgrading from 1.7.0 -> 1.8.x
easier. Only a single user has access to mail_sync.sqlite3,
and R/W at the kernel-level is required for WAL, anyways.
Eric Wong [Sat, 2 Apr 2022 01:13:51 +0000 (01:13 +0000)]
lei_mail_sync: ensure URLs and folder names are stored as binary
Apparently leaving {sqlite_unicode} unset isn't enough, and
there's subtle differences where BLOBs are stored differently
than TEXT when dealing with binary data. We also want to avoid
odd cases where SQLite will attempt to treat a number-like value
as an integer.
This should avoid problems in case non-UTF-8 URLs and pathnames are
used. They'll automatically be upgraded if not, but downgrades
to older lei would cause duplicates to appear.