Eric Wong [Wed, 10 Jun 2020 07:05:03 +0000 (07:05 +0000)]
imap: rely on smsg->{bytes} for RFC822.SIZE
Since we started indexing the CRLF-adjusted size of messages,
we can take an order-of-magnitude speedup for certain MUAs
which fetch this attribute without needing much else.
Admins are encouraged to --reindex existing inboxes for IMAP
support, anyways. It won't be fatal if it's not reindexed, but
some client bugs and warnings can be fixed and they'll be able
to support more of IMAP.
Eric Wong [Wed, 10 Jun 2020 07:05:02 +0000 (07:05 +0000)]
index: account for CRLF conversion when storing bytes
NNTP and IMAP both require CRLF conversions on the wire.
They're also the only components which care about
$smsg->{bytes}, so store the CRLF-adjusted value in over.sqlite3
and Xapian DBs..
This will allow us to optimize RFC822.SIZE fetch item in IMAP
without triggering size mismatch errors in some clients' default
configurations (e.g. Mail::IMAPClient), but not most others.
It could also fix hypothetical problems with NNTP clients that
report discrepancies between overview and article data.
Eric Wong [Wed, 10 Jun 2020 07:05:01 +0000 (07:05 +0000)]
searchidx: v1 (re)-index uses git asynchronously
We can cleanup some of our v1 code slightly and let git do
I/O+decoding in parallel. This gives a slight 2-4%
re-indexing performance boost even on an SSD.
Eric Wong [Wed, 10 Jun 2020 07:05:00 +0000 (07:05 +0000)]
imap: split ->logged_in attribute into a separate class
This is one boolean attribute not worth wasting space for.
With 20000 sockets, this reduces RSS by around 5% at a glance,
and locked hashes doesn't do us much good when clients
use compression, anyways.
Eric Wong [Wed, 10 Jun 2020 07:04:59 +0000 (07:04 +0000)]
imap: 30 minute auto-logout timer
RFC 3501 section 5.4 requires this to be >= 30 minutes,
10x higher than what is recommended for NNTP. Fortunately
our design is reasonably memory-efficient despite being Perl.
Eric Wong [Wed, 10 Jun 2020 07:04:58 +0000 (07:04 +0000)]
imap: IDLE: avoid extraneous wakeups, keep-alive
We should not waste memory for IDLE unless it's used on the most
recent inbox slice. We also need to keep the IDLE connection
alive regardless of $PublicInbox::DS::EXPTIME.
Eric Wong [Wed, 10 Jun 2020 07:04:56 +0000 (07:04 +0000)]
imap: UID FETCH: optimize for smsg-only case
We can avoid loading the entire message from git when mutt makes
a "UID FETCH" request for "(UID FLAGS)". This speeds mutt up by
more than an order-of-magnitude in informal measurements.
Eric Wong [Wed, 10 Jun 2020 07:04:55 +0000 (07:04 +0000)]
imap: compile UID FETCH to opcodes
This is just a hair faster and cacheable in the future, if we
need it. Most notably, this avoids doing PublicInbox::Eml->new
for simple "RFC822", "BODY[]", and "RFC822.SIZE" requests.
Eric Wong [Wed, 10 Jun 2020 07:04:54 +0000 (07:04 +0000)]
imap: remove dummies from sequence number FETCH
Dummy messages make for bad user experience with MUAs which
still use sequence numbers. Not being able to fetch a message
doesn't seem fatal in mutt, so just ignore (sometimes large)
gaps.
Eric Wong [Wed, 10 Jun 2020 07:04:52 +0000 (07:04 +0000)]
search: index byte size of a message for IMAP search
Searching for messages smaller than a certain size is allowed by
offlineimap(1), mbsync(1), and possibly other tools. Maybe
public-inbox-watch will support it, too.
I don't see a reason to expose searching by size via WWW search
right now (but maybe in the future, I could be convinced to).
Note: we only store the byte-size of the message in git,
this is typically LF-only and we won't have the correct
size after CRLF conversion for NNTP or IMAP.
Eric Wong [Wed, 10 Jun 2020 07:04:51 +0000 (07:04 +0000)]
over: get_art: use dbh->prepare_cached
This speeds up xt/imapd-validate.t by around 10% when used with
an abandoned patch to remove ->query_xover. We may also depend
on this further if we abandon storing doc_data in Xapian to save
disk space.
Eric Wong [Wed, 10 Jun 2020 07:04:46 +0000 (07:04 +0000)]
imap: EXAMINE/STATUS: return correct counts
We can share code between them and account for each 50K
mailbox slice. However, we must overreport these for
non-zero slices and just return lots of empty data for
high-numbered slices because some MUAs still insist
on non-UID fetches.
Eric Wong [Wed, 10 Jun 2020 07:04:42 +0000 (07:04 +0000)]
imap: omit $UID_END from mailbox name, use index
Having two large numbers separated by a dash can make visual
comparisons difficult when numbers are in the 3,000,000 range
for LKML. So avoid the $UID_END value, since it can be
calculated from $UID_MIN. And we can avoid large values of
$UID_MIN, too, by instead storing the block index and just
multiplying it by 50000 (and adding 1) on the server side.
Of course, LKML still goes up to 72, at the moment.
Eric Wong [Wed, 10 Jun 2020 07:04:41 +0000 (07:04 +0000)]
imapd: ensure LIST is sorted alphabetically, for now
I'm not sure this matters, and it could be a waste of
CPU cycles if no real clients care. However, it does
make debugging over telnet or s_client a bit easier.
Eric Wong [Wed, 10 Jun 2020 07:04:40 +0000 (07:04 +0000)]
imap: require ".$UID_MIN-$UID_END" suffix
Finish up the IMAP-only portion of iterative config reloading,
which allows us to create all sub-ranges of an inbox up front.
The InboxIdler still uses ->each_inbox which will struggle with
100K inboxes.
Having messages in the top-level newsgroup name of an inbox will
still waste bandwidth for clients which want to do full syncs
once there's a rollover to a new 50K range. So instead, make
every inbox accessible exclusively via 50K slices in the form of
"$NEWSGROUP.$UID_MIN-$UID_END".
This introduces the DummyInbox, which makes $NEWSGROUP
and every parent component a selectable, empty inbox.
This aids navigation with mutt and possibly other MUAs.
Finally, the xt/perf-imap-list maintainer test is broken, now,
so remove it. The grep perlfunc is already proven effective,
and we'll have separate tests for mocking out ~100k inboxes.
Eric Wong [Wed, 10 Jun 2020 07:04:38 +0000 (07:04 +0000)]
imap: break giant inboxes into sub-inboxes of 50K messages
This limit on mailbox size should keep users of tools like
mbsync (isync) and offlineimap happy, since typical filesystems
struggle with giant Maildirs.
I chose 50K since it's a bit more than what LKML typically sees
in a month and still manages to give acceptable performance on
my ancient Centrino laptop.
There were also no responses to my original proposal at:
<https://public-inbox.org/meta/20200519090000.GA24273@dcvr/>
so no objections, either :>
Eric Wong [Wed, 10 Jun 2020 07:04:37 +0000 (07:04 +0000)]
imap: case-insensitive mailbox name comparisons
IMAP RFC 3501 stipulates case-insensitive comparisons, and so
does RFC 977 (NNTP). However, INN (nnrpd) uses case-sensitive
comparisons, so we've always used case-sensitive comparisons for
NNTP to match nnrpd behavior.
Unfortunately, some IMAP clients insist on sending "INBOX" with
caps, which causes problems for us. Since NNTP group names are
typically all lowercase anyways, just force all comparisons to
lowercase for IMAP and warn admins if uppercase-containing
newsgroups won't be accessible over IMAP.
This ensures our existing -nntpd behavior remains unchanged
while being compatible with the expectations of real-world IMAP
clients.
Eric Wong [Wed, 10 Jun 2020 07:04:34 +0000 (07:04 +0000)]
xt: add imapd-validate and imapd-mbsync-oimap
imapd-validate is a beefed up version of our nntpd-validate test
which hammers the server with parallel connections over regular
IMAP, IMAPS, IMAP+STARTTLS; and COMPRESS=DEFLATE variants of
each of those. It uses $START_UID:$END_UID fetch ranges to
reduce requests and slurp many responses at once to saturate
"git cat-file --batch" processes.
mbsync(1) also uses pipelining extensively (but IMHO
unnecessarily), so it was able to shake out some bugs in
the async git code.
Finally, we remove xt/cmp-imapd-compress.t since it's
redundant now that we have PublicInbox::IMAPClient to work
around bugs in Mail::IMAPClient.
Eric Wong [Wed, 10 Jun 2020 07:04:33 +0000 (07:04 +0000)]
imapclient: wrapper for Mail::IMAPClient
We'll be using this wrapper class to workaround some upstream
bugs in Mail::IMAPClient. There may also be experiments with
new APIs for more performance.
Eric Wong [Wed, 10 Jun 2020 07:04:32 +0000 (07:04 +0000)]
git: async: automatic retry on alternates change
This matches the behavior of the existing synchronous ->cat_file
method. In fact, ->cat_file now becomes a small wrapper around
the ->cat_async method.
Eric Wong [Wed, 10 Jun 2020 07:04:31 +0000 (07:04 +0000)]
git: move async_cat reference to PublicInbox::Git
Trying to avoid a circular reference by relying on $ibx object
here makes no sense, since skipping GitCatAsync::close will
result in an FD leak, anyways. So keep GitAsyncCat contained to
git-only operations, since we'll be using it for Solver in the
distant feature.
Eric Wong [Wed, 10 Jun 2020 07:04:29 +0000 (07:04 +0000)]
imap: fix pipelining with async git
Since IMAP yields control to GitAsyncCat, IMAP->event_step may
be invoked with {long_cb} still active. We must be sure to
bail out of IMAP->event_step if that happens and continue to let
GitAsyncCat drive IMAP.
This also improves fairness by never processing more than one
request per ->event_step.
Eric Wong [Wed, 10 Jun 2020 07:04:24 +0000 (07:04 +0000)]
imap: support LSUB command
Since we only support read-only operation, we can't save
subscriptions requested by clients. So just list no inboxes as
subscribed, some MUAs may blindly try to fetch everything its
subscribed to.
Eric Wong [Wed, 10 Jun 2020 07:04:22 +0000 (07:04 +0000)]
imap: use git-cat-file asynchronously
This ought to improve overall performance with multiple clients.
Single client performance suffers a tiny bit due to extra
syscall overhead from epoll.
This also makes the existing async interface easier-to-use,
since calling cat_async_begin is no longer required.
Eric Wong [Wed, 10 Jun 2020 07:04:21 +0000 (07:04 +0000)]
git: do our own read buffering for cat-file
To work with our event loop, we must perform read buffering
ourselves or risk starvation, as there doesn't appear to be
a way to check the amount of data buffered in userspace by
by the PerlIO layers without resorting to C or XS.
This lets us perform fewer syscalls at the expense of more Perl
ops. As it stands, there seems to be a tiny performance
improvement, but more will be possible in the future.
Eric Wong [Wed, 10 Jun 2020 07:04:19 +0000 (07:04 +0000)]
imap: speed up HEADER.FIELDS[.NOT] range fetches
While we can't memoize the regexp forever like we do with other
Eml users, we can still benefit from caching regexp compilation
on a per-request basis.
A FETCH request from mutt on a 4K message inbox is around 8%
faster after this. Since regexp compilation via qr// isn't
unbearably slow, a shared cache probably isn't worth the
trouble of implementing. A per-request cache seems enough.
Eric Wong [Wed, 10 Jun 2020 07:04:15 +0000 (07:04 +0000)]
imap: simplify partial fetch structure
While the contents of normal %want hash keys are bounded in
size, %partial can cause more overhead and lead to repeated sort
calls on multi-message fetches. So sort it once and use
arrayrefs to make the data structure more compact.
Eric Wong [Wed, 10 Jun 2020 07:04:12 +0000 (07:04 +0000)]
imap: always include `resp-text' in responses
Mail::IMAPClient doesn't seem to mind the lack of `resp-text';
but it's required by RFC 3501. Preliminary tests with
offlineimap(1) indicates the presence of `resp-text' is
necessary, even if it's just the freeform `text'.
And make the `text' more consistent, favoring "done" over
"complete" or "completed"; while we're at it.
Eric Wong [Wed, 10 Jun 2020 07:04:10 +0000 (07:04 +0000)]
eml: each_part: single part $idx is 1
Instead of counts starting at 0, we start the single-part
message at 1 like we do with subparts of a multipart message.
This will make it easier to map offsets for "BODY[$SECTION]"
when using IMAP FETCH, since $SECTION must contain non-zero
numbers according to RFC 3501.
This doesn't make any difference for WWW URLs, since single part
messages cannot have downloadable attachments.
Eric Wong [Wed, 10 Jun 2020 07:04:04 +0000 (07:04 +0000)]
imap: delay InboxIdle start, support refresh
InboxIdle should not be holding onto Inbox objects after the
Config object they came from expires, and Config objects may
expire on SIGHUP.
Old Inbox objects still persist due to IMAP clients holding onto
them, but that's a concern we'll deal with at another time, or
not at all, since all clients expire, eventually.
Regardless, stale inotify watch descriptors should not be left
hanging after SIGHUP refreshes.
Eric Wong [Wed, 10 Jun 2020 07:04:03 +0000 (07:04 +0000)]
msgmap: split ->max into its own method
There's enough places where we only care about the max NNTP
article number to warrant avoiding a call into SQLite.
Using ->num_highwater in read-only packages such as
PublicInbox::IMAP is also incorrect, since that memoizes
and won't pick up changes made by other processes.
Eric Wong [Wed, 10 Jun 2020 07:04:01 +0000 (07:04 +0000)]
inboxidle: new class to detect inbox changes
This will be used to implement IMAP IDLE, first.
Eventually, it may be used to trigger other things:
* incremental internal updates for manifest.js.gz
* restart `git cat-file' processes on pack index unlink
* IMAP IDLE-like long-polling HTTP endpoint
And maybe more things we haven't thought of, yet.
It uses Linux::Inotify2 or IO::KQueue depending on what packages
are installed and what the kernel supports. It falls back to
nanosecond-aware Time::HiRes::stat() (available with Perl 5.10.0+)
on systems lacking Linux::Inotify2 and IO::KQueue.
In the future, a pure Perl alternative to Linux::Inotify2 may be
supplied for users of architectures we already support signalfd
and epoll on.
v2 changes:
- avoid O_TRUNC on lock file
- change ctime on Linux systems w/o inotify
- fix naming of comments and fields
Eric Wong [Wed, 10 Jun 2020 07:04:00 +0000 (07:04 +0000)]
preliminary imap server implementation
It shares a bit of code with NNTP. It's copy+pasted for now
since this provides new ground to experiment with APIs for
dealing with slow storage and many inboxes.
Eric Wong [Sun, 7 Jun 2020 20:02:15 +0000 (20:02 +0000)]
index: v2: parallel by default
InboxWritable should only set $v2w->{parallel} if the $parallel
flag is defined to 0 or 1. We want indexing a new inbox to
utilize SMP, just like --reindex.
-index once again allows -j0/--jobs=0 to force single-process
use, and we'll be ensuring that works in tests to maintain
performance on small systems.
Eric Wong [Fri, 5 Jun 2020 02:01:10 +0000 (02:01 +0000)]
searchidx: v1: fix retries when Xapian and Msgmap are out-of-sync
We forcibly stop git-log here, so erroring out on git-log close
failures is wrong since it sees SIGPIPE. Noticed while
reindexing a large v1 inbox for IMAP changes.
Fixes: b32b47fb12a3043d ("index: "git log" failures are fatal")
Eric Wong [Mon, 1 Jun 2020 10:06:57 +0000 (10:06 +0000)]
wwwatomstream: drop smsg->{mid} fallback for non-SQLite
It's no longer necessary to populate the smsg->{mid} field now
that ->smsg_eml calls smsg->populate in rare cases where the
smsg did not originate from SQLite.
Eric Wong [Mon, 1 Jun 2020 10:06:56 +0000 (10:06 +0000)]
smsg: remove remaining accessor methods
We'll continue to favor simpler data models that can be
used directly rather than wasting time and memory with
accessor APIs.
The ->from, ->to, -cc, ->mid, ->subject, >references methods can
all be trivially replaced by hash lookups since all their values
are stored in doc_data. Most remaining callers of those methods
were test cases, anyways.
->from_name is only used in the PSGI code, so we can just
use ->psgi_cull to take care of populating the {from_name}
field.
Eric Wong [Mon, 1 Jun 2020 10:06:55 +0000 (10:06 +0000)]
smsg: remove ->bytes and ->lines methods
They're stored directly in Xapian and SQLite document data.
NNTP accesses those fields directly to avoid method invocation
overhead so there's no reason to waste several kilobytes for
each sub.
Eric Wong [Mon, 1 Jun 2020 10:06:49 +0000 (10:06 +0000)]
smsg: introduce ->populate method
This will eventually replace the __hdr() calling methods and
eradicate {mime} usage from Smsg. For now, we can eliminate
PublicInbox::Smsg->new since most callers already rely on an
open `bless' to avoid the old {mime} arg.
Eric Wong [Mon, 1 Jun 2020 10:06:48 +0000 (10:06 +0000)]
import: modernize to use Perl 5.10 features
First, prefer the leaner "parent" module over the heavy "base"
module to establish ISA relationships, since "base" is only
needed for "fields".
The "//" and "//=" operators allow us simplify our code and fix
minor bugs where a value of "0" was disallowed. Yes, we'll
allow "0" as an email address, too, since some twisted BOFH
could theoretically use it as a local user name.
Going forward, we'll also be avoiding "use warnings" and
instead rely on `-w' in the shebang.
Eric Wong [Mon, 1 Jun 2020 10:06:46 +0000 (10:06 +0000)]
wwwatomstream: convert callers to use smsg_eml
We can simplify WwwAtomStream callbacks by performing ->smsg_eml
calls in the `feed_entry' sub itself. This simplifies callers,
by reducing the number of places which can load an Eml object
into memory.
Eric Wong [Mon, 1 Jun 2020 10:06:45 +0000 (10:06 +0000)]
inbox: introduce smsg_eml method
The goal of this is to eventually remove the $smsg->{mime} field
which is easy-to-misuse and cause memory explosions which
necessitated fixes like commit 7d02b9e64455831d
("view: stop storing all MIME objects on large threads").
Eric Wong [Thu, 28 May 2020 18:37:08 +0000 (18:37 +0000)]
treat $INBOX_DIR/description and gitweb.owner as UTF-8
gitweb does the same with $GIT_DIR/description and gitweb.owner.
Allowing UTF-8 description should not cause problems when used
in responses for to the NNTP "LIST NEWSGROUPS" request, either,
since RFC 3977 section 7.6.6 recommends the description be UTF-8
(but does not require it).
Eric Wong [Tue, 26 May 2020 09:05:24 +0000 (09:05 +0000)]
learn: support --all with `rm'
I found myself wanting to remove a message from all inboxes
while working on a test case in another branch. I figure this
could also be useful for globally removing messages which are in
the grey area or too big for spamc.
Eric Wong [Mon, 25 May 2020 03:30:20 +0000 (03:30 +0000)]
msgmap: tmp_clone: use in-memory journal
This prevents $TMPDIR from being littered with *-journal files
after running the test suite.
This shouldn't cause excessive memory use since $v2w->{mm_tmp}
doesn't see big transactions. There's no need to worry about
data loss, here,either, since this is just a temporary clone
we've even disabled fsync on.
Fixes: 78888d36fb80889f ("msgmap: use TRUNCATE for journal_mode, for now")
Eric Wong [Mon, 25 May 2020 08:21:37 +0000 (08:21 +0000)]
view: do not offer links to 0-byte multipart attachments
Offering links to download 0-byte files is useless. We could
waste memory by preserving $eml->{bdy} during iteration, but
offering attachments of type "multipart" is not very useful,
as users are usually interested in decoded attachments or
the entire raw message.
Fixes: e60231148eb604a3 ("descend into message/(rfc822|news|global) parts")
Eric Wong [Sun, 24 May 2020 03:06:37 +0000 (03:06 +0000)]
v2writable: only load Xapian when a shard is found
We don't need to load Xapian until we have a directory
which looks like a shard, otherwise we're wasting cycles
on memory when running short-lived processes.
Eric Wong [Thu, 21 May 2020 05:12:46 +0000 (05:12 +0000)]
spawn: fix compatibility with old Inline::C
Older versions of Inline (e.g. 0.53 in CentOS 7) did not accept
the `directory' parameter, so use conditional assignment to set
a default value on $ENV{PERL_INLINE_DIRECTORY}, instead.
Eric Wong [Tue, 19 May 2020 09:06:58 +0000 (09:06 +0000)]
scripts/import_*: remove PublicInbox::MIME usage
These aren't really supported and will probably be replaced with
better tools, but PublicInbox::Eml should be readily available
to anybody who already has our source tree.
Eric Wong [Wed, 20 May 2020 07:38:08 +0000 (07:38 +0000)]
convert: describe the release of fast-import pipes
Upon rereading the code, it wasn't immediately obvious to
me why we didn't check for errors with `close($w)' instead
of relying on `undef'. So add a comment for the benefit of
future readers.
Eric Wong [Sun, 17 May 2020 19:48:14 +0000 (19:48 +0000)]
favor readline() and print() as functions
In our inbox-writing code paths, ->getline as an OO method may
be confused with the various definitions of `getline' used by
the PSGI interface. It's also easier to do: "perldoc -f readline"
than to figure out which class "->getline" belongs to (IO::Handle)
and lookup documentation for that.
->print is less confusing than the "readline" vs "getline"
mismatch, but we can still make it clear we're using a real
file handle and not a mock interface.
Finally, functions are a bit faster than their OO counterparts.
Eric Wong [Sun, 17 May 2020 19:37:21 +0000 (19:37 +0000)]
index: add --batch-size=SIZE option
On powerful systems, having this option is preferable to
XAPIAN_FLUSH_THRESHOLD due to lock granularity and contention
with other processes (-learn, -mda, -watch).
Setting XAPIAN_FLUSH_THRESHOLD can cause -learn, -mda, and
-watch to get stuck until an epoch is completely processed.
Eric Wong [Sun, 17 May 2020 00:08:54 +0000 (00:08 +0000)]
index: v2: parallelize if --reindex or --jobs is specified
`--reindex' involves chomping down lots of mail, so it benefits
from parallelization just like the initial indexing. It's
also a bit surprising to specify `--jobs/-j' without parallel
processes, so ensure we turn on parallelization there, too.
We can simplify initialization here, as well, since neither
`eval' nor `V2Writable->new' should be in this code.
Eric Wong [Sat, 16 May 2020 22:53:53 +0000 (22:53 +0000)]
confine Email::MIME use even further
To avoid confusing future readers and users, recommend
PublicInbox::Eml in our Import POD and refer to PublicInbox::Eml
comments at the top of PublicInbox::MIME.
mime_load() confined to t/eml.t, since we won't be using
it anywhere else in our tests.