]> Sergey Matveev's repositories - public-inbox.git/log
public-inbox.git
3 years agot/inbox_idle: switch to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:16 +0000 (12:58 +0100)]
t/inbox_idle: switch to create_inbox

Surprising to me, this is actually significantly faster
even though we're starting off with an empty inbox due
to -no_fsync being the default.

3 years agot/edit: switch to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:15 +0000 (12:58 +0100)]
t/edit: switch to create_inbox

Hardly any time reduction, but code gets more compact.

3 years agot/cgi: create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:14 +0000 (12:58 +0100)]
t/cgi: create_inbox

This only saves a few ms but is nicer to look at.

3 years agot/solver_git: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:13 +0000 (12:58 +0100)]
t/solver_git: use create_inbox

This saves us a dozen or so milliseconds.

3 years agot/imapd: create_inbox (minor)
Eric Wong [Mon, 15 Mar 2021 11:58:12 +0000 (12:58 +0100)]
t/imapd: create_inbox (minor)

This saves over 100ms.

3 years agot/psgi_v2: create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:11 +0000 (12:58 +0100)]
t/psgi_v2: create_inbox

Some lines of code and a dozen milliseconds dealth with.

3 years agot/psgi_scan_all: create_inbox lots saved
Eric Wong [Mon, 15 Mar 2021 11:58:10 +0000 (12:58 +0100)]
t/psgi_scan_all: create_inbox lots saved

A fair mount of setup code goes away and saves us a few
hundred milliseconds.

3 years agot/psgi_multipart_not: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:09 +0000 (12:58 +0100)]
t/psgi_multipart_not: use create_inbox

Over 100ms saved.

3 years agot/psgi_bad_mids: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:08 +0000 (12:58 +0100)]
t/psgi_bad_mids: use create_inbox

Over 100ms saved and it's less code to boot.

3 years agot/feed: switch to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:07 +0000 (12:58 +0100)]
t/feed: switch to create_inbox

This only saves a few milliseconds, but is less code.

3 years agot/psgi_mount: switch to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:06 +0000 (12:58 +0100)]
t/psgi_mount: switch to create_inbox

A few dozen more milliseconds saved.

3 years agot/altid: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:05 +0000 (12:58 +0100)]
t/altid: use create_inbox

Another few dozen milliseconds saved.

3 years agot/convert-compact: create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:04 +0000 (12:58 +0100)]
t/convert-compact: create_inbox

Suprisingly, this saves over 100 milliseconds.

3 years agot/httpd: convert to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:03 +0000 (12:58 +0100)]
t/httpd: convert to create_inbox

No real difference, here, but having less code is nice.

3 years agot/psgi_attach: convert to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:02 +0000 (12:58 +0100)]
t/psgi_attach: convert to create_inbox

Another few dozen milliseconds saved.

3 years agot/plack: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:01 +0000 (12:58 +0100)]
t/plack: use create_inbox

Only a few dozen milliseconds saved, but better than nothing.

3 years agot/html_index: remove now-worthless test
Eric Wong [Mon, 15 Mar 2021 11:58:00 +0000 (12:58 +0100)]
t/html_index: remove now-worthless test

This was for quote-folding behavior we had long ago, but
it ended up just being yet another import test.

3 years agot/admin: switch to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:57:59 +0000 (12:57 +0100)]
t/admin: switch to create_inbox

Over 100ms saved.

v2: quiet down "git gc" warnings on empty inboxes

3 years agot/v2dupindex: create_inbox
Eric Wong [Mon, 15 Mar 2021 11:57:58 +0000 (12:57 +0100)]
t/v2dupindex: create_inbox

Another hundred milliseconds or so saved.

3 years agot/xcpdb-reshard: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:57:57 +0000 (12:57 +0100)]
t/xcpdb-reshard: use create_inbox

Over 100ms saved and fewer LoC to boot

3 years agot/www_altid: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:57:56 +0000 (12:57 +0100)]
t/www_altid: use create_inbox

This barely saves any time due to sqlite3(1) fsync,
but does save some lines of code.

3 years agot/imapd-tls: switch to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:57:55 +0000 (12:57 +0100)]
t/imapd-tls: switch to create_inbox

This saves another few dozen milliseconds and LoC.

3 years agotest_common: minor simplifications to setup_public_inboxes
Eric Wong [Mon, 15 Mar 2021 11:57:54 +0000 (12:57 +0100)]
test_common: minor simplifications to setup_public_inboxes

This will results in a small reduction in on-disk footprint
by removing Xapian docdata and reduction in code by removing
an unnecessary -index invocation.

3 years agot/lei_xsearch: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:57:53 +0000 (12:57 +0100)]
t/lei_xsearch: use create_inbox

Less code and noticeably faster when TMPDIR is on an SSD

3 years agotest_common: add create_inbox helper sub
Eric Wong [Mon, 15 Mar 2021 11:57:52 +0000 (12:57 +0100)]
test_common: add create_inbox helper sub

This saves over 100ms in t/lei-q-remote-import.t so far when
TMPDIR is on an SSD.  If we can memoize inbox creation to save a
few dozen milliseconds every test, this could add up to
noticeable savings across our entire test suite.

3 years agolei: reuse LeiStore object on config changes
Eric Wong [Mon, 15 Mar 2021 09:32:06 +0000 (07:32 -0200)]
lei: reuse LeiStore object on config changes

Unless leistore.dir changes, the same LeiStore object
is should remain reusable and accessible to any clients

This seems to fix problems with t/lei-q-remote-import.t
occasionally getting stuck

3 years agolei q: do not import unnecessarily from externals
Eric Wong [Sun, 14 Mar 2021 11:12:00 +0000 (13:12 +0200)]
lei q: do not import unnecessarily from externals

We only want to auto import messages that are exclusively in
remote externals.  Messages in local externals are not
auto-imported to save space and reduce wear on storage device.

3 years agosearchidx: fix -Lmedium for IDs and filenames
Eric Wong [Sat, 13 Mar 2021 15:40:27 +0000 (15:40 +0000)]
searchidx: fix -Lmedium for IDs and filenames

This fixes "m:", "l:", "f:", "t:", "c:", "dfn:", and "n:" search
prefixes under indexlevel=medium when mixed with indexlevel=full
inboxish.  We need positional data for Message-IDs, List-Id,
email addresses and filenames for exact matches, though we still
want to support wildcards.

Fortunately the storage cost is still small as these prefixes
tend to be small compared to message bodies.  These are NOT
boolean terms since wildcard support and partial matching is
desired.

3 years agolei q: mbox*: disable changing parallelism, add --rsyncable
Eric Wong [Fri, 12 Mar 2021 10:39:43 +0000 (10:39 +0000)]
lei q: mbox*: disable changing parallelism, add --rsyncable

Unfortunately, being mairix-compatible with --threads means we
can't change thread-count of gzip, bzip2, or xz when writing to
compressed mbox with a --threads= parameter.  It's probably not
worth changing, anyways, so another switch or additional value
for --jobs= won't be added.

While we're in the area, add --rsyncable support since
most installations of gzip support it nowadays.

Fixes: 5beb4a5f6585acd ("lei: replace --thread with --threads")
3 years agolei: rearrange OPT_DESC and drop some TBD switches
Eric Wong [Fri, 12 Mar 2021 10:39:42 +0000 (10:39 +0000)]
lei: rearrange OPT_DESC and drop some TBD switches

It'll be easier for us to have the option-spec in front of the
command instead of the other way around.  The option-spec in
front makes it easier to sort and keep track of potentially
confusing/ambiguous use of command-line switches between
different commands.

We'll also update some of the proposed switches while we're
at it.

3 years agolei: add help + completion for --no-external
Eric Wong [Fri, 12 Mar 2021 10:39:41 +0000 (10:39 +0000)]
lei: add help + completion for --no-external

I just needed it.

3 years agoexamples/varnish-4: http => httpd
Eric Wong [Sat, 13 Mar 2021 00:08:47 +0000 (16:08 -0800)]
examples/varnish-4: http => httpd

Our HTTP daemon is `public-inbox-httpd', not
`public-inbox-http'.

3 years agoTODO: update tests item
Eric Wong [Fri, 12 Mar 2021 10:44:03 +0000 (10:44 +0000)]
TODO: update tests item

Our code base is growing, and we could repeat less setup
work in our tests and improve development pace.

3 years agot/v2reindex: avoid reading ~/.public-inbox/config in test
Eric Wong [Fri, 12 Mar 2021 03:52:28 +0000 (03:52 +0000)]
t/v2reindex: avoid reading ~/.public-inbox/config in test

3 years agomsg_part_text: discover text in application/octet-stream
Eric Wong [Thu, 11 Mar 2021 01:45:39 +0000 (19:45 -0600)]
msg_part_text: discover text in application/octet-stream

Some poorly-configured MUAs will send application/octet-stream
even for text-only attachments.  We can't make expect all MUAs
are configured with proper MIME types, and there is plenty of
historical mail that falls into this unfortunate criteria.

v2: simplify the check and ensures returned text is Perl "utf8"

3 years agowww_attach: remove unnecessary parse_content_type import
Eric Wong [Thu, 11 Mar 2021 12:58:01 +0000 (06:58 -0600)]
www_attach: remove unnecessary parse_content_type import

It's not needed since we have the handy eml->ct method.

3 years agov2writable: fix undocumented --xapian-only
Eric Wong [Thu, 11 Mar 2021 10:45:38 +0000 (02:45 -0800)]
v2writable: fix undocumented --xapian-only

We can't pass $self and GLOBs across IPC channels transparently.
I only noticed this because I'm testing the application/octet-stream
fallback with https://public-inbox.org/meta/20210311014539.19756-1-e@80x24.org/

Fixes: bf8df8160076d7a1 ("searchidxshard: use PublicInbox::IPC to kill lots of code")
3 years agoconfig: use '-f' key to store config file pathname
Eric Wong [Thu, 11 Mar 2021 10:45:37 +0000 (02:45 -0800)]
config: use '-f' key to store config file pathname

This fixes ->urlmatch use from lei, which already sets '-f'.
I noticed this because imap.$URL.compress was ignored in
my lei config file.

3 years agoimapclient: disable workaround for Mail::IMAPClient 3.43+
Eric Wong [Thu, 11 Mar 2021 10:45:36 +0000 (02:45 -0800)]
imapclient: disable workaround for Mail::IMAPClient 3.43+

These fixes are in the recently-released Mail::IMAPClient 3.43:

https://metacpan.org/source/PLOBBES/Mail-IMAPClient-3.43/Changes

v2: disable 'once' warning

3 years agodoc: update 1.7 release notes, tuning, TODO
Eric Wong [Thu, 11 Mar 2021 10:45:35 +0000 (02:45 -0800)]
doc: update 1.7 release notes, tuning, TODO

Some stuff done, some stuff still needs doing.

3 years agolei_curl: note proposed master/client mode for curl
Eric Wong [Thu, 11 Mar 2021 10:45:34 +0000 (02:45 -0800)]
lei_curl: note proposed master/client mode for curl

Who knows, maybe stuff learned during lei development
can be used to implement it in curl:
https://curl.se/mail/archive-2021-02/0031.html

3 years agosearchidx: remove smsg_from_doc
Eric Wong [Thu, 11 Mar 2021 10:45:33 +0000 (02:45 -0800)]
searchidx: remove smsg_from_doc

We no longer read Xapian docdata and favor hitting over.sqlite3,
instead, as Xapian is less likely to be available than SQLite.

3 years agodoc: glossary: add information for dates and timestamps
Eric Wong [Thu, 11 Mar 2021 10:45:32 +0000 (02:45 -0800)]
doc: glossary: add information for dates and timestamps

These have been confusing to me in the past, too.

3 years agodoc: start glossary for overlapping concepts
Eric Wong [Wed, 10 Mar 2021 13:23:45 +0000 (13:23 +0000)]
doc: start glossary for overlapping concepts

This is intended to keep track of concepts with different terms
between NNTP, IMAP, config file, lei storage, and upcoming
JMAP support.

3 years agolei import: skip trashed Maildir messages
Eric Wong [Wed, 10 Mar 2021 13:23:44 +0000 (13:23 +0000)]
lei import: skip trashed Maildir messages

This matches IMAP behavior in NetReader in skipping \\Deleted
messages.  Since lei may be used for personal, non-public mail;
Draft messages are NOT skipped by "lei import".

3 years agolei import: simplify Maildir handling
Eric Wong [Wed, 10 Mar 2021 13:23:43 +0000 (13:23 +0000)]
lei import: simplify Maildir handling

Having a one-off Maildir functionality in LeiStore doesn't seem
worth the maintenance burden, especially given an upcoming
change to skip trashed messages.

I expect this will hurt performance slightly with extra IPC
overhead for the socket copy, but "lei import" may eventually
become rare or at least not hit messages redundantly.

3 years agowatch: IMAP: ignore \Deleted and \Draft messages
Eric Wong [Wed, 10 Mar 2021 13:23:42 +0000 (13:23 +0000)]
watch: IMAP: ignore \Deleted and \Draft messages

This matches existing Maildir behavior, as trash and draft
messages have little reason to be exposed publicly.

3 years agodoc: technical/data_structures: update for EOFpipe
Eric Wong [Wed, 10 Mar 2021 13:23:41 +0000 (13:23 +0000)]
doc: technical/data_structures: update for EOFpipe

ParentPipe no longer exists and was replaced by the more
flexible EOFpipe.

3 years agolei q: remove angle brackets around Message-IDs
Eric Wong [Mon, 8 Mar 2021 08:08:16 +0000 (08:08 +0000)]
lei q: remove angle brackets around Message-IDs

They're unnecessary visual noise, and angle brackets don't
always work as intended when going through Xapian's query
parser.

Since we already use "m:" and "refs:" instead of the actual
header names, it should be obvious we're at liberty to
abbreviate such things

Link: https://public-inbox.org/meta/20210304184348.GA19350@dcvr/
3 years agolei q: one -t shouldn't set `flagged' on external mail
Eric Wong [Fri, 5 Mar 2021 04:03:05 +0000 (04:03 +0000)]
lei q: one -t shouldn't set `flagged' on external mail

We only want to set `flagged' if a user requests it via
a two '-t' switches.

Fixes: 232f8e376fe2856c ("lei q: -tt marks direct hits as flagged")
3 years agolei q: fix --import-before default and FIFO output
Eric Wong [Fri, 5 Mar 2021 01:38:29 +0000 (18:38 -0700)]
lei q: fix --import-before default and FIFO output

commit 6c551bffd75afb41d9b5e4774068abe7e06ed0e7
("lei q: --import-augment for mbox and mbox.gz") added a check to
in _pre_augment_mbox for the option being a ref() to distinguish
between default values and user-supplied values (which are
non-ref SCALARs from Getopt::Long).

However, LeiQuery failed to use a SCALAR ref as the default
value, making the check in _pre_augment_mbox useless.  We
now update LeiQuery to use \1 instead of 1 as the default
value so "lei q -f mboxrd ..." to stdout works once again.

Unfortunately, testing with redirects pointed to regular
files didn't trigger the code paths being updated.  Testing
with a FIFO revealed further bugs in the FIFO handling code
which are also fixed in this commit.

We'll also update the $lei->out error message to be
less-specific about "stdout" and use the term "output", instead,
since LeiToMail replaces stdout for all mbox outputs.

3 years agosearch: use "z:" instead of "bytes:" prefix
Eric Wong [Fri, 5 Mar 2021 03:10:58 +0000 (19:10 -0800)]
search: use "z:" instead of "bytes:" prefix

So far, searching by size has never been publicly documented,
and IMHO, of questionable utility.  In any case, "z:" is what
mairix(1) uses, so it may be familiar to existing mairix users
(I've never used this prefix myself).

So far, this prefix is only used internally in tests and in
auto-translated queries from IMAP; thus this incompatible change
is unlikely to affect anyone.

3 years agolei q: s/import-augment/import-before/g
Eric Wong [Thu, 4 Mar 2021 09:03:16 +0000 (17:03 +0800)]
lei q: s/import-augment/import-before/g

Since this importing of keywords is active even when --augment
isn't specified, calling it --import-before seems more
appropriate.

In the future, this will likely default to adding unseen emails
to lei/store, not just updating keywords.

Link: https://public-inbox.org/meta/20210303222930.GA18597@dcvr/T/
3 years agot/lei_to_mail: no need to cat in FIFO test
Eric Wong [Thu, 4 Mar 2021 09:03:15 +0000 (17:03 +0800)]
t/lei_to_mail: no need to cat in FIFO test

We're not forking, here, so there's no need to rely on FD_CLOEXEC
to resolve deadlock issues.

3 years agolei q: --import-augment for mbox and mbox.gz
Eric Wong [Thu, 4 Mar 2021 09:03:14 +0000 (17:03 +0800)]
lei q: --import-augment for mbox and mbox.gz

The trickiest output formats we support due to the possibility
of filesystem FIFOS and pipes for <gzip|xz|bzip2>.

This completes another phase of keyword sync support.

3 years agolei_xsearch: cleanup {pkt_op_p} on exceptions
Eric Wong [Thu, 4 Mar 2021 09:03:13 +0000 (17:03 +0800)]
lei_xsearch: cleanup {pkt_op_p} on exceptions

We must ensure pkt_op_p doesn't live beyond the scope of
->do_query in the top-level lei-daemon, otherwise it can leave a
stray socket hanging around in case of exceptions.

3 years agolei: dclose: do not EPOLL_CTL_DEL w/o event_init
Eric Wong [Thu, 4 Mar 2021 09:03:12 +0000 (17:03 +0800)]
lei: dclose: do not EPOLL_CTL_DEL w/o event_init

It's possible we'll hit a die() statement which triggers
lei->dclose, but aren't in the event loop, yet.

3 years agolei q: support --import-augment for IMAP
Eric Wong [Thu, 4 Mar 2021 09:03:11 +0000 (17:03 +0800)]
lei q: support --import-augment for IMAP

IMAP is similar to Maildir and we can now preserve keyword
updates done on IMAP folders.

3 years agolei q: import flags when clobbering/augmenting Maildirs
Eric Wong [Wed, 3 Mar 2021 13:48:57 +0000 (13:48 +0000)]
lei q: import flags when clobbering/augmenting Maildirs

This will eventually be supported for other mail stores,
but Maildir is the easiest to test and support, here.

This lets us avoid a situation where flag changes get
lost between search results.

3 years agolei: use maildir_each_eml in more places
Eric Wong [Wed, 3 Mar 2021 13:48:56 +0000 (13:48 +0000)]
lei: use maildir_each_eml in more places

This saves us some code and redundant callsites for
eml_from_path.  We'll change maildir_each_eml to include the
filename to facilitate an upcoming change to "lei q" without
--augment

3 years agolei_xsearch: add_eml for remote mboxrd, not set_eml
Eric Wong [Wed, 3 Mar 2021 13:48:55 +0000 (13:48 +0000)]
lei_xsearch: add_eml for remote mboxrd, not set_eml

set_eml will clobber any existing keywords.  Since remote
mboxrds cannot (and should not) be sending keywords to us,
we shouldn't let remote external requests clobber already-set
keywords if they exist.

3 years agoeml: each_part: document IMAP user of the $all parameter
Eric Wong [Wed, 3 Mar 2021 13:48:54 +0000 (13:48 +0000)]
eml: each_part: document IMAP user of the $all parameter

I already forgot what it does and thought I was misusing
the API :x

3 years agods: import croak properly
Eric Wong [Thu, 4 Mar 2021 01:20:39 +0000 (07:20 +0600)]
ds: import croak properly

We use croak in several places but weren't importing it :x

3 years agoinbox: ->mailboxid accessor
Eric Wong [Tue, 2 Mar 2021 09:13:07 +0000 (09:13 +0000)]
inbox: ->mailboxid accessor

This will be necessary for "mailboxIds" as described in RFCs 8620 and
8621 (for JMAP).  We may implement "MAILBOXID" in RFC 8474 for IMAP,
as well.

3 years agolei p2q: fix /dev/null filenames, fix phrase quoting rules
Eric Wong [Mon, 1 Mar 2021 05:47:36 +0000 (11:47 +0600)]
lei p2q: fix /dev/null filenames, fix phrase quoting rules

/dev/null mis-handling was reported by Kyle Meyer.

Phrases quoting rules are also refined to avoid leaving spaces
unquoted when "phrase generator" characters exist.  Also,
context-free hunk headers no longer clobber the in_diff
state of the parser, since git can still generate those.

Link: https://public-inbox.org/meta/87k0qrrhve.fsf@kyleam.com/
3 years agolei q: improve early aborts w/ remote externals
Eric Wong [Sun, 28 Feb 2021 12:25:28 +0000 (18:25 +0600)]
lei q: improve early aborts w/ remote externals

We must issue LeiStore->done if a client disconnects
while we're streaming from a remote external.  This
can happen via SIGPIPE, or if a client process is
interrupted by any other means.

3 years agolei q: fix "-" shortcut for --stdin
Eric Wong [Sun, 28 Feb 2021 12:25:27 +0000 (18:25 +0600)]
lei q: fix "-" shortcut for --stdin

Due to the way our option parser handles this special case, it
must be the first option spec.  This helps us document things
better, even, since many command accept either a pathname or
--stdin|-.

3 years agolei p2q: patch-to-query generator for "lei q --stdin"
Eric Wong [Sun, 28 Feb 2021 12:25:26 +0000 (18:25 +0600)]
lei p2q: patch-to-query generator for "lei q --stdin"

Instead of teaching the to-be-implemented "lei show" to search
threads/messages based commits, this orthogonal sub-command is
designed to generate queries for use with "lei q --stdin".

URI-escaped query parameters may be generated with --uri for
HTTP(S) public-inbox instances, but otherwise the output is
designed for "lei q --stdin".

To find threads for a given git commit from a git worktree:

lei p2q $COMMIT_OID | lei q --stdin -t ...

It can also read via --stdin|-

curl $INBOX_URL/$MSGID/raw | lei p2q - | lei q --stdin -t

Or from the filesystem:

lei p2q $(git format-patch -1) | lei q --stdin -t

This defaults to only generating "dfpost:"-prefixed terms since
I've found those most useful for finding messages relating to a
commit.  This is subject to change.

--want=s@ is a comma-separated or multi-value list of prefixes
that defaults to "dfpost7".  Not all are implemented, yet, but
s, dfn, dfpre, and dfpost all seem to mostly work.  Phrase
handling may need to be tweaked to work with Xapian.

OR, NEAR, ADJ, AND, NOT may be used with --want
(e.g. --want=dfpost,OR,dfn)

Prefixing the field prefix with '+' or '-' (e.g. --want=+dfpost)
generates "+dfpost:$EXTRACTED_OID" for Xapian.   For non-boolean
search prefixes, wildcard (*) may also be supplied: (--want=dfn*)

For boolean search prefixes, suffixing the field prefix with a
digit (e.g. --want=dfpost7) provides a minimum length, allowing
truncated variations to be searched.  This is helpful for
finding older messages as git chooses longer dfpost|dfpre
abbreviations as repos get larger.

Automatic date range generation is not implemented, yet.

3 years agoexamples/cgit-commit-filter: improve quoted text handling
Eric Wong [Sat, 13 Feb 2021 02:15:03 +0000 (02:15 +0000)]
examples/cgit-commit-filter: improve quoted text handling

With an example such as:

something before "quoted phrase" something after

The Xapian will now see:

[ "something before", "quoted phrase", "something after" ]

whereas before it would see:

[ "something before", "quoted", "phrase", "something after" ]

which should improve search results accuracy when looking
up commits by commit title (subject).

3 years agodoc: lei-overview: add performance and bash completion sections
Kyle Meyer [Sat, 27 Feb 2021 18:03:28 +0000 (13:03 -0500)]
doc: lei-overview: add performance and bash completion sections

Take care of a couple of the items mentioned at
<https://public-inbox.org/meta/20210218202818.GA19443@dcvr>.

3 years agodoc: lei-import: drop markup of "stdin"
Kyle Meyer [Sat, 27 Feb 2021 18:03:27 +0000 (13:03 -0500)]
doc: lei-import: drop markup of "stdin"

stdin isn't placed in C<> elsewhere.

3 years agodoc: lei: update manpages
Kyle Meyer [Sat, 27 Feb 2021 18:03:26 +0000 (13:03 -0500)]
doc: lei: update manpages

Catch up with recent developments.

3 years agolei_xsearch: more detail about ->xdb call chain
Eric Wong [Fri, 26 Feb 2021 09:41:41 +0000 (22:41 -1100)]
lei_xsearch: more detail about ->xdb call chain

I was just wondering this myself :x

3 years agot/lei_store: rename $lst to $sto
Eric Wong [Fri, 26 Feb 2021 09:41:40 +0000 (22:41 -1100)]
t/lei_store: rename $lst to $sto

`$sto' is prevalent throughout the rest of the lei code,
and `$lst' seems like an abbreviation for "list".

I don't like the noise from commits like this, but I hope the
long-term payoff being less confusing to new developers is worth
it...

3 years agolei import|convert: support mbox locking on reads
Eric Wong [Fri, 26 Feb 2021 09:41:39 +0000 (22:41 -1100)]
lei import|convert: support mbox locking on reads

In case somebody is writing non-atomically, ensure we
take read locks when opening mbox files for reading.

v2: squash: load MboxLock even for .eml files

3 years agolei q: support mbox locking by default
Eric Wong [Fri, 26 Feb 2021 09:41:38 +0000 (22:41 -1100)]
lei q: support mbox locking by default

While this diverges from from mairix(1) behavior, it's the safer
option.  We'll follow Debian policy by supporting fcntl and
dotlocks by default (in that order).  Users who do not want
locking can use "--lock=none"

This will be used in a read-only capacity for watching
mailboxes for keyword updates via inotify or EVFILT_VNODE.

3 years agolei: style fix for $oldset declaration
Eric Wong [Fri, 26 Feb 2021 09:41:37 +0000 (22:41 -1100)]
lei: style fix for $oldset declaration

We want /^sub oldset/ to match to keep editors and
things like ctags happy.

3 years agolei q: -tt marks direct hits as "flagged"
Eric Wong [Thu, 25 Feb 2021 10:11:06 +0000 (10:11 +0000)]
lei q: -tt marks direct hits as "flagged"

This can be used to quickly distinguish messages which were
direct hits when doing thread expansion vs messages that
were merely part of the same thread.

This is NOT mairix-derived behavior, but I occasionally found
it useful when looking at results in an MUA to know whether
a message was a direct hit or not.

This makes "-t" consistent with non-"-t" cases as far as keyword
reading goes.

3 years agotest_common: io_modes: always support read/write
Eric Wong [Thu, 25 Feb 2021 10:11:05 +0000 (10:11 +0000)]
test_common: io_modes: always support read/write

This avoids warnings when redirecting STDIN to a scalarref
via run_script().

3 years agolei import: use --in-format/-F for consistency
Eric Wong [Thu, 25 Feb 2021 10:11:04 +0000 (10:11 +0000)]
lei import: use --in-format/-F for consistency

Since we recommend $IN_FORMAT:$LOCATION, this is hopefully not
intrusive (not that this is released software, yet).  This is
to be consistent with "lei convert" usage.

We'll keep "-f" only for output formats, since that is used
for "lei q" and "lei convert" for outputs

3 years agolei convert: support IMAP output and "-F eml" inputs
Eric Wong [Thu, 25 Feb 2021 10:11:03 +0000 (10:11 +0000)]
lei convert: support IMAP output and "-F eml" inputs

eml ("message/rfc822" MIME type) is supported by "lei import",
so it probably makes sense to support via convert, at least
for tests.  And IMAP support is supported in "lei q -o $MFOLDER",
so this only required renaming {nrd} => {net} and initializing
outputs before augment preparation (creating the IMAP folder)

3 years agolei q: auto-memoize remote messages into lei/store
Eric Wong [Wed, 24 Feb 2021 23:37:18 +0000 (05:37 +0600)]
lei q: auto-memoize remote messages into lei/store

This lets users avoid network traffic on subsequent searches at
the expense of local disk space.  --no-import-remote may be
specified to reverse this trade-off for users with little
storage.

3 years agolei_external: don't treat IPv6 URLs as globs
Eric Wong [Wed, 24 Feb 2021 23:37:17 +0000 (05:37 +0600)]
lei_external: don't treat IPv6 URLs as globs

IPv6 addresses are hexadecimals and colons inside brackets, so
add some DWIM-ery to ensure we don't attempt to treat addresses
like "http://[dead:beef]/foo/" as a glob.

3 years agonet_reader: trim exports and remove unused uri_new
Eric Wong [Wed, 24 Feb 2021 11:31:54 +0000 (17:31 +0600)]
net_reader: trim exports and remove unused uri_new

More network things for -watch are isolated in NetReader, now,
so fewer exports are necessary.

3 years agowatch: switch IMAP and NNTP fetch loops to NetReader
Eric Wong [Wed, 24 Feb 2021 11:31:53 +0000 (17:31 +0600)]
watch: switch IMAP and NNTP fetch loops to NetReader

NetReader::<imap|nntp>_each were based on the -watch
code they now replace.

v2: do not warn on EINTR if user quit to fix occasional
    test failure in t/imapd.t

3 years agolei <import|convert>: support NNTP sources
Eric Wong [Wed, 24 Feb 2021 11:31:52 +0000 (17:31 +0600)]
lei <import|convert>: support NNTP sources

We can read NNTP in -watch and Net::NNTP is shipped with Perl5,
so lei import and convert have no excuse not to support NNTP
as a client.

Authentication is not tested, yet; but should be close to what
IMAP is like...

3 years agoadd PublicInbox::URInntps package
Eric Wong [Wed, 24 Feb 2021 11:31:51 +0000 (17:31 +0600)]
add PublicInbox::URInntps package

We prefer the IANA-registered form of URIs to avoid confusing
users, but the URI package has yet to support it.

cf. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983419

3 years agolei: avoid needless env passing to subcommands
Eric Wong [Mon, 22 Feb 2021 21:38:22 +0000 (03:38 +0600)]
lei: avoid needless env passing to subcommands

We already localize %ENV before calling dispatch(), so
it's needless overhead in spawn() to be checking env for
undef values in those cases.

3 years agotreewide: avoid "delete local" construct on hashes
Eric Wong [Mon, 22 Feb 2021 21:38:21 +0000 (03:38 +0600)]
treewide: avoid "delete local" construct on hashes

Apparently this feature is only in Perl 5.12+, and we're
still on Perl 5.10.

3 years agowww: use PublicInbox::WwwStream
Uwe Kleine-König [Wed, 24 Feb 2021 08:54:56 +0000 (09:54 +0100)]
www: use PublicInbox::WwwStream

This prevents the following problem logged to the webserver's error log:

E: Undefined subroutine &PublicInbox::WwwStream::code_footer called at /usr/share/perl5/PublicInbox/WwwListing.pm line 102.
 in PublicInbox::ConfigIter=ARRAY(0x557aea68b1a8)::each_section at /usr/share/perl5/PublicInbox/ConfigIter.pm line 37.

Fixes: 7a3946ef122e ("www: support listing of inboxes")
3 years agolei_to_mail: remove unused OnDestroy import
Eric Wong [Tue, 23 Feb 2021 10:01:16 +0000 (04:01 -0600)]
lei_to_mail: remove unused OnDestroy import

3 years agolei q: reduce default lei2mail workers
Eric Wong [Tue, 23 Feb 2021 10:01:15 +0000 (04:01 -0600)]
lei q: reduce default lei2mail workers

While disk I/O is typically buffered for good scheduling,
git blob decoding uses a non-trivial amount of CPU time
and it helps to leave some CPU available for it.

3 years agolei: support "-C" to chdir in all sub commands
Eric Wong [Tue, 23 Feb 2021 10:01:14 +0000 (04:01 -0600)]
lei: support "-C" to chdir in all sub commands

We'll also support "-C" at the end of most commands to give
users a little more flexibility when building command-lines.
This conflicts with "lei daemon-kill -CHLD", so that's
special-cased since "-C" makes no sense with daemon-kill,
anyways.

Unlike "git show", the to-be-implemented "lei show" will diverge
and enable "--find-copies[=<n>]" by default, so "-C[<n>]" won't
be necessary.

3 years agodoc: lei: favor "-o format:$PATHNAME" over "-f"
Kyle Meyer [Tue, 23 Feb 2021 03:45:52 +0000 (22:45 -0500)]
doc: lei: favor "-o format:$PATHNAME" over "-f"

The --format argument is redundant and may be dropped entirely.
Update the lei manpages to prefer the format prefix.

cf. https://public-inbox.org/meta/20210217044032.GA17934@dcvr/

3 years agolei_auth: trim and remove leftover worker code
Eric Wong [Mon, 22 Feb 2021 11:22:59 +0000 (08:22 -0300)]
lei_auth: trim and remove leftover worker code

LeiAuth is no longer a separate worker process.  Instead, it's
used directly by LeiToMail and LeiImport for sharing auth info
from the first worker to the rest of the workers, using
lei-daemon as a message router.  So drop the old code to reduce
human cognitive load and interpreter memory overhead.

3 years agolei convert: inline convert_start
Eric Wong [Mon, 22 Feb 2021 11:22:58 +0000 (08:22 -0300)]
lei convert: inline convert_start

Since we stopped using LeiAuth as a WQ worker, keeping this
around as a single-use sub makes no sense and wastes several
KB of memory.

3 years agonet_reader: mic_get: reuse connections if cache enabled
Eric Wong [Mon, 22 Feb 2021 11:22:57 +0000 (08:22 -0300)]
net_reader: mic_get: reuse connections if cache enabled

We only enable {mic_cached} in WQ workers, and those
aren't expected to fork again going forward.  So cache
here avoid a penalty for the non-augmenting (imap_delete_all)
call with "lei q"

3 years agolei q: reduce wasted IMAP connection for auth
Eric Wong [Mon, 22 Feb 2021 11:22:56 +0000 (08:22 -0300)]
lei q: reduce wasted IMAP connection for auth

We can rework the first lei2mail worker to authenticate, and
then share auth info with the rest of the lei2mail workers.  As
with "lei import", this uses PktOp and lei-daemon to share
updated credentials between the first an subsequent l2m workers.

3 years agolei_auth: migrate common auth code from lei_import
Eric Wong [Mon, 22 Feb 2021 11:22:55 +0000 (08:22 -0300)]
lei_auth: migrate common auth code from lei_import

lei_to_mail will be able to use this, too.

3 years agolei import: no separate auth worker
Eric Wong [Mon, 22 Feb 2021 11:22:54 +0000 (08:22 -0300)]
lei import: no separate auth worker

We'll start sharing auth info from the first worker to the
rest of the workers via wq_broadcast.

This lays the groundwork for getting rid of LeiAuth workers for
authentication work and reducing network round trips required
for IMAP.