]> Sergey Matveev's repositories - public-inbox.git/log
public-inbox.git
3 years agolei: simplify workers_start and callers
Eric Wong [Mon, 22 Mar 2021 07:53:58 +0000 (07:53 +0000)]
lei: simplify workers_start and callers

Since workers_start is in the common PublicInbox::LEI
package, we can just use \&METHOD_NAME instead of relying
on UNIVERSAL->can to avoid a method dispatch.

Most of our worker code can just use lei->dclose, so default
to doing that unless it's been overridden.

3 years agolei: share input code between convert and import
Eric Wong [Mon, 22 Mar 2021 07:53:57 +0000 (07:53 +0000)]
lei: share input code between convert and import

These commands accept mail the same way, and this forces
us to maintain consistent input format support between
commands.

We'll be using this for "lei mark", too.

3 years agonet_reader: escape nasty chars from Net::NNTP->message
Eric Wong [Mon, 22 Mar 2021 07:53:56 +0000 (07:53 +0000)]
net_reader: escape nasty chars from Net::NNTP->message

Net::Cmd::message (used by Net::NNTP) does no escaping at all,
so "\r" was causing confusing/nonsensical error messages when
I tried to import from the wrong group.

3 years agolei: support -c <name>=<value> to overrides
Eric Wong [Mon, 22 Mar 2021 07:53:55 +0000 (07:53 +0000)]
lei: support -c <name>=<value> to overrides

It's a bit nasty, but seems to mostly work for debugging
IMAP and NNTP commands.

3 years agolei: simplify lazy-loading
Eric Wong [Sun, 21 Mar 2021 11:24:05 +0000 (13:24 +0200)]
lei: simplify lazy-loading

This makes it slightly easier to implement future commands,
since there'll be a couple more relatively self-contained
ones.

3 years agolei: fix some warnings in tests
Eric Wong [Sun, 21 Mar 2021 09:50:47 +0000 (15:50 +0600)]
lei: fix some warnings in tests

And then test the contents of $lei_err to ensure it doesn't
happen again.

We'll also make MboxLock emit nicer warnings without the line
number, since the line number is irrelevant to the user fixing
an mbox lock contention problem.

Finally, we'll also allow showing loud warnings via
TEST_LEI_ERR_LOUD=1

3 years agolei q: fix warning on remote imports
Eric Wong [Sun, 21 Mar 2021 09:50:46 +0000 (15:50 +0600)]
lei q: fix warning on remote imports

This will let us tie keywords from remote externals
to those which only exist in local externals.

3 years agolei import: vivify external-only messages
Eric Wong [Sun, 21 Mar 2021 09:50:45 +0000 (15:50 +0600)]
lei import: vivify external-only messages

Keyword storage for external-only messages was preventing
messages from being explicitly imported.  Teach lei_store
to vivify keyword-only entries into fully-indexed messages
on import.

3 years agosearchview: collapse Message-ID links in summary
Eric Wong [Wed, 17 Mar 2021 18:14:08 +0000 (20:14 +0200)]
searchview: collapse Message-ID links in summary

There's no point in showing duplicate links to the same
Message-ID in summary view.  The per-message page will
note the duplication (if any) separately.

Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210317132723.xx4klonordhsb6ve@chatter.i7.local/
3 years agolei q: trim JSON output
Eric Wong [Sat, 20 Mar 2021 12:40:31 +0000 (18:40 +0600)]
lei q: trim JSON output

Stop showing `docid' since it's not useful with shards.

`bytes' and `lines' are probably noise, but maybe could be
visible in some "fuller" view.

v2: t/lei_xsearch: fix warnings from {docid} removal

3 years agolei: tie ALE lifetime to config file
Eric Wong [Sat, 20 Mar 2021 10:04:07 +0000 (19:04 +0900)]
lei: tie ALE lifetime to config file

This should make a future change to "lei import" work more
nicely, since we'll be needing ALE to vivify external-only
messages upon explicit "lei import".

3 years agolei_to_mail: match mutt order of status headers
Eric Wong [Sat, 20 Mar 2021 10:04:06 +0000 (19:04 +0900)]
lei_to_mail: match mutt order of status headers

These changes may make it easier to do byte-for-byte comparisons
with mail copied out of mutt, a popular MUA for our target
audience.

mutt currently outputs the 'R' (seen) flag before the 'O'
character in the Status: header.  We'll assume that stays
the case (it has been for a while).

Status now comes before X-Status, also matching mutt behavior.

3 years agolei q: put keywords on one line in --pretty output
Eric Wong [Sat, 20 Mar 2021 10:04:05 +0000 (19:04 +0900)]
lei q: put keywords on one line in --pretty output

Don't waste precious terminal space when there are only a small
number of possible keywords supported/reserved for JMAP.  In the
future, we may implement more sophisticated wrapping for labels,
but it we'll cross tha bridge when we come to it.

3 years agolei q: support vmd for external-only messages
Eric Wong [Sat, 20 Mar 2021 10:04:04 +0000 (19:04 +0900)]
lei q: support vmd for external-only messages

"lei q" now preserves changes per-message keywords across
invocations when it's --output (Maildir or mbox) is reused
(with or without --augment).

In the future, these changes will be monitored via inotify,
EVFILT_VNODE or IMAP IDLE, too.

Unfortunately, this currently prevents "lei import" from ever
importing a message that's in an external.  That will be fixed
in a future change.

3 years agolei: All Local Externals: bare git dir for alternates
Eric Wong [Sat, 20 Mar 2021 10:04:03 +0000 (19:04 +0900)]
lei: All Local Externals: bare git dir for alternates

This will be used for keyword (and label) storage for externals.
We'll be using this to ensure we don't redundantly auto-import
messages into lei/store if they're already in a local external
(they can still be imported explicitly via "lei import").

3 years agolei_store: initialize IPC lock properly
Eric Wong [Sat, 20 Mar 2021 02:38:00 +0000 (07:38 +0500)]
lei_store: initialize IPC lock properly

This was causing errors in a mass keyword import patch
I'm working on.

3 years agomaildir: avoid redundant slashes
Eric Wong [Sat, 20 Mar 2021 00:56:33 +0000 (20:56 -0400)]
maildir: avoid redundant slashes

Redundant slashes look ugly in strace(1) output.

3 years agolei q: -I/--include overrides --no-(external|local|remote)
Eric Wong [Fri, 19 Mar 2021 22:38:49 +0000 (20:38 -0200)]
lei q: -I/--include overrides --no-(external|local|remote)

Assume that anybody using -I/--include for external locations
will want to override --no-$FOO if they're explicitly including
a location.

With some effort, we could make it order-dependent (e.g.
"-I $LOCATION --no-$FOO" and "--no-$FOO -I $LOCATION"
behave differently).  However that's not straightforward
when using Getopt::Long to parse command-line options into
a hashref.

I'm also not sure if order-dependent switches are a desirable
UI/UX quality.

3 years agoconfig: ignore extindex entries with newlines in paths
Eric Wong [Fri, 19 Mar 2021 12:35:57 +0000 (10:35 -0200)]
config: ignore extindex entries with newlines in paths

git 2.11 and earlier could not handle git directories with
newlines in them, nor does libgit2 support them.

Followup-to: d87dd0e679587043 ("config: reject `\n' in `inboxdir'")
3 years agolei: disallow "\n" in local externals paths
Eric Wong [Fri, 19 Mar 2021 12:35:56 +0000 (10:35 -0200)]
lei: disallow "\n" in local externals paths

git 2.11 and earlier could not handle git directories with
newlines in them, nor does libgit2 support them.

Followup-to: d87dd0e679587043 ("config: reject `\n' in `inboxdir'")
3 years agoxt/create-many-inboxes: adjust for detect_nproc, no fsync
Eric Wong [Fri, 19 Mar 2021 04:22:58 +0000 (07:22 +0300)]
xt/create-many-inboxes: adjust for detect_nproc, no fsync

detect_nproc is in the IPC module, now; and we can safely
disable fsync when creating test data.
And "modernize" up to 5.10.1 while we're at it.

The use fsync was causing this to run for hours instead
of minutes since I forgot to use eatmydata.

3 years agodoc: glossary: add missing over/back POD directives
Eric Wong [Fri, 19 Mar 2021 04:21:03 +0000 (07:21 +0300)]
doc: glossary: add missing over/back POD directives

Oops :x

3 years agolei_overview: unnecessary g2m capture
Eric Wong [Fri, 19 Mar 2021 04:20:32 +0000 (07:20 +0300)]
lei_overview: unnecessary g2m capture

Nothing like the -Wunused C compiler flag in perl, AFAIK...

3 years agoexamples: cgit-commit-filter: drop <tt> HTML tag, use title=
Eric Wong [Fri, 19 Mar 2021 04:18:54 +0000 (04:18 +0000)]
examples: cgit-commit-filter: drop <tt> HTML tag, use title=

<tt> doesn't seem necessary and it's deprecated in HTML, nowadays.
In any case, dillo's CSS support seems to show it as fixed-width
even without <tt>.  Use the title= attribute to highlight that
it goes to the mail thread, too.

In the future, we'll probably link to something like "lei p2q"
(patch-to-query) to include OIDs in the search.

3 years agocgit: fix fallout from lazy coderepo loading
Eric Wong [Thu, 18 Mar 2021 23:27:51 +0000 (23:27 +0000)]
cgit: fix fallout from lazy coderepo loading

We can't completely instantiate our cgit wrapper without knowing
knowing cgit locations for serving static content.

Fixes: a5968dc059f655a ("config: lazy-load coderepos, support extindex")
3 years agolei_store: keywords => vmd (volatile metadata), prepare for labels
Eric Wong [Wed, 17 Mar 2021 09:39:22 +0000 (15:39 +0600)]
lei_store: keywords => vmd (volatile metadata), prepare for labels

Since keywords and mailboxes (AKA labels) are separate things in
JMAP; and only keywords can map reliably to Maildir and mbox;
we'll keep them separate in our internal data representations,
too.

I initially wanted to call this just "meta" for "metadata", but
that might be confused with our mailing list name.  "metadata"
is already used in Xapian's own API, to add another layer of
confusion.

"tags" was also considered, but probably confusing to notmuch
users since our "labels" are analogous to "tags" in notmuch,
and notmuch doesn't seem to cover "keywords" separately...

So "vmd" it is, since we haven't used this particular
three-letter-abbreviation anywhere before; and "volatile" seems
like a good description of this metadata since everything else
up to this point has been mostly WORM (write-once, read-many).

3 years agotests: show lsof output on deleted-file-check failures
Eric Wong [Wed, 17 Mar 2021 07:02:18 +0000 (23:02 -0800)]
tests: show lsof output on deleted-file-check failures

This may help track down some occasional test failures I'm
seeing.

3 years agowww: improve visibility of coderepos
Eric Wong [Wed, 17 Mar 2021 07:02:17 +0000 (23:02 -0800)]
www: improve visibility of coderepos

By adding "+code" next to "mirror" at the top next to the search
box.  Instead of showing "/path/to/$FOO", showing "$FOO.git"
makes it more obvious we're talking about a git repo, here,
instead of some random directory.

3 years agoconfig: lazy-load coderepos, support extindex
Eric Wong [Wed, 17 Mar 2021 07:02:16 +0000 (23:02 -0800)]
config: lazy-load coderepos, support extindex

Extsearch objects are duck-types of Inbox objects, and
are capable of supporting code repos all the same.

3 years agoextindex: add some validation and config knobs for WWW
Eric Wong [Wed, 17 Mar 2021 07:02:15 +0000 (23:02 -0800)]
extindex: add some validation and config knobs for WWW

We'll try to share a bit more configuration with
extindex entries for WWW PSGI usage.

3 years agowww_stream: add trailing slash for help and color links
Eric Wong [Wed, 17 Mar 2021 07:02:14 +0000 (23:02 -0800)]
www_stream: add trailing slash for help and color links

This saves clients a redirect

3 years agoeml: decode Bcc, and Resent-* address variants
Eric Wong [Tue, 16 Mar 2021 10:28:50 +0000 (16:28 +0600)]
eml: decode Bcc, and Resent-* address variants

This is closer to matching RFC 8621 section 4.1.2.3,
though we don't support the "Any header field not defined in
RFC5322 or RFC2369" rule, since that could get tricky...

3 years agolock: remove new_tmp method
Eric Wong [Tue, 16 Mar 2021 09:14:28 +0000 (03:14 -0600)]
lock: remove new_tmp method

Unused as of commit dda8237aeb5722b3a48c31896d9b7398e50823f1
("lei_to_mail: prepare for worker offload") when we switched
to using the LeiOverview output lock.

3 years agogit: drop async_prefetch method
Eric Wong [Tue, 16 Mar 2021 08:48:07 +0000 (03:48 -0500)]
git: drop async_prefetch method

That logic is inlined directly into git_async_prefetch
in GitAsyncCat and I don't see it being useful outside
of a DS event loop.

3 years agonntp: remove unused header_append method
Eric Wong [Tue, 16 Mar 2021 08:48:06 +0000 (03:48 -0500)]
nntp: remove unused header_append method

It was unused since 1bf653ad139bf7bb3d853ab0b5eae3eaa1b13a95
("nntp+www: drop List-* and Archived-At headers")

3 years agolei_store: remove maildir_keywords
Eric Wong [Tue, 16 Mar 2021 01:43:45 +0000 (07:43 +0600)]
lei_store: remove maildir_keywords

It's redundant and the same functionality is in MdirReader.

3 years agombox: move mbox_keywords to MboxReader
Eric Wong [Tue, 16 Mar 2021 01:43:44 +0000 (07:43 +0600)]
mbox: move mbox_keywords to MboxReader

MboxReader is a more appropriate place for it than LeiStore.

3 years agot/*: disable fsync on tests were create_inbox isn't worth it
Eric Wong [Mon, 15 Mar 2021 11:58:26 +0000 (12:58 +0100)]
t/*: disable fsync on tests were create_inbox isn't worth it

Using create_inbox doesn't seem worth the trouble, here, at the
moment, but disabling fsync(2) gives a noticeable speedup on
my system even with an SSD.

3 years agot/altid_v2: create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:25 +0000 (12:58 +0100)]
t/altid_v2: create_inbox

Another 100ms saved

3 years agot/thread-index-gap: create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:24 +0000 (12:58 +0100)]
t/thread-index-gap: create_inbox

Yes, this loses some randomness w.r.t. shuffle, but I
think it's "good enough" and buys us ~100ms speedup on
an SSD TMPDIR.

3 years agot/nntpd: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:23 +0000 (12:58 +0100)]
t/nntpd: use create_inbox

3 years agot/nntpd-tls: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:22 +0000 (12:58 +0100)]
t/nntpd-tls: use create_inbox

Another 100ms or so saved.

3 years agot/miscsearch: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:21 +0000 (12:58 +0100)]
t/miscsearch: use create_inbox

3 years agot/psgi_search: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:20 +0000 (12:58 +0100)]
t/psgi_search: use create_inbox

3 years agot/multi-mid: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:19 +0000 (12:58 +0100)]
t/multi-mid: use create_inbox

3 years agot/indexlevels-mirror: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:18 +0000 (12:58 +0100)]
t/indexlevels-mirror: use create_inbox

This saves hundreds of milliseconds and reduces LoC.

3 years agot/lei_xsearch: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:17 +0000 (12:58 +0100)]
t/lei_xsearch: use create_inbox

This ends up being significantly faster when confined
to a slow TMPDIR.

3 years agot/inbox_idle: switch to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:16 +0000 (12:58 +0100)]
t/inbox_idle: switch to create_inbox

Surprising to me, this is actually significantly faster
even though we're starting off with an empty inbox due
to -no_fsync being the default.

3 years agot/edit: switch to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:15 +0000 (12:58 +0100)]
t/edit: switch to create_inbox

Hardly any time reduction, but code gets more compact.

3 years agot/cgi: create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:14 +0000 (12:58 +0100)]
t/cgi: create_inbox

This only saves a few ms but is nicer to look at.

3 years agot/solver_git: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:13 +0000 (12:58 +0100)]
t/solver_git: use create_inbox

This saves us a dozen or so milliseconds.

3 years agot/imapd: create_inbox (minor)
Eric Wong [Mon, 15 Mar 2021 11:58:12 +0000 (12:58 +0100)]
t/imapd: create_inbox (minor)

This saves over 100ms.

3 years agot/psgi_v2: create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:11 +0000 (12:58 +0100)]
t/psgi_v2: create_inbox

Some lines of code and a dozen milliseconds dealth with.

3 years agot/psgi_scan_all: create_inbox lots saved
Eric Wong [Mon, 15 Mar 2021 11:58:10 +0000 (12:58 +0100)]
t/psgi_scan_all: create_inbox lots saved

A fair mount of setup code goes away and saves us a few
hundred milliseconds.

3 years agot/psgi_multipart_not: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:09 +0000 (12:58 +0100)]
t/psgi_multipart_not: use create_inbox

Over 100ms saved.

3 years agot/psgi_bad_mids: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:08 +0000 (12:58 +0100)]
t/psgi_bad_mids: use create_inbox

Over 100ms saved and it's less code to boot.

3 years agot/feed: switch to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:07 +0000 (12:58 +0100)]
t/feed: switch to create_inbox

This only saves a few milliseconds, but is less code.

3 years agot/psgi_mount: switch to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:06 +0000 (12:58 +0100)]
t/psgi_mount: switch to create_inbox

A few dozen more milliseconds saved.

3 years agot/altid: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:05 +0000 (12:58 +0100)]
t/altid: use create_inbox

Another few dozen milliseconds saved.

3 years agot/convert-compact: create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:04 +0000 (12:58 +0100)]
t/convert-compact: create_inbox

Suprisingly, this saves over 100 milliseconds.

3 years agot/httpd: convert to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:03 +0000 (12:58 +0100)]
t/httpd: convert to create_inbox

No real difference, here, but having less code is nice.

3 years agot/psgi_attach: convert to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:02 +0000 (12:58 +0100)]
t/psgi_attach: convert to create_inbox

Another few dozen milliseconds saved.

3 years agot/plack: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:58:01 +0000 (12:58 +0100)]
t/plack: use create_inbox

Only a few dozen milliseconds saved, but better than nothing.

3 years agot/html_index: remove now-worthless test
Eric Wong [Mon, 15 Mar 2021 11:58:00 +0000 (12:58 +0100)]
t/html_index: remove now-worthless test

This was for quote-folding behavior we had long ago, but
it ended up just being yet another import test.

3 years agot/admin: switch to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:57:59 +0000 (12:57 +0100)]
t/admin: switch to create_inbox

Over 100ms saved.

v2: quiet down "git gc" warnings on empty inboxes

3 years agot/v2dupindex: create_inbox
Eric Wong [Mon, 15 Mar 2021 11:57:58 +0000 (12:57 +0100)]
t/v2dupindex: create_inbox

Another hundred milliseconds or so saved.

3 years agot/xcpdb-reshard: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:57:57 +0000 (12:57 +0100)]
t/xcpdb-reshard: use create_inbox

Over 100ms saved and fewer LoC to boot

3 years agot/www_altid: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:57:56 +0000 (12:57 +0100)]
t/www_altid: use create_inbox

This barely saves any time due to sqlite3(1) fsync,
but does save some lines of code.

3 years agot/imapd-tls: switch to create_inbox
Eric Wong [Mon, 15 Mar 2021 11:57:55 +0000 (12:57 +0100)]
t/imapd-tls: switch to create_inbox

This saves another few dozen milliseconds and LoC.

3 years agotest_common: minor simplifications to setup_public_inboxes
Eric Wong [Mon, 15 Mar 2021 11:57:54 +0000 (12:57 +0100)]
test_common: minor simplifications to setup_public_inboxes

This will results in a small reduction in on-disk footprint
by removing Xapian docdata and reduction in code by removing
an unnecessary -index invocation.

3 years agot/lei_xsearch: use create_inbox
Eric Wong [Mon, 15 Mar 2021 11:57:53 +0000 (12:57 +0100)]
t/lei_xsearch: use create_inbox

Less code and noticeably faster when TMPDIR is on an SSD

3 years agotest_common: add create_inbox helper sub
Eric Wong [Mon, 15 Mar 2021 11:57:52 +0000 (12:57 +0100)]
test_common: add create_inbox helper sub

This saves over 100ms in t/lei-q-remote-import.t so far when
TMPDIR is on an SSD.  If we can memoize inbox creation to save a
few dozen milliseconds every test, this could add up to
noticeable savings across our entire test suite.

3 years agolei: reuse LeiStore object on config changes
Eric Wong [Mon, 15 Mar 2021 09:32:06 +0000 (07:32 -0200)]
lei: reuse LeiStore object on config changes

Unless leistore.dir changes, the same LeiStore object
is should remain reusable and accessible to any clients

This seems to fix problems with t/lei-q-remote-import.t
occasionally getting stuck

3 years agolei q: do not import unnecessarily from externals
Eric Wong [Sun, 14 Mar 2021 11:12:00 +0000 (13:12 +0200)]
lei q: do not import unnecessarily from externals

We only want to auto import messages that are exclusively in
remote externals.  Messages in local externals are not
auto-imported to save space and reduce wear on storage device.

3 years agosearchidx: fix -Lmedium for IDs and filenames
Eric Wong [Sat, 13 Mar 2021 15:40:27 +0000 (15:40 +0000)]
searchidx: fix -Lmedium for IDs and filenames

This fixes "m:", "l:", "f:", "t:", "c:", "dfn:", and "n:" search
prefixes under indexlevel=medium when mixed with indexlevel=full
inboxish.  We need positional data for Message-IDs, List-Id,
email addresses and filenames for exact matches, though we still
want to support wildcards.

Fortunately the storage cost is still small as these prefixes
tend to be small compared to message bodies.  These are NOT
boolean terms since wildcard support and partial matching is
desired.

3 years agolei q: mbox*: disable changing parallelism, add --rsyncable
Eric Wong [Fri, 12 Mar 2021 10:39:43 +0000 (10:39 +0000)]
lei q: mbox*: disable changing parallelism, add --rsyncable

Unfortunately, being mairix-compatible with --threads means we
can't change thread-count of gzip, bzip2, or xz when writing to
compressed mbox with a --threads= parameter.  It's probably not
worth changing, anyways, so another switch or additional value
for --jobs= won't be added.

While we're in the area, add --rsyncable support since
most installations of gzip support it nowadays.

Fixes: 5beb4a5f6585acd ("lei: replace --thread with --threads")
3 years agolei: rearrange OPT_DESC and drop some TBD switches
Eric Wong [Fri, 12 Mar 2021 10:39:42 +0000 (10:39 +0000)]
lei: rearrange OPT_DESC and drop some TBD switches

It'll be easier for us to have the option-spec in front of the
command instead of the other way around.  The option-spec in
front makes it easier to sort and keep track of potentially
confusing/ambiguous use of command-line switches between
different commands.

We'll also update some of the proposed switches while we're
at it.

3 years agolei: add help + completion for --no-external
Eric Wong [Fri, 12 Mar 2021 10:39:41 +0000 (10:39 +0000)]
lei: add help + completion for --no-external

I just needed it.

3 years agoexamples/varnish-4: http => httpd
Eric Wong [Sat, 13 Mar 2021 00:08:47 +0000 (16:08 -0800)]
examples/varnish-4: http => httpd

Our HTTP daemon is `public-inbox-httpd', not
`public-inbox-http'.

3 years agoTODO: update tests item
Eric Wong [Fri, 12 Mar 2021 10:44:03 +0000 (10:44 +0000)]
TODO: update tests item

Our code base is growing, and we could repeat less setup
work in our tests and improve development pace.

3 years agot/v2reindex: avoid reading ~/.public-inbox/config in test
Eric Wong [Fri, 12 Mar 2021 03:52:28 +0000 (03:52 +0000)]
t/v2reindex: avoid reading ~/.public-inbox/config in test

3 years agomsg_part_text: discover text in application/octet-stream
Eric Wong [Thu, 11 Mar 2021 01:45:39 +0000 (19:45 -0600)]
msg_part_text: discover text in application/octet-stream

Some poorly-configured MUAs will send application/octet-stream
even for text-only attachments.  We can't make expect all MUAs
are configured with proper MIME types, and there is plenty of
historical mail that falls into this unfortunate criteria.

v2: simplify the check and ensures returned text is Perl "utf8"

3 years agowww_attach: remove unnecessary parse_content_type import
Eric Wong [Thu, 11 Mar 2021 12:58:01 +0000 (06:58 -0600)]
www_attach: remove unnecessary parse_content_type import

It's not needed since we have the handy eml->ct method.

3 years agov2writable: fix undocumented --xapian-only
Eric Wong [Thu, 11 Mar 2021 10:45:38 +0000 (02:45 -0800)]
v2writable: fix undocumented --xapian-only

We can't pass $self and GLOBs across IPC channels transparently.
I only noticed this because I'm testing the application/octet-stream
fallback with https://public-inbox.org/meta/20210311014539.19756-1-e@80x24.org/

Fixes: bf8df8160076d7a1 ("searchidxshard: use PublicInbox::IPC to kill lots of code")
3 years agoconfig: use '-f' key to store config file pathname
Eric Wong [Thu, 11 Mar 2021 10:45:37 +0000 (02:45 -0800)]
config: use '-f' key to store config file pathname

This fixes ->urlmatch use from lei, which already sets '-f'.
I noticed this because imap.$URL.compress was ignored in
my lei config file.

3 years agoimapclient: disable workaround for Mail::IMAPClient 3.43+
Eric Wong [Thu, 11 Mar 2021 10:45:36 +0000 (02:45 -0800)]
imapclient: disable workaround for Mail::IMAPClient 3.43+

These fixes are in the recently-released Mail::IMAPClient 3.43:

https://metacpan.org/source/PLOBBES/Mail-IMAPClient-3.43/Changes

v2: disable 'once' warning

3 years agodoc: update 1.7 release notes, tuning, TODO
Eric Wong [Thu, 11 Mar 2021 10:45:35 +0000 (02:45 -0800)]
doc: update 1.7 release notes, tuning, TODO

Some stuff done, some stuff still needs doing.

3 years agolei_curl: note proposed master/client mode for curl
Eric Wong [Thu, 11 Mar 2021 10:45:34 +0000 (02:45 -0800)]
lei_curl: note proposed master/client mode for curl

Who knows, maybe stuff learned during lei development
can be used to implement it in curl:
https://curl.se/mail/archive-2021-02/0031.html

3 years agosearchidx: remove smsg_from_doc
Eric Wong [Thu, 11 Mar 2021 10:45:33 +0000 (02:45 -0800)]
searchidx: remove smsg_from_doc

We no longer read Xapian docdata and favor hitting over.sqlite3,
instead, as Xapian is less likely to be available than SQLite.

3 years agodoc: glossary: add information for dates and timestamps
Eric Wong [Thu, 11 Mar 2021 10:45:32 +0000 (02:45 -0800)]
doc: glossary: add information for dates and timestamps

These have been confusing to me in the past, too.

3 years agodoc: start glossary for overlapping concepts
Eric Wong [Wed, 10 Mar 2021 13:23:45 +0000 (13:23 +0000)]
doc: start glossary for overlapping concepts

This is intended to keep track of concepts with different terms
between NNTP, IMAP, config file, lei storage, and upcoming
JMAP support.

3 years agolei import: skip trashed Maildir messages
Eric Wong [Wed, 10 Mar 2021 13:23:44 +0000 (13:23 +0000)]
lei import: skip trashed Maildir messages

This matches IMAP behavior in NetReader in skipping \\Deleted
messages.  Since lei may be used for personal, non-public mail;
Draft messages are NOT skipped by "lei import".

3 years agolei import: simplify Maildir handling
Eric Wong [Wed, 10 Mar 2021 13:23:43 +0000 (13:23 +0000)]
lei import: simplify Maildir handling

Having a one-off Maildir functionality in LeiStore doesn't seem
worth the maintenance burden, especially given an upcoming
change to skip trashed messages.

I expect this will hurt performance slightly with extra IPC
overhead for the socket copy, but "lei import" may eventually
become rare or at least not hit messages redundantly.

3 years agowatch: IMAP: ignore \Deleted and \Draft messages
Eric Wong [Wed, 10 Mar 2021 13:23:42 +0000 (13:23 +0000)]
watch: IMAP: ignore \Deleted and \Draft messages

This matches existing Maildir behavior, as trash and draft
messages have little reason to be exposed publicly.

3 years agodoc: technical/data_structures: update for EOFpipe
Eric Wong [Wed, 10 Mar 2021 13:23:41 +0000 (13:23 +0000)]
doc: technical/data_structures: update for EOFpipe

ParentPipe no longer exists and was replaced by the more
flexible EOFpipe.

3 years agolei q: remove angle brackets around Message-IDs
Eric Wong [Mon, 8 Mar 2021 08:08:16 +0000 (08:08 +0000)]
lei q: remove angle brackets around Message-IDs

They're unnecessary visual noise, and angle brackets don't
always work as intended when going through Xapian's query
parser.

Since we already use "m:" and "refs:" instead of the actual
header names, it should be obvious we're at liberty to
abbreviate such things

Link: https://public-inbox.org/meta/20210304184348.GA19350@dcvr/
3 years agolei q: one -t shouldn't set `flagged' on external mail
Eric Wong [Fri, 5 Mar 2021 04:03:05 +0000 (04:03 +0000)]
lei q: one -t shouldn't set `flagged' on external mail

We only want to set `flagged' if a user requests it via
a two '-t' switches.

Fixes: 232f8e376fe2856c ("lei q: -tt marks direct hits as flagged")
3 years agolei q: fix --import-before default and FIFO output
Eric Wong [Fri, 5 Mar 2021 01:38:29 +0000 (18:38 -0700)]
lei q: fix --import-before default and FIFO output

commit 6c551bffd75afb41d9b5e4774068abe7e06ed0e7
("lei q: --import-augment for mbox and mbox.gz") added a check to
in _pre_augment_mbox for the option being a ref() to distinguish
between default values and user-supplied values (which are
non-ref SCALARs from Getopt::Long).

However, LeiQuery failed to use a SCALAR ref as the default
value, making the check in _pre_augment_mbox useless.  We
now update LeiQuery to use \1 instead of 1 as the default
value so "lei q -f mboxrd ..." to stdout works once again.

Unfortunately, testing with redirects pointed to regular
files didn't trigger the code paths being updated.  Testing
with a FIFO revealed further bugs in the FIFO handling code
which are also fixed in this commit.

We'll also update the $lei->out error message to be
less-specific about "stdout" and use the term "output", instead,
since LeiToMail replaces stdout for all mbox outputs.

3 years agosearch: use "z:" instead of "bytes:" prefix
Eric Wong [Fri, 5 Mar 2021 03:10:58 +0000 (19:10 -0800)]
search: use "z:" instead of "bytes:" prefix

So far, searching by size has never been publicly documented,
and IMHO, of questionable utility.  In any case, "z:" is what
mairix(1) uses, so it may be familiar to existing mairix users
(I've never used this prefix myself).

So far, this prefix is only used internally in tests and in
auto-translated queries from IMAP; thus this incompatible change
is unlikely to affect anyone.

3 years agolei q: s/import-augment/import-before/g
Eric Wong [Thu, 4 Mar 2021 09:03:16 +0000 (17:03 +0800)]
lei q: s/import-augment/import-before/g

Since this importing of keywords is active even when --augment
isn't specified, calling it --import-before seems more
appropriate.

In the future, this will likely default to adding unseen emails
to lei/store, not just updating keywords.

Link: https://public-inbox.org/meta/20210303222930.GA18597@dcvr/T/