10 months agopublic-inbox 1.7.0 v1.7.0
Eric Wong [Thu, 4 Nov 2021 07:43:06 +0000 (07:43 +0000)]
public-inbox 1.7.0

10 months agodoc: relnotes: a few more 1.7.0 related updates
Eric Wong [Wed, 3 Nov 2021 21:09:55 +0000 (21:09 +0000)]
doc: relnotes: a few more 1.7.0 related updates

Note "--all" for -extindex, and some minor wording fixes.

10 months agoAUTHORS: clarify my title
Eric Wong [Thu, 4 Nov 2021 07:03:01 +0000 (07:03 +0000)]
AUTHORS: clarify my title

Being an anti-centralization, anti-authority project; the
traditional meaning of "Benevolent Dictator" never sat well
with me.

Benevolence is relative; and I've never been benevolent towards
monopolist-types who try to consolidate power and influence.
Power corrupts, after all.  In any case, I'll never be more than
a random idiot serving data which anybody can mirror and fork.

10 months agodoc: design_notes: updates for "newer" things
Eric Wong [Thu, 4 Nov 2021 03:40:33 +0000 (03:40 +0000)]
doc: design_notes: updates for "newer" things

public-inbox-imapd, public-inbox-watch, and marketing.txt all
exist, now.

10 months agolei_curl: use http.proxy knob via URL match for curl
Eric Wong [Wed, 3 Nov 2021 20:35:55 +0000 (20:35 +0000)]
lei_curl: use http.proxy knob via URL match for curl

Using the --proxy on the command-line affects the entire
lei invocation, and users searching HTTP(S) remotes and
writing to an IMAP folder may want more fine-grained proxy

  lei q -o imap://no-proxy.example/foo -O https://need-proxy.example/bar ...

10 months agodoc: txt2pre: linkify a add a few more well-known things
Eric Wong [Wed, 3 Nov 2021 21:01:23 +0000 (21:01 +0000)]
doc: txt2pre: linkify a add a few more well-known things

Maybe these will help folks less familiar with some of these things.

10 months agodoc: switch to man(1) for pod => (text|html)
Eric Wong [Wed, 3 Nov 2021 21:01:22 +0000 (21:01 +0000)]
doc: switch to man(1) for pod => (text|html)

pod2text(1) will wrap long .onion URLs and cause resulting HTML
to be linkified improperly.

10 months agodoc: add more 3rd-party refs, use Debian manpages for xapian
Eric Wong [Wed, 3 Nov 2021 21:01:21 +0000 (21:01 +0000)]
doc: add more 3rd-party refs, use Debian manpages for xapian

curl, torsocks, and gitglossary manpages are all newly
referenced, so make sure they're linkified properly in HTML.
We'll be using Debian's manpages as an ad-free, Tor-accessible
host for manpages as a fallback since hosting manpages for all
3rd-party projects we reference doesn't scale.

10 months agodoc: relnotes: 1.7.0: move extindex, note search results change
Eric Wong [Wed, 3 Nov 2021 08:34:46 +0000 (08:34 +0000)]
doc: relnotes: 1.7.0: move extindex, note search results change

extindex is a far more important feature than libgit2 support
(which is actually underperforming and might go away).  The
search results page is also improved (IMHO), nowadays.

10 months agodoc: -clone|lei add-external: add bit about the Makefile
Eric Wong [Wed, 3 Nov 2021 08:34:45 +0000 (08:34 +0000)]
doc: -clone|lei add-external: add bit about the Makefile

It's pretty useful, I think.

10 months agodoc: extindex: document current behavior + knobs
Eric Wong [Wed, 3 Nov 2021 08:34:44 +0000 (08:34 +0000)]
doc: extindex: document current behavior + knobs

I'm not really sure if extindex writing to the config file
is a good idea (since -index doesn't, as -init exists).
Just document what it does and let the user handle it, since
the config file shouldn't be daunting to new users.

10 months agodoc: lei-q: document SEARCH TERMS prefixes
Eric Wong [Tue, 2 Nov 2021 23:55:37 +0000 (12:55 -1100)]
doc: lei-q: document SEARCH TERMS prefixes

The new Documentation/common.perl file will be used for
all manpages in the future.

10 months agodoc: txt2pre: add references to newish manpages
Eric Wong [Tue, 2 Nov 2021 23:55:32 +0000 (12:55 -1100)]
doc: txt2pre: add references to newish manpages

10 months agolei <rediff|rm|tag>: stdin implies `-F eml'
Eric Wong [Tue, 2 Nov 2021 18:14:45 +0000 (18:14 +0000)]
lei <rediff|rm|tag>: stdin implies `-F eml'

These commands are usually run on a single message, so saving
the user the trouble of typing `-F eml' on the command-line
seems reasonable.  I don't think commands like "index" and
"import" will be too useful for single messages, though.

10 months agolei: simplify common LeiInput users with ->wq1_start
Eric Wong [Tue, 2 Nov 2021 18:14:44 +0000 (18:14 +0000)]
lei: simplify common LeiInput users with ->wq1_start

This method replaces a common pattern of starting workers,
preparing internal auth ops, and asynchronous waiting of
command completion.

It also adds missing LeiAuth support to rediff and rm
which rarely need auth.

10 months agolei mail-diff: do not default to 'eml'
Eric Wong [Tue, 2 Nov 2021 18:14:43 +0000 (18:14 +0000)]
lei mail-diff: do not default to 'eml'

In retrospect, this doesn't make sense, since it needs at least
two messages to diff.  So go about "normal" input rules and
require users to specify the format.

10 months agot/lei-refresh-mail-sync: speed up test on FreeBSD 12
Eric Wong [Tue, 2 Nov 2021 09:24:39 +0000 (09:24 +0000)]
t/lei-refresh-mail-sync: speed up test on FreeBSD 12

And improve reliability while we're at it.  It seems closing a
TCP listen socket on FreeBSD 12.2 doesn't cause connect()-ing
clients to fail.  This happens regardless of whether a socket is
IPv4 or IPv6

This non-failure was causing tests to timeout slowly on the
client side instead of failing immediately.  We now fork a new
process which does nothing but accept() + shutdown() to emulate
a dead server.

Reliability improves on all OSes since there's never a point in
time when another process can bind the socket.

10 months agoinit: respect umask when creating description
Eric Wong [Tue, 2 Nov 2021 06:57:43 +0000 (06:57 +0000)]
init: respect umask when creating description

I noticed a description for a new inbox had st_mode=0600.

10 months agombox_reader: do not blindly pass --rsyncable to gzip
Eric Wong [Mon, 1 Nov 2021 23:46:20 +0000 (23:46 +0000)]
mbox_reader: do not blindly pass --rsyncable to gzip

FreeBSD gzip does not support --rsyncable, though my VM
usually has pigz installed.

10 months agotreewide: kill problematic "$h->{k} //= do {" assignments
Eric Wong [Mon, 1 Nov 2021 19:06:09 +0000 (19:06 +0000)]
treewide: kill problematic "$h->{k} //= do {" assignments

As stated in the previous change, conditional hash assignments
which trigger other hash assignments seem problematic, at times.
So replace:

$h->{k} //= do { $h->{x} = ...; $val };

$h->{k} // do {
$h->{x} = ...;
$hk->{k} = $val

"||=" is affected the same way, and some instances of "||=" are
replaced with "//=" or "// do {", now.

10 months agoidx_stack: avoid conditional hash assignment weirdness
Eric Wong [Mon, 1 Nov 2021 19:06:08 +0000 (19:06 +0000)]
idx_stack: avoid conditional hash assignment weirdness

I've been seeing the following error on occasion during "make check-run":
$PWD/t/data-gen/reindex-time-range.v1-master index failed: Modification of a read-only value attempted at $DIR/lib/PublicInbox/ line 899, <$r> line 1.

Perhaps this fixes it.  In any case, a construct of:

 $h->{k} //= do { $h->{x} = ...; $val };

seems wrong and may cause Perl to error out depending on how
hashes are randomized.

10 months agodoc: lei-config: fix missing =back
Eric Wong [Mon, 1 Nov 2021 19:00:25 +0000 (19:00 +0000)]
doc: lei-config: fix missing =back

10 months agodoc: update release notes and INSTALL
Eric Wong [Sun, 31 Oct 2021 20:07:36 +0000 (20:07 +0000)]
doc: update release notes and INSTALL

This is what I can think of at the moment.

10 months agolei_input: disallow uppercase characters for labels
Eric Wong [Sun, 31 Oct 2021 09:26:58 +0000 (09:26 +0000)]
lei_input: disallow uppercase characters for labels

Xapian boolean terms rely on upper-case prefixes, so the terms
themselves need to be all lowercase.

10 months agodoc: add lei-mail-sync-overview manpage
Eric Wong [Sun, 31 Oct 2021 09:10:16 +0000 (09:10 +0000)]
doc: add lei-mail-sync-overview manpage

Mostly illustrating how clunky the process is :p
We'll also tweak some things in existing man pages around
mail synchronization.

10 months agodoc: lei-security: add a note about core dumps
Eric Wong [Sat, 30 Oct 2021 08:11:44 +0000 (08:11 +0000)]
doc: lei-security: add a note about core dumps

Maybe we can avoid them if we stop having buggy code :P

10 months agolei_to_mail: avoid SEGV on worker exit via SIGTERM
Eric Wong [Sat, 30 Oct 2021 08:11:43 +0000 (08:11 +0000)]
lei_to_mail: avoid SEGV on worker exit via SIGTERM

->DESTROY ordering via "exit()" calls is tricky, and dedupe
checks were causing problems.

AFAIK, this only affects users who manually enable WAL on
lei/store/ei*/over.sqlite3.  Fortunately, there is no data
corruption as a result even though "read-only" WAL requires
write permissions.

10 months agolei_xsearch: quiet error message on SIG{PIPE,TERM}
Eric Wong [Sat, 30 Oct 2021 08:11:42 +0000 (08:11 +0000)]
lei_xsearch: quiet error message on SIG{PIPE,TERM}

SIGPIPE and SIGTERM are common and user-induced, so they're
not worth warning on.  Add the value of "$?", though, since
it can help users notice other errors (e.g. SIGSEGV).

10 months agolei_to_mail: limit workers for text, reply and v2 outputs
Eric Wong [Sat, 30 Oct 2021 08:11:41 +0000 (08:11 +0000)]
lei_to_mail: limit workers for text, reply and v2 outputs

"text" and "reply" outputs are intended for the pager, so
parallelizing them is a waste of resources.

v2 has shards, of course, so parallelizing writes to it
is also a waste since the deduplication work is a bit
more complex.

10 months agolei: do not access {sock} after SIGPIPE
Eric Wong [Sat, 30 Oct 2021 08:11:40 +0000 (08:11 +0000)]
lei: do not access {sock} after SIGPIPE

It's possible for this to break out of the event loop if
note_sigpipe fires via PktOp in the same iteration.

10 months agotest_common: clear XDG_CACHE_HOME before lei tests
Eric Wong [Thu, 28 Oct 2021 19:16:50 +0000 (19:16 +0000)]
test_common: clear XDG_CACHE_HOME before lei tests

We don't want to read a users'
$XDG_CACHE_HOME/lei/all_locals_ever.git during tests.

Reported-by: Thomas Weißschuh <>
Tested-by: Thomas Weißschuh <>
10 months agolei rm: move generic input_maildir_cb to LeiInput parent class
Eric Wong [Thu, 28 Oct 2021 11:15:01 +0000 (11:15 +0000)]
lei rm: move generic input_maildir_cb to LeiInput parent class

It's not much of a savings, right now, but maybe it can be in the
future.  I wanted to eliminate the "lei convert" one, too, but
convert needs to preserve keywords which isn't possible with the
generic fallback, so new tests were written for convert, instead.

10 months agolei sucks: show nproc in CPU info
Eric Wong [Thu, 28 Oct 2021 11:15:00 +0000 (11:15 +0000)]
lei sucks: show nproc in CPU info

Some bugs are triggered with more CPUs, some with 1 CPU.

10 months agodoc: lei-add-watch: add warning about unreliability
Eric Wong [Thu, 28 Oct 2021 11:14:59 +0000 (11:14 +0000)]
doc: lei-add-watch: add warning about unreliability

This needs work at some point in the future.

10 months agolei convert: remove redundant input_net_cb
Eric Wong [Thu, 28 Oct 2021 11:14:58 +0000 (11:14 +0000)]
lei convert: remove redundant input_net_cb

Use the one provided by the LeiInput parent class.

10 months agodoc: lei blob: wording fixups, describe --remote
Eric Wong [Thu, 28 Oct 2021 11:14:57 +0000 (11:14 +0000)]
doc: lei blob: wording fixups, describe --remote

There's no current way to retrieve blobs by OID directly
from remote externals.  Maybe the $INBOX_NAME/$OID/s/raw.eml
endpoint could be overloaded for that.

10 months agodoc: lei-convert: various updates and cleanups
Eric Wong [Thu, 28 Oct 2021 11:14:56 +0000 (11:14 +0000)]
doc: lei-convert: various updates and cleanups

Note that "-o OUTPUT" is required in the synopsis.

Leave out "eml:" for now since it doesn't work as an output and
I doubt anybody would use it as a prefix, and it's not really

--no-import-remote is also not accepted by convert, since it
doesn't touch lei/store at all.

10 months agolei convert: use "--output" in failure message
Eric Wong [Thu, 28 Oct 2021 11:14:55 +0000 (11:14 +0000)]
lei convert: use "--output" in failure message

The extra dashes should help users find the correct option
more easily.

10 months agoxt/net_writer_imap: test "lei convert" w/ IMAP source
Eric Wong [Thu, 28 Oct 2021 11:14:54 +0000 (11:14 +0000)]
xt/net_writer_imap: test "lei convert" w/ IMAP source

I just did a double-take and nearly thought authentication
was broken while reading  Add a comment in to clarify things, too.

10 months agolei add-watch: ensure folders are known to mail_sync.sqlite3
Eric Wong [Thu, 28 Oct 2021 06:17:22 +0000 (06:17 +0000)]
lei add-watch: ensure folders are known to mail_sync.sqlite3

This prevents noisy errors in syslog when running t/lei-watch.t

10 months agolei q: fix remote import accounting
Eric Wong [Wed, 27 Oct 2021 21:09:19 +0000 (21:09 +0000)]
lei q: fix remote import accounting

We need to update the {-nr_remote_eml} counter regardless
of progress display being enabled since it's needed for
saved searches.  We'll also split out the {-imported} flag
separately and only call LeiStore->done if a new message
was imported.

Note: this change is NOT expected to fix errors reported by
Thomas in <>

Cc: Thomas Weißschuh <>
10 months agotest_common: key test inboxes to init.defaultBranch
Eric Wong [Wed, 27 Oct 2021 04:07:54 +0000 (04:07 +0000)]
test_common: key test inboxes to init.defaultBranch

This lets users change their global init.defaultBranch config
knob in ~/.gitconfig or similar without breaking tests.

Reported-by: Thomas Weißschuh <>
Tested-by: Thomas Weißschuh <>
10 months agolei mail-diff: support more inputs, split newlines
Eric Wong [Tue, 26 Oct 2021 21:18:05 +0000 (21:18 +0000)]
lei mail-diff: support more inputs, split newlines

Support --in-format like the rest of LeiInput users, and don't
default to .eml if a per-input format was specified.  In any
case, I saved a bunch of messages from mutt which uses mboxcl2.

We'll also split newlines for diff, since it's a pain to read
diffs with escaped "\n" characters in them.

10 months agot/lei-watch: add diagnostics for failure
Eric Wong [Tue, 26 Oct 2021 10:47:34 +0000 (10:47 +0000)]
t/lei-watch: add diagnostics for failure

I just got a difficult-to-reproduce failure, here; so there's
still some issues with the up-to-dateness of the inotify watcher.

10 months agolei_to_mail: only run lms_write_prepare for IMAP+Maildir
Eric Wong [Tue, 26 Oct 2021 10:47:26 +0000 (10:47 +0000)]
lei_to_mail: only run lms_write_prepare for IMAP+Maildir

Mail synchronization in lei_to_mail only works for IMAP and
Maildir; so don't waste time preparing mbox* writers for it.

10 months agoinput_pipe: account for undefined {sock}
Eric Wong [Tue, 26 Oct 2021 10:35:57 +0000 (10:35 +0000)]
input_pipe: account for undefined {sock}

It's possible for ->event_step to fire twice due to ->requeue
with EPOLLET (but not EPOLLONESHOT).  So account for that and
avoid causing event loop errors as a result.

10 months agolei rm|tag: drop redundant mbox+net callbacks
Eric Wong [Tue, 26 Oct 2021 10:35:56 +0000 (10:35 +0000)]
lei rm|tag: drop redundant mbox+net callbacks

These are supplied by the base LeiInput class

10 months agolei p2q: use LeiInput for multi-patch series
Eric Wong [Tue, 26 Oct 2021 10:35:55 +0000 (10:35 +0000)]
lei p2q: use LeiInput for multi-patch series

The LeiInput backend now allows p2q to work like any other
command which reads .eml, .patch, mbox*, Maildir, IMAP, and NNTP
input.  Running "git format-patch --stdout -1 $COMMIT" remains

This is intended to allow lower memory use while parsing
"git log --pretty=mboxrd -p" output.  Previously, the entire
output of "git log" would be slurped into memory at once.

The intended use is to allow easy(-ish :P) searching for
unapplied patches as documented in the new example in the

10 months agolei: add net getopt spec to various commands
Eric Wong [Tue, 26 Oct 2021 10:35:54 +0000 (10:35 +0000)]
lei: add net getopt spec to various commands

All of these commands should support --proxy, at least, if not
other curl options.

10 months agolei inspect: fix atfork hook
Eric Wong [Tue, 26 Oct 2021 10:35:53 +0000 (10:35 +0000)]
lei inspect: fix atfork hook

The misnamed sub wasn't firing, but was unlikely to be
noticeable given the short lifetime of the process.

Fixes: 1f887bd51d92b0d4 ("lei inspect: add atfork hook")
10 months agolei q: enable expensive Xapian flags
Eric Wong [Tue, 26 Oct 2021 10:35:52 +0000 (10:35 +0000)]
lei q: enable expensive Xapian flags

FLAG_PURE_NOT is too expensive for public-facing WWW use, but
lei isn't public-facing.  We'll also unconditionally enable
phrase search on old "chert" DBs since lei doesn't need to
worry about fairness across 10K users.

10 months agoeml: keep body if no headers are found
Eric Wong [Tue, 26 Oct 2021 10:35:51 +0000 (10:35 +0000)]
eml: keep body if no headers are found

This easily allows us to treat "git diff" output as header-less
"messages" for commands such as "lei p2q".

10 months agodoc: lei-store-format: mail sync section, update IPC
Eric Wong [Tue, 26 Oct 2021 10:35:50 +0000 (10:35 +0000)]
doc: lei-store-format: mail sync section, update IPC

mail_sync.sqlite3 needs to be documented, and brings the IPC
section up-to-date while we're in the area.

10 months agodoc: tuning: additional notes for many inboxes
Eric Wong [Tue, 26 Oct 2021 10:35:49 +0000 (10:35 +0000)]
doc: tuning: additional notes for many inboxes

-extindex is the most important piece for dealing with many
inboxes, so note it first.  Also, frequent use of "git gc" is
important for both loose object performance and reducing memory

11 months agolei p2q: document --uri, add examples
Eric Wong [Mon, 25 Oct 2021 19:31:47 +0000 (19:31 +0000)]
lei p2q: document --uri, add examples

This is useful for users lacking in local storage.  Also,
referencing lei-add-external(1) seems to make less sense
than referencing lei-q(1).

We'll also start dropping years from the copyright statement
to reduce future churn.

11 months agowww: mirror: fix rendering of NNTP URLs
Kyle Meyer [Tue, 26 Oct 2021 00:48:10 +0000 (20:48 -0400)]
www: mirror: fix rendering of NNTP URLs

As of commit 738c4a65, the code for reporting NNTP information in
_/text/mirror/ incorrectly uses ->imap_url rather than ->nntp_url.

Fixes: 738c4a65719e6278 ("www: various help text updates")
11 months agot/index-git-times: support non-master default branch
Thomas Weißschuh [Mon, 25 Oct 2021 22:24:53 +0000 (00:24 +0200)]
t/index-git-times: support non-master default branch

11 months agolei_to_mail: write directly to mail_sync.sqlite3
Eric Wong [Mon, 25 Oct 2021 08:59:19 +0000 (08:59 +0000)]
lei_to_mail: write directly to mail_sync.sqlite3

No need to go through the lei/store process when we write
mail_sync.sqlite3.  This ought to reduce ENOBUFS errors (and the
sleep workaround) on RAM-starved systems.

11 months agocontrib/css/216light: add more contrast to foreground text
Eric Wong [Mon, 25 Oct 2021 17:53:51 +0000 (14:53 -0300)]
contrib/css/216light: add more contrast to foreground text

333 on dimmed displays doesn't show up well.  I still
find 000 foregrounds too harsh, though, but 003 is available.
It seems dark enough to not cause problems while not being too

003 should be available on more displays, even, and could fit
a 22-color "safest" color scheme.

11 months agowww: $MSGID/raw: set charset in HTTP response
Eric Wong [Mon, 25 Oct 2021 02:45:53 +0000 (02:45 +0000)]
www: $MSGID/raw: set charset in HTTP response

By using the charset specified in the message, web browsers are
more likely to display the raw text properly for human readers.

Inspired by a patch by Thomas Weißschuh:

Cc: Thomas Weißschuh <>
11 months agogzip_filter: delay async wcb call
Eric Wong [Mon, 25 Oct 2021 02:45:52 +0000 (02:45 +0000)]
gzip_filter: delay async wcb call

This will let us modify the response header later to set
a proper charset for Content-Type when displaying raw

Cc: Thomas Weißschuh <>
11 months agot/git: support non-master default branch
Thomas Weißschuh [Sun, 24 Oct 2021 21:43:36 +0000 (23:43 +0200)]
t/git: support non-master default branch

11 months agot/watch_maildir: support non-master default branch
Thomas Weißschuh [Sun, 24 Oct 2021 21:43:35 +0000 (23:43 +0200)]
t/watch_maildir: support non-master default branch

11 months agoviewvcs: die on tmpfile() errors
Eric Wong [Sun, 24 Oct 2021 01:45:22 +0000 (01:45 +0000)]
viewvcs: die on tmpfile() errors

Just let Plack::Util::run_app catch the error and generate
a 500 response for it.

11 months agogit: avoid Perl5 internal scratchpad target cache
Eric Wong [Sun, 24 Oct 2021 00:20:45 +0000 (18:20 -0600)]
git: avoid Perl5 internal scratchpad target cache

Creating a scalar ref directly off substr() seemed to be causing
the underlying non-ref scalar to end up in Perl's scratchpad.
Assign the substr result to a local variable seems sufficient to
prevent multi-megabyte SVs from lingering indefinitely when a
read-only daemon serves rare, oversized blobs.

11 months agothread: avoid Perl5 internal scratchpad target cache
Eric Wong [Sun, 24 Oct 2021 00:20:44 +0000 (18:20 -0600)]
thread: avoid Perl5 internal scratchpad target cache

The use of array-returning built-ins such as `grep' inside
arrayref declarations appears to result in permanently allocated
scratchpad space for caching according to my malloc inspector.

Thread skeletons get discarded every response, but multiple
skeletons can exist in memory at once, so do what we can to
prevent long-lived allocations from being made, here.

In other words, replacing constructs such as:

my $foo = [ grep(...) ];


my @foo = grep(...);

Seems to ensure the mortality of the underlying array.

11 months agolistener: emit warnings on EPERM
Eric Wong [Sun, 24 Oct 2021 00:20:43 +0000 (18:20 -0600)]
listener: emit warnings on EPERM

In retrospect, warnings for EPERM on accept4(2) failure may
help detect misconfigured firewalls, so start emitting warnings
for EPERM.  Fwiw, I've never known excessive EPERM warnings
to be excessively noisy in other TCP services I've run over
the years.

11 months agohttp: use a larger buffer for ->getline responses
Eric Wong [Sun, 24 Oct 2021 00:20:42 +0000 (18:20 -0600)]
http: use a larger buffer for ->getline responses

64K matches the Linux pipe default, and matches what we use in
httpd/async and qspawn.  This should reduce syscalls used for
serving git packs via dumb HTTP and any ->getline code paths
used by other PSGI code.

This appears to speed up HTML rendering by w3m when serving
giant HTML responsees from the Devel::Mwrap::PSGI memory

11 months agoshared_kv: remove cache_size attribute support
Eric Wong [Sun, 24 Oct 2021 00:20:41 +0000 (18:20 -0600)]
shared_kv: remove cache_size attribute support

We're not using it, anywhere.

11 months agolei export-kw: skip read-only IMAP folders
Eric Wong [Sun, 24 Oct 2021 00:20:40 +0000 (18:20 -0600)]
lei export-kw: skip read-only IMAP folders

Since we want to store IMAP flags asynchronously and not wait
for results, we can't check for IMAP errors this way and end up
wasting bandwidth on public-inbox-imapd.  Now, we just check
PERMANENTFLAGS up front to ensure a folder can handle IMAP flag
storage before proceeding.

11 months agolei: always pass $lei to LeiAuth->op_merge
Eric Wong [Sun, 24 Oct 2021 00:20:39 +0000 (18:20 -0600)]
lei: always pass $lei to LeiAuth->op_merge

This will make future developments easier.

11 months agocmd_ipc4: retry sendmsg on ENOBUFS/ENOMEM/ETOOMANYREFS
Eric Wong [Sat, 23 Oct 2021 21:53:46 +0000 (21:53 +0000)]
cmd_ipc4: retry sendmsg on ENOBUFS/ENOMEM/ETOOMANYREFS

I'm seeing ENOBUFS on a RAM-starved system, and slowing the
sender down enough for the receiver to drain the buffers seems
to work.  ENOMEM and ETOOMANYREFS could be in the same boat

Watching for POLLOUT events via select/poll/epoll_wait doesn't
seem to work, since the kernel can already sleep (or return
EAGAIN) for cases where POLLOUT would work.

11 months agowww: respect coderepo.*.url during cgit init
Eric Wong [Sat, 23 Oct 2021 20:19:39 +0000 (20:19 +0000)]
www: respect coderepo.*.url during cgit init

This is necessary for showing "found $OID in $CODEREPO_URL"
in solver-generated pages ($INBOX_URL/$OID/s/).

11 months agoconfig: remove *_url_format support for cgit
Eric Wong [Sat, 23 Oct 2021 20:19:38 +0000 (20:19 +0000)]
config: remove *_url_format support for cgit

We're not using them, anywhere.

11 months agogit: simplify local_nick, avoid "foo.git.git"
Eric Wong [Sat, 23 Oct 2021 20:19:37 +0000 (20:19 +0000)]
git: simplify local_nick, avoid "foo.git.git"

We need to use a non-greedy regexp to avoid capturing the
".git" suffix in the pathname before blindly appending our

11 months agot/v2index-late-dedupe: don't read user's ~/.public-inbox/config
Eric Wong [Sat, 23 Oct 2021 19:09:35 +0000 (04:09 +0900)]
t/v2index-late-dedupe: don't read user's ~/.public-inbox/config

Otherwise things can get noisy if bad entries exist in that
file, because they do.

11 months agosearchidx: v1: raise on msgmap init failure
Eric Wong [Sat, 23 Oct 2021 19:08:41 +0000 (19:08 +0000)]
searchidx: v1: raise on msgmap init failure

Indexing any inboxes requires SQLite and msgmap, so don't hide
exceptions if it fails.

11 months agodoc: lei-forget-search: fix option name in --prune description
Kyle Meyer [Sat, 23 Oct 2021 00:22:40 +0000 (20:22 -0400)]
doc: lei-forget-search: fix option name in --prune description

Fixes: 6f8e16a266b30819 ("lei forget-search: support --prune=<local|remote>")
11 months agolei forget-search: support --prune=<local|remote>
Eric Wong [Fri, 22 Oct 2021 08:22:47 +0000 (08:22 +0000)]
lei forget-search: support --prune=<local|remote>

Instead of:

lei forget-search $OUTPUT && rm -r $OUTPUT

we'll also allow a user to do:

rm -r $OUTPUT && lei forget-search --prune

This gives users flexibility to choose whatever flow
is most natural to them.

11 months agolei export-kw: completion returns all Maildir+IMAP
Eric Wong [Fri, 22 Oct 2021 08:22:46 +0000 (08:22 +0000)]
lei export-kw: completion returns all Maildir+IMAP

It's theoretically possible an AUTH=ANONYMOUS login could be
writable and allowed to store flags for various people (e.g.
within a private network).

11 months agolei export-kw: don't recreate deleted IMAP folders
Eric Wong [Fri, 22 Oct 2021 08:22:45 +0000 (08:22 +0000)]
lei export-kw: don't recreate deleted IMAP folders

In case an IMAP folder is deleted, just set an error and
ignore it rather than creating an empty folder which we
attempt to export keywords to for non-existent messages.

11 months agowwwatomstream: call gmtime with scalar
Kyle Meyer [Fri, 22 Oct 2021 04:49:35 +0000 (00:49 -0400)]
wwwatomstream: call gmtime with scalar

When the gmtime() calls were moved from feed_entry() and atom_header()
into feed_updated() in c447bbbd, @_ rather than a scalar was passed to
gmtime().  As a result, feed <updated> values end up as

Switch back to using a scalar argument to restore the correct

Fixes: c447bbbddb4ac8e1 ("wwwatomstream: simplify feed_update callers")
11 months agolei: use RENAME_NOREPLACE on Linux 3.15+
Eric Wong [Thu, 21 Oct 2021 21:10:32 +0000 (21:10 +0000)]
lei: use RENAME_NOREPLACE on Linux 3.15+

One syscall is better than two for atomicity in Maildirs.  This
means there's no window where another process can see both the
old and new file at the same time (link && unlink), nor a window
where we might inadvertantly clobber an existing file if we were
to do `stat && rename'.

11 months agolei_mail_sync: mv_src: use transaction, check UNIQUE
Eric Wong [Thu, 21 Oct 2021 21:10:31 +0000 (21:10 +0000)]
lei_mail_sync: mv_src: use transaction, check UNIQUE

We need a transaction across two SQL statements so readers
(which don't use flock) will see the result as atomic.

This may help against some occasional test failures I'm seeing
from t/lei-auto-watch.t and t/lei-watch.t, or make the problem
more apparent.

11 months agolei: no Perl FileHandle for `undef' w/ ECONNRESET
Eric Wong [Thu, 21 Oct 2021 21:10:30 +0000 (21:10 +0000)]
lei: no Perl FileHandle for `undef' w/ ECONNRESET

Error reporting for recv_cmd4 methods is a bit wonky.

11 months agodir_idle: treat IN_MOVED_FROM as a gone event
Eric Wong [Thu, 21 Oct 2021 21:10:29 +0000 (21:10 +0000)]
dir_idle: treat IN_MOVED_FROM as a gone event

Whether an MUA uses rename(2) or link(2)+unlink(2) combination
should not matter to us.  We should be able to handle both

11 months agolei note-event: clear_src on ENOENT
Eric Wong [Thu, 21 Oct 2021 21:10:28 +0000 (21:10 +0000)]
lei note-event: clear_src on ENOENT

When a file goes away, try to make sure we don't waste
time trying to access or store it.

11 months agodoc: lei-overview: add CAVEATS section
Eric Wong [Thu, 21 Oct 2021 21:10:27 +0000 (21:10 +0000)]
doc: lei-overview: add CAVEATS section

IMAP and NNTP client performance absolutely sucks compared to what
the read-only daemons are capable of...

11 months agowatch: remove redundant signal mask manipulation
Eric Wong [Thu, 21 Oct 2021 21:10:26 +0000 (21:10 +0000)]
watch: remove redundant signal mask manipulation

The top-level daemon process already blocks all signals,
so there's no reason to block them around fork() calls.

11 months agowatch: check for {quit} before IDLE
Eric Wong [Thu, 21 Oct 2021 21:10:25 +0000 (21:10 +0000)]
watch: check for {quit} before IDLE

This may make it less likely for watch-dependent tests to get
stuck.  Unfortunately, due to the synchronous API of
Mail::IMAPClient, ->idle is still susceptible to missing

11 months agolei_search: try harder to associate "lei index"-ed messages
Eric Wong [Thu, 21 Oct 2021 21:10:24 +0000 (21:10 +0000)]
lei_search: try harder to associate "lei index"-ed messages

Allow checking for keyword changes if we have an known OID,
even if the blob isn't currently reachable.

11 months agolei note-event: wq_io_do => wq_do
Eric Wong [Thu, 21 Oct 2021 21:10:23 +0000 (21:10 +0000)]
lei note-event: wq_io_do => wq_do

No need to pass extra arrayref args, here.

11 months agolei note-event: drop unnecessary eval guard
Eric Wong [Thu, 21 Oct 2021 21:10:22 +0000 (21:10 +0000)]
lei note-event: drop unnecessary eval guard

We don't want to lose the failure message in case note-event

11 months agolei/store: check for any unexpected process death
Eric Wong [Thu, 21 Oct 2021 21:10:21 +0000 (21:10 +0000)]
lei/store: check for any unexpected process death

The lei/store process should only exit from EOF on the
socket, so make sure we note any unintended signals

11 months agot/lei-p2q: extra diagnostics
Eric Wong [Thu, 21 Oct 2021 21:10:20 +0000 (21:10 +0000)]
t/lei-p2q: extra diagnostics

I got one mysterious test failure here, once, and can't seem
to reproduce it...

11 months agot/lei-import-maildir: rename fix (SR -> RS)
Eric Wong [Thu, 21 Oct 2021 21:10:19 +0000 (21:10 +0000)]
t/lei-import-maildir: rename fix (SR -> RS)

While it doesn't matter to us, the Maildir spec specifies
characters are to be sorted in alphabetical order.

11 months agot/lei-{auto-watch,export-kw}: extra diagnostics on failure
Eric Wong [Thu, 21 Oct 2021 21:10:18 +0000 (21:10 +0000)]
t/lei-{auto-watch,export-kw}: extra diagnostics on failure

Maybe these will help track down some failures and make
diagnosing bugs easier.  "lei export-kw" should also become
optional, even, so allow disabling it easily in the test.

11 months agohttpd: reject requests with spaces in header names
Eric Wong [Tue, 19 Oct 2021 21:26:15 +0000 (21:26 +0000)]
httpd: reject requests with spaces in header names

Malicious clients may attempt HTTP request smuggling this way.
This doesn't affect our current code as we only look for exact
matches, but it could affect other servers behind a
to-be-implemented reverse proxy built around our -httpd.

This doesn't affect users behind varnish at all, nor the
HTTPS/HTTP reverse proxy I use (I don't know about nginx), but
could be passed through by other reverse proxies.

This change is only needed for HTTP::Parser::XS which most users
probably use.  Users of the pure Perl parser (via
PLACK_HTTP_PARSER_PP=1) already hit 400 errors in this case,
so this makes the common XS case consistent with the pure Perl


11 months agolei_mail_sync: show non-matching SHA
Eric Wong [Tue, 19 Oct 2021 09:33:46 +0000 (09:33 +0000)]
lei_mail_sync: show non-matching SHA

It could prove useful for diagnosing bugs (either on our
end or an MUA's), or storage device failures.

11 months agolei inspect: show ISO8601 {rt} and {dt}, too
Eric Wong [Tue, 19 Oct 2021 09:33:45 +0000 (09:33 +0000)]
lei inspect: show ISO8601 {rt} and {dt}, too

While inspect is intended for debugging, the Unix epoch in
seconds requires extra steps for human consumption; just
steal what we used for "lei q -f json" output.