public-inbox.git
20 months agopublic-inbox 1.6.1 - minor bugfix release v1.6.1
Eric Wong [Thu, 31 Dec 2020 20:15:28 +0000 (20:15 +0000)]
public-inbox 1.6.1 - minor bugfix release

20 months agoimport: drop X-Status in addition to Status
Eric Wong [Thu, 17 Dec 2020 09:14:48 +0000 (09:14 +0000)]
import: drop X-Status in addition to Status

It's actually supported by mutt, dovecot[1], and likely some other
software to augment the Status: header.  While dovecot doesn't
expose X-Status to clients, mutt will write 'A' (answered) and
'F' to X-Status (but not T (draft)).

So we'll drop it like we do Status since it's not suitable for
public mail, but stick it in an @UNWANTED_HEADERS array will
allow us to configure an override if needed.

[1]  https://doc.dovecot.org/configuration_manual/mail_location/mbox/

20 months agoeml: fix undefined vars on <Perl 5.28
Eric Wong [Sat, 26 Dec 2020 12:25:42 +0000 (12:25 +0000)]
eml: fix undefined vars on <Perl 5.28

Encode::MIME::Header::_decode_octets did not correctly default
to Encode::FB_DEFAULT until Encode 2.93 (perl5.git commit
0c541dc5633a341cf44b818014b58e7f8be532e9).  Provide the default
again to work with older Perls.

Reported-by: Ali Alnubani <alialnu@nvidia.com>
Link: https://public-inbox.org/meta/DM6PR12MB49106F8E3BD697B63B943A22DADB0@DM6PR12MB4910.namprd12.prod.outlook.com/
Tested-by: Ali Alnubani <alialnu@nvidia.com>
(cherry picked from commit 46bd595f57cc3d425754b0d0770c125616e74448)

20 months agot/config: test --get-urlmatch for git <2.26
Eric Wong [Sat, 26 Dec 2020 12:30:35 +0000 (12:30 +0000)]
t/config: test --get-urlmatch for git <2.26

While git 1.8.5 learned --get-urlmatch, git did not learn to
match URLs against wildcards until 2.26.  So only depend on
1.8.5 for this test since 2.26 is too new.

Reported-by: Ali Alnubani <alialnu@nvidia.com>
Link: https://public-inbox.org/meta/DM6PR12MB49106F8E3BD697B63B943A22DADB0@DM6PR12MB4910.namprd12.prod.outlook.com/
Tested-by: Ali Alnubani <alialnu@nvidia.com>
(cherry picked from commit 382a2bdd54cfb6c28a935c2b9fe4f1b1c2f469c4)

20 months agoinboxidle: avoid needless syscalls on refresh
Eric Wong [Sat, 26 Dec 2020 09:34:39 +0000 (09:34 +0000)]
inboxidle: avoid needless syscalls on refresh

We don't have to replace a bunch of existing watches
with identical new ones.  On Linux with Linux::Inotify2
installed, this avoids a storm of inotify_add_watch(2)
and inotify_rm_watch(2) syscalls on SIGHUP with -imapd
and "-extindex --watch"

(cherry picked from commit 10bf54305da8422d9ece6b809996092c1c4b1786)

Note: this seems to fix missed wakeups with many watches,
so it's in the stable branch.

Link: https://public-inbox.org/meta/20201226201115.GA30142@dcvr/
20 months agoinboxidle: clue users into resolving ENOSPC from inotify
Eric Wong [Sat, 26 Dec 2020 05:59:22 +0000 (05:59 +0000)]
inboxidle: clue users into resolving ENOSPC from inotify

It may not be obvious to users a ENOSPC error is from hitting
a (tunable) kernel-imposed limit on inotify watches, and not
some storage device running out of space.  Give them a hint
here to reduce our own support burden.

(cherry picked from commit 5e05c2eb58a450849f1826f3d02ed62b814b6617)

20 months agoinbox: name variable for values loop iterator
Eric Wong [Sat, 26 Dec 2020 08:12:52 +0000 (08:12 +0000)]
inbox: name variable for values loop iterator

->on_inbox_unlock callbacks could clobber $_, and this seems to
fix a problem with -extindex --watch failing to index some
inboxes after SIGHUP reload.

(cherry picked from commit b5e960f50289434025f5904c8c1311e4c8a02b82)

20 months agopublic-inbox-v[12]-format.pod: make lexgrog happy
Uwe Kleine-König [Tue, 22 Dec 2020 17:18:10 +0000 (18:18 +0100)]
public-inbox-v[12]-format.pod: make lexgrog happy

The Debian package linter (lintian) emits the following warning:

W: bad-whatis-entry
N:
N:   A manual page should start with a NAME section, which lists the
N:   program name and a brief description. The NAME section is used to
N:   generate a database that can be queried by commands like apropos and
N:   whatis. You are seeing this tag because lexgrog was unable to parse
N:   the NAME section.
N:
N:   Manual pages for multiple programs, functions, or files should list
N:   each separated by a comma and a space, followed by \- and a common
N:   description.
N:
N:   Listed items may not contain any spaces. A manual page for a two-level
N:   command such as fs listacl must look like fs_listacl so the list is
N:   read correctly.
N:
N:   Refer to the lexgrog(1) manual page, the groff_man(7) manual page, and
N:   the groff_mdoc(7) manual page for details.
N:
N:   Severity: warning
N:
N:   Check: documentation/manual
N:
N:   Renamed from: manpage-has-bad-whatis-entry
N:

for public-inbox-v1-format and public-inbox-v2-format.

Adapt the descriptions to make lexgrog and so lintian happy.

(cherry picked from commit 1350f5ab09f72c75ac2cd6c88f6a2b9e198fef55)

20 months agomanifest.js.gz: fix per-inbox /$INBOX/manifest.js.gz
Eric Wong [Mon, 21 Dec 2020 19:41:21 +0000 (19:41 +0000)]
manifest.js.gz: fix per-inbox /$INBOX/manifest.js.gz

/$INBOX/manifest.js.gz should not attempt to match every inbox
in the domain (or every inbox); that is for /manifest.js.gz
(without a /$INBOX prefix).

Fixes: f303b4add8ea1883 ("wwwlisting: avoid hogging event loop")
(cherry picked from commit 3ce4c38119f13d419bb865a0aa9b66feff339308)

20 months agoFix manpage section of perl module documentation
Uwe Kleine-König [Fri, 18 Dec 2020 11:56:14 +0000 (12:56 +0100)]
Fix manpage section of perl module documentation

On Debian (at least) perl documentation is supposed to be installed in
section 3pm. With the build system hardcoding this to 3 instead this
results in a warning by the Debian package linter:

W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::Git.3.gz:74 3 != 3pm
W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::Import.3.gz:74 3 != 3pm
W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::SaPlugin::ListMirror.3.gz:74 3 != 3pm
W: public-inbox: wrong-manual-section ... use --no-tag-display-limit to see all (or pipe to a file/program)

So use "$(MAN3EXT)" instead of "3".

(cherry picked from commit cd12cf22c4d161b7aec819f45628b983f0bd675d)

20 months agot/psgi_v2: ignore warnings on missing P::M::ReverseProxy
Eric Wong [Wed, 16 Dec 2020 04:39:37 +0000 (04:39 +0000)]
t/psgi_v2: ignore warnings on missing P::M::ReverseProxy

Plack::Test::ExternalServer doesn't depend on
Plack::Middleware::ReverseProxy, so we need to account for
some warnings in stderr if P::M::RP is missing.

(cherry picked from commit d26c2837f479b41182946a6540aad95d34b2b594)

20 months agodaemon: support --daemonize without Net::Server::Daemonize
Eric Wong [Tue, 15 Dec 2020 11:47:16 +0000 (11:47 +0000)]
daemon: support --daemonize without Net::Server::Daemonize

We don't actually need Net::Server::Daemonize to support
the --daemonize flag, since the daemonize() sub provided
by N::S::D doesn't exactly do the things we want.

(cherry picked from commit ab11caeb8ae114087636748df4c956d0e625b237)

20 months agodoc: v2-format: drop repeated word
Kyle Meyer [Sat, 12 Dec 2020 18:24:48 +0000 (18:24 +0000)]
doc: v2-format: drop repeated word

(cherry picked from commit 4df1904582c4d7bec64e05d1566cd48c8c2e888e)

20 months agoover: ensure old, merged {tid} is really gone
Eric Wong [Fri, 4 Dec 2020 12:09:29 +0000 (12:09 +0000)]
over: ensure old, merged {tid} is really gone

We must use the result of link_refs() since it can trigger
merge_threads() and invalidate $old_tid.  In case
merge_threads() isn't triggered, link_refs() will return
$old_tid anyways.

When rethreading and allocating new {tid}, we also must update
the row where the now-expired {tid} came from to ensure only the
new {tid} is seen when reindexing subsequent messages in
history.  Otherwise, every subsequently reindexed+rethreaded
message could end up getting a new {tid}.

Reported-by: Kyle Meyer <kyle@kyleam.com>
Link: https://public-inbox.org/meta/87360nlc44.fsf@kyleam.com/
(cherry picked from commit 9356ec0cc5afc95a8fd398ddf898942ef0acdb74)

20 months agowwwattach: prevent deep-linking via Referer match
Eric Wong [Mon, 23 Nov 2020 14:15:35 +0000 (14:15 +0000)]
wwwattach: prevent deep-linking via Referer match

This prevents `<img src=' tags from being used to deep-link
image attachments from HTML outside of the current host and
reduces potential for abuse.

Some browsers (e.g. Firefox) favor content detection and will
display images irrespective of the Content-Type header being
"application/octet-stream", and "Content-Disposition: attachment"
doesn't stop them, either.

Tested with dillo and Firefox.

Reported-by: Leah Neukirchen <leah@vuxu.org>
(cherry picked from commit 46cbc5a7a4ba917bd7154be3b6e6898420ff85d3)

20 months agot/eml.t: workaround newer Email::MIME* behavior
Eric Wong [Sun, 15 Nov 2020 08:56:09 +0000 (08:56 +0000)]
t/eml.t: workaround newer Email::MIME* behavior

Recent (2020) versions of Email::MIME (and/or dependencies)
have different behavior than historical versions which seem
to be less DWIM and perhaps technically more correct.  We'll
retain historical behavior for now, since it doesn't seem to
cause real problems and DWIM-ness is often required to make
sense of historical mail.

Tested on a FreeBSD 11.4 VM with the following packages:

p5-Email-MIME-1.949
p5-Email-MIME-ContentType-1.024_1
p5-Email-MIME-Encodings-1.315_2

(cherry picked from commit 5c4874309c4dd7aa8b9acd6bb97bd512360ab08a)

20 months agonntp: attempt RFC 5536 3.1.5-conformant Path: headers
Eric Wong [Wed, 4 Nov 2020 10:44:48 +0000 (10:44 +0000)]
nntp: attempt RFC 5536 3.1.5-conformant Path: headers

Perhaps some NNTP clients would be unhappy with the old value
"y".  So use a bit more bandwidth+space to use the server-name
and historical "!not-for-mail" tail-entry to better conform to
a published RFC.

Reported-by: Andrey Melnikov <temnota.am@gmail.com>
(cherry picked from commit 47559005ee678592808c351daa1c9f2682beb248)

20 months agonntp: delimit Newsgroup: header with commas
Eric Wong [Tue, 3 Nov 2020 22:55:59 +0000 (22:55 +0000)]
nntp: delimit Newsgroup: header with commas

...instead of spaces.  This is specified in RFC 5536 3.1.4.

Include references to RFC 1036, 5536 and 5537 in our docs while
we're at it.

Reported-by: Andrey Melnikov <temnota.am@gmail.com>
Link: https://public-inbox.org/meta/CA+PODjpUN5Q4gBFQhAzUNuMasVEdmp9f=8Uo0Ej0mFumdSwi4w@mail.gmail.com/
(cherry picked from commit 44227c2624e4f954943d632cd5335396351373be)

20 months agotls: epollbit: account for miscellaneous OpenSSL errors
Eric Wong [Fri, 30 Oct 2020 02:13:58 +0000 (02:13 +0000)]
tls: epollbit: account for miscellaneous OpenSSL errors

Apparently they happen (triggered by my -imapd instance), so
bail out by closing the underlying socket rather than stopping
the event loop and daemon process.

(cherry picked from commit c51c22c349529d9c377160abcc7961a6ca7b7d5c)

20 months agoscripts/dupe-finder: restore $dbh variable
Kyle Meyer [Fri, 16 Oct 2020 02:08:52 +0000 (22:08 -0400)]
scripts/dupe-finder: restore $dbh variable

When dupe-finder was switched from ->search->{over_ro} to ->over, the
database handle was dropped.  Restore it because a spot downstream
uses it.

Fixes: 73e3a6ed6e95adc6 (use more idiomatic internal API for ->over access)
(cherry picked from commit 0438ee2b1a5fbeb587ee62aa3e267ec09d281bd9)

20 months agosearchidx: index lower-case List-Id value
Eric Wong [Mon, 28 Sep 2020 05:15:19 +0000 (05:15 +0000)]
searchidx: index lower-case List-Id value

We don't want a List-Id value being confused with a Xapian
term prefix, here.

Followup-to: 8b06cda3a3af3f0e ("mda: match List-Id insensitively")
(cherry picked from commit 447d049b70c6723a0de8e28aa7db1a4130394c4b)

20 months agods: add missing label for systems w/o EPOLLEXCLUSIVE
Eric Wong [Sun, 27 Sep 2020 21:26:08 +0000 (21:26 +0000)]
ds: add missing label for systems w/o EPOLLEXCLUSIVE

Oops :x
(cherry picked from commit 28dec3a76687617a3a697058964dc0bd1351eb6b)

20 months agoimap: avoid raising exception if client disconnects
Eric Wong [Sat, 26 Sep 2020 08:08:37 +0000 (08:08 +0000)]
imap: avoid raising exception if client disconnects

This ought to save a few cycles if a client disconnects while
in the middle of a (UID) FETCH.  This avoids:

  Can't call method "git" on an undefined value at .../PublicInbox/IMAP.pm

errors in stderr.

(cherry picked from commit 34880bc83077eac5739deca69f66df7685965064)

20 months agoidxstack: fix comment about file_char
Eric Wong [Tue, 22 Sep 2020 18:49:49 +0000 (18:49 +0000)]
idxstack: fix comment about file_char

It's `d' for deletes, not `a'.

(cherry picked from commit 9dd93e4399c67abf74df6832181d442f81e0aa33)

20 months agomda: match List-Id insensitively
Eric Wong [Mon, 21 Sep 2020 20:58:09 +0000 (20:58 +0000)]
mda: match List-Id insensitively

This follows -watch commit b70473ab8296d31ebb600adb4fa8fe0ac5935ca8
to match List-Id headers case-insensitively.

Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20200921180152.uyqluod7qxbwqubo@chatter.i7.local/
(cherry picked from commit 8b06cda3a3af3f0e88b599a01f03edb7fec4f2c4)

20 months agomid: drop repeated ';' in mid_escape() regular expression
Kyle Meyer [Sun, 20 Sep 2020 18:13:58 +0000 (14:13 -0400)]
mid: drop repeated ';' in mid_escape() regular expression

(cherry picked from commit dc93c36eb62d36e649b9500b7f190687a3fbcffd)

20 months agodoc: post-1.6 updates, start 1.7
Eric Wong [Sat, 19 Sep 2020 21:42:01 +0000 (21:42 +0000)]
doc: post-1.6 updates, start 1.7

I should've dropped "PENDING" notes before the 1.6 release;
they're dropped now, and a note is added to remind my future
self to drop them before 1.7.

(cherry picked from commit 3b5d3d1910f1db526a488142c01f42db5255ac72)

20 months agoconfig: warn on multiple values for some fields
Eric Wong [Sun, 20 Sep 2020 01:43:15 +0000 (01:43 +0000)]
config: warn on multiple values for some fields

Our code doesn't support multi-values for these, and having
unexpected arrays leads to unexpected results (e.g. showing
stuff like "ARRAY(0xDEADBEEFADD12E55)" in user interfaces).  So
warn and only use the last value (matching git-config(1)
behavior without `--get-all').

(cherry picked from commit c8fe4673056571ed251c3a20cb0bbec586c60bd6)

20 months agodoc: txt2pre: more manpage URLs
Eric Wong [Thu, 17 Sep 2020 21:25:22 +0000 (21:25 +0000)]
doc: txt2pre: more manpage URLs

We host our own -imapd manpage, and we started using a few more
git commands (fast-import for ages).  We'll also need to link to
manpages.debian.org and live with long URLs for a few
non-standard manpages in software we reference.

(cherry picked from commit 31065006a3654968c0c4bbfe2d7399a3b326ce18)

20 months agodoc: flow: include -imapd
Eric Wong [Thu, 17 Sep 2020 21:25:21 +0000 (21:25 +0000)]
doc: flow: include -imapd

It's another read-only daemon, and it may see more usage than
-nntpd as more users have IMAP support than NNTP.

(cherry picked from commit 1fcb5694c2fdd4200820f777f27f9f55cd51b6ae)

20 months agot/indexlevels-mirror: fix improperly skipped test
Eric Wong [Wed, 16 Sep 2020 22:17:08 +0000 (22:17 +0000)]
t/indexlevels-mirror: fix improperly skipped test

Oops :x
(cherry picked from commit a7cd8c6a2bfa1ebfe2edeb0d7a93de17fd334683)

2 years agopublic-inbox 1.6.0 v1.6.0
Eric Wong [Wed, 16 Sep 2020 20:02:20 +0000 (20:02 +0000)]
public-inbox 1.6.0

2 years agogit_async_cat: fix outdated comment
Eric Wong [Wed, 16 Sep 2020 07:18:53 +0000 (07:18 +0000)]
git_async_cat: fix outdated comment

We replaced Danga::Socket with PublicInbox::DS roughly a year
before GitAsyncCat was introduced into our git history.

2 years agowwwtext: link to public-inbox.org/meta archives
Eric Wong [Tue, 15 Sep 2020 19:48:50 +0000 (19:48 +0000)]
wwwtext: link to public-inbox.org/meta archives

Since we're advertising our address at meta@public-inbox.org,
we should advertise the archives, too.

2 years agowwwstream: link to cgit URLs for coderepo
Eric Wong [Tue, 15 Sep 2020 20:15:24 +0000 (20:15 +0000)]
wwwstream: link to cgit URLs for coderepo

Hopefully this reduces the ambiguity between code for the
project(s) using public-inbox and the code for public-inbox
itself.

2 years agotreewide: relax allow >=40 chars for git OID
Eric Wong [Tue, 15 Sep 2020 19:51:37 +0000 (19:51 +0000)]
treewide: relax allow >=40 chars for git OID

This will help with eventual git SHA-256 transitions.

2 years agomid: rename MID_MAX to ID_MAX
Eric Wong [Tue, 15 Sep 2020 19:51:36 +0000 (19:51 +0000)]
mid: rename MID_MAX to ID_MAX

It's only used for HTML anchors which we will need indefinitely.

2 years agoimap: quiet uninitialized variable warning on FETCH
Eric Wong [Tue, 15 Sep 2020 06:11:18 +0000 (06:11 +0000)]
imap: quiet uninitialized variable warning on FETCH

This was triggered by blindly trying to FETCH an MSN (not
"UID FETCH") on an empty dummy inbox.  It's harmless, and
probably triggered by a wayward client or misbehaving bot.

2 years agoci/deps: add Plack::Test::ExternalServer for devtest
Eric Wong [Mon, 14 Sep 2020 22:02:23 +0000 (22:02 +0000)]
ci/deps: add Plack::Test::ExternalServer for devtest

More of our Plack tests exercise public-inbox-httpd, nowadays;
and ExternalServer lets us test it easily alongside generic PSGI
stuff.

2 years agot/imapd.t: skip dependent test on failure
Eric Wong [Mon, 14 Sep 2020 21:29:40 +0000 (21:29 +0000)]
t/imapd.t: skip dependent test on failure

We don't want to cascade failures/warnings when something else
breaks.  There's likely more of these to be fixed as we
encounter them.

2 years agodoc: TODO and release notes updates ahead of 1.6
Eric Wong [Mon, 14 Sep 2020 06:39:36 +0000 (06:39 +0000)]
doc: TODO and release notes updates ahead of 1.6

Some more things have happened...

And drop some items which are too expensive to support,
such as automatic mirroring.

2 years agotests: consistently check for xapian-compact
Eric Wong [Mon, 14 Sep 2020 06:29:26 +0000 (06:29 +0000)]
tests: consistently check for xapian-compact

We may need to test against development versions of Xapian,
which may rely on setting `XAPIAN_COMPACT=xapian-compact-1.5'.
Ensure it's possible to do that.

And add a missing check in t/xcpdb-reshard.t, too.

2 years agosigfd: fix typos and scoping on systems w/o epoll+kqueue
Eric Wong [Mon, 14 Sep 2020 03:42:30 +0000 (03:42 +0000)]
sigfd: fix typos and scoping on systems w/o epoll+kqueue

Unfortunately, I'm not sure how easy catching these at
compile-time, is.  Prototypes do not seem to check these
at compile time when crossing packages (not even with
exported subroutines).

2 years agodoc: Add piem to list of clients
Kyle Meyer [Sun, 13 Sep 2020 19:31:55 +0000 (15:31 -0400)]
doc: Add piem to list of clients

2 years agonntp: share more code between art_lookup callers
Eric Wong [Fri, 11 Sep 2020 07:32:33 +0000 (07:32 +0000)]
nntp: share more code between art_lookup callers

This prepares us for future changes to improve scalability to
many inboxes.

2 years agot/nntpd: add test for the XPATH command
Eric Wong [Fri, 11 Sep 2020 07:32:32 +0000 (07:32 +0000)]
t/nntpd: add test for the XPATH command

It's only in RFC 2980 (not 977 or 3977), but Net::NNTP has
supported it since 2001, at least.  We'll be making changes
to avoid pathological behavior, so test it, first.

2 years agotreewide: avoid `goto &NAME' for tail recursion
Eric Wong [Fri, 11 Sep 2020 07:32:31 +0000 (07:32 +0000)]
treewide: avoid `goto &NAME' for tail recursion

While Perl implements tail recursion via `goto' which allows
avoiding warnings on deep recursion.  It doesn't (as of 5.28)
optimize the speed of such dispatches, though it may reduce
ephemeral memory usage.

Make the code less alien to hackers coming from other languages
by using normal subroutine dispatch.  It's actually slightly
faster in micro benchmarks due to the complexity of `goto &NAME'.

2 years agowwwstream: show init + index instructions for -V1, too
Eric Wong [Thu, 10 Sep 2020 05:46:04 +0000 (05:46 +0000)]
wwwstream: show init + index instructions for -V1, too

This should've always been there.  I'm not sure how widely
spread 1.0 and earlier releases were, but we'll keep documenting
the version requirement.

2 years agosolver: async blob retrieval for diff extraction
Eric Wong [Thu, 10 Sep 2020 01:51:53 +0000 (01:51 +0000)]
solver: async blob retrieval for diff extraction

Like the rest of the WWW code, public-inbox-httpd now uses
git_async_cat to retrieve blobs without blocking the event loop.
This improves fairness when git blobs are on slow storage and
allows us to take better advantage of SMP systems.

2 years agosolver: break apart inbox blob retrieval
Eric Wong [Wed, 9 Sep 2020 06:26:18 +0000 (06:26 +0000)]
solver: break apart inbox blob retrieval

To avoid hogging the event loop in public-inbox-httpd when
many candidate messages match, we'll separate the steps to
ensure fairness on slow storage.

2 years agosolver: check one git coderepo and inbox at a time
Eric Wong [Wed, 9 Sep 2020 06:26:17 +0000 (06:26 +0000)]
solver: check one git coderepo and inbox at a time

With public-inbox-httpd, this mitigates the effect of slow git
blob storage with multiple coderepos configured for an inbox.
It's still synchronous for now (and may need to remain that way
for ->last_check_err), but no longer monopolizes the event loop
when checking multiple coderepos.

We don't yet support multi-inbox scanning, yet; but this also
prepares us for a future where we do.

We'll also support >=40 char blob OIDs in preparation for future
git SHA-256 support, too.

2 years agowwwlisting: avoid hogging event loop
Eric Wong [Wed, 9 Sep 2020 06:26:16 +0000 (06:26 +0000)]
wwwlisting: avoid hogging event loop

By using the just-introduced ConfigIter class.
And make ManifestJsGz a subclass of it to reduce duplication.

2 years agoextmsg: prevent cross-inbox matches from hogging event loop
Eric Wong [Wed, 9 Sep 2020 06:26:15 +0000 (06:26 +0000)]
extmsg: prevent cross-inbox matches from hogging event loop

With many inboxes, checking multiple SQLite repos will be slow
and time-consuming, so ensure we can schedule it fairly between
multiple inboxes.

2 years agot/cgi.t: show stderr on failures
Eric Wong [Wed, 9 Sep 2020 06:26:14 +0000 (06:26 +0000)]
t/cgi.t: show stderr on failures

This helped me diagnose an error I would've introduced
in the next commit.

2 years agoconfig: split out iterator into separate object
Eric Wong [Wed, 9 Sep 2020 06:26:13 +0000 (06:26 +0000)]
config: split out iterator into separate object

We will need to allow simultaneous iterators on the same
config object, since we'll need this for ExtMsg, NNTPD,
WwwListing, NewsWWW, and other places.

2 years agoconfig: flatten each_inbox and iterate_start args
Eric Wong [Wed, 9 Sep 2020 06:26:12 +0000 (06:26 +0000)]
config: flatten each_inbox and iterate_start args

In Perl, we can simplify callers by passing a single array
all the way down the stack instead of a single array ref which
needs to be expanded every call.

2 years agowww: manifest.js.gz generation no longer hogs event loop
Eric Wong [Wed, 9 Sep 2020 06:26:11 +0000 (06:26 +0000)]
www: manifest.js.gz generation no longer hogs event loop

It's still as slow as before with hundreds/thousands of inboxes,
but at least it's fair.  Future changes will allow it to be
cached and memoized with persistent HTTP servers.

2 years agouse "\&" where possible when referring to subroutines
Eric Wong [Wed, 9 Sep 2020 06:26:10 +0000 (06:26 +0000)]
use "\&" where possible when referring to subroutines

"*foo" is ambiguous in that it may refer to a bareword file handle;
so we'll use it where we can without triggering warnings.

PublicInbox::TestCommon::run_script_exit required dropping the
prototype, however.  We'll also future-proof by dropping "use
warnings" in Cgit.pm and use the less-ambiguous "//=" in Inbox.pm
while we're in the area.

2 years agosolver: drop warnings, modernize use v5.10.1, use SEEK_SET
Eric Wong [Wed, 9 Sep 2020 06:26:09 +0000 (06:26 +0000)]
solver: drop warnings, modernize use v5.10.1, use SEEK_SET

With Perl upstream preparing to deprecate things, we'll move
towards only enabling warnings during development via shebang
and stop enabling them via "use".

We'll also favor "use v5.10.1" over the Perl 5.6-compatible "use
5.010_001", since our code base never worked on 5.6.

Finally, were also importing SEEK_SET without using it, just use it
for readability since we can't avoid loading Fcntl in other
places and it'll get constant-folded, anyways.

2 years agoxt/solver: test with public-inbox-httpd, too
Eric Wong [Wed, 9 Sep 2020 06:26:08 +0000 (06:26 +0000)]
xt/solver: test with public-inbox-httpd, too

We'll be making changes to solver to make it even fairer
to slow clients on slow storage.  Ensure we test with
public-inbox-httpd-specific codepaths, since the generic
PSGI code paths are rare in production use.

2 years agowwwtext: config comment improvements
Eric Wong [Wed, 9 Sep 2020 21:23:26 +0000 (21:23 +0000)]
wwwtext: config comment improvements

Use the full URL of the inbox being mirrored to reduce ambiguity
(instead of just the inbox name).

Using asymmetric quotes (e.g `foo') improves readability for me
in that it's more obvious when a quote begins and ends.  It also
lights up fewer pixels and reduces visual noise compared to
double-quotes.

We'll also reflow the `mainrepo' vs `inboxdir' comment slightly
to emphasize the word `instead'.

2 years agowwwtext: don't blindly quote "git clone" destination
Eric Wong [Wed, 9 Sep 2020 21:23:25 +0000 (21:23 +0000)]
wwwtext: don't blindly quote "git clone" destination

Save screen space and light up fewer pixels to reduce visual noise.

2 years agowwwtext: describe the use of `coderepo' entries
Eric Wong [Wed, 9 Sep 2020 21:23:24 +0000 (21:23 +0000)]
wwwtext: describe the use of `coderepo' entries

The `solver' feature is not very obvious, give potential
users a hint about it.

2 years agonntp: fix cross-newsgroup Message-ID lookups
Eric Wong [Thu, 10 Sep 2020 09:38:39 +0000 (09:38 +0000)]
nntp: fix cross-newsgroup Message-ID lookups

We cannot blindly use the selected newsgroup for
HEAD/ARTICLE/BODY requests using Message-ID, since
those commands look across all newsgroups; not just
the selected one (if any).

So stuff a reference to the Inbox object into $smsg.
We can reduce args passed into set_nntp_headers() and
msg_hdr_write(), too.

Fixes: 0e6ceff37fc38f28 ("nntp: support slow blob retrievals")
2 years agowwwstream: fix "Atom feed" link
Eric Wong [Wed, 9 Sep 2020 21:38:17 +0000 (21:38 +0000)]
wwwstream: fix "Atom feed" link

Oops, I wanted to stop escaping double-quotes with `qq()' but
used `q()' instead :x

Fixes: 2f61828fcb727e51 ("www: make mirror instructions more prominent")
2 years agocontrib/css: limit <a> coloring to links, only
Eric Wong [Wed, 9 Sep 2020 20:24:55 +0000 (20:24 +0000)]
contrib/css: limit <a> coloring to links, only

We don't want <a> tags without href= attributes to be colored,
since the `<a id=mirror>' tag in the HTML footer is intended
as an anchor destination for `<a href=#mirror>' link at the
top.

2 years agowww: make mirror instructions more prominent
Eric Wong [Tue, 8 Sep 2020 08:29:14 +0000 (08:29 +0000)]
www: make mirror instructions more prominent

In order to fight the misconception that public-inboxes are
centralized, anchor "#mirror" to the clone instructions and
place an emphasis on "mirror", not just cloning.

While we're at it, better describe multi-epoch -V2 inboxes,
since some users do not seem to realize epochs consist of
different data.

2 years agov2writable: reuse read-only shard counting code
Eric Wong [Wed, 2 Sep 2020 11:04:21 +0000 (11:04 +0000)]
v2writable: reuse read-only shard counting code

We'll also fix the read-only code to ensure we notice missing
Xapian shards, since gaps would throw off our expectation that
Xapian document IDs and NNTP article numbers are interchangeable.

2 years agooveridx: document column uses
Eric Wong [Wed, 2 Sep 2020 11:04:20 +0000 (11:04 +0000)]
overidx: document column uses

This may be useful for keeping our heads on straight dealing
with IMAP, NNTP, JMAP, etc.

2 years agowwwaltid: drop unused sqlite3_missing function
Eric Wong [Wed, 2 Sep 2020 11:04:19 +0000 (11:04 +0000)]
wwwaltid: drop unused sqlite3_missing function

It's inlined into the main function, which we'll shorten
slightly with the defined-or (`//') operator.  Also noticed
and fixed a mismatched HTML tag.

2 years agoimap: drop old, pre-Parse::RecDescent search parser
Eric Wong [Wed, 2 Sep 2020 11:04:18 +0000 (11:04 +0000)]
imap: drop old, pre-Parse::RecDescent search parser

We switched to Parse::RecDescent during development and left
some dead code behind.

2 years agosearch: remove {over_ro} field
Eric Wong [Wed, 2 Sep 2020 11:04:17 +0000 (11:04 +0000)]
search: remove {over_ro} field

Only inbox accesses the read-only {over}, now, instead of going
through ->search.  This simplifies our object graph and avoids
potentially redundant FDs and DB handles pointing to the same
over.sqlite3 file.

2 years agosearch: replace ->query with ->mset
Eric Wong [Wed, 2 Sep 2020 11:04:16 +0000 (11:04 +0000)]
search: replace ->query with ->mset

Nearly all of the search uses in the production code rely on
a Xapian mset iterator being returned (instead of an array
of $smsg objects).  So default to returning the mset and move
the burden of smsg array conversion into the test cases.

2 years agotests: add "use strict" and declare v5.10.1 compatibility
Eric Wong [Wed, 2 Sep 2020 11:04:15 +0000 (11:04 +0000)]
tests: add "use strict" and declare v5.10.1 compatibility

strict.pm helped me find a typo in an upcoming recent change, so
ensure we use it since it does more good than harm.  We'll also
take the opportunity here to declare v5.10.1 compatibility level
to future-proof against Perl incompatibilities.

2 years agosearch: remove special case for blank query
Eric Wong [Wed, 2 Sep 2020 11:04:14 +0000 (11:04 +0000)]
search: remove special case for blank query

The special case (if any) belongs at a higher-level,
and this is another step towards removing {over_ro}-dependence
in our Search object.

2 years agouse more idiomatic internal API for ->over access
Eric Wong [Wed, 2 Sep 2020 11:04:13 +0000 (11:04 +0000)]
use more idiomatic internal API for ->over access

{over_ro} being a part of the Search object is a historical
oddity which will go away, soon.  Lets start removing its use in
tests and rarely-used helper scripts.

2 years agodisambiguate OverIdx and Over by field name
Eric Wong [Wed, 2 Sep 2020 11:04:12 +0000 (11:04 +0000)]
disambiguate OverIdx and Over by field name

We'll use {oidx} as the common field name for the read-write
OverIdx, here, to disambiguate it from the read-only {over}
field.  This hopefully makes it clearer which code paths are
read-only and which are read-write.

2 years agomsgmap: note how we use ->created_at
Eric Wong [Wed, 2 Sep 2020 11:04:11 +0000 (11:04 +0000)]
msgmap: note how we use ->created_at

It'll likely be used in the future for JMAP, detached indices,
and maybe other things.

2 years agot/run: Perl future proofing
Eric Wong [Mon, 31 Aug 2020 23:41:56 +0000 (23:41 +0000)]
t/run: Perl future proofing

Bareword file handles outside of STD(IN|OUT|ERR) seem to be on
the chopping block for Perl 8.  We'll also "use v5.10.1" to
guard against future incompatibilities.

2 years agoinit+convert: create non-existing directory hierarchies
Eric Wong [Tue, 1 Sep 2020 01:15:07 +0000 (01:15 +0000)]
init+convert: create non-existing directory hierarchies

Following "git init" as an example, we'll create every parent
path up to the one specified, instead of attempting to continue
on when Cwd::abs_path returns `undef'.

2 years agodoc: remove B<> (bold) markup from the remaining POD
Eric Wong [Tue, 1 Sep 2020 01:15:06 +0000 (01:15 +0000)]
doc: remove B<> (bold) markup from the remaining POD

B<> decreases readability of the POD source and is of dubious
usefulness in the man page.

2 years agowatch: add --help/-h support
Eric Wong [Tue, 1 Sep 2020 01:15:05 +0000 (01:15 +0000)]
watch: add --help/-h support

And avoid unnecessary POD markup in the man page.

2 years agoconfig: use defined-or (//) in a few places
Eric Wong [Tue, 1 Sep 2020 01:15:04 +0000 (01:15 +0000)]
config: use defined-or (//) in a few places

Just some golfing to reduce scrolling and hopefully readability.

2 years agomda+learn: add --help / -h support
Eric Wong [Tue, 1 Sep 2020 01:15:03 +0000 (01:15 +0000)]
mda+learn: add --help / -h support

"use Getopt::Long" doesn't seem too slow on a hot page cache,
and it's probably used frequently enough to be in cache.

We'll also start reducing the amount of markup in the .pod and
favoring verbatim text in documentation for readability in
source form, since the bold text seems excessive.

2 years agodaemon: support --help/-h in -httpd/imapd/nntpd
Eric Wong [Tue, 1 Sep 2020 01:15:02 +0000 (01:15 +0000)]
daemon: support --help/-h in -httpd/imapd/nntpd

For consistency with other commands, though the
protocol-specific options should refer users to
the manpage.

2 years agoscript/*: fold $usage into $help, support `-h' instead of -?
Eric Wong [Tue, 1 Sep 2020 01:15:01 +0000 (01:15 +0000)]
script/*: fold $usage into $help, support `-h' instead of -?

`-h' doesn't conflict with anything, and some users (including
git users) may be more accustomed to using it rather than the
rarely-seen-outside-of-Getopt::Long `-?' switch.

We can also rely on the GetOptions() function to emit a proper
error message instead of just "bad command-line args".

2 years agoedit+purge: support `--help' and `-h' like other commands
Eric Wong [Tue, 1 Sep 2020 01:15:00 +0000 (01:15 +0000)]
edit+purge: support `--help' and `-h' like other commands

And while we're at it, note edit is *destructive* to encourage
reading the fine manual.

2 years agoadmin: improve minimum version text
Eric Wong [Tue, 1 Sep 2020 01:14:59 +0000 (01:14 +0000)]
admin: improve minimum version text

"inboxes 1 inboxes not supported by ..." was non-sensical.
Now it'll show "-V1 inbox not supported by ...", instead.

2 years agoscript/*: set executable bit on -learn and -imapd
Eric Wong [Tue, 1 Sep 2020 01:14:58 +0000 (01:14 +0000)]
script/*: set executable bit on -learn and -imapd

It's useful to mark they're meant to be executable, even
if the shebang is useless.

2 years agot/v2dupindex: test indexing mirrors with duplicate messages
Eric Wong [Tue, 1 Sep 2020 05:55:45 +0000 (05:55 +0000)]
t/v2dupindex: test indexing mirrors with duplicate messages

While it's not a known problem, our deduplicating logic may
change in the future; or a BOFH could be manually injecting
duplicate messages directly into the git epoch repositories.

Ensure indexing in mirrors doesn't break when there's
duplicates.  This is in preparation for detached indices
for multi-inbox search.

2 years agoindex: check for xapian-compact when using --compact
Eric Wong [Tue, 1 Sep 2020 16:54:31 +0000 (16:54 +0000)]
index: check for xapian-compact when using --compact

Otherwise, users may be frustrated to discover it missing
a long indexing run.

2 years agoreplace ParentPipe with EOFpipe
Eric Wong [Mon, 31 Aug 2020 04:41:40 +0000 (04:41 +0000)]
replace ParentPipe with EOFpipe

ParentPipe was a subset of EOFpipe, except EOFpipe correctly
accounts for theoretical(*) spurious wakeups on the pipe.

(*) AFAIK, spurious wakeups are/were more likely on TCP sockets
    due to checksum failures, something that's not a problem on
    local pipes.  We're also not sharing pipes like we do with
    listen sockets on accept(2), so there's no chance of another
    process grabbing bytes (unless we have bugs in our code).

2 years agods: avoid unnecessary timer for waitpid
Eric Wong [Mon, 31 Aug 2020 04:41:39 +0000 (04:41 +0000)]
ds: avoid unnecessary timer for waitpid

It doesn't seem necessary, since we won't call dwaitpid()
until we see an EOF.

2 years agowatch: use EOFpipe to reduce dwaitpid wakeups
Eric Wong [Mon, 31 Aug 2020 04:41:38 +0000 (04:41 +0000)]
watch: use EOFpipe to reduce dwaitpid wakeups

It's a bit inefficient to use a pipe, here.  However, using
dwaitpid() on a process that's not expected to exit soon is
also inefficient as it causes excessive wakeups as most of
our inbox-writing code expects synchronous waitpid().

This only affects -watch instances configured for NNTP and IMAP
clients.

2 years agods: avoid excessive queueing when reaping PIDs
Eric Wong [Mon, 31 Aug 2020 04:41:37 +0000 (04:41 +0000)]
ds: avoid excessive queueing when reaping PIDs

We should not enqueue reap_pids() to run more than once per
EventLoop iteration.  We'll start reformatting reap_pids
to tabs, too, since we're no longer Danga::Socket.

We should also be able to remove timer usage for reaping
down-the-line once we stop abusing dwaitpid() in -watch.

2 years agowatch: comments and tiny cleanups
Eric Wong [Mon, 31 Aug 2020 04:41:36 +0000 (04:41 +0000)]
watch: comments and tiny cleanups

Get rid of an unused variable, prefix a warning and try to
better document control flow around various callbacks.

2 years agowatch: block signals before fork on non-signalfd/kevent systems
Eric Wong [Mon, 31 Aug 2020 04:41:35 +0000 (04:41 +0000)]
watch: block signals before fork on non-signalfd/kevent systems

In case there's non-Linux or BSD users w/o IO::KQueue, we
shouldn't let signal handlers fire in the child processes.

The child processes always assumed signals were blocked by
the parent, so no changes were necessary, there.

2 years agowatch: avoid unnecessary spawning on spam removals
Eric Wong [Mon, 31 Aug 2020 04:41:34 +0000 (04:41 +0000)]
watch: avoid unnecessary spawning on spam removals

This should further mitigate lock contention problems
when -watch is configured to watch on a Maildir for spam
while performing a large NNTP import.

There is now a small risk a message won't get removed because if
it's in the current (uncommitted) fast-import batch, but
unlikely given the batch size is now only 10 messages.

If a that small window is hit, flipping the \Seen flag
(e.g. marking it unread, and then read again) will trigger
another removal attempt via IMAP or Maildir.

2 years agowatch: log signal activities to STDERR
Eric Wong [Mon, 31 Aug 2020 04:41:33 +0000 (04:41 +0000)]
watch: log signal activities to STDERR

Sometimes it may not be apparent when/if a signal is
processed, this hopefully improves the situation.

We'll also change the process title when we're quitting
to better inform users.

2 years agorename WatchMaildir => Watch
Eric Wong [Mon, 31 Aug 2020 04:41:32 +0000 (04:41 +0000)]
rename WatchMaildir => Watch

This is no longer limited to Maildirs now that IMAP and NNTP
support exist; so give it a shorter name.