Eric Wong [Sun, 20 Sep 2020 01:43:15 +0000 (01:43 +0000)]
config: warn on multiple values for some fields
Our code doesn't support multi-values for these, and having
unexpected arrays leads to unexpected results (e.g. showing
stuff like "ARRAY(0xDEADBEEFADD12E55)" in user interfaces). So
warn and only use the last value (matching git-config(1)
behavior without `--get-all').
Eric Wong [Sat, 19 Sep 2020 09:37:14 +0000 (09:37 +0000)]
gcf2: wire up read-only daemons and rm -gcf2 script
It seems easiest to have a singleton Gcf2Client client object
per daemon worker for all inboxes to use. This reduces overall
FD usage from pipes.
The `public-inbox-gcf2' command + manpage are gone and a `$^X'
one-liner is used, instead. This saves inodes for internal
commands and hopefully makes it easier to avoid mismatched
PERL5LIB include paths (as noticed during development :x).
We'll also make the existing cat-file process management
infrastructure more resilient to BOFHs on process killing
sprees (or in case our libgit2-based code fails on us).
(Rare) PublicInbox::WWW PSGI users NOT using public-inbox-httpd
won't automatically benefit from this change, and extra
configuration will be required (to be documented later).
Eric Wong [Sat, 19 Sep 2020 09:37:11 +0000 (09:37 +0000)]
gcf2: transparently retry on missing OID
Since we only get OIDs from trusted local data sources
(over.sqlite3), we can safely retry within the -gcf2 process
without worry about clients spamming us with requests for
invalid OIDs and triggering reopens.
Eric Wong [Sat, 19 Sep 2020 09:37:09 +0000 (09:37 +0000)]
t/gcf2: test changes to alternates
Calling ->add_alternate won't pick up new additions to
$OBJDIR/info/alternates, unfornately. Thus v2 inboxes will
need to do something to invalidate Gcf2 objects.
Eric Wong [Sat, 19 Sep 2020 09:37:08 +0000 (09:37 +0000)]
gcf2: libgit2-based git cat-file alternative
Having tens of thousands of inboxes and associated git processes
won't work well, so we'll use libgit2 to access the object DB
directly. We only care about OID lookups and won't need to rely
on per-repo revision names or paths.
The Git::Raw XS package won't be used since its manpages don't
promise a stable API. Since we already use Inline::C and have
experience with I::C when it comes to compatibility, this only
introduces libgit2 itself as a source of new incompatibilities.
This also provides an excuse for me to writev(2) to reduce
syscalls, but liburing is on the horizon for next year.
Eric Wong [Thu, 17 Sep 2020 08:57:51 +0000 (08:57 +0000)]
git_async_cat: inline + drop redundant batch_prepare call
$git->cat_async already calls $git->batch_prepare iff needed, so
we can reduce subroutine calls and inline a one-off subroutine
to save some memory, here.
Eric Wong [Thu, 17 Sep 2020 21:25:22 +0000 (21:25 +0000)]
doc: txt2pre: more manpage URLs
We host our own -imapd manpage, and we started using a few more
git commands (fast-import for ages). We'll also need to link to
manpages.debian.org and live with long URLs for a few
non-standard manpages in software we reference.
Eric Wong [Tue, 15 Sep 2020 06:11:18 +0000 (06:11 +0000)]
imap: quiet uninitialized variable warning on FETCH
This was triggered by blindly trying to FETCH an MSN (not
"UID FETCH") on an empty dummy inbox. It's harmless, and
probably triggered by a wayward client or misbehaving bot.
Eric Wong [Mon, 14 Sep 2020 06:29:26 +0000 (06:29 +0000)]
tests: consistently check for xapian-compact
We may need to test against development versions of Xapian,
which may rely on setting `XAPIAN_COMPACT=xapian-compact-1.5'.
Ensure it's possible to do that.
And add a missing check in t/xcpdb-reshard.t, too.
Eric Wong [Mon, 14 Sep 2020 03:42:30 +0000 (03:42 +0000)]
sigfd: fix typos and scoping on systems w/o epoll+kqueue
Unfortunately, I'm not sure how easy catching these at
compile-time, is. Prototypes do not seem to check these
at compile time when crossing packages (not even with
exported subroutines).
Eric Wong [Fri, 11 Sep 2020 07:32:32 +0000 (07:32 +0000)]
t/nntpd: add test for the XPATH command
It's only in RFC 2980 (not 977 or 3977), but Net::NNTP has
supported it since 2001, at least. We'll be making changes
to avoid pathological behavior, so test it, first.
Eric Wong [Fri, 11 Sep 2020 07:32:31 +0000 (07:32 +0000)]
treewide: avoid `goto &NAME' for tail recursion
While Perl implements tail recursion via `goto' which allows
avoiding warnings on deep recursion. It doesn't (as of 5.28)
optimize the speed of such dispatches, though it may reduce
ephemeral memory usage.
Make the code less alien to hackers coming from other languages
by using normal subroutine dispatch. It's actually slightly
faster in micro benchmarks due to the complexity of `goto &NAME'.
Eric Wong [Thu, 10 Sep 2020 01:51:53 +0000 (01:51 +0000)]
solver: async blob retrieval for diff extraction
Like the rest of the WWW code, public-inbox-httpd now uses
git_async_cat to retrieve blobs without blocking the event loop.
This improves fairness when git blobs are on slow storage and
allows us to take better advantage of SMP systems.
Eric Wong [Wed, 9 Sep 2020 06:26:17 +0000 (06:26 +0000)]
solver: check one git coderepo and inbox at a time
With public-inbox-httpd, this mitigates the effect of slow git
blob storage with multiple coderepos configured for an inbox.
It's still synchronous for now (and may need to remain that way
for ->last_check_err), but no longer monopolizes the event loop
when checking multiple coderepos.
We don't yet support multi-inbox scanning, yet; but this also
prepares us for a future where we do.
We'll also support >=40 char blob OIDs in preparation for future
git SHA-256 support, too.
Eric Wong [Wed, 9 Sep 2020 06:26:13 +0000 (06:26 +0000)]
config: split out iterator into separate object
We will need to allow simultaneous iterators on the same
config object, since we'll need this for ExtMsg, NNTPD,
WwwListing, NewsWWW, and other places.
Eric Wong [Wed, 9 Sep 2020 06:26:12 +0000 (06:26 +0000)]
config: flatten each_inbox and iterate_start args
In Perl, we can simplify callers by passing a single array
all the way down the stack instead of a single array ref which
needs to be expanded every call.
Eric Wong [Wed, 9 Sep 2020 06:26:11 +0000 (06:26 +0000)]
www: manifest.js.gz generation no longer hogs event loop
It's still as slow as before with hundreds/thousands of inboxes,
but at least it's fair. Future changes will allow it to be
cached and memoized with persistent HTTP servers.
Eric Wong [Wed, 9 Sep 2020 06:26:10 +0000 (06:26 +0000)]
use "\&" where possible when referring to subroutines
"*foo" is ambiguous in that it may refer to a bareword file handle;
so we'll use it where we can without triggering warnings.
PublicInbox::TestCommon::run_script_exit required dropping the
prototype, however. We'll also future-proof by dropping "use
warnings" in Cgit.pm and use the less-ambiguous "//=" in Inbox.pm
while we're in the area.
Eric Wong [Wed, 9 Sep 2020 06:26:09 +0000 (06:26 +0000)]
solver: drop warnings, modernize use v5.10.1, use SEEK_SET
With Perl upstream preparing to deprecate things, we'll move
towards only enabling warnings during development via shebang
and stop enabling them via "use".
We'll also favor "use v5.10.1" over the Perl 5.6-compatible "use
5.010_001", since our code base never worked on 5.6.
Finally, were also importing SEEK_SET without using it, just use it
for readability since we can't avoid loading Fcntl in other
places and it'll get constant-folded, anyways.
Eric Wong [Wed, 9 Sep 2020 06:26:08 +0000 (06:26 +0000)]
xt/solver: test with public-inbox-httpd, too
We'll be making changes to solver to make it even fairer
to slow clients on slow storage. Ensure we test with
public-inbox-httpd-specific codepaths, since the generic
PSGI code paths are rare in production use.
Eric Wong [Wed, 9 Sep 2020 21:23:26 +0000 (21:23 +0000)]
wwwtext: config comment improvements
Use the full URL of the inbox being mirrored to reduce ambiguity
(instead of just the inbox name).
Using asymmetric quotes (e.g `foo') improves readability for me
in that it's more obvious when a quote begins and ends. It also
lights up fewer pixels and reduces visual noise compared to
double-quotes.
We'll also reflow the `mainrepo' vs `inboxdir' comment slightly
to emphasize the word `instead'.
Eric Wong [Thu, 10 Sep 2020 09:38:39 +0000 (09:38 +0000)]
nntp: fix cross-newsgroup Message-ID lookups
We cannot blindly use the selected newsgroup for
HEAD/ARTICLE/BODY requests using Message-ID, since
those commands look across all newsgroups; not just
the selected one (if any).
So stuff a reference to the Inbox object into $smsg.
We can reduce args passed into set_nntp_headers() and
msg_hdr_write(), too.
Fixes: 0e6ceff37fc38f28 ("nntp: support slow blob retrievals")
Eric Wong [Wed, 9 Sep 2020 20:24:55 +0000 (20:24 +0000)]
contrib/css: limit <a> coloring to links, only
We don't want <a> tags without href= attributes to be colored,
since the `<a id=mirror>' tag in the HTML footer is intended
as an anchor destination for `<a href=#mirror>' link at the
top.
Eric Wong [Tue, 8 Sep 2020 08:29:14 +0000 (08:29 +0000)]
www: make mirror instructions more prominent
In order to fight the misconception that public-inboxes are
centralized, anchor "#mirror" to the clone instructions and
place an emphasis on "mirror", not just cloning.
While we're at it, better describe multi-epoch -V2 inboxes,
since some users do not seem to realize epochs consist of
different data.
Eric Wong [Wed, 2 Sep 2020 11:04:21 +0000 (11:04 +0000)]
v2writable: reuse read-only shard counting code
We'll also fix the read-only code to ensure we notice missing
Xapian shards, since gaps would throw off our expectation that
Xapian document IDs and NNTP article numbers are interchangeable.
Eric Wong [Wed, 2 Sep 2020 11:04:17 +0000 (11:04 +0000)]
search: remove {over_ro} field
Only inbox accesses the read-only {over}, now, instead of going
through ->search. This simplifies our object graph and avoids
potentially redundant FDs and DB handles pointing to the same
over.sqlite3 file.
Eric Wong [Wed, 2 Sep 2020 11:04:16 +0000 (11:04 +0000)]
search: replace ->query with ->mset
Nearly all of the search uses in the production code rely on
a Xapian mset iterator being returned (instead of an array
of $smsg objects). So default to returning the mset and move
the burden of smsg array conversion into the test cases.
Eric Wong [Wed, 2 Sep 2020 11:04:15 +0000 (11:04 +0000)]
tests: add "use strict" and declare v5.10.1 compatibility
strict.pm helped me find a typo in an upcoming recent change, so
ensure we use it since it does more good than harm. We'll also
take the opportunity here to declare v5.10.1 compatibility level
to future-proof against Perl incompatibilities.
Eric Wong [Wed, 2 Sep 2020 11:04:13 +0000 (11:04 +0000)]
use more idiomatic internal API for ->over access
{over_ro} being a part of the Search object is a historical
oddity which will go away, soon. Lets start removing its use in
tests and rarely-used helper scripts.
Eric Wong [Wed, 2 Sep 2020 11:04:12 +0000 (11:04 +0000)]
disambiguate OverIdx and Over by field name
We'll use {oidx} as the common field name for the read-write
OverIdx, here, to disambiguate it from the read-only {over}
field. This hopefully makes it clearer which code paths are
read-only and which are read-write.
Eric Wong [Mon, 31 Aug 2020 23:41:56 +0000 (23:41 +0000)]
t/run: Perl future proofing
Bareword file handles outside of STD(IN|OUT|ERR) seem to be on
the chopping block for Perl 8. We'll also "use v5.10.1" to
guard against future incompatibilities.
Following "git init" as an example, we'll create every parent
path up to the one specified, instead of attempting to continue
on when Cwd::abs_path returns `undef'.
Eric Wong [Tue, 1 Sep 2020 01:15:03 +0000 (01:15 +0000)]
mda+learn: add --help / -h support
"use Getopt::Long" doesn't seem too slow on a hot page cache,
and it's probably used frequently enough to be in cache.
We'll also start reducing the amount of markup in the .pod and
favoring verbatim text in documentation for readability in
source form, since the bold text seems excessive.
Eric Wong [Tue, 1 Sep 2020 01:15:01 +0000 (01:15 +0000)]
script/*: fold $usage into $help, support `-h' instead of -?
`-h' doesn't conflict with anything, and some users (including
git users) may be more accustomed to using it rather than the
rarely-seen-outside-of-Getopt::Long `-?' switch.
We can also rely on the GetOptions() function to emit a proper
error message instead of just "bad command-line args".
Eric Wong [Tue, 1 Sep 2020 05:55:45 +0000 (05:55 +0000)]
t/v2dupindex: test indexing mirrors with duplicate messages
While it's not a known problem, our deduplicating logic may
change in the future; or a BOFH could be manually injecting
duplicate messages directly into the git epoch repositories.
Ensure indexing in mirrors doesn't break when there's
duplicates. This is in preparation for detached indices
for multi-inbox search.
Eric Wong [Mon, 31 Aug 2020 04:41:40 +0000 (04:41 +0000)]
replace ParentPipe with EOFpipe
ParentPipe was a subset of EOFpipe, except EOFpipe correctly
accounts for theoretical(*) spurious wakeups on the pipe.
(*) AFAIK, spurious wakeups are/were more likely on TCP sockets
due to checksum failures, something that's not a problem on
local pipes. We're also not sharing pipes like we do with
listen sockets on accept(2), so there's no chance of another
process grabbing bytes (unless we have bugs in our code).
Eric Wong [Mon, 31 Aug 2020 04:41:38 +0000 (04:41 +0000)]
watch: use EOFpipe to reduce dwaitpid wakeups
It's a bit inefficient to use a pipe, here. However, using
dwaitpid() on a process that's not expected to exit soon is
also inefficient as it causes excessive wakeups as most of
our inbox-writing code expects synchronous waitpid().
This only affects -watch instances configured for NNTP and IMAP
clients.
Eric Wong [Mon, 31 Aug 2020 04:41:37 +0000 (04:41 +0000)]
ds: avoid excessive queueing when reaping PIDs
We should not enqueue reap_pids() to run more than once per
EventLoop iteration. We'll start reformatting reap_pids
to tabs, too, since we're no longer Danga::Socket.
We should also be able to remove timer usage for reaping
down-the-line once we stop abusing dwaitpid() in -watch.
Eric Wong [Mon, 31 Aug 2020 04:41:34 +0000 (04:41 +0000)]
watch: avoid unnecessary spawning on spam removals
This should further mitigate lock contention problems
when -watch is configured to watch on a Maildir for spam
while performing a large NNTP import.
There is now a small risk a message won't get removed because if
it's in the current (uncommitted) fast-import batch, but
unlikely given the batch size is now only 10 messages.
If a that small window is hit, flipping the \Seen flag
(e.g. marking it unread, and then read again) will trigger
another removal attempt via IMAP or Maildir.
Eric Wong [Mon, 31 Aug 2020 04:41:31 +0000 (04:41 +0000)]
watchmaildir: use v5.10.1, drop warnings
Declare 5.10.1 to avoid potential compatibility problems with
Perl 7/8 down the line. We'll rely on the command-line to set
or drop warnings during development, at least.
Eric Wong [Mon, 31 Aug 2020 04:41:30 +0000 (04:41 +0000)]
watch: limit batch size of NNTP and IMAP workers, too
We don't want to monopolize locks because processes can easily
block each other if using `watchspam' on a Maildir while a big
NNTP or IMAP import is happening.
This can also happen if somebody configured a single inbox to
watch from several sources to merge several mailboxes into one
(e.g. both an IMAP and Maildir are watched).
Eric Wong [Fri, 28 Aug 2020 10:12:59 +0000 (10:12 +0000)]
imaptracker: update_last: simplify callers
By making it a no-op if last_uid is not defined. This isn't a
hot code path, so the extra method dispatch isn't an issue.
It'll save some indentation/wrapping in future commits.
Eric Wong [Fri, 28 Aug 2020 10:12:58 +0000 (10:12 +0000)]
watch: flush changes to inbox before updating IMAPTracker
Data needs to hit inboxes, first. Otherwise it's possible to
skip messages in case git-fast-import is killed before it sees
"done\n". Now, -watch will just waste a little bandwidth in
re-downloading a seen message if it's interrupted immediately
before updating IMAPTracker.
Eric Wong [Fri, 28 Aug 2020 04:22:00 +0000 (04:22 +0000)]
Makefile.PL: run check-man for <= 80 columns on check-run, too
I mostly use "make check-run" instead of the slower "make check"
target, nowadays, so add this check to ensure the rendered
manpage is always be visible to more users who need big fonts.
Eric Wong [Thu, 27 Aug 2020 22:05:00 +0000 (22:05 +0000)]
www: more descriptive pagination
Being an easily confused person, I find "next" and "prev"
ambiguous as to whether messages on the next or previous page
will be newer or older than the current page. Clarify that for
the threaded /$INBOX/ view and search results.
For search results sorted by relevance, we'll use "[>= $SCORE]"
or "[<= $SCORE]" to indicate to indicate directionality.
This also fixes $INBOX/new.html for unindexed v1 inboxes.
Eric Wong [Thu, 27 Aug 2020 22:04:59 +0000 (22:04 +0000)]
www: improve navigation around contemporary threads
Sometimes it's useful to quickly get to threads and messages
which are contemporaries of the current thread/message being
focused on. This hopefully improves navigation by making:
a) the top line (where $INBOX_DIR/description) is shown
a link to the latest topics in search results and
per-thread/per-message views.
b) providing a link to contemporaries ("~YYYY-MM-DD") at
around the thread overview skeleton area for per-thread
and per-message views
Eric Wong [Thu, 27 Aug 2020 12:17:05 +0000 (12:17 +0000)]
doc: move watch config docs to -watch manpage
The -config manpage is a bit long and the -watch stuff is
isolated from the rest of it while we start documenting NNTP and
IMAP support.
I'm not entirely happy with the way IMAP and NNTP are
configured, it's still good enough for small setups.
This also fixes a long-standing misplaced comment about
`publicinboxwatch.spamcheck' affecting all configured inboxes,
that comment was actually for `publicinboxwatch.watchspam'.
We'll omit documenting NNTP for `watchspam', for now, given the
lack of \Seen flags in NNTP and I'm not sure if it's even
useful. There may not be any newsgroups for sharing confirmed
spam, either...
Eric Wong [Thu, 27 Aug 2020 12:17:04 +0000 (12:17 +0000)]
watch: imap: only remove \Seen spam
This matches the behavior of Maildir `watchspam' handling in not
removing unseen messages. NNTP can't match this behavior, since
NNTP servers don't store flags, clients do.