]> Sergey Matveev's repositories - public-inbox.git/log
public-inbox.git
3 years agoextindex: remove skip-docdata option
Eric Wong [Wed, 18 Nov 2020 08:22:27 +0000 (08:22 +0000)]
extindex: remove skip-docdata option

Since extindex is entirely new, it doesn't have backwards
compatibility concerns and never stored docdata, anyways.

3 years agov2writable: avoid initiating leftover unindex if interrupted
Eric Wong [Sun, 15 Nov 2020 23:27:23 +0000 (23:27 +0000)]
v2writable: avoid initiating leftover unindex if interrupted

We can also avoid a needless progress message on log2stack
interruptions, too.

3 years agosearchidx: check for graceful shutdown in log2stack
Eric Wong [Sun, 15 Nov 2020 23:20:17 +0000 (23:20 +0000)]
searchidx: check for graceful shutdown in log2stack

The initial "git log" invocation for a git epoch can be time
consuming, so check for graceful shutdown at each line to ensure
timely shutdowns and avoid SSD/HDD wear.

3 years agot/eml.t: workaround newer Email::MIME* behavior
Eric Wong [Sun, 15 Nov 2020 08:56:09 +0000 (08:56 +0000)]
t/eml.t: workaround newer Email::MIME* behavior

Recent (2020) versions of Email::MIME (and/or dependencies)
have different behavior than historical versions which seem
to be less DWIM and perhaps technically more correct.  We'll
retain historical behavior for now, since it doesn't seem to
cause real problems and DWIM-ness is often required to make
sense of historical mail.

Tested on a FreeBSD 11.4 VM with the following packages:

p5-Email-MIME-1.949
p5-Email-MIME-ContentType-1.024_1
p5-Email-MIME-Encodings-1.315_2

3 years agoextindex: support graceful shutdown via QUIT/INT/TERM
Eric Wong [Fri, 13 Nov 2020 11:11:44 +0000 (11:11 +0000)]
extindex: support graceful shutdown via QUIT/INT/TERM

Just like the daemon processes, -extindex now supports graceful
shutdown via the same signals.  This lets users avoid having to
repeat indexing messages when a power outage strikes during a
long (multi-hour/day) indexing run.

Per-inbox (v1/v2) -index graceful shutdowns are not supported,
yet, but is planned for later.

3 years ago*index: discard sync->{todo} on iteration
Eric Wong [Fri, 13 Nov 2020 11:11:43 +0000 (11:11 +0000)]
*index: discard sync->{todo} on iteration

There's no need to continuously append to {todo} when indexing
multiple inboxes.  They're not redundantly indexed (because the
IdxStack is discarded, making it a noop), but it's still a waste
of memory keeping the $unit hashrefs around.

3 years ago*index: avoid per-epoch --batch-check processes
Eric Wong [Fri, 13 Nov 2020 11:11:42 +0000 (11:11 +0000)]
*index: avoid per-epoch --batch-check processes

Since all.git (v2) and ALL.git (extindex) encompass every single
epoch or indexed inbox; and is_ancestor() only uses hexadecimal
OIDs; there is no good reason to use $unit->{git} for an
epoch-local $git->check.

This prevents dozens/hundreds of --batch-check processes from
being left running after indexing and can improve locality
if size checks are being done (since that uses --batch-check,
too).

Theoretically several epochs may have conflicting OIDs, but
we're screwed in those cases, anyways, so we might as well
detect it earlier (though I'm not sure what the behavior would
be :x).

3 years ago*index: checkpoints write last_commit metadata
Eric Wong [Fri, 13 Nov 2020 11:11:41 +0000 (11:11 +0000)]
*index: checkpoints write last_commit metadata

This will set us up for supporting graceful shutdown
on -index without repeating any work.

3 years agosearchidx: fix fallback on unindex miss
Eric Wong [Tue, 10 Nov 2020 03:20:29 +0000 (03:20 +0000)]
searchidx: fix fallback on unindex miss

In case of other bugs or intentional corruption of over.sqlite3,
we don't want to attempt dereferencing a non-ref scalar when
calling ->mid_delete in the fallback code path.

Noticed while chasing another bug in extindex development...

3 years agoextindex: fix --batch-size support
Eric Wong [Sat, 7 Nov 2020 10:57:00 +0000 (10:57 +0000)]
extindex: fix --batch-size support

Calling PublicInbox::Admin::index_prepare is required for
--batch-size (k|m|g) modifiiers and indexBatchSize in the config
file.  Otherwise, the default 1m batch size stuck and led
to unexpectedly bad performance on a machine which could index
v2 inboxes faster with larger batch sizes.

3 years agoextindex: SIGUSR1 supports checkpoint
Eric Wong [Sat, 7 Nov 2020 10:56:59 +0000 (10:56 +0000)]
extindex: SIGUSR1 supports checkpoint

Matching the behavior of git-fast-import(1), we'll allow a user
to send SIGUSR1 to checkpoint over.sqlite3 and Xapian.

3 years agov2writable: more accurate {current_info} warnings/progress
Eric Wong [Sat, 7 Nov 2020 10:56:58 +0000 (10:56 +0000)]
v2writable: more accurate {current_info} warnings/progress

With async git blob retrievals, the OID being enqueued and the
OID being processed can be totally unrelated and misleading.

We'll also prefix $INBOX_DIR for v2, and not just the epoch
since we could be indexing multiple inboxes via both -index
and -extindex.

3 years agoextsearch: canonicalize topdir
Eric Wong [Sat, 7 Nov 2020 10:56:57 +0000 (10:56 +0000)]
extsearch: canonicalize topdir

This makes `ps' output look a bit nicer if there's trailing
slashes involved from the command-line.

3 years agoextsearchidx: quiet warning for unindexed `d' messages
Eric Wong [Sat, 7 Nov 2020 10:56:56 +0000 (10:56 +0000)]
extsearchidx: quiet warning for unindexed `d' messages

"deleted" messages (via -learn <spam|rm>) in the source inboxes
are likely to already be unindexed, so avoid triggering needless
warnings about the spam message being missing.

3 years agov2writable: less expensive checkpoint for extindex
Eric Wong [Sat, 7 Nov 2020 10:56:55 +0000 (10:56 +0000)]
v2writable: less expensive checkpoint for extindex

Since extindex holds no locks on parallel inbox writers,
we can simply use "barrier" IPC shard commands to checkpoint
and avoid respawning shard or git processes.

3 years agosearchidxshard: further improve {current_info} readability
Eric Wong [Sat, 7 Nov 2020 10:56:54 +0000 (10:56 +0000)]
searchidxshard: further improve {current_info} readability

Add a space after \0 to visually disambiguate it from the
{bytes} field.

3 years agosearchidxshard: reduce syscalls when writing ->eidx_key
Eric Wong [Sat, 7 Nov 2020 10:56:53 +0000 (10:56 +0000)]
searchidxshard: reduce syscalls when writing ->eidx_key

We use ->autoflush(1) on this pipe to ensure the shard workers
see data immediately on print; so this means we have to do our
own buffering for optional data.

3 years agoextsearchidx: avoid needless alternates rewrite in ALL.git
Eric Wong [Sat, 7 Nov 2020 10:56:52 +0000 (10:56 +0000)]
extsearchidx: avoid needless alternates rewrite in ALL.git

As with fill_alternates in V2Writable, we do not need to update
$GIT_DIR/objects/info/alternates if nothing is changed.

3 years agoextsearch: rename -eindex to -extindex
Eric Wong [Sat, 7 Nov 2020 10:56:51 +0000 (10:56 +0000)]
extsearch: rename -eindex to -extindex

Upon "eindex" rhymes with "reindex", which could be confusing;
so name the command and config prefix to use "extindex" which
is hopefully less confusing.

3 years agosearchidxshard: make warnings with eidx_key less confusing
Eric Wong [Tue, 27 Oct 2020 07:54:53 +0000 (07:54 +0000)]
searchidxshard: make warnings with eidx_key less confusing

Seeing "Xorg.foo.bar" can be confusing in warnings if the
eidx_key is only "org.foo.bar" with no relation to "Xorg" at
all.  Furthermore, printing "\0" to log or terminal output isn't
very nice and could throw off some users/tools.

3 years agoextsearchidx: support --batch-size checkpoints
Eric Wong [Tue, 27 Oct 2020 07:54:52 +0000 (07:54 +0000)]
extsearchidx: support --batch-size checkpoints

This is needed to limit the RSS of processes and ensure the
stored data in over.sqlite3 and Xapian DBs are consistent if
interrupted.  Without checkpoints, indexing lore causes shard
workers to take several GB of memory and thrash/OOM smaller
systems.

3 years agoextsearchidx: set current_info in warning callbacks
Eric Wong [Tue, 27 Oct 2020 07:54:51 +0000 (07:54 +0000)]
extsearchidx: set current_info in warning callbacks

This bit is duplicated with per-Inbox indexing in Admin,
undecided if it's the right place for it.

3 years agosearchidx: ignore exceptions from ->remove_term
Eric Wong [Tue, 27 Oct 2020 07:54:50 +0000 (07:54 +0000)]
searchidx: ignore exceptions from ->remove_term

This seems necessary for some cross-posted messages (and we did
it historically before we used over.sqlite3).

3 years agoextsearch: wire up remaining Inbox-like methods for WWW
Eric Wong [Tue, 27 Oct 2020 07:54:49 +0000 (07:54 +0000)]
extsearch: wire up remaining Inbox-like methods for WWW

This lets us pretend an ExtSearch object is an Inbox object
in most of the existing WWW code.

3 years agoextsearchidx: handle edits
Eric Wong [Tue, 27 Oct 2020 07:54:48 +0000 (07:54 +0000)]
extsearchidx: handle edits

We can now handle cases where messages are edited in one inbox
but not another, bifurcating the message.

V2Writable::log_range handles some edge-cases which could happen
in v2-only code paths, as well, but weren't usually triggered
due to default git-gc knobs not pruning immediately

3 years agoextsearch: wire up smsg_eml
Eric Wong [Tue, 27 Oct 2020 07:54:47 +0000 (07:54 +0000)]
extsearch: wire up smsg_eml

We'll probably still need synchronous message retrieval
in a few places (tests, at least).

3 years agot/v2writable: remove pointless ->barrier call
Eric Wong [Tue, 27 Oct 2020 07:54:46 +0000 (07:54 +0000)]
t/v2writable: remove pointless ->barrier call

We don't actually use it anywhere, and may not need it in
the future.

3 years agot/extsearch.t: verify results and xref3 ordering
Eric Wong [Tue, 27 Oct 2020 07:54:45 +0000 (07:54 +0000)]
t/extsearch.t: verify results and xref3 ordering

We want NNTP clients to see consistent Xref: headers to ensure
client-side caches don't get confused.

3 years agosearchidx: remove xref3 support for Xapian
Eric Wong [Tue, 27 Oct 2020 07:54:44 +0000 (07:54 +0000)]
searchidx: remove xref3 support for Xapian

It doesn't seem worth storing xref3 data in Xapian now that
the same info is in over.sqlite3.

3 years agoover: store xref3 data in over.sqlite3
Eric Wong [Tue, 27 Oct 2020 07:54:43 +0000 (07:54 +0000)]
over: store xref3 data in over.sqlite3

We may not end up storing xref3 data in Xapian, actually.
This will make indexlevel=basic possible, and along with
--sequential-shard indexing support for slow storage.

Making oidmap a separate table seems unnecessary, too, so
fold it into the xref3 table since it's unlikely a git blob
will be responsible for multiple xref3 rows.

3 years agoindex: eindex wiring
Eric Wong [Tue, 27 Oct 2020 07:54:42 +0000 (07:54 +0000)]
index: eindex wiring

This doesn't do anything, yet, but it will once the rest
of the eindex stuff works.

3 years agoscript: add preliminary eindex implementation
Eric Wong [Tue, 27 Oct 2020 07:54:41 +0000 (07:54 +0000)]
script: add preliminary eindex implementation

Not documented, yet, but it runs...

3 years agoMakefile.PL: do not build manpage if POD is missing
Eric Wong [Tue, 27 Oct 2020 07:54:40 +0000 (07:54 +0000)]
Makefile.PL: do not build manpage if POD is missing

But warn on it, this lets us test new or throwaway commands more
easily if we don't have to start a new POD for everything we
want to dump in script/.

3 years agosearchidx: favor $sync->{ibx} (over $self->{ibx})
Eric Wong [Tue, 27 Oct 2020 07:54:39 +0000 (07:54 +0000)]
searchidx: favor $sync->{ibx} (over $self->{ibx})

In case we want to reuse code with ExtSearchIdx or V2Writable.

3 years agosearchidx: reduce inbox-dependency, wrap ->with_umask
Eric Wong [Tue, 27 Oct 2020 07:54:38 +0000 (07:54 +0000)]
searchidx: reduce inbox-dependency, wrap ->with_umask

This will let us work consistently with both existing inboxes
and external indices.

3 years agoextsearchidx: sync updates
Eric Wong [Tue, 27 Oct 2020 07:54:37 +0000 (07:54 +0000)]
extsearchidx: sync updates

A couple of more things to prepare us to run syncs on
both v1 and v2 inboxes.

3 years agosearchidx: export prepare_stack
Eric Wong [Tue, 27 Oct 2020 07:54:36 +0000 (07:54 +0000)]
searchidx: export prepare_stack

We'll be needing it in ExtSearchIdx for the next commit.

3 years agoextsearchidx: sync unit updates
Eric Wong [Tue, 27 Oct 2020 07:54:35 +0000 (07:54 +0000)]
extsearchidx: sync unit updates

Now that the V2Writable code is more generic, we can
sync with it to use `units' which represent either
a v2 epoch or an entire v1 inbox.

3 years agov2writable: pass oid to uindex_oid
Eric Wong [Tue, 27 Oct 2020 07:54:34 +0000 (07:54 +0000)]
v2writable: pass oid to uindex_oid

We'll be validating against this in the future to stop
bugs from creeping in.

3 years agoextsearchidx: remove {unindex_range} field
Eric Wong [Tue, 27 Oct 2020 07:54:33 +0000 (07:54 +0000)]
extsearchidx: remove {unindex_range} field

Moved to per-epoch "units".

3 years agov2writable: reduce scope of epoch-aware code
Eric Wong [Tue, 27 Oct 2020 07:54:32 +0000 (07:54 +0000)]
v2writable: reduce scope of epoch-aware code

And clearly label it.  We may try to reuse some of this for v1
indexing code paths.

3 years agoextsearchidx: more compatibility with V2Writable callers
Eric Wong [Tue, 27 Oct 2020 07:54:31 +0000 (07:54 +0000)]
extsearchidx: more compatibility with V2Writable callers

We'll use `index_oid' and `unindex_oid' as our method names
so V2Writable methods may use `$self->can' to access them.

3 years agov2writable: move size check init to sync_prepare
Eric Wong [Tue, 27 Oct 2020 07:54:30 +0000 (07:54 +0000)]
v2writable: move size check init to sync_prepare

This will let us use it from ExtSearchIdx.

3 years agov2writable: make *last_commits and sync_prepare OO methods
Eric Wong [Tue, 27 Oct 2020 07:54:29 +0000 (07:54 +0000)]
v2writable: make *last_commits and sync_prepare OO methods

This will allow ExtSearchIdx to override or reuse them more
easily.  Unfortunately we lose prototype validation, but that
seems to be discouraged anyways given the 'signatures' feature
in Perl 5.20+.

3 years agov2writable: rename {v2w} field to {self}
Eric Wong [Tue, 27 Oct 2020 07:54:28 +0000 (07:54 +0000)]
v2writable: rename {v2w} field to {self}

This will make it easier to reuse some indexing code for ExtSearchIdx.

3 years agov2writable: allow OO method references
Eric Wong [Tue, 27 Oct 2020 07:54:27 +0000 (07:54 +0000)]
v2writable: allow OO method references

Using `->can(method)' allows subclasses to override `index_oid'
and `unindex_oid' methods.

3 years agov2writable: more generic sync setup code
Eric Wong [Tue, 27 Oct 2020 07:54:26 +0000 (07:54 +0000)]
v2writable: more generic sync setup code

We want to reuse this code for ExtSearchIdx, eventually.

3 years agosearchidx: log2stack: simplify callers
Eric Wong [Tue, 27 Oct 2020 07:54:25 +0000 (07:54 +0000)]
searchidx: log2stack: simplify callers

Since we store {ibx} in $sync state, we no longer have to
pass it as an argument to log2stack.

3 years agosearchidx: put {ibx} into $sync state
Eric Wong [Tue, 27 Oct 2020 07:54:24 +0000 (07:54 +0000)]
searchidx: put {ibx} into $sync state

This will allow reusability with ExtSearchIdx

3 years agosearchidxshard: special init for eidx
Eric Wong [Tue, 27 Oct 2020 07:54:23 +0000 (07:54 +0000)]
searchidxshard: special init for eidx

Having a special init path for external indices is probably
easier than further overloading SearchIdx->new initialization
to work without an Inbox object.

3 years agosearchidx: xref3 delete support
Eric Wong [Tue, 27 Oct 2020 07:54:22 +0000 (07:54 +0000)]
searchidx: xref3 delete support

Not yet tested, but Perl compiles it!

3 years agosearchidx: index eidx_key as a boolean term
Eric Wong [Tue, 27 Oct 2020 07:54:21 +0000 (07:54 +0000)]
searchidx: index eidx_key as a boolean term

Using `O' (owner) here (according Xapian omega's
termprefixes.rst) since we could say the newsgroup or inbox is
the owner of the given message.

3 years agoextsearchidx: initial implementation
Eric Wong [Tue, 27 Oct 2020 07:54:20 +0000 (07:54 +0000)]
extsearchidx: initial implementation

It compiles...

3 years agov2writable: checkpoint: account for lack of {mm}
Eric Wong [Tue, 27 Oct 2020 07:54:19 +0000 (07:54 +0000)]
v2writable: checkpoint: account for lack of {mm}

ExtSearchIdx will not have Msgmap, since it may index
non email blobs in the future (it'll still be usable
with IMAP, but not NNTP).

3 years agov2writable: rename remaining "remote" terminology
Eric Wong [Tue, 27 Oct 2020 07:54:18 +0000 (07:54 +0000)]
v2writable: rename remaining "remote" terminology

"remote" used to imply "child process on the same machine" which
was somewhat non-sensical, anyways.  And OverIdx has been in the
same process since v2 was finalized.  So use the suffix "aux"
for "auxiliary" since it can be safely jettisoned without
breaking URLs.

3 years agoinboxwritable: eidx_key for external index
Eric Wong [Tue, 27 Oct 2020 07:54:17 +0000 (07:54 +0000)]
inboxwritable: eidx_key for external index

This is preferable to open-coding "newsgroup // inboxdir" everywhere.

3 years agov2: some changes for ExtSearchIdx compatibility
Eric Wong [Tue, 27 Oct 2020 07:54:16 +0000 (07:54 +0000)]
v2: some changes for ExtSearchIdx compatibility

We'll be using per-sync-state {ibx} refs instead, so make parts
of the v2 indexing code less-dependent on $self->{ibx} where
$self is a V2Writable object.

3 years agooveridx: introduce changes for external index
Eric Wong [Tue, 27 Oct 2020 07:54:15 +0000 (07:54 +0000)]
overidx: introduce changes for external index

Since external indices won't have msgmap.sqlite3, we'll need to
store last_commit-* metadata in over.sqlite3 instead.  This
has a longer limits to account for path names or newsgroup names
stored in keys.

We'll also rely on built-in counters for Xapian document IDs,
since msgmap.sqlite3 no longer provides an AUTOINCREMENT column.

3 years agov2writable: count_shards: allow working without {ibx}
Eric Wong [Tue, 27 Oct 2020 07:54:14 +0000 (07:54 +0000)]
v2writable: count_shards: allow working without {ibx}

This will be needed for ExtSearchIdx which doesn't have a
persistent PublicInbox::Inbox object.

3 years agov2writable: idx_shard: simplify callers
Eric Wong [Tue, 27 Oct 2020 07:54:13 +0000 (07:54 +0000)]
v2writable: idx_shard: simplify callers

This will make it easier-to-use in ExtSearchIdx.

3 years agosearchidxshard: allow msgref to be undef
Eric Wong [Tue, 27 Oct 2020 07:54:12 +0000 (07:54 +0000)]
searchidxshard: allow msgref to be undef

We don't need to keep it in code paths which are guaranteed to
only see PublicInbox::Eml (and not Email::MIME or PublicInbox::MIME
which did not round-trip properly).  However, we must set
{raw_bytes} since PublicInbox::Eml may add an extra "\n" for
rare messages with no bodies.

3 years agov2writable: hoist out write_alternates
Eric Wong [Tue, 27 Oct 2020 07:54:11 +0000 (07:54 +0000)]
v2writable: hoist out write_alternates

We'll be reusing this for external indices and possibly
other places.

3 years agov2writable: prepare initialization for external indices
Eric Wong [Tue, 27 Oct 2020 07:54:10 +0000 (07:54 +0000)]
v2writable: prepare initialization for external indices

External indices won't have $self->{ibx} since it needs to
deal with multiple inboxes.  We can also hoist out
->parallel_init to make it easier to distinguish the
non-parallel control flow.

3 years agosearchidx: introduce "xref3" concept
Eric Wong [Tue, 27 Oct 2020 07:54:09 +0000 (07:54 +0000)]
searchidx: introduce "xref3" concept

This will be used to track cross-posted messages in the
external/detached index.

3 years agosearch: xdb_sharded: make this a public method for ExtSearch
Eric Wong [Tue, 27 Oct 2020 07:54:08 +0000 (07:54 +0000)]
search: xdb_sharded: make this a public method for ExtSearch

We can simplify callers by using $self->{xpfx} instead of
passing another arg on the stack.

3 years agov2writable: make OO calls to last_commit-related methods
Eric Wong [Tue, 27 Oct 2020 07:54:07 +0000 (07:54 +0000)]
v2writable: make OO calls to last_commit-related methods

We'll try to reuse as much V2Writable code as possible for
external indices, but the way "last_commit" info is stored
must be different as external indices will deal with last_commit
info for multiple inboxes.

3 years agov2writable: add git method
Eric Wong [Tue, 27 Oct 2020 07:54:06 +0000 (07:54 +0000)]
v2writable: add git method

This will make it easier to share code with ExtSearchIdx.

3 years agosearchidx: expose INDEXLEVELS as `our'
Eric Wong [Tue, 27 Oct 2020 07:54:05 +0000 (07:54 +0000)]
searchidx: expose INDEXLEVELS as `our'

This will be used by external/detached indices, too.

3 years agoextsearch: start mocking out
Eric Wong [Tue, 27 Oct 2020 07:54:04 +0000 (07:54 +0000)]
extsearch: start mocking out

This will provide a similar API to PublicInbox::Inbox for
read-only WWW, -imapd, and -nntpd interfaces.

3 years agosearch: hoist out _xdb_sharded for v2 inboxes
Eric Wong [Tue, 27 Oct 2020 07:54:03 +0000 (07:54 +0000)]
search: hoist out _xdb_sharded for v2 inboxes

We'll be using this in detached (ext) Xapian indexes
in cross inbox search.

3 years agodoc/standards: add RFCs for URL schemes
Eric Wong [Tue, 27 Oct 2020 07:54:02 +0000 (07:54 +0000)]
doc/standards: add RFCs for URL schemes

We linkify these in the WWW UI, and will support them in other
places.  These URL schemes may end up being stored in
external/detached indices for indexing non-git-based mail
stores.

3 years agonntp: attempt RFC 5536 3.1.5-conformant Path: headers
Eric Wong [Wed, 4 Nov 2020 10:44:48 +0000 (10:44 +0000)]
nntp: attempt RFC 5536 3.1.5-conformant Path: headers

Perhaps some NNTP clients would be unhappy with the old value
"y".  So use a bit more bandwidth+space to use the server-name
and historical "!not-for-mail" tail-entry to better conform to
a published RFC.

Reported-by: Andrey Melnikov <temnota.am@gmail.com>
3 years agonntp: delimit Newsgroup: header with commas
Eric Wong [Tue, 3 Nov 2020 22:55:59 +0000 (22:55 +0000)]
nntp: delimit Newsgroup: header with commas

...instead of spaces.  This is specified in RFC 5536 3.1.4.

Include references to RFC 1036, 5536 and 5537 in our docs while
we're at it.

Reported-by: Andrey Melnikov <temnota.am@gmail.com>
Link: https://public-inbox.org/meta/CA+PODjpUN5Q4gBFQhAzUNuMasVEdmp9f=8Uo0Ej0mFumdSwi4w@mail.gmail.com/
4 years agotls: epollbit: account for miscellaneous OpenSSL errors
Eric Wong [Fri, 30 Oct 2020 02:13:58 +0000 (02:13 +0000)]
tls: epollbit: account for miscellaneous OpenSSL errors

Apparently they happen (triggered by my -imapd instance), so
bail out by closing the underlying socket rather than stopping
the event loop and daemon process.

4 years agoactually remove xt/eml_check_roundtrip.t
Eric Wong [Sat, 17 Oct 2020 21:27:06 +0000 (21:27 +0000)]
actually remove xt/eml_check_roundtrip.t

Fixes: 6550226296e9db79 ("xt: remove eml_check_roundtrip")
4 years agoxt: remove eml_check_roundtrip
Eric Wong [Sat, 17 Oct 2020 08:17:01 +0000 (08:17 +0000)]
xt: remove eml_check_roundtrip

If there's no body ({bdy} field), ->each_part set the {bdy}
field to "\n" and the ->as_string result afterwards is one
extra "\n" byte longer than the original.

It's not worth extra cycles in common ->each_part calls to
ensure 100% round-trip matches of header-only messages (which
are likely spam), especially when the only difference is a
trailing "\n".

4 years agogit: introduce async_wait_all
Eric Wong [Sat, 17 Oct 2020 08:04:24 +0000 (08:04 +0000)]
git: introduce async_wait_all

->cat_async and ->check_async may trigger each other (in future
callers) while waiting, so we need a unified method to ensure
both complete.  This doesn't affect current code, but allows us
to slightly simplify existing callers.

4 years agotmpfile: modernize to 5.10.1+, note O_APPEND workaround
Eric Wong [Fri, 16 Oct 2020 07:05:10 +0000 (07:05 +0000)]
tmpfile: modernize to 5.10.1+, note O_APPEND workaround

Once again we'll need O_APPEND on a temporary file, so note we
support it, here; since Perl 5.32 is way too new to depend on
our users having.

4 years agogit: async: loop inflight checks for nested callbacks
Eric Wong [Fri, 16 Oct 2020 06:59:34 +0000 (06:59 +0000)]
git: async: loop inflight checks for nested callbacks

We need to loop the inflight check for nested callback
invocations to ensure we don't clog the pipe that feeds
`git cat-file'.

This bug was obscured by the fact that we're already
accounting for 64-char git OIDs with SHA-256 in the
pipe space calculation; perhaps we shouldn't do that.

4 years agogit: *_async: support nested callback invocations
Eric Wong [Fri, 16 Oct 2020 06:59:33 +0000 (06:59 +0000)]
git: *_async: support nested callback invocations

For external indices, we'll need to support nested cat_async
invocations to deduplicate cross-posted messages.

Thus we need to ensure we do not clobber the {inflight*} queues
while stepping through and ensure {cat_rbuf} is stored before
invoking callbacks.

This fixes the ->cat_async-only case, but does not yet
account for the mix of ->check_async interspersed with
->cat_async calls, yet.  More work will be needed on that
front at a later date.

4 years agogit: ensure ->destroy clobbers check_async read buffer
Eric Wong [Fri, 16 Oct 2020 06:59:32 +0000 (06:59 +0000)]
git: ensure ->destroy clobbers check_async read buffer

It's currently not a problem as ->destroy doesn't
happen for no reason, we'll need to ensure future uses of
->destroy correctly discard the check_async buffer.

4 years agoinbox: add uidvalidity method
Eric Wong [Fri, 16 Oct 2020 06:59:31 +0000 (06:59 +0000)]
inbox: add uidvalidity method

This will make it easier to deal with ExtSearchIdx, which
won't have msgmap.

4 years agoscripts/dupe-finder: restore $dbh variable
Kyle Meyer [Fri, 16 Oct 2020 02:08:52 +0000 (22:08 -0400)]
scripts/dupe-finder: restore $dbh variable

When dupe-finder was switched from ->search->{over_ro} to ->over, the
database handle was dropped.  Restore it because a spot downstream
uses it.

Fixes: 73e3a6ed6e95adc6 (use more idiomatic internal API for ->over access)
4 years agoadmin: preserve config ordering of `--all' switch
Eric Wong [Tue, 13 Oct 2020 01:34:30 +0000 (01:34 +0000)]
admin: preserve config ordering of `--all' switch

When `--all' is passed to -index and similar commands, process
them in the same order as what is given in the config file.
This ensures predictable behavior so admins can ensure certain
inboxes see updated indices before others.  For (upcoming)
external indices, this will ensure stable Xref: ordering for
predictable caching/memoization by NNTP clients.

4 years agomanifest: favor Cpanel::JSON::XS
Eric Wong [Sat, 3 Oct 2020 23:12:46 +0000 (23:12 +0000)]
manifest: favor Cpanel::JSON::XS

JSON::MaybeXS already favors Cpanel::JSON::XS (and has for many
years, now).  Allow users to skip installing JSON::MaybeXS if
they want an XS-based JSON implementation.

4 years agov2writable: use "HEAD" to match v1 indexing behavior
Eric Wong [Tue, 29 Sep 2020 19:43:50 +0000 (19:43 +0000)]
v2writable: use "HEAD" to match v1 indexing behavior

Users may want to change the default branch used for git epochs
in v2 (v1 SearchIdx always used whatever "HEAD" pointed to).

4 years agosearchidx: index lower-case List-Id value
Eric Wong [Mon, 28 Sep 2020 05:15:19 +0000 (05:15 +0000)]
searchidx: index lower-case List-Id value

We don't want a List-Id value being confused with a Xapian
term prefix, here.

Followup-to: 8b06cda3a3af3f0e ("mda: match List-Id insensitively")
4 years agogcf2: improve error handling and do not ->fail on wbuf
Eric Wong [Sun, 27 Sep 2020 22:12:48 +0000 (22:12 +0000)]
gcf2: improve error handling and do not ->fail on wbuf

For historical reasons, both Danga::Socket::write and
PublicInbox::DS::write will return 0 when data is buffered;
so Gcf2Client must not call ->fail when DS::write returns 0.

We'll also improve robustness by recreating the entire
Gcf2Client object if it does die for other reasons, instead of
risking mismatched fields due to deferred close.

We also need to ensure we only get one EPOLLERR wakeup and
issue EPOLL_CTL_DEL if ->event_step is triggered by a dying
Gcf2 process, so always register the FD with EPOLLONESHOT.

4 years agods: add missing label for systems w/o EPOLLEXCLUSIVE
Eric Wong [Sun, 27 Sep 2020 21:26:08 +0000 (21:26 +0000)]
ds: add missing label for systems w/o EPOLLEXCLUSIVE

Oops :x

4 years agoimap: avoid raising exception if client disconnects
Eric Wong [Sat, 26 Sep 2020 08:08:37 +0000 (08:08 +0000)]
imap: avoid raising exception if client disconnects

This ought to save a few cycles if a client disconnects while
in the middle of a (UID) FETCH.  This avoids:

  Can't call method "git" on an undefined value at .../PublicInbox/IMAP.pm

errors in stderr.

4 years agoxt: add eml ->as_string round trip checker
Eric Wong [Thu, 24 Sep 2020 20:51:45 +0000 (20:51 +0000)]
xt: add eml ->as_string round trip checker

Unlike Email::MIME, PublicInbox::Eml::as_string should be able
to round trip from the Perl object to a raw scalar and back
without changes.

4 years agosearchidx: fix (undocumented) --skip-docdata handling
Eric Wong [Thu, 24 Sep 2020 10:13:39 +0000 (10:13 +0000)]
searchidx: fix (undocumented) --skip-docdata handling

This switch is still undocumented, but we can reduce the scope
of our Xapian docdata dependency by moving its only caller to
SearchIdx.  This reduces the amount of code loaded by read-only
code paths.

4 years agov2writable: drop outdated {unindex_range} check
Eric Wong [Tue, 22 Sep 2020 18:49:50 +0000 (18:49 +0000)]
v2writable: drop outdated {unindex_range} check

{unindex_range} only exists in the $sync state, nowadays, not the
V2Writable ($self) object.  $sync->{unindex_range} won't be
populated if $regen_max is zero, either, unless somebody is
injecting importable commits into an epoch history, in which
this change will result in no-op indexing doing no work.

4 years agoidxstack: fix comment about file_char
Eric Wong [Tue, 22 Sep 2020 18:49:49 +0000 (18:49 +0000)]
idxstack: fix comment about file_char

It's `d' for deletes, not `a'.

4 years agomda: match List-Id insensitively
Eric Wong [Mon, 21 Sep 2020 20:58:09 +0000 (20:58 +0000)]
mda: match List-Id insensitively

This follows -watch commit b70473ab8296d31ebb600adb4fa8fe0ac5935ca8
to match List-Id headers case-insensitively.

Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20200921180152.uyqluod7qxbwqubo@chatter.i7.local/
4 years agomid: drop repeated ';' in mid_escape() regular expression
Kyle Meyer [Sun, 20 Sep 2020 18:13:58 +0000 (14:13 -0400)]
mid: drop repeated ';' in mid_escape() regular expression

4 years agodoc: post-1.6 updates, start 1.7
Eric Wong [Sat, 19 Sep 2020 21:42:01 +0000 (21:42 +0000)]
doc: post-1.6 updates, start 1.7

I should've dropped "PENDING" notes before the 1.6 release;
they're dropped now, and a note is added to remind my future
self to drop them before 1.7.

4 years agoconfig: warn on multiple values for some fields
Eric Wong [Sun, 20 Sep 2020 01:43:15 +0000 (01:43 +0000)]
config: warn on multiple values for some fields

Our code doesn't support multi-values for these, and having
unexpected arrays leads to unexpected results (e.g. showing
stuff like "ARRAY(0xDEADBEEFADD12E55)" in user interfaces).  So
warn and only use the last value (matching git-config(1)
behavior without `--get-all').

4 years agogcf2: wire up read-only daemons and rm -gcf2 script
Eric Wong [Sat, 19 Sep 2020 09:37:14 +0000 (09:37 +0000)]
gcf2: wire up read-only daemons and rm -gcf2 script

It seems easiest to have a singleton Gcf2Client client object
per daemon worker for all inboxes to use.  This reduces overall
FD usage from pipes.

The `public-inbox-gcf2' command + manpage are gone and a `$^X'
one-liner is used, instead.  This saves inodes for internal
commands and hopefully makes it easier to avoid mismatched
PERL5LIB include paths (as noticed during development :x).

We'll also make the existing cat-file process management
infrastructure more resilient to BOFHs on process killing
sprees (or in case our libgit2-based code fails on us).

(Rare) PublicInbox::WWW PSGI users NOT using public-inbox-httpd
won't automatically benefit from this change, and extra
configuration will be required (to be documented later).

4 years agogcf2: require git dir with OID
Eric Wong [Sat, 19 Sep 2020 09:37:13 +0000 (09:37 +0000)]
gcf2: require git dir with OID

This amortizes the cost of recreating PublicInbox::Gcf2 objects
when alternates change in v2 all.git.