Eric Wong [Sun, 27 Dec 2020 02:53:06 +0000 (02:53 +0000)]
ds: simplify EventLoop implementation
More importantly, make it easier-to-find the sub by avoiding
runtime manipulation of subroutine names. There's no point in
avoiding a potential call to _InitPoller in EventLoop since
entering EventLoop is rare.
On the contrary, PublicInbox::DS->new is called often and this
change to avoid entering _InitPoller there may have more
benefits (which may still be unmeasurable).
Eric Wong [Sun, 27 Dec 2020 19:38:29 +0000 (19:38 +0000)]
search: remove {mset} option for ->mset method
The ->mset method always returns a Xapian mset nowadays, so
naming a parameter {mset} is too confusing. As it does with
MiscSearch, setting the {relevance} parameter to -1 now sorts by
ascending docid order. -2 is now supported for descending
docid order, too, since it may be useful for lei users.
Eric Wong [Sun, 27 Dec 2020 11:01:42 +0000 (11:01 +0000)]
miscsearch: take reopen from Search and use it
As with ExtSearch, MiscSearch lacks a janky cleanup timer of
PublicInbox::Inbox objects, leading to info about
inboxes/newsgroups going stale. Fortunately, we don't use
MiscSearch very heavily, yet.
In the future, we may be able to detect new inboxes without
having to SIGHUP or restart daemons using MiscSearch.
Eric Wong [Sun, 27 Dec 2020 11:01:41 +0000 (11:01 +0000)]
extsearch: unconditionally reopen on access
Since ExtSearch lacks the janky cleanup timer of
PublicInbox::Inbox objects, its search results get stale.
Reopen the Xapian DB on every ->search call for now, as
reducing reopen calls doesn't seem worth the complexity.
The Xapian::Database::reopen operation itself takes only ~50us
on my old workstation with 3 shards totaling <200GB. Other
parts of Xapian dominates the search time, so the reopen seems
inconsequential with single-digit shard counts.
Eric Wong [Sat, 26 Dec 2020 10:16:24 +0000 (10:16 +0000)]
extindex: allow using --all without EXTINDEX_DIR
If "--all" is specified to index all inboxes, implicitly choose
the configured [extindex "all"] external index since "--all" is
incompatible with specifying inbox directories on the
command-line.
Eric Wong [Sat, 26 Dec 2020 10:16:23 +0000 (10:16 +0000)]
extindex: add undocumented --no-scan switch
This makes diagnosing --watch problems easier when there's
50K inboxes by avoiding the lengthy scan (which is the reason
--watch exists in the first place).
Eric Wong [Sat, 26 Dec 2020 10:16:22 +0000 (10:16 +0000)]
extindex: enable autoflush on STDOUT/STDERR
With --watch, the output may be redirected to a pipe or socket
which Perl may decide to buffer. Ensure Perl doesn't buffer
these outputs since they can provide real-time status updates
in response to signals or FS activity.
Eric Wong [Sat, 26 Dec 2020 10:16:21 +0000 (10:16 +0000)]
extindex: various --watch signal handling fixes
We need to clobber the SIGUSR1 resync queue on SIGHUP to
invalidate old inbox objects. Furthermore, the lengthy
initial scan needs to ignore signals intended for the
event loop to avoid unexpected behavior. Finally, add
some progress output to inform users on the terminal
to inform users' of progress.
Eric Wong [Sat, 26 Dec 2020 01:44:37 +0000 (01:44 +0000)]
extindex: --watch for inotify-based updates
This reuses existing InboxIdle infrastructure to update external
indices based on per-inbox updates. This is an alternative to
auto-updating external indices via the -index command and also
works with existing uses of -mda and public-inbox-watch.
Using inotify (or EVFILT_VNODE) allows watching thousands of
inboxes without having to scan every single one at every
invocation.
This is especially beneficial in cases where an external index
is not writable to the users writing to per-inbox indices.
Eric Wong [Sat, 26 Dec 2020 12:25:42 +0000 (12:25 +0000)]
eml: fix undefined vars on <Perl 5.28
Encode::MIME::Header::_decode_octets did not correctly default
to Encode::FB_DEFAULT until Encode 2.93 (perl5.git commit 0c541dc5633a341cf44b818014b58e7f8be532e9). Provide the default
again to work with older Perls.
Eric Wong [Sat, 26 Dec 2020 12:30:35 +0000 (12:30 +0000)]
t/config: test --get-urlmatch for git <2.26
While git 1.8.5 learned --get-urlmatch, git did not learn to
match URLs against wildcards until 2.26. So only depend on
1.8.5 for this test since 2.26 is too new.
Eric Wong [Sat, 26 Dec 2020 01:44:36 +0000 (01:44 +0000)]
default to CORE::warn in $SIG{__WARN__} handlers
As with CORE::die and $SIG{__DIE__}, it turns out CORE::warn is
safe to use inside $SIG{__WARN__} handlers without triggering
infinite recursion. So fall back to reusing CORE::warn instead
of creating a new sub.
Eric Wong [Sat, 26 Dec 2020 08:12:52 +0000 (08:12 +0000)]
inbox: name variable for values loop iterator
->on_inbox_unlock callbacks could clobber $_, and this seems to
fix a problem with -extindex --watch failing to index some
inboxes after SIGHUP reload.
Eric Wong [Sat, 26 Dec 2020 09:34:39 +0000 (09:34 +0000)]
inboxidle: avoid needless syscalls on refresh
We don't have to replace a bunch of existing watches
with identical new ones. On Linux with Linux::Inotify2
installed, this avoids a storm of inotify_add_watch(2)
and inotify_rm_watch(2) syscalls on SIGHUP with -imapd
and "-extindex --watch"
Eric Wong [Sat, 26 Dec 2020 05:59:22 +0000 (05:59 +0000)]
inboxidle: clue users into resolving ENOSPC from inotify
It may not be obvious to users a ENOSPC error is from hitting
a (tunable) kernel-imposed limit on inotify watches, and not
some storage device running out of space. Give them a hint
here to reduce our own support burden.
Eric Wong [Fri, 25 Dec 2020 10:21:15 +0000 (10:21 +0000)]
index: filter out indexlevel=basic from extindex
extindex users will likely want to use indexlevel=basic for
per-inbox indices, however extindex itself doesn't support basic
index level (yet?). Let's ensure we don't trip up extindex
users who specify "-L basic" on the -index command-line.
Eric Wong [Fri, 25 Dec 2020 10:21:14 +0000 (10:21 +0000)]
v2writable: don't verify tip if reindexing
We only rely on git-rev-parse to resolve symbolic names ("HEAD")
to a SHA-* git commit ID. We'll assume any git commit IDs we
get from SQLite DBs are valid and let "git-log" fail if it
isn't.
Eric Wong [Fri, 25 Dec 2020 10:21:12 +0000 (10:21 +0000)]
index: do not attach inbox to extindex unless updated
We'll count the number of log changes (regardless of index or
unindex) and only attach inboxes to ExtSearchIdx objects when
they get new work. We'll also reduce lock bouncing and only
update external indices after all per-inbox indexing is done.
This also updates existing v2 indexing/unindexing callers
to be more consistent and ensures unindex log entries update
per-inbox last commit information.
Eric Wong [Fri, 25 Dec 2020 10:21:11 +0000 (10:21 +0000)]
extsearchidx: close DB handles after use if FD constrained
Most distros ship with low RLIMIT_NOFILE limits and surprises
may lurk for admins who configure many inboxes. Keep FD usage
under control to avoid EMFILE errors at inopportune times during
reindex.
From what I can tell, this is the only place where extindex can
have unpredictable FD growth when there's thousands of inboxes,
and it's in an extremely rare code path.
Eric Wong [Fri, 25 Dec 2020 10:21:10 +0000 (10:21 +0000)]
extsearchidx: delay SQLite availability checks
This will make attach_inbox faster for no-op calls. It also
helps us avoid races in case msgmap or over.sqlite3 gets
unlinked while -extindex is running.
Eric Wong [Thu, 24 Dec 2020 10:09:19 +0000 (10:09 +0000)]
index: support --fast-noop / -F switch
Note: I'm not sure if it's worth documenting and supporting this
long-term.
We can can avoid taking locks for invocations of "index --all"
and rely on high-resolution ctime (struct timespec st_ctim)
comparisons of msgmap.sqlite3 and the packed-refs + refs/heads
directory of the newest epoch.
This cuts public-inbox-index invocations with
"--all --no-update-extindex -L basic" down from 0.92s to 0.31s.
The change with "-L medium" or "-L full" and (default) non-zero
jobs is even more drastic, reducing a 12-13s no-op invocation
down to the same 0.31s
Eric Wong [Wed, 23 Dec 2020 23:02:55 +0000 (23:02 +0000)]
index: update [extindex "all"] by default, support -E
In most cases, this ensures users will only have to opt-in to
using -extindex once and won't have to issue extra commands
to keep external indices up-to-date when using
public-inbox-index.
Since we support arbitrary numbers of external indices for
ease-of-development, we'll support repeating "-E"
("--update-extindex=") in case users want to test changes in
parallel.
Eric Wong [Wed, 23 Dec 2020 08:38:53 +0000 (08:38 +0000)]
config: config_fh_parse: micro-optimize harder
Instead of relying on split() and a regexp, we'll drop split()
entirely and rely on index() + two substr() calls to operate on
fixed strings. This brings PublicInbox::Config->new time down
from 0.98s down to 0.84s.
Eric Wong [Wed, 23 Dec 2020 08:38:52 +0000 (08:38 +0000)]
config: config_fh_parse: micro-optimize
We can avoid a slow regexp capture and instead and rely on
rindex + substr to extract the section from the config file.
Then we use the defined-or-assignment (//=) operator combined
with the documented return value of `push' to ensure @section_order
is unique without repeating a hash lookup.
Finally, we avoid short-lived variables inside the loop and
declare them subroutine-wide to knock a teeny bit of allocation
time.
Combined, these optimizations bring the ~1.22s
PublicInbox::Config->new time down to ~0.98s with 50K inboxes.
Eric Wong [Wed, 23 Dec 2020 08:38:51 +0000 (08:38 +0000)]
config: git_config_dump: pre-compile RE for split
It appears the Perl split() operator is not optimized for fixed
strings at all. With this change, PublicInbox::Config->new (w/o
->fill_all) time is reduced from 1.81s to 1.22s on a config file
with 50K inboxes.
Eric Wong [Wed, 23 Dec 2020 08:38:49 +0000 (08:38 +0000)]
extsearchidx: close SQLite handles after attaching
This is needed to prevent us from running out of FDs when
indexing many inboxes. Perhaps checking these on attach_inbox
is unnecessary and may be removed entirely down the line.
Eric Wong [Wed, 23 Dec 2020 08:38:48 +0000 (08:38 +0000)]
miscsearch: index UIDVALIDITY, use as startup cache
This brings -nntpd startup time down from ~35s to ~5s with 50K
inboxes.
Further improvements ought to be possible with deeper changes to
MiscIdx, since -mda having to load every inbox seems unreasonable;
but this general change is fairly unintrusive.
Eric Wong [Wed, 23 Dec 2020 08:38:46 +0000 (08:38 +0000)]
inbox: git_epoch: correct false comment
The original comment hasn't been true since
PublicInbox::Git->modified was changed to use cat_async blob
responses. In any case, manifest.js.gz generation already
cleans up per-epoch git processes used for ->modified.
Eric Wong [Tue, 22 Dec 2020 05:04:16 +0000 (05:04 +0000)]
wwwstream: show relative coderepo URLs correctly
Trying to link "foo.git" relative to the current URL usually
does not provide correct results, so prefix it by going
into the parent directory if an absolute (or protocol-relative)
URL is not supplied.
Eric Wong [Tue, 22 Dec 2020 06:01:44 +0000 (06:01 +0000)]
admin: resolve inboxes to absolute paths for index
Some of my ancient v1-only scripts called public-inbox-index
to operate on GIT_DIR:
GIT_DIR=/path/to/foo.git public-inbox-index
This change ensures they keep working, otherwise "." will be
passed to the --git-dir= switch of git(1) because that's the
default directory if no inboxes are specified on the
command-line.
Uwe Kleine-König [Tue, 22 Dec 2020 17:18:10 +0000 (18:18 +0100)]
public-inbox-v[12]-format.pod: make lexgrog happy
The Debian package linter (lintian) emits the following warning:
W: bad-whatis-entry
N:
N: A manual page should start with a NAME section, which lists the
N: program name and a brief description. The NAME section is used to
N: generate a database that can be queried by commands like apropos and
N: whatis. You are seeing this tag because lexgrog was unable to parse
N: the NAME section.
N:
N: Manual pages for multiple programs, functions, or files should list
N: each separated by a comma and a space, followed by \- and a common
N: description.
N:
N: Listed items may not contain any spaces. A manual page for a two-level
N: command such as fs listacl must look like fs_listacl so the list is
N: read correctly.
N:
N: Refer to the lexgrog(1) manual page, the groff_man(7) manual page, and
N: the groff_mdoc(7) manual page for details.
N:
N: Severity: warning
N:
N: Check: documentation/manual
N:
N: Renamed from: manpage-has-bad-whatis-entry
N:
for public-inbox-v1-format and public-inbox-v2-format.
Adapt the descriptions to make lexgrog and so lintian happy.
Eric Wong [Mon, 21 Dec 2020 07:51:22 +0000 (07:51 +0000)]
extsearch*: drop unnecessary path canonicalization
Unlike inboxdir, the canonical-ness of -extindex paths is not
relevant at the moment, and may never be relevant at all. So
don't mislead others into thinking these paths being
canonicalized matters.
Eric Wong [Mon, 21 Dec 2020 07:51:21 +0000 (07:51 +0000)]
searchidx: rename get_val to int_val and return IV
Values can be strings in Xapian, although we currently use
integer values exclusively. Give the wrapper a more appropriate
name in case we start using string columns.
For future-proofing, we'll now return `undef' on missing columns
and coerce the return value to an IV (integer value) to save
memory, as sortable_unserialise returns a PV (pointer value)
scalar despite it existing to support numeric values.
Eric Wong [Mon, 21 Dec 2020 07:51:19 +0000 (07:51 +0000)]
isearch: use numeric sort for article numbers
Perl sort is alphabetical by default and Xapian uses numeric
document IDs, so sort must be told explicitly to use numeric
comparisons even if the scalars are integer values (IV)
internally.
And eliminate extra hash marks ("#") since they're probably too
noisy if there are many IDs.
Note: I haven't seen this warning message in syslog, yet :>
Eric Wong [Mon, 21 Dec 2020 07:51:18 +0000 (07:51 +0000)]
inbox: delay ->version detection
Our read-only code won't need to know the version until an inbox
is accessed. This is a small step towards eliminating many
stat() calls on read-only daemon startup.
We'll try to avoid calling Cwd::abs_path and use
File::Spec->rel2abs instead, since abs_path will resolve
symlinks the user specified on the command-line.
Unfortunately, ->rel2abs still leaves "/.." and "/../"
uncollapsed, so we still need to fall back to Cwd::abs_path in
those cases.
While we are at it, we'll also resolve inboxdir from deep inside
v2 directories instead of misdetecting them as v1 bare git
repos.
In any case, stop matching directories by name and instead rely
on the unique combination of st_dev + st_ino on stat() as we
started doing in the extindex code.
Eric Wong [Thu, 17 Dec 2020 19:40:06 +0000 (19:40 +0000)]
extsearchidx: improve missing machine-id fallback
It's likely most GNU/Linux systems have /etc/machine-id these
days, so anything missing it is likely a *BSD, most of which
support and favor "sysctl -n kern.hostid". We'll also support
"ghostid" since GNU utils are commonly prefixed with 'g' on
non-GNU platforms.
In any case, we'll suppress stderr from missing commands and
fall back to hard coding an $OSNAME-based identifier as a last
resort and hope the hostname is unique.
Uwe Kleine-König [Fri, 18 Dec 2020 11:56:14 +0000 (12:56 +0100)]
Fix manpage section of perl module documentation
On Debian (at least) perl documentation is supposed to be installed in
section 3pm. With the build system hardcoding this to 3 instead this
results in a warning by the Debian package linter:
W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::Git.3.gz:74 3 != 3pm
W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::Import.3.gz:74 3 != 3pm
W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::SaPlugin::ListMirror.3.gz:74 3 != 3pm
W: public-inbox: wrong-manual-section ... use --no-tag-display-limit to see all (or pipe to a file/program)
Eric Wong [Thu, 17 Dec 2020 09:14:48 +0000 (09:14 +0000)]
import: drop X-Status in addition to Status
It's actually supported by mutt, dovecot[1], and likely some other
software to augment the Status: header. While dovecot doesn't
expose X-Status to clients, mutt will write 'A' (answered) and
'F' to X-Status (but not T (draft)).
So we'll drop it like we do Status since it's not suitable for
public mail, but stick it in an @UNWANTED_HEADERS array will
allow us to configure an override if needed.
Eric Wong [Wed, 16 Dec 2020 23:19:06 +0000 (23:19 +0000)]
extsearchidx: no need to make InboxWritable
extindex treats v1/v2 public inboxes as read-only, so there's
no need to scare people by using the InboxWritable package
now that ->git_dir_n is gone and we can use ->max_git_epoch
instead of ->git_dir_latest.
Eric Wong [Wed, 16 Dec 2020 23:19:04 +0000 (23:19 +0000)]
inbox: simplify v2 epoch counting
Perl readdir detects list context and can return an array
suitable for the grep op. From there, we can rely on
substr to remove the ".git" suffix and integerize the value
to save a few bytes before letting List::Util::max return
the value.
This is how we detect Xapian shards nowadays, too, and
we'll also use defined-or (//) to simplify the return
value there.
We'll also simplify InboxWritable->git_dir_latest,
remove some callers, and consider removing it entirely.
Eric Wong [Wed, 16 Dec 2020 23:04:53 +0000 (23:04 +0000)]
extsearchidx: lock eidxq on full --reindex
Incremental indexing can use the `eidxq' reindexing queue for
handling deletes and resuming interrupted indexing. Ensure
those incremental -extindex invocations do not steal (and
prematurely perform) work that an "-extindex --reindex"
invocation is handling.
When checkpointing and yielding the lock to other processes,
we need to ensure any open DB statement handles are closed,
since they reference and prevent DB FDs from being closed
and unlocked.
And clean up some progress reporting while we're at it.
Eric Wong [Tue, 15 Dec 2020 02:02:22 +0000 (02:02 +0000)]
extsearchidx: simplify reindex code paths
Since we're inside a Xapian transaction, calling ->index_raw
followed by ->shard_add_eidx_info calls on the same docid
doesn't seem to hurt indexing performance. It definitely
reduces FS read traffic and IPC from git at the cost of some
more IPC between the parent and workers. Nevertheless, the code
and FD reductions seem worth it.
Eric Wong [Tue, 15 Dec 2020 02:02:21 +0000 (02:02 +0000)]
extsearchidx: checkpoint releases locks
--reindex can take many hours or days, ensure we release
locks according to --batch-size so automated fetch+index
jobs can write new data to indices while we update old data.
Eric Wong [Tue, 15 Dec 2020 02:02:20 +0000 (02:02 +0000)]
extsearchidx: reindex works on Xapian, too
Instead of just working on over.sqlite3, we need to work on
the Xapian DBs as well. While no changes to our Xapian use
have taken place recently, they could in the future and
--reindex exists to account for that.
Eric Wong [Tue, 15 Dec 2020 02:02:19 +0000 (02:02 +0000)]
extindex: support --rethread and content bifurcation
--rethread is useful for dealing with bugs and behaves
just like it does with current inboxes.
This is in case our content deduplication logic changes for
whatever reason and causes previously merged messages to be
considered "different". As with v2, this won't allow us to
merge messages in a way that allows deduplicating messages which
were previously considered different, but v2 inboxes do not
allow that, either.
In other words, this makes the --reindex and --rethread
switches of -extindex match the behavior of v2 -index.
Eric Wong [Tue, 15 Dec 2020 02:02:18 +0000 (02:02 +0000)]
over: sort xref3 by xnum if ibx_id repeats
While unlikely to happen, it may be possible for messages
from the same inbox to get indexed multiple times. Provide
consistent results in this case for ease-of-testing.
Eric Wong [Tue, 15 Dec 2020 02:02:16 +0000 (02:02 +0000)]
extindex: preliminary --reindex support
--reindex allows us to catch missed and stale messages due to
-extindex vs -index races prior to commit 02b2fcc46f364b51
("extsearchidx: enforce -index before -extindex").
We'll also rely on reindex to internally deal with v1/v2 inbox
removals and partial-unindexing of messages which are only
removed from one inbox out of many.
This reindex design is completely different than how normal
v1/v2 inbox reindex operates due to extindex having multiple
histories to work with. Instead of scanning git history, this
relies exclusively on comparing over.sqlite3 contents between
the v1/v2 inboxes and the extindex.
Changes to Xapian behavior also get picked up, now. Xapian indexing
is handled by workers with minimal IPC to the parent process.
This results in more read I/O but fewer writes when dealing
with cross-posted messages.
Changes to $smsg->populate and --rethread still need further
work.
Eric Wong [Wed, 16 Dec 2020 04:39:37 +0000 (04:39 +0000)]
t/psgi_v2: ignore warnings on missing P::M::ReverseProxy
Plack::Test::ExternalServer doesn't depend on
Plack::Middleware::ReverseProxy, so we need to account for
some warnings in stderr if P::M::RP is missing.
Eric Wong [Wed, 16 Dec 2020 23:56:29 +0000 (23:56 +0000)]
inbox: ->uidvalidity returns undef w/o ->mm
While totally unindexed inboxes are rare, we still support
them for v1 and may hit code which calls this method. Just
return `undef' when ->mm access fails.
Eric Wong [Tue, 15 Dec 2020 11:47:16 +0000 (11:47 +0000)]
daemon: support --daemonize without Net::Server::Daemonize
We don't actually need Net::Server::Daemonize to support
the --daemonize flag, since the daemonize() sub provided
by N::S::D doesn't exactly do the things we want.
Eric Wong [Thu, 10 Dec 2020 22:38:47 +0000 (22:38 +0000)]
nntp+www: drop List-* and Archived-At headers
These headers can conflict with headers in the DKIM signature;
and parsing the DKIM-Signature header to determine whether or
not we can safely add a header would be more code and CPU
cycles.
Since IMAP seems fine without these headers (and JMAP will
likely be, too), there's likely no need to continue appending
these to every message. Nowadays, developers seem sufficiently
trained to use URLs with Message-IDs in them. So drop the
headers and save some cycles and bandwidth all around.
Eric Wong [Thu, 10 Dec 2020 23:04:50 +0000 (23:04 +0000)]
extmsg: avoid exceptions when /all/$MSGID/ fails
If a message can't be found in ->ALL, we shouldn't attempt to
enter code paths which iterate normal inboxes or attempt to
access non-existent fields (e.g. {name}, {newsgroup},
{inboxdir}) in the ExtSearch object.
Eric Wong [Thu, 10 Dec 2020 08:35:47 +0000 (08:35 +0000)]
manifest: account for future cache in MiscIdx docdata
We'll be storing private data inside the "" (empty string) key
of the JSON doc we use for store for manifest.js.gz generation.
This private data will allow us to reduce FS activity at and
speed up startup times, but some will also be in Xapian boolean
terms and values for searching and filtering.
Eric Wong [Wed, 9 Dec 2020 09:25:12 +0000 (09:25 +0000)]
extsearchidx: enforce -index before -extindex
We cannot set xref3 data without the `xnum' column to
tie it to the per-inbox over.sqlite3 DB. So ensure we don't
read brand-new history that only exists in git, but instead
rely on last_commit and last_xap15-$EPOCH metadata in msgmap
to decide how far we can index.
Before this change, it was possible to miss messages in
the extindex if -index did not run (which will be fixable by
upcoming --reindex support in -extindex).
Eric Wong [Wed, 9 Dec 2020 09:25:10 +0000 (09:25 +0000)]
searchidx: all indexers check for bad blobs
This should help us detect bugs in our code or storage
synchronization problems more easily. This probably won't
detect corrupted git storage, but can detect corrupted SQLite
files.
"Bad blobs, bad blobs, whatcha gonna do when they come for you?"
Eric Wong [Wed, 9 Dec 2020 23:33:00 +0000 (23:33 +0000)]
www+nntp: deal with lack of addresses for ->ALL
Since extindex is an amalgamation of several inboxes, discerning
an appropriate address for List-Post: would be expensive and
most likely unnecessary. Some legacy/historical inboxes may
have no active address, either, so don't attempt to set the
List-Post header if no addresses are configured.
Eric Wong [Tue, 8 Dec 2020 21:21:28 +0000 (21:21 +0000)]
extindex: do not use current dir like -index does
At least not for resolving inboxes, since there's no good way
for a user to specify what is an inbox or extindex directory
without a command-line switch.
Instead of changing the -extindex command, we change the -index
command internals to rely on the new {-use_cwd} flag to avoid
internal use of negation, since double-negatives and the like
are confusing to me.
Eric Wong [Tue, 8 Dec 2020 21:21:27 +0000 (21:21 +0000)]
rename {pi_config} fields to {pi_cfg}
{pi_config} may be confused with the documented `PI_CONFIG'
environment variable, and we'll favor vowel-removal to be
consistent with our usage of object references.
The `pi_' prefix may stay in some places, for now; since a
separate namespace may come into this codebase for local/private
client-tooling.
For InboxIdle, we'll also remove an invalid comment about
holding a reference to the PublicInbox::Config object, too.
Eric Wong [Tue, 8 Dec 2020 21:21:26 +0000 (21:21 +0000)]
nntp: replace {ng} with {ibx} for consistency
They're PublicInbox::Inbox objects just like the rest of
the non-NNTP code. So rename the NNTP code for consistency
with the rest of the codebase. Furthermore, {ng} and $ng
may be confused with the `--ng' switch for -init, and that's
a non-ref scalar string.
Eric Wong [Tue, 8 Dec 2020 21:21:25 +0000 (21:21 +0000)]
treewide: replace {-inbox} with {ibx} for consistency
{ibx} is shorter and is the most prevalent abbreviation
in indexing and IMAP code, and the `$ibx' local variable
is already prevalent throughout.
In general, the codebase favors removal of vowels in variable
and field names to denote non-references (because references are
"lighter" than non-references).
So update WWW and Filter users to use the same code since
it reduces confusion and may allow easier code sharing.
Eric Wong [Sat, 5 Dec 2020 22:22:46 +0000 (22:22 +0000)]
search: reinstate "uid:" internal search prefix
User-supplied queries (via PublicInbox::IMAPsearchqp) may
restrict messages to certain UID ranges in addition to the
limits we impose ourselves for mailbox slices. So we'll
continue to ask Xapian::QueryParser to "uid:" numeric ranges.
Fixes: 4b551c884a648b45 ("imap: support isearch and reduce Xapian queries")
Eric Wong [Mon, 7 Dec 2020 07:40:53 +0000 (07:40 +0000)]
shard_add_eidx_info: pass $eidx_key instead of $ibx object
This improves consistency with sibling methods such as
->shard_remove_eidx_info and ->add_xref3. Passing the
$eidx_key scalar is preferable to the entire $ibx object
for IPC-friendliness.
Eric Wong [Mon, 7 Dec 2020 07:40:52 +0000 (07:40 +0000)]
searchidx: remove $oid parameter from most calls
Xapian docids have been tied to the over {num} column for
nearly 3 years, now; and OIDs are no longer stored in Xapian
document data. There's no need to increase code and IPC
complexity by passing the OID around.