Eric Wong [Mon, 27 May 2019 18:45:45 +0000 (18:45 +0000)]
v2: fix reindex skipping NNTP article numbers
`public-inbox-index --reindex' could cause NNTP article number
gaps to form when it also has to deal with new,
never-before-seen commits in mirrors running off `git fetch'.
Fix this by running two distinct invocations of ->index_sync;
once to only reindex old commits, and a second time to index
new commits.
This does not appear to be a problem on v1 at the moment,
but I'll need more time to analyze this.
Eric Wong [Sun, 26 May 2019 08:24:46 +0000 (08:24 +0000)]
viewvcs: keep temporary Solver dir for large streams
Streaming large blobs can take multiple iterations of the event
loop in our -httpd; so we must not let the File::Temp::Dir
result go out-of-scope when streaming large blobs created from
patches.
Eric Wong [Fri, 24 May 2019 08:55:56 +0000 (08:55 +0000)]
TODO: more stuff: bundles, synonyms, dogfooding
git bundles could/should make self-hosting easier.
Being able to configure synonym (and spelling) lists would make
some searches more useful.
Might as well dogfood kernel stuff, too, given the overlap and
history between this project, git and the Linux kernel. Would
be interesting to have *BSD folks throw their hat in the ring,
too.
Building/testing userspace stuff is often the most
time-consuming, but necessary to ensure future compatibility.
Eric Wong [Fri, 24 May 2019 02:57:57 +0000 (02:57 +0000)]
doc: xcpdb: add switch documentation
In particular, the '--compact' switch is really useful since it
works without holding the inbox-wide lock for minutes at a time
on giant inboxes (inboxes where copies can take dozens, if not
hundreds of minutes).
Eric Wong [Fri, 24 May 2019 02:57:56 +0000 (02:57 +0000)]
doc: generate manpages for some Xapian commands
They're nowhere to be found on Xapian.org, and links to
external services are either too long (for manpages.debian.org)
or have privacy-invasive tracking JS on them.
Eric Wong [Fri, 24 May 2019 01:12:08 +0000 (01:12 +0000)]
search: don't log all warnings on retry_reopen
Some users (or bots :P) can trigger horrible queries which
the caller can choose to either log or ignore. This prevents
horrible queries from ExtMsg from logging confusing "ref: "
messages when $@ is not a Perl reference.
Eric Wong [Thu, 23 May 2019 10:37:38 +0000 (10:37 +0000)]
doc: various updates to reflect current state
-index documentation avoid redundant v1 information and refers
readers to apropriate v1/v2 manpages. Search::Xapian can also
be optional, now, as only the PSGI search interface uses it.
Favor "INBOX_DIR" where appropriate, since "REPO_DIR" can be
confused for code repos which we also support.
XAPIAN_FLUSH_THRESHOLD is documented for all relevant
bulk commands.
Eric Wong [Thu, 23 May 2019 09:37:04 +0000 (09:37 +0000)]
xapcmd: do not reset %SIG until last Xtmpdir is done
To properly handle compact tmpdir cleanup in single process
situations, we need to carefully account for Xtmpdir not
being a singleton and ensuring we don't clobber signal
handlers which belong to other Xtmpdirs.
Eric Wong [Thu, 23 May 2019 09:37:03 +0000 (09:37 +0000)]
xcpdb|compact: support --jobs/-j flag like gmake(1)
We don't have to be tied to the number of partitions in case
we made a bad choice at initialization. This doesn't affect
reindexing, but the copying phase is already intensive.
And optimize away the extra process when we only have a single
job which won't parallelize.
The wording for the (v2) reindexing phase could be improved,
later. I also plan to allow repartitioning of existing
Xapian DBs.
Eric Wong [Thu, 23 May 2019 09:37:01 +0000 (09:37 +0000)]
xcpdb|compact: support some xapian-compact switches
Allow users to specify the --blocksize <B>, --no-full, --fuller
options for xapian-compact(1) for fine-tuning compact behavior
for low-traffic/inactive inboxes.
We also won't support --multipass, since it doesn't seem
compatible with our requirement to use --no-renumber.
We also won't support --single-file, since it only seems
intended for totally dead inboxes; and it doesn't seem
worth the support overhead when "totally dead" turns out
to be a misdiagnosis.
Eric Wong [Thu, 23 May 2019 09:37:00 +0000 (09:37 +0000)]
compact: reuse infrastructure from xcpdb
Since -xcpdb is a superset of -compact, we can reuse much of
that code used for driving compact.
For compact (only), this is slightly less memory efficient since
it requires an extra process per-partition, but we get to prefix
the output with the partition name for more readable output.
Eric Wong [Thu, 23 May 2019 09:36:59 +0000 (09:36 +0000)]
xcpdb: remove temporary directories on aborts
Cleanup temporary directories on common termination signals
(INT, HUP, PIPE, TERM), but only if it's not in the process
of being committed via rename() sequence.
Eric Wong [Thu, 23 May 2019 09:36:58 +0000 (09:36 +0000)]
xcpdb: show re-indexing progress
Emit information about reindexing git revision ranges when used
with xcpdb. Additionally, distinguish Xapian copy output from
v2 git epoch counting by increasing directory context info.
For now, v1 batches batches are emitted. v2 indexing is still
missing progress reporting for batches, as the data structures
for reindexing would benefit from a refactoring, first.
This does not currently affect the use of public-inbox-index,
but may in the future.
Eric Wong [Thu, 23 May 2019 09:36:57 +0000 (09:36 +0000)]
xapcmd: use "print STDERR" for progress reporting
`warn' is reserved for actual warnings, as it respects
$SIG{__WARN__} and we rely on that override to print
message context information when we are indexing.
Eric Wong [Thu, 23 May 2019 09:36:55 +0000 (09:36 +0000)]
xapcmd: avoid EXDEV when finalizing changes
By creating temporary directories as deep as possible,
we can allow v2 repositories to have `xap$SCHEMA_VERSION'
(e.g. `xap15') reside on a separate FS.
We also check st_dev ahead-of-time to avoid doing work which
will fail with EXDEV. Of course, another process may still
move/change things around.
Eric Wong [Thu, 23 May 2019 09:36:53 +0000 (09:36 +0000)]
xcpdb: implement progress reporting
Copying an entire Xapian DB is horribly slow whether it's done
via Perl or copydatabase(1). So displaying some progress
indication is good for user experience.
While we're at it, prefix xapian-compact output, too; since
parallel processes end up clobbering each other.
Eric Wong [Thu, 23 May 2019 09:36:52 +0000 (09:36 +0000)]
xcpdb: use fine-grained locking
Copying an entire Xapian DB takes a long time, so update our
reindexing code to support partial reindexing, snapshot the
pre-copydatabase git revisions, perform the lengthy copy,
and do a partial reindex when the copy + renames are done.
Eric Wong [Thu, 23 May 2019 09:36:50 +0000 (09:36 +0000)]
xapcmd: xcpdb supports compaction
To minimize the delay on active inboxes, it's actually ideal to
run xapian-compact at the end of the per-partition cpdb process;
since the new DB isn't accessible yet and so we don't have to
deal with lock contention with -mda or -watch processes. The
downside is temporary file overhead (3x instead of 2x) required.
Eric Wong [Thu, 23 May 2019 09:36:49 +0000 (09:36 +0000)]
xcpdb: implement using Perl bindings
By avoid copydatabase(1) entirely, we can make further changes
to avoid locking the entire inbox for a long operation and
switch to fine-grained locking.
Eric Wong [Thu, 23 May 2019 09:36:47 +0000 (09:36 +0000)]
xapcmd: do not cleanup on errors
We move the old directory into the new directory, so avoid the
situation where a bug or error could cause the tempdir cleanup to run
and destroy both our old and new directories.
Eric Wong [Thu, 23 May 2019 09:36:46 +0000 (09:36 +0000)]
xcpdb: new tool which wraps Xapian's copydatabase(1)
copydatabase(1) is an existing Xapian tool which is the
recommended way to upgrade existing DBs to the latest Xapian
database format (currently "glass" for stable/released
versions). Our use of Xapian relies on preserving document IDs,
so we'll wrap it like we do xapian-compact(1) and use the
"--no-renumber" switch.
I could not name the tool "public-inbox-copydatabase" since it
would be ambiguous as to which DB it's actually copying. So, I
abbreviated the suffix to "xcpdb" (Xapian CoPy DataBase), which
I hope is acceptable and unambiguous.
Eric Wong [Thu, 23 May 2019 09:36:45 +0000 (09:36 +0000)]
xapcmd: support spawn options
copydatabase(1) is exceptionally noisy and it's output is
confusing when run in parallel. Support redirects at least, and
env while we're at it to give us future options.
We can also stuff a -jobs parameter into the options to limit
parallelism since it can be useful for low-priority upgrade
jobs.
Eric Wong [Thu, 23 May 2019 09:36:40 +0000 (09:36 +0000)]
v1writable: retire in favor of InboxWritable
In retrospect, introducing V1Writable was unnecessary and
InboxWritable->importer is in a better position to abstract
away differences between v1 and v2 writers.
So teach InboxWritable to initialize inboxes and get rid
of V1Writable.
Eric Wong [Wed, 22 May 2019 21:54:21 +0000 (21:54 +0000)]
Merge branch 'ds-cleanup'
* ds-cleanup:
DS: warn on deprecations
DS: remove IPPROTO_TCP import
DS: drop $VERSION var
DS: remove support OtherFds code
DS: get rid of unused methods and aliases
Eric Wong [Wed, 22 May 2019 21:46:56 +0000 (21:46 +0000)]
DS: remove support OtherFds code
It's easy enough to wrap FDs in classes that can use
all of the functionality of the event loop, not just
the read-only interface AddOtherFds provided.
Eric Wong [Tue, 21 May 2019 23:22:05 +0000 (23:22 +0000)]
Merge remote-tracking branch 'origin/xap-optional' into master
* origin/xap-optional:
admin: improve warnings and errors for missing modules
searchidx: do not create empty Xapian partitions for basic
lazy load Xapian and make it optional for v2
www: use Inbox->over where appropriate
nntp: use Inbox->over directly
inbox: add ->over method to ease access
Eric Wong [Thu, 16 May 2019 01:23:29 +0000 (01:23 +0000)]
view: perform highlighting for space-prefixed diffs
"git format-patch --interdiff" and similar can prefix diffs
with leading white space. Teach our diff parser to account
for it and set appropriate CSS classes for them.
Eric Wong [Wed, 15 May 2019 01:18:07 +0000 (01:18 +0000)]
www: use Inbox->over where appropriate
We don't need to rely on Xapian search functionality for the
majority of the WWW code, even. subject_normalized is moved to
SearchMsg, where it (probably) makes more sense, anyways.
Eric Wong [Tue, 14 May 2019 03:32:00 +0000 (03:32 +0000)]
tests: remove unnecessary loading of ::DS and Socket
PublicInbox::DS works for every platform we we care about,
nowadays; so checking for it is a waste of time. Cleanup a
few POSIX and Socket imports while we're in the area.
Eric Wong [Tue, 14 May 2019 02:04:43 +0000 (02:04 +0000)]
searchidx: fix incremental index with indexlevel=basic on v1
We were reindexing the full history every invocation of -index
when Xapian was not used because we were incorrectly relying on
'last_commit' metadata stored in Xapian.
Rewrite the indexing logic to be less confusing while we're
at it, since we rely on `git merge-base --is-ancestor' nowadays.
Furthermore, we need to handle message removals from the
overview index correctly when Xapian is not in use.
Co-authored-by: Eric W. Biederman <ebiederm@xmission.com>
Eric Wong [Tue, 14 May 2019 02:04:41 +0000 (02:04 +0000)]
v1writable: new wrapper which is closer to v2writable
Import initialization is a little strange from history, but we
also can't change it too much because it's technically a public
API which external code may rely on...
And we may need to support v1 repos indefinitely. This should
make it easier to write tests for both formats.
Eric Wong [Tue, 7 May 2019 09:09:54 +0000 (09:09 +0000)]
ci: include some scripts to make CI easier
This should make it easier to test a bunch of package
installation profiles across whatever OS isolation
one chooses (chroots, containers, jails, VMs).
Eric Wong [Wed, 8 May 2019 08:34:36 +0000 (08:34 +0000)]
doc/include.mk: remove git use and redundant declarations
And document that we still have GNU-isms in that
include.mk Makefile (and may continue to do so).
Finally, take advantage of GNU-isms to warn users
to run "gmake" to build all manpages.
Eric Wong [Wed, 8 May 2019 02:47:20 +0000 (02:47 +0000)]
INSTALL: depend on Encode instead of Encode::MIME::Header
Email::MIME uses Encode::MIME::Header and depends on that
appropriately; however we depend on other parts of the Encode
distribution, but that's bundled with Perl by upstream, anyways;
and should place no additional burden on users.
Eric Wong [Sun, 5 May 2019 01:37:00 +0000 (01:37 +0000)]
INSTALL: update with FreeBSD pkg names and reorder
Fortunately, there is a pattern to most of these package names
in all distros I've tested (and hopefully other BSDs have them,
too).
Then, reorder the INSTALL document to mention the top-level
modules, first, so users can avoid needing to type extra
dependencies. However, we still list some implicit dependencies
in case the upstream package drops dependencies independently of
us.
Finally, Devel::Peek is not a dependency worth making optional
since it's bundled by Perl upstream. Fedora/RH-based distros
are the only one which turn it into a non-standard package when
Perl5 is installed.
* origin/danga-bundle:
DS: epoll: fix misordered EPOLL_CTL_DEL call
DS: drop unused "_undef" sub
syscall: drop readahead wrapper
build: do not manify DS and Syscall pods
DS: handle EINTR in IO::Poll path, too
DS: workaround IO::Kqueue EINTR (mis-)handling
DS: drop profiling support
DS: remove unused fields and functions
listener: use EPOLLEXCLUSIVE for listen sockets
bundle Danga::Socket and Sys::Syscall
Eric Wong [Wed, 8 May 2019 19:04:17 +0000 (19:04 +0000)]
DS: epoll: fix misordered EPOLL_CTL_DEL call
Any operations on an fd after POSIX::close() are invalid, so
epoll_ctl will fail. Worse off, in a multi-threaded Perl, the
fd may be reused by another thread and EPOLL_CTL_DEL can hit the
wrong file description as a result.