Eric Wong [Sat, 1 Jun 2019 21:43:47 +0000 (21:43 +0000)]
ds: fix and test for FD leaks with kqueue on ->Reset
Even though we currently don't use it repeatedly, ->Reset
should close() kqueue FDs and not cause the process to run
out of descriptors.
Add a close-on-exec test while we're at it.
Eric Wong [Sat, 1 Jun 2019 07:28:17 +0000 (07:28 +0000)]
doc/v2: note that Xapian is now optional in v2
This is true as of
e220b8b2ee5cfd458167dc2c6c92726352c4c80e
("Merge remote-tracking branch 'origin/xap-optional' into master")
Eric Wong [Sat, 1 Jun 2019 00:23:22 +0000 (00:23 +0000)]
ds: set close-on-exec flag on epoll descriptors
We should not be leaking these FDs to git(1) processes,
in case git has a bug that causes it to access the wrong FD.
Eric Wong [Sat, 1 Jun 2019 03:27:55 +0000 (03:27 +0000)]
git: drop the deleted err_c file
No reason to leave that (usually) empty file open after killing off
"cat-file --batch-check". This wasn't an unbound leak, though,
as respawning the --batch-check process would've clobbered the
old err_c file.
Eric Wong [Sat, 1 Jun 2019 00:20:51 +0000 (00:20 +0000)]
git: unconditional expiry
A constant stream of traffic to either httpd/nntpd would mean
git-cat-file processes never expire. Things can go bad after a
full repack, as a full repack will unlink old pack indices and
git-cat-file does not currently detect unlinked files.
We could do something complicated by recursively stat-ing
objects/pack of every git directory and alternate;
but that's probably not worth the trouble compared to
occasionally restarting the cat-file process.
So simplify the code and let httpd/nntpd expire them
periodically, since spawning a "git-cat-file --batch" process
isn't too expensive. We already spawn for every request which
hits git-http-backend, cgit, and git-apply.
In the future, we may optionally support the Git::Raw module
to avoid IPC; but we must remain careful to not leave lingering
FDs open to unlinked files after repack.
Eric Wong [Fri, 31 May 2019 16:58:46 +0000 (16:58 +0000)]
TODO: add item for optional Cache::FastMmap
Taking one step out of setting up a performant deployment could
make setup and administration easier (at the cost of installing
an extra-but-common XS module). This can also be useful for
the day NNTP servers see hug-of-death events.
Eric Wong [Fri, 31 May 2019 16:46:04 +0000 (16:46 +0000)]
viewdiff: avoid repeat variable expansion
This is worth a 1-2% speedup in t/perf-msgview.t rendering 2620
messages currently in https://public-inbox.org/meta/
Eric Wong [Thu, 30 May 2019 19:50:04 +0000 (19:50 +0000)]
Merge remote-tracking branch 'origin/v2-noop-speedup'
* origin/v2-noop-speedup:
v2writable: short-circuit is_ancestor check on equality
v2writable: avoid mm_tmp creation without regen
v2writable: hoist out index_epoch sub
v2writable: split off unindex_range mapping
Eric Wong [Thu, 30 May 2019 06:54:06 +0000 (06:54 +0000)]
doc/hosted: drop some links and clarify wording
I don't have time to check and train spam for all these
projects.
Spam filtering is especially difficult on ruby-core: it
enters via Redmine, so it doesn't have a distinct Received:
chain, and also gets mixed with non-spam bug-report text,
throwing off Bayes training.
And I'm not sure if those mirrors did anybody any good, even;
so lets not say its' a "service" to anybody :P
The actual mirrors remain up, for now, but who knows...
I care about decentralization too much to ask anybody
to trust me to keep anything up :P
Eric Wong [Thu, 30 May 2019 06:35:05 +0000 (06:35 +0000)]
v2writable: short-circuit is_ancestor check on equality
We don't need to use git to check ancestry if object IDs
match on a string comparison.
This saves 100ms or so and brings down the ~0.5s no-op time on
lore.kernel.org/lkml down to ~0.4s.
Eric Wong [Thu, 30 May 2019 03:59:40 +0000 (03:59 +0000)]
v2writable: avoid mm_tmp creation without regen
Creating mm_tmp is an expensive operation with large inboxes
and can be avoided if there are no new messages to process.
Since git-fetch(1) currently lacks an --exit-code option(*),
mirrors will run `public-inbox-index' unconditionally after
fetch, which is an expensive op if it needs to duplicate
a large SQLite DB.
This speeds up the mirror case of:
git --git-dir=git/$EPOCH.git fetch && public-inbox-index
This reduces the no-op `public-inbox-index' time from over 8s to
~0.5s on a (currently) 7-epoch clone of https://lore.kernel.org/lkml/
on my system.
(*) WIP --exit-code for git-fetch:
https://public-inbox.org/git/87ftphw7mv.fsf@evledraar.gmail.com/
Eric Wong [Thu, 30 May 2019 03:59:39 +0000 (03:59 +0000)]
v2writable: hoist out index_epoch sub
This will make future changes easier-to-follow.
Eric Wong [Thu, 30 May 2019 03:59:38 +0000 (03:59 +0000)]
v2writable: split off unindex_range mapping
It'll make it easier to detect if we have anything to
unindex and run git-log on, at all.
Eric Wong [Wed, 29 May 2019 20:56:32 +0000 (20:56 +0000)]
searchidx: store indexlevel=medium as metadata
And use it from Admin.
It's easy to tell what indexlevel=basic is from unconfigured
inboxes, but distinguishing between 'medium' and 'full' would
require stat()-ing position.* files which is fragile and
Xapian-implementation-dependent.
So use the metadata facility of Xapian and store it in the main
partition so Admin tools can deal better with unconfigured
inboxes copied using generic tools like cp(1) or rsync(1).
Eric Wong [Wed, 29 May 2019 20:56:31 +0000 (20:56 +0000)]
index: remove warning on unconfigured inboxes
It's annoying for people using "git fetch && public-inbox-index"
as one user; and running -httpd/-nntpd as a different user
(where users see different config files).
Eric Wong [Wed, 29 May 2019 08:45:49 +0000 (08:45 +0000)]
Merge branch 'v2-idx-progress'
* v2-idx-progress:
v2writable: show progress updates for index_sync
index: support --verbose option
v2writable: move index_sync options to sync state
v2writable: use prototypes for internal subs
v2writable: localize unindex-range.$EPOCH to $sync state
v2writable: move {ranges} into $sync state
v2writable: move {regen} into $sync state
v2writable: move {reindex} field to $sync state
v2writable: sync: move delete markers into $sync state
v2writable: introduce $sync state and put mm_tmp in it
Eric Wong [Wed, 29 May 2019 08:37:54 +0000 (08:37 +0000)]
Makefile.PL: enable prove warnings
We already "use warnings" everywhere, but could miss some spots.
This ought to cover that, and usually Perl module authors are
consistent about avoiding warnings that we won't clutter our
test outputs.
Eric Wong [Wed, 29 May 2019 08:29:30 +0000 (08:29 +0000)]
Makefile.PL: allow `N' variable to be set in local config.mak
This can useful for limiting test resource use without relying
on remembering the variable command-line.
Eric Wong [Tue, 28 May 2019 23:48:55 +0000 (23:48 +0000)]
v2writable: show progress updates for index_sync
We can show progress whenever we commit changes to the FS.
Eric Wong [Tue, 28 May 2019 23:48:52 +0000 (23:48 +0000)]
index: support --verbose option
It doesn't implement progress of batches, yet, but it wires
up the parsing of the command-line while preserving output
compatibility.
This output is NOT meant to be stable.
Eric Wong [Tue, 28 May 2019 23:48:51 +0000 (23:48 +0000)]
v2writable: move index_sync options to sync state
And use singular `opt' to be consistent with the common name
of 'getopt'.
Eric Wong [Tue, 28 May 2019 23:48:50 +0000 (23:48 +0000)]
v2writable: use prototypes for internal subs
Hopefully this improves maintainability by allowing Perl
to do some arg checking for us.
Eric Wong [Tue, 28 May 2019 23:48:49 +0000 (23:48 +0000)]
v2writable: localize unindex-range.$EPOCH to $sync state
We don't need to stuff that into $self (V2Writable) which can be
longer-lived than a ->index_sync invocation.
Eric Wong [Tue, 28 May 2019 23:48:48 +0000 (23:48 +0000)]
v2writable: move {ranges} into $sync state
Yet another temporary variable with no use outside of index_sync.
Eric Wong [Tue, 28 May 2019 23:48:47 +0000 (23:48 +0000)]
v2writable: move {regen} into $sync state
regen is always enabled for index_sync nowadays (and has
been for a while).
Rename `index_prepare' to `sync_prepare' to show it's for
->index_sync; and not the online indexing we do for ->add.
Eric Wong [Tue, 28 May 2019 23:48:46 +0000 (23:48 +0000)]
v2writable: move {reindex} field to $sync state
reindexing info is not used outside of the index_sync code path.
Eric Wong [Tue, 28 May 2019 23:48:45 +0000 (23:48 +0000)]
v2writable: sync: move delete markers into $sync state
Another small step to reduce parameters passed to reindex_oid.
Eric Wong [Tue, 28 May 2019 23:48:44 +0000 (23:48 +0000)]
v2writable: introduce $sync state and put mm_tmp in it
A first step towards making the v2 index_sync code
easier-to-follow. More fields to follow...
Eric Wong [Mon, 27 May 2019 18:45:45 +0000 (18:45 +0000)]
v2: fix reindex skipping NNTP article numbers
`public-inbox-index --reindex' could cause NNTP article number
gaps to form when it also has to deal with new,
never-before-seen commits in mirrors running off `git fetch'.
Fix this by running two distinct invocations of ->index_sync;
once to only reindex old commits, and a second time to index
new commits.
This does not appear to be a problem on v1 at the moment,
but I'll need more time to analyze this.
Eric Wong [Mon, 27 May 2019 18:45:44 +0000 (18:45 +0000)]
searchidx: fix obvious typo
We can't pass an empty string to `git merge-base --is-ancestor'
AFAIK, this did NOT present issues in the current test suite.
Eric Wong [Mon, 27 May 2019 18:45:43 +0000 (18:45 +0000)]
t/v1reindex.t: fix typo in setting `indexlevel'
It did not cause a test failure because the default fallback
is `indexlevel=full'
Eric Wong [Sun, 26 May 2019 08:24:46 +0000 (08:24 +0000)]
viewvcs: keep temporary Solver dir for large streams
Streaming large blobs can take multiple iterations of the event
loop in our -httpd; so we must not let the File::Temp::Dir
result go out-of-scope when streaming large blobs created from
patches.
Eric Wong [Sat, 25 May 2019 19:18:09 +0000 (19:18 +0000)]
v2writable: fix assertions around reindexing
Fix a misspelling and ensure line context is printed by
`die' by leaving out the final '\n'. Also, `delete' was
pointless.
Eric Wong [Sat, 25 May 2019 19:10:00 +0000 (19:10 +0000)]
contrib/css: mark as CC0 (public domain)
No reason to copyright colour schemes :P
Eric Wong [Sat, 25 May 2019 09:42:03 +0000 (09:42 +0000)]
v2writable: drop unused $last_commits var
Apparently it's never been used and we write to msgmap directly.
Eric Wong [Sat, 25 May 2019 09:42:08 +0000 (09:42 +0000)]
t/indexlevels: fix indexlevel of ro_mirror
Don't hard-code "basic", since we already ran -init with the
intended indexlevel.
Eric Wong [Sat, 25 May 2019 04:34:20 +0000 (04:34 +0000)]
msgmap: remove double negative
I have never not found double negatives to be confusing...
Eric Wong [Fri, 24 May 2019 08:55:56 +0000 (08:55 +0000)]
TODO: more stuff: bundles, synonyms, dogfooding
git bundles could/should make self-hosting easier.
Being able to configure synonym (and spelling) lists would make
some searches more useful.
Might as well dogfood kernel stuff, too, given the overlap and
history between this project, git and the Linux kernel. Would
be interesting to have *BSD folks throw their hat in the ring,
too.
Building/testing userspace stuff is often the most
time-consuming, but necessary to ensure future compatibility.
Eric Wong [Fri, 24 May 2019 07:55:29 +0000 (07:55 +0000)]
MANIFEST: add extman.perl
Oops :x
Eric Wong [Fri, 24 May 2019 02:57:58 +0000 (02:57 +0000)]
doc: add URLs for Xapian manpages
Since we go through the effort of hosting these manpages,
link to them.
Eric Wong [Fri, 24 May 2019 02:57:57 +0000 (02:57 +0000)]
doc: xcpdb: add switch documentation
In particular, the '--compact' switch is really useful since it
works without holding the inbox-wide lock for minutes at a time
on giant inboxes (inboxes where copies can take dozens, if not
hundreds of minutes).
Eric Wong [Fri, 24 May 2019 02:57:56 +0000 (02:57 +0000)]
doc: generate manpages for some Xapian commands
They're nowhere to be found on Xapian.org, and links to
external services are either too long (for manpages.debian.org)
or have privacy-invasive tracking JS on them.
Eric Wong [Fri, 24 May 2019 02:57:55 +0000 (02:57 +0000)]
doc: sync .txt mtime to .pod mtime
Otherwise timestamps for .html files get screwed up, too;
and that hurts caching.
Eric Wong [Fri, 24 May 2019 02:57:54 +0000 (02:57 +0000)]
doc: don't barf on missing `git set-file-times'
It's not critical, but it's nice to have for cache-friendliness
(otherwise I would not have written it :P)
I guess I should follow up on getting it into 'git contrib/':
https://public-inbox.org/git/
20100702033709.GA6818@burratino/
Eric Wong [Fri, 24 May 2019 02:57:53 +0000 (02:57 +0000)]
doc: daemon: fix manpage section for nginx
The nginx manpage is in section 8.
Eric Wong [Fri, 24 May 2019 02:57:52 +0000 (02:57 +0000)]
doc: index: fix miscapitalization of "SQLite"
Oops :x
Eric Wong [Fri, 24 May 2019 01:12:08 +0000 (01:12 +0000)]
search: don't log all warnings on retry_reopen
Some users (or bots :P) can trigger horrible queries which
the caller can choose to either log or ignore. This prevents
horrible queries from ExtMsg from logging confusing "ref: "
messages when $@ is not a Perl reference.
Eric Wong [Thu, 23 May 2019 10:37:38 +0000 (10:37 +0000)]
doc: various updates to reflect current state
-index documentation avoid redundant v1 information and refers
readers to apropriate v1/v2 manpages. Search::Xapian can also
be optional, now, as only the PSGI search interface uses it.
Favor "INBOX_DIR" where appropriate, since "REPO_DIR" can be
confused for code repos which we also support.
XAPIAN_FLUSH_THRESHOLD is documented for all relevant
bulk commands.
Eric Wong [Thu, 23 May 2019 09:37:04 +0000 (09:37 +0000)]
xapcmd: do not reset %SIG until last Xtmpdir is done
To properly handle compact tmpdir cleanup in single process
situations, we need to carefully account for Xtmpdir not
being a singleton and ensuring we don't clobber signal
handlers which belong to other Xtmpdirs.
Eric Wong [Thu, 23 May 2019 09:37:03 +0000 (09:37 +0000)]
xcpdb|compact: support --jobs/-j flag like gmake(1)
We don't have to be tied to the number of partitions in case
we made a bad choice at initialization. This doesn't affect
reindexing, but the copying phase is already intensive.
And optimize away the extra process when we only have a single
job which won't parallelize.
The wording for the (v2) reindexing phase could be improved,
later. I also plan to allow repartitioning of existing
Xapian DBs.
Eric Wong [Thu, 23 May 2019 09:37:02 +0000 (09:37 +0000)]
xapcmd: cleanup on interrupted xcpdb "--compact"
We should not have leftover junk on interrupted invocations.
Eric Wong [Thu, 23 May 2019 09:37:01 +0000 (09:37 +0000)]
xcpdb|compact: support some xapian-compact switches
Allow users to specify the --blocksize <B>, --no-full, --fuller
options for xapian-compact(1) for fine-tuning compact behavior
for low-traffic/inactive inboxes.
We also won't support --multipass, since it doesn't seem
compatible with our requirement to use --no-renumber.
We also won't support --single-file, since it only seems
intended for totally dead inboxes; and it doesn't seem
worth the support overhead when "totally dead" turns out
to be a misdiagnosis.
Eric Wong [Thu, 23 May 2019 09:37:00 +0000 (09:37 +0000)]
compact: reuse infrastructure from xcpdb
Since -xcpdb is a superset of -compact, we can reuse much of
that code used for driving compact.
For compact (only), this is slightly less memory efficient since
it requires an extra process per-partition, but we get to prefix
the output with the partition name for more readable output.
Eric Wong [Thu, 23 May 2019 09:36:59 +0000 (09:36 +0000)]
xcpdb: remove temporary directories on aborts
Cleanup temporary directories on common termination signals
(INT, HUP, PIPE, TERM), but only if it's not in the process
of being committed via rename() sequence.
Eric Wong [Thu, 23 May 2019 09:36:58 +0000 (09:36 +0000)]
xcpdb: show re-indexing progress
Emit information about reindexing git revision ranges when used
with xcpdb. Additionally, distinguish Xapian copy output from
v2 git epoch counting by increasing directory context info.
For now, v1 batches batches are emitted. v2 indexing is still
missing progress reporting for batches, as the data structures
for reindexing would benefit from a refactoring, first.
This does not currently affect the use of public-inbox-index,
but may in the future.
Eric Wong [Thu, 23 May 2019 09:36:57 +0000 (09:36 +0000)]
xapcmd: use "print STDERR" for progress reporting
`warn' is reserved for actual warnings, as it respects
$SIG{__WARN__} and we rely on that override to print
message context information when we are indexing.
Eric Wong [Thu, 23 May 2019 09:36:56 +0000 (09:36 +0000)]
doc: xcpdb: update to reflect the current state
It is no longer a wrapper around copydatabase(1), since
copydatabase did not recover from DatabaseModifiedError.
Eric Wong [Thu, 23 May 2019 09:36:55 +0000 (09:36 +0000)]
xapcmd: avoid EXDEV when finalizing changes
By creating temporary directories as deep as possible,
we can allow v2 repositories to have `xap$SCHEMA_VERSION'
(e.g. `xap15') reside on a separate FS.
We also check st_dev ahead-of-time to avoid doing work which
will fail with EXDEV. Of course, another process may still
move/change things around.
Eric Wong [Thu, 23 May 2019 09:36:54 +0000 (09:36 +0000)]
xcpdb: cleanup error handling and diagnosis
Running a full "public-inbox-index --reindex" in parallel
with "public-inbox-xcpdb" on the same inbox can still cause
problems, though.
Eric Wong [Thu, 23 May 2019 09:36:53 +0000 (09:36 +0000)]
xcpdb: implement progress reporting
Copying an entire Xapian DB is horribly slow whether it's done
via Perl or copydatabase(1). So displaying some progress
indication is good for user experience.
While we're at it, prefix xapian-compact output, too; since
parallel processes end up clobbering each other.
Eric Wong [Thu, 23 May 2019 09:36:52 +0000 (09:36 +0000)]
xcpdb: use fine-grained locking
Copying an entire Xapian DB takes a long time, so update our
reindexing code to support partial reindexing, snapshot the
pre-copydatabase git revisions, perform the lengthy copy,
and do a partial reindex when the copy + renames are done.
Eric Wong [Thu, 23 May 2019 09:36:51 +0000 (09:36 +0000)]
v2writable: hoist out log_range sub for readability
This is preparation to to support partial reindexing
Eric Wong [Thu, 23 May 2019 09:36:50 +0000 (09:36 +0000)]
xapcmd: xcpdb supports compaction
To minimize the delay on active inboxes, it's actually ideal to
run xapian-compact at the end of the per-partition cpdb process;
since the new DB isn't accessible yet and so we don't have to
deal with lock contention with -mda or -watch processes. The
downside is temporary file overhead (3x instead of 2x) required.
Eric Wong [Thu, 23 May 2019 09:36:49 +0000 (09:36 +0000)]
xcpdb: implement using Perl bindings
By avoid copydatabase(1) entirely, we can make further changes
to avoid locking the entire inbox for a long operation and
switch to fine-grained locking.
Eric Wong [Thu, 23 May 2019 09:36:48 +0000 (09:36 +0000)]
admin: move index_inbox over
We will be reindexing after copydatabase
Eric Wong [Thu, 23 May 2019 09:36:47 +0000 (09:36 +0000)]
xapcmd: do not cleanup on errors
We move the old directory into the new directory, so avoid the
situation where a bug or error could cause the tempdir cleanup to run
and destroy both our old and new directories.
Eric Wong [Thu, 23 May 2019 09:36:46 +0000 (09:36 +0000)]
xcpdb: new tool which wraps Xapian's copydatabase(1)
copydatabase(1) is an existing Xapian tool which is the
recommended way to upgrade existing DBs to the latest Xapian
database format (currently "glass" for stable/released
versions). Our use of Xapian relies on preserving document IDs,
so we'll wrap it like we do xapian-compact(1) and use the
"--no-renumber" switch.
I could not name the tool "public-inbox-copydatabase" since it
would be ambiguous as to which DB it's actually copying. So, I
abbreviated the suffix to "xcpdb" (Xapian CoPy DataBase), which
I hope is acceptable and unambiguous.
Eric Wong [Thu, 23 May 2019 09:36:45 +0000 (09:36 +0000)]
xapcmd: support spawn options
copydatabase(1) is exceptionally noisy and it's output is
confusing when run in parallel. Support redirects at least, and
env while we're at it to give us future options.
We can also stuff a -jobs parameter into the options to limit
parallelism since it can be useful for low-priority upgrade
jobs.
Eric Wong [Thu, 23 May 2019 09:36:44 +0000 (09:36 +0000)]
admin: hoist out resolve_inboxes for -compact and -index
Both of these index-affecting commands should work similarly
on the command-line.
public-inbox-index no longer complains about unconfigured
~/.public-inbox/config; but often I found myself being
annoyed by that, anyways...
Eric Wong [Thu, 23 May 2019 09:36:43 +0000 (09:36 +0000)]
xapcmd: new module for wrapping Xapian commands
Port public-inbox-compact(1) over to using it, and we will need
to wrap copydatabase(1) to ease glass migrations, too.
Eric Wong [Thu, 23 May 2019 09:36:42 +0000 (09:36 +0000)]
search: reenable phrase search on non-chert Xapian
This is assuming nobody uses flint or earlier, anymore;
as flint predates the existence of this project.
Eric Wong [Thu, 23 May 2019 09:36:41 +0000 (09:36 +0000)]
doc: document the reason for --no-renumber
We're going to need copydatabase, too
Eric Wong [Thu, 23 May 2019 09:36:40 +0000 (09:36 +0000)]
v1writable: retire in favor of InboxWritable
In retrospect, introducing V1Writable was unnecessary and
InboxWritable->importer is in a better position to abstract
away differences between v1 and v2 writers.
So teach InboxWritable to initialize inboxes and get rid
of V1Writable.
Eric Wong [Thu, 23 May 2019 09:36:39 +0000 (09:36 +0000)]
t/convert-compact: skip on missing xapian-compact(1)
Can't run the test if the required Xapian tools are missing.
Eric Wong [Wed, 22 May 2019 21:54:21 +0000 (21:54 +0000)]
Merge branch 'ds-cleanup'
* ds-cleanup:
DS: warn on deprecations
DS: remove IPPROTO_TCP import
DS: drop $VERSION var
DS: remove support OtherFds code
DS: get rid of unused methods and aliases
Eric Wong [Wed, 22 May 2019 21:46:59 +0000 (21:46 +0000)]
DS: warn on deprecations
Enabling deprecation warnings didn't seem to have any
noticeable effects with "perl -w -c", so whatever reason
Danga had for it is long irrelevant.
Eric Wong [Wed, 22 May 2019 21:46:58 +0000 (21:46 +0000)]
DS: remove IPPROTO_TCP import
Unlike Danga::Socket, we do not support TCP_CORK, either
Eric Wong [Wed, 22 May 2019 21:46:57 +0000 (21:46 +0000)]
DS: drop $VERSION var
It was only relevant to Danga::Socket.
Eric Wong [Wed, 22 May 2019 21:46:56 +0000 (21:46 +0000)]
DS: remove support OtherFds code
It's easy enough to wrap FDs in classes that can use
all of the functionality of the event loop, not just
the read-only interface AddOtherFds provided.
Eric Wong [Wed, 22 May 2019 21:46:55 +0000 (21:46 +0000)]
DS: get rid of unused methods and aliases
"make syntax" is clean, now
Eric Wong [Wed, 22 May 2019 07:27:52 +0000 (07:27 +0000)]
usercontent: stop relying on autodie
It's a non-standard package on CentOS-7, actually; and we
shouldn't bloat the PSGI server by loading a module which
isn't strictly needed.
Eric Wong [Wed, 22 May 2019 02:14:02 +0000 (02:14 +0000)]
ci: support CentOS-7
Tested on an amd64 chroot built with rinse 3.4
Eric Wong [Wed, 22 May 2019 02:14:01 +0000 (02:14 +0000)]
t/search*: require DBI and DBD::SQLite, too
None of the Search::Xapian-dependent stuff works without DBI
and DBD::SQLite.
There are no plans to support Xapian w/o DBD::SQLite since
SQLite is more common and less resource-intensive than Xapian.
Eric Wong [Wed, 22 May 2019 02:14:00 +0000 (02:14 +0000)]
git: workaround old git-rev-parse(1) (--git-path)
git < 2.5.0 was missing --git-path support. This means any
users relying on some rare environment variables will need git
2.5.0+
Eric Wong [Wed, 22 May 2019 02:13:59 +0000 (02:13 +0000)]
t/watch_filter_rubylang: disable v2 test for git < 2.6
This test was not disabled properly for ancient versions of
git without get-mark support.
Eric Wong [Wed, 22 May 2019 02:13:58 +0000 (02:13 +0000)]
init: preserve permissions for git prior to 2.1.0
"git config" did not preserve permissions of the config file it
modifies prior to git 2.1.0, so workaround that.
Eric Wong [Tue, 21 May 2019 23:22:05 +0000 (23:22 +0000)]
Merge remote-tracking branch 'origin/xap-optional' into master
* origin/xap-optional:
admin: improve warnings and errors for missing modules
searchidx: do not create empty Xapian partitions for basic
lazy load Xapian and make it optional for v2
www: use Inbox->over where appropriate
nntp: use Inbox->over directly
inbox: add ->over method to ease access
Eric Wong [Tue, 21 May 2019 06:54:27 +0000 (06:54 +0000)]
searchidx: remove unused Compress::Zlib import
We only compress in OverIdx, now; since we no longer do overview
stuff in Xapian (and Xapian compresses document data, anyways).
Eric W. Biederman [Fri, 17 May 2019 01:53:06 +0000 (20:53 -0500)]
PublicInbox::Import::add: Consolidate subject handling
Consolidate subject handling in the add function to make it easier to
read and understand.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
Eric Wong [Thu, 16 May 2019 07:06:21 +0000 (07:06 +0000)]
www: unescape '+' => ' ' before general URI unescape
This allows searching for terms with "+" in them properly.
Eric Wong [Thu, 16 May 2019 06:54:08 +0000 (06:54 +0000)]
Revert "view: perform highlighting for space-prefixed diffs"
This was buggy and was causing non-diff text to have extra
leading spaces. The diff parsing code needs to be cleaned up,
so this will be fixed, later.
This reverts commit
1a67b91c1326efa372d1ec957e2494849d894f0b.
Eric Wong [Thu, 16 May 2019 03:40:47 +0000 (03:40 +0000)]
search: disable phrase searching, for now
There probably needs to be an option to enable this
independently of indexlevel; but for now this is
the safest option.
And, as I discovered during the development of the
indexlevel option, Xapian does a pretty good job of
finding phrases without position data, anyways.
Eric Wong [Thu, 16 May 2019 01:23:29 +0000 (01:23 +0000)]
view: perform highlighting for space-prefixed diffs
"git format-patch --interdiff" and similar can prefix diffs
with leading white space. Teach our diff parser to account
for it and set appropriate CSS classes for them.
Eric Wong [Wed, 15 May 2019 10:39:19 +0000 (10:39 +0000)]
remove hard Devel::Peek dependency and lazy load for daemons
It's only useful for a corner case in long-running daemons when
an admin decides to compact or vacuum a Xapian or SQLite DB.
As a result, other scripts should run slightly faster. For
instance, this saves about 80ms (2.710s => 2.630s) in t/mda.t
on my remote workstation.
While we're at it, make sure EvCleanup is properly require'd
in Daemon.pm and HTTP.pm and document our use of Devel::Peek.
Eric Wong [Wed, 15 May 2019 10:39:01 +0000 (10:39 +0000)]
inbox: remove POSIX strftime import
We no longer need it since Inbox->recent hits the overview
DB instead of Xapian
Eric Wong [Thu, 9 May 2019 09:57:37 +0000 (09:57 +0000)]
doc: add marketing guide
Preventative measures; since marketing is almost always annoying
to me. And trying to avoid unintended consequences.
Eric Wong [Wed, 15 May 2019 01:18:09 +0000 (01:18 +0000)]
admin: improve warnings and errors for missing modules
Since we lazy-load Xapian now, some errors may become
more cryptic or buried. Try to improve that by making
Admin show better errors.
Eric Wong [Wed, 15 May 2019 01:18:10 +0000 (01:18 +0000)]
searchidx: do not create empty Xapian partitions for basic
No point in leaving a mess of empty directories when Xapian
doesn't load.
Eric Wong [Wed, 15 May 2019 01:18:08 +0000 (01:18 +0000)]
lazy load Xapian and make it optional for v2
More tests work without Search::Xapian, now.
Usability issues still need to be fixed
Eric Wong [Wed, 15 May 2019 01:18:07 +0000 (01:18 +0000)]
www: use Inbox->over where appropriate
We don't need to rely on Xapian search functionality for the
majority of the WWW code, even. subject_normalized is moved to
SearchMsg, where it (probably) makes more sense, anyways.