]> Sergey Matveev's repositories - public-inbox.git/log
public-inbox.git
5 years agov2writable: move {ranges} into $sync state
Eric Wong [Tue, 28 May 2019 23:48:48 +0000 (23:48 +0000)]
v2writable: move {ranges} into $sync state

Yet another temporary variable with no use outside of index_sync.

5 years agov2writable: move {regen} into $sync state
Eric Wong [Tue, 28 May 2019 23:48:47 +0000 (23:48 +0000)]
v2writable: move {regen} into $sync state

regen is always enabled for index_sync nowadays (and has
been for a while).

Rename `index_prepare' to `sync_prepare' to show it's for
->index_sync; and not the online indexing we do for ->add.

5 years agov2writable: move {reindex} field to $sync state
Eric Wong [Tue, 28 May 2019 23:48:46 +0000 (23:48 +0000)]
v2writable: move {reindex} field to $sync state

reindexing info is not used outside of the index_sync code path.

5 years agov2writable: sync: move delete markers into $sync state
Eric Wong [Tue, 28 May 2019 23:48:45 +0000 (23:48 +0000)]
v2writable: sync: move delete markers into $sync state

Another small step to reduce parameters passed to reindex_oid.

5 years agov2writable: introduce $sync state and put mm_tmp in it
Eric Wong [Tue, 28 May 2019 23:48:44 +0000 (23:48 +0000)]
v2writable: introduce $sync state and put mm_tmp in it

A first step towards making the v2 index_sync code
easier-to-follow.  More fields to follow...

5 years agov2: fix reindex skipping NNTP article numbers
Eric Wong [Mon, 27 May 2019 18:45:45 +0000 (18:45 +0000)]
v2: fix reindex skipping NNTP article numbers

`public-inbox-index --reindex' could cause NNTP article number
gaps to form when it also has to deal with new,
never-before-seen commits in mirrors running off `git fetch'.

Fix this by running two distinct invocations of ->index_sync;
once to only reindex old commits, and a second time to index
new commits.

This does not appear to be a problem on v1 at the moment,
but I'll need more time to analyze this.

5 years agosearchidx: fix obvious typo
Eric Wong [Mon, 27 May 2019 18:45:44 +0000 (18:45 +0000)]
searchidx: fix obvious typo

We can't pass an empty string to `git merge-base --is-ancestor'
AFAIK, this did NOT present issues in the current test suite.

5 years agot/v1reindex.t: fix typo in setting `indexlevel'
Eric Wong [Mon, 27 May 2019 18:45:43 +0000 (18:45 +0000)]
t/v1reindex.t: fix typo in setting `indexlevel'

It did not cause a test failure because the default fallback
is `indexlevel=full'

5 years agoviewvcs: keep temporary Solver dir for large streams
Eric Wong [Sun, 26 May 2019 08:24:46 +0000 (08:24 +0000)]
viewvcs: keep temporary Solver dir for large streams

Streaming large blobs can take multiple iterations of the event
loop in our -httpd; so we must not let the File::Temp::Dir
result go out-of-scope when streaming large blobs created from
patches.

5 years agov2writable: fix assertions around reindexing
Eric Wong [Sat, 25 May 2019 19:18:09 +0000 (19:18 +0000)]
v2writable: fix assertions around reindexing

Fix a misspelling and ensure line context is printed by
`die' by leaving out the final '\n'.  Also, `delete' was
pointless.

5 years agocontrib/css: mark as CC0 (public domain)
Eric Wong [Sat, 25 May 2019 19:10:00 +0000 (19:10 +0000)]
contrib/css: mark as CC0 (public domain)

No reason to copyright colour schemes :P

5 years agov2writable: drop unused $last_commits var
Eric Wong [Sat, 25 May 2019 09:42:03 +0000 (09:42 +0000)]
v2writable: drop unused $last_commits var

Apparently it's never been used and we write to msgmap directly.

5 years agot/indexlevels: fix indexlevel of ro_mirror
Eric Wong [Sat, 25 May 2019 09:42:08 +0000 (09:42 +0000)]
t/indexlevels: fix indexlevel of ro_mirror

Don't hard-code "basic", since we already ran -init with the
intended indexlevel.

5 years agomsgmap: remove double negative
Eric Wong [Sat, 25 May 2019 04:34:20 +0000 (04:34 +0000)]
msgmap: remove double negative

I have never not found double negatives to be confusing...

5 years agoTODO: more stuff: bundles, synonyms, dogfooding
Eric Wong [Fri, 24 May 2019 08:55:56 +0000 (08:55 +0000)]
TODO: more stuff: bundles, synonyms, dogfooding

git bundles could/should make self-hosting easier.

Being able to configure synonym (and spelling) lists would make
some searches more useful.

Might as well dogfood kernel stuff, too, given the overlap and
history between this project, git and the Linux kernel.  Would
be interesting to have *BSD folks throw their hat in the ring,
too.

Building/testing userspace stuff is often the most
time-consuming, but necessary to ensure future compatibility.

5 years agoMANIFEST: add extman.perl
Eric Wong [Fri, 24 May 2019 07:55:29 +0000 (07:55 +0000)]
MANIFEST: add extman.perl

Oops :x

5 years agodoc: add URLs for Xapian manpages
Eric Wong [Fri, 24 May 2019 02:57:58 +0000 (02:57 +0000)]
doc: add URLs for Xapian manpages

Since we go through the effort of hosting these manpages,
link to them.

5 years agodoc: xcpdb: add switch documentation
Eric Wong [Fri, 24 May 2019 02:57:57 +0000 (02:57 +0000)]
doc: xcpdb: add switch documentation

In particular, the '--compact' switch is really useful since it
works without holding the inbox-wide lock for minutes at a time
on giant inboxes (inboxes where copies can take dozens, if not
hundreds of minutes).

5 years agodoc: generate manpages for some Xapian commands
Eric Wong [Fri, 24 May 2019 02:57:56 +0000 (02:57 +0000)]
doc: generate manpages for some Xapian commands

They're nowhere to be found on Xapian.org, and links to
external services are either too long (for manpages.debian.org)
or have privacy-invasive tracking JS on them.

5 years agodoc: sync .txt mtime to .pod mtime
Eric Wong [Fri, 24 May 2019 02:57:55 +0000 (02:57 +0000)]
doc: sync .txt mtime to .pod mtime

Otherwise timestamps for .html files get screwed up, too;
and that hurts caching.

5 years agodoc: don't barf on missing `git set-file-times'
Eric Wong [Fri, 24 May 2019 02:57:54 +0000 (02:57 +0000)]
doc: don't barf on missing `git set-file-times'

It's not critical, but it's nice to have for cache-friendliness
(otherwise I would not have written it :P)

I guess I should follow up on getting it into 'git contrib/':
https://public-inbox.org/git/20100702033709.GA6818@burratino/

5 years agodoc: daemon: fix manpage section for nginx
Eric Wong [Fri, 24 May 2019 02:57:53 +0000 (02:57 +0000)]
doc: daemon: fix manpage section for nginx

The nginx manpage is in section 8.

5 years agodoc: index: fix miscapitalization of "SQLite"
Eric Wong [Fri, 24 May 2019 02:57:52 +0000 (02:57 +0000)]
doc: index: fix miscapitalization of "SQLite"

Oops :x

5 years agosearch: don't log all warnings on retry_reopen
Eric Wong [Fri, 24 May 2019 01:12:08 +0000 (01:12 +0000)]
search: don't log all warnings on retry_reopen

Some users (or bots :P) can trigger horrible queries which
the caller can choose to either log or ignore.  This prevents
horrible queries from ExtMsg from logging confusing "ref: "
messages when $@ is not a Perl reference.

5 years agodoc: various updates to reflect current state
Eric Wong [Thu, 23 May 2019 10:37:38 +0000 (10:37 +0000)]
doc: various updates to reflect current state

-index documentation avoid redundant v1 information and refers
readers to apropriate v1/v2 manpages.  Search::Xapian can also
be optional, now, as only the PSGI search interface uses it.

Favor "INBOX_DIR" where appropriate, since "REPO_DIR" can be
confused for code repos which we also support.

XAPIAN_FLUSH_THRESHOLD is documented for all relevant
bulk commands.

5 years agoxapcmd: do not reset %SIG until last Xtmpdir is done
Eric Wong [Thu, 23 May 2019 09:37:04 +0000 (09:37 +0000)]
xapcmd: do not reset %SIG until last Xtmpdir is done

To properly handle compact tmpdir cleanup in single process
situations, we need to carefully account for Xtmpdir not
being a singleton and ensuring we don't clobber signal
handlers which belong to other Xtmpdirs.

5 years agoxcpdb|compact: support --jobs/-j flag like gmake(1)
Eric Wong [Thu, 23 May 2019 09:37:03 +0000 (09:37 +0000)]
xcpdb|compact: support --jobs/-j flag like gmake(1)

We don't have to be tied to the number of partitions in case
we made a bad choice at initialization.  This doesn't affect
reindexing, but the copying phase is already intensive.

And optimize away the extra process when we only have a single
job which won't parallelize.

The wording for the (v2) reindexing phase could be improved,
later.  I also plan to allow repartitioning of existing
Xapian DBs.

5 years agoxapcmd: cleanup on interrupted xcpdb "--compact"
Eric Wong [Thu, 23 May 2019 09:37:02 +0000 (09:37 +0000)]
xapcmd: cleanup on interrupted xcpdb "--compact"

We should not have leftover junk on interrupted invocations.

5 years agoxcpdb|compact: support some xapian-compact switches
Eric Wong [Thu, 23 May 2019 09:37:01 +0000 (09:37 +0000)]
xcpdb|compact: support some xapian-compact switches

Allow users to specify the --blocksize <B>, --no-full, --fuller
options for xapian-compact(1) for fine-tuning compact behavior
for low-traffic/inactive inboxes.

We also won't support --multipass, since it doesn't seem
compatible with our requirement to use --no-renumber.

We also won't support --single-file, since it only seems
intended for totally dead inboxes; and it doesn't seem
worth the support overhead when "totally dead" turns out
to be a misdiagnosis.

5 years agocompact: reuse infrastructure from xcpdb
Eric Wong [Thu, 23 May 2019 09:37:00 +0000 (09:37 +0000)]
compact: reuse infrastructure from xcpdb

Since -xcpdb is a superset of -compact, we can reuse much of
that code used for driving compact.

For compact (only), this is slightly less memory efficient since
it requires an extra process per-partition, but we get to prefix
the output with the partition name for more readable output.

5 years agoxcpdb: remove temporary directories on aborts
Eric Wong [Thu, 23 May 2019 09:36:59 +0000 (09:36 +0000)]
xcpdb: remove temporary directories on aborts

Cleanup temporary directories on common termination signals
(INT, HUP, PIPE, TERM), but only if it's not in the process
of being committed via rename() sequence.

5 years agoxcpdb: show re-indexing progress
Eric Wong [Thu, 23 May 2019 09:36:58 +0000 (09:36 +0000)]
xcpdb: show re-indexing progress

Emit information about reindexing git revision ranges when used
with xcpdb.  Additionally, distinguish Xapian copy output from
v2 git epoch counting by increasing directory context info.

For now, v1 batches batches are emitted.  v2 indexing is still
missing progress reporting for batches, as the data structures
for reindexing would benefit from a refactoring, first.

This does not currently affect the use of public-inbox-index,
but may in the future.

5 years agoxapcmd: use "print STDERR" for progress reporting
Eric Wong [Thu, 23 May 2019 09:36:57 +0000 (09:36 +0000)]
xapcmd: use "print STDERR" for progress reporting

`warn' is reserved for actual warnings, as it respects
$SIG{__WARN__} and we rely on that override to print
message context information when we are indexing.

5 years agodoc: xcpdb: update to reflect the current state
Eric Wong [Thu, 23 May 2019 09:36:56 +0000 (09:36 +0000)]
doc: xcpdb: update to reflect the current state

It is no longer a wrapper around copydatabase(1), since
copydatabase did not recover from DatabaseModifiedError.

5 years agoxapcmd: avoid EXDEV when finalizing changes
Eric Wong [Thu, 23 May 2019 09:36:55 +0000 (09:36 +0000)]
xapcmd: avoid EXDEV when finalizing changes

By creating temporary directories as deep as possible,
we can allow v2 repositories to have `xap$SCHEMA_VERSION'
(e.g. `xap15') reside on a separate FS.

We also check st_dev ahead-of-time to avoid doing work which
will fail with EXDEV.  Of course, another process may still
move/change things around.

5 years agoxcpdb: cleanup error handling and diagnosis
Eric Wong [Thu, 23 May 2019 09:36:54 +0000 (09:36 +0000)]
xcpdb: cleanup error handling and diagnosis

Running a full "public-inbox-index --reindex" in parallel
with "public-inbox-xcpdb" on the same inbox can still cause
problems, though.

5 years agoxcpdb: implement progress reporting
Eric Wong [Thu, 23 May 2019 09:36:53 +0000 (09:36 +0000)]
xcpdb: implement progress reporting

Copying an entire Xapian DB is horribly slow whether it's done
via Perl or copydatabase(1).  So displaying some progress
indication is good for user experience.

While we're at it, prefix xapian-compact output, too; since
parallel processes end up clobbering each other.

5 years agoxcpdb: use fine-grained locking
Eric Wong [Thu, 23 May 2019 09:36:52 +0000 (09:36 +0000)]
xcpdb: use fine-grained locking

Copying an entire Xapian DB takes a long time, so update our
reindexing code to support partial reindexing, snapshot the
pre-copydatabase git revisions, perform the lengthy copy,
and do a partial reindex when the copy + renames are done.

5 years agov2writable: hoist out log_range sub for readability
Eric Wong [Thu, 23 May 2019 09:36:51 +0000 (09:36 +0000)]
v2writable: hoist out log_range sub for readability

This is preparation to to support partial reindexing

5 years agoxapcmd: xcpdb supports compaction
Eric Wong [Thu, 23 May 2019 09:36:50 +0000 (09:36 +0000)]
xapcmd: xcpdb supports compaction

To minimize the delay on active inboxes, it's actually ideal to
run xapian-compact at the end of the per-partition cpdb process;
since the new DB isn't accessible yet and so we don't have to
deal with lock contention with -mda or -watch processes.  The
downside is temporary file overhead (3x instead of 2x) required.

5 years agoxcpdb: implement using Perl bindings
Eric Wong [Thu, 23 May 2019 09:36:49 +0000 (09:36 +0000)]
xcpdb: implement using Perl bindings

By avoid copydatabase(1) entirely, we can make further changes
to avoid locking the entire inbox for a long operation and
switch to fine-grained locking.

5 years agoadmin: move index_inbox over
Eric Wong [Thu, 23 May 2019 09:36:48 +0000 (09:36 +0000)]
admin: move index_inbox over

We will be reindexing after copydatabase

5 years agoxapcmd: do not cleanup on errors
Eric Wong [Thu, 23 May 2019 09:36:47 +0000 (09:36 +0000)]
xapcmd: do not cleanup on errors

We move the old directory into the new directory, so avoid the
situation where a bug or error could cause the tempdir cleanup to run
and destroy both our old and new directories.

5 years agoxcpdb: new tool which wraps Xapian's copydatabase(1)
Eric Wong [Thu, 23 May 2019 09:36:46 +0000 (09:36 +0000)]
xcpdb: new tool which wraps Xapian's copydatabase(1)

copydatabase(1) is an existing Xapian tool which is the
recommended way to upgrade existing DBs to the latest Xapian
database format (currently "glass" for stable/released
versions).  Our use of Xapian relies on preserving document IDs,
so we'll wrap it like we do xapian-compact(1) and use the
"--no-renumber" switch.

I could not name the tool "public-inbox-copydatabase" since it
would be ambiguous as to which DB it's actually copying.  So, I
abbreviated the suffix to "xcpdb" (Xapian CoPy DataBase), which
I hope is acceptable and unambiguous.

5 years agoxapcmd: support spawn options
Eric Wong [Thu, 23 May 2019 09:36:45 +0000 (09:36 +0000)]
xapcmd: support spawn options

copydatabase(1) is exceptionally noisy and it's output is
confusing when run in parallel.  Support redirects at least, and
env while we're at it to give us future options.

We can also stuff a -jobs parameter into the options to limit
parallelism since it can be useful for low-priority upgrade
jobs.

5 years agoadmin: hoist out resolve_inboxes for -compact and -index
Eric Wong [Thu, 23 May 2019 09:36:44 +0000 (09:36 +0000)]
admin: hoist out resolve_inboxes for -compact and -index

Both of these index-affecting commands should work similarly
on the command-line.

public-inbox-index no longer complains about unconfigured
~/.public-inbox/config; but often I found myself being
annoyed by that, anyways...

5 years agoxapcmd: new module for wrapping Xapian commands
Eric Wong [Thu, 23 May 2019 09:36:43 +0000 (09:36 +0000)]
xapcmd: new module for wrapping Xapian commands

Port public-inbox-compact(1) over to using it, and we will need
to wrap copydatabase(1) to ease glass migrations, too.

5 years agosearch: reenable phrase search on non-chert Xapian
Eric Wong [Thu, 23 May 2019 09:36:42 +0000 (09:36 +0000)]
search: reenable phrase search on non-chert Xapian

This is assuming nobody uses flint or earlier, anymore;
as flint predates the existence of this project.

5 years agodoc: document the reason for --no-renumber
Eric Wong [Thu, 23 May 2019 09:36:41 +0000 (09:36 +0000)]
doc: document the reason for --no-renumber

We're going to need copydatabase, too

5 years agov1writable: retire in favor of InboxWritable
Eric Wong [Thu, 23 May 2019 09:36:40 +0000 (09:36 +0000)]
v1writable: retire in favor of InboxWritable

In retrospect, introducing V1Writable was unnecessary and
InboxWritable->importer is in a better position to abstract
away differences between v1 and v2 writers.

So teach InboxWritable to initialize inboxes and get rid
of V1Writable.

5 years agot/convert-compact: skip on missing xapian-compact(1)
Eric Wong [Thu, 23 May 2019 09:36:39 +0000 (09:36 +0000)]
t/convert-compact: skip on missing xapian-compact(1)

Can't run the test if the required Xapian tools are missing.

5 years agoMerge branch 'ds-cleanup'
Eric Wong [Wed, 22 May 2019 21:54:21 +0000 (21:54 +0000)]
Merge branch 'ds-cleanup'

* ds-cleanup:
  DS: warn on deprecations
  DS: remove IPPROTO_TCP import
  DS: drop $VERSION var
  DS: remove support OtherFds code
  DS: get rid of unused methods and aliases

5 years agoDS: warn on deprecations
Eric Wong [Wed, 22 May 2019 21:46:59 +0000 (21:46 +0000)]
DS: warn on deprecations

Enabling deprecation warnings didn't seem to have any
noticeable effects with "perl -w -c", so whatever reason
Danga had for it is long irrelevant.

5 years agoDS: remove IPPROTO_TCP import
Eric Wong [Wed, 22 May 2019 21:46:58 +0000 (21:46 +0000)]
DS: remove IPPROTO_TCP import

Unlike Danga::Socket, we do not support TCP_CORK, either

5 years agoDS: drop $VERSION var
Eric Wong [Wed, 22 May 2019 21:46:57 +0000 (21:46 +0000)]
DS: drop $VERSION var

It was only relevant to Danga::Socket.

5 years agoDS: remove support OtherFds code
Eric Wong [Wed, 22 May 2019 21:46:56 +0000 (21:46 +0000)]
DS: remove support OtherFds code

It's easy enough to wrap FDs in classes that can use
all of the functionality of the event loop, not just
the read-only interface AddOtherFds provided.

5 years agoDS: get rid of unused methods and aliases
Eric Wong [Wed, 22 May 2019 21:46:55 +0000 (21:46 +0000)]
DS: get rid of unused methods and aliases

"make syntax" is clean, now

5 years agousercontent: stop relying on autodie
Eric Wong [Wed, 22 May 2019 07:27:52 +0000 (07:27 +0000)]
usercontent: stop relying on autodie

It's a non-standard package on CentOS-7, actually; and we
shouldn't bloat the PSGI server by loading a module which
isn't strictly needed.

5 years agoci: support CentOS-7
Eric Wong [Wed, 22 May 2019 02:14:02 +0000 (02:14 +0000)]
ci: support CentOS-7

Tested on an amd64 chroot built with rinse 3.4

5 years agot/search*: require DBI and DBD::SQLite, too
Eric Wong [Wed, 22 May 2019 02:14:01 +0000 (02:14 +0000)]
t/search*: require DBI and DBD::SQLite, too

None of the Search::Xapian-dependent stuff works without DBI
and DBD::SQLite.

There are no plans to support Xapian w/o DBD::SQLite since
SQLite is more common and less resource-intensive than Xapian.

5 years agogit: workaround old git-rev-parse(1) (--git-path)
Eric Wong [Wed, 22 May 2019 02:14:00 +0000 (02:14 +0000)]
git: workaround old git-rev-parse(1) (--git-path)

git < 2.5.0 was missing --git-path support.  This means any
users relying on some rare environment variables will need git
2.5.0+

5 years agot/watch_filter_rubylang: disable v2 test for git < 2.6
Eric Wong [Wed, 22 May 2019 02:13:59 +0000 (02:13 +0000)]
t/watch_filter_rubylang: disable v2 test for git < 2.6

This test was not disabled properly for ancient versions of
git without get-mark support.

5 years agoinit: preserve permissions for git prior to 2.1.0
Eric Wong [Wed, 22 May 2019 02:13:58 +0000 (02:13 +0000)]
init: preserve permissions for git prior to 2.1.0

"git config" did not preserve permissions of the config file it
modifies prior to git 2.1.0, so workaround that.

5 years agoMerge remote-tracking branch 'origin/xap-optional' into master
Eric Wong [Tue, 21 May 2019 23:22:05 +0000 (23:22 +0000)]
Merge remote-tracking branch 'origin/xap-optional' into master

* origin/xap-optional:
  admin: improve warnings and errors for missing modules
  searchidx: do not create empty Xapian partitions for basic
  lazy load Xapian and make it optional for v2
  www: use Inbox->over where appropriate
  nntp: use Inbox->over directly
  inbox: add ->over method to ease access

5 years agosearchidx: remove unused Compress::Zlib import
Eric Wong [Tue, 21 May 2019 06:54:27 +0000 (06:54 +0000)]
searchidx: remove unused Compress::Zlib import

We only compress in OverIdx, now; since we no longer do overview
stuff in Xapian (and Xapian compresses document data, anyways).

5 years agoPublicInbox::Import::add: Consolidate subject handling
Eric W. Biederman [Fri, 17 May 2019 01:53:06 +0000 (20:53 -0500)]
PublicInbox::Import::add: Consolidate subject handling

Consolidate subject handling in the add function to make it easier to
read and understand.

Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
5 years agowww: unescape '+' => ' ' before general URI unescape
Eric Wong [Thu, 16 May 2019 07:06:21 +0000 (07:06 +0000)]
www: unescape '+' => ' ' before general URI unescape

This allows searching for terms with "+" in them properly.

5 years agoRevert "view: perform highlighting for space-prefixed diffs"
Eric Wong [Thu, 16 May 2019 06:54:08 +0000 (06:54 +0000)]
Revert "view: perform highlighting for space-prefixed diffs"

This was buggy and was causing non-diff text to have extra
leading spaces.  The diff parsing code needs to be cleaned up,
so this will be fixed, later.

This reverts commit 1a67b91c1326efa372d1ec957e2494849d894f0b.

5 years agosearch: disable phrase searching, for now
Eric Wong [Thu, 16 May 2019 03:40:47 +0000 (03:40 +0000)]
search: disable phrase searching, for now

There probably needs to be an option to enable this
independently of indexlevel; but for now this is
the safest option.

And, as I discovered during the development of the
indexlevel option, Xapian does a pretty good job of
finding phrases without position data, anyways.

5 years agoview: perform highlighting for space-prefixed diffs
Eric Wong [Thu, 16 May 2019 01:23:29 +0000 (01:23 +0000)]
view: perform highlighting for space-prefixed diffs

"git format-patch --interdiff" and similar can prefix diffs
with leading white space.  Teach our diff parser to account
for it and set appropriate CSS classes for them.

5 years agoremove hard Devel::Peek dependency and lazy load for daemons
Eric Wong [Wed, 15 May 2019 10:39:19 +0000 (10:39 +0000)]
remove hard Devel::Peek dependency and lazy load for daemons

It's only useful for a corner case in long-running daemons when
an admin decides to compact or vacuum a Xapian or SQLite DB.

As a result, other scripts should run slightly faster.  For
instance, this saves about 80ms (2.710s => 2.630s) in t/mda.t
on my remote workstation.

While we're at it, make sure EvCleanup is properly require'd
in Daemon.pm and HTTP.pm and document our use of Devel::Peek.

5 years agoinbox: remove POSIX strftime import
Eric Wong [Wed, 15 May 2019 10:39:01 +0000 (10:39 +0000)]
inbox: remove POSIX strftime import

We no longer need it since Inbox->recent hits the overview
DB instead of Xapian

5 years agodoc: add marketing guide
Eric Wong [Thu, 9 May 2019 09:57:37 +0000 (09:57 +0000)]
doc: add marketing guide

Preventative measures; since marketing is almost always annoying
to me.  And trying to avoid unintended consequences.

5 years agoadmin: improve warnings and errors for missing modules
Eric Wong [Wed, 15 May 2019 01:18:09 +0000 (01:18 +0000)]
admin: improve warnings and errors for missing modules

Since we lazy-load Xapian now, some errors may become
more cryptic or buried.  Try to improve that by making
Admin show better errors.

5 years agosearchidx: do not create empty Xapian partitions for basic
Eric Wong [Wed, 15 May 2019 01:18:10 +0000 (01:18 +0000)]
searchidx: do not create empty Xapian partitions for basic

No point in leaving a mess of empty directories when Xapian
doesn't load.

5 years agolazy load Xapian and make it optional for v2
Eric Wong [Wed, 15 May 2019 01:18:08 +0000 (01:18 +0000)]
lazy load Xapian and make it optional for v2

More tests work without Search::Xapian, now.
Usability issues still need to be fixed

5 years agowww: use Inbox->over where appropriate
Eric Wong [Wed, 15 May 2019 01:18:07 +0000 (01:18 +0000)]
www: use Inbox->over where appropriate

We don't need to rely on Xapian search functionality for the
majority of the WWW code, even.  subject_normalized is moved to
SearchMsg, where it (probably) makes more sense, anyways.

5 years agonntp: use Inbox->over directly
Eric Wong [Wed, 15 May 2019 01:18:06 +0000 (01:18 +0000)]
nntp: use Inbox->over directly

None of the NNTP code actually relies on Xapian, anymore.

5 years agoinbox: add ->over method to ease access
Eric Wong [Wed, 15 May 2019 01:18:05 +0000 (01:18 +0000)]
inbox: add ->over method to ease access

One small step towards making installing Xapian optional for v2
and providing more WWW and NNTP functionality without it.

5 years agotests: get rid of unnecessary Cwd module use
Eric Wong [Tue, 14 May 2019 03:32:03 +0000 (03:32 +0000)]
tests: get rid of unnecessary Cwd module use

We only need it for tests that chdir, and maybe for ENV{PATH}
portability (dash seems fine, not sure about others).

v2: revert change to solver_git.t for FreeBSD 11.2 and document

5 years agot/nntp.t: skip if Data::Dumper is missing
Eric Wong [Tue, 14 May 2019 03:32:02 +0000 (03:32 +0000)]
t/nntp.t: skip if Data::Dumper is missing

We can revisit this, later; but Data::Dumper requires a separate
package for CentOS-7 users, at least.

5 years agot/config.t: remove Data::Dumper dependency
Eric Wong [Tue, 14 May 2019 03:32:01 +0000 (03:32 +0000)]
t/config.t: remove Data::Dumper dependency

CentOS-7 needs the perl-Data-Dumper package, and the
test is small enough to roll our own escaping, here.

5 years agotests: remove unnecessary loading of ::DS and Socket
Eric Wong [Tue, 14 May 2019 03:32:00 +0000 (03:32 +0000)]
tests: remove unnecessary loading of ::DS and Socket

PublicInbox::DS works for every platform we we care about,
nowadays; so checking for it is a waste of time.  Cleanup a
few POSIX and Socket imports while we're in the area.

5 years agohttpd: get rid of Deflater warning
Eric Wong [Sun, 5 May 2019 20:42:20 +0000 (20:42 +0000)]
httpd: get rid of Deflater warning

Deflating responses may be done by the reverse proxy (e.g. varnish
or nginx), so the warning for it could be invalid.

5 years agosearchidx: fix incremental index with indexlevel=basic on v1
Eric Wong [Tue, 14 May 2019 02:04:43 +0000 (02:04 +0000)]
searchidx: fix incremental index with indexlevel=basic on v1

We were reindexing the full history every invocation of -index
when Xapian was not used because we were incorrectly relying on
'last_commit' metadata stored in Xapian.

Rewrite the indexing logic to be less confusing while we're
at it, since we rely on `git merge-base --is-ancestor' nowadays.

Furthermore, we need to handle message removals from the
overview index correctly when Xapian is not in use.

Co-authored-by: Eric W. Biederman <ebiederm@xmission.com>
5 years agov2writable: allow setting nproc via creat options
Eric Wong [Tue, 14 May 2019 02:04:42 +0000 (02:04 +0000)]
v2writable: allow setting nproc via creat options

Avoiding reliance on environment variables is a bit cleaner
for writing tests

5 years agov1writable: new wrapper which is closer to v2writable
Eric Wong [Tue, 14 May 2019 02:04:41 +0000 (02:04 +0000)]
v1writable: new wrapper which is closer to v2writable

Import initialization is a little strange from history, but we
also can't change it too much because it's technically a public
API which external code may rely on...

And we may need to support v1 repos indefinitely.  This should
make it easier to write tests for both formats.

5 years agoci: include some scripts to make CI easier
Eric Wong [Tue, 7 May 2019 09:09:54 +0000 (09:09 +0000)]
ci: include some scripts to make CI easier

This should make it easier to test a bunch of package
installation profiles across whatever OS isolation
one chooses (chroots, containers, jails, VMs).

5 years agodoc/include.mk: remove git use and redundant declarations
Eric Wong [Wed, 8 May 2019 08:34:36 +0000 (08:34 +0000)]
doc/include.mk: remove git use and redundant declarations

And document that we still have GNU-isms in that
include.mk Makefile (and may continue to do so).
Finally, take advantage of GNU-isms to warn users
to run "gmake" to build all manpages.

5 years agoMakefile.PL: improve make(1) portability
Eric Wong [Tue, 7 May 2019 23:43:42 +0000 (23:43 +0000)]
Makefile.PL: improve make(1) portability

The `shell' function appears missing, so we'll rely on Bourne
shell expansioN, instead.

Use "$?" instead of "$<" since the latter is only specified
for inference and .DEFAULT rules, not target rules.

Tested on FreeBSD make(1) and bmake(1) on Debian.

5 years agodoc: give txt files proper titles
Eric Wong [Tue, 7 May 2019 20:40:28 +0000 (20:40 +0000)]
doc: give txt files proper titles

We can fix the redundant rule in include.mk which causes
make(1) on FreeBSD to complain; but HTML docs will likely
still require GNU make.

5 years agodoc: use bullet list for wwwlisting options
Eric Wong [Wed, 8 May 2019 22:32:43 +0000 (22:32 +0000)]
doc: use bullet list for wwwlisting options

Otherwise, pod2man complains about "=item 404" not starting
with a letter and thinking it's part of a numbered list.

5 years agot/purge.t: fix unreferenced variable
Eric Wong [Wed, 8 May 2019 19:52:36 +0000 (19:52 +0000)]
t/purge.t: fix unreferenced variable

5 years agoINSTALL: note xapian-compact(1) tool
Eric Wong [Wed, 8 May 2019 20:03:33 +0000 (20:03 +0000)]
INSTALL: note xapian-compact(1) tool

The Perl Xapian bindings do not support compacting, yet,
so we rely on the xapian-compact tool.

5 years agoINSTALL: depend on Encode instead of Encode::MIME::Header
Eric Wong [Wed, 8 May 2019 02:47:20 +0000 (02:47 +0000)]
INSTALL: depend on Encode instead of Encode::MIME::Header

Email::MIME uses Encode::MIME::Header and depends on that
appropriately; however we depend on other parts of the Encode
distribution, but that's bundled with Perl by upstream, anyways;
and should place no additional burden on users.

5 years agoINSTALL: update with FreeBSD pkg names and reorder
Eric Wong [Sun, 5 May 2019 01:37:00 +0000 (01:37 +0000)]
INSTALL: update with FreeBSD pkg names and reorder

Fortunately, there is a pattern to most of these package names
in all distros I've tested (and hopefully other BSDs have them,
too).

Then, reorder the INSTALL document to mention the top-level
modules, first, so users can avoid needing to type extra
dependencies.  However, we still list some implicit dependencies
in case the upstream package drops dependencies independently of
us.

Finally, Devel::Peek is not a dependency worth making optional
since it's bundled by Perl upstream.  Fedora/RH-based distros
are the only one which turn it into a non-standard package when
Perl5 is installed.

5 years agoMerge remote-tracking branch 'origin/danga-bundle'
Eric Wong [Wed, 8 May 2019 19:24:07 +0000 (19:24 +0000)]
Merge remote-tracking branch 'origin/danga-bundle'

* origin/danga-bundle:
  DS: epoll: fix misordered EPOLL_CTL_DEL call
  DS: drop unused "_undef" sub
  syscall: drop readahead wrapper
  build: do not manify DS and Syscall pods
  DS: handle EINTR in IO::Poll path, too
  DS: workaround IO::Kqueue EINTR (mis-)handling
  DS: drop profiling support
  DS: remove unused fields and functions
  listener: use EPOLLEXCLUSIVE for listen sockets
  bundle Danga::Socket and Sys::Syscall

5 years agoDS: epoll: fix misordered EPOLL_CTL_DEL call
Eric Wong [Wed, 8 May 2019 19:04:17 +0000 (19:04 +0000)]
DS: epoll: fix misordered EPOLL_CTL_DEL call

Any operations on an fd after POSIX::close() are invalid, so
epoll_ctl will fail.  Worse off, in a multi-threaded Perl, the
fd may be reused by another thread and EPOLL_CTL_DEL can hit the
wrong file description as a result.

cf. https://rt.cpan.org/Ticket/Display.html?id=129487

5 years agoDS: drop unused "_undef" sub
Eric Wong [Wed, 8 May 2019 18:58:58 +0000 (18:58 +0000)]
DS: drop unused "_undef" sub

No longer used since we removed the *_ip_string fields

5 years agosyscall: drop readahead wrapper
Eric Wong [Wed, 8 May 2019 18:10:25 +0000 (18:10 +0000)]
syscall: drop readahead wrapper

No backwards compatibility to worry about for us; and fadvise
is superior anyways.