]> Sergey Matveev's repositories - public-inbox.git/log
public-inbox.git
5 years agoMerge branch 'charclass'
Eric Wong [Tue, 4 Jun 2019 10:38:20 +0000 (10:38 +0000)]
Merge branch 'charclass'

* charclass: (24 commits)
  www: require ASCII word characters for CSS filenames
  www: require ASCII range for mbox downloads
  githttpbackend: require ASCII in path
  require ASCII digits for local FS items
  www: require ASCII digit for git epoch
  solver|viewdiff: restrict digit matches to ASCII
  inbox: require ASCII digits for feedmax var
  filter/rubylang: require ASCII digit for mailcount
  msgtime: require ASCII digits for parsing dates
  searchview: do not allow non-ASCII offsets and limits
  githttpbackend: require Range:, Status: to be ASCII digits
  view: require YYYYmmDD(HHMMSS) timestamps to be ASCII
  newswww: only accept ASCII digits as article numbers
  config: do not accept non-ASCII digits in cgitrc params
  www: require ASCII filenames in git blob downloads
  www: only emit ASCII chars in attachment filenames
  wwwattach: only pass the charset through if ASCII
  wwwlisting: require ASCII digit for port number
  http: require SERVER_PORT to be ASCII digit
  feed: only accept ASCII digits for ref~$N
  ...

5 years agoMANIFEST: add examples/nginx_proxy
Eric Wong [Tue, 4 Jun 2019 10:37:52 +0000 (10:37 +0000)]
MANIFEST: add examples/nginx_proxy

5 years agoexamples: add sample nginx configuration
Ali Alnubani [Tue, 4 Jun 2019 09:22:27 +0000 (09:22 +0000)]
examples: add sample nginx configuration

The sample configuration can be used to proxy-pass requests
to public-inbox-httpd or to a standalone PSGI/Plack server.

5 years agowww: require ASCII word characters for CSS filenames
Eric Wong [Tue, 4 Jun 2019 10:19:34 +0000 (10:19 +0000)]
www: require ASCII word characters for CSS filenames

Allowing admins to set non-ASCII CSS filenames could
cause unnecessary problems for client and proxies.

5 years agowww: require ASCII range for mbox downloads
Eric Wong [Tue, 4 Jun 2019 09:05:51 +0000 (09:05 +0000)]
www: require ASCII range for mbox downloads

We do not support many mboxrd download range specifications at
the moment; but parsing non-ASCII characters isn't planned.
This makes no difference aside from being able to return 404
slightly earlier than we would've in the past.

5 years agogithttpbackend: require ASCII in path
Eric Wong [Tue, 4 Jun 2019 08:58:32 +0000 (08:58 +0000)]
githttpbackend: require ASCII in path

We mainly support git-upload-pack; and maybe somebody uses
git-receive-pack with this.  Perhaps other (experimental)
command names are acceptable.  But it's unlikely anybody will
want Unicode command names for git services.

5 years agorequire ASCII digits for local FS items
Eric Wong [Tue, 4 Jun 2019 08:40:34 +0000 (08:40 +0000)]
require ASCII digits for local FS items

In case some BOFH decides to randomly create directories
using non-ASCII digits all over the place.

5 years agowww: require ASCII digit for git epoch
Eric Wong [Tue, 4 Jun 2019 08:36:18 +0000 (08:36 +0000)]
www: require ASCII digit for git epoch

Don't inadvertantly serve git repos containing non-ASCII
digit characters.

5 years agosolver|viewdiff: restrict digit matches to ASCII
Eric Wong [Tue, 4 Jun 2019 08:30:55 +0000 (08:30 +0000)]
solver|viewdiff: restrict digit matches to ASCII

git would not generate non-ASCII digits to describe
hunk offsets, so don't waste more time than necessary
to make sense of non-ASCII digit chars for line offsets.

5 years agoinbox: require ASCII digits for feedmax var
Eric Wong [Tue, 4 Jun 2019 08:29:36 +0000 (08:29 +0000)]
inbox: require ASCII digits for feedmax var

Don't waste more cycles than necessary if somebody decides to
put non-ASCII digits in their ~/.public-inbox/config

5 years agofilter/rubylang: require ASCII digit for mailcount
Eric Wong [Tue, 4 Jun 2019 08:25:10 +0000 (08:25 +0000)]
filter/rubylang: require ASCII digit for mailcount

Unlikely to matter, but who knows...

5 years agomsgtime: require ASCII digits for parsing dates
Eric Wong [Tue, 4 Jun 2019 02:04:24 +0000 (02:04 +0000)]
msgtime: require ASCII digits for parsing dates

User input contains the darndest things.  Don't waste more time
than necessary trying to parse dates out of non-ASCII digits.

5 years agosearchview: do not allow non-ASCII offsets and limits
Eric Wong [Tue, 4 Jun 2019 02:04:33 +0000 (02:04 +0000)]
searchview: do not allow non-ASCII offsets and limits

Non-ASCII digits would be interpreted as zero when used as integers.

5 years agogithttpbackend: require Range:, Status: to be ASCII digits
Eric Wong [Tue, 4 Jun 2019 02:04:32 +0000 (02:04 +0000)]
githttpbackend: require Range:, Status: to be ASCII digits

Non-ASCII digits would be interpreted as a zeroes as integers.

While we're at it, ensure the Status: code is an ASCII digit,
too; though I would not expect git-http-backend(1) or cgit(1)
start spewing non-ASCII digits at us.

5 years agoview: require YYYYmmDD(HHMMSS) timestamps to be ASCII
Eric Wong [Tue, 4 Jun 2019 02:04:31 +0000 (02:04 +0000)]
view: require YYYYmmDD(HHMMSS) timestamps to be ASCII

Passing digits to `timegm' which it does not understand would
be a waste of time.

5 years agonewswww: only accept ASCII digits as article numbers
Eric Wong [Tue, 4 Jun 2019 02:04:30 +0000 (02:04 +0000)]
newswww: only accept ASCII digits as article numbers

Non-ASCII digits aren't specified in RFC3977 for article numbers;
so don't waste a trip to SQLite only to turn up empty.

5 years agoconfig: do not accept non-ASCII digits in cgitrc params
Eric Wong [Tue, 4 Jun 2019 02:04:29 +0000 (02:04 +0000)]
config: do not accept non-ASCII digits in cgitrc params

cgit uses atoi(3), and now we can retain compatibility.

5 years agowww: require ASCII filenames in git blob downloads
Eric Wong [Tue, 4 Jun 2019 09:02:01 +0000 (09:02 +0000)]
www: require ASCII filenames in git blob downloads

Our Hval::to_filename sub has always been strict about emitting
ASCII-only characters for ViewVCS "raw" links.

However, somebody could manually generate a filename with
non-ASCII words for somebody else to download (we have no
cheap and fast way of mapping filenames back to blobs for
validation).

5 years agowww: only emit ASCII chars in attachment filenames
Eric Wong [Tue, 4 Jun 2019 02:04:28 +0000 (02:04 +0000)]
www: only emit ASCII chars in attachment filenames

We don't want to emit funky URLs which can be lost in
translation or cause problems with non-Unicode-aware
clients.

Then, don't accept non-ASCII filenames in URLs, since
a manually-generated URL/filename in attachment downloads
could be used for Unicode homographs to confuse folks who
down the attachment.

5 years agowwwattach: only pass the charset through if ASCII
Eric Wong [Tue, 4 Jun 2019 02:04:27 +0000 (02:04 +0000)]
wwwattach: only pass the charset through if ASCII

AFAIK all names of charsets are ASCII, so passing non-ASCII
characters from emails to clients would probably confuse clients.

5 years agowwwlisting: require ASCII digit for port number
Eric Wong [Tue, 4 Jun 2019 08:32:27 +0000 (08:32 +0000)]
wwwlisting: require ASCII digit for port number

We only care about the hostname portion for matching,
so this change is probably inconsequential.

5 years agohttp: require SERVER_PORT to be ASCII digit
Eric Wong [Tue, 4 Jun 2019 02:04:26 +0000 (02:04 +0000)]
http: require SERVER_PORT to be ASCII digit

I'm not sure what middlewares care for for SERVER_PORT; but
allowing non-ASCII digits seems non-sensical, here.

5 years agofeed: only accept ASCII digits for ref~$N
Eric Wong [Tue, 4 Jun 2019 02:04:22 +0000 (02:04 +0000)]
feed: only accept ASCII digits for ref~$N

We don't want to waste cycles passing non-ASCII characters
to git.

5 years agomid: id_compress requires ASCII-clean words
Eric Wong [Tue, 4 Jun 2019 02:04:23 +0000 (02:04 +0000)]
mid: id_compress requires ASCII-clean words

Its result is used for HTML anchors and such.

5 years agonntp: ensure we only handle ASCII whitespace
Eric Wong [Tue, 4 Jun 2019 08:20:40 +0000 (08:20 +0000)]
nntp: ensure we only handle ASCII whitespace

RFC3977 does not have provisions for whitespace beyond ASCII
TAB, SP, CR and LF.  I doubt there's any NNTP clients broken
enough to be sending non-ASCII whitespace delimiters.

We're probably excessively liberal regarding TAB acceptance,
even; but it's probably too late to change at this point...

5 years agonntp: be explicit about ASCII digit matches
Eric Wong [Tue, 4 Jun 2019 02:04:25 +0000 (02:04 +0000)]
nntp: be explicit about ASCII digit matches

We aren't able to make sense of non-ASCII digits

cf. perlrecharclass(1) / "Digits" section

5 years agolinkify: support Internationalized Domain Names in URLs
Eric Wong [Tue, 4 Jun 2019 02:04:21 +0000 (02:04 +0000)]
linkify: support Internationalized Domain Names in URLs

The "\w" character class in Perl matches any word characters
in the Unicode database, not just ASCII characters.  So we
must be prepared for that and generate links to IDNs.

5 years agoMerge remote-tracking branch 'origin/git-cleanup'
Eric Wong [Mon, 3 Jun 2019 10:06:04 +0000 (10:06 +0000)]
Merge remote-tracking branch 'origin/git-cleanup'

* origin/git-cleanup:
  git: drop the deleted err_c file
  git: unconditional expiry

5 years agoMerge remote-tracking branch 'origin/ds'
Eric Wong [Mon, 3 Jun 2019 09:04:01 +0000 (09:04 +0000)]
Merge remote-tracking branch 'origin/ds'

* origin/ds:
  ds: remove PLCMap and per-socket PostLoopCallback
  ds: drop write_set_watch field
  ds: drop unused EVENT: label in epoll code path
  ds: drop checks for invalid descriptors
  ds: drop set_writer_func support
  ds: add a note about planned future changes
  ds: drop more unused subs

5 years agods: remove PLCMap and per-socket PostLoopCallback
Eric Wong [Mon, 3 Jun 2019 09:00:28 +0000 (09:00 +0000)]
ds: remove PLCMap and per-socket PostLoopCallback

We don't need and won't be needing per-socket PostLoopCallbacks.

5 years agot/psgi_search.t: require DBD::SQLite
Eric Wong [Mon, 3 Jun 2019 01:40:06 +0000 (01:40 +0000)]
t/psgi_search.t: require DBD::SQLite

In case we encounter an odd system which has Search::Xapian
but not DBD::SQLite.

5 years agods: drop write_set_watch field
Eric Wong [Sun, 2 Jun 2019 03:39:37 +0000 (03:39 +0000)]
ds: drop write_set_watch field

We never enable write watches ourselves for HTTP and NNTP,
and only enable the write watch with EvCleanup because it's
an "always on" watch.

5 years agods: drop unused EVENT: label in epoll code path
Eric Wong [Sun, 2 Jun 2019 03:39:36 +0000 (03:39 +0000)]
ds: drop unused EVENT: label in epoll code path

This was never used in Danga::Socket 1.61, either.

5 years agods: drop checks for invalid descriptors
Eric Wong [Sun, 2 Jun 2019 03:39:35 +0000 (03:39 +0000)]
ds: drop checks for invalid descriptors

I've used Danga::Socket for well over a decade in various
projects at this point and have never seen the need for it.

If such a bug ever happens; the process should fall over so
it gets fixed ASAP.

5 years agods: drop set_writer_func support
Eric Wong [Sun, 2 Jun 2019 03:39:34 +0000 (03:39 +0000)]
ds: drop set_writer_func support

This is not used by perlbal for OpenSSL support, either;
and it does not appear to be the right layer for doing
write translations anyways (IO::Socket::SSL uses `tie').

5 years agods: add a note about planned future changes
Eric Wong [Sun, 2 Jun 2019 03:39:33 +0000 (03:39 +0000)]
ds: add a note about planned future changes

Sometimes I get bored with the email part of this project and
need a distraction :P

5 years agods: drop more unused subs
Eric Wong [Sun, 2 Jun 2019 03:39:32 +0000 (03:39 +0000)]
ds: drop more unused subs

ToClose and HaveEpoll are of no use to us and I see no
future use for them, either.

5 years agods: fix and test for FD leaks with kqueue on ->Reset
Eric Wong [Sat, 1 Jun 2019 21:43:47 +0000 (21:43 +0000)]
ds: fix and test for FD leaks with kqueue on ->Reset

Even though we currently don't use it repeatedly, ->Reset
should close() kqueue FDs and not cause the process to run
out of descriptors.

Add a close-on-exec test while we're at it.

5 years agodoc/v2: note that Xapian is now optional in v2
Eric Wong [Sat, 1 Jun 2019 07:28:17 +0000 (07:28 +0000)]
doc/v2: note that Xapian is now optional in v2

This is true as of e220b8b2ee5cfd458167dc2c6c92726352c4c80e
("Merge remote-tracking branch 'origin/xap-optional' into master")

5 years agods: set close-on-exec flag on epoll descriptors
Eric Wong [Sat, 1 Jun 2019 00:23:22 +0000 (00:23 +0000)]
ds: set close-on-exec flag on epoll descriptors

We should not be leaking these FDs to git(1) processes,
in case git has a bug that causes it to access the wrong FD.

5 years agogit: drop the deleted err_c file
Eric Wong [Sat, 1 Jun 2019 03:27:55 +0000 (03:27 +0000)]
git: drop the deleted err_c file

No reason to leave that (usually) empty file open after killing off
"cat-file --batch-check".  This wasn't an unbound leak, though,
as respawning the --batch-check process would've clobbered the
old err_c file.

5 years agogit: unconditional expiry
Eric Wong [Sat, 1 Jun 2019 00:20:51 +0000 (00:20 +0000)]
git: unconditional expiry

A constant stream of traffic to either httpd/nntpd would mean
git-cat-file processes never expire.  Things can go bad after a
full repack, as a full repack will unlink old pack indices and
git-cat-file does not currently detect unlinked files.

We could do something complicated by recursively stat-ing
objects/pack of every git directory and alternate;
but that's probably not worth the trouble compared to
occasionally restarting the cat-file process.

So simplify the code and let httpd/nntpd expire them
periodically, since spawning a "git-cat-file --batch" process
isn't too expensive.  We already spawn for every request which
hits git-http-backend, cgit, and git-apply.

In the future, we may optionally support the Git::Raw module
to avoid IPC; but we must remain careful to not leave lingering
FDs open to unlinked files after repack.

5 years agogit: drop the deleted err_c file
Eric Wong [Sat, 1 Jun 2019 03:27:55 +0000 (03:27 +0000)]
git: drop the deleted err_c file

No reason to leave that (usually) empty file open after killing off
"cat-file --batch-check".  This wasn't an unbound leak, though,
as respawning the --batch-check process would've clobbered the
old err_c file.

5 years agogit: unconditional expiry
Eric Wong [Sat, 1 Jun 2019 00:20:51 +0000 (00:20 +0000)]
git: unconditional expiry

A constant stream of traffic to either httpd/nntpd would mean
git-cat-file processes never expire.  Things can go bad after a
full repack, as a full repack will unlink old pack indices and
git-cat-file does not currently detect unlinked files.

We could do something complicated by recursively stat-ing
objects/pack of every git directory and alternate;
but that's probably not worth the trouble compared to
occasionally restarting the cat-file process.

So simplify the code and let httpd/nntpd expire them
periodically, since spawning a "git-cat-file --batch" process
isn't too expensive.  We already spawn for every request which
hits git-http-backend, cgit, and git-apply.

In the future, we may optionally support the Git::Raw module
to avoid IPC; but we must remain careful to not leave lingering
FDs open to unlinked files after repack.

5 years agoTODO: add item for optional Cache::FastMmap
Eric Wong [Fri, 31 May 2019 16:58:46 +0000 (16:58 +0000)]
TODO: add item for optional Cache::FastMmap

Taking one step out of setting up a performant deployment could
make setup and administration easier (at the cost of installing
an extra-but-common XS module).  This can also be useful for
the day NNTP servers see hug-of-death events.

5 years agoviewdiff: avoid repeat variable expansion
Eric Wong [Fri, 31 May 2019 16:46:04 +0000 (16:46 +0000)]
viewdiff: avoid repeat variable expansion

This is worth a 1-2% speedup in t/perf-msgview.t rendering 2620
messages currently in https://public-inbox.org/meta/

5 years agoMerge remote-tracking branch 'origin/v2-noop-speedup'
Eric Wong [Thu, 30 May 2019 19:50:04 +0000 (19:50 +0000)]
Merge remote-tracking branch 'origin/v2-noop-speedup'

* origin/v2-noop-speedup:
  v2writable: short-circuit is_ancestor check on equality
  v2writable: avoid mm_tmp creation without regen
  v2writable: hoist out index_epoch sub
  v2writable: split off unindex_range mapping

5 years agodoc/hosted: drop some links and clarify wording
Eric Wong [Thu, 30 May 2019 06:54:06 +0000 (06:54 +0000)]
doc/hosted: drop some links and clarify wording

I don't have time to check and train spam for all these
projects.

Spam filtering is especially difficult on ruby-core: it
enters via Redmine, so it doesn't have a distinct Received:
chain, and also gets mixed with non-spam bug-report text,
throwing off Bayes training.

And I'm not sure if those mirrors did anybody any good, even;
so lets not say its' a "service" to anybody :P

The actual mirrors remain up, for now, but who knows...
I care about decentralization too much to ask anybody
to trust me to keep anything up :P

5 years agov2writable: short-circuit is_ancestor check on equality
Eric Wong [Thu, 30 May 2019 06:35:05 +0000 (06:35 +0000)]
v2writable: short-circuit is_ancestor check on equality

We don't need to use git to check ancestry if object IDs
match on a string comparison.

This saves 100ms or so and brings down the ~0.5s no-op time on
lore.kernel.org/lkml down to ~0.4s.

5 years agov2writable: avoid mm_tmp creation without regen
Eric Wong [Thu, 30 May 2019 03:59:40 +0000 (03:59 +0000)]
v2writable: avoid mm_tmp creation without regen

Creating mm_tmp is an expensive operation with large inboxes
and can be avoided if there are no new messages to process.

Since git-fetch(1) currently lacks an --exit-code option(*),
mirrors will run `public-inbox-index' unconditionally after
fetch, which is an expensive op if it needs to duplicate
a large SQLite DB.

This speeds up the mirror case of:

git --git-dir=git/$EPOCH.git fetch && public-inbox-index

This reduces the no-op `public-inbox-index' time from over 8s to
~0.5s on a (currently) 7-epoch clone of https://lore.kernel.org/lkml/
on my system.

(*) WIP --exit-code for git-fetch:
    https://public-inbox.org/git/87ftphw7mv.fsf@evledraar.gmail.com/

5 years agov2writable: hoist out index_epoch sub
Eric Wong [Thu, 30 May 2019 03:59:39 +0000 (03:59 +0000)]
v2writable: hoist out index_epoch sub

This will make future changes easier-to-follow.

5 years agov2writable: split off unindex_range mapping
Eric Wong [Thu, 30 May 2019 03:59:38 +0000 (03:59 +0000)]
v2writable: split off unindex_range mapping

It'll make it easier to detect if we have anything to
unindex and run git-log on, at all.

5 years agosearchidx: store indexlevel=medium as metadata
Eric Wong [Wed, 29 May 2019 20:56:32 +0000 (20:56 +0000)]
searchidx: store indexlevel=medium as metadata

And use it from Admin.

It's easy to tell what indexlevel=basic is from unconfigured
inboxes, but distinguishing between 'medium' and 'full' would
require stat()-ing position.* files which is fragile and
Xapian-implementation-dependent.

So use the metadata facility of Xapian and store it in the main
partition so Admin tools can deal better with unconfigured
inboxes copied using generic tools like cp(1) or rsync(1).

5 years agoindex: remove warning on unconfigured inboxes
Eric Wong [Wed, 29 May 2019 20:56:31 +0000 (20:56 +0000)]
index: remove warning on unconfigured inboxes

It's annoying for people using "git fetch && public-inbox-index"
as one user; and running -httpd/-nntpd as a different user
(where users see different config files).

5 years agoMerge branch 'v2-idx-progress'
Eric Wong [Wed, 29 May 2019 08:45:49 +0000 (08:45 +0000)]
Merge branch 'v2-idx-progress'

* v2-idx-progress:
  v2writable: show progress updates for index_sync
  index: support --verbose option
  v2writable: move index_sync options to sync state
  v2writable: use prototypes for internal subs
  v2writable: localize unindex-range.$EPOCH to $sync state
  v2writable: move {ranges} into $sync state
  v2writable: move {regen} into $sync state
  v2writable: move {reindex} field to $sync state
  v2writable: sync: move delete markers into $sync state
  v2writable: introduce $sync state and put mm_tmp in it

5 years agoMakefile.PL: enable prove warnings
Eric Wong [Wed, 29 May 2019 08:37:54 +0000 (08:37 +0000)]
Makefile.PL: enable prove warnings

We already "use warnings" everywhere, but could miss some spots.
This ought to cover that, and usually Perl module authors are
consistent about avoiding warnings that we won't clutter our
test outputs.

5 years agoMakefile.PL: allow `N' variable to be set in local config.mak
Eric Wong [Wed, 29 May 2019 08:29:30 +0000 (08:29 +0000)]
Makefile.PL: allow `N' variable to be set in local config.mak

This can useful for limiting test resource use without relying
on remembering the variable command-line.

5 years agov2writable: show progress updates for index_sync
Eric Wong [Tue, 28 May 2019 23:48:55 +0000 (23:48 +0000)]
v2writable: show progress updates for index_sync

We can show progress whenever we commit changes to the FS.

5 years agoindex: support --verbose option
Eric Wong [Tue, 28 May 2019 23:48:52 +0000 (23:48 +0000)]
index: support --verbose option

It doesn't implement progress of batches, yet, but it wires
up the parsing of the command-line while preserving output
compatibility.

This output is NOT meant to be stable.

5 years agov2writable: move index_sync options to sync state
Eric Wong [Tue, 28 May 2019 23:48:51 +0000 (23:48 +0000)]
v2writable: move index_sync options to sync state

And use singular `opt' to be consistent with the common name
of 'getopt'.

5 years agov2writable: use prototypes for internal subs
Eric Wong [Tue, 28 May 2019 23:48:50 +0000 (23:48 +0000)]
v2writable: use prototypes for internal subs

Hopefully this improves maintainability by allowing Perl
to do some arg checking for us.

5 years agov2writable: localize unindex-range.$EPOCH to $sync state
Eric Wong [Tue, 28 May 2019 23:48:49 +0000 (23:48 +0000)]
v2writable: localize unindex-range.$EPOCH to $sync state

We don't need to stuff that into $self (V2Writable) which can be
longer-lived than a ->index_sync invocation.

5 years agov2writable: move {ranges} into $sync state
Eric Wong [Tue, 28 May 2019 23:48:48 +0000 (23:48 +0000)]
v2writable: move {ranges} into $sync state

Yet another temporary variable with no use outside of index_sync.

5 years agov2writable: move {regen} into $sync state
Eric Wong [Tue, 28 May 2019 23:48:47 +0000 (23:48 +0000)]
v2writable: move {regen} into $sync state

regen is always enabled for index_sync nowadays (and has
been for a while).

Rename `index_prepare' to `sync_prepare' to show it's for
->index_sync; and not the online indexing we do for ->add.

5 years agov2writable: move {reindex} field to $sync state
Eric Wong [Tue, 28 May 2019 23:48:46 +0000 (23:48 +0000)]
v2writable: move {reindex} field to $sync state

reindexing info is not used outside of the index_sync code path.

5 years agov2writable: sync: move delete markers into $sync state
Eric Wong [Tue, 28 May 2019 23:48:45 +0000 (23:48 +0000)]
v2writable: sync: move delete markers into $sync state

Another small step to reduce parameters passed to reindex_oid.

5 years agov2writable: introduce $sync state and put mm_tmp in it
Eric Wong [Tue, 28 May 2019 23:48:44 +0000 (23:48 +0000)]
v2writable: introduce $sync state and put mm_tmp in it

A first step towards making the v2 index_sync code
easier-to-follow.  More fields to follow...

5 years agov2: fix reindex skipping NNTP article numbers
Eric Wong [Mon, 27 May 2019 18:45:45 +0000 (18:45 +0000)]
v2: fix reindex skipping NNTP article numbers

`public-inbox-index --reindex' could cause NNTP article number
gaps to form when it also has to deal with new,
never-before-seen commits in mirrors running off `git fetch'.

Fix this by running two distinct invocations of ->index_sync;
once to only reindex old commits, and a second time to index
new commits.

This does not appear to be a problem on v1 at the moment,
but I'll need more time to analyze this.

5 years agosearchidx: fix obvious typo
Eric Wong [Mon, 27 May 2019 18:45:44 +0000 (18:45 +0000)]
searchidx: fix obvious typo

We can't pass an empty string to `git merge-base --is-ancestor'
AFAIK, this did NOT present issues in the current test suite.

5 years agot/v1reindex.t: fix typo in setting `indexlevel'
Eric Wong [Mon, 27 May 2019 18:45:43 +0000 (18:45 +0000)]
t/v1reindex.t: fix typo in setting `indexlevel'

It did not cause a test failure because the default fallback
is `indexlevel=full'

5 years agoviewvcs: keep temporary Solver dir for large streams
Eric Wong [Sun, 26 May 2019 08:24:46 +0000 (08:24 +0000)]
viewvcs: keep temporary Solver dir for large streams

Streaming large blobs can take multiple iterations of the event
loop in our -httpd; so we must not let the File::Temp::Dir
result go out-of-scope when streaming large blobs created from
patches.

5 years agov2writable: fix assertions around reindexing
Eric Wong [Sat, 25 May 2019 19:18:09 +0000 (19:18 +0000)]
v2writable: fix assertions around reindexing

Fix a misspelling and ensure line context is printed by
`die' by leaving out the final '\n'.  Also, `delete' was
pointless.

5 years agocontrib/css: mark as CC0 (public domain)
Eric Wong [Sat, 25 May 2019 19:10:00 +0000 (19:10 +0000)]
contrib/css: mark as CC0 (public domain)

No reason to copyright colour schemes :P

5 years agov2writable: drop unused $last_commits var
Eric Wong [Sat, 25 May 2019 09:42:03 +0000 (09:42 +0000)]
v2writable: drop unused $last_commits var

Apparently it's never been used and we write to msgmap directly.

5 years agot/indexlevels: fix indexlevel of ro_mirror
Eric Wong [Sat, 25 May 2019 09:42:08 +0000 (09:42 +0000)]
t/indexlevels: fix indexlevel of ro_mirror

Don't hard-code "basic", since we already ran -init with the
intended indexlevel.

5 years agomsgmap: remove double negative
Eric Wong [Sat, 25 May 2019 04:34:20 +0000 (04:34 +0000)]
msgmap: remove double negative

I have never not found double negatives to be confusing...

5 years agoTODO: more stuff: bundles, synonyms, dogfooding
Eric Wong [Fri, 24 May 2019 08:55:56 +0000 (08:55 +0000)]
TODO: more stuff: bundles, synonyms, dogfooding

git bundles could/should make self-hosting easier.

Being able to configure synonym (and spelling) lists would make
some searches more useful.

Might as well dogfood kernel stuff, too, given the overlap and
history between this project, git and the Linux kernel.  Would
be interesting to have *BSD folks throw their hat in the ring,
too.

Building/testing userspace stuff is often the most
time-consuming, but necessary to ensure future compatibility.

5 years agoMANIFEST: add extman.perl
Eric Wong [Fri, 24 May 2019 07:55:29 +0000 (07:55 +0000)]
MANIFEST: add extman.perl

Oops :x

5 years agodoc: add URLs for Xapian manpages
Eric Wong [Fri, 24 May 2019 02:57:58 +0000 (02:57 +0000)]
doc: add URLs for Xapian manpages

Since we go through the effort of hosting these manpages,
link to them.

5 years agodoc: xcpdb: add switch documentation
Eric Wong [Fri, 24 May 2019 02:57:57 +0000 (02:57 +0000)]
doc: xcpdb: add switch documentation

In particular, the '--compact' switch is really useful since it
works without holding the inbox-wide lock for minutes at a time
on giant inboxes (inboxes where copies can take dozens, if not
hundreds of minutes).

5 years agodoc: generate manpages for some Xapian commands
Eric Wong [Fri, 24 May 2019 02:57:56 +0000 (02:57 +0000)]
doc: generate manpages for some Xapian commands

They're nowhere to be found on Xapian.org, and links to
external services are either too long (for manpages.debian.org)
or have privacy-invasive tracking JS on them.

5 years agodoc: sync .txt mtime to .pod mtime
Eric Wong [Fri, 24 May 2019 02:57:55 +0000 (02:57 +0000)]
doc: sync .txt mtime to .pod mtime

Otherwise timestamps for .html files get screwed up, too;
and that hurts caching.

5 years agodoc: don't barf on missing `git set-file-times'
Eric Wong [Fri, 24 May 2019 02:57:54 +0000 (02:57 +0000)]
doc: don't barf on missing `git set-file-times'

It's not critical, but it's nice to have for cache-friendliness
(otherwise I would not have written it :P)

I guess I should follow up on getting it into 'git contrib/':
https://public-inbox.org/git/20100702033709.GA6818@burratino/

5 years agodoc: daemon: fix manpage section for nginx
Eric Wong [Fri, 24 May 2019 02:57:53 +0000 (02:57 +0000)]
doc: daemon: fix manpage section for nginx

The nginx manpage is in section 8.

5 years agodoc: index: fix miscapitalization of "SQLite"
Eric Wong [Fri, 24 May 2019 02:57:52 +0000 (02:57 +0000)]
doc: index: fix miscapitalization of "SQLite"

Oops :x

5 years agosearch: don't log all warnings on retry_reopen
Eric Wong [Fri, 24 May 2019 01:12:08 +0000 (01:12 +0000)]
search: don't log all warnings on retry_reopen

Some users (or bots :P) can trigger horrible queries which
the caller can choose to either log or ignore.  This prevents
horrible queries from ExtMsg from logging confusing "ref: "
messages when $@ is not a Perl reference.

5 years agodoc: various updates to reflect current state
Eric Wong [Thu, 23 May 2019 10:37:38 +0000 (10:37 +0000)]
doc: various updates to reflect current state

-index documentation avoid redundant v1 information and refers
readers to apropriate v1/v2 manpages.  Search::Xapian can also
be optional, now, as only the PSGI search interface uses it.

Favor "INBOX_DIR" where appropriate, since "REPO_DIR" can be
confused for code repos which we also support.

XAPIAN_FLUSH_THRESHOLD is documented for all relevant
bulk commands.

5 years agoxapcmd: do not reset %SIG until last Xtmpdir is done
Eric Wong [Thu, 23 May 2019 09:37:04 +0000 (09:37 +0000)]
xapcmd: do not reset %SIG until last Xtmpdir is done

To properly handle compact tmpdir cleanup in single process
situations, we need to carefully account for Xtmpdir not
being a singleton and ensuring we don't clobber signal
handlers which belong to other Xtmpdirs.

5 years agoxcpdb|compact: support --jobs/-j flag like gmake(1)
Eric Wong [Thu, 23 May 2019 09:37:03 +0000 (09:37 +0000)]
xcpdb|compact: support --jobs/-j flag like gmake(1)

We don't have to be tied to the number of partitions in case
we made a bad choice at initialization.  This doesn't affect
reindexing, but the copying phase is already intensive.

And optimize away the extra process when we only have a single
job which won't parallelize.

The wording for the (v2) reindexing phase could be improved,
later.  I also plan to allow repartitioning of existing
Xapian DBs.

5 years agoxapcmd: cleanup on interrupted xcpdb "--compact"
Eric Wong [Thu, 23 May 2019 09:37:02 +0000 (09:37 +0000)]
xapcmd: cleanup on interrupted xcpdb "--compact"

We should not have leftover junk on interrupted invocations.

5 years agoxcpdb|compact: support some xapian-compact switches
Eric Wong [Thu, 23 May 2019 09:37:01 +0000 (09:37 +0000)]
xcpdb|compact: support some xapian-compact switches

Allow users to specify the --blocksize <B>, --no-full, --fuller
options for xapian-compact(1) for fine-tuning compact behavior
for low-traffic/inactive inboxes.

We also won't support --multipass, since it doesn't seem
compatible with our requirement to use --no-renumber.

We also won't support --single-file, since it only seems
intended for totally dead inboxes; and it doesn't seem
worth the support overhead when "totally dead" turns out
to be a misdiagnosis.

5 years agocompact: reuse infrastructure from xcpdb
Eric Wong [Thu, 23 May 2019 09:37:00 +0000 (09:37 +0000)]
compact: reuse infrastructure from xcpdb

Since -xcpdb is a superset of -compact, we can reuse much of
that code used for driving compact.

For compact (only), this is slightly less memory efficient since
it requires an extra process per-partition, but we get to prefix
the output with the partition name for more readable output.

5 years agoxcpdb: remove temporary directories on aborts
Eric Wong [Thu, 23 May 2019 09:36:59 +0000 (09:36 +0000)]
xcpdb: remove temporary directories on aborts

Cleanup temporary directories on common termination signals
(INT, HUP, PIPE, TERM), but only if it's not in the process
of being committed via rename() sequence.

5 years agoxcpdb: show re-indexing progress
Eric Wong [Thu, 23 May 2019 09:36:58 +0000 (09:36 +0000)]
xcpdb: show re-indexing progress

Emit information about reindexing git revision ranges when used
with xcpdb.  Additionally, distinguish Xapian copy output from
v2 git epoch counting by increasing directory context info.

For now, v1 batches batches are emitted.  v2 indexing is still
missing progress reporting for batches, as the data structures
for reindexing would benefit from a refactoring, first.

This does not currently affect the use of public-inbox-index,
but may in the future.

5 years agoxapcmd: use "print STDERR" for progress reporting
Eric Wong [Thu, 23 May 2019 09:36:57 +0000 (09:36 +0000)]
xapcmd: use "print STDERR" for progress reporting

`warn' is reserved for actual warnings, as it respects
$SIG{__WARN__} and we rely on that override to print
message context information when we are indexing.

5 years agodoc: xcpdb: update to reflect the current state
Eric Wong [Thu, 23 May 2019 09:36:56 +0000 (09:36 +0000)]
doc: xcpdb: update to reflect the current state

It is no longer a wrapper around copydatabase(1), since
copydatabase did not recover from DatabaseModifiedError.

5 years agoxapcmd: avoid EXDEV when finalizing changes
Eric Wong [Thu, 23 May 2019 09:36:55 +0000 (09:36 +0000)]
xapcmd: avoid EXDEV when finalizing changes

By creating temporary directories as deep as possible,
we can allow v2 repositories to have `xap$SCHEMA_VERSION'
(e.g. `xap15') reside on a separate FS.

We also check st_dev ahead-of-time to avoid doing work which
will fail with EXDEV.  Of course, another process may still
move/change things around.

5 years agoxcpdb: cleanup error handling and diagnosis
Eric Wong [Thu, 23 May 2019 09:36:54 +0000 (09:36 +0000)]
xcpdb: cleanup error handling and diagnosis

Running a full "public-inbox-index --reindex" in parallel
with "public-inbox-xcpdb" on the same inbox can still cause
problems, though.

5 years agoxcpdb: implement progress reporting
Eric Wong [Thu, 23 May 2019 09:36:53 +0000 (09:36 +0000)]
xcpdb: implement progress reporting

Copying an entire Xapian DB is horribly slow whether it's done
via Perl or copydatabase(1).  So displaying some progress
indication is good for user experience.

While we're at it, prefix xapian-compact output, too; since
parallel processes end up clobbering each other.

5 years agoxcpdb: use fine-grained locking
Eric Wong [Thu, 23 May 2019 09:36:52 +0000 (09:36 +0000)]
xcpdb: use fine-grained locking

Copying an entire Xapian DB takes a long time, so update our
reindexing code to support partial reindexing, snapshot the
pre-copydatabase git revisions, perform the lengthy copy,
and do a partial reindex when the copy + renames are done.