]> Sergey Matveev's repositories - public-inbox.git/log
public-inbox.git
5 years agods: import IO::KQueue namespace
Eric Wong [Mon, 24 Jun 2019 02:52:19 +0000 (02:52 +0000)]
ds: import IO::KQueue namespace

Make the rest of our IO::KQueue-using code less verbose and
closer to the C equivalent.

5 years agods: set event flags directly at initialization
Eric Wong [Mon, 24 Jun 2019 02:52:18 +0000 (02:52 +0000)]
ds: set event flags directly at initialization

We can avoid the EPOLL_CTL_ADD && EPOLL_CTL_MOD sequence with
a single EPOLL_CTL_ADD.

5 years agosyscall: get rid of unnecessary uname local vars
Eric Wong [Mon, 24 Jun 2019 02:52:17 +0000 (02:52 +0000)]
syscall: get rid of unnecessary uname local vars

We don't need to keep information from uname(2) around outside
of startup.

5 years agosyscall: get rid of unused EPOLL* constants
Eric Wong [Mon, 24 Jun 2019 02:52:16 +0000 (02:52 +0000)]
syscall: get rid of unused EPOLL* constants

EPOLLRDBAND is used for DECnet; and I'm pretty sure I won't be
updating any of our code to work with DECnet.

I've never found use for EPOLLHUP or EPOLLERR, either; so
disable those for now and add comments for things I might
actually use: EPOLLET and EPOLLONESHOT.

5 years agods: get rid of redundant and unnecessary POLL* constants
Eric Wong [Mon, 24 Jun 2019 02:52:15 +0000 (02:52 +0000)]
ds: get rid of redundant and unnecessary POLL* constants

EPOLL* constants already match their POLL* counterparts and
there's no way Linux can ever diverge or change the values
of those constants.  So we'll favor the EPOLL* ones since we
use EPOLLEXCLUSIVE, already.

For weird stuff like kqueue, we'd need to keep maintaining
the mapping, anyways.

5 years agods: switch write buffering to use a tempfile
Eric Wong [Mon, 24 Jun 2019 02:52:14 +0000 (02:52 +0000)]
ds: switch write buffering to use a tempfile

Data which can't fit into a generously-sized socket buffer,
has no business being stored in heap.

5 years agods: share send(..., MSG_MORE) logic
Eric Wong [Mon, 24 Jun 2019 02:52:13 +0000 (02:52 +0000)]
ds: share send(..., MSG_MORE) logic

No sense in having similar Linux-specific functionality in
both our NNTP.pm and HTTP.pm

5 years agohttp: favor DS->write(strref) when reasonable
Eric Wong [Mon, 24 Jun 2019 02:52:12 +0000 (02:52 +0000)]
http: favor DS->write(strref) when reasonable

This can avoid large memory copies when strings can't be
copy-on-write and saves us the trouble of creating new
refs in the code.

5 years agods: remove support for DS->write(undef)
Eric Wong [Mon, 24 Jun 2019 02:52:11 +0000 (02:52 +0000)]
ds: remove support for DS->write(undef)

We call ->flush_write directly, now; so we can eliminate a
needless check.

5 years agods: don't pass `events' arg to EPOLL_CTL_DEL
Eric Wong [Mon, 24 Jun 2019 02:52:10 +0000 (02:52 +0000)]
ds: don't pass `events' arg to EPOLL_CTL_DEL

There's no point in passing a mask of interesting events
when removing an item from the epoll watch set.

5 years agods: lazy-initialize wbuf
Eric Wong [Mon, 24 Jun 2019 02:52:09 +0000 (02:52 +0000)]
ds: lazy-initialize wbuf

We don't need write buffering unless we encounter slow clients
requesting large responses.  So don't waste a hash slot or
(empty) arrayref for it.

5 years agods: split out from ->flush_write and ->write
Eric Wong [Mon, 24 Jun 2019 02:52:08 +0000 (02:52 +0000)]
ds: split out from ->flush_write and ->write

Get rid of the confusing $need_queue variable and all
the associated documentation for it.  Instead, make it
obvious that we're either skipping the write buffer or
flushing the write buffer by splitting the sub in two.

5 years agods: lazy initialize wbuf_off
Eric Wong [Mon, 24 Jun 2019 02:52:07 +0000 (02:52 +0000)]
ds: lazy initialize wbuf_off

Since Perl 5.10+, "fields" makes a restricted hash; not a
compile-time-defined array (struct) with fixed offsets as
it did in Perl <= 5.8.

Thus in-use fields cost memory, and since the write buffer
offset is rarely needed; stop relying on it.

5 years agods: get rid of on_incomplete_write wrapper
Eric Wong [Mon, 24 Jun 2019 02:52:06 +0000 (02:52 +0000)]
ds: get rid of on_incomplete_write wrapper

Wrong place to be wrapping this method.

5 years agoAddTimer: avoid clock_gettime for the '0' case
Eric Wong [Mon, 24 Jun 2019 02:52:05 +0000 (02:52 +0000)]
AddTimer: avoid clock_gettime for the '0' case

We rely on immediate timers often, so we can avoid the overhead
of an extra subroutine call to retrieve the monotonic time (and
a sometimes-system call on some platforms).

5 years agods: use and export monotonic now()
Eric Wong [Mon, 24 Jun 2019 02:52:04 +0000 (02:52 +0000)]
ds: use and export monotonic now()

All of our internal timing code should use monotonic clocks
for consistency against system clock adjustments.

This can be shared by our Daemon and NNTP packages.

5 years agods: get rid of more unused debug instance methods
Eric Wong [Mon, 24 Jun 2019 02:52:03 +0000 (02:52 +0000)]
ds: get rid of more unused debug instance methods

Over a decade of using Danga::Socket and I never found the
built-in debug functionality useful.

5 years agods: get rid of {closed} field
Eric Wong [Mon, 24 Jun 2019 02:52:02 +0000 (02:52 +0000)]
ds: get rid of {closed} field

Merely checking the presence of the {sock} field is
enough, and having multiple sources of truth increases
confusion and the likelyhood of bugs.

5 years agot/httpd-corner: ensure chunk payload read doesn't overreach
Eric Wong [Thu, 20 Jun 2019 01:03:47 +0000 (01:03 +0000)]
t/httpd-corner: ensure chunk payload read doesn't overreach

It never has, AFAIK, but I'm making some changes to this code in
another branch and nearly introduced a bug where it would be
overreading and discarding the pipelined request.

5 years agot/httpd-corner.t: fix braino :x
Eric Wong [Thu, 20 Jun 2019 00:16:17 +0000 (00:16 +0000)]
t/httpd-corner.t: fix braino :x

Plack is for Perl, Rack is for Ruby; this a Perl project :x

5 years agods: stop distinguishing event read and write callbacks
Eric Wong [Sun, 16 Jun 2019 06:11:28 +0000 (06:11 +0000)]
ds: stop distinguishing event read and write callbacks

Having separate read/write callbacks in every class is too
confusing to my easily-confused mind.  Instead, give every class
an "event_step" callback which is easier to wrap my head around.

This will make future code to support IO::Socket::SSL-wrapped
sockets easier-to-digest, since SSL_write() can require waiting
on POLLIN events, and SSL_read() can require waiting on POLLOUT
events.

5 years agot/replace.t: fix SKIP label for testing w/o Xapian
Eric Wong [Sun, 16 Jun 2019 03:22:25 +0000 (03:22 +0000)]
t/replace.t: fix SKIP label for testing w/o Xapian

5 years agoxcpdb: don't warn on --jobs != --reshard
Eric Wong [Sun, 16 Jun 2019 01:32:01 +0000 (01:32 +0000)]
xcpdb: don't warn on --jobs != --reshard

It's slightly confusing since we dedicate one job
to dealing with fast-import + SQLite indexing; and
it's not worth complaining about when it happens.

5 years agoMerge remote-tracking branch 'origin/newspeak' into xcpdb
Eric Wong [Sun, 16 Jun 2019 01:04:28 +0000 (01:04 +0000)]
Merge remote-tracking branch 'origin/newspeak' into xcpdb

* origin/newspeak:
  comments: replace "partition" with "shard"
  t/xcpdb-reshard: use 'shard' term in local variables
  xapcmd: favor 'shard' over 'part' in local variables
  search: use "shard" for local variable
  v2writable: use "epoch" consistently when referring to git repos
  adminedit: "part" => "shard" for local variables
  v2writable: rename local vars to match Xapian terminology
  v2writable: avoid "part" in internal subs and fields
  search*: rename {partition} => {shard}
  xapcmd: update comments referencing "partitions"
  v2: rename SearchIdxPart => SearchIdxShard
  inboxwritable: s/partitions/shards/ in local var
  tests: change messages to use "shard" instead of partition
  v2writable: rename {partitions} field to {shards}
  v2writable: count_partitions => count_shards
  searchidxpart: start using "shard" in user-visible places
  rename reference to git epochs as "partitions"
  admin|xapcmd: user-facing messages say "shard"
  v2writable: update comments regarding xcpdb --reshard
  doc: rename our Xapian "partitions" to "shards"

5 years agot/psgi_search.t: use higher-level APIs
Eric Wong [Sun, 16 Jun 2019 00:57:36 +0000 (00:57 +0000)]
t/psgi_search.t: use higher-level APIs

No point in using lower-level APIs for a PSGI test.

5 years agosearchview: add link at bottom to reverse results
Eric Wong [Sat, 15 Jun 2019 20:23:42 +0000 (20:23 +0000)]
searchview: add link at bottom to reverse results

I could not find a place to put the link the top without
making navigation too cluttered.  Putting it at the bottom
of the page seems reasonable...

5 years agosearchview: support negative offsets to reverse ordering
Eric Wong [Sat, 15 Jun 2019 20:23:41 +0000 (20:23 +0000)]
searchview: support negative offsets to reverse ordering

Taking a hint from Perl array access, we'll allow negative
offsets for the 'o' parameter and to reverse the sort order.

5 years agoMerge remote-tracking branch 'origin/ds'
Eric Wong [Sat, 15 Jun 2019 17:38:42 +0000 (17:38 +0000)]
Merge remote-tracking branch 'origin/ds'

* origin/ds:
  ds: stop caring about event flags set by epoll/poll/kqueue
  ds: do not distinguish between POLLHUP and POLLERR
  ds: remove read method, here, too
  nntp: use sysread to append to existing buffer
  ds: remove steal_socket method
  ds: remove {fd} field
  ds: reduce Errno imports and drop ->close reason
  ds: cleanup Errno imports and favor constant comparisons
  ds: simplify write buffer accounting

5 years agot/git-http-backend: explain purpose of test
Eric Wong [Fri, 14 Jun 2019 10:09:13 +0000 (10:09 +0000)]
t/git-http-backend: explain purpose of test

I found myself tempted to switch to HTTP::Tiny, here, since
it's distributed with Perl since 5.14, unlike Net::HTTP
(which AFAIK was never a part of Perl proper).

But we really want to use Net::HTTP, here, since it's
lower-level and allows us to trigger server-side buffering
by not reading the entity body.

5 years agocomments: replace "partition" with "shard"
Eric Wong [Sat, 15 Jun 2019 08:25:44 +0000 (08:25 +0000)]
comments: replace "partition" with "shard"

Now that the code matches Xapian terminology, ensure
our comments match, too.

5 years agot/xcpdb-reshard: use 'shard' term in local variables
Eric Wong [Fri, 14 Jun 2019 21:30:31 +0000 (21:30 +0000)]
t/xcpdb-reshard: use 'shard' term in local variables

Another step in maintaining consistency with Xapian docs.

5 years agoxapcmd: favor 'shard' over 'part' in local variables
Eric Wong [Fri, 14 Jun 2019 21:29:37 +0000 (21:29 +0000)]
xapcmd: favor 'shard' over 'part' in local variables

Yet another step to keeping our naming consistent with Xapian
terminology.

5 years agosearch: use "shard" for local variable
Eric Wong [Fri, 14 Jun 2019 18:18:06 +0000 (18:18 +0000)]
search: use "shard" for local variable

Another small step towards terminology consistency with Xapian.

5 years agov2writable: use "epoch" consistently when referring to git repos
Eric Wong [Fri, 14 Jun 2019 18:16:09 +0000 (18:16 +0000)]
v2writable: use "epoch" consistently when referring to git repos

Be consistent with our own terminology and use "epoch" for
[0-9]+\.git repos.  The term "partition" is going away entirely.

5 years agoadminedit: "part" => "shard" for local variables
Eric Wong [Fri, 14 Jun 2019 18:14:42 +0000 (18:14 +0000)]
adminedit: "part" => "shard" for local variables

5 years agov2writable: rename local vars to match Xapian terminology
Eric Wong [Fri, 14 Jun 2019 18:00:20 +0000 (18:00 +0000)]
v2writable: rename local vars to match Xapian terminology

5 years agov2writable: avoid "part" in internal subs and fields
Eric Wong [Fri, 14 Jun 2019 17:50:45 +0000 (17:50 +0000)]
v2writable: avoid "part" in internal subs and fields

We'll be using the term "shard" from now on to be consistent
with Xapian terminology.

5 years agosearch*: rename {partition} => {shard}
Eric Wong [Fri, 14 Jun 2019 17:42:57 +0000 (17:42 +0000)]
search*: rename {partition} => {shard}

Another step towards keeping our internal data structures
consistent with Xapian naming.

5 years agoxapcmd: update comments referencing "partitions"
Eric Wong [Fri, 14 Jun 2019 17:38:01 +0000 (17:38 +0000)]
xapcmd: update comments referencing "partitions"

Don't confuse future readers of our code.

5 years agov2: rename SearchIdxPart => SearchIdxShard
Eric Wong [Fri, 14 Jun 2019 17:35:04 +0000 (17:35 +0000)]
v2: rename SearchIdxPart => SearchIdxShard

Another step towards keeping our file and package names
consistent with Xapian terminology.

5 years agoinboxwritable: s/partitions/shards/ in local var
Eric Wong [Fri, 14 Jun 2019 17:31:28 +0000 (17:31 +0000)]
inboxwritable: s/partitions/shards/ in local var

More work towards being consistent with Xapian's own terminology

5 years agotests: change messages to use "shard" instead of partition
Eric Wong [Fri, 14 Jun 2019 08:02:32 +0000 (08:02 +0000)]
tests: change messages to use "shard" instead of partition

Another potentially user-facing piece made consistent with
Xapian terminology.

5 years agov2writable: rename {partitions} field to {shards}
Eric Wong [Fri, 14 Jun 2019 07:59:53 +0000 (07:59 +0000)]
v2writable: rename {partitions} field to {shards}

Our internal data structure should be consistent with Xapian
terminology.

5 years agov2writable: count_partitions => count_shards
Eric Wong [Fri, 14 Jun 2019 07:56:58 +0000 (07:56 +0000)]
v2writable: count_partitions => count_shards

Another step towards becoming consistent with Xapian terminology

5 years agosearchidxpart: start using "shard" in user-visible places
Eric Wong [Fri, 14 Jun 2019 07:55:35 +0000 (07:55 +0000)]
searchidxpart: start using "shard" in user-visible places

We'll name our process title with "shard" instead, and
update a few error messages and comments to match.

5 years agorename reference to git epochs as "partitions"
Eric Wong [Fri, 14 Jun 2019 07:51:45 +0000 (07:51 +0000)]
rename reference to git epochs as "partitions"

Try to remain consistent with our own documentation regarding
v2 git "epochs", first.

5 years agoadmin|xapcmd: user-facing messages say "shard"
Eric Wong [Fri, 14 Jun 2019 07:37:03 +0000 (07:37 +0000)]
admin|xapcmd: user-facing messages say "shard"

We're slowly getting rid of the word "partition" when it
comes to remain consistent with Xapian docs.

5 years agov2writable: update comments regarding xcpdb --reshard
Eric Wong [Fri, 14 Jun 2019 07:34:07 +0000 (07:34 +0000)]
v2writable: update comments regarding xcpdb --reshard

Using compact to change shard count was abandoned during
the v2 development phase.

5 years agodoc: rename our Xapian "partitions" to "shards"
Eric Wong [Fri, 14 Jun 2019 06:38:58 +0000 (06:38 +0000)]
doc: rename our Xapian "partitions" to "shards"

For consistency with Xapian documentation (in the "master"
branch).

5 years agov2writable: fix brainfart when capping NPROC_MAX_DEFAULT
Eric Wong [Fri, 14 Jun 2019 21:43:16 +0000 (21:43 +0000)]
v2writable: fix brainfart when capping NPROC_MAX_DEFAULT

Oops :x

5 years agosearch: require PublicInbox::Inbox ref here
Eric Wong [Fri, 14 Jun 2019 17:13:04 +0000 (17:13 +0000)]
search: require PublicInbox::Inbox ref here

No sense in supporting multiple methods of initialization
for an internal class.

5 years agosearchidx: require PublicInbox::Inbox (or InboxWritable) ref
Eric Wong [Fri, 14 Jun 2019 17:01:55 +0000 (17:01 +0000)]
searchidx: require PublicInbox::Inbox (or InboxWritable) ref

PublicInbox::Inbox objects have minimal dependencies, so
drop code to support old tests which existed before the
PublicInbox::Inbox object came into existence.

5 years agoMakefile.PL: add test scripts to syntax checks
Eric Wong [Fri, 14 Jun 2019 16:47:44 +0000 (16:47 +0000)]
Makefile.PL: add test scripts to syntax checks

I make syntax errors all the time :x

5 years agot/www_listing: favor HTTP::Tiny over Net::HTTP
Eric Wong [Fri, 14 Jun 2019 16:42:26 +0000 (16:42 +0000)]
t/www_listing: favor HTTP::Tiny over Net::HTTP

More testers are likely to have HTTP::Tiny than Net::HTTP, since
HTTP::Tiny is a dual-life module and distributed with Perl since
Perl 5.14 (2011-05-14), whereas Net::HTTP will likely live in
a separate package forever.

5 years agoedit: fix portability of editor invocation
Eric Wong [Mon, 10 Jun 2019 23:53:46 +0000 (23:53 +0000)]
edit: fix portability of editor invocation

The eval was unnecessary, and $0 can't be "--".
Tested with /bin/sh on FreeBSD 11.2

5 years agoMerge remote-tracking branch 'origin/reshard' into next
Eric Wong [Fri, 14 Jun 2019 16:25:39 +0000 (16:25 +0000)]
Merge remote-tracking branch 'origin/reshard' into next

* origin/reshard:
  xcpdb: support resharding v2 repos
  xcpdb: use destination shard as progress prefix
  xapcmd: preserve indexlevel based on the destination
  v2writable: use a smaller default for Xapian partitions

5 years agoMerge remote-tracking branch 'origin/manifest' into next
Eric Wong [Fri, 14 Jun 2019 16:23:13 +0000 (16:23 +0000)]
Merge remote-tracking branch 'origin/manifest' into next

* origin/manifest:
  git: ensure ->modified returns an integer
  www: support $INBOX/git/$EPOCH.git for v2 cloning
  www: wire up /$INBOX/manifest.js.gz, too
  wwwlisting: generate grokmirror-compatible manifest.js.gz
  wwwlisting: allow hiding entries from manifest

5 years agoMerge remote-tracking branch 'origin/edit' into next
Eric Wong [Fri, 14 Jun 2019 16:23:00 +0000 (16:23 +0000)]
Merge remote-tracking branch 'origin/edit' into next

* origin/edit:
  edit: unlink temporary file when done
  v2writable: replace: kill git processes before reindexing
  edit: drop unwanted headers before noop check
  edit|purge: improve output on rewrites
  edit: new tool to perform edits
  doc: document the --prune option for -index
  admin: expose ->config
  AdminEdit: move editability checks from -purge
  admin: beef up resolve_inboxes to handle purge options
  purge: start moving common options to AdminEdit module
  admin: remove warning arg for unconfigured inboxes
  v2writable: implement ->replace call
  import: switch to "replace_oids" interface for purge
  import: extract_author_info becomes extract_commit_info
  v2writable: consolidate overview and indexing call

5 years agoxcpdb: support resharding v2 repos
Eric Wong [Thu, 13 Jun 2019 00:29:37 +0000 (00:29 +0000)]
xcpdb: support resharding v2 repos

v2 repos are sometimes created on machines where CPU
parallelization exceeds the capability of the storage devices.

In that case, users may reshard the Xapian DB to any smaller,
positive integer to avoid excessive overhead and contention when
bottlenecked by slow storage.

Resharding can also be used to increase shard count after
hardware upgrades.

5 years agoxcpdb: use destination shard as progress prefix
Eric Wong [Wed, 12 Jun 2019 08:10:04 +0000 (08:10 +0000)]
xcpdb: use destination shard as progress prefix

For M:N resharding, we'll want to display the number from
the new shard number.

5 years agoxapcmd: preserve indexlevel based on the destination
Eric Wong [Wed, 12 Jun 2019 08:05:33 +0000 (08:05 +0000)]
xapcmd: preserve indexlevel based on the destination

To support M:N resharding, we need to ensure we store the
indexlevel in the destination shard, rather than the
originating one.

5 years agov2writable: use a smaller default for Xapian partitions
Eric Wong [Wed, 12 Jun 2019 00:35:32 +0000 (00:35 +0000)]
v2writable: use a smaller default for Xapian partitions

Apparently 16 CPUs (probably HT) and SATA storage is common
these days.  Having excessive Xapian partitions leads to
contention and excessive FD/space use.  So set a smaller
default but continue allowing user-specified values to bump
this up.

5 years agogit: remove cat_file sub callback interface
Eric Wong [Thu, 13 Jun 2019 08:10:02 +0000 (08:10 +0000)]
git: remove cat_file sub callback interface

We weren't using it, and in retrospect, it makes no sense to use
this API cat_file for giant responses which can't read quickly
with minimal context-switching (or sanely fit into memory for
Email::Simple/Email::MIME).

For giant blobs which we don't want slurped in memory, we'll
spawn a short-lived git-cat-file process like we do in ViewVCS.

Otherwise, monopolizing a git-cat-file process for a giant
blob is harmful to other PSGI/NNTP users.

A better interface is coming which will be more suitable for
for batch processing of "small" objects such as commits and
email blobs.

5 years agonntp: filter out duplicate Message-IDs for leafnode
Eric Wong [Fri, 14 Jun 2019 00:27:31 +0000 (00:27 +0000)]
nntp: filter out duplicate Message-IDs for leafnode

It's the unfortunate reality that there are some clients which
reuse Message-IDs (in which we generate + use another) or set
multiple Message-IDs on their own.  While the v2 format
addresses that, NNTP clients such as leafnode are not always
prepared to deal with that case.

So, ensure NNTP clients only see a single Message-ID, and
show the others as 'X-Alt-Message-ID'.

5 years agonntp: ensure Message-ID is not folded for leafnode
Eric Wong [Thu, 13 Jun 2019 20:46:52 +0000 (20:46 +0000)]
nntp: ensure Message-ID is not folded for leafnode

Leafnode cannot handle Message-ID headers which are too long and
require folding via Email::Simple::Header.  Since there are
already many of these messages in git with the header already
folded, we need to handle the unfolding when emitting the
message via NNTP.

As far as we know, Leafnode is the only client software
incapable of handling this case.

5 years agonntp: add Path: header for leafnode
Eric Wong [Thu, 13 Jun 2019 06:54:13 +0000 (06:54 +0000)]
nntp: add Path: header for leafnode

Apparently leafnode just needs any junk in the Path: header.
Lets not waste bandwidth and just use a single byte to keep
leafnode happy.

Cc: Dave Taht <dave@taht.net>
5 years agoMakefile.PL: "dsyn" target to check syntax of changed files
Eric Wong [Thu, 13 Jun 2019 03:44:41 +0000 (03:44 +0000)]
Makefile.PL: "dsyn" target to check syntax of changed files

We have lots of files and syntax-checking every single one of
them is slow.  Enable "perl -w" in the existing syntax check
while we're at it.

5 years agodoc: update dependencies for CentOS 7.x
Eric Wong [Thu, 13 Jun 2019 02:26:30 +0000 (02:26 +0000)]
doc: update dependencies for CentOS 7.x

Digest::SHA is the most notable missing package at runtime
for a minimal system.

Tests don't run at all without Test::Simple (or Test::More).
Plack::Test is also a separate package, too...

Also, the package for IO::Compress::Gzip should be IO::Compress;
as perl-PerlIO-gzip is a different thing entirely which is not
relevant to our needs.

Test::HTTP::Server::Simple doesn't seem required at all for Plack
tests.

ExtUtils::MakeMaker needs to be documented as a install dependency
for people installing this, too; since AFAIK public-inbox is not
yet in any distros.

5 years agot/common.perl: fix error message for git requirements
Eric Wong [Thu, 13 Jun 2019 01:53:18 +0000 (01:53 +0000)]
t/common.perl: fix error message for git requirements

And enable strict + warnings in the scope of t/common.perl, too.

5 years agosearchmsg: remove unused ->get subroutine
Eric Wong [Thu, 13 Jun 2019 01:49:11 +0000 (01:49 +0000)]
searchmsg: remove unused ->get subroutine

It's obsolete and unusable since our search schema version 15;
which made the Xapian document ID correspond to the NNTP article
number.

5 years agosearchidx: improve error message when Xapian fails
Eric Wong [Wed, 12 Jun 2019 00:18:01 +0000 (00:18 +0000)]
searchidx: improve error message when Xapian fails

Make it easier to detect if a partition is corrupt.

5 years agoedit: unlink temporary file when done
Eric Wong (Contractor, The Linux Foundation) [Tue, 11 Jun 2019 18:34:58 +0000 (18:34 +0000)]
edit: unlink temporary file when done

We don't need to leave temporary files lying around.

5 years agov2writable: replace: kill git processes before reindexing
Eric Wong (Contractor, The Linux Foundation) [Mon, 10 Jun 2019 21:43:52 +0000 (21:43 +0000)]
v2writable: replace: kill git processes before reindexing

Xapian on Linux <3.15 has trouble with coprocesses since it used
fork() for locking and would hold onto pipes used for git
unnecessarily.

5 years agoedit: drop unwanted headers before noop check
Eric Wong (Contractor, The Linux Foundation) [Mon, 10 Jun 2019 18:09:27 +0000 (18:09 +0000)]
edit: drop unwanted headers before noop check

mutt will set Content-Length, Lines, and Status headers
unconditionally, so we need to account for that before
doing header comparisons to avoid making expensive changes
when noop edits are made.

5 years agoedit|purge: improve output on rewrites
Eric Wong (Contractor, The Linux Foundation) [Mon, 10 Jun 2019 16:58:55 +0000 (16:58 +0000)]
edit|purge: improve output on rewrites

Fill in undef as "(unchanged)" when displaying commits
and prefix the epoch name.

5 years agods: stop caring about event flags set by epoll/poll/kqueue
Eric Wong [Mon, 10 Jun 2019 03:21:37 +0000 (03:21 +0000)]
ds: stop caring about event flags set by epoll/poll/kqueue

If we got something to write, then write it.  Otherwise, try
reading; and continue dealing with errors which normally occur
along the way.

Trying to read requests while we need to buffer in luserspace
is suicidal from a memory management standpoint.

The only adjustment needed for existing callers is EvCleanup;
where we need to ensure we're always calling the dummy
EvCleanup::event_write callback to accomplish nothing.

5 years agods: do not distinguish between POLLHUP and POLLERR
Eric Wong [Mon, 10 Jun 2019 03:02:11 +0000 (03:02 +0000)]
ds: do not distinguish between POLLHUP and POLLERR

In my experience, both are worthless as any normal read/write
call path will be wanting to check errors and deal with them
appropriately; so we can just call event_read, for now.

Eventually, there'll probably be only one callback for dealing
with all in/out/err/hup events to simplify logic, especially w.r.t
TLS socket negotiation.

5 years agods: remove read method, here, too
Eric Wong [Mon, 10 Jun 2019 02:39:24 +0000 (02:39 +0000)]
ds: remove read method, here, too

Since we stop using it in NNTP, we don't need it at all.

5 years agonntp: use sysread to append to existing buffer
Eric Wong [Mon, 10 Jun 2019 02:34:48 +0000 (02:34 +0000)]
nntp: use sysread to append to existing buffer

We already do this in PublicInbox::HTTP, as it's superior to
DS::read in this regard.  Initially (when I started writing
NNTP.pm, I wanted to use Danga::Socket's read buffering and
push_back_read (removed in DS) but quickly figured out it wasn't
useful at all for dealing with trickling clients.

5 years agods: remove steal_socket method
Eric Wong [Mon, 10 Jun 2019 02:18:44 +0000 (02:18 +0000)]
ds: remove steal_socket method

We won't be needing it, not even for TLS support.

5 years agods: remove {fd} field
Eric Wong [Mon, 10 Jun 2019 02:07:26 +0000 (02:07 +0000)]
ds: remove {fd} field

Storing the file descriptor was redundant as we can quickly call
fileno($self->{sock}) and not have to store an extra hash table
entry.  Multiple sources of truth leads to confusion, confusion
leads to bugs.

5 years agods: reduce Errno imports and drop ->close reason
Eric Wong [Tue, 4 Jun 2019 12:15:45 +0000 (12:15 +0000)]
ds: reduce Errno imports and drop ->close reason

ECONNRESET and EPIPE are common on a big Internet filled with
unreliable connections, and there's nothing our code can do
about it.

So no point in wasting code to log them and there are plenty of
tracing tools to choose from if such diagnostics are needed.

5 years agods: cleanup Errno imports and favor constant comparisons
Eric Wong [Mon, 3 Jun 2019 09:11:00 +0000 (09:11 +0000)]
ds: cleanup Errno imports and favor constant comparisons

Stop importing unused constants, and favor integer comparisons
of `$!' over `$!{EFOO}' hash lookups.  Integer comparisons are
slightly faster, even:

Benchmark: timing 30 iterations of cmp_eq, cmp_ne, hash_hit, hash_miss...
    cmp_eq:  1 wallclock secs ( 1.61 usr +  0.00 sys =  1.61 CPU) @ 18.63/s (n=30)
    cmp_ne:  2 wallclock secs ( 1.57 usr +  0.00 sys =  1.57 CPU) @ 19.11/s (n=30)
  hash_hit:  4 wallclock secs ( 3.85 usr +  0.00 sys =  3.85 CPU) @  7.79/s (n=30)
 hash_miss:  4 wallclock secs ( 3.74 usr +  0.00 sys =  3.74 CPU) @  8.02/s (n=30)

#!/usr/bin/perl -w
use Benchmark qw(:all);
use Errno qw(EAGAIN EINTR);
my ($r, $w);
pipe($r, $w) or die 'pipe';
require IO::Handle;
$r->blocking(0);
my $buf;
my $n = 30000;
timethese(30, {
hash_hit => sub {
sysread($r, $buf, 1);
for (0..$n) {
next if $!{EAGAIN};
die 'FAIL';
}
}
,
'cmp_eq' => sub {
sysread($r, $buf, 1);
for (0..$n) {
next if $! == EAGAIN;
die 'FAIL';
}
},
hash_miss => sub {
sysread($r, $buf, 1);
for (0..$n) {
die 'FAIL' if $!{EINTR};
}
},
'cmp_ne' => sub {
sysread($r, $buf, 1);
for (0..$n) {
die 'FAIL' if $! == EINTR;
}
},
});

5 years agods: simplify write buffer accounting
Eric Wong [Mon, 10 Jun 2019 01:33:30 +0000 (01:33 +0000)]
ds: simplify write buffer accounting

Keeping track of write_buf_size was redundant and pointless when
we can simply check the number of elements in the buffer array.
Multiple sources of truth leads to confusion; confusion leads to
bugs.

Finally, rename the prefixes to 'wbuf' to ensure we loudly
(instead of silently) break any external dependencies being
ported over from Danga::Socket, as further changes are pending.

5 years agogit: ensure ->modified returns an integer
Eric Wong (Contractor, The Linux Foundation) [Sun, 9 Jun 2019 00:53:29 +0000 (00:53 +0000)]
git: ensure ->modified returns an integer

We don't want to serialize timestamps as strings to JSON.
I only noticed this bug on a 32-bit system.

5 years agodoc: index.pod: fix mismatched =back
Eric Wong [Mon, 10 Jun 2019 02:44:07 +0000 (02:44 +0000)]
doc: index.pod: fix mismatched =back

5 years agodoc: generate manpage and HTML from -convert POD
Eric Wong [Sun, 9 Jun 2019 09:10:30 +0000 (09:10 +0000)]
doc: generate manpage and HTML from -convert POD

It's been written for over a year, but I forgot to include
it in the build so it did not get installed or put on the site.

5 years agodoc: clean-doc target removes *.8 manpages
Eric Wong [Sun, 9 Jun 2019 09:10:26 +0000 (09:10 +0000)]
doc: clean-doc target removes *.8 manpages

5 years agoedit: new tool to perform edits
Eric Wong (Contractor, The Linux Foundation) [Sun, 9 Jun 2019 02:51:47 +0000 (02:51 +0000)]
edit: new tool to perform edits

This wrapper around V2Writable->replace provides a user-interface
for editing messages as single-message mboxes (or the raw text
via $EDITOR).

5 years agodoc: document the --prune option for -index
Eric Wong (Contractor, The Linux Foundation) [Sun, 9 Jun 2019 02:51:46 +0000 (02:51 +0000)]
doc: document the --prune option for -index

We've had it around for a while, but I forgot to document it :x

5 years agoadmin: expose ->config
Eric Wong (Contractor, The Linux Foundation) [Sun, 9 Jun 2019 02:51:45 +0000 (02:51 +0000)]
admin: expose ->config

No point in forcing admin programs to reparse the config
themselves; and we won't support multiple instances of it;
unlike the WWW code.

5 years agoAdminEdit: move editability checks from -purge
Eric Wong (Contractor, The Linux Foundation) [Sun, 9 Jun 2019 02:51:44 +0000 (02:51 +0000)]
AdminEdit: move editability checks from -purge

We'll be reusing the same logic for -edit

5 years agoadmin: beef up resolve_inboxes to handle purge options
Eric Wong (Contractor, The Linux Foundation) [Sun, 9 Jun 2019 02:51:43 +0000 (02:51 +0000)]
admin: beef up resolve_inboxes to handle purge options

We'll be using this in -edit, and maybe other admin-oriented
tools for UI-consistency.

5 years agopurge: start moving common options to AdminEdit module
Eric Wong (Contractor, The Linux Foundation) [Sun, 9 Jun 2019 02:51:42 +0000 (02:51 +0000)]
purge: start moving common options to AdminEdit module

Editing and purging are similar operations involving history
rewrites, so there'll be common options and code between them.

5 years agoadmin: remove warning arg for unconfigured inboxes
Eric Wong (Contractor, The Linux Foundation) [Sun, 9 Jun 2019 02:51:41 +0000 (02:51 +0000)]
admin: remove warning arg for unconfigured inboxes

We no longer make -index warn on it, no other code uses it;
and working on unconfigured inboxes is totally reasonable
for admins who are setting things up.

5 years agov2writable: implement ->replace call
Eric Wong (Contractor, The Linux Foundation) [Sun, 9 Jun 2019 02:51:40 +0000 (02:51 +0000)]
v2writable: implement ->replace call

Much of the existing purge code is repurposed to a general
"replace" functionality.

->purge is simpler because it can just drop the information.
Unlike ->purge, ->replace needs to edit existing git commits (in
case of From: and Subject: headers) and reindex the modified
message.

We currently disallow editing of References:, In-Reply-To: and
Message-ID headers because it can cause bad side effects with
our threading (and our lack of rethreading support to deal with
excessive matching from incorrect/invalid References).

5 years agoimport: switch to "replace_oids" interface for purge
Eric Wong (Contractor, The Linux Foundation) [Sun, 9 Jun 2019 02:51:39 +0000 (02:51 +0000)]
import: switch to "replace_oids" interface for purge

Continuing the work by Eric Biederman in commit a118d58a402bd31b
("Import.pm: When purging replace a purged file with a zero length file"),
we can use a generic OID replacement mechanism to implement
purge.

5 years agoimport: extract_author_info becomes extract_commit_info
Eric Wong (Contractor, The Linux Foundation) [Sun, 9 Jun 2019 02:51:38 +0000 (02:51 +0000)]
import: extract_author_info becomes extract_commit_info

We will be reusing the same logic for extracting all
the authorship and commit title logic for edits; so
put it all into one sub.

5 years agov2writable: consolidate overview and indexing call
Eric Wong (Contractor, The Linux Foundation) [Sun, 9 Jun 2019 02:51:37 +0000 (02:51 +0000)]
v2writable: consolidate overview and indexing call

It's one ugly sub with lots of parameters, but it's better
than calling a bunch of ugly subs with lots of parameters;
as we'll be needing to call it again when reindexing for
message replacements.

5 years agowww: support $INBOX/git/$EPOCH.git for v2 cloning
Eric Wong (Contractor, The Linux Foundation) [Sun, 9 Jun 2019 04:31:05 +0000 (04:31 +0000)]
www: support $INBOX/git/$EPOCH.git for v2 cloning

And use it in manifest.js.

To ease maintaining mirrors with grokmirror(1), we can accept
a "git/" directory prefix before the epoch, and ".git" suffix
after the epoch number.

We maintain compatibility with "$INBOX/$EPOCH" cloning, of
course, and it's still easier-to-type on the command-line.