doc: various updates to reflect current state

-index documentation avoid redundant v1 information and refers
readers to apropriate v1/v2 manpages.  Search::Xapian can also
be optional, now, as only the PSGI search interface uses it.

Favor "INBOX_DIR" where appropriate, since "REPO_DIR" can be
confused for code repos which we also support.

XAPIAN_FLUSH_THRESHOLD is documented for all relevant
bulk commands.

diff --git a/Documentation/public-inbox-compact.pod b/Documentation/public-inbox-compact.pod
index 4a519ce999e4c30d6251e6edd2b7bf4b8ead3dd8..7d37f6fb154506785e5475415cae0c5dd6e17fbf 100644 (file)
--- a/Documentation/public-inbox-compact.pod
+++ b/Documentation/public-inbox-compact.pod
@@ -9,15 +9,12 @@ public-inbox-compact - compact Xapian DBs
 =head1 DESCRIPTION
 
 public-inbox-compact is a wrapper for L<xapian-compact(1)>
 =head1 DESCRIPTION
 
 public-inbox-compact is a wrapper for L<xapian-compact(1)>

-designed for "v2" inboxes.  It combines multiple Xapian
-partitions into one to reduce space overhead after an initial
-mass import (using multiple partitions) is done.
+which locks the inbox and prevents other processes such as
+L<public-inbox-watch(1)> or L<public-inbox-mda(1)> from
+writing while it operates.

 
 

-It locks the inbox and prevents other processes such as
-L<public-inbox-watch(1)> from writing while it operates.
-
-It also supports "v1" (ssoma) inboxes with limited
-usefulness over L<xapian-compact(1)>
+It enforces the use of the C<--no-renumber> option of
+L<xapian-compact(1)>

 
 =head1 ENVIRONMENT
 
 
 =head1 ENVIRONMENT
 


@@ -28,9 +25,15 @@ usefulness over L<xapian-compact(1)>
 The default config file, normally "~/.public-inbox/config".
 See L<public-inbox-config(5)>
 
 The default config file, normally "~/.public-inbox/config".
 See L<public-inbox-config(5)>
 

-=back
+=item XAPIAN_FLUSH_THRESHOLD
+
+The number of documents to update before committing changes to
+disk.  This environment is handled directly by Xapian, refer to
+Xapian API documentation for more details.

 
 

-=head1 UPGRADING
+Default: 10000
+
+=back

 
 =head1 CONTACT
 
 
 =head1 CONTACT
 


@@ -41,7 +44,7 @@ and L<http://hjrcffqmbrq6wope.onion/meta/>
 
 =head1 COPYRIGHT
 
 
 =head1 COPYRIGHT
 

-Copyright 2018 all contributors L<mailto:meta@public-inbox.org>
+Copyright 2018-2019 all contributors L<mailto:meta@public-inbox.org>

 
 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 
 
 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 

diff --git a/Documentation/public-inbox-index.pod b/Documentation/public-inbox-index.pod
index acc903921e94e8a4a641fcc6c34b62a420e06872..2e0ff69322781633621f68378171c7054f189f91 100644 (file)
--- a/Documentation/public-inbox-index.pod
+++ b/Documentation/public-inbox-index.pod
@@ -4,14 +4,15 @@ public-inbox-index - create and update search indices
 
 =head1 SYNOPSIS
 
 
 =head1 SYNOPSIS
 

-public-inbox-index [OPTIONS] REPO_DIR
+public-inbox-index [OPTIONS] INBOX_DIR

 
 =head1 DESCRIPTION
 
 
 =head1 DESCRIPTION
 

-public-inbox-index creates and updates the search and NNTP
-article number database used by the read-only public-inbox HTTP
-and NNTP interfaces.  Currently, this requires L<Search::Xapian>
-and L<DBD::SQlite> and L<DBI> Perl modules.
+public-inbox-index creates and updates the search, overview and
+NNTP article number database used by the read-only public-inbox
+HTTP and NNTP interfaces.  Currently, this requires
+L<DBD::SQlite> and L<DBI> Perl modules.  L<Search::Xapian>
+is optional, only to support the PSGI search interface.

 
 Once the initial indices are created by public-inbox-index,
 L<public-inbox-mda(1)> and L<public-inbox-watch(1)> will
 
 Once the initial indices are created by public-inbox-index,
 L<public-inbox-mda(1)> and L<public-inbox-watch(1)> will


@@ -22,10 +23,10 @@ relying on L<git-fetch(1)> to mirror an existing public-inbox;
 or if upgrading to a new version of public-inbox using
 the C<--reindex> option.
 
 or if upgrading to a new version of public-inbox using
 the C<--reindex> option.
 

-Having a search and article number database is essential to
+Having the overview and article number database is essential to

 running the NNTP interface, and strongly recommended for the
 running the NNTP interface, and strongly recommended for the

-HTTP interface as it provides thread grouping in addition
-to normal search functionality.
+HTTP interface as it provides thread grouping in addition to
+normal search functionality.

 
 =head1 OPTIONS
 
 
 =head1 OPTIONS
 


@@ -45,50 +46,11 @@ This does not touch the NNTP article number database.
 
 =head1 FILES
 
 
 =head1 FILES
 

+For v1 (ssoma) repositories described in L<public-inbox-v1-format>.

 All public-inbox-specific files are contained within the
 All public-inbox-specific files are contained within the

-C<$REPO_DIR/public-inbox/> directory.  All files are expected to
-grow in size as more messages are archived, so using compaction
-commands (e.g. L<xapian-compact(1)>) is not recommended unless
-the list is no longer active.
+C<$GIT_DIR/public-inbox/> directory.

 
 

-=over
-
-=item $REPO_DIR/public-inbox/msgmap.sqlite3
-
-The stable NNTP article number to Message-ID mapping is
-stored in an SQLite3 database.
-
-This is required for users of L<public-inbox-nntpd(1)>, but
-users of the L<PublicInbox::WWW> interface will find it
-useful for attempting recovery from copy-paste truncations of
-URLs containing long Message-IDs.
-
-Avoid removing this file and regenerating it; it may cause
-existing NNTP readers to lose sync and miss (or see duplicate)
-messages.
-
-This file is relatively small, and typically less than 5%
-of the space of the mail stored in a packed git repository.
-
-=item $REPO_DIR/public-inbox/xapian*
-
-The database used by L<Search::Xapian>.  This directory name is
-followed by a number indicating the index schema version this
-installation of public-inbox uses.
-
-These directories may be safely deleted or removed in full
-while the NNTP and HTTP interfaces are no longer accessing
-them.
-
-In addition to providing a search interface for the HTTP
-interface, the Xapian database is used to group and combine
-related messages into threads.  For NNTP servers, it also
-provides a cache of metadata and header information often
-requested by NNTP clients.
-
-This directory is large, often two to three times the size of
-the objects stored in a packed git repository.  Using the
-C<--reindex> option makes it larger, still.
+v2 repositories are described in L<public-inbox-v2-format>.

 
 =back
 
 
 =back
 


@@ -100,8 +62,24 @@ C<--reindex> option makes it larger, still.
 
 Used to override the default "~/.public-inbox/config" value.
 
 
 Used to override the default "~/.public-inbox/config" value.
 

+=item XAPIAN_FLUSH_THRESHOLD
+
+The number of documents to update before committing changes to
+disk.  This environment is handled directly by Xapian, refer to
+Xapian API documentation for more details.
+
+Default: our indexing code flushes every megabyte of mail seen
+to keep memory usage low.  Setting this environment variable to
+any positive value will switch to a document count-based
+threshold in Xapian.
+

 =back
 
 =back
 

+=head1 UPGRADING
+
+Occasionally, public-inbox will update it's schema version and
+require a full index by running this command.
+

 =head1 CONTACT
 
 Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
 =head1 CONTACT
 
 Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>


@@ -111,7 +89,7 @@ and L<http://hjrcffqmbrq6wope.onion/meta/>
 
 =head1 COPYRIGHT
 
 
 =head1 COPYRIGHT
 

-Copyright 2016-2018 all contributors L<mailto:meta@public-inbox.org>
+Copyright 2016-2019 all contributors L<mailto:meta@public-inbox.org>

 
 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 
 
 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 

diff --git a/Documentation/public-inbox-v1-format.pod b/Documentation/public-inbox-v1-format.pod
index 3b0e70e1fc5f68039a114062af4b3b7282c4cc87..c960913d332893ef63c7ca1e56220c9c392abbb2 100644 (file)
--- a/Documentation/public-inbox-v1-format.pod
+++ b/Documentation/public-inbox-v1-format.pod
@@ -104,6 +104,10 @@ SQLite3 database maintaining a stable mapping of Message-IDs to NNTP
 article numbers.  Used by L<public-inbox-nntpd(1)> and created
 and updated by L<public-inbox-index(1)>.
 
 article numbers.  Used by L<public-inbox-nntpd(1)> and created
 and updated by L<public-inbox-index(1)>.
 

+Users of the L<PublicInbox::WWW> interface will find it
+useful for attempting recovery from copy-paste truncations of
+URLs containing long Message-IDs.
+

 Automatically updated by L<public-inbox-mda(1)>,
 L<public-inbox-learn(1)> and L<public-inbox-watch(1)>.
 
 Automatically updated by L<public-inbox-mda(1)>,
 L<public-inbox-learn(1)> and L<public-inbox-watch(1)>.
 


@@ -135,8 +139,12 @@ the "overview" DB also exists in the xapian directory for v1
 repositories.  See L<public-inbox-v2-format(5)/OVERVIEW DB>
 
 Our use of the L</OVERVIEW DB> requires Xapian document IDs to
 repositories.  See L<public-inbox-v2-format(5)/OVERVIEW DB>
 
 Our use of the L</OVERVIEW DB> requires Xapian document IDs to

-remain stable.  Thus, use of L<xapian-compact(1)> and
-L<copydatabase(8)> require the use of C<--no-renumber> switch.
+remain stable.  Using L<public-inbox-compact(1)> and
+L<public-inbox-xcpdb(1)> wrappers are recommended over tools
+provided by Xapian.
+
+This directory is large, often two to three times the size of
+the objects stored in a packed git repository.

 
 =item $GIT_DIR/ssoma.index
 
 
 =item $GIT_DIR/ssoma.index
 

diff --git a/Documentation/public-inbox-v2-format.pod b/Documentation/public-inbox-v2-format.pod
index bc58074e0de56fdfe0c791fea2ed44ad1a5eef2d..65a85c19470835ef6529515421cdefca6575538a 100644 (file)
--- a/Documentation/public-inbox-v2-format.pod
+++ b/Documentation/public-inbox-v2-format.pod
@@ -118,8 +118,9 @@ large mail archives; but are fine for backup and usable for
 small instances.
 
 Our use of the L</OVERVIEW DB> requires Xapian document IDs to
 small instances.
 
 Our use of the L</OVERVIEW DB> requires Xapian document IDs to

-remain stable.  Thus, use of L<xapian-compact(1)> and
-L<copydatabase(8)> require the use of C<--no-renumber> switch.
+remain stable.  Using L<public-inbox-compact(1)> and
+L<public-inbox-xcpdb(1)> wrappers are recommended over tools
+provided by Xapian.

 
 =head2 OVERVIEW DB
 
 
 =head2 OVERVIEW DB
 

diff --git a/Documentation/public-inbox-xcpdb.pod b/Documentation/public-inbox-xcpdb.pod
index c47500b629a3516df5a790a4a677962b5e9fc20d..5697dcddb67d31ef96acc9266e52c8539a51ae0a 100644 (file)
--- a/Documentation/public-inbox-xcpdb.pod
+++ b/Documentation/public-inbox-xcpdb.pod
@@ -1,6 +1,6 @@
 =head1 NAME
 
 =head1 NAME
 

-public-inbox-xcpdb - copy Xapian DBs (for format upgrades)
+public-inbox-xcpdb - upgrade Xapian DB formats

 
 =head1 SYNOPSIS
 
 
 =head1 SYNOPSIS
 


@@ -16,7 +16,8 @@ L<public-inbox-watch(1)> or L<public-inbox-mda(1)>.
 
 This is intended for upgrading the database format used by
 Xapian.  It DOES NOT upgrade the schema used by the
 
 This is intended for upgrading the database format used by
 Xapian.  It DOES NOT upgrade the schema used by the

-public-inbox search interface (see L<public-inbox-index(1)>).
+public-inbox PSGI search interface (see
+L<public-inbox-index(1)>).

 
 =head1 ENVIRONMENT
 
 
 =head1 ENVIRONMENT
 

diff --git a/lib/PublicInbox/Inbox.pm b/lib/PublicInbox/Inbox.pm
index 0d86771fbd33677f08e652bcbbe9cfa87852d144..2771a241aa5361f3067ef15f22a1463c4d6b75dc 100644 (file)
--- a/lib/PublicInbox/Inbox.pm
+++ b/lib/PublicInbox/Inbox.pm
@@ -225,7 +225,7 @@ sub description {
        local $/ = "\n";
        chomp $desc;
        $desc =~ s/\s+/ /smg;
        local $/ = "\n";
        chomp $desc;
        $desc =~ s/\s+/ /smg;

-       $desc = '($REPO_DIR/description missing)' if $desc eq '';
+       $desc = '($INBOX_DIR/description missing)' if $desc eq '';

        $self->{description} = $desc;
 }
 
        $self->{description} = $desc;
 }
 

diff --git a/script/public-inbox-compact b/script/public-inbox-compact
index 4bdadfc38fa1fcbbcfd7567c4813207a3cb4aa46..e8bf31e973781c816ce6756cfbc476f566be45bf 100755 (executable)
--- a/script/public-inbox-compact
+++ b/script/public-inbox-compact
@@ -8,7 +8,7 @@ use PublicInbox::InboxWritable;
 use PublicInbox::Xapcmd;
 use PublicInbox::Admin;
 PublicInbox::Admin::require_or_die('-index');
 use PublicInbox::Xapcmd;
 use PublicInbox::Admin;
 PublicInbox::Admin::require_or_die('-index');

-my $usage = "Usage: public-inbox-compact REPO_DIR\n";
+my $usage = "Usage: public-inbox-compact INBOX_DIR\n";

 my $opt = { compact => 1, -coarse_lock => 1 };
 GetOptions($opt, @PublicInbox::Xapcmd::COMPACT_OPT) or
        die "bad command-line args\n$usage";
 my $opt = { compact => 1, -coarse_lock => 1 };
 GetOptions($opt, @PublicInbox::Xapcmd::COMPACT_OPT) or
        die "bad command-line args\n$usage";

diff --git a/script/public-inbox-index b/script/public-inbox-index
index b6e3052824c320b3b05d00900c18d3250ff4271e..40187b35018f84d6cd86e18959455ce877be365f 100755 (executable)
--- a/script/public-inbox-index
+++ b/script/public-inbox-index
@@ -4,12 +4,12 @@
 # Basic tool to create a Xapian search index for a git repository
 # configured for public-inbox.
 # Usage with libeatmydata <https://www.flamingspork.com/projects/libeatmydata/>
 # Basic tool to create a Xapian search index for a git repository
 # configured for public-inbox.
 # Usage with libeatmydata <https://www.flamingspork.com/projects/libeatmydata/>

-# highly recommended: eatmydata public-inbox-index REPO_DIR
+# highly recommended: eatmydata public-inbox-index INBOX_DIR

 
 use strict;
 use warnings;
 use Getopt::Long qw(:config gnu_getopt no_ignore_case auto_abbrev);
 
 use strict;
 use warnings;
 use Getopt::Long qw(:config gnu_getopt no_ignore_case auto_abbrev);

-my $usage = "public-inbox-index REPO_DIR";
+my $usage = "public-inbox-index INBOX_DIR";

 use PublicInbox::Admin;
 PublicInbox::Admin::require_or_die('-index');
 
 use PublicInbox::Admin;
 PublicInbox::Admin::require_or_die('-index');
 

diff --git a/script/public-inbox-init b/script/public-inbox-init
index 5724c52ac90e1c46474adb227df9a4e9936a1abb..985a09fbcbb4f49d6b24f5237b8b817190c917ee 100755 (executable)
--- a/script/public-inbox-init
+++ b/script/public-inbox-init
@@ -5,7 +5,7 @@
 # Initializes a public-inbox, basically a wrapper for git-init(1)
 use strict;
 use warnings;
 # Initializes a public-inbox, basically a wrapper for git-init(1)
 use strict;
 use warnings;

-my $usage = "public-inbox-init NAME REPO_DIR HTTP_URL ADDRESS [ADDRESS..]";
+my $usage = "public-inbox-init NAME INBOX_DIR HTTP_URL ADDRESS [ADDRESS..]";

 use Getopt::Long qw/:config gnu_getopt no_ignore_case auto_abbrev/;
 use PublicInbox::Admin;
 PublicInbox::Admin::require_or_die('-base');
 use Getopt::Long qw/:config gnu_getopt no_ignore_case auto_abbrev/;
 use PublicInbox::Admin;
 PublicInbox::Admin::require_or_die('-base');