X-Git-Url: http://www.git.stargrave.org/?a=blobdiff_plain;f=Documentation%2Fpublic-inbox-index.pod;h=ff2e5486748889667667de1ef0f86d0d6f390115;hb=2ca7db34a51b858c9d7f6f7366afb9fffee86b6e;hp=554346de1b7259f562368a5258a4bc225449f005;hpb=66c6699142d95cce9e5c9d49e0498161a776cc22;p=public-inbox.git diff --git a/Documentation/public-inbox-index.pod b/Documentation/public-inbox-index.pod index 554346de..ff2e5486 100644 --- a/Documentation/public-inbox-index.pod +++ b/Documentation/public-inbox-index.pod @@ -4,70 +4,150 @@ public-inbox-index - create and update search indices =head1 SYNOPSIS -public-inbox-index GIT_DIR +public-inbox-index [OPTIONS] INBOX_DIR... =head1 DESCRIPTION -public-inbox-index creates and updates the search and NNTP -article number database used by the read-only public-inbox HTTP -and NNTP interfaces. Currently, this requires L -and L and L Perl modules. +public-inbox-index creates and updates the search, overview and +NNTP article number database used by the read-only public-inbox +HTTP and NNTP interfaces. Currently, this requires +L and L Perl modules. L +is optional, only to support the PSGI search interface. Once the initial indices are created by public-inbox-index, L and L will automatically maintain them. Running this manually to update indices is only required if -relying on L to mirror an existing public-inbox. +relying on L to mirror an existing public-inbox; +or if upgrading to a new version of public-inbox using +the C<--reindex> option. -Having a search and article number database is essential to +Having the overview and article number database is essential to running the NNTP interface, and strongly recommended for the -HTTP interface as it provides thread grouping in addition -to normal search functionality. +HTTP interface as it provides thread grouping in addition to +normal search functionality. + +=head1 OPTIONS + +=over + +=item --jobs=JOBS, -j + +Control the number of Xapian indexing jobs in a +(L) inbox. + +C<--jobs=0> is accepted as of public-inbox 1.6.0 (PENDING) +to disable parallel indexing. + +Default: the number of existing Xapian shards + +=item --compact / -c + +Compacts the Xapian DBs after indexing. This is recommended +when using C<--reindex> to avoid running out of disk space +while indexing multiple inboxes. + +While option takes a negligible amount of time compared to +C<--reindex>, it requires temporarily duplicating the entire +contents of the Xapian DB. + +This switch may be specified twice, in which case compaction +happens both before and after indexing to minimize the temporal +footprint of the (re)indexing operation. + +Available since public-inbox 1.4.0. + +=item --reindex + +Forces a re-index of all messages in the inbox. +This can be used for in-place upgrades and bugfixes while +NNTP/HTTP server processes are utilizing the index. Keep in +mind this roughly doubles the size of the already-large +Xapian database. Using this with C<--compact> or running +L afterwards is recommended to +release free space. + +public-inbox protects writes to various indices with L, +so it is safe to reindex while L, +L or L run. + +This does not touch the NNTP article number database or +affect threading. + +=item --prune + +Run L to prune and expire reflogs if discontiguous history +is detected. This is intended to be used in mirrors after running +L or L to ensure data +is expunged from mirrors. + +Available since public-inbox 1.2.0. + +=item --max-size SIZE + +Sets or overrides L on a +per-invocation basis. See L +below. + +Available since public-inbox 1.5.0. + +=item --batch-size SIZE + +Sets or overrides L on a +per-invocation basis. See L +below. + +Available in public-inbox 1.6.0 (PENDING). + +=back =head1 FILES +For v1 (ssoma) repositories described in L. All public-inbox-specific files are contained within the -C<$GIT_DIR/public-inbox/> directory. All files are expected to -grow in size as more messages are archived, so using compaction -commands (e.g. L) is not recommended unless -the list is no longer active. +C<$GIT_DIR/public-inbox/> directory. + +v2 inboxes are described in L. -=item $GIT_DIR/public-inbox/msgmap.sqlite3 +=head1 CONFIGURATION + +=over 8 -The stable NNTP article number to Message-ID mapping is -stored in an SQLite3 database. +=item publicinbox.indexMaxSize -This is required for users of L, but -users of the L interface will find it -useful for attempting recovery from copy-paste truncations of -URLs containing long Message-IDs. +Prevents indexing of messages larger than the specified size +value. A single suffix modifier of C, C or C is +supported, thus the value of C<1m> to prevents indexing of +messages larger than one megabyte. -Avoid removing this file and regenerating it; it may cause -existing NNTP readers to lose sync and miss (or duplicate) -messages. +This is useful for avoiding memory exhaustion in mirrors. +This option is only available in public-inbox 1.5 or later. -This file is relatively small, and typically less than 5% -of the space of the mail stored in a packed git repository. +Default: none -=item $GIT_DIR/public-inbox/xapian* +=item publicinbox.indexBatchSize -The database used by L. This directory name is -followed by a number indicating the index schema version this -installation of public-inbox uses. +Flushes changes to the filesystem and releases locks after +indexing the given number of bytes. The default value of C<1m> +(one megabyte) is low to minimize memory use and reduce +contention with parallel invocations of L, +L, and L. -These directories may be safely deleted or removed in full -while the NNTP and HTTP interfaces are no longer accessing -them. +Increase this value on powerful systems to improve throughput at +the expense of memory use. The reduction of lock granularity +may not be noticeable on fast systems. -In addition to providing a search interface for the HTTP -interface, the Xapian database is used to group and combine -related messages into threads. For NNTP servers, it also -provides a cache of metadata and header information often -requested by NNTP clients. +This option is available in public-inbox 1.6 or later. +public-inbox 1.5 and earlier used the current default, C<1m>. -This directory is large, often two to three times the size of -the objects stored in a packed git repository. +For L inboxes, this value is +multiplied by the number of Xapian shards. Thus a typical v2 +inbox with 3 shards will flush every 3 megabytes by default. + +Default: 1m (one megabyte) + +=back =head1 ENVIRONMENT @@ -77,8 +157,27 @@ the objects stored in a packed git repository. Used to override the default "~/.public-inbox/config" value. +=item XAPIAN_FLUSH_THRESHOLD + +The number of documents to update before committing changes to +disk. This environment is handled directly by Xapian, refer to +Xapian API documentation for more details. + +For public-inbox 1.6 and later, use C +instead. Setting C for a large C<--reindex> +may cause L, L and +L tasks to wait long periods of time +during C<--reindex>. + +Default: none, uses C + =back +=head1 UPGRADING + +Occasionally, public-inbox will update it's schema version and +require a full index by running this command. + =head1 CONTACT Feedback welcome via plain-text mail to L @@ -88,7 +187,7 @@ and L =head1 COPYRIGHT -Copyright 2016 all contributors L +Copyright 2016-2020 all contributors L License: AGPL-3.0+ L