X-Git-Url: http://www.git.stargrave.org/?a=blobdiff_plain;f=Documentation%2Fpublic-inbox-index.pod;h=7c04f753a408f373d99ef73d33b569adf2597bea;hb=95bdac7f09c69036efed537a4d03d5bdd2ae4eb6;hp=554346de1b7259f562368a5258a4bc225449f005;hpb=66c6699142d95cce9e5c9d49e0498161a776cc22;p=public-inbox.git diff --git a/Documentation/public-inbox-index.pod b/Documentation/public-inbox-index.pod index 554346de..7c04f753 100644 --- a/Documentation/public-inbox-index.pod +++ b/Documentation/public-inbox-index.pod @@ -4,70 +4,61 @@ public-inbox-index - create and update search indices =head1 SYNOPSIS -public-inbox-index GIT_DIR +public-inbox-index [OPTIONS] INBOX_DIR =head1 DESCRIPTION -public-inbox-index creates and updates the search and NNTP -article number database used by the read-only public-inbox HTTP -and NNTP interfaces. Currently, this requires L -and L and L Perl modules. +public-inbox-index creates and updates the search, overview and +NNTP article number database used by the read-only public-inbox +HTTP and NNTP interfaces. Currently, this requires +L and L Perl modules. L +is optional, only to support the PSGI search interface. Once the initial indices are created by public-inbox-index, L and L will automatically maintain them. Running this manually to update indices is only required if -relying on L to mirror an existing public-inbox. +relying on L to mirror an existing public-inbox; +or if upgrading to a new version of public-inbox using +the C<--reindex> option. -Having a search and article number database is essential to +Having the overview and article number database is essential to running the NNTP interface, and strongly recommended for the -HTTP interface as it provides thread grouping in addition -to normal search functionality. +HTTP interface as it provides thread grouping in addition to +normal search functionality. -=head1 FILES - -All public-inbox-specific files are contained within the -C<$GIT_DIR/public-inbox/> directory. All files are expected to -grow in size as more messages are archived, so using compaction -commands (e.g. L) is not recommended unless -the list is no longer active. +=head1 OPTIONS -=item $GIT_DIR/public-inbox/msgmap.sqlite3 +=over -The stable NNTP article number to Message-ID mapping is -stored in an SQLite3 database. +=item --reindex -This is required for users of L, but -users of the L interface will find it -useful for attempting recovery from copy-paste truncations of -URLs containing long Message-IDs. +Forces a search engine re-index of all messages in the +repository. This can be used for in-place upgrades while +NNTP/HTTP server processes are utilizing the index. Keep in +mind this roughly doubles the size of the already-large +Xapian database. Running L +afterwards is recommended to release free space. -Avoid removing this file and regenerating it; it may cause -existing NNTP readers to lose sync and miss (or duplicate) -messages. +This does not touch the NNTP article number database. -This file is relatively small, and typically less than 5% -of the space of the mail stored in a packed git repository. +=item --prune -=item $GIT_DIR/public-inbox/xapian* +Run L to prune and expire reflogs if discontiguous history +is detected. This is intended to be used in mirrors after running +L or L to ensure data +is expunged from mirrors. -The database used by L. This directory name is -followed by a number indicating the index schema version this -installation of public-inbox uses. +=back -These directories may be safely deleted or removed in full -while the NNTP and HTTP interfaces are no longer accessing -them. +=head1 FILES -In addition to providing a search interface for the HTTP -interface, the Xapian database is used to group and combine -related messages into threads. For NNTP servers, it also -provides a cache of metadata and header information often -requested by NNTP clients. +For v1 (ssoma) repositories described in L. +All public-inbox-specific files are contained within the +C<$GIT_DIR/public-inbox/> directory. -This directory is large, often two to three times the size of -the objects stored in a packed git repository. +v2 repositories are described in L. =head1 ENVIRONMENT @@ -77,8 +68,24 @@ the objects stored in a packed git repository. Used to override the default "~/.public-inbox/config" value. +=item XAPIAN_FLUSH_THRESHOLD + +The number of documents to update before committing changes to +disk. This environment is handled directly by Xapian, refer to +Xapian API documentation for more details. + +Default: our indexing code flushes every megabyte of mail seen +to keep memory usage low. Setting this environment variable to +any positive value will switch to a document count-based +threshold in Xapian. + =back +=head1 UPGRADING + +Occasionally, public-inbox will update it's schema version and +require a full index by running this command. + =head1 CONTACT Feedback welcome via plain-text mail to L @@ -88,7 +95,7 @@ and L =head1 COPYRIGHT -Copyright 2016 all contributors L +Copyright 2016-2020 all contributors L License: AGPL-3.0+ L