3 public-inbox-extindex - create and update external search indices
7 public-inbox-extindex [OPTIONS] EXTINDEX_DIR INBOX_DIR...
9 public-inbox-extindex [OPTIONS] [EXTINDEX_DIR] --all
13 public-inbox-extindex creates and updates an external search and
14 overview database used by the read-only public-inbox PSGI (HTTP),
15 NNTP, and IMAP interfaces. This requires either the
16 L<Search::Xapian> XS bindings OR the L<Xapian> SWIG bindings,
17 along with L<DBD::SQLite> and L<DBI> Perl modules.
27 ... TODO, see L<public-inbox-index(5)>
31 Index all C<publicinbox> entries in C<PI_CONFIG>.
33 C<publicinbox> entries indexed by C<public-inbox-extindex> can
34 have full Xapian searching abilities with the per-C<publicinbox>
35 C<indexlevel> set to C<basic> and their respective Xapian
36 (C<xap15> or C<xapian15>) directories removed. For multiple
37 public-inboxes where cross-posting is common, this allows
38 significant space savings on Xapian indices.
42 Perform garbage collection instead of indexing. Use this if
43 inboxes are removed from the extindex, or if messages are
44 purged or removed from some inboxes.
48 Forces a re-index of all messages in the extindex. This can be
49 used for in-place upgrades and bugfixes while read-only server
50 processes are utilizing the index. Keep in mind this roughly
51 doubles the size of the already-large Xapian database.
53 The extindex locks will be released roughly every 10s to
54 allow L<public-inbox-mda(1)> and L<public-inbox-watch(1)>
55 processes to write to the extindex.
59 Used with C<--reindex>, it will only look for new and stale
60 entries and not touch already-indexed messages.
66 L<public-inbox-extindex-format(5)>
70 public-inbox-extindex does not currently write to the
71 L<public-inbox-config(5)> file, configuration may be entered
72 manually. The extindex name of C<all> is a special case which
73 corresponds to indexing C<--all> inboxes. An example for
74 C<--all> is as follows:
77 topdir = /path/to/extindex_dir
82 See L<public-inbox-config(5)> for more details.
90 Used to override the default "~/.public-inbox/config" value.
92 =item XAPIAN_FLUSH_THRESHOLD
94 The number of documents to update before committing changes to
95 disk. This environment is handled directly by Xapian, refer to
96 Xapian API documentation for more details.
98 Setting C<XAPIAN_FLUSH_THRESHOLD> or
99 C<publicinbox.indexBatchSize> for a large C<--reindex> may cause
100 L<public-inbox-mda(1)>, L<public-inbox-learn(1)> and
101 L<public-inbox-watch(1)> tasks to wait long and unpredictable
102 periods of time during C<--reindex>.
104 Default: none, uses C<publicinbox.indexBatchSize>
110 Occasionally, public-inbox will update it's schema version and
111 require a full index by running this command.
115 Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
117 The mail archives are hosted at L<https://public-inbox.org/meta/> and
118 L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
122 Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
124 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
128 L<Search::Xapian>, L<DBD::SQLite>