3 public-inbox-extindex - create and update external search indices
7 public-inbox-extindex [OPTIONS] EXTINDEX_DIR INBOX_DIR...
9 public-inbox-extindex [OPTIONS] [EXTINDEX_DIR] --all
13 public-inbox-extindex creates and updates an external search and
14 overview database used by the read-only public-inbox PSGI (HTTP),
15 NNTP, and IMAP interfaces. This requires either the
16 L<Search::Xapian> XS bindings OR the L<Xapian> SWIG bindings,
17 along with L<DBD::SQLite> and L<DBI> Perl modules.
35 =item --batch-size SIZE
37 These switches behave as they do for L<public-inbox-index(1)>
41 Index all C<publicinbox> entries in C<PI_CONFIG>.
43 C<publicinbox> entries indexed by C<public-inbox-extindex> can
44 have full Xapian searching abilities with the per-C<publicinbox>
45 C<indexlevel> set to C<basic> and their respective Xapian
46 (C<xap15> or C<xapian15>) directories removed. For multiple
47 public-inboxes where cross-posting is common, this allows
48 significant space savings on Xapian indices.
52 Perform garbage collection instead of indexing. Use this if
53 inboxes are removed from the extindex, or if messages are
54 purged or removed from some inboxes.
58 Forces a re-index of all messages in the extindex. This can be
59 used for in-place upgrades and bugfixes while read-only server
60 processes are utilizing the index. Keep in mind this roughly
61 doubles the size of the already-large Xapian database.
63 The extindex locks will be released roughly every 10s to
64 allow L<public-inbox-mda(1)> and L<public-inbox-watch(1)>
65 processes to write to the extindex.
69 Used with C<--reindex>, it will only look for new and stale
70 entries and not touch already-indexed messages.
76 L<public-inbox-extindex-format(5)>
80 public-inbox-extindex does not currently write to the
81 L<public-inbox-config(5)> file, configuration may be entered
82 manually. The extindex name of C<all> is a special case which
83 corresponds to indexing C<--all> inboxes. An example for
84 C<--all> is as follows:
87 topdir = /path/to/extindex_dir
92 See L<public-inbox-config(5)> for more details.
100 Used to override the default "~/.public-inbox/config" value.
102 =item XAPIAN_FLUSH_THRESHOLD
104 The number of documents to update before committing changes to
105 disk. This environment is handled directly by Xapian, refer to
106 Xapian API documentation for more details.
108 Setting C<XAPIAN_FLUSH_THRESHOLD> or
109 C<publicinbox.indexBatchSize> for a large C<--reindex> may cause
110 L<public-inbox-mda(1)>, L<public-inbox-learn(1)> and
111 L<public-inbox-watch(1)> tasks to wait long and unpredictable
112 periods of time during C<--reindex>.
114 Default: none, uses C<publicinbox.indexBatchSize>
120 Occasionally, public-inbox will update it's schema version and
121 require a full index by running this command.
125 Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
127 The mail archives are hosted at L<https://public-inbox.org/meta/> and
128 L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
132 Copyright all contributors L<mailto:meta@public-inbox.org>
134 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
138 L<Search::Xapian>, L<DBD::SQLite>