3 public-inbox-index - create and update search indices
7 public-inbox-index [OPTIONS] INBOX_DIR...
11 public-inbox-index creates and updates the search, overview and
12 NNTP article number database used by the read-only public-inbox
13 HTTP and NNTP interfaces. Currently, this requires
14 L<DBD::SQLite> and L<DBI> Perl modules. L<Search::Xapian>
15 is optional, only to support the PSGI search interface.
17 Once the initial indices are created by public-inbox-index,
18 L<public-inbox-mda(1)> and L<public-inbox-watch(1)> will
19 automatically maintain them.
21 Running this manually to update indices is only required if
22 relying on L<git-fetch(1)> to mirror an existing public-inbox;
23 or if upgrading to a new version of public-inbox using
24 the C<--reindex> option.
26 Having the overview and article number database is essential to
27 running the NNTP interface, and strongly recommended for the
28 HTTP interface as it provides thread grouping in addition to
29 normal search functionality.
37 Control the number of Xapian indexing jobs in a
38 (L<public-inbox-v2-format(5)>) inbox.
40 C<--jobs=0> is accepted as of public-inbox 1.6.0 (PENDING)
41 to disable parallel indexing.
43 Default: the number of existing Xapian shards
47 Compacts the Xapian DBs after indexing. This is recommended
48 when using C<--reindex> to avoid running out of disk space
49 while indexing multiple inboxes.
51 While option takes a negligible amount of time compared to
52 C<--reindex>, it requires temporarily duplicating the entire
53 contents of the Xapian DB.
55 This switch may be specified twice, in which case compaction
56 happens both before and after indexing to minimize the temporal
57 footprint of the (re)indexing operation.
59 Available since public-inbox 1.4.0.
63 Forces a re-index of all messages in the inbox.
64 This can be used for in-place upgrades and bugfixes while
65 NNTP/HTTP server processes are utilizing the index. Keep in
66 mind this roughly doubles the size of the already-large
67 Xapian database. Using this with C<--compact> or running
68 L<public-inbox-compact(1)> afterwards is recommended to
71 public-inbox protects writes to various indices with
72 L<flock(2)>, so it is safe to reindex (and rethread) while
73 L<public-inbox-watch(1)>, L<public-inbox-mda(1)> or
74 L<public-inbox-learn(1)> run.
76 This does not touch the NNTP article number database.
77 It does not affect threading unless C<--rethread> is
82 Regenerate internal THREADID and message thread associations
85 This fixes some bugs in older versions of public-inbox. While
86 it is possible to use this without C<--reindex>, it makes little
89 Available in public-inbox 1.6.0 (PENDING).
93 Run L<git-gc(1)> to prune and expire reflogs if discontiguous history
94 is detected. This is intended to be used in mirrors after running
95 L<public-inbox-edit(1)> or L<public-inbox-purge(1)> to ensure data
96 is expunged from mirrors.
98 Available since public-inbox 1.2.0.
100 =item --max-size SIZE
102 Sets or overrides L</publicinbox.indexMaxSize> on a
103 per-invocation basis. See L</publicinbox.indexMaxSize>
106 Available since public-inbox 1.5.0.
108 =item --batch-size SIZE
110 Sets or overrides L</publicinbox.indexBatchSize> on a
111 per-invocation basis. See L</publicinbox.indexBatchSize>
114 Available in public-inbox 1.6.0 (PENDING).
118 Disables L<fsync(2)> and L<fdatasync(2)> operations on SQLite
119 and Xapian. This is only effective with Xapian 1.4+.
121 Available in public-inbox 1.6.0 (PENDING).
127 For v1 (ssoma) repositories described in L<public-inbox-v1-format>.
128 All public-inbox-specific files are contained within the
129 C<$GIT_DIR/public-inbox/> directory.
131 v2 inboxes are described in L<public-inbox-v2-format>.
137 =item publicinbox.indexMaxSize
139 Prevents indexing of messages larger than the specified size
140 value. A single suffix modifier of C<k>, C<m> or C<g> is
141 supported, thus the value of C<1m> to prevents indexing of
142 messages larger than one megabyte.
144 This is useful for avoiding memory exhaustion in mirrors.
145 This option is only available in public-inbox 1.5 or later.
149 =item publicinbox.indexBatchSize
151 Flushes changes to the filesystem and releases locks after
152 indexing the given number of bytes. The default value of C<1m>
153 (one megabyte) is low to minimize memory use and reduce
154 contention with parallel invocations of L<public-inbox-mda(1)>,
155 L<public-inbox-learn(1)>, and L<public-inbox-watch(1)>.
157 Increase this value on powerful systems to improve throughput at
158 the expense of memory use. The reduction of lock granularity
159 may not be noticeable on fast systems.
161 This option is available in public-inbox 1.6 or later.
162 public-inbox 1.5 and earlier used the current default, C<1m>.
164 For L<public-inbox-v2-format(5)> inboxes, this value is
165 multiplied by the number of Xapian shards. Thus a typical v2
166 inbox with 3 shards will flush every 3 megabytes by default.
168 Default: 1m (one megabyte)
178 Used to override the default "~/.public-inbox/config" value.
180 =item XAPIAN_FLUSH_THRESHOLD
182 The number of documents to update before committing changes to
183 disk. This environment is handled directly by Xapian, refer to
184 Xapian API documentation for more details.
186 For public-inbox 1.6 and later, use C<publicinbox.indexBatchSize>
187 instead. Setting C<XAPIAN_FLUSH_THRESHOLD> for a large C<--reindex>
188 may cause L<public-inbox-mda(1)>, L<public-inbox-learn(1)> and
189 L<public-inbox-watch(1)> tasks to wait long periods of time
192 Default: none, uses C<publicinbox.indexBatchSize>
198 Occasionally, public-inbox will update it's schema version and
199 require a full index by running this command.
203 Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
205 The mail archives are hosted at L<https://public-inbox.org/meta/>
206 and L<http://hjrcffqmbrq6wope.onion/meta/>
210 Copyright 2016-2020 all contributors L<mailto:meta@public-inbox.org>
212 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
216 L<Search::Xapian>, L<DBD::SQLite>