3 public-inbox-index - create and update search indices
7 public-inbox-index [OPTIONS] INBOX_DIR...
11 public-inbox-index creates and updates the search, overview and
12 NNTP article number database used by the read-only public-inbox
13 HTTP and NNTP interfaces. Currently, this requires
14 L<DBD::SQLite> and L<DBI> Perl modules. L<Search::Xapian>
15 is optional, only to support the PSGI search interface.
17 Once the initial indices are created by public-inbox-index,
18 L<public-inbox-mda(1)> and L<public-inbox-watch(1)> will
19 automatically maintain them.
21 Running this manually to update indices is only required if
22 relying on L<git-fetch(1)> to mirror an existing public-inbox;
23 or if upgrading to a new version of public-inbox using
24 the C<--reindex> option.
26 Having the overview and article number database is essential to
27 running the NNTP interface, and strongly recommended for the
28 HTTP interface as it provides thread grouping in addition to
29 normal search functionality.
37 Influences the number of Xapian indexing shards in a
38 (L<public-inbox-v2-format(5)>) inbox.
40 C<--jobs=0> is accepted as of public-inbox 1.6.0 (PENDING)
41 to disable parallel indexing.
43 If the inbox has not been indexed or initialized, C<JOBS - 1>
44 shards will be created (one job is always needed for indexing
45 the overview and article number mapping).
47 Default: the number of existing Xapian shards
51 Compacts the Xapian DBs after indexing. This is recommended
52 when using C<--reindex> to avoid running out of disk space
53 while indexing multiple inboxes.
55 While option takes a negligible amount of time compared to
56 C<--reindex>, it requires temporarily duplicating the entire
57 contents of the Xapian DB.
59 This switch may be specified twice, in which case compaction
60 happens both before and after indexing to minimize the temporal
61 footprint of the (re)indexing operation.
63 Available since public-inbox 1.4.0.
67 Forces a re-index of all messages in the inbox.
68 This can be used for in-place upgrades and bugfixes while
69 NNTP/HTTP server processes are utilizing the index. Keep in
70 mind this roughly doubles the size of the already-large
71 Xapian database. Using this with C<--compact> or running
72 L<public-inbox-compact(1)> afterwards is recommended to
75 public-inbox protects writes to various indices with
76 L<flock(2)>, so it is safe to reindex (and rethread) while
77 L<public-inbox-watch(1)>, L<public-inbox-mda(1)> or
78 L<public-inbox-learn(1)> run.
80 This does not touch the NNTP article number database.
81 It does not affect threading unless C<--rethread> is
86 Regenerate internal THREADID and message thread associations
89 This fixes some bugs in older versions of public-inbox. While
90 it is possible to use this without C<--reindex>, it makes little
93 Available in public-inbox 1.6.0 (PENDING).
97 Run L<git-gc(1)> to prune and expire reflogs if discontiguous history
98 is detected. This is intended to be used in mirrors after running
99 L<public-inbox-edit(1)> or L<public-inbox-purge(1)> to ensure data
100 is expunged from mirrors.
102 Available since public-inbox 1.2.0.
104 =item --max-size SIZE
106 Sets or overrides L</publicinbox.indexMaxSize> on a
107 per-invocation basis. See L</publicinbox.indexMaxSize>
110 Available since public-inbox 1.5.0.
112 =item --batch-size SIZE
114 Sets or overrides L</publicinbox.indexBatchSize> on a
115 per-invocation basis. See L</publicinbox.indexBatchSize>
118 When using rotational storage but abundant RAM, using a large
119 value (e.g. C<500m>) with C<--sequential-shard> can
120 significantly speed up the initial index and full C<--reindex>
121 invocations (but not incremental updates).
123 Available in public-inbox 1.6.0 (PENDING).
127 Disables L<fsync(2)> and L<fdatasync(2)> operations on SQLite
128 and Xapian. This is only effective with Xapian 1.4+.
130 Available in public-inbox 1.6.0 (PENDING).
132 =item --sequential-shard
134 Sets or overrides L</publicinbox.indexSequentialShard> on a
135 per-invocation basis. See L</publicinbox.indexSequentialShard>
138 Available in public-inbox 1.6.0 (PENDING).
144 For v1 (ssoma) repositories described in L<public-inbox-v1-format(5)>.
145 All public-inbox-specific files are contained within the
146 C<$GIT_DIR/public-inbox/> directory.
148 v2 inboxes are described in L<public-inbox-v2-format(5)>.
154 =item publicinbox.indexMaxSize
156 Prevents indexing of messages larger than the specified size
157 value. A single suffix modifier of C<k>, C<m> or C<g> is
158 supported, thus the value of C<1m> to prevents indexing of
159 messages larger than one megabyte.
161 This is useful for avoiding memory exhaustion in mirrors
162 via git. It does not prevent L<public-inbox-mda(1)> or
163 L<public-inbox-watch(1)> from importing (and indexing)
166 This option is only available in public-inbox 1.5 or later.
170 =item publicinbox.indexBatchSize
172 Flushes changes to the filesystem and releases locks after
173 indexing the given number of bytes. The default value of C<1m>
174 (one megabyte) is low to minimize memory use and reduce
175 contention with parallel invocations of L<public-inbox-mda(1)>,
176 L<public-inbox-learn(1)>, and L<public-inbox-watch(1)>.
178 Increase this value on powerful systems to improve throughput at
179 the expense of memory use. The reduction of lock granularity
180 may not be noticeable on fast systems. With SSDs, values above
181 C<4m> have little benefit.
183 For L<public-inbox-v2-format(5)> inboxes, this value is
184 multiplied by the number of Xapian shards. Thus a typical v2
185 inbox with 3 shards will flush every 3 megabytes by default
186 unless parallelism is disabled via C<--sequential-shard>
189 This influences memory usage of Xapian, but it is not exact.
190 The actual memory used by Xapian and Perl has been observed
191 in excess of 10x this value.
193 This option is available in public-inbox 1.6 or later.
194 public-inbox 1.5 and earlier used the current default, C<1m>.
196 Default: 1m (one megabyte)
198 =item publicinbox.indexSequentialShard
200 For L<public-inbox-v2-format(5)> inboxes, setting this to C<true>
201 allows indexing Xapian shards in multiple passes. This speeds up
202 indexing on rotational storage with high seek latency by allowing
203 individual shards to fit into the kernel page cache.
205 Using a higher-than-normal number of C<--jobs> with
206 L<public-inbox-init(1)> may be required to ensure individual
207 shards are small enough to fit into cache.
209 Warning: interrupting C<public-inbox-index(1)> while this option
210 is in use may leave the search indices out-of-date with respect
211 to SQLite databases. WWW and IMAP users may notice incomplete
212 search results, but it is otherwise non-fatal. Using C<--reindex>
213 will bring everything back up-to-date.
215 Available in public-inbox 1.6.0 (PENDING).
217 This is ignored on L<public-inbox-v1-format(5)> inboxes.
219 Default: false, shards are indexed in parallel
221 =item publicinbox.<name>.indexSequentialShard
223 Identical to L</publicinbox.indexSequentialShard>,
224 but only affect the inbox matching E<lt>nameE<gt>.
234 Used to override the default "~/.public-inbox/config" value.
236 =item XAPIAN_FLUSH_THRESHOLD
238 The number of documents to update before committing changes to
239 disk. This environment is handled directly by Xapian, refer to
240 Xapian API documentation for more details.
242 For public-inbox 1.6 and later, use C<publicinbox.indexBatchSize>
245 Setting C<XAPIAN_FLUSH_THRESHOLD> or
246 C<publicinbox.indexBatchSize> for a large C<--reindex> may cause
247 L<public-inbox-mda(1)>, L<public-inbox-learn(1)> and
248 L<public-inbox-watch(1)> tasks to wait long and unpredictable
249 periods of time during C<--reindex>.
251 Default: none, uses C<publicinbox.indexBatchSize>
257 Occasionally, public-inbox will update it's schema version and
258 require a full index by running this command.
262 Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
264 The mail archives are hosted at L<https://public-inbox.org/meta/>
265 and L<http://hjrcffqmbrq6wope.onion/meta/>
269 Copyright 2016-2020 all contributors L<mailto:meta@public-inbox.org>
271 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
275 L<Search::Xapian>, L<DBD::SQLite>