doc: lei-lcat: document --stdin behavior

[public-inbox.git] / Documentation / public-inbox-tuning.pod

diff --git a/Documentation/public-inbox-tuning.pod b/Documentation/public-inbox-tuning.pod
index abc53d1ee0722c7561190bc805dcf5b4e6b81993..67316fb11109b4071197a2db9788dc254eb9a1d0 100644 (file)
--- a/Documentation/public-inbox-tuning.pod
+++ b/Documentation/public-inbox-tuning.pod
@@ -34,6 +34,10 @@ Performance on solid state drives
 
 Read-only daemons
 
 
 Read-only daemons
 

+=item 7
+
+Other OS tuning knobs
+

 =back
 
 =head2 New inboxes: public-inbox-init -V2
 =back
 
 =head2 New inboxes: public-inbox-init -V2


@@ -55,6 +59,15 @@ public-inbox processes.
 More (optional) L<Inline::C> use will be introduced in the future
 to lower memory use and improve scalability.
 
 More (optional) L<Inline::C> use will be introduced in the future
 to lower memory use and improve scalability.
 

+=head2 libgit2 usage via Inline::C
+
+If libgit2 development files are installed and L<Inline::C>
+is enabled (described above), per-inbox C<git cat-file --batch>
+processes are replaced with a single L<perl(1)> process running
+C<PublicInbox::Gcf2::loop> in read-only daemons.
+
+Available as of public-inbox 1.7.0.
+

 =head2 Performance on rotational hard disk drives
 
 Random I/O performance is poor on rotational HDDs.  Xapian indexing
 =head2 Performance on rotational hard disk drives
 
 Random I/O performance is poor on rotational HDDs.  Xapian indexing


@@ -69,11 +82,12 @@ footprint when indexing on HDDs.
 
 Initializing a mirror with a high C<--jobs> count to create more
 shards (in C<-V2> inboxes) will keep each shard smaller and
 
 Initializing a mirror with a high C<--jobs> count to create more
 shards (in C<-V2> inboxes) will keep each shard smaller and

-reduce its kernel page cache footprint.
+reduce its kernel page cache footprint.  Keep in mind excessive
+sharding imposes a performance penalty for read-only queries.

 
 Users with large amounts of RAM are advised to set a large value
 for C<publicinbox.indexBatchSize> as documented in
 
 Users with large amounts of RAM are advised to set a large value
 for C<publicinbox.indexBatchSize> as documented in

-L<public-inbox-config(5)>.
+L<public-inbox-index(1)>.

 
 C<dm-crypt> users on Linux 4.0+ are advised to try the
 C<--perf-same_cpu_crypt> C<--perf-submit_from_crypt_cpus>
 
 C<dm-crypt> users on Linux 4.0+ are advised to try the
 C<--perf-same_cpu_crypt> C<--perf-submit_from_crypt_cpus>


@@ -88,12 +102,23 @@ used by public-inbox are no exception to that.
 
 public-inbox 1.6.0+ disables copy-on-write (CoW) on Xapian and SQLite
 indices on btrfs to achieve acceptable performance (even on SSD).
 
 public-inbox 1.6.0+ disables copy-on-write (CoW) on Xapian and SQLite
 indices on btrfs to achieve acceptable performance (even on SSD).

-Disabling copy-on-write also disables checksumming, thus raid1
-(or higher) configurations may corrupt on unsafe shutdowns.
+Disabling copy-on-write also disables checksumming, thus C<raid1>
+(or higher) configurations may be corrupt after unsafe shutdowns.

 
 Fortunately, these SQLite and Xapian indices are designed to
 recoverable from git if missing.
 
 
 Fortunately, these SQLite and Xapian indices are designed to
 recoverable from git if missing.
 

+Disabling CoW does not prevent all fragmentation.  Large values
+of C<publicInbox.indexBatchSize> also limit fragmentation during
+the initial index.
+
+Avoid snapshotting subvolumes containing Xapian and/or SQLite indices.
+Snapshots use CoW despite our efforts to disable it, resulting
+in fragmentation.
+
+L<filefrag(8)> can be used to monitor fragmentation, and
+C<btrfs filesystem defragment -fr $INBOX_DIR> may be necessary.
+

 Large filesystems benefit significantly from the C<space_cache=v2>
 mount option documented in L<btrfs(5)>.
 
 Large filesystems benefit significantly from the C<space_cache=v2>
 mount option documented in L<btrfs(5)>.
 


@@ -106,6 +131,11 @@ While SSD read performance is generally good, SSD write performance
 degrades as the drive ages and/or gets full.  Issuing C<TRIM> commands
 via L<fstrim(8)> or similar is required to sustain write performance.
 
 degrades as the drive ages and/or gets full.  Issuing C<TRIM> commands
 via L<fstrim(8)> or similar is required to sustain write performance.
 

+Users of the Flash-Friendly File System
+L<F2FS|https://en.wikipedia.org/wiki/F2FS> may benefit from
+optimizations found in SQLite 3.21.0+.  Benchmarks are greatly
+appreciated.
+

 =head2 Read-only daemons
 
 L<public-inbox-httpd(1)>, L<public-inbox-imapd(1)>, and
 =head2 Read-only daemons
 
 L<public-inbox-httpd(1)>, L<public-inbox-imapd(1)>, and


@@ -116,12 +146,21 @@ for parallelism.
 
 The open file descriptor limit (C<RLIMIT_NOFILE>, C<ulimit -n> in L<sh(1)>,
 C<LimitNOFILE=> in L<systemd.exec(5)>) may need to be raised to
 
 The open file descriptor limit (C<RLIMIT_NOFILE>, C<ulimit -n> in L<sh(1)>,
 C<LimitNOFILE=> in L<systemd.exec(5)>) may need to be raised to

-accomodate many concurrent clients.
+accommodate many concurrent clients.

 
 Transport Layer Security (IMAPS, NNTPS, or via STARTTLS) significantly
 increases memory use of client sockets, sure to account for that in
 capacity planning.
 
 
 Transport Layer Security (IMAPS, NNTPS, or via STARTTLS) significantly
 increases memory use of client sockets, sure to account for that in
 capacity planning.
 

+=head2 Other OS tuning knobs
+
+Linux users: the C<sys.vm.max_map_count> sysctl may need to be increased if
+handling thousands of inboxes (with L<public-inbox-extindex(1)>) to avoid
+out-of-memory errors from git.
+
+Other OSes may have similar tuning knobs (patches appreciated).
+
+

 =head1 CONTACT
 
 Feedback encouraged via plain-text mail to L<mailto:meta@public-inbox.org>
 =head1 CONTACT
 
 Feedback encouraged via plain-text mail to L<mailto:meta@public-inbox.org>


@@ -130,10 +169,10 @@ Information for *BSDs and non-traditional filesystems especially
 welcome.
 
 Our archives are hosted at L<https://public-inbox.org/meta/>,
 welcome.
 
 Our archives are hosted at L<https://public-inbox.org/meta/>,

-L<http://hjrcffqmbrq6wope.onion/meta/>, and other places
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>, and other places

 
 =head1 COPYRIGHT
 
 
 =head1 COPYRIGHT
 

-Copyright 2020 all contributors L<mailto:meta@public-inbox.org>
+Copyright 2020-2021 all contributors L<mailto:meta@public-inbox.org>

 
 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 
 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>