doc: tuning: additional notes for many inboxes

[public-inbox.git] / Documentation / public-inbox-tuning.pod

diff --git a/Documentation/public-inbox-tuning.pod b/Documentation/public-inbox-tuning.pod
index f5a25676b7aab830b91c5679fd13e70041b10b28..53668eccb7cb85a7a7c0f6149c0c8c54189614e2 100644 (file)
--- a/Documentation/public-inbox-tuning.pod
+++ b/Documentation/public-inbox-tuning.pod
@@ -16,7 +16,7 @@ New inboxes: public-inbox-init -V2
 
 =item 2
 
-Process spawning
+Optional Inline::C use
 
 =item 3
 
@@ -34,6 +34,14 @@ Performance on solid state drives
 
 Read-only daemons
 
+=item 7
+
+Other OS tuning knobs
+
+=item 8
+
+Scalability to many inboxes
+
 =back
 
 =head2 New inboxes: public-inbox-init -V2
@@ -42,7 +50,7 @@ If you're starting a new inbox (and not mirroring an existing one),
 the L<-V2|public-inbox-v2-format(5)> requires L<DBD::SQLite>, but is
 orders of magnitude more scalable than the original C<-V1> format.
 
-=head2 Process spawning
+=head2 Optional Inline::C use
 
 Our optional use of L<Inline::C> speeds up subprocess spawning from
 large daemon processes.
@@ -52,9 +60,17 @@ environment variable to point to a writable directory, or create
 C<~/.cache/public-inbox/inline-c> for any user(s) running
 public-inbox processes.
 
+If libgit2 development files are installed and L<Inline::C>
+is enabled (described above), per-inbox C<git cat-file --batch>
+processes are replaced with a single L<perl(1)> process running
+C<PublicInbox::Gcf2::loop> in read-only daemons.  libgit2 use
+will be available in public-inbox 1.7.0+
+
 More (optional) L<Inline::C> use will be introduced in the future
 to lower memory use and improve scalability.
 
+Note: L<Inline::C> is required for L<lei(1)>, but not public-inbox-*
+
 =head2 Performance on rotational hard disk drives
 
 Random I/O performance is poor on rotational HDDs.  Xapian indexing
@@ -139,6 +155,26 @@ Transport Layer Security (IMAPS, NNTPS, or via STARTTLS) significantly
 increases memory use of client sockets, sure to account for that in
 capacity planning.
 
+=head2 Other OS tuning knobs
+
+Linux users: the C<sys.vm.max_map_count> sysctl may need to be increased if
+handling thousands of inboxes (with L<public-inbox-extindex(1)>) to avoid
+out-of-memory errors from git.
+
+Other OSes may have similar tuning knobs (patches appreciated).
+
+=head2 Scalability to many inboxes
+
+L<public-inbox-extindex(1)> allows any number of public-inboxes
+to share the same Xapian indices.
+
+git 2.33+ startup time is orders-of-magnitude faster and uses
+less memory when dealing with thousands of alternates required
+for thousands of inboxes with L<public-inbox-extindex(1)>.
+
+Frequent packing (via L<git-gc(1)>) both improves performance
+and reduces the need to increase C<sys.vm.max_map_count>.
+
 =head1 CONTACT
 
 Feedback encouraged via plain-text mail to L<mailto:meta@public-inbox.org>
@@ -147,10 +183,10 @@ Information for *BSDs and non-traditional filesystems especially
 welcome.
 
 Our archives are hosted at L<https://public-inbox.org/meta/>,
-L<http://hjrcffqmbrq6wope.onion/meta/>, and other places
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>, and other places
 
 =head1 COPYRIGHT
 
-Copyright 2020 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
 
 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>