pop3: reduce memory use while generating the mailbox cache

[public-inbox.git] / Documentation / public-inbox-v2-format.pod

diff --git a/Documentation/public-inbox-v2-format.pod b/Documentation/public-inbox-v2-format.pod
index d87a717d40bcce2ddeade5c5e3bb7f63abba4c38..e93d7fc701d9f3191081629c6ecfab744d5d7c4c 100644 (file)
--- a/Documentation/public-inbox-v2-format.pod
+++ b/Documentation/public-inbox-v2-format.pod
@@ -2,7 +2,7 @@
 
 =head1 NAME
 
-public-inbox v2 format description
+public-inbox-v2-format - structure of public inbox v2 archives
 
 =head1 DESCRIPTION
 
@@ -113,9 +113,14 @@ improved with high-quality and high-quantity solid-state storage.
 Issuing TRIM commands with L<fstrim(8)> was necessary to maintain
 consistent performance while developing this feature.
 
-Rotational storage devices are NOT recommended for indexing of
-large mail archives; but are fine for backup and usable for
-small instances.
+Rotational storage devices perform significantly worse than
+solid state storage for indexing of large mail archives; but are
+fine for backup and usable for small instances.
+
+As of public-inbox 1.6.0, the C<publicInbox.indexSequentialShard>
+option of L<public-inbox-index(1)> may be used with a high shard
+count to ensure individual shards fit into page cache when the entire
+Xapian DB cannot.
 
 Our use of the L</OVERVIEW DB> requires Xapian document IDs to
 remain stable.  Using L<public-inbox-compact(1)> and
@@ -133,7 +138,7 @@ OVER/XOVER commands).
 
 The overview DB maintains all the header information necessary
 to implement the NNTP OVER/XOVER commands and non-search
-endpoints of of the PSGI UI.
+endpoints of the PSGI UI.
 
 Xapian has become completely optional for v2 (as it is for v1), but
 SQLite remains required for v2.  SQLite turns out to be powerful
@@ -159,7 +164,7 @@ top-level of the directory.
 
 =head1 OBJECT IDENTIFIERS
 
-There are three distinct type of identifiers.  content_id is the
+There are three distinct type of identifiers.  content_hash is the
 new one for v2 and should make message removal and deduplication
 easier.  object_id and Message-ID are already known.
 
@@ -179,11 +184,11 @@ The email header; duplicates allowed for archival purposes.
 This remains a searchable field in Xapian.  Note: it's possible
 for emails to have multiple Message-ID headers (and L<git-send-email(1)>
 had that bug for a bit); so we take all of them into account.
-In case of conflicts detected by content_id below, we generate a new
-Message-ID based on content_id; if the generated Message-ID still
+In case of conflicts detected by content_hash below, we generate a new
+Message-ID based on content_hash; if the generated Message-ID still
 conflicts, a random one is generated.
 
-=item content_id
+=item content_hash
 
 A hash of relevant headers and raw body content for
 purging of unwanted content.  This is not stored anywhere,
@@ -193,7 +198,7 @@ For now, the relevant headers are:
 
        Subject, From, Date, References, In-Reply-To, To, Cc
 
-Received, List-Id, and similar headers are NOT part of content_id as
+Received, List-Id, and similar headers are NOT part of content_hash as
 they differ across lists and we will want removal to be able to cross
 lists.
 
@@ -203,7 +208,7 @@ raw body risks being broken by list signatures; but we can use
 filters (e.g. PublicInbox::Filter::Vger) to clean the body for
 imports.
 
-content_id is SHA-256 for now; but can be changed at any time
+content_hash is SHA-256 for now; but can be changed at any time
 without making DB changes.
 
 =back
@@ -230,7 +235,7 @@ and testing of the v2 format.
 
 =head1 COPYRIGHT
 
-Copyright 2018-2020 all contributors L<mailto:meta@public-inbox.org>
+Copyright 2018-2021 all contributors L<mailto:meta@public-inbox.org>
 
 License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt>