nntp: resolve inboxes immediately on group listings

[public-inbox.git] / Documentation / public-inbox-v2-format.pod

diff --git a/Documentation/public-inbox-v2-format.pod b/Documentation/public-inbox-v2-format.pod
index 65a85c19470835ef6529515421cdefca6575538a..e93d7fc701d9f3191081629c6ecfab744d5d7c4c 100644 (file)
--- a/Documentation/public-inbox-v2-format.pod
+++ b/Documentation/public-inbox-v2-format.pod
@@ -2,7 +2,7 @@
 
 =head1 NAME
 
 
 =head1 NAME
 

-public-inbox v2 repository description
+public-inbox-v2-format - structure of public inbox v2 archives

 
 =head1 DESCRIPTION
 
 
 =head1 DESCRIPTION
 


@@ -16,21 +16,21 @@ Message-IDs.
 The key change in v2 is the inbox is no longer a bare git
 repository, but a directory with two or more git repositories.
 v2 divides git repositories by time "epochs" and Xapian
 The key change in v2 is the inbox is no longer a bare git
 repository, but a directory with two or more git repositories.
 v2 divides git repositories by time "epochs" and Xapian

-databases for parallelism by "partitions".
+databases for parallelism by "shards".

 
 =head2 INBOX OVERVIEW AND DEFINITIONS
 
 
 =head2 INBOX OVERVIEW AND DEFINITIONS
 

-$EPOCH - Integer starting with 0 based on time
-$SCHEMA_VERSION - PublicInbox::Search::SCHEMA_VERSION used by Xapian
-$PART - Integer (0..NPROCESSORS)
+  $EPOCH - Integer starting with 0 based on time
+  $SCHEMA_VERSION - DB schema version (for Xapian)
+  $SHARD - Integer starting with 0 based on parallelism

 
 

-foo/ # assuming "foo" is the name of the list
-- inbox.lock                 # lock file (flock) to protect global state
-- git/$EPOCH.git             # normal git repositories
-- all.git                    # empty git repo, alternates to git/$EPOCH.git
-- xap$SCHEMA_VERSION/$PART   # per-partition Xapian DB
-- xap$SCHEMA_VERSION/over.sqlite3 # OVER-view DB for NNTP and threading
-- msgmap.sqlite3             # same the v1 msgmap
+  foo/                              # "foo" is the name of the inbox
+  - inbox.lock                      # lock file to protect global state
+  - git/$EPOCH.git                  # normal git repositories
+  - all.git                         # empty, alternates to $EPOCH.git
+  - xap$SCHEMA_VERSION/$SHARD       # per-shard Xapian DB
+  - xap$SCHEMA_VERSION/over.sqlite3 # OVER-view DB for NNTP, threading
+  - msgmap.sqlite3                  # same the v1 msgmap

 
 For blob lookups, the reader only needs to open the "all.git"
 repository with $GIT_DIR/objects/info/alternates which references
 
 For blob lookups, the reader only needs to open the "all.git"
 repository with $GIT_DIR/objects/info/alternates which references


@@ -95,16 +95,16 @@ are documented at:
 
 L<https://public-inbox.org/meta/20180209205140.GA11047@dcvr/>
 
 
 L<https://public-inbox.org/meta/20180209205140.GA11047@dcvr/>
 

-=head2 XAPIAN PARTITIONS
+=head2 XAPIAN SHARDS

 
 Another second scalability problem in v1 was the inability to
 utilize multiple CPU cores for Xapian indexing.  This is
 
 Another second scalability problem in v1 was the inability to
 utilize multiple CPU cores for Xapian indexing.  This is

-addressed by using partitions in Xapian to perform import
+addressed by using shards in Xapian to perform import

 indexing in parallel.
 
 As with git alternates, Xapian natively supports a read-only
 interface which transparently abstracts away the knowledge of
 indexing in parallel.
 
 As with git alternates, Xapian natively supports a read-only
 interface which transparently abstracts away the knowledge of

-multiple partitions.  This allows us to simplify our read-only
+multiple shards.  This allows us to simplify our read-only

 code paths.
 
 The performance of the storage device is now the bottleneck on
 code paths.
 
 The performance of the storage device is now the bottleneck on


@@ -113,9 +113,14 @@ improved with high-quality and high-quantity solid-state storage.
 Issuing TRIM commands with L<fstrim(8)> was necessary to maintain
 consistent performance while developing this feature.
 
 Issuing TRIM commands with L<fstrim(8)> was necessary to maintain
 consistent performance while developing this feature.
 

-Rotational storage devices are NOT recommended for indexing of
-large mail archives; but are fine for backup and usable for
-small instances.
+Rotational storage devices perform significantly worse than
+solid state storage for indexing of large mail archives; but are
+fine for backup and usable for small instances.
+
+As of public-inbox 1.6.0, the C<publicInbox.indexSequentialShard>
+option of L<public-inbox-index(1)> may be used with a high shard
+count to ensure individual shards fit into page cache when the entire
+Xapian DB cannot.

 
 Our use of the L</OVERVIEW DB> requires Xapian document IDs to
 remain stable.  Using L<public-inbox-compact(1)> and
 
 Our use of the L</OVERVIEW DB> requires Xapian document IDs to
 remain stable.  Using L<public-inbox-compact(1)> and


@@ -133,12 +138,12 @@ OVER/XOVER commands).
 
 The overview DB maintains all the header information necessary
 to implement the NNTP OVER/XOVER commands and non-search
 
 The overview DB maintains all the header information necessary
 to implement the NNTP OVER/XOVER commands and non-search

-endpoints of of the PSGI UI.
+endpoints of the PSGI UI.

 
 

-In the future, Xapian will become completely optional for v2 (as
-it is for v1) as SQLite turns out to be powerful enough to
-maintain overview information.  Most of the PSGI and all of the
-NNTP functionality will be possible with only SQLite in addition
+Xapian has become completely optional for v2 (as it is for v1), but
+SQLite remains required for v2.  SQLite turns out to be powerful
+enough to maintain overview information.  Most of the PSGI and all
+of the NNTP functionality is possible with only SQLite in addition

 to git.
 
 The overview DB was an instrumental piece in maintaining near
 to git.
 
 The overview DB was an instrumental piece in maintaining near


@@ -159,7 +164,7 @@ top-level of the directory.
 
 =head1 OBJECT IDENTIFIERS
 
 
 =head1 OBJECT IDENTIFIERS
 

-There are three distinct type of identifiers.  content_id is the
+There are three distinct type of identifiers.  content_hash is the

 new one for v2 and should make message removal and deduplication
 easier.  object_id and Message-ID are already known.
 
 new one for v2 and should make message removal and deduplication
 easier.  object_id and Message-ID are already known.
 


@@ -168,7 +173,7 @@ easier.  object_id and Message-ID are already known.
 =item object_id
 
 The blob identifier git uses (currently SHA-1).  No need to
 =item object_id
 
 The blob identifier git uses (currently SHA-1).  No need to

-publically expose this outside of normal git ops (cloning) and
+publicly expose this outside of normal git ops (cloning) and

 there's no need to make this searchable.  As with v1 of
 public-inbox, this is stored as part of the Xapian document so
 expensive name lookups can be avoided for document retrieval.
 there's no need to make this searchable.  As with v1 of
 public-inbox, this is stored as part of the Xapian document so
 expensive name lookups can be avoided for document retrieval.


@@ -179,11 +184,11 @@ The email header; duplicates allowed for archival purposes.
 This remains a searchable field in Xapian.  Note: it's possible
 for emails to have multiple Message-ID headers (and L<git-send-email(1)>
 had that bug for a bit); so we take all of them into account.
 This remains a searchable field in Xapian.  Note: it's possible
 for emails to have multiple Message-ID headers (and L<git-send-email(1)>
 had that bug for a bit); so we take all of them into account.

-In case of conflicts detected by content_id below, we generate a new
-Message-ID based on content_id; if the generated Message-ID still
+In case of conflicts detected by content_hash below, we generate a new
+Message-ID based on content_hash; if the generated Message-ID still

 conflicts, a random one is generated.
 
 conflicts, a random one is generated.
 

-=item content_id
+=item content_hash

 
 A hash of relevant headers and raw body content for
 purging of unwanted content.  This is not stored anywhere,
 
 A hash of relevant headers and raw body content for
 purging of unwanted content.  This is not stored anywhere,


@@ -193,7 +198,7 @@ For now, the relevant headers are:
 
        Subject, From, Date, References, In-Reply-To, To, Cc
 
 
        Subject, From, Date, References, In-Reply-To, To, Cc
 

-Received, List-Id, and similar headers are NOT part of content_id as
+Received, List-Id, and similar headers are NOT part of content_hash as

 they differ across lists and we will want removal to be able to cross
 lists.
 
 they differ across lists and we will want removal to be able to cross
 lists.
 


@@ -203,7 +208,7 @@ raw body risks being broken by list signatures; but we can use
 filters (e.g. PublicInbox::Filter::Vger) to clean the body for
 imports.
 
 filters (e.g. PublicInbox::Filter::Vger) to clean the body for
 imports.
 

-content_id is SHA-256 for now; but can be changed at any time
+content_hash is SHA-256 for now; but can be changed at any time

 without making DB changes.
 
 =back
 without making DB changes.
 
 =back


@@ -226,11 +231,11 @@ no sense in a public archive.
 =head1 THANKS
 
 Thanks to the Linux Foundation for sponsoring the development
 =head1 THANKS
 
 Thanks to the Linux Foundation for sponsoring the development

-and testing of the v2 repository format.
+and testing of the v2 format.

 
 =head1 COPYRIGHT
 
 
 =head1 COPYRIGHT
 

-Copyright 2018-2019 all contributors L<mailto:meta@public-inbox.org>
+Copyright 2018-2021 all contributors L<mailto:meta@public-inbox.org>

 
 License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt>
 
 
 License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt>