-From: Eric Wong <e@yhbt.net>
+From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
-Subject: [WIP] public-inbox 1.6.0
+Subject: [ANNOUNCE] public-inbox 1.6.0
+Date: Wed, 16 Sep 2020 20:03:09 +0000
+Message-ID: <20200916200309.public-inbox-1.6.0-rele@sed>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
+A big release containing several performance optimizations, a
+new anonymous IMAP server, and more. It represents an
+incremental improvement over 1.5 in several areas with more to
+come in 1.7.
+
+The read-only httpd and nntpd daemons no longer block the event
+loop when retrieving blobs from git, making better use of SMP
+systems while accomodating slow storage.
+
+Indexing can be now be tuned to give somewhat usable performance
+on HDD storage, though we can't defy the laws of physics, either.
+
* General changes:
- ~/.cache/public-inbox/inline-c is automatically used for Inline::C
- msgmap.sqlite3 uses journal_mode=TRUNCATE, matching over.sqlite3
behavior for a minor reduction in VFS traffic
+ - public-inbox-tuning(7) - new manpage containing pointers to
+ various tuning options and tips for certain HW and OS setups.
+
+ - Copy-on-write is disabled on BTRFS for new indices to avoid
+ fragmentation. See the new public-inbox-tuning(7) manpage.
+
- message/{rfc822,news,global} attachments are decoded recursively
- and indexed for search. Use `public-inbox-index --reindex' to
- ensure these attachments are indexed in old messages.
+ and indexed for search. Reindexing (see below) is required
+ to ensure these attachments are indexed in old messages.
+
+ - inbox.lock (v2) and ssoma.lock (v1) files are written to
+ on message delivery (or spam removal) to wake up read-only
+ daemons via inotify or kqueue.
+
+ - `--help' switch supported by command-line tools
+
+* Upgrading for new features in 1.6
+
+ The ordering of these steps is only necessary if you intend to
+ use some new features in 1.6. Future releases may have
+ different instructions (or be entirely transparent).
+
+ 0. install (use your OS package manager, or "make install")
+
+ 1. restart public-inbox-watch instances if you have any
+
+ 2. Optional: remove Plack::Middleware::Deflater if you're using
+ a custom .psgi file for PublicInbox::WWW. This only saves
+ some memory and CPU cycles, and you may also skip this step
+ if you expect to roll back to 1.5.0 for any reason.
+
+ Steps 3a and 3b may happen in any order, 3b is optional
+ and is only required to use new WWW and IMAP features.
+
+ 3a. restart existing read-only daemons if you have them
+ (public-inbox-nntpd, public-inbox-httpd)
+
+ 3b. run "public-inbox-index -c --reindex --rethread --all"
+ to reindex all configured inboxes
+
+ 4. configure and start the new public-inbox-imapd. This
+ requires reindexing in 3b, but there's no obligation to
+ run an IMAP server, either.
* public-inbox-index
- - --batch-size=BYTES or publicinbox.indexBatchSize parameter
+ There are several new options to improve usability on slow,
+ rotational storage.
+
+ - `--batch-size=BYTES' or publicinbox.indexBatchSize parameter
+ to reduce frequency of random writes on HDDs
+
+ - `--sequential-shard' or publicInbox.sequentialShard parameter
+ to improve OS page cache utilization on HDDs.
+
+ - `--no-fsync' when combined with Xapian 1.4+ can be used to
+ speed up indexing on SSDs and small (default) `--batch-size'
+
+ - `--rethread' option to go with `--reindex' (use sparringly,
+ see manpage)
+
+ - parallelize v2 updates by default, `--sequential-shard' and
+ `-j0' is (once again) allowed to disable parallelization
- - parallelize updates by default, "-j0" is (once again) allowed
- parallelization
+ - (re-)indexing parallelizes blob reads from git
+
+ - `--all' may be specified to index all configured inboxes
* public-inbox-learn
- `rm' supports `--all' to remove from all configured inboxes
+* public-inbox-imapd
+
+ - new read-only IMAP daemon similar to public-inbox-nntpd
+ `AUTH=ANONYMOUS' is supported, but any username and
+ password for clients without `AUTH=ANONYMOUS' support.
+
+* public-inbox-nntpd
+
+ - blob reads from git are handled asynchronously
+
+* public-inbox-httpd
+
+ - Plack::Middleware::Deflater is no longer loaded by default
+ when no .psgi file is specified; PublicInbox::WWW can rely
+ on gzip for buffering (see below)
+
* PublicInbox::WWW
- use consistent blank line around attachment links
- $INBOX_DIR/description is treated as UTF-8
+ - HTML, Atom, and text/plain responses are gzipped without
+ relying on Plack::Middleware::Deflater
+
+ - Multi-message endpoints (/t.mbox.gz, /T/, /t/, etc) are ~10% faster
+ when running under public-inbox-httpd with asynchronous blob
+ retrieval
+
+ - mbox search results may now include all messages pertaining to that
+ thread. Needs `--reindex' mentioned above in
+ `Upgrading for new features in 1.6'.
+
+ - fix mbox.gz search results downloads for lynx users
+
+ - small navigation tweaks, more prominent mirroring instructions
+
+* public-inbox-watch
+
+ - Linux::Inotify2 or IO::KQueue is used directly,
+ Filesys::Notify::Simple is no longer required
+
+ - NNTP groups and IMAP mailboxes may be watched in addition
+ to Maildirs (lightly tested).
+
+* Ongoing internal changes
+
+ - reduce event loop hogging for many-inbox support
+
+ - use more Perl v5.10-isms, future-proof against Perl 8
+
+ - more consistent variable and field naming, improve internal
+ documentation and comments
+
+ - start supporting >=40 char git identifiers for SHA-256
+
+ - test -httpd-specific code paths via Plack::Test::ExternalServer
+ in addition to generic PSGI paths.
+
Please report bugs via plain-text mail to: meta@public-inbox.org
See archives at https://public-inbox.org/meta/ for all history.