Documentation/RelNotes/v1.7.0.wip | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++- Documentation/public-inbox-tuning.pod | 9 +++++++++ TODO | 16 +++++----------- diff --git a/Documentation/RelNotes/v1.7.0.wip b/Documentation/RelNotes/v1.7.0.wip index a35ff227daf764edab2e779e21e7f738a7bfe0b9..f71f447feb4d44cd1d4388ac9bc641ac272ca173 100644 --- a/Documentation/RelNotes/v1.7.0.wip +++ b/Documentation/RelNotes/v1.7.0.wip @@ -4,11 +4,68 @@ MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline -TODO: gcf2, detached indices, JMAP, ... +Another big release focused on multi-inbox search and scalability. + +* general changes + + config file parsing is 2x faster with 50K inboxes + +* read-only public-inbox-daemon (-httpd, -nntpd, -imapd): + + libgit2 may be used via Inline::C to avoid hitting system pipe + and process limits. See public-inbox-tuning(7) manpage + for more details. + +* public-inbox-extindex + + A new Xapian + SQLite index able to search across several inboxes. + This may be configured to replace per-inbox Xapian DBs, + (but not per-inbox SQLite indices) and speed up manifest.js.gz + generation. + + See public-inbox-extindex-format(5) and + public-inbox-extindex(1) manpages for more details. + +* public-inbox-nntpd + + - startup is 6x faster with 50K inboxes if using -extindex + +* PublicInbox::WWW + + - mboxrd search results are returned in reverse Xapian docid order, + so more recent results are more likely to show up first + + - d: and dt: search prefixes allow "approxidate" formats supported + by "git log --since=" + + - manifest.js.gz generation is ~25x faster with -extindex + +* lei - local email interface + + An experimental, subject-to-change, likely-to-eat-your-mail tool for + personal mail as well as interacting with public-inboxes on the local + filesystem or over HTTP(S). See lei(1), lei-overview(7), and other + lei-* manpages for details. + +* public-inbox-watch + + - IMAP and NNTP code shared with lei, fixing an off-by-one error + in IMAP synchronization for single-message IMAP folders. + + - \Deleted and \Draft messages ignored for IMAP, as they are for + Maildir. + + - IMAP and NNTP connection establishment (including git-credential + prompts) ordering is now tied to config file order. Compatibility: * Rollbacks all the way to public-inbox 1.2.0 remain supported + +Internal changes + +* public-inbox-index switched to new internal IPC code shared + with lei Please report bugs via plain-text mail to: meta@public-inbox.org diff --git a/Documentation/public-inbox-tuning.pod b/Documentation/public-inbox-tuning.pod index e970241636936ea13eec9d9248912a67d844ab20..b3a2b411e320faef819473a01e5d181c76415de1 100644 --- a/Documentation/public-inbox-tuning.pod +++ b/Documentation/public-inbox-tuning.pod @@ -55,6 +55,15 @@ More (optional) L use will be introduced in the future to lower memory use and improve scalability. +=head2 libgit2 usage via Inline::C + +If libgit2 development files are installed and L +is enabled (described above), per-inbox C +processes are replaced with a single L process running +C in read-only daemons. + +Available as of public-inbox 1.7.0. + =head2 Performance on rotational hard disk drives Random I/O performance is poor on rotational HDDs. Xapian indexing diff --git a/TODO b/TODO index 53907efd4f90b445eb1358e02e18a914460aa624..4993b02c2837a5dc64edeb6d4aff244490c30ab0 100644 --- a/TODO +++ b/TODO @@ -86,7 +86,10 @@ with Message-IDs which include '%' (done?) * more and better test cases (use git fast-import to speed up creation) -* large mbox/Maildir/MH/NNTP spool import (see PublicInbox::Import) +* large mbox/Maildir/MH/NNTP spool import (in lei, but not + for public-facing inboxes) + +* MH import support (read-only, at least) * Read-only WebDAV interface to the git repo so it can be mounted via davfs2 or fusedav to avoid full clones. @@ -133,18 +136,9 @@ - pagination for WwwListing - inotify-based manifest.js.gz updates - - process/FD reduction (needs to be slow-storage friendly) - ... -* command-line tool (similar to mairix/notmuch, but solver+git-aware) - -* consider removing doc_data from Xapian, redundant with over.sqlite3 - It's no longer read as of public-inbox 1.6.0, but still written for - compatibility. - -* share "git cat-file --batch" processes across inboxes to avoid - bumping into /proc/sys/fs/pipe-user-pages-* limits +* lei - see %CMD in lib/PublicInbox/LEI.pm * make "git cat-file --batch" detect unlinked packfiles so we don't have to restart processes (very long-term)