X-Git-Url: http://www.git.stargrave.org/?p=public-inbox.git;a=blobdiff_plain;f=TODO;h=7a27fdd2f716e80d1542c94a00662e42c5b90ba3;hp=94f690efdbd619f7902690163b614bc6a8e31c5a;hb=HEAD;hpb=faa0f744db6047db0594baf00535cc8122211ede diff --git a/TODO b/TODO index 94f690ef..1537179e 100644 --- a/TODO +++ b/TODO @@ -13,26 +13,20 @@ all need to be considered for everything we introduce) * support remapping of expired URLs similar to mailmap (coordinate with git.git with this?) -* POP3 server, since some webmail providers support external POP3: - https://public-inbox.org/meta/20160411034104.GA7817@dcvr.yhbt.net/ - Perhaps make this depend solely the NNTP server and work as a proxy. - Meaning users can run this without needing a full copy of the - archives in a git repository. - -* HTTP and NNTP proxy support. Allow us to be a frontend for +* HTTP, IMAP, NNTP, POP3 proxy support. Allow us to be a frontend for firewalled off (or Tor-exclusive) instances. The use case is for offering a publicly accessible IP with a cheap VPS, yet storing large amounts of data on computers without a public IP behind a home Internet connection. -* support HTTP(S) CONNECT proxying to NNTP for users with +* support HTTP(S) CONNECT proxying to IMAP/NNTP/POP3 for users with firewall problems * DHT (distributed hash table) for mapping Message-IDs to various archive locations to avoid SPOF. * optional Cache::FastMmap support so production deployments won't - need Varnish (Varnish doesn't protect NNTP, either) + need Varnish (Varnish doesn't protect NNTP nor IMAP, either) * dogfood and take advantage of new kernel APIs (while maintaining portability to older Linux, free BSDs and maybe Hurd). @@ -42,8 +36,7 @@ all need to be considered for everything we introduce) while retaining compatibility with old versions. * Support more of RFC 3977 (NNTP) - -* Combined "super server" for NNTP/HTTP/POP3 to reduce memory overhead + Is there anything left for read-only support? * Configurable linkification for per-inbox shorthands: "$gmane/123456" could be configured to expand to the @@ -75,16 +68,20 @@ all need to be considered for everything we introduce) * linkify thread skeletons better https://public-inbox.org/git/6E3699DEA672430CAEA6DEFEDE6918F4@PhilipOakley/ -* low-memory Email::MIME replacement: currently we generate many - allocations/strings for headers we never look at and slurp - entire message bodies into memory. GMime+Inline::C could work. +* Further lower mail parser memory usage. We still slurp entire + message bodies into memory and incur 2-3x overhead on + multipart messages. Inline::C (and maybe gmime) could work. * use REQUEST_URI properly for CGI / mod_perl2 compatibility with Message-IDs which include '%' (done?) -* more and better test cases (use git fast-import to speed up creation) +* better test cases, make faster by reusing more setup + code across tests -* large mbox/Maildir/MH/NNTP spool import (see PublicInbox::Import) +* large mbox/Maildir/MH/NNTP spool import (in lei, but not + for public-facing inboxes) + +* MH import support (read-only, at least) * Read-only WebDAV interface to the git repo so it can be mounted via davfs2 or fusedav to avoid full clones. @@ -105,41 +102,59 @@ all need to be considered for everything we introduce) * code repository integration (cgit: done, TODO: gitweb, etc...) -* migration path to v2 without breaking v1 "git fetch" cronjobs +* migration path to v2 (making it transparent for "git fetch" + may not be possible, but "public-inbox-fetch" will handle it) * imperfect scraper importers for obfuscated list archives (e.g. obfuscated Mailman stuff, Google Groups, etc...) * improve performance and avoid head-of-line blocking on slow storage + (done for most git blob retrievals, Xapian needs work) + +* allow optional use of separate Xapian worker process to implement + timeouts and avoid head-of-line blocking problems. Consider + just-ahead-of-time builds to take advantage of custom date parsers + (approxidate) and other features not available to Perl bindings. + +* integrate git approxidate parsing into Xapian w/o spawning git + +* HTTP(S) search API (likely JMAP, but GraphQL could be an option) + It should support git-specific prefixes (dfpre:, dfpost:, dfn:, etc) + as extensions. If JMAP, it should have HTTP(S) analogues to + various IMAP extensions. + +* scalability to tens/hundreds of thousands of inboxes -* share "git cat-file --batch" processes across inboxes to avoid - bumping into /proc/sys/fs/pipe-user-pages-* limits + - inotify-based manifest.js.gz updates + + ... + +* lei - see %CMD in lib/PublicInbox/LEI.pm + (there's a truckload here..) * make "git cat-file --batch" detect unlinked packfiles so we don't have to restart processes (very long-term) -* support searching based on `git-patch-id --stable` to improve - bidirectional mapping of commits <=> emails - * linter to check validity of config file * linter option and WWW endpoint to graph relationships and flows - between inboxes, addresses maildirs, coderepos, etc... + between inboxes, addresses, Maildirs, coderepos, newsgroups, + IMAP mailboxes, etc... * pygments support - via Python script similar to `git cat-file --batch' to avoid startup penalty. pygments.rb (Ruby) can be inspiration, too. * highlighting + linkification for "git format-patch --interdiff" output -* highlighting + linkification for "git format-patch --range-diff" output - (requires mirroring of git repos) +* highlighting for "git format-patch --range-diff" output + (linkification is too expensive, as it requires mirroring) -* parse and allow (semi)automatic-mirroring of "git request-pull" output - for coderepos +* support UUCP addresses for legacy archives -* configurable diff output for solver-generated blobs +* support pipelining as an IMAP/NNTP client for -watch + lei -* figure out how search for messages with multiple Date: headers - should work (some wacky examples out there...) +* expose lei contents via read/write IMAP/JMAP server for personal use -* support UUCP addresses for legacy archives +* git SHA-256 migration/coexistence path + +* decode RFC 3676 format=flowed + DelSp properly (see mflow (mblaze), mutt, ...)