X-Git-Url: http://www.git.stargrave.org/?p=public-inbox.git;a=blobdiff_plain;f=TODO;h=7a27fdd2f716e80d1542c94a00662e42c5b90ba3;hp=7bd68c7b816bd5e13a44119fe70d068663be3c74;hb=9b0c238f887475d920a8589b492ec15c63770152;hpb=2c5ef3910834b1a931bc83d294181dc6baddddd3

diff --git a/TODO b/TODO
index 7bd68c7b..7a27fdd2 100644
--- a/TODO
+++ b/TODO
@@ -5,26 +5,34 @@ performance, ease-of-setup, installation, maintainability, etc
 all need to be considered for everything we introduce)
 
 * general performance improvements, but without relying on
-  XS or compiled code any more than we currently do.
+  XS or pre-built modules any more than we currently do.
+  (Optional Inline::C and user-compiled re2c acceptable)
 
 * mailmap support (same as git) for remapping expired email addresses
 
+* support remapping of expired URLs similar to mailmap
+  (coordinate with git.git with this?)
+
 * POP3 server, since some webmail providers support external POP3:
   https://public-inbox.org/meta/20160411034104.GA7817@dcvr.yhbt.net/
   Perhaps make this depend solely the NNTP server and work as a proxy.
   Meaning users can run this without needing a full copy of the
-  archives in a git repository.
+  archives in git repositories.
 
-* HTTP and NNTP proxy support.  Allow us to be a frontend for
+* HTTP, IMAP and NNTP proxy support.  Allow us to be a frontend for
   firewalled off (or Tor-exclusive) instances.  The use case is
-  for offering a publically accessible IP with a cheap VPS,
+  for offering a publicly accessible IP with a cheap VPS,
   yet storing large amounts of data on computers without a
   public IP behind a home Internet connection.
 
-* TLS support for various daemons (including STARTTLS for NNTP and POP3)
+* support HTTP(S) CONNECT proxying to NNTP for users with
+  firewall problems
+
+* DHT (distributed hash table) for mapping Message-IDs to various
+  archive locations to avoid SPOF.
 
 * optional Cache::FastMmap support so production deployments won't
-  need Varnish (Varnish doesn't protect NNTP, either)
+  need Varnish (Varnish doesn't protect NNTP nor IMAP, either)
 
 * dogfood and take advantage of new kernel APIs (while maintaining
   portability to older Linux, free BSDs and maybe Hurd).
@@ -33,15 +41,15 @@ all need to be considered for everything we introduce)
   ensure things continue working as they should (or more better)
   while retaining compatibility with old versions.
 
-* NNTP COMPRESS extension (see innd)
-
 * Support more of RFC 3977 (NNTP)
+  Is there anything left for read-only support?
 
-* Combined "super server" for NNTP/HTTP/POP3 to reduce memory overhead
+* Combined "super server" for NNTP/HTTP/POP3/IMAP to reduce memory,
+  process, and FD overhead
 
 * Configurable linkification for per-inbox shorthands:
   "$gmane/123456" could be configured to expand to the
-  appropriate link pointing to the gmane.org list archives,
+  appropriate link pointing to the gmane.io list archives,
   likewise "[Bug #123456]" could be configured to expand to
   point to some project's bug tracker at http://example.com/bug/123456
 
@@ -69,17 +77,20 @@ all need to be considered for everything we introduce)
 * linkify thread skeletons better
   https://public-inbox.org/git/6E3699DEA672430CAEA6DEFEDE6918F4@PhilipOakley/
 
-* streaming Email::MIME replacement: currently we generate many
-  allocations/strings for headers we never look at and slurp
-  entire message bodies into memory.
-  (this is pie-in-the-sky territory...)
+* Further lower mail parser memory usage.  We still slurp entire
+  message bodies into memory and incur 2-3x overhead on
+  multipart messages.  Inline::C (and maybe gmime) could work.
 
 * use REQUEST_URI properly for CGI / mod_perl2 compatibility
   with Message-IDs which include '%' (done?)
 
-* more and better test cases (use git fast-import to speed up creation)
+* better test cases, make faster by reusing more setup
+  code across tests
 
-* large mbox/Maildir/MH/NNTP spool import (see PublicInbox::Import)
+* large mbox/Maildir/MH/NNTP spool import (in lei, but not
+  for public-facing inboxes)
+
+* MH import support (read-only, at least)
 
 * Read-only WebDAV interface to the git repo so it can be mounted
   via davfs2 or fusedav to avoid full clones.
@@ -98,14 +109,50 @@ all need to be considered for everything we introduce)
   Sometimes an indexing bug only affects a handful of messages,
   so it's not worth the trouble of doing a full reindex.
 
-* code repository integration (with cgit, gitweb, etc...)
+* code repository integration (cgit: done, TODO: gitweb, etc...)
 
-* migration path to v2 without breaking v1 "git fetch" cronjobs
+* migration path to v2 (making it transparent for "git fetch"
+  may not be possible, but "public-inbox-fetch" will handle it)
 
 * imperfect scraper importers for obfuscated list archives
   (e.g. obfuscated Mailman stuff, Google Groups, etc...)
 
-* support hooks, since low-level git-fast-import does not run them
-  https://public-inbox.org/meta/20190405174329.GA21472@chatter.qube.local/
+* improve performance and avoid head-of-line blocking on slow storage
+  (done for most git blob retrievals, Xapian needs work)
+
+* HTTP(S) search API (likely JMAP, but GraphQL could be an option)
+  It should support git-specific prefixes (dfpre:, dfpost:, dfn:, etc)
+  as extensions.  If JMAP, it should have HTTP(S) analogues to
+  various IMAP extensions.
+
+* scalability to tens/hundreds of thousands of inboxes
+
+  - inotify-based manifest.js.gz updates
+
+  ...
+
+* lei - see %CMD in lib/PublicInbox/LEI.pm
+  (there's a truckload here..)
+
+* make "git cat-file --batch" detect unlinked packfiles so we don't
+  have to restart processes (very long-term)
+
+* linter to check validity of config file
+
+* linter option and WWW endpoint to graph relationships and flows
+  between inboxes, addresses, Maildirs, coderepos, newsgroups,
+  IMAP mailboxes, etc...
+
+* pygments support - via Python script similar to `git cat-file --batch'
+  to avoid startup penalty.  pygments.rb (Ruby) can be inspiration, too.
+
+* highlighting + linkification for "git format-patch --interdiff" output
+
+* highlighting for "git format-patch --range-diff" output
+  (linkification is too expensive, as it requires mirroring)
+
+* support UUCP addresses for legacy archives
+
+* support pipelining as an IMAP/NNTP client for -watch + lei
 
-* investigate native grokmirror support/integration
+* auto-detect and reload on TLS cert+key changes in daemons