]> Sergey Matveev's repositories - public-inbox.git/log
public-inbox.git
2 years agoxt/solver: improve diagnostics
Eric Wong [Mon, 22 Aug 2022 02:33:41 +0000 (02:33 +0000)]
xt/solver: improve diagnostics

I'm making some tweaks to solver and will probably find extra
output useful, and also update to v5.12 while we're at it.

2 years agoconfig: fix confusing space in ->repo_objs
Eric Wong [Mon, 22 Aug 2022 02:33:40 +0000 (02:33 +0000)]
config: fix confusing space in ->repo_objs

It's actually valid Perl to have "$foo ->{field} = ..."
but it's confusing and I noticed it while tracking down
a configuration error.

2 years agowww: support `+' in inbox names
Eric Wong [Sun, 21 Aug 2022 22:21:00 +0000 (22:21 +0000)]
www: support `+' in inbox names

`+' already seemed to works for IMAP mailboxes and NNTP newsgroup
names and git-config doesn't complain, either.  So allow it as
the path components of WWW URLs so projects like `libstdc++' can
use it.

Reported-by: Mark Wielaard <mark@klomp.org>
Tested-by: Mark Wielaard <mark@klomp.org>
Link: https://public-inbox.org/meta/YwKnFCvganW7ErXU@wildebeest.org/
2 years agopublic-inbox 1.9.0 v1.9.0
Eric Wong [Sun, 21 Aug 2022 02:35:27 +0000 (02:35 +0000)]
public-inbox 1.9.0

2 years agodoc: 1.9 release notes update
Eric Wong [Sat, 20 Aug 2022 08:25:32 +0000 (08:25 +0000)]
doc: 1.9 release notes update

Getting close to another release, I think...

2 years agowww: mbox* drop unneeded {base_url} memoizations
Eric Wong [Sat, 20 Aug 2022 08:01:35 +0000 (08:01 +0000)]
www: mbox* drop unneeded {base_url} memoizations

That field is not needed since List-* and Archived-At headers
are no longer appended as of commit:
1bf653ad139bf7bb (nntp+www: drop List-* and Archived-At headers, 2020-12-10)

2 years agowww: use absolute URLs for coderepo URLs
Eric Wong [Sat, 20 Aug 2022 08:01:34 +0000 (08:01 +0000)]
www: use absolute URLs for coderepo URLs

Showing "../../foo.git" looks awkward and isn't conducive to
users who want to "git clone" a URL.

2 years agoview: do not show pagination footer for small inboxes
Eric Wong [Sat, 20 Aug 2022 08:01:33 +0000 (08:01 +0000)]
view: do not show pagination footer for small inboxes

For new public inboxes with few messages, the dead pagination
footer is a worthless and confusing waste of space: "page: \n";
without `next' or `prev' links for users to follow.

2 years agoimap: remove some intermediate arrays
Eric Wong [Sat, 20 Aug 2022 08:01:32 +0000 (08:01 +0000)]
imap: remove some intermediate arrays

We can assign arrays directly without `[]' creating an extra
immortal pad allocation.

2 years agoimport: take advantage of some Perl 5.10.x features
Eric Wong [Sat, 20 Aug 2022 08:01:31 +0000 (08:01 +0000)]
import: take advantage of some Perl 5.10.x features

We can save a few lines of code this way and reduce the
verbosity of some lines we keep.

2 years agolei/store: reindex culls over-indexed messages
Eric Wong [Fri, 19 Aug 2022 09:07:45 +0000 (09:07 +0000)]
lei/store: reindex culls over-indexed messages

I may be the only lei user who has redundantly-indexed messages
needing this, though...

2 years agosmsg: ->populate falls back to old {ds}/{ts} values
Eric Wong [Fri, 19 Aug 2022 09:07:44 +0000 (09:07 +0000)]
smsg: ->populate falls back to old {ds}/{ts} values

This will be useful for re-indexing external messages which
were over-indexed in lei/store.

2 years agotests: add some basic "lei reindex" tests
Eric Wong [Fri, 19 Aug 2022 09:07:43 +0000 (09:07 +0000)]
tests: add some basic "lei reindex" tests

This is a bit hard-to-test, but at least we must ensure
volatile-metadata is preserved.

2 years agolei reindex: account for parallel lei/store users
Eric Wong [Fri, 19 Aug 2022 09:07:42 +0000 (09:07 +0000)]
lei reindex: account for parallel lei/store users

We need to call eidx_init in each git->cat_async callback
since another requestor may've stopped the shard processes.

2 years agolei reindex: new command to reindex lei/store
Eric Wong [Wed, 17 Aug 2022 09:33:17 +0000 (09:33 +0000)]
lei reindex: new command to reindex lei/store

2 years agolei/store: reduce work when accessing mail_sync.sqlite3
Eric Wong [Wed, 17 Aug 2022 09:33:16 +0000 (09:33 +0000)]
lei/store: reduce work when accessing mail_sync.sqlite3

There's no need to initialize eidx if we already have an open
handle for mail_sync.sqlite3

2 years agolei inspect: less scary exception for invalid "docid:" inspect
Eric Wong [Wed, 17 Aug 2022 09:33:15 +0000 (09:33 +0000)]
lei inspect: less scary exception for invalid "docid:" inspect

It still says "Exception:", but doesn't pointlessly print out
the line number and file of the exception when it's a data/input
problem, and not a code problem on our end.

2 years agosearchidx: fix spelling error in comment
Eric Wong [Wed, 17 Aug 2022 09:33:14 +0000 (09:33 +0000)]
searchidx: fix spelling error in comment

2 years agolei: do not wait for sto->done on disconnected EOF
Eric Wong [Tue, 16 Aug 2022 03:44:03 +0000 (03:44 +0000)]
lei: do not wait for sto->done on disconnected EOF

lei-daemon (the top-level daemon process) should not have
synchronous waits, and this was causing a deadlock with
interrupted commands.  There may still be a bug lurking in
lei/store despite this fix, though.  I originally thought commit
fd261b9e65674505 (lei_store_err: use level-trigger for error pipe, 2022-08-15)
was sufficient, but at least this change is needed, as well.

2 years agopop3: speed up STAT slightly (~1%)
Eric Wong [Fri, 12 Aug 2022 22:09:19 +0000 (22:09 +0000)]
pop3: speed up STAT slightly (~1%)

We can calculate the total size of the mailbox while generating
the cache, which allows us to iterate the cache again to
calculate the size of the mailbox slice.  While we're in the
area, simplify the loop and avoid needlessly updating the `$beg'
variable.

This adds a small amount of constant time overhead to DELE,
however that is amortized across multiple requests for fairness.

2 years agolei_store_err: use level-trigger for error pipe
Eric Wong [Mon, 15 Aug 2022 19:28:33 +0000 (19:28 +0000)]
lei_store_err: use level-trigger for error pipe

This fixes deadlocks from errors inside lei/store when multiple
errors are spewed.

2 years agopop3: fix off-by-one error when handling `EXPIRE 0'
Eric Wong [Fri, 12 Aug 2022 21:30:37 +0000 (21:30 +0000)]
pop3: fix off-by-one error when handling `EXPIRE 0'

mark_dele already works on the cache offset, so there's no need
to make further adjustments to the offset (as we do with POP3
sequence numbers).

2 years agopop3: quiet warning for cached active statements
Eric Wong [Fri, 12 Aug 2022 09:14:48 +0000 (09:14 +0000)]
pop3: quiet warning for cached active statements

Setting the $if_active parameter of ->prepare_cached to `1'
seemed to be the best option many years ago, so it's probably
the best option going forward when caching prepared statements.

Fixes: cab36ebd00ca72f8 ("pop3: remove untouched rows on QUIT/disconnect")
2 years agosyscall: add support for riscv64
Eric Wong [Thu, 11 Aug 2022 22:33:12 +0000 (22:33 +0000)]
syscall: add support for riscv64

Tested on gcc92.fsffrance.org from cfarm.

2 years agotests: support require_mods 'v2'
Eric Wong [Thu, 11 Aug 2022 22:33:05 +0000 (22:33 +0000)]
tests: support require_mods 'v2'

This fixes t/v2index-late-dupe.t to be skipped properly on
systems without DBD::SQLite

2 years agodoc: spell out "PROTOCOL" in examples
Eric Wong [Thu, 11 Aug 2022 20:13:11 +0000 (20:13 +0000)]
doc: spell out "PROTOCOL" in examples

"PROTO" is probably ambiguous, and we'll use the same
terminology as used in curl(1) documentation.

2 years agoexamples: add systemd files for -netd
Eric Wong [Thu, 11 Aug 2022 20:13:10 +0000 (20:13 +0000)]
examples: add systemd files for -netd

It's important show that a single systemd service and socket file
can replace all other read-only daemons for ease-of-management.

2 years agoexamples: consolidate systemd socket examples
Eric Wong [Thu, 11 Aug 2022 20:13:09 +0000 (20:13 +0000)]
examples: consolidate systemd socket examples

systemd.socket(5) files can actually contain multiple listen
sockets, so shave down inode overhead and simplify config
file management by consolidating all applicable ports into
a single file for each daemon.

2 years agodoc: drop ancient Apache and WEBrick examples
Eric Wong [Thu, 11 Aug 2022 20:13:08 +0000 (20:13 +0000)]
doc: drop ancient Apache and WEBrick examples

Having old, unmaintained docs for other HTTP servers is likely
harmful at this point.  public-inbox-httpd is specifically
designed to handle git repos on slow storage and stream giant
mbox.gz files fairly to slow clients.

2 years agodevel/syscall-list: support non-Linux, show sizeof(pid_t)
Eric Wong [Thu, 11 Aug 2022 20:33:39 +0000 (20:33 +0000)]
devel/syscall-list: support non-Linux, show sizeof(pid_t)

While I have no intention of using syscall numbers for
non-Linux, sizeof(pid_t) was useful for OpenBSD.  And maybe
Linux can have real competition from other OSes with stable
syscall numbers someday.

2 years agopop3d: enable native fcntl locks on all *BSDs
Eric Wong [Thu, 11 Aug 2022 20:00:21 +0000 (20:00 +0000)]
pop3d: enable native fcntl locks on all *BSDs

...as we've already done for the simpler case of mbox locking in lei.
I've just confirmed NetBSD and OpenBSD share the same "struct flock"
with FreeBSD, and assume DragonflyBSD is the same.  sizeof(pid_t) == 4
in all places I've checked, and it's unlikely we'll need 64-bit
pid_t any time soon...

2 years agowww: inbox: favor "pop3://" over "pop://"
Eric Wong [Thu, 11 Aug 2022 20:00:20 +0000 (20:00 +0000)]
www: inbox: favor "pop3://" over "pop://"

curl only supports "pop3://" and "pop3s://", despite RFC 2384
existing for "pop://".  AFAIK, there's no RFCs for "pop3://"
and "pop3s://", but please let us know if there are.

In any case, real-world cases like curl are more relevant.

2 years agodoc: add WIP 1.9 release notes
Eric Wong [Thu, 11 Aug 2022 17:36:19 +0000 (17:36 +0000)]
doc: add WIP 1.9 release notes

I think "lei reindex" and some doc cleanups will be the last
things before 1.9...

2 years agowww_text: fix #nntp anchor for there's a single NNTP server
Eric Wong [Thu, 11 Aug 2022 15:47:30 +0000 (15:47 +0000)]
www_text: fix #nntp anchor for there's a single NNTP server

We use "Newsgroup" (singular) when there's only one NNTP
server address configured.

2 years agodaemon: rely on $SIG{__WARN__} for error output
Eric Wong [Wed, 10 Aug 2022 15:58:01 +0000 (15:58 +0000)]
daemon: rely on $SIG{__WARN__} for error output

warn/carp usage is unavoidable given Perl itself and standard
libraries, so just rely on localized $SIG{__WARN__} from
60d262483a4d6ddf (daemon: use per-listener SIG{__WARN__} callbacks, 2022-08-08)
for all error reporting.

While we're in the area, make some of the error handling more
consistent between IMAP/NNTP/POP3.

2 years agowww_text: add AUTH=ANONYMOUS to IMAP URLs
Eric Wong [Wed, 10 Aug 2022 07:40:31 +0000 (07:40 +0000)]
www_text: add AUTH=ANONYMOUS to IMAP URLs

While the ';' requires escaping on the command-line, the
presence of ";AUTH=ANONYMOUS" communicates clearly that
anonymous access is supported in accordance to RFC 4505.

2 years agowww_text: clarify the password+username is for POP3
Eric Wong [Wed, 10 Aug 2022 07:40:30 +0000 (07:40 +0000)]
www_text: clarify the password+username is for POP3

NNTP and IMAP can also exist in the same area, so clarify
that the username + password is only for POP3

2 years agowww_text: add #nntp, #pop3, and #imap anchors to help HTML
Eric Wong [Wed, 10 Aug 2022 07:40:29 +0000 (07:40 +0000)]
www_text: add #nntp, #pop3, and #imap anchors to help HTML

This will make it easier to link to these sections in
3rd-party documentation.

2 years agopop3: remove untouched rows on QUIT/disconnect
Eric Wong [Wed, 10 Aug 2022 06:00:53 +0000 (06:00 +0000)]
pop3: remove untouched rows on QUIT/disconnect

Some POP3 clients may connect and never retrieve messages nor
trigger deletes.  In that case, save some storage by removing
unused rows from the `deletes' and `users' tables.

2 years agoimapd|nntpd: drop ->base_url preload
Eric Wong [Mon, 8 Aug 2022 23:53:11 +0000 (23:53 +0000)]
imapd|nntpd: drop ->base_url preload

It was never useful for IMAP, and NNTP hasn't needed it since:
1bf653ad139bf7bb (nntp+www: drop List-* and Archived-At headers, 2020-12-10)

2 years agoimap: mailboxes list across listeners
Eric Wong [Mon, 8 Aug 2022 23:53:10 +0000 (23:53 +0000)]
imap: mailboxes list across listeners

Since IMAP mailbox lists are tied to the PublicInbox::Config
object, we can share them the same way the config object is
shared when an -imapd or -netd instance has multiple listeners.

This ought to reduce memory use and startup time when binding
multiple sockets which share a common config file.

2 years agodaemon: cleanup internal data structures
Eric Wong [Mon, 8 Aug 2022 23:53:09 +0000 (23:53 +0000)]
daemon: cleanup internal data structures

This avoids dangling {''} entries in $xnetd and
%tls_opt hashes.  Furthermore, we can safely undef
%tls_opt once it's associated with each $xnetd object.

2 years agodaemon: use per-listener SIG{__WARN__} callbacks
Eric Wong [Mon, 8 Aug 2022 23:53:08 +0000 (23:53 +0000)]
daemon: use per-listener SIG{__WARN__} callbacks

This allows "-l $ADDRESS?err=/path/to/err.log to isolate normal
warn() (and carp()) messages for a particular listen address to
track down errors more easily.

2 years agodaemon: use default address + well-known ports for scheme
Eric Wong [Mon, 8 Aug 2022 23:53:07 +0000 (23:53 +0000)]
daemon: use default address + well-known ports for scheme

This ensures the "bound $URL" diagnostic message at startup
always shows the URL scheme handled if not relying on socket
inheritance.

This also avoids duplicate/unused data structures when binding
sockets ourselves, as bound socket names can expand from short
names to longer names (e.g. "0:119" => "0.0.0.0:119").

2 years agoREADME: recommend AUTH=ANONYMOUS on IMAP URLs
Eric Wong [Mon, 8 Aug 2022 23:16:48 +0000 (23:16 +0000)]
README: recommend AUTH=ANONYMOUS on IMAP URLs

public-inbox-imapd prioritizes AUTH=ANONYMOUS clients, nowadays,
since it's a good heuristic for legitimate client traffic.

2 years agoimap: prioritize AUTH=ANONYMOUS clients
Eric Wong [Mon, 8 Aug 2022 23:16:47 +0000 (23:16 +0000)]
imap: prioritize AUTH=ANONYMOUS clients

...by deprioritizing clients using a username + password.

As IMAP provides AUTH=ANONYMOUS for designating anonymous
access, we'll rely on it as a heuristic for favoring "good"
clients.  Clients using a username + password seem to (more
often than not) be malicious and looking for info which doesn't
belong in public inboxes.

This copies the technique used by WWW + -httpd to deprioritize
expensive mbox.gz downloads.

2 years agoimap: only give AUTH=ANONYMOUS clients prefetch
Eric Wong [Mon, 8 Aug 2022 23:16:46 +0000 (23:16 +0000)]
imap: only give AUTH=ANONYMOUS clients prefetch

Looking at IMAP traffic on public-inbox.org, it seems there is a
fair amount of traffic coming from malicious clients assuming
the IMAP server is compromised and searching for private
information.  Since AUTH=ANONYMOUS clients are more likely to
be legitimate clients looking for publicly-archived mail,
give them priority.

2 years agoimap: limit ibx_async_prefetch to idle git processes
Eric Wong [Mon, 8 Aug 2022 23:16:45 +0000 (23:16 +0000)]
imap: limit ibx_async_prefetch to idle git processes

This improves fairness while having no measurable performance
impact for a single uncached IMAP client (mutt) opening a folder
for the first time.

I noticed this problem with the public-inbox.org IMAP server where
a few IMAP clients were unfairly monopolizing the -netd process.

2 years agodaemon: dedupe PublicInbox::Config objects by pathname
Eric Wong [Fri, 5 Aug 2022 08:29:54 +0000 (08:29 +0000)]
daemon: dedupe PublicInbox::Config objects by pathname

This means all Inbox, Git, Over, Msgmap, Search objects also get
deduplicated if they belong to the same config file, reducing
memory and FD usage.  This helps save memory and improve cache
hit rates in -netd setups where NNTP, IMAP, HTTP, and POP3
servers run in the same process.

InboxIdle was the only bit which needed adjustment, but there
may be other bugs lurking despite all tests passing.

2 years agowww: gzip_filter: avoid errors after ->write failure
Eric Wong [Thu, 4 Aug 2022 20:08:21 +0000 (20:08 +0000)]
www: gzip_filter: avoid errors after ->write failure

->zflush must return a string to its caller, not undef.
Additionally, {http_out} may be deleted on ->write if ->close
recurses.

This should fix the following errors:
  Use of uninitialized value $_[1] in string eq at PublicInbox/HTTP.pm line 211.
  E: Can't call method "close" on an undefined value at GzipFilter.pm line 167.

Fixes: a6d50dc1098c01a1 (www: gzip_filter: gracefully handle socket ->write failures, 2022-08-03)
2 years agoview: avoid intermediate array when streaming thread
Eric Wong [Thu, 4 Aug 2022 08:17:03 +0000 (08:17 +0000)]
view: avoid intermediate array when streaming thread

We can rely on auto-vivification to avoid an intermediate
array for the map result.

2 years agofeed: avoid unnecessary map loop in non-over path
Eric Wong [Thu, 4 Aug 2022 08:17:02 +0000 (08:17 +0000)]
feed: avoid unnecessary map loop in non-over path

We can bless objects while doing the initial insertion to avoid
extra the extra map iteration and temporary array(s).  Fewer ops
means memory savings for the likely case of ->over users, too.

2 years agoimap: ensure_slices_exist: drop needless map and array
Eric Wong [Thu, 4 Aug 2022 08:17:01 +0000 (08:17 +0000)]
imap: ensure_slices_exist: drop needless map and array

We can reduce ops and temporary objects here by folding the
stringification into the `for' loop and push directly into the
{mailboxlist} array; relying on autovivification to turn it into
a noop for the initial population.

2 years agolei_overview: remove pointless map {} op
Eric Wong [Thu, 4 Aug 2022 08:17:00 +0000 (08:17 +0000)]
lei_overview: remove pointless map {} op

We can rely on //g and autovivification, here.

2 years agoisearch: mset_to_artnums: avoid unnecessary ops
Eric Wong [Thu, 4 Aug 2022 08:16:59 +0000 (08:16 +0000)]
isearch: mset_to_artnums: avoid unnecessary ops

We can use DBI's selectcol_arrayref directly (as we do in other
places) to avoid unnecessary arrays and ops on our end.

2 years agoover: get_xref3: modify rows in-place
Eric Wong [Thu, 4 Aug 2022 08:16:58 +0000 (08:16 +0000)]
over: get_xref3: modify rows in-place

There's no need to create two intermediate arrays when we can
modify the existing arrayref.

2 years agohttp: coerce SERVER_PORT to integer
Eric Wong [Thu, 4 Aug 2022 08:16:57 +0000 (08:16 +0000)]
http: coerce SERVER_PORT to integer

This may save a few bytes with many connected clients.
Noticed while working on the JMAP endpoint.

2 years agoTODO: remove done items, adjust/add/abandon some
Eric Wong [Thu, 4 Aug 2022 07:23:49 +0000 (07:23 +0000)]
TODO: remove done items, adjust/add/abandon some

public-inbox-pop3d (and -netd) gives us POP3 support, and
it seems to work.  Proxy support can come independently,
probably after JMAP.

public-inbox-netd provides the multi-protocol "super server"
which allows code memory savings.  Work is ongoing to further
reduce memory use...

Automatically updating on TLS cert and key changes on
inotify/EVFILT_VNODE won't be done, since (IMHO) there's too
much risk of inadvertent updates on incomplete changes.
My same train-of-thought applies to auto-reloading on config
file changes: an admin may save a file halfway through a
multi-step change and auto-reloading can be too surprising and
break things.

I don't think lei+FUSE will be as portable or useful as a
local IMAP server (and maybe JMAP, eventually); but r/w IMAP
support would be nice..

Finally, git SHA-256 repo support will need to be taken into
account.

2 years agodaemon: handle per-listener options on inherited, well-known ports
Eric Wong [Thu, 4 Aug 2022 06:27:39 +0000 (06:27 +0000)]
daemon: handle per-listener options on inherited, well-known ports

We must not clobber already-parsed per-listener options when
handling inherited sockets which are well-known.  Unfortunately,
this isn't easy to test in a non-intrusive way for regular
users.

2 years agoimapd: use nntpd_cache to speed up startup/reload time
Eric Wong [Wed, 3 Aug 2022 20:03:57 +0000 (20:03 +0000)]
imapd: use nntpd_cache to speed up startup/reload time

ConfigIter was still too slow despite being fair.  The addition of
ART_MIN in ALL->misc means it can be used as a startup/reload cache
for -imapd, too.

This results in a ~3x faster startup for -imapd with 50K inboxes.

2 years agonntp: speed up group listings via ->ALL->misc
Eric Wong [Wed, 3 Aug 2022 20:03:56 +0000 (20:03 +0000)]
nntp: speed up group listings via ->ALL->misc

By taking advantage of the new ART_MIN/ART_MAX value in MiscIdx,
we can avoid the overhead of opening per-inbox msgmap DB files.
The result gives us a ~40 speedup with 50K newgroups.

2 years agomiscidx: index inbox min/max article numbers
Eric Wong [Wed, 3 Aug 2022 20:03:55 +0000 (20:03 +0000)]
miscidx: index inbox min/max article numbers

This will be used to speed up NNTP group listings and IMAP startup
with thousands of inboxes.

2 years agonntpd: do not delete newsgroup name from inbox object
Eric Wong [Wed, 3 Aug 2022 20:03:54 +0000 (20:03 +0000)]
nntpd: do not delete newsgroup name from inbox object

While PublicInbox::NNTP doesn't use it, config sharing inside
public-inbox-netd will mean inbox objects also get shared.

2 years agodaemon: reload TLS certs and keys on SIGHUP
Eric Wong [Wed, 3 Aug 2022 08:06:03 +0000 (08:06 +0000)]
daemon: reload TLS certs and keys on SIGHUP

This allows new TLS certificates to be loaded for new clients
without having to timeout nor drop existing clients with
established connections made with the old certs.  This should
benefit users with admins who expire certificates frequently (as
encouraged by Let's Encrypt).

2 years agowww: simplify GzipFilter->zflush callers
Eric Wong [Wed, 3 Aug 2022 07:59:12 +0000 (07:59 +0000)]
www: simplify GzipFilter->zflush callers

->zflush can take a buffer arg, so there's no need to
make a separate call to ->translate in some cases.

2 years agods: use ->dflush to distinguish from ->zflush
Eric Wong [Wed, 3 Aug 2022 07:59:11 +0000 (07:59 +0000)]
ds: use ->dflush to distinguish from ->zflush

->zflush is already for GzipFilter in PublicInbox::WWW,
while we use DEFLATE for NNTP and IMAP.  This ought to
make the code easier-to-follow.

2 years agowww: gzip_filter: update a few comments
Eric Wong [Wed, 3 Aug 2022 07:59:10 +0000 (07:59 +0000)]
www: gzip_filter: update a few comments

A few things I noticed while reviewing and evaluating
the PSGI code for JMAP support.

2 years agowww: gzip_filter: gracefully handle socket ->write failures
Eric Wong [Wed, 3 Aug 2022 07:59:09 +0000 (07:59 +0000)]
www: gzip_filter: gracefully handle socket ->write failures

Socket ->write failures are expected and common for TCP traffic,
especially if it's facing unreliable remote connections.  So
just bail out silently if our {gz} field was already clobbered
during the small bit of recursion we hit on ->write failures
from async responses.

This ought to fix some GzipFilter::zflush errors (via $forward
->close from PublicInbox::HTTP) I've been noticing on
deployments running -netd.  I'm still unsure as to why I hadn't
seen them before, but it might've only been ignorance on my
part...

Link: https://public-inbox.org/meta/20220802065436.GA13935@dcvr/
2 years agodaemon: share FDs for identical log paths
Eric Wong [Mon, 1 Aug 2022 21:24:47 +0000 (21:24 +0000)]
daemon: share FDs for identical log paths

We rely on the %logs hash for SIGUSR1 log reopening.  Without this sharing,
some FDs would be hidden inside its respective {HTTP,IMAP,POP3}D
object and not reopened on USR2

2 years agodaemon: allow listening on well-known ports based on protocol
Eric Wong [Mon, 1 Aug 2022 21:24:46 +0000 (21:24 +0000)]
daemon: allow listening on well-known ports based on protocol

This allows admins to use "-l nntp://0.0.0.0/" to bind on port 119
without specifying ":119" on the CLI.

2 years agodaemon: add diagnostics about inherited/bound listeners
Eric Wong [Mon, 1 Aug 2022 21:24:45 +0000 (21:24 +0000)]
daemon: add diagnostics about inherited/bound listeners

These are helpful for diagnosing configuration problems,
as well as a bug (to be fixed in the following commit).

2 years agodaemon: require absolute cert/key paths with --daemonize
Eric Wong [Mon, 1 Aug 2022 21:24:44 +0000 (21:24 +0000)]
daemon: require absolute cert/key paths with --daemonize

This is preparation for supporting loading new certs on SIGHUP.

2 years agodaemon: support per-listener env, .psgi, out, err
Eric Wong [Mon, 1 Aug 2022 21:24:43 +0000 (21:24 +0000)]
daemon: support per-listener env, .psgi, out, err

This allows memory savings by allowing multiple, completely
unrelated-PSGI apps to run within the same process as IMAP,
NNTP, and POP3.

2 years agohttpd: make internals slightly more generic
Eric Wong [Mon, 1 Aug 2022 21:24:42 +0000 (21:24 +0000)]
httpd: make internals slightly more generic

This brings the HTTP server closer to the IMAP/NNTP/POP3
implementations and eliminates package-wide globals in
PublicInbox::HTTPD.  The end goal is to be able to host
completely different PSGI applications on different listen
ports.

2 years agosolver: avoid deprecation warnings in git 2.36.0+
Eric Wong [Sat, 30 Jul 2022 09:38:24 +0000 (09:38 +0000)]
solver: avoid deprecation warnings in git 2.36.0+

git deprecated core.fsyncObjectFiles in favor of core.fsync
with 2.36.0+, while GIT_TEST_FSYNC was added in 2.35.0.  So
use the environment variable since it's been supported slightly
longer than the new configuration knob.

2 years agotests: maintainer test for using mpop
Eric Wong [Fri, 29 Jul 2022 20:41:04 +0000 (20:41 +0000)]
tests: maintainer test for using mpop

This ought to be a good stress test to ensure our POP3
implementation works against the POP3 client I've found.

2 years agodoc|www: flesh out POP3 documentation for servers and users
Eric Wong [Fri, 29 Jul 2022 20:41:03 +0000 (20:41 +0000)]
doc|www: flesh out POP3 documentation for servers and users

Hopefully it makes sense to new users deploying or using POP3...

2 years agodoc: httpd: document GIT_HTTP_MAX_REQUEST_BUFFER
Eric Wong [Thu, 28 Jul 2022 08:10:31 +0000 (08:10 +0000)]
doc: httpd: document GIT_HTTP_MAX_REQUEST_BUFFER

We've always shared this environment with git-http-backend(1)
(but don't (yet) support http.maxRequestBuffer anywhere)

2 years agowww: drop --subject from "git send-email" instructions
Eric Wong [Fri, 22 Jul 2022 20:18:09 +0000 (20:18 +0000)]
www: drop --subject from "git send-email" instructions

Apparently, --subject doesn't work[1] with "git send-email" in
this context.  So drop the CLI arg and add a note to tell the
user to set a "Subject:" line in their response body, instead.

[1] I'm not sure if --subject ever worked as I thought it would,
    or if it's a regression.  In either case, there are current
    versions of git where it doesn't, so just tell users to use
    the currently supported method.

Link: https://80x24.org/lore/git/CAC4O8c-Tf11CpwuRudyrpXv5bGshuyEenV9kKrs0zRWER-+yHA@mail.gmail.com/
2 years agoadd xt/mem-nntpd-tls maintainer test
Eric Wong [Sat, 23 Jul 2022 15:52:09 +0000 (15:52 +0000)]
add xt/mem-nntpd-tls maintainer test

This ensures memory usage is reasonable when DEFLATE and TLS are
enabled.  It's also our only coverage for NNTP COMPRESS since
Net::NNTP has yet to implement compression support:

https://rt.cpan.org/Public/Bug/Display.html?id=129967

2 years agodsdeflate: shorten scope of initial buffer
Eric Wong [Sat, 23 Jul 2022 15:52:08 +0000 (15:52 +0000)]
dsdeflate: shorten scope of initial buffer

There's no need to keep the initial buffer alive in package-wide
scope once it's replaced by `$next' in ->write or ->zflush.

2 years agoxt/mem-imapd-tls: update aliases to DSdeflate subs
Eric Wong [Sat, 23 Jul 2022 15:52:07 +0000 (15:52 +0000)]
xt/mem-imapd-tls: update aliases to DSdeflate subs

Fixes: 23af251dd607c4e7 (imap+nntp: share COMPRESS implementation, 2022-07-23)
2 years agonntp: use substr to check for trailing CRLF
Eric Wong [Sat, 23 Jul 2022 06:13:07 +0000 (06:13 +0000)]
nntp: use substr to check for trailing CRLF

Regexps consume more CPU cycles and memory, and aren't
necessary here since we just converted the entire buffer
to CRLF.

2 years agopop3: reduce memory use while generating the mailbox cache
Eric Wong [Sat, 23 Jul 2022 06:12:16 +0000 (06:12 +0000)]
pop3: reduce memory use while generating the mailbox cache

While the cache itself is relatively compact for 50K messages,
generating it was inefficient due to our schema and Over.pm APIs
being designed for NNTP.  While we won't change our schema for
now, we can choose better DBI APIs to use and limit our ephemeral
memory use.

This amounts to a 60% reduction in memory usage and a 5-10%
speedup against org.kernel.vger.git.0:

{
echo 'USER '$(uuidgen)'@org.kernel.vger.git.0'
echo PASS anonymous
echo STAT
echo QUIT
} | nc $HOST $PORT

2 years agoimap+nntp: share COMPRESS implementation
Eric Wong [Sat, 23 Jul 2022 04:41:55 +0000 (04:41 +0000)]
imap+nntp: share COMPRESS implementation

Their code was nearly identical to begin with, so save some
memory in -netd and disk space for all of our tarball/distro
users, at least.

And I seem to have used multiple inheritance successfully, here,
maybe...

2 years agonntp: resolve inboxes immediately on group listings
Eric Wong [Sat, 23 Jul 2022 04:41:54 +0000 (04:41 +0000)]
nntp: resolve inboxes immediately on group listings

This prevents potential races between SIGHUP config reloads
while gigantic group listings are streaming, allowing us to
avoid many invalidation checks.

This also reduces send(2) syscalls and avoid Perl internal pad
allocations in a few places where it's not beneficial.  There
might be a slight (0.5%) speedup, but I'm not sure if that's
down to system noise, power/thermal management, or other users
on my VM.

2 years agods: share long_step between NNTP and IMAP
Eric Wong [Sat, 23 Jul 2022 04:41:53 +0000 (04:41 +0000)]
ds: share long_step between NNTP and IMAP

It's not actually used by our POP3 code at the moment,
but it may be soon to reduce memory usage when loading
50K smsg objects into memory.

2 years agonntp: inline CRLF in all response lines
Eric Wong [Sat, 23 Jul 2022 04:41:52 +0000 (04:41 +0000)]
nntp: inline CRLF in all response lines

This brings NNTP closer to POP3 and IMAP implementations
to allow CoW avoidance on constants.

2 years agonntp: listgroup_range_i: remove useless `map' op
Eric Wong [Sat, 23 Jul 2022 04:41:51 +0000 (04:41 +0000)]
nntp: listgroup_range_i: remove useless `map' op

No need to iterate through the array twice; and this even seems
a hair faster than what I got with commit 726d6e71aee5d974
(nntp: small speed up for multi-line responses, 2020-12-04)

2 years agods: move requeue_once
Eric Wong [Sat, 23 Jul 2022 04:41:50 +0000 (04:41 +0000)]
ds: move requeue_once

It's the same subroutine everywhere.

2 years agods: move no-op ->zflush to common base class
Eric Wong [Sat, 23 Jul 2022 04:41:49 +0000 (04:41 +0000)]
ds: move no-op ->zflush to common base class

More deduplication, and POP3 never needed it.

2 years agods: support greeting protocols
Eric Wong [Sat, 23 Jul 2022 04:41:48 +0000 (04:41 +0000)]
ds: support greeting protocols

We can share some common code between IMAP, NNTP, and POP3
without too much trouble, so cut down our LoC.

2 years agonntp: remove more() wrapper
Eric Wong [Sat, 23 Jul 2022 04:41:47 +0000 (04:41 +0000)]
nntp: remove more() wrapper

Using PublicInbox::DS->msg_more directly can avoid unnecessary
CoW memory traffic since there's no appending "\r\n".

2 years agonntp: start adding CRLF to responses natively
Eric Wong [Sat, 23 Jul 2022 04:41:46 +0000 (04:41 +0000)]
nntp: start adding CRLF to responses natively

With IMAP and POP3, I've started to embed CRLF into constant
response codes to avoid triggering CoW and extra memory traffic
in Perl.

The end goal is to enable more code sharing between IMAP, NNTP,
and POP3 inside one -netd process.

2 years agonntp: pass regexp to split() callers
Eric Wong [Sat, 23 Jul 2022 04:41:45 +0000 (04:41 +0000)]
nntp: pass regexp to split() callers

Current implementations of Perl5 don't have optimizations for
single-character field separators.

2 years agopop3: drop File::FcntlLock requirement for FreeBSD and Linux
Eric Wong [Thu, 21 Jul 2022 05:36:12 +0000 (05:36 +0000)]
pop3: drop File::FcntlLock requirement for FreeBSD and Linux

I know Linux has a stable ABI for this, and FreeBSD seems to,
too (*BSDs don't have stable syscall numbers, though).
I suspect this is safe enough for all *BSDs.

This is stricter than the MboxLock one since we use exact byte
ranges with these locks.

2 years agowww: note "x=m" and "t=1" (mis)use for GET requests
Eric Wong [Wed, 20 Jul 2022 22:57:07 +0000 (22:57 +0000)]
www: note "x=m" and "t=1" (mis)use for GET requests

We require "x=m" (requests for mboxes) to be POST requests to
avoid unnecessary traffic from crawlers.  "t=1" only collapses
threads in the summary view, which isn't normally accessible
from <form> elements.

This also fixes the missing "[summary|nested]" element when
"x=m" is used.

2 years agogcf2: avoid excessive checks for unlinked files
Eric Wong [Wed, 20 Jul 2022 18:01:28 +0000 (18:01 +0000)]
gcf2: avoid excessive checks for unlinked files

We were misusing the timer and not expiring it before checking
for unlinked files.  Now, we check for unlinked files every 60s,
instead.

2 years agopop3: advertise STLS in CAPA if appropriate
Eric Wong [Wed, 20 Jul 2022 09:24:13 +0000 (09:24 +0000)]
pop3: advertise STLS in CAPA if appropriate

This is documented in RFC 2595, and POP3 clients may rely on
seeing "STLS" in CAPA output to initiate TLS negotiation.

2 years agonetd: setup TLS bits for well-known STARTTLS ports
Eric Wong [Wed, 20 Jul 2022 09:24:12 +0000 (09:24 +0000)]
netd: setup TLS bits for well-known STARTTLS ports

Unfortunately, I can't think of an easy way to test this in
our test suite since binding these ports are privileged and
are often in use, anyways.