lei q|up: limit default write --jobs for IMAP(S)

Eric Wong <e@80x24.org> wrote:
> Thanks for confirming things work as intended.  I think the
> default should be clamped, though... 15 seems a bit high for
> smaller IMAP servers *shrug*

--------8<-------
Subject: [PATCH] lei q|up: limit default write --jobs for IMAP(S)

IMAP(S) servers often limit per-user connections, so avoid
bumping into limits to improve the out-of-the-box experience.
4 seems like a conservative default, since we already chose
that number for remote HTTP(S) endpoints.

Link: https://public-inbox.org/meta/20220910201958.GA12212@dcvr/

diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 8134223e8bc333d5224e300fb1b04f4f9baf2b71..d52c5b046d1da4c2f1f530899aadca06de9a303a 100644 (file)
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -135,8 +135,8 @@ Set the number of query and write worker processes for parallelism.
 C<QUERY_WORKERS> defaults to the number of CPUs available, but 4 per
 remote (HTTP/HTTPS) host.
 
-C<WRITE_WORKERS> defaults to the number of CPUs available for Maildir,
-IMAP/IMAPS, and mbox* destinations.
+C<WRITE_WORKERS> defaults to 75% of the number of CPUs available for
+Maildir and mbox* destinations, but 4 per IMAP/IMAPS host.
 
 Omitting C<QUERY_WORKERS> but leaving the comma (C<,>) allows
 one to only set C<WRITE_WORKERS>

diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index df9c32b301d5be97c07eb4e4543267826f376bb2..0f8392362e4bc9ec2fb213a3a349484bf3794798 100644 (file)
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -39,8 +39,11 @@ sub _start_query { # used by "lei q" and "lei up"
                        $lms->lms_write_prepare->lms_pause; # just create
                }
        }
-       $l2m and $l2m->{-wq_nr_workers} //= $mj //
-               int($nproc * 0.75 + 0.5); # keep some CPU for git
+       $l2m and $l2m->{-wq_nr_workers} //= $mj // do {
+               # keep some CPU for git, and don't overload IMAP destinations
+               my $n = int($nproc * 0.75 + 0.5);
+               $self->{net} && $n > 4 ? 4 : $n;
+       };
 
        # descending docid order is cheapest, MUA controls sorting order
        $self->{mset_opt}->{relevance} //= -2 if $l2m || $opt->{threads};