From: Eric Wong Date: Wed, 4 Jan 2023 03:49:34 +0000 (+0000) Subject: git: fix asynchronous batching for deep pipelines X-Git-Url: http://www.git.stargrave.org/?a=commitdiff_plain;h=d4ba8828ab23f2785be54493495bbf7e1d62c0b0;hp=d4ba8828ab23f2785be54493495bbf7e1d62c0b0;p=public-inbox.git git: fix asynchronous batching for deep pipelines ...By using non-blocking pipe writes. This avoids problems for musl (and other libc) where getdelim(3) used by `git cat-file --batch*' uses a smaller input buffer than glibc or FreeBSD libc. My key mistake was our check against MAX_INFLIGHT is only useful for the initial batch of requests. It is not useful for subsequent requests since git will drain the pipe at unpredictable rates due to libc differences. To fix this problem, I initially tried to drain the read pipe as long as readable data was pending. However, reading git output without giving git more work would also limit parallelism opportunities since we don't want git to sit idle, either. This change ensures we keep both pipes reasonably full to reduce stalls and maximize parallelism between git and public-inbox. While the limit set a few weeks ago in commit 56e6e587745c (git: cap MAX_INFLIGHT value to POSIX minimum, 2022-12-21) remains in place, any higher or lower limit will work. It may be worth it to use an even lower limit to improve interactivity w.r.t. Ctrl-C interrupts. I've tested the pre-56e6e587745c and even higher values on an Alpine VM in the GCC Farm Reported-by: Chris Brannon Link: https://public-inbox.org/meta/87edssl7u0.fsf@the-brannons.com/T/ ---