]> Sergey Matveev's repositories - public-inbox.git/commit
lei q: parallelize Maildir and mbox writing
authorEric Wong <e@80x24.org>
Mon, 18 Jan 2021 10:30:31 +0000 (04:30 -0600)
committerEric Wong <e@80x24.org>
Mon, 18 Jan 2021 21:20:25 +0000 (21:20 +0000)
commit51191d611e918ff3ef6e9ce8ee52ba7b2cd2144c
treec787b82356d3f7e42d003a478d8c3a3ce0ee956e
parent21671ed82f8d1a7b6de593e073079e29c5675aa8
lei q: parallelize Maildir and mbox writing

With 4 dedicated workers, this seems to provide a 100-120%
speedup on a 4 core machine when writing thousands of search
results to a Maildir or mbox.  This also sets us up for
high-latency IMAP destinations in the future.

This opens the door to more speedup opportunities such
as optimizing dedupe locking and other ways to reduce
contention.

This change is fairly complex and convoluted, unfortunately.
Further work may allow us to simplify it and even improve
performance.
lib/PublicInbox/IPC.pm
lib/PublicInbox/LEI.pm
lib/PublicInbox/LeiOverview.pm
lib/PublicInbox/LeiQuery.pm
lib/PublicInbox/LeiToMail.pm
lib/PublicInbox/LeiXSearch.pm
lib/PublicInbox/Spawn.pm