]> Sergey Matveev's repositories - public-inbox.git/commit
projects / public-inbox.git / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 01888d6) | patch
extsearchidx: support --batch-size checkpoints
authorEric Wong <e@80x24.org>
Tue, 27 Oct 2020 07:54:52 +0000 (07:54 +0000)
committerEric Wong <e@80x24.org>
Sat, 7 Nov 2020 10:22:14 +0000 (10:22 +0000)
commit7ce1a0d97294c5603cf8c6cf89cab491cc5840d3
tree39deb59c56e7e1a0e52f768f9ee0c2a87913629etree
parent01888d60a8f958f6ae23fffd8541011b5ef74a19commit | diff
extsearchidx: support --batch-size checkpoints

This is needed to limit the RSS of processes and ensure the
stored data in over.sqlite3 and Xapian DBs are consistent if
interrupted.  Without checkpoints, indexing lore causes shard
workers to take several GB of memory and thrash/OOM smaller
systems.
lib/PublicInbox/ExtSearchIdx.pm diff | blob | history
My patches to https://public-inbox.org/public-inbox.git