OurBigBookTODO: get this working. Maybe we should also bake it into the
ourbigbook CLI tool as well for greater portability. Starting like this as a faster way to prototype:
rm -rf out/parallel
mkdir -p out/parallel
# ID extraction.
git ls-files | grep -E '\.bigb$' | parallel -X ourbigbook --no-render --no-check-db --outdir 'out/parallel/{%}' '{}'
./merge-dbs out/db.sqlite3 out/parallel/*/db.sqlite3
ourbigbook --check-db
# Render.
git ls-files | grep -E '\.bigb$' | parallel -X ourbigbook --no-check-db '{}'Observed
--no-render speedup on 1k small files from the Wikipedia bot and 8 cores: 3x. So not bad.Observed render speedup on 1k small files from the Wikipedia bot and 8 cores: none. TODO. Is this because of database contention?