Background build

-W, --web: use the original per-article HTTP requests for remote ID extraction, database checks, and rendering instead of background batches, and rebuild the nested set synchronously in the foreground. It also applies to ourbigbook --web-nested-set --web-individual-upload, providing a foreground fallback for standalone tree rebuilding. Foreground operations remain subject to the web router's request timeout.

By default, -W, --web manages one current build per user through PUT /api/articles/bulk/build, stages extraction, database-check, and render batches through PUT /api/articles/bulk, then submits the complete plan through PUT /api/articles/bulk/build/commit. Starting a replacement removes the user's previous jobs and history, but never articles. An in-flight article or tree transaction finishes before replacement can proceed; old workers cannot start further articles. Build identity in the CLI and UI is simply the user. An internal generation token rejects delayed requests belonging to a replaced upload. Nothing runs until submission verifies that every batch is present and in phase order. Each batch contains at most 100 articles and the CLI limits its serialized payload to approximately 4 MiB. Each phase can stage at most the larger of the user's article quota and existing article count, bounding job storage even for one-article batches. Source bodies stay on disk until their individual staging request. Local source splitting and its checks still happen in the CLI. Asset uploads and cleanup of removed articles retain their existing behavior.

Workers save progress after each article. Successful pre-render database checks are saved with the source hash, so restarting -W, --web skips checks already committed for unchanged sources even if rendering is unfinished. ID extraction clears that checkpoint, including forced extraction; failed checks are never saved. Rendering still validates against the current database. After the CLI prints web_upload: @USERNAME: submitted, it only monitors progress: the server performs all extraction batches in parent-first order, then all checks, then all renders, followed by the final tree rebuild. Closing the CLI or restarting the server does not lose the submitted plan. Jobs waiting for an earlier batch have status waiting; each next batch joins the existing shared or dedicated worker queue, preserving fairness between users. A failed batch stops the build and cancels its remaining work. Before local conversion, an existing unfinished build prompts for confirmation: accepting discards it and starts fresh; declining only watches it, without uploading afterwards. Non-interactive invocations default to watching. --web-force replaces without prompting, and --web-watch only watches. Both imply -W, --web. Abandoned staged jobs remain until the next replacement; there is no periodic history cleanup or staged-upload expiry. Source bodies are discarded when their extraction batch finishes. --web-id, --web-start-id, --web-max-renders, --web-force-render, and --no-web-render still apply; --no-web-render runs extraction and database checks without rendering.

The Build jobs tab in user settings has TODO and Done subtabs. TODO is the default and shows ongoing jobs first, followed by queued, waiting, and staged jobs in submission order. Done shows completed and failed jobs, latest finish first. Both lists show the current build, are paginated, and include tree rebuilds, batch position, Job ID, phase, status, progress, creation timestamp in UTC, runtime, and errors. Runtime measures elapsed time from the first worker start to completion or failure, including retry/recovery delays but not the initial queue wait; older jobs without a recorded start show a dash. The table refreshes every three seconds while open. Site settings has the same table for all users, with usernames and date-only timestamps, and without private error details. Local workers run automatically. Production uses the same Heroku credentials as background nested-set rebuilding; update the server and CLI together. The individual-upload fallback also works against older servers.

By default, all users share one background build worker slot, covering both article batches and tree rebuilds in a single FIFO queue. Administrators can enable "Dedicated build worker" in a user's settings to grant that user one separate slot; this defaults to disabled, including for administrators. Waiting jobs show as "queued" and their execution timeout starts only when dispatched. Workers reuse their slot for waiting batches and exit when idle. The web server checks the durable queue every five seconds, so already queued work does not require a connected CLI. Before replacing a worker, it waits for local process exit or confirmation that the Heroku dyno stopped; an uncertain launch reserves its slot until the recovery deadline. This bounds concurrent workers, not total monthly spending: sustained uploads can keep the shared worker busy. Browser edits and the foreground debug fallback keep their existing synchronous behavior and do not launch workers.