Conversation
…s#2139) SQLite opened with no pragmas: journal_mode=delete plus the driver's 5s busy_timeout. With all scrapers selected, dozens of concurrent writers and UI polling reads pile up past 5s, SaveWithRetry's 10x100ms all fail, and log.Fatal kills the server. Set _busy_timeout=30000&_journal_mode=WAL& _synchronous=NORMAL on every sqlite handle via the DSN (sqlite3 driver only) and back off SaveWithRetry to 30 attempts, 1s-15s. Fatal backstop kept: call sites ignore the error return, so returning it would silently skip saves. Assisted-By: muse-spark-1.3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2139.
Running all selected scrapers dies with
Failed to save ... database is locked(10 retries) and the process exits. Root cause, all inpkg/models/db.go:concurrent_scrapersis 9999) writer queues routinely exceed 5s.SaveWithRetryretried 10x at ~100ms, thenlog.Fatalkilled the whole server.--concurrent_scrapers 1cannot help: it only batches scraper goroutines, it never serialized DB writes.Changes:
sqliteWriteDSN()appends_busy_timeout=30000&_journal_mode=WAL&_synchronous=NORMALfor the sqlite3 driver only (MySQL untouched), used by bothGetDB()andGetCommonDB(). DSN params apply to every pooled connection. WAL stops readers blocking the writer; 30s covers writer queues.SaveWithRetry: 30 attempts with 1s-15s backoff. Thelog.Fatalbackstop is kept deliberately: every call site ignores the error return, so returning it would turn loud death into silently skipped saves. Transient locks no longer reach it.pkg/models/db_sqlite_test.go: DSN unit test, WAL/timeout assertions, and a lock-contention regression test (8s write hold; fails at ~5s on the plain DSN with the reported error, waits and succeeds with the fix).Notes: existing databases flip to WAL on first open after upgrade (backward compatible, SQLite-recommended pairing with synchronous=NORMAL). No changes to scan, edit, or scrapers.