fix: extract snapshot tarballs into $IOTEX_HOME/data - #337
Open
envestcc wants to merge 3 commits into
Open
Conversation
The snapshot tarballs are flat — their root members are chain-*.db / bloomfilter.index.db, with no leading data/ directory. Several call sites still extracted them with `-C $IOTEX_HOME`, a leftover from the pre-split single-package era. eec89cb fixed all_in_one_*.sh when it switched to the split URLs, but missed the two setup scripts and the docs. The container mounts $IOTEX_HOME/data as /var/data and the install/upgrade detection in setup_fullnode.sh keys off $IOTEX_HOME/data/chain.db, so extracting one level too high dumps ~190GB into $IOTEX_HOME while the mounted data dir stays empty. The node then syncs from genesis, and every subsequent run of the script still sees no data/chain.db, so it is treated as a fresh install again and re-downloads the snapshot. - scripts/setup_fullnode.sh, scripts/setup_fullnode_marketplace.sh: extract into $IOTEX_HOME/data, and mkdir -p it first (tar -C fails on a missing directory, and only the fresh-install branch created it). - README.md, README_CN.md, README_testnet.md, README_CN_testnet.md: Option 1 (gateway snapshot) had no -C at all, so it extracted into the current directory; it also read a relative path while aria2c writes to $IOTEX_HOME. Step 5 in each was already correct. - archive-node.md: same missing -C for all three archive tarballs, right above a listing that claims the files land in $IOTEX_HOME/data. - changelog/v1.2-instruction.md: the combined *-data-with-idx-latest snapshots now 404. Repointed at the gateway snapshot, fixed the same extraction bug, and added a banner marking the page historical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This path had no automated coverage, which is why the extraction target could drift out of sync with the published tarball layout and stay broken. The failure needs two facts to disagree, and each is fine on its own: the tarballs are flat, and the extractions name a destination. So assert them separately instead of trying to run the thing end to end — a real --auto --snapshot run needs ~190GB compressed and ~265GB extracted, well beyond a hosted runner. - scripts/ci/check_extract_targets.sh: static scan of shell and markdown for tar invocations in extract mode, requiring each to target $IOTEX_HOME/data. No network. Extractions unrelated to chain data (Go toolchain, prometheus, diagnostics bundle) are allowlisted. Verified against the pre-fix tree: flags exactly the 11 sites this branch fixes and passes the 13 that were already correct. - scripts/ci/check_snapshot_layout.sh: reads the leading members of each of the four published tarballs from a 1MB range request (a tar header is the first 512 bytes) and fails if any sit under data/. ~4MB total. Catches the inverse regression, where the publishing side starts wrapping contents in a data/ directory and the now-correct extractions silently become wrong. The layout check drops the archive's own "./" root entry before judging. An initial version read only the first member and was fooled by that entry into passing a tarball nested under data/ — the exact regression it exists to catch. Tested against four archive shapes (flat and nested, each with and without a leading root entry) plus unreachable and empty-body responses, which fail rather than silently pass. The layout check also runs on a daily schedule: the tarballs can change without any commit here, so it cannot be gated on PRs alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
check_extract_targets.sh quotes the correct tar form in its own failure message, so once the file was tracked by git it matched its own scan and reported itself as a violation. scripts/ci/ is now excluded — it is tooling, not something anyone runs against a node. This did not show up in local testing because the check builds its file list from `git ls-files`, and the script was still untracked when it was run locally. The first run that saw the committed tree was CI. Also corrects the counts in the previous commit message: the check flags 12 sites on the pre-fix tree and passes the 12 that were already correct (24 total), not 11 and 13. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The snapshot tarballs are flat — the root members are
chain-*.db/bloomfilter.index.db, with no leadingdata/directory. Verified by streaming the first member of all four published tarballs:mainnet-data-snapshot-core-latestchain-00000001.dbmainnet-data-snapshot-gateway-latestbloomfilter.index.dbtestnet-data-snapshot-core-latestchain-00000001.dbtestnet-data-snapshot-gateway-latestbloomfilter.index.dbSeveral call sites still extracted them with
-C $IOTEX_HOME, a leftover from the pre-split single-package era. eec89cb fixedall_in_one_*.shwhen it moved to the split URLs, but the two setup scripts and the docs never followed.The container mounts
$IOTEX_HOME/dataas/var/data, and the install/upgrade detection insetup_fullnode.shkeys off$IOTEX_HOME/data/chain.db. Extracting one level too high meansbash setup_fullnode.sh --auto --snapshotdumps ~190GB into$IOTEX_HOMEwhile the mounteddata/stays empty — the node then syncs from genesis.Note the failure order:
checkPrivateKeyruns beforedonwloadBlockDataFile, so thedata/chain.dbcheck is not what breaks first. The empty mount is. The check-failure is the second-order symptom on the next run: the script still sees nodata/chain.db, treats the node as a fresh install again, and re-downloads the whole snapshot — and since_IS_UPGRADE_also requiresdata/chain.db, it can never reach the upgrade path.Fixing the extraction target restores detection:
chain.dbis present in the snapshot alongside thechain-%08d.dbaux files (filedaotreats it as the master file), so[ -f data/chain.db ]holds after a correct extraction.Changes
scripts/setup_fullnode.sh,scripts/setup_fullnode_marketplace.sh— extract into$IOTEX_HOME/data, andmkdir -pit first:tar -Cfails on a missing directory, and today only the fresh-install branch creates it.README.md,README_CN.md,README_testnet.md,README_CN_testnet.md— Option 1 (gateway snapshot) had no-Cat all, so it extracted into the current directory; it also read a relativedata_index.tar.gzwhilearia2cwrites to$IOTEX_HOME. Step 5 in each was already correct.archive-node.md— same missing-Cfor all three archive tarballs, directly above a listing that claims the files land in$IOTEX_HOME/data.changelog/v1.2-instruction.md—t.iotex.me/{mainnet,testnet}-data-with-idx-latestnow returns HTTP 404 (302 →storage.iotex.io/...tar.gz→ 404). Repointed at the gateway snapshot, fixed the same extraction bug on that page, and added a banner marking it historical.AGENT.mdandall_in_one_*.shwere already correct and are unchanged.Verification
bash -nclean on both modified scripts.chain.db,chain-00000001.db,trie.db,index.db): files land indata/, and[ -f $IOTEX_HOME/data/chain.db ]passes.$IOTEX_HOME/data.CI
This PR adds the repo's first
.github/workflows/. The bug needed two facts to disagree, each fine alone, so each is asserted separately — neither check downloads a snapshot or starts a node:scripts/ci/check_extract_targets.sh— static scan of shell and markdown for tar invocations in extract mode, requiring each to target$IOTEX_HOME/data. No network. Run against the pre-fix tree it flags 12 sites and passes the 12 that were already correct; on this branch all 24 pass.scripts/ci/check_snapshot_layout.sh— reads the leading members of each of the four published tarballs from a 1MB range request and fails if any sit underdata/. ~4MB total, runs on PRs and daily. Catches the inverse regression, where the publishing side starts wrapping contents indata/and the now-correct extractions silently become wrong.Both jobs are green on this branch.
End-to-end verification
Ran the fixed
setup_fullnode.sh --auto --snapshotagainst the testnet snapshot (75.7GB — same code path as mainnet, an eighth the data) on a scratch directory of an internal host. Results:$IOTEX_HOMEroot — onlydata/ etc/ log/ monitor/ tmp/. Before the fix the 53 chain files land here and the mounteddata/stays empty.$IOTEX_HOME/data/chain.dbpresent, 53 files, 94GB.State factory started height=46079958, thenblock sync intervals Start=46079959, committing blocks). With the bug this reads height 1.All test artifacts were removed afterwards.
One prerequisite issue surfaced during the run and is not fixed here:
checkDockerCompose()requires the EOLdocker-composev1 binary, which is absent on hosts with a current Docker (Compose is a plugin,docker compose). Three internal hosts all failed this check and needed a shim. That blockssetup_fullnode.shentirely on any modern install and deserves its own issue.