Skip to content

fix: extract snapshot tarballs into $IOTEX_HOME/data - #337

Open
envestcc wants to merge 3 commits into
masterfrom
fix/snapshot-extract-path
Open

fix: extract snapshot tarballs into $IOTEX_HOME/data#337
envestcc wants to merge 3 commits into
masterfrom
fix/snapshot-extract-path

Conversation

@envestcc

@envestcc envestcc commented Jul 31, 2026

Copy link
Copy Markdown
Member

Problem

The snapshot tarballs are flat — the root members are chain-*.db / bloomfilter.index.db, with no leading data/ directory. Verified by streaming the first member of all four published tarballs:

tarball first member
mainnet-data-snapshot-core-latest chain-00000001.db
mainnet-data-snapshot-gateway-latest bloomfilter.index.db
testnet-data-snapshot-core-latest chain-00000001.db
testnet-data-snapshot-gateway-latest bloomfilter.index.db

Several call sites still extracted them with -C $IOTEX_HOME, a leftover from the pre-split single-package era. eec89cb fixed all_in_one_*.sh when it moved to the split URLs, but the two setup scripts and the docs never followed.

The container mounts $IOTEX_HOME/data as /var/data, and the install/upgrade detection in setup_fullnode.sh keys off $IOTEX_HOME/data/chain.db. Extracting one level too high means bash setup_fullnode.sh --auto --snapshot dumps ~190GB into $IOTEX_HOME while the mounted data/ stays empty — the node then syncs from genesis.

Note the failure order: checkPrivateKey runs before donwloadBlockDataFile, so the data/chain.db check is not what breaks first. The empty mount is. The check-failure is the second-order symptom on the next run: the script still sees no data/chain.db, treats the node as a fresh install again, and re-downloads the whole snapshot — and since _IS_UPGRADE_ also requires data/chain.db, it can never reach the upgrade path.

Fixing the extraction target restores detection: chain.db is present in the snapshot alongside the chain-%08d.db aux files (filedao treats it as the master file), so [ -f data/chain.db ] holds after a correct extraction.

Changes

  • scripts/setup_fullnode.sh, scripts/setup_fullnode_marketplace.sh — extract into $IOTEX_HOME/data, and mkdir -p it first: tar -C fails on a missing directory, and today only the fresh-install branch creates it.
  • README.md, README_CN.md, README_testnet.md, README_CN_testnet.md — Option 1 (gateway snapshot) had no -C at all, so it extracted into the current directory; it also read a relative data_index.tar.gz while aria2c writes to $IOTEX_HOME. Step 5 in each was already correct.
  • archive-node.md — same missing -C for all three archive tarballs, directly above a listing that claims the files land in $IOTEX_HOME/data.
  • changelog/v1.2-instruction.mdt.iotex.me/{mainnet,testnet}-data-with-idx-latest now returns HTTP 404 (302 → storage.iotex.io/...tar.gz → 404). Repointed at the gateway snapshot, fixed the same extraction bug on that page, and added a banner marking it historical.

AGENT.md and all_in_one_*.sh were already correct and are unchanged.

Verification

  • Confirmed all four tarballs are flat (table above).
  • bash -n clean on both modified scripts.
  • Exercised the fixed extraction against a synthetic flat tarball shaped like production (chain.db, chain-00000001.db, trie.db, index.db): files land in data/, and [ -f $IOTEX_HOME/data/chain.db ] passes.
  • Grepped the tree — every remaining snapshot extraction now targets $IOTEX_HOME/data.

CI

This PR adds the repo's first .github/workflows/. The bug needed two facts to disagree, each fine alone, so each is asserted separately — neither check downloads a snapshot or starts a node:

  • scripts/ci/check_extract_targets.sh — static scan of shell and markdown for tar invocations in extract mode, requiring each to target $IOTEX_HOME/data. No network. Run against the pre-fix tree it flags 12 sites and passes the 12 that were already correct; on this branch all 24 pass.
  • scripts/ci/check_snapshot_layout.sh — reads the leading members of each of the four published tarballs from a 1MB range request and fails if any sit under data/. ~4MB total, runs on PRs and daily. Catches the inverse regression, where the publishing side starts wrapping contents in data/ and the now-correct extractions silently become wrong.

Both jobs are green on this branch.

End-to-end verification

Ran the fixed setup_fullnode.sh --auto --snapshot against the testnet snapshot (75.7GB — same code path as mainnet, an eighth the data) on a scratch directory of an internal host. Results:

  • Nothing landed at $IOTEX_HOME root — only data/ etc/ log/ monitor/ tmp/. Before the fix the 53 chain files land here and the mounted data/ stays empty.
  • $IOTEX_HOME/data/chain.db present, 53 files, 94GB.
  • Node started at height 46,079,958 and synced forward (State factory started height=46079958, then block sync intervals Start=46079959, committing blocks). With the bug this reads height 1.
  • No panics or fatals; container healthy on the testnet p2p port.

All test artifacts were removed afterwards.

One prerequisite issue surfaced during the run and is not fixed here: checkDockerCompose() requires the EOL docker-compose v1 binary, which is absent on hosts with a current Docker (Compose is a plugin, docker compose). Three internal hosts all failed this check and needed a shim. That blocks setup_fullnode.sh entirely on any modern install and deserves its own issue.

envestcc and others added 3 commits July 31, 2026 10:04
The snapshot tarballs are flat — their root members are chain-*.db /
bloomfilter.index.db, with no leading data/ directory. Several call sites
still extracted them with `-C $IOTEX_HOME`, a leftover from the pre-split
single-package era. eec89cb fixed all_in_one_*.sh when it switched to the
split URLs, but missed the two setup scripts and the docs.

The container mounts $IOTEX_HOME/data as /var/data and the install/upgrade
detection in setup_fullnode.sh keys off $IOTEX_HOME/data/chain.db, so
extracting one level too high dumps ~190GB into $IOTEX_HOME while the
mounted data dir stays empty. The node then syncs from genesis, and every
subsequent run of the script still sees no data/chain.db, so it is treated
as a fresh install again and re-downloads the snapshot.

- scripts/setup_fullnode.sh, scripts/setup_fullnode_marketplace.sh:
  extract into $IOTEX_HOME/data, and mkdir -p it first (tar -C fails on a
  missing directory, and only the fresh-install branch created it).
- README.md, README_CN.md, README_testnet.md, README_CN_testnet.md:
  Option 1 (gateway snapshot) had no -C at all, so it extracted into the
  current directory; it also read a relative path while aria2c writes to
  $IOTEX_HOME. Step 5 in each was already correct.
- archive-node.md: same missing -C for all three archive tarballs, right
  above a listing that claims the files land in $IOTEX_HOME/data.
- changelog/v1.2-instruction.md: the combined *-data-with-idx-latest
  snapshots now 404. Repointed at the gateway snapshot, fixed the same
  extraction bug, and added a banner marking the page historical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This path had no automated coverage, which is why the extraction target
could drift out of sync with the published tarball layout and stay broken.

The failure needs two facts to disagree, and each is fine on its own: the
tarballs are flat, and the extractions name a destination. So assert them
separately instead of trying to run the thing end to end — a real
--auto --snapshot run needs ~190GB compressed and ~265GB extracted, well
beyond a hosted runner.

- scripts/ci/check_extract_targets.sh: static scan of shell and markdown
  for tar invocations in extract mode, requiring each to target
  $IOTEX_HOME/data. No network. Extractions unrelated to chain data (Go
  toolchain, prometheus, diagnostics bundle) are allowlisted. Verified
  against the pre-fix tree: flags exactly the 11 sites this branch fixes
  and passes the 13 that were already correct.
- scripts/ci/check_snapshot_layout.sh: reads the leading members of each
  of the four published tarballs from a 1MB range request (a tar header
  is the first 512 bytes) and fails if any sit under data/. ~4MB total.
  Catches the inverse regression, where the publishing side starts
  wrapping contents in a data/ directory and the now-correct extractions
  silently become wrong.

The layout check drops the archive's own "./" root entry before judging.
An initial version read only the first member and was fooled by that
entry into passing a tarball nested under data/ — the exact regression it
exists to catch. Tested against four archive shapes (flat and nested,
each with and without a leading root entry) plus unreachable and
empty-body responses, which fail rather than silently pass.

The layout check also runs on a daily schedule: the tarballs can change
without any commit here, so it cannot be gated on PRs alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
check_extract_targets.sh quotes the correct tar form in its own failure
message, so once the file was tracked by git it matched its own scan and
reported itself as a violation. scripts/ci/ is now excluded — it is
tooling, not something anyone runs against a node.

This did not show up in local testing because the check builds its file
list from `git ls-files`, and the script was still untracked when it was
run locally. The first run that saw the committed tree was CI.

Also corrects the counts in the previous commit message: the check flags
12 sites on the pre-fix tree and passes the 12 that were already correct
(24 total), not 11 and 13.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant