Retry resize2fs on transient Permission denied during persistent disk grow

[neddp]

What is this change about?

When a persistent disk is extended online, the agent calls resize2fs to grow the ext4 filesystem after resizing the partition. We noticed this on OpenStack and it fails transiently with:

Error: Action Failed get_task: Task <xxxxxxxxx> result: Adjusting persistent disk partitioning: Failed to grow filesystem: Failed to grow Ext4 filesystem: Running command: 'resize2fs -f /dev/sdb1', stdout: 'Filesystem at /dev/sdb1 is mounted on /var/vcap/store; on-line resizing required
old_desc_blocks = 13, new_desc_blocks = 128
', stderr: 'resize2fs 1.46.5 (30-Dec-2021)
resize2fs: Permission denied to resize filesystem
': exit status 1

The EXT4_IOC_RESIZE_FS ioctl returns EPERM when the kernel cannot yet confirm the block device's new size - a race between the storage layer acknowledging the extension and the guest-visible device reflecting it. The error resolves on its own once the storage layer settles.

This change retries resize2fs up to 10 times with a 5s sleep between attempts, but only on "Permission denied to resize filesystem" - any other error still fails immediately. Maximum additional wait before giving up: 50 seconds.

Please provide contextual information.

Observed repeatedly in production during a disk type change (100G → 1T).

The fix follows the existing clock-injection pattern used by NewPartedPartitioner and NewSfdiskPartitioner in the same package, making the retry delay injectable for tests. The original NewLinuxFormatter(runner, fs) signature is preserved; a new NewLinuxFormatterWithClock(runner, fs, clock) is added for test injection.

What tests have you run against this PR?

• Full platform/disk unit test suite: 173 passed, 0 failed
• Three new specs covering:
  • Non-retryable errors fail immediately (SleepCallCount() == 0)
  • Transient EPERM retries until success (sleep count matches failed attempts)
  • Exhausted retries returns the error after 10 sleeps

How should this change be described in bosh-agent release notes?

Persistent disk grows that fail with a transient "Permission denied" from resize2fs are now retried automatically (up to 10 times, 5s apart), avoiding deploy failures during online disk extension on affected IaaS environments.

Does this PR introduce a breaking change?

No. The NewLinuxFormatter(runner, fs) signature is unchanged. A new NewLinuxFormatterWithClock(runner, fs, clock) constructor is added alongside it for clock injection in tests.

[coderabbitai]

Walkthrough

The Linux formatter now accepts a clock service. Ext4 growth delegates to a retry helper. Permission-denied resize2fs failures retry up to ten times with five-second delays. Non-retryable errors return immediately. Tests cover immediate failure, successful retry, retry exhaustion, clock usage, and existing filesystem scenarios.

Suggested reviewers: copilot

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly describes the primary change: retrying resize2fs after transient permission-denied errors during persistent disk growth.
Description check	✅ Passed	The description covers the change, context, tests, release notes, and breaking-change status, but it omits the template's AI Review Feedback section.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/grow-filesystem-permission-denied-retry

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@platform/disk/linux_formatter.go`:
- Around line 112-120: Update the retry loop around resize2fs in
platform/disk/linux_formatter.go lines 112-120 to sleep only when another
attempt remains, preserving immediate returns for success and non-permission
errors. Update the corresponding expectation in
platform/disk/linux_formatter_test.go lines 308-315 to assert nine sleeps for
ten failed attempts.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 9b65b799-b77b-4ae5-bdf3-dcbcb623e607

📥 Commits

Reviewing files that changed from the base of the PR and between 4f9ec7c and aa8ebcd.

📒 Files selected for processing (3)

• platform/disk/linux_disk_manager.go
• platform/disk/linux_formatter.go
• platform/disk/linux_formatter_test.go

[Copilot]

Pull request overview

Adds targeted retry logic around resize2fs during online ext4 filesystem growth to mitigate transient EPERM (“Permission denied to resize filesystem”) observed on some IaaS environments. This is implemented with an injected clock to make retry delays testable and updates the disk manager wiring and unit tests accordingly.

Changes:

• Inject clock.Clock into NewLinuxFormatter and use it to sleep between resize2fs retries.
• Add ext4 grow retry logic gated specifically on the “Permission denied to resize filesystem” error substring.
• Extend platform/disk unit specs to cover retryable vs non-retryable resize2fs failures.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
platform/disk/linux_formatter.go	Adds retry wrapper for ext4 grow and injects a clock for testable delays.
platform/disk/linux_formatter_test.go	Updates formatter construction and adds specs validating retry behavior.
platform/disk/linux_disk_manager.go	Updates formatter instantiation to pass a real clock.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

platform/disk/linux_formatter.go:30

• NewLinuxFormatter is an exported constructor in a non-internal/ package, so changing its signature is a breaking API change for any downstream Go code that imports github.com/cloudfoundry/bosh-agent/v2/platform/disk. The PR description says this is internal/no external consumers; that may be true in practice, but it’s not enforced by the package structure. Consider keeping the existing NewLinuxFormatter(runner, fs) signature as a compatibility wrapper (calling a new clock-injectable constructor) if you want to avoid a breaking change.

func NewLinuxFormatter(runner boshsys.CmdRunner, fs boshsys.FileSystem, timeService clock.Clock) Formatter {
	return linuxFormatter{
		runner:      runner,
		fs:          fs,
		timeService: timeService,
	}

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

[neddp]

The root cause was actually a corrupted file system. Unfortunately, the error from resize2fs is generic and the operator needs to check the kernel logs for the actual problem.

[ramonskie]

We could do a if it fails to check for a quick file system corruption? Or add a extra debugging line. That mentions that filesystem could be corrupted

[neddp]

Hi @ramonskie,

I looked into it and we'll need to check the kernel logs. It didn't sound like it's worth the effort and additional code for such an edge case. I think the resize2fs: Permission denied to resize filesystem log should be enough to guide the user to isolate it further.

Something else we noticed. We ran into this issue in a CI pipeline and it fails the first time (partition resized, the filesystem fails to be resized). Any following runs actually succeed, since the code does not check if there is a discrepancy between the partition and filesystem size. I tried to add a new check with #465, but I didn't like the implementation and couldn't think of a better one.

[ramonskie]

Maby put it within the bosh docs then?

[neddp]

Documented with cloudfoundry/docs-bosh#918.