Skip to content

[filesystem] avoid zero-skip loop in HadoopDataInputStream#3738

Draft
wolfkill wants to merge 1 commit into
apache:mainfrom
wolfkill:fix/hadoop-seek-past-eof
Draft

[filesystem] avoid zero-skip loop in HadoopDataInputStream#3738
wolfkill wants to merge 1 commit into
apache:mainfrom
wolfkill:fix/hadoop-seek-past-eof

Conversation

@wolfkill

Copy link
Copy Markdown

Fixes #3718.

Root Cause

HadoopDataInputStream.seek() optimizes small forward seeks by calling skipFully(delta) instead of issuing a Hadoop seek().

skipFully() assumed every FSDataInputStream.skip(bytes) call makes forward progress. Some streams return 0 at EOF or when no bytes can currently be skipped. In that case bytes never decreases and the loop can spin forever instead of letting the underlying filesystem handle the requested target position.

Changes

  • Pass the requested target position into skipFully.
  • If skip() returns 0 or a negative value, fall back to forceSeek(seekPos) and return.
  • Add a regression test using a seekable stream that returns one zero-byte skip and fails on a second skip, proving the implementation no longer retries zero-progress skips.

Validation

  • RED: JAVA_HOME=/Users/cc/Library/Java/JavaVirtualMachines/corretto-18.0.2/Contents/Home mvn -pl fluss-filesystems/fluss-fs-hadoop -am -Dtest=HadoopDataInputStreamTest#testSeekFallsBackWhenSkipMakesNoProgress -DfailIfNoTests=false test failed before the fix because skip() was retried after making no progress.
  • GREEN: JAVA_HOME=/Users/cc/Library/Java/JavaVirtualMachines/corretto-18.0.2/Contents/Home mvn -pl fluss-filesystems/fluss-fs-hadoop -am -Dtest=HadoopDataInputStreamTest#testSeekFallsBackWhenSkipMakesNoProgress -DfailIfNoTests=false test
  • JAVA_HOME=/Users/cc/Library/Java/JavaVirtualMachines/corretto-18.0.2/Contents/Home mvn -pl fluss-filesystems/fluss-fs-hadoop -am -Dtest=HadoopDataInputStreamTest -DfailIfNoTests=false test
  • JAVA_HOME=/Users/cc/Library/Java/JavaVirtualMachines/corretto-18.0.2/Contents/Home mvn -pl fluss-filesystems/fluss-fs-hadoop -am -DfailIfNoTests=false test
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[filesystem] HadoopDataInputStream.seek loops forever past EOF

1 participant