[Feature] Support independent TTL configuration for local log segments

[gyang94]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Currently, table.log.ttl is used for both:

• Retaining log segments in remote storage.
• Retaining local log segments when tiered storage is enabled.

This couples the local and remote retention periods. In practice, users may want to retain data in remote storage for a long time while deleting already-uploaded local segments earlier to reduce local disk usage.

There is also an ambiguity around remoteLogEndOffset = -1. It may mean either:

1. No log segment has ever been copied to remote storage.
2. Log segments were copied previously, but all remote segments have since expired and been deleted.

Local segment cleanup must distinguish these cases because a local segment should only be deleted after it has been successfully copied to remote storage.

Solution

Introduce a new table configuration:

table.log.local-ttl

This configuration controls how long local log segments are retained after they have been copied to remote storage.

The expected behavior is:

• table.log.ttl continues to control remote log retention.
• table.log.local-ttl controls local log segment retention.
• If table.log.local-ttl is not configured, it defaults to table.log.ttl for backward compatibility.
• table.log.local-ttl must be less than or equal to table.log.ttl.
• Local segments that have not been copied to remote storage must not be deleted, even if their local TTL has expired.

Remote Offset Tracking

Track two different remote offsets:

• The current remote log end offset derived from the active remote log manifest.
• The highest offset that has ever been successfully copied to remote storage.

The highest copied offset should be persisted and remain monotonic even after all corresponding remote segments expire.

This allows local cleanup to determine whether a segment was previously uploaded without relying on the ambiguous remoteLogEndOffset = -1 value.

After a TabletServer or replica restart, the highest copied offset should be restored from persistent metadata.

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!