Syncing from kakfa and doing ETL should run as a continuous service so we don't need to poll and recompute progress state on each run.
Step 1 is to get from-kafka to provide a continuous service where it listens on each configured topic and syncs data as it arrives. There should be two parameters to drive commits: a data limit and a timeout. When data arrives but does not exceed the data limit, a timeout triggers processing.
Step 2 is to automate ETL based on from-kafka commits. Here the service is running continuously and whenever data arrives that could be consumed by an ETL, the logic is run automatically. This way, we don't need to run ETLs on a polling loop as they are run only when they have new data to process.
Syncing from kakfa and doing ETL should run as a continuous service so we don't need to poll and recompute progress state on each run.
Step 1 is to get
from-kafkato provide a continuous service where it listens on each configured topic and syncs data as it arrives. There should be two parameters to drive commits: a data limit and a timeout. When data arrives but does not exceed the data limit, a timeout triggers processing.Step 2 is to automate ETL based on from-kafka commits. Here the service is running continuously and whenever data arrives that could be consumed by an ETL, the logic is run automatically. This way, we don't need to run ETLs on a polling loop as they are run only when they have new data to process.