Thoughts
Because a standard Kinesis stream uses four shards by default, and because we have been able to work around a serialization order flaw per #101 (comment), it would be nice to consume events from those four shards (or more) in parallel, to speed up data loading.
It looks like the async-kinesis package we are using for tapping into a Kinesis stream works well for this purpose of coordinating multiple consumers.
Thoughts
Because a standard Kinesis stream uses four shards by default, and because we have been able to work around a serialization order flaw per #101 (comment), it would be nice to consume events from those four shards (or more) in parallel, to speed up data loading.
It looks like the async-kinesis package we are using for tapping into a Kinesis stream works well for this purpose of coordinating multiple consumers.
async-kinesisto resume properly after processor restarts? cratedb-toolkit#475