To test the service defined in #83, this issue is to create a performance test rig using large amounts of synthetic data. We will create a large number of data across thousands of topics and run these perf tests by hand. We can ingest the test data into confluent with a script and just leave the data in the kafka cloud service available for testing whenever we need it.
This will expose some issues with the built-in queries that zync does to track and update progress between the raw and staging pools. We need to make sure the efficiency of these queries is O(work to do) and never O(all data in pool). This may require adding some optimizations to the zed lake as we continue to improve the DAG planner and optimizer.
To test the service defined in #83, this issue is to create a performance test rig using large amounts of synthetic data. We will create a large number of data across thousands of topics and run these perf tests by hand. We can ingest the test data into confluent with a script and just leave the data in the kafka cloud service available for testing whenever we need it.
This will expose some issues with the built-in queries that zync does to track and update progress between the raw and staging pools. We need to make sure the efficiency of these queries is O(work to do) and never O(all data in pool). This may require adding some optimizations to the zed lake as we continue to improve the DAG planner and optimizer.