Right now it is done on cpu by different workers.
Has a big impact on small graphs where normalization and transform is a big chunk of the total time, and where workers don't have time to prepare the next batches "in the background" when the GPU is busy doing inference.
Right now it is done on cpu by different workers.
Has a big impact on small graphs where normalization and transform is a big chunk of the total time, and where workers don't have time to prepare the next batches "in the background" when the GPU is busy doing inference.