Is your feature request related to a problem? Please describe.
Currently, the server instantiates all models on boot, notably it downloads the pretrained GPT-2 model from google cloud which weighs 500MiB. This caused the google server to crash and requires increasing its memory limit. It also slows down the startup time a lot.
Describe the solution you'd like
The server could lazily instantiate models when a client joins the task (the server is in charge of sending the base model to new participants).
In the case of the GPT-2 model, clients could potentially be in charge of downloading the model themselves (right now the server downloads it, serializes it and each clients downloads it from the server).
Is your feature request related to a problem? Please describe.
Currently, the server instantiates all models on boot, notably it downloads the pretrained GPT-2 model from google cloud which weighs 500MiB. This caused the google server to crash and requires increasing its memory limit. It also slows down the startup time a lot.
Describe the solution you'd like
The server could lazily instantiate models when a client joins the task (the server is in charge of sending the base model to new participants).
In the case of the GPT-2 model, clients could potentially be in charge of downloading the model themselves (right now the server downloads it, serializes it and each clients downloads it from the server).