Idea
Add an option to unload the built-in model after X minutes of inactivity.
Today the built-in model stays loaded while Cai runs (~1.9 GB for the default Ministral 3B) so Option+C actions respond instantly (see #37). For setups where that idle memory matters, Cai could free it after a period of no use and reload it on the next action.
Trade-off
Frees ~2 GB while idle, at the cost of a few seconds of reload delay on the first action after sleep.
Rough shape
- Setting in Settings → AI: off by default, with a configurable timeout (e.g. 5 / 15 / 30 min).
- Only applies to the Built-in provider; external providers hold no model in memory.
If this trade-off would help you, give it a 👍 or comment with your setup. Happy to implement it for the next release.
Idea
Add an option to unload the built-in model after X minutes of inactivity.
Today the built-in model stays loaded while Cai runs (~1.9 GB for the default Ministral 3B) so Option+C actions respond instantly (see #37). For setups where that idle memory matters, Cai could free it after a period of no use and reload it on the next action.
Trade-off
Frees ~2 GB while idle, at the cost of a few seconds of reload delay on the first action after sleep.
Rough shape
If this trade-off would help you, give it a 👍 or comment with your setup. Happy to implement it for the next release.