Add some form of caching to improve the performance of intensive file system reads including:
- File Explorer - which has to recursively read all directories
- Views (sources) - which has to recursively read all directories and many files to load data
Desktop has no noticeable performance issues right now, but on mobile with my existing personal vault the file explorer took around 3-5 seconds to load and a view source searching all markdown files took 10+ seconds. These are not feasible numbers to be able to even use the app right now.
I'm not 100% sure on where the performance issues are, however my guess is that passing data between the native mobile and web layers in Capacitor is probably slow, and because I'm recursively loading directories via separate Filesystem.readdir calls this makes it much worse... before even then loading each file contents too for views.
Solutions
Improve native integrations
This is likely the best option, but also the most complex and time consuming. If I could implement my own native code for performing file system actions such as tree and glob I assume these would be much more efficient.
Long term this should likely be done regardless of other options, but maybe isn't the best quick short-term fix for progressing the app.
Managed "index" cache for quick querying
Create a dedicated "index" which could be used for file system queries, rather than relying on slower file system calls.
This could allow quick file system reads in all aspects, but comes with complexity costs such as storage management (is all data indexed, is that feasible for big vaults etc), and synchronization/invalidation issues with files changing.
It also removes the "file system as the single source of truth" behavior of the app which is a massive advantage.
Result caching in vault
Rather than managing a full "index" in memory or similar, instead the results of expensive calls could be individually cached as files within the vault itself (/.headbase/cache/). These could then be synced alongside the files to all devices.
Devices could then choose if they want to read from the cache files (if present) and if they want to update them after running actions.
A device would load an expensive result (like a file system data source), create a cache file, and return the results.
On the next call, the cache file contents could be returned rather than performing the query.
This could still lead to stale results like other caching mechanisms, but this could be handled via a mix of automatic re-validation (long term) and exposing cache info to users such as when the cache was created and on what device, and giving users the ability to reload the given query.
Add some form of caching to improve the performance of intensive file system reads including:
Desktop has no noticeable performance issues right now, but on mobile with my existing personal vault the file explorer took around 3-5 seconds to load and a view source searching all markdown files took 10+ seconds. These are not feasible numbers to be able to even use the app right now.
I'm not 100% sure on where the performance issues are, however my guess is that passing data between the native mobile and web layers in Capacitor is probably slow, and because I'm recursively loading directories via separate
Filesystem.readdircalls this makes it much worse... before even then loading each file contents too for views.Solutions
Improve native integrations
This is likely the best option, but also the most complex and time consuming. If I could implement my own native code for performing file system actions such as tree and glob I assume these would be much more efficient.
Long term this should likely be done regardless of other options, but maybe isn't the best quick short-term fix for progressing the app.
Managed "index" cache for quick querying
Create a dedicated "index" which could be used for file system queries, rather than relying on slower file system calls.
This could allow quick file system reads in all aspects, but comes with complexity costs such as storage management (is all data indexed, is that feasible for big vaults etc), and synchronization/invalidation issues with files changing.
It also removes the "file system as the single source of truth" behavior of the app which is a massive advantage.
Result caching in vault
Rather than managing a full "index" in memory or similar, instead the results of expensive calls could be individually cached as files within the vault itself (
/.headbase/cache/). These could then be synced alongside the files to all devices.Devices could then choose if they want to read from the cache files (if present) and if they want to update them after running actions.
A device would load an expensive result (like a file system data source), create a cache file, and return the results.
On the next call, the cache file contents could be returned rather than performing the query.
This could still lead to stale results like other caching mechanisms, but this could be handled via a mix of automatic re-validation (long term) and exposing cache info to users such as when the cache was created and on what device, and giving users the ability to reload the given query.