Feature: add-ons for using data (APIs and database backends)

[sc0ttj]

Add-ons for managing data and APIs

Component needs an easy way of importing, querying, saving, and syncing external data.

The benefits of easily managing external data:

• access large datasets
• auto updating data, triggers re-render
• querying data for custom results
• saving/retrieving persistent data
• syncing data between page users

External data could be:

• third party APIs
• "flat file" content (JSON files, etc)
• databases
• distributed P2P data stored or held by all users of a page

Add-ons related to external data handling could be:

• a useURL add-on, for URL (query string and hash path) data, based on sc0ttj/router
• a useAPI or useQuery add-on for accessing third party APIs/endpoints, based on fetch
• a useDatabase add-on for saving/retreiving component data to/from a persistent database backend

The add-ons should enable the following, with simple, concise code:

• use same code in Node and Browser
• fetch, parse and cache data
• handle login and auth stuff
• save data to a persistent store
• real-time updating of data (triggering page re-renders)
• syncing data for real-time interactions/collaboration between multiple users on a page

---

Other Libraries

Data from page URL (inc query strings and hash path):

• sc0ttj/router - steal code from it, to save state as URL query string

Fetching from third-party APIs:

• videojs/xhr - nice XHR wrapper
• donavon/thwack - axios like API, 1.5kb, isomorphic, very nice
• tannerlinsley/react-query
• shaolinmkz/useQueryParam
• lquixada/cross-fetch - isomorphic, standards-compliant fetch replacement
• developit/redaxios - 800 byte axios alternative (nice API)
• Swizec/better-fetch - tiny wrapper around fetch "to make it easier"
• axross/ffetch - nice API, works in Node and browser
• tiaanduplessis/tenacious-fetch - 600 byte fetch wrapper supports retries and timeouts
• marionebl/omni-fetch isomorphic fetch
• matthew-andrews/isomorphic-fetch
• kokororin/honoka - another isomorphic fetch
• typicode/fetchival - 0.5kb min/gz, fetch wrapper, browser (IE9+) and Node
• developit/unfetch - 500b fetch polyfill

Persistent data stores, with P2P data syncing:

• PouchDB
  • cdaringe/pouchy - syntactic sugar frontend for pouchdb
  • garbados/comdb - encrypt/decrypt pouchdb data
  • UXtemple/redux-pouch - binds redux state updates to pouchdb updates
  • glynbird/couchshell - interact with couchdb as if it were a UNIX filesystem
• IPFS
  • orbitdb/orbit-db - looks great in general, IPFS p2p db
  • ipfs-shipyard/tevere - decentralized db over IPFS
  • dmotz/trystero - "makes any webpage multiplayer", supports BitTorrent, IPFS, Firebase
• Supabase
  • supabase - Postgres backend, Firebase alternative, see supabase/supabase-js

Persistent data stores:

• ipfs-mini JS client - uses IPFS, 2kb (minified, not gzipped)

Non-persistent data stores, with P2P data syncing:

• amark/gun - 9kb, encrypted, secure, decentralized, p2p db
• mafintosh/hyperdb - Distributed scalable database, uses with karissa/hyperdiscovery for syncing

Others:

• hyperswarm/hyperswarm - join p2p swarms, by topic (not sure about persistence/db/saving)

"In-memory" only databases (no backend, no collab):

• typicaljoe/taffydb - 1kb, very nice API, works in browsers, Node, Electron
• lowdb/lowdb - tiny, lightweight, works in browsers, Node, Electron
• TomPrograms/stormdb - 1.1kb, works in browsers, Node, Electron
• nodeGame/NDDB - 60kb, nice API, lots of features, works in browser and Node, can use localStorage

Reading

• promise-api - specifically, using Promise.all to resolve multiple requests
• StackOverflow-return-fetch-results-from-a-function
• https://dev.to/sidv93/cleaner-data-fetching-with-react-query-4klg
• https://dev.to/sruhleder/using-react-query-with-supabase-a03

[sc0ttj]

Ideal code examples:

useURL would be something like:

const data = useURL(
  // give the URL to parse
  'https://site.com/foo/bar.html?var=true#/profile/1'),
  // optionally give an array of "routes", to shape the resulting data how you want it
  [ '/profile/:userId', '/page/:pageId']
);

console.log(data); 
// {
//     protocol: 'https',
//     domain: 'site.com',
//     port: '',
//     path: '/foo/',
//     page: 'bar.html',
//     var: true,
//     profile: {
//       userId: 1,
//    },
// }

the useAPI add-on would be something like:

const opts = { ... };

const products = await useAPI('https://some.com/api/products', opts);
products.map(product => console.log(product)); // output JSON of each product

// ..or

useAPI('https://some.com/api/products', opts)
.then(products => {
  products.map(product => console.log(product)); // output JSON of each product
});

a database backend would be something like:

const db = useDatabase({
  persistent: true,                          // refresh the page and the data persists
  collaborative: true,                       // all users on page can read/write to the backend
  realtime: true,                            // update (and re-render) page whenever the db changes
  // the data source (choose one of these)
  source: 'ipfs://oijwr9i8joisdnjfownf0nf',  // use IPFS as storage
  source: 'magnet://aefs46ad465we4e323es',   // bitTorrent protocol as a backend, see dmotz/tystero
  source: 'https://some.com/mydb.json',      // read/write to JSON file on a [local or remote] web server
  source: 'pouchdb://some.com/mydb',         // in-memory first, syncs to remote
  source: 'couchdb://some.com/mydb',         // always uses remote
  source: 'supabase://some.com/mydb',        // a remote supabase db
  source: 'localStorage',                    // in-memory only
  backup: 'couchbdb://some.com/mydb2',       // auto backup your database some where else
});

const name = 'Ted';
const details = db.get(`SELECT * FROM users WHERE name = ${name}`);

// ...and usage something like this 

db.add( ... ).then(res => { ... })

db.update( ... ).then(res => { ... })

db.delete( ... ).then(res => { ... })

db.get( ... ).then(data => { ... })

db.query( ... ).then(data => { ... })

db.getJoin( ..., ... ).then(data => { ... })

[sc0ttj]

https://github.com/mozilla-b2g/gaia/blob/master/shared/js/async_storage.js

• defines an asynchronous version of the localStorage API, backed by an IndexedDB database.
• It creates a global asyncStorage object that has methods like the localStorage object.