Feature Request: Support for Custom Type Projections in ShareDB

[csbenjamin]

Hello,

I'm currently working with a custom type in ShareDB and am keen on implementing projections for it. Upon reviewing the source code, I noticed that the functions within lib/projections.js appear to be tailored specifically for the json0 type.

To enhance the versatility of projections in ShareDB, would it be possible to add an option in the ShareDB constructor that permits users to provide custom functions tailored to their specific type operations?

Additionally, I'd like to clarify if the functions present in lib/projections.js are the sole methods that need adaptation for the successful integration of projections with custom types?

Thank you for considering this request.

[csbenjamin]

Additional Suggestion for Custom Type Projections Integration

Another approach might be to structure these functions as methods of the custom type itself, similar to how we currently implement the apply, transform, create, and compose methods. In fact, I believe this could be a more streamlined solution compared to passing these functions via the ShareDB constructor. It would also offer the added advantage of enabling projections for multiple types, with all associated logic encapsulated within the type implementation itself.

[alecgibson]

Hi 👋🏼 thanks for raising the issue. I can't see a reason to not add this in principle. Unfortunately, I think this is a huge task; as I'm sure you've seen, projection logic is scattered throughout the codebase and will need some work to tidy up into a more generically-usable shape.

I agree that the correct place for the logic is on the type itself; ideally sharedb core should be as type-agnostic as possible (somewhere where projections don't excel at all...).

If you want to start work on a PR, we can try to help you along the way.

[csbenjamin]

Hi,

Thank you for the prompt response and for considering the feature request. I appreciate your insights on the current state of the codebase.

As a temporary workaround, given that I'm using Webpack, I've overridden the projections.js file with my own implementation. It seems to be working well for now. Here's a snippet of my custom projections code for reference:

import {AOAction, AOActionTransformPaths} from './modules/backend/websocket/conexty-type';

exports.projectSnapshot = projectSnapshot;
exports.projectSnapshots = projectSnapshots;
exports.projectOp = projectOp;
exports.isSnapshotAllowed = isSnapshotAllowed;
exports.isOpAllowed = isOpAllowed;


// Project a snapshot in place to only include specified fields
function projectSnapshot(fields: {[key: string]: true}, snapshot: any) {
    // Only json0 supported right now
    // if (snapshot.type && snapshot.type !== json0.uri) {
    //     throw new Error(ERROR_CODE.ERR_TYPE_CANNOT_BE_PROJECTED, 'Cannot project snapshots of type ' + snapshot.type);
    // }
    snapshot.data = projectData(fields, snapshot.data);
}

function projectSnapshots(fields: {[key: string]: true}, snapshots: any) {
    for (let i = 0; i < snapshots.length; i++) {
        const snapshot = snapshots[i];
        projectSnapshot(fields, snapshot);
    }
}

function projectOp(fields: {[key: string]: true}, op: {create?: any, op?: AOAction[]}) {
    if (op.create) {
        projectSnapshot(fields, op.create);
    }
    if (op.op) {
        op.op = projectEdit(fields, op.op);
    }
}

function projectEdit(fields: {[key: string]: true}, op: AOAction[]) {
    // So, we know the op is a JSON op
    const result = [];

    for (let i = 0; i < op.length; i++) {
        const c = op[i];
        if (c.op !== 'transformPaths') {
            const path = c.path;

            if (path.length > 0) {
                // The path has a first element. Just check it against the fields.
                if (fields[path[0]]) {
                    result.push(c);
                }
            }
        } else {
            // Transform paths
            const paths = c.paths;
            const newPaths = [];
            for (let j = 0; j < paths.length; j++) {
                const p = paths[j];
                if (p.length > 0) {
                    if (fields[p[0]]) {
                        newPaths.push(p);
                    }
                }
            }
            if (newPaths.length > 0) {
                result.push({...c, paths: newPaths} as AOActionTransformPaths);
            }
        }
    }
    return result;
}

function isOpAllowed(knownType: any, fields: {[key: string]: true}, op: {create?: any, op?: AOAction[]}) {
    if (op.create) {
        return isSnapshotAllowed(fields, op.create);
    }
    if (op.op) {
        // if (knownType && knownType !== json0.uri) return false;
        return isEditAllowed(fields, op.op);
    }
    // Noop and del are both ok.
    return true;
}

// Basically, would the projected version of this data be the same as the original?
function isSnapshotAllowed(fields: {[key: string]: true}, snapshot: any) {
    // if (snapshot.type && snapshot.type !== json0.uri) {
    //     return false;
    // }
    if (snapshot.data == null) {
        return true;
    }
    // Data must be an object if not null
    if (typeof snapshot.data !== 'object' || Array.isArray(snapshot.data)) {
        return false;
    }
    for (const k in snapshot.data) {
        if (!fields[k]) return false;
    }
    return true;
}

function isEditAllowed(fields: {[key: string]: true}, op: AOAction[]) {
    for (let i = 0; i < op.length; i++) {
        const c = op[i];
        if (c.op !== 'transformPaths') {
            const path = c.path;
            if (path.length === 0) {
                return false;
            } else if (!fields[path[0]]) {
                return false;
            }
        } else {
            const paths = c.paths;
            for (let j = 0; j < paths.length; j++) {
                const p = paths[j];
                if (p.length === 0) {
                    return false;
                } else if (!fields[p[0]]) {
                    return false;
                }
            }
        }
    }
    return true;
}

function projectData(fields: {[key: string]: true}, data: any) {
    // Return back null or undefined
    if (data == null) {
        return data;
    }
    // If data is not an object, the projected version just looks like null.
    if (typeof data !== 'object' || Array.isArray(data)) {
        return null;
    }
    // Shallow copy of each field
    const result: any = {};
    for (const key in fields) {
        if (data.hasOwnProperty(key)) {
            result[key] = data[key];
        }
    }
    return result;
}

I plan to delve deeper into the ShareDB code to explore the possibility of implementing the proposed solution, aiming to move the logic to the type itself.

Again, thanks for your guidance. I'll keep you updated as I make progress and might reach out for assistance along the way.

Best regards.