Sample with unique PK generation

[wardi]

chromo definitions with edit_using__id: true will generate separate templates for create and update using the datastore's internal _id field to identify records being edited.

This allows direct updates to primary key values from excel and the form and prevents accidental overwriting of data from the reuse of a saved excel form (records can only be created once)

The _id field is normally a simple sequence per table, but the sample_pk_deluxe.yaml in this PR shows how to use a shared sequence and a bit of math to generate _id values that are:

• unique across all orgs and record types
• short and easy to type, if we ever ask users to enter them
• resistant to typos with added check digits

[JVickery-TBS]

@wardi okay so the datastore_create_sequence: sample_pk_deluxe_seq would create the sequence (via open-data/ckan#234) during the recombinant create-triggers command?

In the example schema you have:

NEW._id := nextval('sample_pk_deluxe_seq') * 100 + 99;
NEW._id := NEW._id - (NEW._id - 5) % 97;

is that just to show that you can modify the return of the sample_pk_deluxe_seq psql sequence? otherwise, if not provided, it would use the default auto increment _id right? (see idea comment for datastore_info field to set a sequence...maybe? open-data/ckan#234 (comment))

So in the terms of Canada stack, let's say we have Grants:

dataset_type: grants

datastore_create_sequence: grants_open_canada_id_seq

To my understanding, all org tables would use the same sequence right? And that IS what we want? Like in that case, they would be fully unique in the Recombinant Combined table right? So if TBS makes a Grants record, the record _id would be 1, and then if CRA makes one afterwards, that record would be _id 2? Do you foresee users questioning jumps in their sequence numbers (in the Registry)?

And if we want to get "fancy" with it we can concatenate to have some string formatted thing? e.g.

org_name text NOT NULL := (SELECT org_name FROM datastore_user LIMIT 1);
.....
NEW._id := format(
    'grants-%s-%s',
    org_name,
    nextval('grants_open_canada_id_seq')
);

(org_name capability coming from https://github.com/open-data/ckanext-canada/pull/1668/changes#diff-01ed71d02b0989f2bf56503bf31eab66be96a288626487db801964eaad4a9242R318)

then we would have "grants-tbs-sct-1" ...but _id is not a text column right? so that would not work? I think our argument for not having those silly formatted reference numbers of "--" is that we have date fields and values, and have the org identifier in another field too right? like we don't need users dissecting a reference number / PK for that info?

[JVickery-TBS]

@wardi when uploading Excel file, I get Example record on row 5 is missing

[wardi]

@JVickery-TBS sorry, forgot to push the last commit

[wardi]

The open_canada_id calculation I've suggested would make these ids not only unique across any PD type, but unique across the whole catalog if we are careful to always use a different mod 97 remainder value for each type of things (each sequence we create)

While we could change the type of the _id field to text we would then lose the sorting-by-record-created-time we get for free with incrementing integer values

If a client insists on a specially formatted ids like grants-tbs-sct-004321 or something then that's easier to implement in the filter script we use for publishing the data. We just prefix some 0's (to maintin sorting) and add the other info in one line of python, but keep the clean _ids in the DB