Skip to content

Serialize GTS types to MessagePack and Protobuf #129

Description

@MikeFalcon77

GTS types serialize to JSON only. We want to send GTS entities as MessagePack and over gRPC. This can be added later on top of gts-macros without touching the JSON path.

Current state

  • #[struct_to_gts_schema] adds Serialize/Deserialize derives to base structs. Nested structs (base = Parent) can't implement serde traits directly. The macro generates GtsSerialize/GtsDeserialize for them instead (gts-macros/src/lib.rs, traits in gts/src/schema.rs).
  • Both traits are generic over serde::Serializer/serde::Deserializer. Nothing in the type signatures ties them to serde_json.
  • The generated gts_serialize calls serialize_struct. The generated gts_deserialize calls deserialize_struct and implements visit_map only, no visit_seq.
  • A nested type is serialized as an object under the base struct's generic field (payload). The macro doesn't use #[serde(flatten)].
  • Entity I/O in gts uses serde_json, plus serde-saphyr for YAML input in files_reader.rs. There are no protobuf or MessagePack dependencies.
  • Schemas are JSON Schema draft-07.

Using gts-rust with gRPC today

No changes to gts-rust are needed. Carry the JSON as bytes in a proto field:

message GtsEvent {
  string type_id = 1;
  bytes payload = 2; // JSON produced by serde_json
}

The sender serializes the full base struct, for example BaseEventV1<AuditPayloadV1<D>>, with serde_json::to_vec. A nested type can't be serialized on its own, so always send the base struct. type_id repeats the type field from the JSON so the receiver can route the message without parsing the payload.

A receiver that knows the Rust type calls serde_json::from_slice into it. A receiver that doesn't parses the payload into serde_json::Value and checks it with GtsStore::validate_payload(type_id, &value). The type's schemas must be registered in the store first.

This is the envelope option from Stage 2, done in application code.

Stage 1: MessagePack

MessagePack is a serde format, so the existing impls should work through rmp-serde without macro changes. This hasn't been tested.

One known problem: rmp-serde encodes structs as arrays by default. The generated deserializer has no visit_seq, so nested types won't decode in that mode. Either require map mode (Serializer::with_struct_map()) or generate visit_seq as well. Map mode keeps field names on the wire and matches the JSON shape. Array mode is smaller but depends on field order.

Work:

  • optional msgpack feature in gts with encode/decode helpers
  • round-trip tests over the existing inheritance fixtures: base, nested, generic payload, unit structs
  • pick map mode only, or add visit_seq

Stage 2: Protobuf / gRPC

Protobuf isn't a serde format. prost has its own Message trait and identifies fields by tag number, so the serde impls can't be reused. Two options.

Envelope. A fixed message with the serialized payload inside:

message GtsEntity {
  string type_id = 1;
  bytes payload = 2; // JSON or MessagePack
}

No macro changes. Payloads stay untyped on the wire, and clients can't use generated types for them.

Generated messages. A new macro attribute or derive emits a .proto message per GTS struct and From/TryFrom conversions to the prost types. gts-macros-cli would write the .proto files the same way it writes .schema.json today. Because nested types already serialize as a separate object under the generic field, each struct maps to its own message and the generic field maps to a message field.

Open questions for generated messages:

  • How to represent a generic field (payload: P): one message per concrete instantiation, a oneof over known child types, or google.protobuf.Any.
  • Where tag numbers come from. Deriving them from field order breaks when fields are reordered. An explicit per-field attribute is stable but adds annotations to every GTS struct.
  • How tag numbers stay stable across minor versions of a type.
  • Whether schema_evolution.rs should also check .proto compatibility, or whether JSON Schema compatibility is enough.

If a gRPC consumer needs this before generated messages are designed, ship the envelope first. It doesn't block the second option.

Out of scope

  • Changes to the JSON format or JSON Schema output.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions