[FEATURE] Storing OpenTelemetry map<string, AnyValue> fields in OpenSearch

[penghuo]

1. Problems Statements

In OpenTelemetry, the fields that carry the diagnostic signal — a log or span's attributes, and
the resource and scope attributes attached to them — are all typed map<string, AnyValue>: an
application-defined map from string keys to values of any type, whose set of keys keeps emerging.
attributes is only the most prominent such field; the problem is general to any
map<string, AnyValue> field. Storing this data has only two fundamental difficulties: its value
types change from one producer to the next and cannot be stored by unifying them into one type,
and its number of distinct keys has no upper bound.

1.1. No type unification — one path carries many types

The same path can be a different type in different documents: status is sometimes 200 and
sometimes "OK"; duration is sometimes 1.5 and sometimes "1.5s". This is an inevitable
product of multi-language SDKs, version skew, and the fact that JSON itself does not distinguish int
from float — not a bug that can be fixed on the application side. The right goal is to preserve
every type, not to force them into a single least-common type.

1.2. Unbounded paths — the number of distinct keys has no ceiling

The set of attribute keys is decided by applications and keeps growing: thousands of keys, changing
daily, plus unbounded-cardinality paths like k8s.labels.<anything> and
http.request.header.<anything>, and keys with IDs embedded in their names. Any layout that gives
each path its own column must therefore either cap how many columns it will create or pay a metadata
cost that grows without bound.

---

1.3. How OpenSearch stores this today

Every current option resolves one axis by giving up the other, or gives up the type entirely. Take
the type conflict first: dynamic mapping infers a field's type from the first value it sees and
freezes it, so when service A writes status: 200 (inferred as long) and service B later writes
status: "OK", service B's entire log line is rejected — a silent loss whose symptom ("service B's
logs are missing") is far removed from its cause ("what some other service wrote earlier"), across
teams and across time. The alternatives trade that away in different directions:

Approach	Avoids type-conflict data loss?	Bounds the field count?	Limitations
Dynamic mapping (typed fields)	No	No	A document whose type differs from the first value seen is rejected in full; every new key also adds a mapping entry that the cluster manager must broadcast to every node.
ignore_malformed	Partly	No	The conflicting value is silently dropped from the index — it survives only in _source, where it cannot be queried; the field count still grows without bound.
Map every field to keyword	Yes	No	Every value is stored as a string, so the type is gone: no numeric aggregation and no numeric range. Each key is still its own mapping entry, so the mapping grows without bound.
flat_object	Yes	Yes	The whole subtree becomes one field and every value a string, so numeric aggregation and numeric range are impossible; the paths inside are invisible to the field browser, and it cannot feed a star-tree.
Derived / runtime fields	Not applicable (reads _source)	Not applicable	The value is cast from _source by a per-document script at query time — correct, but with no doc_values or star-tree behind it, it does not scale to dashboards.
Normalize types at ingest	Yes	No	Works only for keys you can predict and configure ahead of time — which unpredictable, application-defined attributes are not.
dynamic: false	Yes (kept only in _source)	Yes	Dynamic fields are stored but not indexed, so they cannot be filtered, aggregated, or searched at all.

2. Expected Behavior: variant access over a flat_object field

2.1. Creating the index

Map the fields that hold map<string, AnyValue> as flat_object.

PUT otel-logs
{
  "mappings": {
    "properties": {
      "@timestamp":   { "type": "date"        },
      "severityText": { "type": "keyword"     },
      "body":         { "type": "text"        },
      "resource":     { "type": "flat_object" },
      "attributes":   { "type": "flat_object" }
    }
  }
}

That is the whole mapping. flat_object accepts no parameters — there is nothing else to
configure. Whatever keys arrive under resource and attributes are discovered from the data.

Two things to know before you start:

• flat_object contributes one entry to the mapping, no matter how many keys arrive. This is what
  keeps an unbounded key set from growing the mapping.

OTel keys contain dots — http.status_code, service.name. flat_object addresses a leaf by its full
dotted path, so a key literally named "http.status_code" and a nested object {"http":{"status_code":…}}
are both reached as attributes.http.status_code. You do not need to know which shape the producer sent.

2.2. Indexing documents

Two log records from two services. They disagree about the type of attributes.http.status_code and
about which keys exist.

POST otel-logs/_bulk
{"index":{"_id":"1"}}
{"@timestamp":"2026-08-21T10:00:00Z","severityText":"INFO","body":"checkout completed",
 "resource":{"service.name":"checkout"},
 "attributes":{"http.status_code":200,"duration_ms":12.5,"retry_count":0}}
{"index":{"_id":"2"}}
{"@timestamp":"2026-08-21T10:00:01Z","severityText":"ERROR","body":"upstream unavailable",
 "resource":{"service.name":"payments"},
 "attributes":{"http.status_code":"503","duration_ms":4,"error":"timeout"}}

Both documents are indexed. What happens:

http.status_code is 200 in doc 1 and "503" in doc 2	both preserved as written — one a number, one a string
duration_ms is 12.5 in doc 1 and 4 in doc 2	both preserved as written; 12.5 does not become 12
retry_count exists only in doc 1, error only in doc 2	no effect on each other
The mapping after indexing	unchanged — still 5 fields

Nothing here is a special case. A document is never rejected and a value is never altered because
another document used a different type for the same path. That is the property variant access
depends on.

---

2.3. Querying

A path under a flat_object field has PPL type variant. The type of the value is per document.

3.1 The common case needs no types

Paths are written the way you would write any field. The type comes from the operator.

source=otel-logs | where attributes.http.status_code = 200
source=otel-logs | where attributes.duration_ms > 10
source=otel-logs | stats avg(attributes.duration_ms) by resource.service.name
source=otel-logs | sort attributes.duration_ms

Query	Result
where attributes.http.status_code = 200	doc 1
where attributes.http.status_code = 503	doc 2 — "503" converts to 503
where attributes.duration_ms > 10	doc 1
stats avg(attributes.duration_ms)	8.25
stats count() by attributes.http.status_code	200 → 1, "503" → 1
stats count() by resource.service.name	checkout → 1, payments → 1
where isnotnull(attributes.retry_count)	doc 1

3.2 How the operators behave

Operator	What it does	Values of another type
+, -, *	converts each value to a number	become NULL
sum, avg, min, max	converts each value to a number	excluded, and counted (§3.4)
=, >, <, between	converts each value to the literal's type, then compares	do not match
sort by	orders across types: null < string < number < boolean < array < object	every row is placed
by (group by), count(distinct)	one group per distinct value; type is part of the value	own group per type
isnull, isnotnull	tests whether the path is present	—
like, regex	converts each value to a string	—

Two consequences worth knowing:

• Numbers of different widths are one value. 200 and 200.0 are equal, group together, and
  compare numerically — 9 < 1000, never "9" > "10".
• = is more forgiving than by. where http.status_code = 503 finds doc 2 even though the value
  is the string "503", but by http.status_code keeps 200 and "503" in separate groups — so a
  group-by shows you when a path holds mixed types.

3.3 When you want control: variant_get and try_variant_get

The operators above silently skip what they cannot convert. When you want to choose that behavior
yourself, name the type:

	Value does not fit the type	Path is absent
variant_get(v, path, type)	error	NULL
try_variant_get(v, path, type)	NULL	NULL
path::type — shorthand for try_variant_get	NULL	NULL

Use variant_get when a wrong type is something you want to hear about. Use try_variant_get (or
::) when dirty rows should be dropped quietly.

Expression	doc 1	doc 2
variant_type(attributes.http.status_code)	number	string
attributes.http.status_code::long	200	503
variant_get(attributes, '$.duration_ms', 'long')	error — 12.5 would lose the fraction	4
try_variant_get(attributes, '$.duration_ms', 'long')	NULL	4
variant_get(attributes, '$.duration_ms', 'double')	12.5	4
variant_get(attributes, '$.error', 'long')	NULL — absent	error — "timeout" is not a number

variant_type(path) tells you what a value actually is, and returns NULL when the path is absent.
Grouping by it shows how a path is split:

source=otel-logs | stats count() by variant_type(attributes.http.status_code)
-- number → 1, string → 1

12.5 requested as long never returns 12. A narrowing that loses information is an error, or a
NULL with try_variant_get — never a quietly changed number.

3.4 Aggregations report what they skipped

An aggregation that converts values reports two counts with its result:

• coerced — the type differed but the value converted exactly, e.g. "503" → 503
• excluded — the value could not be converted, e.g. "timeout"

source=otel-logs | stats sum(attributes.http.status_code)
-- 703 · coerced 1 · excluded 0

Converting numeric strings is on by default. Turning it off moves those rows from coerced to
excluded and changes the result — the same query would return 200 · coerced 0 · excluded 1.

[ps48]

Related issues:

• Data prepper ingestion failure due to mapping conflicts - [BUG] Otel Trace Source - Field Mapping Conflict for OTel Attributes with Dotted Keys data-prepper#5616
• Update the OTel mappings for OpenSearch - [FEATURE] Second look at OTel logs and traces schema technical-steering#116