Json decoder factory by hareshkh · Pull Request #10670 · apache/arrow-rs

hareshkh · 2026-08-12T23:27:39Z

Which issue does this PR close?

Related: Allow extensions to arrow-json decoder and include an extension for variant #9021, [WIP] Json decoder factory v2 #9272 (prior art, both went stale); a step toward [arrow-json] deserialize Variant fields #8987 but does not close it.
Closes Support custom decoders in the JSON reader (DecoderFactory, reader counterpart of EncoderFactory) #10739

Rationale for this change

The JSON writer has been extensible since Add hooks to json encoder to override default encoding or add support for unsupported types #7015 (EncoderFactory, plus a public make_encoder to delegate to defaults). The reader has no equivalent.
Small now because [WIP] Json decoder factory v2 #9272's prefactors already merged (Move extension type construction logic out of Field #9266, make_decoder accepts borrowed DataType instead of owned #9270, Factor out json reader's static make_decoder args to a struct #9271): DecoderContext already sits on main.

What changes are included in this PR?

TapeElement #[non_exhaustive], documenting numbers as Number (JSON text) or native i32/i64/f32/f64 (serde path), 64-bit spanning two elements.
ArrayDecoder public; pos holds one tape index per output row.
New DecoderFactory, consulted before the reader's dispatch; Ok(None) accepts the default.
DecoderContext::make_decoder public, so a factory can delegate to the decoder the reader would otherwise use - without it, overriding a nested type means reimplementing its children (@scovich's point on Allow extensions to arrow-json decoder and include an extension for variant #9021).
ReaderBuilder::with_decoder_factory.

Are these changes tested?

Yes. Unit tests and a doctest decoding Binary from a JSON int array - the inverse of the existing EncoderFactory doctest, so the two round-trip, no new dependency.

Are there any user-facing changes?

New: arrow_json::{Tape, TapeElement, ArrayDecoder, DecoderFactory}, DecoderContext::{make_decoder, decoder_factory}, ReaderBuilder::with_decoder_factory.

alamb · 2026-08-18T17:52:41Z

Thanks @hareshkh -- this looks pretty good, though it is still marked as draft

I filed a ticket to track this work

Support custom decoders in the JSON reader (DecoderFactory, reader counterpart of EncoderFactory) #10739

hareshkh · 2026-08-19T18:15:36Z

Thanks @alamb! Can I please have a review now?

# Which issue does this PR close? N/A -- documentation only. # Rationale for this change The ability to customize JSON encoding via [`EncoderFactory`] was added in #7015, but neither the crate-level docs nor the writer module docs mention it, so users only find the hook by stumbling on `WriterBuilder::with_encoder_factory`. [`EncoderFactory`]: https://docs.rs/arrow-json/latest/arrow_json/trait.EncoderFactory.html # What changes are included in this PR? Documentation only, no code changes: - Crate-level docs: list a custom `EncoderFactory` as an alternative for binary data encoding - Misc other wording fixeers It would be nice to add a matching pointer for customizing *decoding* as part of #10670 # Are these changes tested? Covered by existing doc tests and CI rustdoc link checking. # Are there any user-facing changes? Documentation only.

…10741) # Which issue does this PR close? N/A -- documentation only. # Rationale for this change The ability to customize JSON encoding via [`EncoderFactory`] was added in apache#7015, but neither the crate-level docs nor the writer module docs mention it, so users only find the hook by stumbling on `WriterBuilder::with_encoder_factory`. [`EncoderFactory`]: https://docs.rs/arrow-json/latest/arrow_json/trait.EncoderFactory.html # What changes are included in this PR? Documentation only, no code changes: - Crate-level docs: list a custom `EncoderFactory` as an alternative for binary data encoding - Misc other wording fixeers It would be nice to add a matching pointer for customizing *decoding* as part of apache#10670 # Are these changes tested? Covered by existing doc tests and CI rustdoc link checking. # Are there any user-facing changes? Documentation only.

alamb

Thanks @hareshkh -- I left some comments. Let me know if they make sense

alamb · 2026-08-27T19:26:06Z

+    fn make_custom_decoder(
+        &self,
+        _ctx: &DecoderContext,
+        _data_type: &DataType,


Rather than pass in the fields of a DataType, how about just pass in the &Field reference directly:

That would also allow the decoder to key off Metadata (where the extension type information is stored)

@@ -861,17 +864,24 @@ pub trait ArrayDecoder: Send { /// /// [`EncoderFactory`]: crate::EncoderFactory pub trait DecoderFactory: std::fmt::Debug + Send + Sync { - /// Make a decoder for `data_type`, or `Ok(None)` to use the reader's default. + /// Make a decoder for `field`, or `Ok(None)` to use the reader's default. + /// + /// Receives the [`FieldRef`] rather than just its [`DataType`] so decoder + /// selection can consider the field's metadata, e.g. to identify [extension + /// types]. The root of a [`ReaderBuilder::new`] schema is presented as a + /// synthesized nameless `Struct` field. /// /// Use [`DecoderContext::make_decoder`] on `ctx` to build child decoders. Calling - /// it for the `data_type` this was invoked with recurses back here and loops. + /// it with the field this was invoked with recurses back here and loops. /// - /// `is_nullable` folds in ancestor nullability, so it may be `true` even where the - /// corresponding field is not. + /// `is_nullable` folds in ancestor nullability, so it may be `true` even when + /// `field.is_nullable()` is not. + /// + /// [extension types]: https://arrow.apache.org/docs/format/Columnar.html#extension-types fn make_custom_decoder( &self, _ctx: &DecoderContext, - _data_type: &DataType, + _field: &FieldRef, _is_nullable: bool, ) -> Result<Option<Box<dyn ArrayDecoder>>, ArrowError> { Ok(None)

Done, however is_nullable can be different from field.is_nullable so keeping that around. But this will still unlock metadata based dispatch which can be useful.

alamb · 2026-08-27T19:28:01Z

-    fn make_decoder(
+    /// The standard way to create child decoders from within a decoder, and how a
+    /// [`DecoderFactory`] delegates to the decoder the reader would otherwise use.
+    /// The factory is consulted first, so calling this for the same data type the


this seems like a non trivial foot gun and would be nice to have a way to get the standard decoder from within the factor (e.g. to delegate to the default decoders)

Could we add some method like make_default_decoder that bypasses the Decoder factor?

Added make_builtin_decoder, please let me know if naming looks off

alamb · 2026-08-27T19:32:27Z

+    ///
+    /// `is_nullable` folds in ancestor nullability, so it may be `true` even where the
+    /// corresponding field is not.
+    fn make_custom_decoder(


It would be nice to make this name symmetric with the writer side -

arrow-rs/arrow-json/src/writer/encoder.rs

Line 251 in 4cc296b

fn make_default_encoder<'a>(

So something like

fn make_default_decoder(

alamb · 2026-08-27T19:33:11Z

I think this is pretty close

…10741) # Which issue does this PR close? N/A -- documentation only. # Rationale for this change The ability to customize JSON encoding via [`EncoderFactory`] was added in apache#7015, but neither the crate-level docs nor the writer module docs mention it, so users only find the hook by stumbling on `WriterBuilder::with_encoder_factory`. [`EncoderFactory`]: https://docs.rs/arrow-json/latest/arrow_json/trait.EncoderFactory.html # What changes are included in this PR? Documentation only, no code changes: - Crate-level docs: list a custom `EncoderFactory` as an alternative for binary data encoding - Misc other wording fixeers It would be nice to add a matching pointer for customizing *decoding* as part of apache#10670 # Are these changes tested? Covered by existing doc tests and CI rustdoc link checking. # Are there any user-facing changes? Documentation only.

Preparatory work for allowing custom decoders in the arrow-json reader. A custom `ArrayDecoder` necessarily reads the tape, so `Tape` and `TapeElement` have to be public for such a hook to be usable. Rather than making `mod tape` public, this re-exports just the two types that a decoder needs. `TapeDecoder` stays private, so downstream code can read a `Tape` it is handed but cannot construct one, keeping the tape's production an implementation detail. `TapeElement` is marked `#[non_exhaustive]`. Its representation is an implementation detail — 64-bit values are split across two consecutive elements, and the type's own docs note that offsets may become a custom `u56` type — so downstream matches need a wildcard arm for that to remain a non-breaking change. Also documents two aspects of the tape that become public contract: string data is copied into the tape with escapes resolved, and numbers appear either as `Number` (parsing JSON text) or as the native `I32`/`I64`/`F32`/`F64` variants (serializing Rust values), so decoders must handle both.

The JSON writer has supported overriding how a type is encoded since apache#7015 via `EncoderFactory`. The reader has had no equivalent, so anything the built-in decoders don't do — a different binary encoding, an extension type — requires forking the crate. This adds the reader-side counterpart: * `ArrayDecoder` is now public, so callers can implement a decoder. * `DecoderFactory` is consulted for every data type before the reader's own dispatch, and returns `Ok(None)` to accept the default. * `DecoderContext::make_decoder` is now public, so a factory can delegate to the decoder the reader would otherwise have used. Without this, overriding anything nested would mean reimplementing all of its children. * `ReaderBuilder::with_decoder_factory` registers a factory. Prior art: apache#9021 and apache#9272, both of which went stale.

hareshkh · 2026-08-28T17:38:24Z

+struct CheckedDecoder {
+    inner: Box<dyn ArrayDecoder>,
+    data_type: DataType,
+}


@alamb: This is net new from the previous pass and adds validation to custom decoders - what are your thoughts on this?

I think it is a good idea to check that the array type returned is what was declared

hareshkh · 2026-08-28T17:39:29Z

Thanks for the review @alamb! This should be ready for another look.

alamb

Thanks @hareshkh

I went over this again and I think the code parts look good to me.

I have some suggestions on the examples and the documentation that I think are worth considering, but we could also handle them after merging

Note I will be away for the next week so I may be slow to respond

alamb · 2026-08-30T10:32:11Z

+/// Decodes `Binary` from a JSON array of integers rather than the default hex string.
+///
+/// ```
+/// use std::sync::Arc;


I recommend we prefix the use statements with # so that they are hidden in the documentation rendering

E.g instead of

/// use arrow_array::{Array, ArrayRef, BinaryArray};

like

/// # use arrow_array::{Array, ArrayRef, BinaryArray};

alamb · 2026-08-30T10:35:21Z

+///
+/// # Examples
+///
+/// Decodes `Binary` from a JSON array of integers rather than the default hex string.


I think this example does a bunch of other stuff too

To keep the example simpler, I reccomend splitting it up -- one example that shows how to decode Binary and then one example that shows how to implement the SHOUT extension type

That will be more lines overall as the setup mut be shared, but splitting the concepts I think will make it easier to understand as those are two separate usecases

alamb · 2026-08-30T10:36:25Z

+pub trait DecoderFactory: std::fmt::Debug + Send + Sync {
+    /// Make a decoder for `field`, or `Ok(None)` to use the reader's default.
+    ///
+    /// Receives the [`FieldRef`] rather than just its [`DataType`] so decoder


I think the fact that Field ref is passed in is obvious from the code -- maybe we can here just focus on "Note: the extension metadata can be found from the field argument

alamb · 2026-08-30T10:37:13Z

+    /// Use [`DecoderContext::make_decoder`] on `ctx` to build child decoders, and
+    /// [`DecoderContext::make_builtin_decoder`] to build on the reader's own decoder
+    /// for this field. Calling `make_decoder` with the field this was invoked with
+    /// recurses back here and loops.


maybe mention "causes infinite recursion and stack overflow"?

Maybe as a follow on PR we could find some simple way to detect this case and error (rather than stack overflow)

alamb · 2026-08-30T10:38:20Z

+struct CheckedDecoder {
+    inner: Box<dyn ArrayDecoder>,
+    data_type: DataType,
+}


I think it is a good idea to check that the array type returned is what was declared

alamb · 2026-08-30T10:39:07Z

        }
    }
+
+    /// Declining everything must be indistinguishable from no factory at all.


what is the value of this test (I wonder what type of error it would catch)?

Maybe it is worth just removing

alamb · 2026-08-30T10:39:48Z

 /// iteration may increase this to a custom `u56` type.
 ///
+/// Numbers take more than one form, and matches must handle all of them. Parsing JSON
+/// text always yields [`Self::Number`], holding the value textually (read via


what does "holding the value "textually" mean?

The detail here about calling Tape::get_string and Decoder::serialize seems too much and not in the right place

I think the point of this comment is supposed to be "Numbers take more than one form, so your decoder must handle all of them. For example, JSON text yields Self::Number, but deserializing Rust values yields Self::I64, ...."

The other details just makes this harder for me to understand.

If you want to document that Tape::ge_string should be used to get the value from Self::number , it should be documented on Self::number I would think.

alamb · 2026-08-30T10:43:14Z

 /// This approach to decoding JSON is inspired by [simdjson]
 ///
+/// String data is copied into the tape with escapes resolved, so [`Tape::get_string`]
+/// borrows from the tape, not the input. A `Tape` is read, never constructed.


maybe point out that a Tape is only constructed internally to the crate, and is exposed to consumers to read, but that other crates can not construct this

alamb · 2026-09-01T18:15:00Z

run benchmark json_reader

alamb

Thank you @hareshkh

I went over this again and it looks good. I just kicked off a benchmark run. Assuming that looks good I think we are ready to merge this

alamb · 2026-09-01T18:15:39Z

run benchmark json_reader

adriangbot · 2026-09-01T18:17:37Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5498354530-2062-4khw4 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing json-decoder-factory (09cb374) to 27f68b1 (merge-base) diff

Run configuration

run benchmark json_reader

BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench json_reader
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-01T18:18:45Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5498362145-2063-bvjxw 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing json-decoder-factory (09cb374) to 27f68b1 (merge-base) diff

Run configuration

run benchmark json_reader

BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench json_reader
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-01T18:34:04Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing json-decoder-factory (09cb374) to 27f68b1 (merge-base) diff

Run configuration

run benchmark json_reader

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        json-decoder-factory                   main
-----                                        --------------------                   ----
decode_binary_hex_json                       1.00     14.0±0.02ms        ? ?/sec    1.00     14.0±0.03ms        ? ?/sec
decode_binary_view_hex_json                  1.00     14.5±0.02ms        ? ?/sec    1.00     14.5±0.04ms        ? ?/sec
decode_fixed_binary_hex_json                 1.00     14.0±0.03ms        ? ?/sec    1.00     14.1±0.03ms        ? ?/sec
decode_list_long_json/131072                 1.01    305.8±1.31ms   256.1 MB/sec    1.00    303.8±1.41ms   257.7 MB/sec
decode_list_long_serialize                   1.00    204.1±1.25ms        ? ?/sec    1.00    203.6±0.86ms        ? ?/sec
decode_list_short_json/131072                1.01     20.6±0.12ms   253.3 MB/sec    1.00     20.5±0.08ms   255.0 MB/sec
decode_list_short_serialize                  1.00     11.5±0.09ms        ? ?/sec    1.01     11.6±0.08ms        ? ?/sec
decode_list_view_long_json/131072            1.01    306.1±1.08ms   255.8 MB/sec    1.00    303.5±1.42ms   258.0 MB/sec
decode_list_view_long_serialize              1.01    205.8±0.90ms        ? ?/sec    1.00    204.3±1.11ms        ? ?/sec
decode_list_view_short_json/131072           1.00     20.5±0.06ms   254.8 MB/sec    1.01     20.7±0.15ms   252.6 MB/sec
decode_list_view_short_serialize             1.00     11.8±0.09ms        ? ?/sec    1.01     11.8±0.10ms        ? ?/sec
decode_map_large_json/131072                 1.00    266.0±0.97ms   285.6 MB/sec    1.01    269.1±0.47ms   282.3 MB/sec
decode_map_large_serialize                   1.00    284.0±1.96ms        ? ?/sec    1.02    289.3±2.04ms        ? ?/sec
decode_map_small_json/131072                 1.00     31.0±0.11ms   264.9 MB/sec    1.02     31.5±0.07ms   260.6 MB/sec
decode_map_small_serialize                   1.00     20.0±0.18ms        ? ?/sec    1.04     20.8±0.22ms        ? ?/sec
decode_ree_long_json/131072                  1.00      6.1±0.01ms   248.1 MB/sec    1.01      6.2±0.01ms   245.8 MB/sec
decode_ree_long_serialize                    1.00      4.4±0.05ms        ? ?/sec    1.02      4.4±0.07ms        ? ?/sec
decode_ree_short_json/131072                 1.00      6.7±0.01ms   259.7 MB/sec    1.06      7.1±0.14ms   244.5 MB/sec
decode_ree_short_serialize                   1.00      4.8±0.06ms        ? ?/sec    1.00      4.7±0.06ms        ? ?/sec
decode_wide_object_json/131072               1.00    473.3±4.15ms   203.3 MB/sec    1.03    486.0±4.74ms   198.0 MB/sec
decode_wide_object_serialize                 1.00    433.9±5.53ms        ? ?/sec    1.01   439.3±12.31ms        ? ?/sec
decode_wide_projection_full_json/131072      1.00    784.6±8.36ms   221.8 MB/sec    1.01   792.8±10.46ms   219.5 MB/sec
decode_wide_projection_narrow_json/131072    1.00    453.5±1.22ms   383.7 MB/sec    1.02    462.6±1.39ms   376.2 MB/sec
infer_json_schema/1000                       1.03  1554.1±12.35µs    81.2 MB/sec    1.00  1503.9±16.69µs    83.9 MB/sec
large_bench_primitive                        1.00   1526.4±4.02µs        ? ?/sec    1.01   1536.5±3.01µs        ? ?/sec
small_bench_list                             1.00      6.9±0.02µs        ? ?/sec    1.00      6.9±0.01µs        ? ?/sec
small_bench_primitive                        1.01      4.1±0.02µs        ? ?/sec    1.00      4.0±0.01µs        ? ?/sec
small_bench_primitive_with_utf8view          1.04      4.1±0.01µs        ? ?/sec    1.00      4.0±0.02µs        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	455.1s
Peak memory	1.4 GiB
Avg memory	923.2 MiB
CPU user	414.1s
CPU sys	36.0s
Peak spill	0 B

branch

Metric	Value
Wall time	460.1s
Peak memory	1.4 GiB
Avg memory	920.6 MiB
CPU user	415.4s
CPU sys	37.1s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-09-01T18:34:20Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing json-decoder-factory (09cb374) to 27f68b1 (merge-base) diff

Run configuration

run benchmark json_reader

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        json-decoder-factory                   main
-----                                        --------------------                   ----
decode_binary_hex_json                       1.01     13.9±0.06ms        ? ?/sec    1.00     13.7±0.05ms        ? ?/sec
decode_binary_view_hex_json                  1.00     14.3±0.05ms        ? ?/sec    1.00     14.4±0.05ms        ? ?/sec
decode_fixed_binary_hex_json                 1.00     13.9±0.05ms        ? ?/sec    1.00     13.9±0.06ms        ? ?/sec
decode_list_long_json/131072                 1.01    305.0±1.30ms   256.7 MB/sec    1.00    302.0±1.35ms   259.3 MB/sec
decode_list_long_serialize                   1.00    205.3±1.10ms        ? ?/sec    1.01    206.4±1.23ms        ? ?/sec
decode_list_short_json/131072                1.00     20.3±0.13ms   257.2 MB/sec    1.00     20.3±0.07ms   257.2 MB/sec
decode_list_short_serialize                  1.00     11.4±0.12ms        ? ?/sec    1.00     11.4±0.22ms        ? ?/sec
decode_list_view_long_json/131072            1.01    305.1±1.07ms   256.6 MB/sec    1.00    302.2±1.21ms   259.1 MB/sec
decode_list_view_long_serialize              1.00    205.8±1.08ms        ? ?/sec    1.01    207.7±3.14ms        ? ?/sec
decode_list_view_short_json/131072           1.00     20.5±0.06ms   254.2 MB/sec    1.00     20.5±0.06ms   254.0 MB/sec
decode_list_view_short_serialize             1.00     11.6±0.12ms        ? ?/sec    1.00     11.6±0.08ms        ? ?/sec
decode_map_large_json/131072                 1.00    266.6±0.80ms   284.9 MB/sec    1.00    267.8±0.44ms   283.7 MB/sec
decode_map_large_serialize                   1.01    291.6±2.14ms        ? ?/sec    1.00    287.8±2.93ms        ? ?/sec
decode_map_small_json/131072                 1.00     31.1±0.10ms   263.9 MB/sec    1.01     31.6±0.08ms   260.0 MB/sec
decode_map_small_serialize                   1.03     20.6±0.17ms        ? ?/sec    1.00     20.0±0.24ms        ? ?/sec
decode_ree_long_json/131072                  1.00      6.1±0.01ms   248.7 MB/sec    1.06      6.5±0.13ms   233.8 MB/sec
decode_ree_long_serialize                    1.01      4.4±0.03ms        ? ?/sec    1.00      4.3±0.04ms        ? ?/sec
decode_ree_short_json/131072                 1.04      7.0±0.01ms   245.7 MB/sec    1.00      6.7±0.01ms   256.1 MB/sec
decode_ree_short_serialize                   1.00      4.6±0.04ms        ? ?/sec    1.00      4.6±0.04ms        ? ?/sec
decode_wide_object_json/131072               1.00    473.9±5.28ms   203.1 MB/sec    1.02    482.7±5.48ms   199.3 MB/sec
decode_wide_object_serialize                 1.00    437.0±6.85ms        ? ?/sec    1.01   440.8±11.38ms        ? ?/sec
decode_wide_projection_full_json/131072      1.00   784.2±10.43ms   221.9 MB/sec    1.00   786.1±13.37ms   221.3 MB/sec
decode_wide_projection_narrow_json/131072    1.00    451.8±1.15ms   385.1 MB/sec    1.02    459.8±1.31ms   378.5 MB/sec
infer_json_schema/1000                       1.03  1550.7±14.95µs    81.4 MB/sec    1.00  1510.3±16.15µs    83.6 MB/sec
large_bench_primitive                        1.00   1515.4±2.01µs        ? ?/sec    1.01   1532.5±2.81µs        ? ?/sec
small_bench_list                             1.01      6.9±0.02µs        ? ?/sec    1.00      6.9±0.02µs        ? ?/sec
small_bench_primitive                        1.01      4.0±0.02µs        ? ?/sec    1.00      4.0±0.01µs        ? ?/sec
small_bench_primitive_with_utf8view          1.03      4.1±0.01µs        ? ?/sec    1.00      4.0±0.06µs        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	455.1s
Peak memory	1.4 GiB
Avg memory	926.7 MiB
CPU user	414.4s
CPU sys	36.9s
Peak spill	0 B

branch

Metric	Value
Wall time	455.1s
Peak memory	1.4 GiB
Avg memory	924.9 MiB
CPU user	412.7s
CPU sys	36.8s
Peak spill	0 B

File an issue against this benchmark runner

…10741) # Which issue does this PR close? N/A -- documentation only. # Rationale for this change The ability to customize JSON encoding via [`EncoderFactory`] was added in apache#7015, but neither the crate-level docs nor the writer module docs mention it, so users only find the hook by stumbling on `WriterBuilder::with_encoder_factory`. [`EncoderFactory`]: https://docs.rs/arrow-json/latest/arrow_json/trait.EncoderFactory.html # What changes are included in this PR? Documentation only, no code changes: - Crate-level docs: list a custom `EncoderFactory` as an alternative for binary data encoding - Misc other wording fixeers It would be nice to add a matching pointer for customizing *decoding* as part of apache#10670 # Are these changes tested? Covered by existing doc tests and CI rustdoc link checking. # Are there any user-facing changes? Documentation only.

alamb · 2026-09-02T10:56:48Z

Thanks again @hareshkh

hareshkh · 2026-09-02T11:06:16Z

Thanks @alamb!
You doc suggestions make sense to me as well, creating a follow up PR for that now.

Adds crate-level and reader-module doc pointers to the `DecoderFactory` extension point added in apache#10670, mirroring the encoder documentation added in apache#10741.

alamb · 2026-09-02T11:21:15Z

Thanks @alamb! You doc suggestions make sense to me as well, creating a follow up PR for that now.

Thanks @hareshkh -- I am actually working on such a PR here

Document how to customize JSON decoding via DecoderFactory #10897

Perhaps you can review (I'll mark it ready for review in a few minutes after I do a final pass)?

Adds crate-level and reader-module doc pointers to the `DecoderFactory` extension point added in apache#10670, mirroring the encoder documentation added in apache#10741.

github-actions Bot added arrow Changes to the arrow crate arrow-json labels Aug 12, 2026

hareshkh force-pushed the json-decoder-factory branch 2 times, most recently from 63e65ba to d38a0dd Compare August 12, 2026 23:53

alamb mentioned this pull request Aug 18, 2026

Support custom decoders in the JSON reader (DecoderFactory, reader counterpart of EncoderFactory) #10739

Closed

alamb mentioned this pull request Aug 18, 2026

Document how to customize JSON encoding via EncoderFactory #10741

Merged

hareshkh marked this pull request as ready for review August 19, 2026 18:14

alamb mentioned this pull request Aug 27, 2026

Document how to customize JSON decoding via DecoderFactory #10897

Open

alamb reviewed Aug 27, 2026

View reviewed changes

hareshkh added 4 commits August 28, 2026 11:55

Remove a small whitespace

f4230c6

Address comments

0c5989d

hareshkh force-pushed the json-decoder-factory branch from 6b326e5 to 0c5989d Compare August 28, 2026 14:34

hareshkh added 3 commits August 28, 2026 17:43

Add checked decoder to validate

54aadba

Enrich the doctest with nested example

cb4cafa

Fold two more tests into doctest

09cb374

hareshkh commented Aug 28, 2026

View reviewed changes

alamb approved these changes Aug 30, 2026

View reviewed changes

alamb approved these changes Sep 1, 2026

View reviewed changes

alamb merged commit b84fc5c into apache:main Sep 2, 2026
31 checks passed

hareshkh deleted the json-decoder-factory branch September 2, 2026 11:50

Uh oh!

Conversation

hareshkh commented Aug 12, 2026 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Aug 18, 2026

Uh oh!

hareshkh commented Aug 19, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hareshkh Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Aug 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hareshkh commented Aug 28, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Sep 1, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Sep 1, 2026

Uh oh!

adriangbot commented Sep 1, 2026

Uh oh!

adriangbot commented Sep 1, 2026

Uh oh!

adriangbot commented Sep 1, 2026

Uh oh!

adriangbot commented Sep 1, 2026

Uh oh!

Uh oh!

alamb commented Sep 2, 2026

Uh oh!

hareshkh commented Aug 12, 2026 •

edited by alamb

Loading

hareshkh Aug 28, 2026 •

edited

Loading