Scan character classes with SIMD in the JIT on x86-64

[mattst88]

fast_forward_start_bits() tests the start bitmap one code unit at a time on
every architecture, so a pattern whose first character is a class scans far more
slowly than one starting with a literal, even though the JIT already has a
vectorized scan for the literal case.

This adds a hook for it and implements it for x86-64 with SSE2, both for a class
at the start of the match and for one at a relative offset, so that .{3}[QXZ]
is searched as well as [QXZ].

Commits

Add a helper to split a class bitmap into ranges. optimize_class_ranges()
already reduced a 256 bit bitmap to the code unit values at which membership
changes. That scan moves into extract_class_ranges(), placed before the
include of pcre2_jit_simd_inc.h so the vectorized scan can use it too, and
takes a limit on the number of entries rather than the fixed
MAX_CLASS_RANGE_SIZE. No functional change.

Emit the x86 vector compares through one helper.
fast_forward_char_pair_sse2_compare() assembled each instruction byte by byte,
patching the prefix in place for the AVX2 form, so the same four bytes were put
together seven times over in slightly different ways. The encoding moves into
emit_vector_op(), which takes an opcode and three register indices and picks
the two operand SSE2 form or the three operand AVX2 one from the register width.
It also emits a REX prefix, which the byte-by-byte code had no need for, so that
later users can reach the upper vector registers. The emitted bytes are
unchanged for every combination of compare type, register width, step and
register assignment.

Scan character classes with SIMD in the JIT on x86-64. The bitmap is reduced
to a list of ranges at compile time; the scan then tests each range with the
usual unsigned range idiom, where subtracting the low bound makes the range
start at zero and a saturating subtract of the span leaves zero exactly for the
bytes inside it, since anything below the low bound wraps to a value larger than
the span. That is PSUBB, PSUBUSB and PCMPEQB per range, OR'd together,
with PCMPEQB alone for a single character. Bitmaps needing more than four
ranges keep the existing scan.

Scan for a character class at a relative offset in the JIT. scan_prefix()
already walks the class bitmap at each position, but expands it into the chars
array and keeps at most MAX_DIFF_CHARS characters, so a larger class left
nothing behind but "any character". It now records the bitmap itself alongside
the array. A position keeps its bitmap only while one and the same class reaches
it, since a second alternative, a literal or an unbounded class all make it
describe less than the position accepts.

fast_forward_first_n_chars() then picks the sparsest such position, but only
where it would otherwise give up: a position holding one or two characters gets
a cheaper scan of its own, and a run of them gets the shift table, both of which
beat testing a handful of ranges.

Numbers

Measured over a 4MB subject on an i7-1370P, scanning for a class that does not
occur:

pattern	before	after
[QXZ]	1445 MB/s	25961 MB/s
[0-9]{6}	3337 MB/s	33440 MB/s
.{3}[QXZ]	450 MB/s	25798 MB/s
.{2}[0-9]{6}	647 MB/s	33424 MB/s

Every other pattern measured is unchanged, including a.{4}[QXZ] at 1486 MB/s,
which keeps its single character scan rather than being taken over by the class
scan.

Notes

Vectorizing only pays for a sparse class. The byte-at-a-time loop stops at the
first code unit in the class, so where the class is dense it stops almost
immediately, while a vector loop has already tested a whole block. Measured on
\b\w{12,}\b, whose class accepts 63 code units, the vector scan was 38.6%
slower. Classes accepting more than 48 code units are therefore left alone, and
\b\w{12,}\b and .{3}\w{12} are unchanged.

The bounds of each range live in a vector register of their own, so the scan
needs twelve of them, more than the Win64 ABI leaves as scratch registers. It is
limited to the other x86-64 ABIs, as fast_forward_char_pair_simd() already is,
with a compile time assertion tying the range limit to
SLJIT_NUMBER_OF_SCRATCH_VECTOR_REGISTERS.

SSE4.2's PCMPESTRI was measured as an alternative. It handles up to eight
ranges in one instruction and so is flat in the number of ranges, but it is
slower than this sequence for one or two ranges, which is what real classes
mostly are: 11.5 GB/s against 28.5 GB/s for a single range. It also has no
256-bit form, so it would foreclose widening this loop to AVX2 later.
PCMPISTRI, the faster of the two, cannot be used at all, because it treats a
zero byte as the end of the subject.

RunTest and pcre2_jit_test pass at all three code unit widths, on x86-32, and
with the SIMD scans compiled out.

[mattst88]

This was something that I noticed could be improved when investigating one of the regressions on Alpha.

Not sure if I should add support for necessary ops to sljit first and then use them here or whether the approach this patch takes is okay. Happy to do either.

[zherczeg]

This looks like a good improvement.

The other similar functions also support relative offset, so not only [range] but also .{n}[range] is also supported where n is < 15. I.e. the first few characters are too complex to search, but there is a simple range after them.

[zherczeg]

I think there is a similar function in the code somewhere. Not sure it can be used here.

[mattst88]

It was the scan at the top of optimize_class_ranges(). The first commit of the
series now moves that into extract_class_ranges(), which takes a limit on the
number of entries instead of the fixed MAX_CLASS_RANGE_SIZE, and sits before
the include of pcre2_jit_simd_inc.h so both users see it.

extract_start_bits_ranges() then only has to turn the transition list into
inclusive pairs. optimize_class_ranges() is otherwise unchanged: the
membership of the highest code unit, which its loop used to leave behind in
bit, is read from the bitmap instead.

[zherczeg]

This looks like a good idea

[zherczeg]

Maybe such helpers could be used to improve code readability.

[mattst88]

Done in the second commit of the series. emit_sse2_op() became
emit_vector_op(), taking an opcode and three register indices and picking the
two operand SSE2 form or the three operand AVX2 one from the register width, and
fast_forward_char_pair_sse2_compare() is rewritten on top of it, losing about
60 lines of hand assembled prefix bytes.

The one place the two shapes genuinely differ is the two character compare,
where the 256 bit form does with one instruction what the 128 bit form needs two
for, so that case no longer falls into the shared switch.

Since the AVX2 path is disabled at runtime I could not exercise it the usual way,
so I compiled the old and the new function side by side against a recording stub
and compared their output: 98304 cases, every combination of compare type,
register width, step and register assignment, byte identical.

[NWilson]

This looks nice.

However, I recommend that we bump it to the next release, rather than merge it and release it in a hurry.

I'm just trying to get all the required bugfixes in, and stabilise the branch ahead of the upcoming release.

[mattst88]

Thanks for the review. The two inline comments are answered in their threads;
this covers the rest.

The other similar functions also support relative offset, so not only [range] but also .{n}[range] is also supported where n is < 15.

This is the fourth commit of the series. scan_prefix() already walks the class
bitmap at each position, but expands it into the chars array and keeps at most
MAX_DIFF_CHARS characters, so a larger class left nothing behind but "any
character". It now records the bitmap itself alongside the array, and a position
keeps its bitmap only while one and the same class reaches it, since a second
alternative, a literal or an unbounded class all make it describe less than the
position accepts.

fast_forward_first_n_chars() picks the sparsest such position, but only where
it would otherwise give up. A position holding one or two characters gets a
cheaper scan of its own, and a run of them gets the shift table, and both beat
testing a handful of ranges, so this takes no work away from them. a.{4}[QXZ]
keeps its single character scan, unchanged at 1486 MB/s.

.{3}[QXZ] goes from 450 to 25798 MB/s and .{2}[0-9]{6} from 647 to 33424
MB/s over a 4MB subject on an i7-1370P. A dense class at an offset, .{3}\w{12},
is left alone as before.

Two things worth flagging separately.

The scan as I first posted it was broken on Win64. It holds the bounds of each
range in a vector register of its own, so it needs twelve of them, but
SLJIT_NUMBER_OF_SAVED_FLOAT_REGISTERS is 10 there and only five vector scratch
registers are requested, so it reached registers the ABI requires to be
preserved. It is now limited to the other x86-64 ABIs the same way
fast_forward_char_pair_simd() is, with a compile time assertion tying the range
limit to SLJIT_NUMBER_OF_SCRATCH_VECTOR_REGISTERS so the budget cannot
silently overflow again.

I also tried handing the bound registers out densely rather than two per range,
so that a class of single characters stays below xmm8 and needs no REX prefix.
It saves four bytes on [QXZ], but costs 7.7% on .{3}[QXZ], stable over
repeated runs, purely from the loop body shifting relative to its alignment. Not
worth it, so it is not in the series.

The branch is force-pushed and rebased on current main. Tested at all three code
unit widths, on x86-32, and with the SIMD scans compiled out.