Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions corpus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,18 @@ npm run models:evaluate
npm run models:verify
```

Benchmark all inputs or isolate a stable recognizer family workload with:

```sh
npm run benchmark
npm run benchmark -- --family=singlebyte
npm run benchmark -- --family=multibyte
```

Benchmark output includes the input bytes per iteration and a workload SHA-256.
Performance results are comparable only when the family, split, and workload
hash match.

The build script generates the corpus and performs an iconv round trip for every
document, allowing canonically equivalent Unicode composition where an encoding
such as windows-1258 stores combining marks. The check script independently rebuilds into a temporary
Expand Down Expand Up @@ -45,5 +57,11 @@ reports every held-out test document, while `models:verify` checks
reproducibility and fails if any encoding or language result is incorrect. A
different encoding label is accepted only when both encodings decode the actual
held-out bytes identically; the report distinguishes these byte-equivalent
results from exact matches. CP949 is excluded because its multibyte recogniser
requires character frequency data rather than byte trigrams.
results from exact matches.

For multibyte encodings, the structural parsers identify valid raw code units
and the compiler selects the 128 most frequent units from the training split.
Runtime confidence is the structurally valid proportion of input code units
covered by that generated model. Shift_JIS, EUC-JP, Big5, EUC-KR, and GB18030
use this path. CP949 remains in the corpus but is excluded from the runtime
models until its extended byte sequences have a dedicated parser.
3 changes: 3 additions & 0 deletions corpus/generated/Big5/zh/test/community-garden.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
���Ϫ��

�F�~�̧�Ű|�l��y�������A�ؤU�A��B����M�p��C�Ĥl�̵e�F�m��еP�A�U�a�C�g�@�_���Ӫ�����C
3 changes: 3 additions & 0 deletions corpus/generated/Big5/zh/train/city-morning.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
����������

�����C�C���ӡA�����E���F�W�Z���H�A�@�ة����}�j���A�Ĥ@�Z���@�T���}�l��L�w�R����D�C
3 changes: 3 additions & 0 deletions corpus/generated/Big5/zh/train/library.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�w�R���Ϯ��]

���ϹϮ��]�̥u��½�Ѫ��n���CŪ�̴M����v�B��ǩM�Ȧ�譱���ѡA�޲z�����U�ǥͬd���ơC
3 changes: 3 additions & 0 deletions corpus/generated/Big5/zh/train/market.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�P����������

�s���W�������\���s�A����B���G�B���e�M�ѥ]�C�U�ȦV�A���߰ݦ����A�å�y�a�`�檺���k�C
3 changes: 3 additions & 0 deletions corpus/generated/Big5/zh/train/workshop.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�Ʀ�u�@��

�@�Ӥp�ζ����b�s�@�ժ��]���n�C�]�p�v��z�����A�}�o���ˬd�{���A�s�謰���P�~�֪����[�̼��g�M���������C
3 changes: 3 additions & 0 deletions corpus/generated/Big5/zh/validation/river-trip.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�e��Ȧ�

�B�̱ͭa�ۦa�ϩM������L��L�C���F�ܤ[�H��A�L�̧��b�w�R���e��𮧡Ať�ıߪ�����C
3 changes: 3 additions & 0 deletions corpus/generated/EUC-JP/ja/test/community-garden.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�ϰ����

���οͤ����϶����Ƥ��������������֤���ڤ�㤤�ڤ򿢤������Ҥɤ⤿���Ͽ����䤫�ʻ����ꡢ��²���轵��ʪ�˿���롣
3 changes: 3 additions & 0 deletions corpus/generated/EUC-JP/ja/train/city-morning.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
����ī

ī�γ��Ϥ�ä����ܤ�Фޤ����ؤˤ��̶Ф���ͤ����ޤꡢ����Ź����򳫤����ǽ�ΥХ����Ť����̤������Ϥ�롣
3 changes: 3 additions & 0 deletions corpus/generated/EUC-JP/ja/train/library.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�Ť��ʿ޽��

�ϰ�ο޽�ۤǤ��ܤΥڡ�����᤯�벻��ʹ�����롣�ɼԤ���ˤ�ʳؤ�ι�Ԥ��ܤ�õ�����ʽ�ϳ�����Ĵ��ʪ���������
3 changes: 3 additions & 0 deletions corpus/generated/EUC-JP/ja/train/market.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�������λԾ�

����λԾ�ˤϿ�������ڡ���ʪ��˪̪���Ƥ����ƤΥѥ��¤֡��㤤ʪ�Ҥ����Ȥȵ���������ˤĤ����ä���
3 changes: 3 additions & 0 deletions corpus/generated/EUC-JP/ja/train/workshop.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�ǥ����빩˼

�����ʥ����ब��ʪ�ۤΰ�����äƤ��롣�߷׼Ԥ����̤���������ȯ�Ԥ������ɤ��ǧ�����Խ��Ԥ�ʬ����䤹��������񤯡�
3 changes: 3 additions & 0 deletions corpus/generated/EUC-JP/ja/validation/river-trip.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
��ؤξ�ι��

ͧ�ͤ������ϿޤȲ������������äƿ����⤤����Ĺ������θ塢�Ť������դǵ٤ߤʤ���ͼ����Ļ������ʹ������
3 changes: 3 additions & 0 deletions corpus/generated/EUC-KR/ko/test/community-garden.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
���� ���� ����

�ֹε��� ��� �ִ� ������ �����ϰ� �ɰ� ä��, � ������ �ɾ���. ���̵��� �˷ϴ޷��� �̸�ǥ�� ������� �̿����� ���� �� ���� �ְ� ������ �����Ѵ�.
3 changes: 3 additions & 0 deletions corpus/generated/EUC-KR/ko/train/city-morning.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
������ ��ħ

���ô� õõ�� ��ħ�� �´´�. ī�䰡 ���� ���� ������ ù ������ �����ϸ� ������� �޴���ȭ�� ������ �д´�. �ǹ� ���̷� ���� �޺��� ������.
3 changes: 3 additions & 0 deletions corpus/generated/EUC-KR/ko/train/library.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
������ ������

���� ���������� å�� �ѱ�� �Ҹ��� �����ϰ� �鸰��. ���ڵ��� ����� ����, ���࿡ ���� å�� ã�´�. �缭�� �л����� ���� ���� ���ø� �غ��Ѵ�.
3 changes: 3 additions & 0 deletions corpus/generated/EUC-KR/ko/train/market.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
����� ����

���忡 ���� ���忡�� ����� ��, ġ��� �� ���� ���� ���δ�. �մԵ��� ä�Ҹ� ������ ��ο� ���� �̾߱⸦ ������ ������ ���� �丮���� �Ұ��Ѵ�.
3 changes: 3 additions & 0 deletions corpus/generated/EUC-KR/ko/train/workshop.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
������ �۾���

���� ���� �ڹ��� �ȳ� ���α׷��� �����. �����̳ʴ� �޴��� �����ϰ� �����ڴ� �ڵ带 Ȯ���ϸ� �����ڴ� �湮������ ���� ������ ����. Ȯ�� ���� �۰� �ڵ� �˻��Ѵ�.
3 changes: 3 additions & 0 deletions corpus/generated/EUC-KR/ko/validation/river-trip.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
������ ���� ����

�Ͽ��Ͽ� ģ������ ���� ���� ������ ������ �ɾ��. ������ ������ ��, �����⸦ ì���. �� ��å �ڿ��� ������ �ɾ� ���� ���Ҹ��� �����.
3 changes: 3 additions & 0 deletions corpus/generated/GB18030/zh/test/community-garden.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
��^���@

���ӂ��ѿ�Ժ�Ӹ���ɾGɫ���@���N���r�����߲˺�С�䡣���ӂ����˲�ɫ���ƣ�����ÿ�Lһ��oֲ�ˮ��
3 changes: 3 additions & 0 deletions corpus/generated/GB18030/zh/train/city-morning.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
���е��糿

���������с���܇վ�ۼ����ϰ���ˣ����ȵ���_���T����һ�๫����܇�_ʼ���^���o�Ľֵ���
3 changes: 3 additions & 0 deletions corpus/generated/GB18030/zh/train/library.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
���o�ĈD���^

��^�D���^�eֻ�з�����•�����x�ߌ��Қvʷ���ƌW�����з���ĕ�������T�����W�������Y�ϡ�
3 changes: 3 additions & 0 deletions corpus/generated/GB18030/zh/train/market.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
���������Ј�

�V���ϵ��Ј��[�M���r�߲ˡ�ˮ�������ۺ��I��������r��ԃ���ճɣ��K�����ҳ��˵�������
3 changes: 3 additions & 0 deletions corpus/generated/GB18030/zh/train/workshop.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�������

һ��С�F������u�������^ָ�ϡ��OӋ��������棬�_�l�ߙz���ʽ����݋�鲻ͬ���g�ą��^��׫��������f����
3 changes: 3 additions & 0 deletions corpus/generated/GB18030/zh/validation/river-trip.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
��߅����

���т������؈D�͟�贩�^���֡����˺ܾ����ᣬ�������ڰ��o�ĺ�߅��Ϣ�� �������B�Q��
3 changes: 3 additions & 0 deletions corpus/generated/Shift_JIS/ja/test/community-garden.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�n��̒�

�ߏ��̐l�����͋󂢂Ă�����𐮂��A�Ԃ��؂�Ⴂ�؂�A�����B�q�ǂ������͐F�N�₩�ȎD�����A�Ƒ��͖��T�A���ɐ������B
3 changes: 3 additions & 0 deletions corpus/generated/Shift_JIS/ja/train/city-morning.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�X�̒�

���̊X�͂������ڂ��o�܂��B�w�ɂ͒ʋ΂���l���W�܂�A�i���X�������J���A�ŏ��̃o�X���Â��Ȓʂ�𑖂�n�߂�B
3 changes: 3 additions & 0 deletions corpus/generated/Shift_JIS/ja/train/library.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�Â��Ȑ}����

�n��̐}���قł͖{�̃y�[�W���߂��鉹����������B�ǎ҂͗��j��Ȋw�◷�s�̖{��T���A�i���͊w���̒��ו�����`���B
3 changes: 3 additions & 0 deletions corpus/generated/Shift_JIS/ja/train/market.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�y�j���̎s��

�L��̎s��ɂ͐V�N�Ȗ�؁A�ʕ��A�I���A�Ă����Ẵp�������ԁB�������q�͔_�ƂƋG�߂̗����ɂ‚��Ęb���B
3 changes: 3 additions & 0 deletions corpus/generated/Shift_JIS/ja/train/workshop.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
�f�W�^���H�[

�����ȃ`�[���������ق̈ē�������Ă���B�݌v�҂���ʂ𐮂��A�J���҂��R�[�h���m�F���A�ҏW�҂�������₷�������������B
3 changes: 3 additions & 0 deletions corpus/generated/Shift_JIS/ja/validation/river-trip.bin
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
��ւ̏����s

�F�l�����͒n�}�Ɖ����������������ĐX��������B�����U���̌�A�Â��Ȑ�ӂŋx�݂Ȃ���[���̒��̐��𕷂����B
Loading