Variant grids on the native DataTables engine #1815

[davmlaw]

🤖 Written by Claude

Closes out the work planned in claude/plans/1815_variant_grids_native_datatables_plan.md (issue #1815).

Why

The jqGrid client went in #1785, but the variant grids still spoke jqGrid on the server — colmodel dicts, rows/page/sidx/sord, a {page, total, rows, records} envelope — with jqgrid_datatable_adapter.py translating to and from the DataTables protocol on every request. Two server side engines that had already drifted (null ordering, count strategy, CSV semantics, page size cap, NaN handling), and a second vocabulary every variant grid feature had to be written in and then mapped.

What

AbstractVariantGrid and its subclasses (VariantGrid, ExportVariantGrid, AllVariantsGrid, NearbyVariantsGrid, GeneSymbolVariantsGrid, TaggedVariantGrid) are now DatatableConfigs served by DatabaseTableView. The pieces they genuinely needed became first-class engine features any grid can use:

RichColumn	width, header_title, column_filter, null_order, csv_rendered, sort_menu, client_renderer_kwargs
DatatableConfig	get_extra(), filter_fields() / apply_filter_rules(), known_count() / approximate_count(), post_data(), initial_order(), render_rows() / iter_export_rows(), plus the definition flags (table class, compact controls, deferred loading, cache-stable params, ajax type, max page length)

datatable_response() and datatable_definition() are module functions, so the analysis node grid views — which sit on their own mixin for error handling and locking — build the same envelope as the view does.

New library/django_utils/filter_rules.py holds the rule vocabulary and rules_to_q, shared by the grids' filter builder and FilterNode (FakeFilterGrid goes). resolve_field_path moves to library/django_utils.

Columns are built from a CustomColumnsCollection by get_variant_grid_columns(), which resolves each catalogue path to its Django field for the label, the filter type and the server side formatting the CSV shares with the grid. Node column contributions return RichColumns. Packed genotype columns sort on an alias annotated per column rather than a ':'-packed colmodel index.

The <slug:op> URL segment goes — one URL per grid, ?dataTableCsv=1 is the download, and tagged_variant_export folds into it.

library/jqgrid/, jqgrid_datatable_adapter.py and jqgrid_view.py are deleted. CACHE_VERSION is bumped — node grid definitions and handler responses are cached for a week under the old param names and envelope.

Three deviations from the plan worth reviewing

• renderKwargs stays, as RichColumn.client_renderer_kwargs. The plan called it unused server side, but VariantGridFormat.sampleZygosity reads samplePrefix and dbZygosityCounts reads countPrefix. The one genuinely dead consumer, VariantGridFormat.link, is deleted.
• AnalysisNode.get_extra_columns() drops its cache.set — RichColumns hold renderer closures (the per-VCF filter lookup) which don't pickle. Both node grid endpoints are behind @cache_page anyway.
• Nearby and gene symbol variant CSVs now export in the grid's default order (locus__position) rather than -pk, since default_sort applies server side. Tagged variants keeps the genomic order its old export view forced.

Testing

python3 manage.py test --keepdb — 2605 tests, OK.

analysis/tests/test_jqgrid_datatable_adapter.py is replaced by analysis/tests/test_variant_grid_datatable.py (definition shape, per-column widths, filter builder fields/operations, rows envelope, draw handling, download URL). The behaviour tests it shared with the grids — known count, sort limit, packed genotype sort round trip, node endpoints, CSV/VCF exports, all-variants estimate — were ported in place.

Not yet done: the plan's manual verification in a running app (node grid sort/filter/exports, all-variants estimate, gene symbol hotspot filters, and a before/after byte diff of a node CSV and VCF).

🤖 Generated with Claude Code

https://claude.ai/code/session_01NHKKAWGpw8uwE1r9KTumHs