Skip to content

Generating Annotation Data

Jump to bottom
Dave Lawrence edited this page Aug 6, 2026 · 1 revision

Installing a deployment does not need anything on this page — python3 manage.py download_annotation_data pulls prebuilt files off variantgrid.com, see Install Annotation.

This page is for producing a new release of that data: a new gnomAD/dbNSFP release, a new genome build, or the data behind a new columns_version. The scripts are in annotation/annotation_data/generate_annotation/.

The three kinds of annotation data file

  1. Downloaded as-is from a public source — conservation bigwigs, TOPMED, UK10K, MaxEntScan, reference fastas. annotation/annotation_data/vep_download_37_and_38.sh (which calls vep_all_builds_download.sh, vep_grch37_data_download.sh, vep_grch38_data_download.sh) is the record of where each one comes from. It cds to /data/annotation and lays files out to match settings.ANNOTATION.
  2. Downloaded from a source needing registration — COSMIC, Mastermind, SpliceAI. See VEP annotation data manual download.
  3. Generated by us from a bulk source that is too big or the wrong shape to use directly — gnomAD, dbNSFP, denovo-db, the GRCh37 placental conservation tracks. That's the rest of this page.

Whatever you generate, the filename has to match the vep_config entry in variantgrid/settings/components/annotation_settings.py (relative to ANNOTATION_VEP_BASE_DIR), or that entry has to be changed to match the file. python3 manage.py vep_data_check prints the full path of anything it expects and can't find.

gnomAD

The raw release is ~500G (GRCh37) / ~800G (GRCh38), so it is downloaded and stripped per chromosome, in parallel, then merged.

gnomad2.1_download.sh / gnomad3.1.2_download.sh / gnomad4.0_download.sh

fetch the per-chrom exome/genome/SV VCFs for a release.

gnomad_data.py does the merge. It works in two passes — --scripts writes the per-chromosome shell scripts (which strip INFO fields down to what we keep, then merge exomes + genomes summing the counts), and --af recalculates allele frequency from the merged counts and writes the combined VCF:

gnomad_data.py --scripts --genome-build=GRCh38 --version=4.0 --chrom-mapping-file=snpdb/genome/chrom_mapping_GRCh38.map
gnomad_data.py --af --gnomad-input-vcf=... --af-output-vcf=...

AF is recalculated rather than taken from the source, so AF/grpmax are consistent across the merged exome+genome counts.

v4.1 ships "joint" VCFs, handled by a separate set of scripts:

  • gnomad41_fields.sh — sourced, not run: the list of _joint INFO fields to keep and the bcftools --rename-annots mapping back to the legacy v4.0 names, so downstream code doesn't have to know which release it came from.
  • strip_gnomad41_one.sh INPUT_VCF OUTPUT_VCF CHROM_MAP — strips, renames the annotations, renames contigs (NC_000001.11 → 1) with a bcftools --rename-chrs map from snpdb/genome/, bgzips and tabixes one chromosome.
  • strip_gnomad41.sh INPUT_DIR OUTPUT_DIR CHROM_MAP [PARTITION] — writes a per-chromosome script for each. With no PARTITION it stops there and you run them however you like; given a SLURM partition it adds #SBATCH --partition= and submits them.

Structural variants are separate again — gnomad4_process_sv.sh downloads the per-chrom SV VCFs, keeps the SV columns (SVLEN/SVTYPE/END) plus counts and sub-population AFs, and merges.

Common / uncommon partition filter VCFs

VCF import splits variants into "common" and "uncommon" partitions using a gnomAD AF>5% site list — see Cohort Genotype Common Filters. Those site lists are generated from the merged gnomAD VCFs above by:

export VEP_ANNOTATION_DIR=/data/annotation/VEP
generate_gnomad_upload_preprocess_grch37.sh
generate_gnomad_upload_preprocess_grch38.sh
generate_gnomad_upload_preprocess_t2t.sh

Each extracts AF>0.05 with bcftools view, strips it to a bare site list and tabixes it.

GRCh38 has an extra step: generate_gnomad_upload_preprocess_grch38_intersection.sh runs bcftools isec -n=2 -w1 over the 4.0 and 4.1 outputs so the common partition stays valid under either gnomAD version (#1582). That intersection is the file VCF_IMPORT_COMMON_FILTERS["GRCh38"] points at.

If you build a new one, the gnomad_version / additional_gnomad_versions on the CohortGenotypeCommonFilterVersion record are provenance of the file on disk — they describe what you intersected, so keep them honest.

dbNSFP

dbnsfp_strip.py produces the stripped, bgzipped, tabixed file the VEP dbNSFP plugin reads. It looks column indices up by name from the header, so it survives dbNSFP reordering or adding columns between releases. It prints a shell pipeline (cut/sort/bgzip/tabix) to stdout rather than streaming the data through Python — redirect it to a file and run it with bash.

# GRCh37 / GRCh38 - the pre-sorted single-file download
dbnsfp_strip.py --build GRCh38 --version 5.3.1a --sorted dbNSFP5.3.1a_grch38.gz > run.sh

# T2T - per-chrom inputs, needs sorting
dbnsfp_strip.py --build T2T-CHM13v2.0 --version 5.3.1a dbNSFP5.3.1a_variant.chr*.gz > run.sh

dbNSFP also ships a gene-level file, which is loaded into the database rather than used by VEP:

python3 manage.py import_dbnsfp_gene_annotation --dbnsfp-version=5.3 /data/annotation/incoming/dbNSFP5.3_gene.complete.gz

Conservation (phastCons / phyloP)

The 100-way and 30-way tracks are downloaded ready-made as bigwigs from UCSC (in vep_grch37_data_download.sh / vep_grch38_data_download.sh). The GRCh37 46-way placental mammals tracks are only published as per-chromosome wigFix, so they are merged into a genome-wide bigwig:

vg_annotation_generate_grch37.sh   # wgets the wigFix dirs + UCSC wigToBigWig/bigWigCat, then:
merge_wigs_to_bigwig.sh hg19.chrom.sizes hg19.46way.phastCons.bw <dir>/*.wigFix.gz

merge_wigs_to_bigwig.sh converts each chromosome with wigToBigWig into a temp dir then concatenates with bigWigCat.

bigwig_max_bins.py pre-computes binned summary values from a bigwig. It isn't part of the installed data set — SV conservation maxima are computed live from the source bigwigs by annotation/sv_conservation.py (#1657).

denovo-db

The upstream denovo-db VCF drops StudyName / PubmedID / PrimaryPhenotype (they're only in the TSV), so we rebuild the VCF from the TSVs to carry them through to VEP --custom:

export VEP_ANNOTATION_DIR=/data/annotation/VEP
denovo_db/denovo_db_generate_grch37.sh    # TSVs -> GRCh37 VCF (version read from the TSV header)
export ANNOTATION_DIR=/data/annotation
denovo_db/denovo_db_liftover_grch37_to_grch38.sh   # bcftools +liftover to GRCh38

denovo-db only ships hg19, so GRCh38 is a liftover — which needs Install bcftools liftover and the chromosome-named fastas below.

Reference fastas

The reference_fasta VEP and cdot use is an NCBI accession-named fasta, downloaded by the vep_grch3*_data_download.sh scripts. The liftover fasta is a different, chromosome-named file (matching the contig names in the chain files) built by:

vep_fasta_grch37.sh   # Homo_sapiens.GRCh37.75.dna.primary_assembly.fa.gz
vep_fasta_grch38.sh   # Homo_sapiens.GRCh38.dna.toplevel.fa.gz

Both pipe the download through bgzip (not gzip) and samtools faidx it, which is what VEP requires.

Plugin data

plugin_data.sh is a scratch record of how the smaller plugin data files were made — MaxEntScan, RepeatMasker BED (exported from the UCSC table browser), historical dbNSFP and dbscSNV commands. Treat it as notes, not a runnable script.

Grantham.pm is a VEP plugin (not data) kept here so it can be wgot straight into the plugins dir — see Install VEP.

Publishing generated data

Files are served from http://variantgrid.com/download/annotation/..., mirroring the /data/annotation layout — so /data/annotation/VEP/annotation_data/GRCh38/foo.vcf.gz is at http://variantgrid.com/download/annotation/VEP/annotation_data/GRCh38/foo.vcf.gz. That mapping is what download_annotation_data (and the deployment_check fix instructions) generate, in variantgrid/deployment_validation/annotation_files_check.py. Put a new file at the path settings expect and it is downloadable with no code change.

For a data release it's worth adding an md5 manifest to annotation/annotation_data/md5sum_check/ (named like vep_110_columns_version3_md5sum.txt, paths relative to the annotation data dir) so deployments can verify contents rather than just existence — see Annotation Column Versions.

See also

Clone this wiki locally