Skip to content

Install Annotation

Jump to bottom
Dave Lawrence edited this page Aug 7, 2026 · 8 revisions

Previously Required Steps

Annotation directory and permissions

Everything below writes into settings.ANNOTATION_BASE_DIR, which defaults to /data/annotation (variantgrid/settings/components/settings_paths.py). Create it up front and give it to the user you run VariantGrid as — the download commands create the sub-directories themselves, but they can't create the top of the tree, and running them as root leaves root-owned files that the variantgrid user later can't update:

sudo mkdir -p /data/annotation
sudo chown ${USER} /data/annotation   # or 'variantgrid' on a server - the user that runs VG

It needs a lot of space (gnomAD and dbNSFP dominate — allow ~1TB for GRCh37 + GRCh38), so it's usually a mounted data disk rather than the root partition. To put it somewhere else, set ANNOTATION_BASE_DIR in your env settings file and re-derive the paths built from it — the component settings computed them at import time, so setting the base alone changes nothing. variantgrid/settings/env/runx1db2.py is the worked example (/mnt/annotation): it redefines ANNOTATION_VEP_BASE_DIR, the VEP code/cache dirs, the fasta dir and the per-build reference_fasta. vep_config entries are relative to ANNOTATION_VEP_BASE_DIR so they follow automatically.

The same user has to be able to read it wherever VEP runs — including the Celery annotation_workers, which on a server are the variantgrid user (see Install (Server)).

Download VEP cache

Download your choice of refseq or ensembl vep cache, matching the VEP version in settings.ANNOTATION_VEP_VERSION and the consortium in settings.ANNOTATION[build]["annotation_consortium"].

Extract to settings.ANNOTATION_VEP_CACHE_DIR — by default /data/annotation/VEP/vep_cache (should make a homo_sapiens or homo_sapiens_refseq dir)

Download VEP annotation data

This is the reference genomes, liftover chains, and the extra data for the plugins/custom annotations.

python3 manage.py download_annotation_data

It works out what's missing from your settings (every vep_config entry for each enabled build, plus the fastas and chains), and downloads it from variantgrid.com. Because the list comes from settings, it always fetches the file versions your columns_version wants — you don't have to keep a download list in sync by hand.

Useful options:

python3 manage.py download_annotation_data --dry-run       # list what's missing, download nothing
python3 manage.py download_annotation_data --force         # re-download even if present
python3 manage.py download_annotation_data --bash-script downloads.sh  # write the wgets out to run elsewhere

Verify with:

python3 manage.py vep_data_check

A vep_config entry of None means "not applicable for this build" and is skipped, so it isn't an error. If you want a different release of a data file to the packaged one, see VEP annotation data manual download.

The packaged settings are the latest annotation (currently VEP 116 / columns_version 5), so a fresh install that just downloaded the data needs no annotation settings at all. Pinning back is only for deployments that have older data on disk — eg after Install from database dump — see Annotation Column Versions.

Where the files come from

The download URLs are the /data/annotation layout served under http://variantgrid.com/download/annotation/ — so a file at /data/annotation/VEP/annotation_data/GRCh38/foo.vcf.gz comes from http://variantgrid.com/download/annotation/VEP/annotation_data/GRCh38/foo.vcf.gz. That's how download_annotation_data builds its list (variantgrid/deployment_validation/annotation_files_check.py), and it's what you'd wget by hand if you wanted a single file.

vep_data_check only checks that files exist. To verify contents, run the matching manifest from annotation/annotation_data/md5sum_check/ from the annotation data dir:

cd /data/annotation/VEP/annotation_data
md5sum -c ${VARIANTGRID_DIR}/annotation/annotation_data/md5sum_check/vep_110_columns_version3_md5sum.txt

Manifests are per VEP/columns version, and only exist for versions someone published one for.

Generating annotation data

The public data — conservation bigwigs, TOPMED, UK10K, MaxEntScan, reference fastas — is downloaded straight from source by the shell scripts under annotation/annotation_data/ (vep_download_37_and_38.sh and the per-build scripts it calls). The rest (gnomAD, dbNSFP, denovo-db) is bulk data we process into shape first.

download_annotation_data is what you run to install any of it. See Generating Annotation Data if you're producing a new release of the data itself.

Reference genomes

Each build has two fastas, and they aren't interchangeable:

  • ANNOTATION[build]["reference_fasta"] — an NCBI fasta whose sequences are named by contig accession (eg GCF_000001405.25_GRCh37.p13_genomic.fna.gz). This is what VEP and cdot use.
  • ANNOTATION[build]["liftover"]["fasta"] — sequences named by chromosome (eg Homo_sapiens.GRCh37.75.dna.primary_assembly.fa.gz), to match the contig names in the liftover chain files. Only needed when LIFTOVER_BCFTOOLS_ENABLED — bcftools +liftover has no --rename-chrs equivalent, so using the accession-named fasta here would mean rewriting the chains instead.

download_annotation_data gets the reference_fasta for every enabled build, and the liftover fasta and chains as well when LIFTOVER_BCFTOOLS_ENABLED. The public sources are NCBI (ftp.ncbi.nlm.nih.gov/genomes/refseq/vertebrate_mammalian/Homo_sapiens/all_assembly_versions/) and Ensembl (ftp.ensembl.org/pub/) — if you fetch them by hand they must be bgzipped (not gzipped) and indexed with samtools faidx, as VEP requires.

Genes / Transcripts

Run the following to get the latest version of cdot transcripts. The command works out the latest cdot data release itself, and skips the work if you already have it (--force to override):

python3 manage.py import_cdot_latest

You can restrict it with --genome-build / --annotation-consortium.

For consistent analyses you also want a GeneAnnotationRelease snapshot matching your VEP version — see Gene and Transcript Version Data.

Annotation install steps

View the "annotation" page, this will be "/annotation" from the server path, eg for a local connection https://localhost:8000/annotation

See also

Clone this wiki locally