- Version: 1.7.0
- Released: 2026/08/03
- Author(s): Bryan Gee (UT Libraries, University of Texas at Austin; bryan.gee@austin.utexas.edu; ORCID: 0000-0003-4517-3290)
- Contributor(s): None
- License: GNU GPLv3
- README last updated: 2026/08/05
This repository includes scripts that are designed for reporting and assessment purposes for the Texas Data Repository (TDR). They are intended for both institution-level and TDR-level analysis.
This repository uses uv to manage its Python version and dependencies (declared in pyproject.toml and pinned in uv.lock).
- Install uv, if you don't already have it, by following the official installation instructions.
- Clone this repository and open a terminal (either within an IDE or separately)in its root directory.
- Run
uv syncto create a project-local virtual environment (.venv) and install the exact dependency versions fromuv.lock. If you are running in a OneDrive-linked system, you may need to runuv sync --link-mode=copy. - Save
.env.exampleas.envand populate it with your own API key, institution details, and toggles (see Config file and .env file). - Make sure to always navigate to the folder with the scripts (either via a terminal or within an IDE) to keep running the venv, otherwise it may fail if you do not have local versions of certain modules.
If you add or update a dependency, edit pyproject.toml and run uv lock followed by uv sync to keep uv.lock current.
- dataverse-file-assessment.ipynb: Jupyter notebook with code to retrieve metadata at the file, dataset, and collection levels. This is accomplished through multiple API calls through the Search API and the Native API and includes metadata that is not present in the monthly institutional reports generated by TDL (though conversely, some information available in those reports, like metadata on unpublished datasets for all institutions, cannot be retrieved from these scripts). The script can either be run at the institution-level, in which case a regular liaison user would be able to retrieve unpublished collections, datasets, and files, or it can be run at the pan-TDR level. Core parts of the codebase have been modified or created in parallel with the Metadata Re-Curation Workflow such that the current version differs a fair bit from previous versions.
- tcdl-graphs.ipynb: Jupyter notebook with code to generate graphics for annual TCDL usage reporting. Right now, it also contains all of the code in tcdl-institutional-reports.ipynb for a single-shot run.
- tcdl-institutional-reports.ipynb: Jupyter notebook with code to generate institution-specific slide-decks for annual TCDL usage reporting.
- dataverse-metadata-completeness.ipynb: Jupyter notebook
- utils.py: This file contains all of the functions needed for the scripts. As a note, this is a function file being used by the developer across many different projects, so it includes many functions irrelevant to this repository, which is why only the necessary ones are imported in the scripts. It should not be modified except by users with detailed knowledge of Python and this workflow.
- data-dictionary.csv: This file contains descriptions of the columns in four files, the data dictionary itself, the date_institution_all-dataverses.csv file, the date_institution_all-datasets-combined-with-dataverses.csv, and the date_institution_all-files-deduplicated.csv file.
This script will return eight direct output files (listed in the order in which they are generated):
- date_institution_all-deposits.csv: a dataset-level dataframe with an entry for every dataset that is returned from the Search API. For users with the appropriate permissions, this can include unpublished and deaccessioned datasets. This dataframe is merged with one of the TDL data dumps for additional dataset-level metadata.
- date_institution_all-files-deduplicated.csv: a file-level dataframe with an entry for each file retrieved from the search process. If you are only retrieving published records, this will have '-PUBLISHED' appended to the end of the filename.
- date_institution_all-datasets-combined.csv: a dataset-level dataframe that is constructed by aggregating all file-level information into dataset-level entries and then merging it with one of the TDL data dumps for additional dataset-level metadata.
- date_institution_all-collections.csv: a collection-level dataframe with an entry for every collection that is returned from the Search API. For users with the appropriate permissions, this can include unpublished and deaccessioned collections. This dataframe is merged with one of the TDL data dumps for additional collection-level metadata.
- date_institution_SUMMARY-unique-format.csv: a dataframe with a summary of the number of unique datasets in which each file format occurs.
- date_institution_SUMMARY-annual-size.csv: a dataframe with a summary of the total file size of files created in a given year.
- date_institution_all-datasets-combined-with-collections.csv: a dataset-level dataframe that is essentially files 3 and 4 combined. If you enable metrics retrieval, these will be appended to this file.
- date_institution_all-datasets-combined-with-collections-PUBLISHED.csv: the same as file 7 but only for published datasets.
- date_new-file-formats.csv: a dataframe with a list of all files that have a mimeType that was not mapped to a friendly format. You only need to look at the
file_mime_typecolumn, identify the corresponding friendly format, and add it to theconfig.jsonfile. - *date_dataset-metrics_index_institution.csv. if you retrieve dataset metrics, a separate output file with only those metrics and the DOI will be output.
This script generates graphics used in the annual Usage Report for TCDL. Right now, this script is not set to write any outputs, as generated graphics can be directly copied out of the Jupyter notebook interface and into any desired program. In the future, the script will be set up to save these images.
This script will return one PowerPoint file for each institution with the filename formatted as {institution}report{date}.pptx. The slide-decks are in 16:9 format to facilitate direct adaptation of the pan-TDR graph code, which is mostly set to output plots at 14 x 7 dimensions.
This script generates two summary graphics depicting (in)completeness of metadata fields across TDR - it is broadly similar to the DataCite Metadat Dashboard, though it includes fields not cross-walked to DataCite. Right now, this script is not set to write any outputs, as generated graphics can be directly copied out of the Jupyter notebook interface and into any desired program.
These scripts can be freely re-used, re-distributed, and modified in line with the associated GNU GPLv3 license. If a re-user is only seeking to replicate a UT-Austin-specific output or to retrieve an equivalent output for a different institution, the script will require very little modification - essentially only the defining of affiliation parameters will be necessary. A superuser could have greater functionality in some instances, but superuser-specific functionality has largely not been developed because I have no way to test it.
Parameters are split across two files:
config.json: numerical API query parameters (e.g., records per page, page limit), graph/table defaults, and institution standardization maps. Nothing secret lives here, so it's safe to share/commit as-is..env: your Dataverse API token, your institution's identifying details, and run-time toggles (which live here rather than in config.json since they tend to change frequently between runs). The file included in this repository called.env.exampleshould be saved as.envand populated with your own API key and institution-specific information..envis gitignored and should never be distributed to anyone else.
| Variable | Required? | Default | Description |
|---|---|---|---|
DATAVERSE_TOKEN |
Required | — | Your personal Dataverse API key, sent as the X-Dataverse-key header on all TDR API requests. Obtain one from your Dataverse account. |
MY_INSTITUTION |
Required | — | Short-hand name for your institution (e.g., UT Austin), used in filenames and console output. Must pick value from controlled vocabulary |
INSTITUTION_FILENAME |
Required | — | Filename-safe version of your institution's name (e.g., UT-austin), used to build output filenames. |
INSTITUTION_SUBTREE |
Required | — | Alias of your institution's Dataverse collection root (e.g., utexas), used to scope Search API queries. |
TEST_ENVIRONMENT |
Boolean toggle | false |
If true, retrieves only a handful of pages instead of a full run - useful for testing and troubleshooting. |
ONLY_MY_INSTITUTION |
Boolean toggle | false |
If true, restricts retrieval/reporting to your institution only, rather than all of TDR. When false, EXCLUDE_DRAFTS is automatically forced to true. |
VERSIONS_API |
Boolean toggle | false |
If true, enables the additional "stage 3" version-level API retrieval in dataverse-file-assessment.ipynb. |
EXCLUDE_DRAFTS |
Boolean toggle | false |
If true, restricts queries to published records only (filters out unpublished/deaccessioned records). Forced to true whenever ONLY_MY_INSTITUTION is false. |
METRICS_DC |
Boolean toggle | false |
If true, retrieves/uses DataCite-sourced usage metrics. |
METRICS_DV |
Boolean toggle | false |
If true, retrieves/uses native Dataverse usage metrics. |
CURRENT_MEMBERS |
Boolean toggle | false |
If true, restricts graphs/reports to current TDR member institutions, excluding former or newly added members. |
SPLIT_INSTITUTION_OUTPUT |
Boolean toggle (in development) | false |
If true, splits combined dataset output by institution. Automatically disabled when ONLY_MY_INSTITUTION is true. |
MY_INSTITUTION must be entered from this controlled vocabulary:
- 'Baylor U'
- 'Lamar U'
- 'SMU'
- 'TAMU'
- 'TAMU Galveston'
- 'TAMIU'
- 'Texas State U'
- 'Texas Tech U'
- 'Texas Women's U'
- 'U Houston'
- 'UNT-HSC'
- 'UT Arlington'
- 'UT Austin'
- 'UTHSCSA'
- 'UT Southwestern'
It is possible to update this if you really do not like your institution's shorthand name, but it has to be changed in many other places (mainly config.json but also some of the actual notebooks to avoid breaking the entire script)
Users will need to create accounts for Dataverse in order to obtain personalized API keys, add those to the .env file (as DATAVERSE_TOKEN).
A Boolean variable called test, defined by TEST_ENVIRONMENT in .env, can be used to create a 'test environment.' If this setting is set to TRUE, the script is set to only retrieve a handful of pages of the full response. It is useful for testing new functionality and trouble-shooting, provided that any bugs are not edge cases that would be unlikely to be retrieved in a small sample size.
Following requests to implement manual rate limiting, large batches of iterative API calls have had manually rate limiting implemented in the code (via time.sleep commands). This should not be modified.
In addition to the technical infrastructure needed to run this script, two different files provided by TDL are necessary:
- dataverse-reports-YYYYMMDD: this folder contains the biweekly (now monthly?) reports run for each institution. The primary script here will concatenate all of the datasets and dataverses by importing each file's relevant sheets and will output a single concatenated file for each into that same folder.
- Dataverse-users-YYYYMMDD.xlsx: this Excel file contains all users in the system and cannot be reproduced by concatenating the 'users' tab from the biweekly reports. It is only necessary for the graphing components - there are no additional data retrieval components involved with this.
- logos: this folder contains PNG or JPG images of each institution's logo. This is not shared on GitHub for trademark purposes and can either be requested from this repository's maintainer (Bryan) or recreated yourself by adding a logos subfolder within the same directory as the script and adding images with the name {institution}_logo. For standardization, you should use the TDR collection abbreviation (e.g., 'utexas' for UT Austin) that is used as the alias for your institution's collection.