Action-Analyzer

A TypeScript-based static analysis tool that parses GitHub Actions workflow YAML files using @actions/workflow-parser and analyzes CI patterns across top-starred GitHub repositories.

Corpus metadata for all experiment repos is shipped in this repository (no parent CWC checkout required). Tool fingerprint: tool-manifest.json (npm run repro:manifest).

Experiment corpus (repos & versions)

The study analyzes 2,000 frozen GitHub repositories:

Tree	Count	Role
`typescript-repos/`	1,000	TypeScript / Node.js projects
`python-repos/`	1,000	Python projects

How each repo is named and located

Declared in this package’s corpus.gitmodules (export of the study .gitmodules entries for the two corpora):

[submodule "typescript-repos/colinhacks__zod"]
	path = typescript-repos/colinhacks__zod
	url = https://github.com/colinhacks/zod.git

Folder name owner__repo → GitHub project owner/repo
URL is the clone remote (from corpus.gitmodules)
Entries cover python-repos/* and typescript-repos/* only (2,000 modules)

How the version (commit) is fixed

corpus.gitmodules stores path + URL. The pinned commit SHA for each repo is in corpus-pins.tsv (language, folder, commit, url, path).

Example:

Field	Example
Path	`typescript-repos/colinhacks__zod`
URL (`corpus.gitmodules`)	`https://github.com/colinhacks/zod.git`
Pinned commit (`corpus-pins.tsv`)	`c7805073fef5b6b8857307c3d4b3597a70613bc2`

Action-Analyzer does not re-select or re-pin repositories; it only reads whatever is checked out under CWC_TS_REPOS / CWC_PY_REPOS.

Fetch the corpus (one command)

Works from a standalone clone of this repo (uses corpus.gitmodules + corpus-pins.tsv; needs network access to GitHub):

npm run fetch:corpus

Equivalent: ./scripts/fetch-corpus.sh

Clones each pinned commit into $CWC_ROOT/python-repos and $CWC_ROOT/typescript-repos.
CWC_ROOT defaults to the parent study tree when present (../..), otherwise to this package directory (standalone).

# optional knobs
./scripts/fetch-corpus.sh --jobs 16
SHALLOW=0 ./scripts/fetch-corpus.sh          # fuller history fetch
CWC_ROOT=. npm run fetch:corpus              # corpora next to this package
CWC_ROOT=/path/to/outdir npm run fetch:corpus

Smoke test with a few repos (`--limit`)

Full corpus is 1,000 repos per language. For a quick local check, fetch only the first N repos per language (sorted by path, so the subset is deterministic):

# 10 Python + 10 TypeScript (20 clones total)
npm run fetch:corpus -- --limit 10

# TypeScript only, 5 repos
npm run fetch:corpus -- --limit 5 --lang typescript

# Python only, 20 repos
./scripts/fetch-corpus.sh --limit 20 --lang python

Then run the usual pipelines; analyzers only see directories that exist on disk. Paths fetched in that run are listed in corpus-fetched.txt. For paper-scale reproduction, run npm run fetch:corpus again without --limit.

After a successful fetch, point analyzers at the same root (.env):

# standalone example
CWC_ROOT=.

Corpora paths: $CWC_ROOT/typescript-repos, $CWC_ROOT/python-repos.

Project Structure

Action-Analyzer/
├── src/
│   ├── cli.ts                # Preferred CLI entry (task + --lang)
│   ├── config.ts             # Path / API token resolution (env-overridable)
│   ├── index.ts              # Main analyzer (version, library, self-hosted, readability checks)
│   ├── use-analyzer.ts       # Validates reusable action commit existence via GitHub API
│   ├── package-parser.ts     # Matches Node.js version against package.json declarations
│   └── package-parser.py     # Matches Python version against pyproject.toml declarations
├── scripts/
│   ├── fetch-corpus.sh       # One-command submodule fetch for python/typescript repos
│   ├── run-pipeline.sh       # End-to-end parsing → detection → CSV aggregation
│   └── generate-tool-manifest.sh
├── result/
│   ├── python/               # Analysis results for Python repositories
│   └── node/                 # Analysis results for TypeScript repositories
├── patches/                  # patch-package patch files
├── dist/                     # esbuild bundle output
├── VERSION                   # Tool release identifier
├── corpus.gitmodules         # Path + URL for all 2000 experiment repos
├── corpus-pins.tsv           # Per-repo pinned commit inventory (2000 rows)
├── corpus-fetched.txt        # Paths fetched by the last fetch:corpus run
├── tool-manifest.json        # Generated tool fingerprint (commit, lock hash, API params)
├── .env.example              # CWC_*_REPOS and GITHUB_TOKEN
└── .github/workflows/
    └── sample.yml            # Sample workflow for local single-file testing

How to Run

1. Install dependencies

npm ci    # preferred for reproduction
# or: npm install

The postinstall hook automatically runs patch-package, applying the required patches to @actions/workflow-parser and @actions/expressions.

2. Fetch the frozen corpus

npm run fetch:corpus
# quick smoke test (N repos per language):
# npm run fetch:corpus -- --limit 10

Configure paths via .env (auto-loaded; no hardcoded machine paths):

cp .env.example .env
# Edit if needed — relative paths are from this package directory (pwd):
#   CWC_ROOT=../..
#   CWC_TS_REPOS=../../typescript-repos
#   CWC_PY_REPOS=../../python-repos
#   GITHUB_TOKEN=...
npm run analyze -- config   # prints resolved absolute paths

Defaults: CWC_ROOT=../.. → <package>/../../typescript-repos and …/python-repos.

3. Preferred: CLI tasks (no source edits)

npm run analyze -- config
npm run analyze -- parse --lang python
npm run analyze -- node-version
npm run analyze -- readability --lang typescript
npm run analyze -- match-node
npm run analyze -- match-python
npm run analyze -- reuse-validate   # requires GITHUB_TOKEN

Full pipelines:

npm run pipeline:typescript
npm run pipeline:python

Record the tool fingerprint for archival:

npm run repro:manifest

4. Legacy: direct module execution

npm run start

Or uncomment a function at the bottom of src/index.ts (still supported). Set isLocal: true to use CWC_LOCAL_ANALYZER.

5. Update CSV aggregations

python3 result/python/update.py
python3 result/node/update.py

Reads each txt result file and updates the aggregation columns in py-github-stars.csv / ts-github-stars.csv.

Analysis Pipeline

Frozen corpus (after npm run fetch:corpus → CWC_TS_REPOS / CWC_PY_REPOS)
        │
        ▼
 npm run analyze / scripts/run-pipeline.sh
   · parse, *-version, library-*, uses, self-hosted, readability
        │
        ▼
*_version_result.txt / use_command_result.txt
        │
        ├─► match-python (package-parser.py)  → result/python/*.txt
        ├─► match-node   (package-parser.ts)  → result/node/*.txt
        └─► reuse-validate (use-analyzer.ts)  → result.txt  (GitHub Commits API)
                │
                ▼
        result/*/update.py  →  py-github-stars.csv / ts-github-stars.csv

Core Code

`src/index.ts` — Main Analyzer

Provides shared utilities to recursively scan workflow directories and parse each YAML file.

Function	What it checks	Output
`nodeVersionAnalyzer`	Extracts `node-version` from `actions/setup-node`, evaluates matrix expressions	`node_version_result.txt`
`pythonVersionAnalyzer`	Extracts `python-version` from `actions/setup-python`	`python_version_result.txt`
`libraryInstallAnalyzer`	Detects missing npm/yarn/pnpm/bun install before lint, build, test, deploy, release	Console output
`pythonLibraryInstallAnalyzer`	Detects missing `pip install` before `python *.py` execution	Console output
`useCommandExtractor`	Collects all `uses:` values across all workflows	`use_command_result.txt`
`selfHostChecker`	Checks whether `self-hosted` is the first element in the `runs-on` array	`self_host_checker_result.txt`
`nameChecker`	Detects jobs containing a step longer than 300 characters with no `name`, and at least 2 steps	`result/readability_issue_workflows.txt`
`parseChecker`	Generates a list of successfully parsed workflow file paths	`result/python/parsed_workflows_list.txt`

Since the base types in @actions/workflow-parser do not include with or step_len fields, the patches in patches/ add _with and step_len respectively.

`src/package-parser.py` — Python Version Matching

Compares python_version_result.txt (versions declared in workflows) against each repository's pyproject.toml / .python-version (declared required version).

Mismatch: result/python/version_not_matched_workflows.txt
Pre-release versions: result/python/prerelease_workflow_versions.txt
Unparseable versions: result/python/invalid_workflow_versions.txt

`src/package-parser.ts` — Node.js Version Matching

Compares node_version_result.txt against each repository's package.json engines.node / volta.node / .nvmrc.

Version mismatch: result/node/1. version_not_found_workflows.txt
Odd major version (non-LTS): result/node/2. odd_version_workflows.txt

`src/use-analyzer.ts` — Reusable Action Commit Validation

Reads use_command_result.txt and validates each owner/repo@sha action reference against the GitHub Commits API (GITHUB_TOKEN / GH_TOKEN). References whose commits cannot be found are written to result.txt.

`src/config.ts` — Paths and API credentials

Loads .env from the package directory, resolves relative paths against that root (pwd when using npm scripts), and builds GitHub API headers. Detection code stays free of machine-specific absolute paths.

`result/*/update.py` — CSV Aggregation

Extracts repository folder names from each txt result file and appends per-category counts as columns to py-github-stars.csv / ts-github-stars.csv.

Result Folder Contents

`result/python/` — Python Repository Analysis Results

File	Contents
`parsed_workflows_list.txt`	Absolute paths of all successfully parsed workflow files (6,545 entries)
`1. version_not_matched_workflows.txt`	Workflows where the Python version in the workflow does not match `requires-python` in `pyproject.toml`
`3. reuse_action_not_found_workflows copy.txt`	Workflows containing reusable actions whose commits could not be found via the GitHub API
`4. reuse_action_not_exist_workflows.txt`	Workflows referencing reusable action repositories that do not exist
`5. invalid_self_host_workflows.txt`	Workflows where `self-hosted` is not the first element in the `runs-on` array
`6. readability_issue_workflows.txt`	Workflows containing jobs with unnamed steps longer than 300 characters
`py-github-stars.csv`	Metadata and per-category issue counts for 1,000 Python repositories
`update.py`	Aggregation script that reads txt results and updates CSV columns

CSV aggregation columns: workflow_count, version_not_matched_workflows, prerequisite_violation_workflows, reuse_action_not_found_workflows, reuse_action_not_exist_workflows, invalid_self_host_workflows, readability_issue_workflows

`result/node/` — TypeScript Repository Analysis Results

File	Contents
`parsed_workflows_list.txt`	Absolute paths of all successfully parsed workflow files (7,081 entries)
`1. version_not_found_workflows.txt`	Workflows where the Node.js version does not match `engines` in `package.json` or `.nvmrc`
`2. odd_version_workflows.txt`	Workflows using an odd (non-LTS) Node.js major version
`3. library_workflows.txt`	Workflows running lint/build/test without a preceding package install step
`4. reuse_action_not_found_workflows.txt`	Workflows containing reusable actions that failed GitHub API validation
`5. invalid_self_host_workflows.txt`	Workflows where `self-hosted` is not the first element in the `runs-on` array
`6. readability_issue_workflows.txt`	Workflows containing jobs with unnamed steps longer than 300 characters
`ts-github-stars.csv`	Metadata and per-category issue counts for 1,000 TypeScript repositories
`update.py`	Aggregation script that reads txt results and updates CSV columns

CSV aggregation columns: workflow_count, version_not_found_workflows, odd_version_workflows, prerequisite_violation_workflows, reuse_action_not_found_workflows, invalid_self_host_workflows, readability_issue_workflows

Reproducibility

Addresses manuscript Major Comment 9 for the tool side. Corpus clone URLs and pinned commits are shipped here as corpus.gitmodules + corpus-pins.tsv; on-disk clones are produced by npm run fetch:corpus.

Immutable tool identifier

Field	How to obtain
Tool version	`VERSION` (also `package.json` `version`)
Git commit	`git -C analyzer/Action-Analyzer rev-parse HEAD`
Dependency lock	`package-lock.json` (+ SHA-256 in manifest)
Parser patches	`patches/@actions+workflow-parser+0.3.19.patch`, `patches/@actions+expressions+0.3.20.patch`

npm run repro:manifest
# → tool-manifest.json

Recommended permanent release

Tag this package: git tag -a action-analyzer-vX.Y.Z <commit>.
Attach tool-manifest.json, corpus.gitmodules, and corpus-pins.tsv in the GitHub Release notes.
Deposit the tagged source + corpus metadata on Zenodo (see Getting a DOI below) and cite the DOI in the manuscript.
Optionally deposit a separate full-clone tarball DOI; do not mix mutable live GitHub main with the paper numbers.

Getting a DOI (what it is and how to mint one)

A DOI (Digital Object Identifier) is a permanent ID for a fixed snapshot of a digital object (software release, dataset tarball, etc.). Unlike a GitHub URL, it still resolves after repos move or main changes. Reviewers asking for a DOI want an archival copy of exactly the tool/corpus used in the paper.

You typically need two DOIs (or one combined deposit with two versioned files):

Deposit	What to archive	Fills manuscript placeholder
Tool	This `Action-Analyzer` release + `tool-manifest.json` + `VERSION`	`[TOOL_DOI]`
Dataset	This repo’s `corpus.gitmodules` + `corpus-pins.tsv` (and/or a tarball of frozen clones)	`[DATASET_DOI]`

Easiest path for GitHub code: Zenodo ↔ GitHub

Make the GitHub repo public (Zenodo’s GitHub integration only archives public repos).
Sign in at zenodo.org with Log in with GitHub and authorize access.
Open Zenodo → GitHub settings, click Sync now, and toggle On for:
- MGPOCKY/Action-Analyzer (tool + corpus.gitmodules / corpus-pins.tsv).
On GitHub, create a Release (e.g. tag v1.0.0) that contains the exact commit you evaluated. Include tool-manifest.json (run npm run repro:manifest first) in that commit.
Zenodo archives that release and mints a DOI (often within a few minutes). You get:
- a version DOI (cite this in the paper for the exact snapshot), and
- a concept DOI (points at all versions; optional for “latest”).
Copy the version DOI into the manuscript / manuscript_comment9_fragment.tex placeholders ([TOOL_DOI], [DATASET_DOI]).
Optional: add a DOI badge to the README and a CITATION.cff once the DOI exists.

Dataset / large clone trees

Prefer archiving pin metadata (corpus.gitmodules + corpus-pins.tsv) plus instructions (npm run fetch:corpus) rather than uploading ~2000 full clones if size is an issue.
If the journal wants the full trees: upload a compressed tarball (or multiple Zenodo records) via Zenodo’s New upload UI, fill creators/title/license, publish → DOI.
Record the collection date (UTC) in the Zenodo description and in [COLLECTION_DATE_UTC].

After you have DOIs

# regenerate fingerprint for the released commit
npm run repro:manifest
# then fill manuscript_comment9_fragment.tex:
#   [TOOL_VERSION] [TOOL_COMMIT] [TOOL_DOI]
#   [DATASET_VERSION] [DATASET_DOI] [COLLECTION_DATE_UTC]

Inputs this tool expects

Input	Role
`corpus.gitmodules`	Clone URLs + paths (shipped in this repo)
`corpus-pins.tsv`	Pinned commit SHA per repo (shipped in this repo)
`CWC_TS_REPOS` / `CWC_PY_REPOS`	On-disk clone trees after `npm run fetch:corpus`

Fetch with npm run fetch:corpus (see Experiment corpus above). Action-Analyzer does not re-select or re-pin repositories.

Repository selection, language classification, and exclusion policy

Implemented in the parent study tree (scripts/clone_ts_repos.js, scripts/clone_py_repos.js, star/metadata collectors). For the manuscript, report at least:

Item	Study policy (record exact values in the dataset release)
Collection date	UTC date when Search API pages were fetched
API	`GET /search/repositories` + Contents API for `.github/workflows`
Language classification	GitHub Search `language:TypeScript` / `language:Python` (GitHub linguist)
Forks	Excluded (`fork:false`)
Archived repositories	Excluded (`archived:false`)
Mirrors	Not specially queried; GitHub `mirror_url` should be filtered if present in metadata dump
Duplicate projects	One clone per `owner/repo`; folder name `owner__repo`
Generated workflows	Not excluded a priori; all `.yml`/`.yaml` under `.github/workflows` are parsed
Stars / ranking	Exact query string, `sort=stars`, `order=desc`, and any star thresholds
Workflow presence	Only repos with a non-empty `.github/workflows` on the default branch

Keep the exact query strings, page parameters, and collection timestamp in the parent dataset README / manifest. This package consumes the resulting frozen trees only.

Tool runtime parameters

Parameter	Value / source
Node.js	`engines.node`: `lts/*` (record exact `node -v` in `tool-manifest.json`)
Workflow parser	`@actions/workflow-parser@0.3.19` + local patch (`_with`, `step_len`)
Expressions	`@actions/expressions@0.3.20` + local patch
Readability threshold	`step_len > 300` and job has `steps.length > 1` and step has no `name`
Reuse validation API	`GET /repos/{owner}/{repo}/commits/{ref}`
API headers	`Accept: application/vnd.github+json`, `X-GitHub-Api-Version: 2022-11-28`
Auth	`Authorization: Bearer $GITHUB_TOKEN` (or `GH_TOKEN`)

End-to-end commands (parsing → detection → aggregation)

cd analyzer/Action-Analyzer
npm ci                          # deterministic install + patch-package
npm run fetch:corpus            # python-repos + typescript-repos at pinned SHAs
# optional: cp .env.example .env  and set GITHUB_TOKEN for reuse-validate

npm run repro:manifest          # optional: record tool fingerprint
npm run pipeline:typescript
npm run pipeline:python

Individual stages:

npm run analyze -- config
npm run analyze -- parse --lang python
npm run analyze -- python-version
npm run analyze -- match-python
npm run analyze -- readability --lang typescript
npm run analyze -- reuse-validate    # needs GITHUB_TOKEN

Manual validation

Manual validation artifacts for the paper (sampling protocol, spreadsheets, agreement notes) should live with the dataset / study archive. From this tool:

Use the category txt files under result/{node,python}/ as the sampling frame.
Open the referenced workflow paths inside the frozen clones (same commit as the archive).
Record decisions without re-fetching main from GitHub at a later date.
Store the validation sheet next to the dataset DOI materials.

Why a snapshot is required

GitHub rankings, default-branch workflows, action SHAs, and Linguist labels change continuously. Paper numbers must be tied to: (dataset DOI + pinned clone commits) × (tool tag/commit + tool-manifest.json).

Manuscript checklist (Comment 9)

Repository-collection date (UTC) stated
Dataset version / DOI / immutable ID stated
Tool commit or release tag (+ DOI/SWHID) stated
Search / Contents / Commits API parameters stated
Language-classification procedure stated (GitHub Search linguist labels)
Treatment of forks, mirrors, archived repos, duplicates, generated workflows stated
Scripts + instructions for selection (parent), parsing, detection, manual validation, result generation linked
tool-manifest.json included in the tool archive

Name		Name	Last commit message	Last commit date
Latest commit History 14 Commits
.github/workflows		.github/workflows
patches		patches
result		result
scripts		scripts
src		src
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
VERSION		VERSION
corpus-pins.tsv		corpus-pins.tsv
corpus.gitmodules		corpus.gitmodules
eslint.config.ts		eslint.config.ts
package-lock.json		package-lock.json
package.json		package.json
tool-manifest.json		tool-manifest.json
tsconfig.json		tsconfig.json

Folders and files

Latest commit

History

Repository files navigation

Action-Analyzer

Experiment corpus (repos & versions)

How each repo is named and located

How the version (commit) is fixed

Fetch the corpus (one command)

Smoke test with a few repos (--limit)

Project Structure

How to Run

1. Install dependencies

2. Fetch the frozen corpus

3. Preferred: CLI tasks (no source edits)

4. Legacy: direct module execution

5. Update CSV aggregations

Analysis Pipeline

Core Code

src/index.ts — Main Analyzer

src/package-parser.py — Python Version Matching

src/package-parser.ts — Node.js Version Matching

src/use-analyzer.ts — Reusable Action Commit Validation

src/config.ts — Paths and API credentials

result/*/update.py — CSV Aggregation

Result Folder Contents

result/python/ — Python Repository Analysis Results

result/node/ — TypeScript Repository Analysis Results

Reproducibility

Immutable tool identifier

Getting a DOI (what it is and how to mint one)

Inputs this tool expects

Repository selection, language classification, and exclusion policy

Tool runtime parameters

End-to-end commands (parsing → detection → aggregation)

Manual validation

Why a snapshot is required

Manuscript checklist (Comment 9)

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages