Open-Source Release Intelligence

A data pipeline and analytics application for collecting GitHub release and issue activity, assigning issues to release periods, measuring support and reliability trends, classifying issue descriptions, detecting recurring categories, forecasting weekly support demand, and producing metric-grounded release summaries.

The project is designed for observational analysis. It reports that a release was followed by a measured issue volume; it does not claim that the release caused those issues.

What is implemented

• Authenticated GitHub REST client with pagination, retries, exponential backoff with jitter, primary/secondary rate-limit handling, request logging, conditional requests, and configurable pauses.
• Append-only JSONL raw archive by repository and entity.
• SQLite operational checkpoint store with endpoint/repository progress, page recovery, incremental updated_at watermarks, request logs, ETags, and comment-completion state.
• DuckDB raw tables with idempotent MERGE loading, including extraction checkpoints and the HTTP request log.
• dbt staging, dimensions, facts, release cohorts, repository-health marts, category trends, weekly-demand features, and tests.
• Manual-labeling sample generation, keyword baseline, majority baseline, TF-IDF/logistic regression, structured-output LLM classifier, fixed holdout evaluation, confidence and cost fields, and low-confidence review support.
• Previous-week, four-week-moving-average, Ridge, and gradient-boosting forecasting with rolling-origin validation and release-week/non-release-week errors.
• Six Streamlit analytical pages plus a data-operations page.
• Airflow DAG and GitHub Actions CI.
• Synthetic demo generator so the full analytical layer and dashboard can be tested without consuming GitHub API quota.

Default repository set

Repository	Category
MarlinFirmware/Marlin	3D-printer firmware
espressif/arduino-esp32	Microcontroller / IoT framework
micropython/micropython	Embedded runtime
platformio/platformio-core	Embedded development tools
hathach/tinyusb	Embedded USB driver stack

The selection rationale and limitations are documented in reports/repository_selection.md. Run the audit against candidates:

release-intel audit

The audit records approximate release, closed issue/PR, and label counts. Because GitHub's Issues endpoint includes pull requests, the audit is a screening step; the production loader filters pull requests using the pull_request key.

Architecture

GitHub REST API
       |
       v
Python extraction client
       |
       +--> append-only JSONL archive
       +--> SQLite extraction state + request logs
       |
       v
DuckDB raw schema
       |
       v
dbt staging -> dimensions/facts -> analytical marts
       |                              |
       v                              v
Issue classifiers               Forecasting models
       |                              |
       +---------------+--------------+
                       |
                       v
               Streamlit dashboard

Local setup

Python 3.11 or newer is required.

git clone <your-repository-url>
cd open-source-release-intelligence
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
cp .env.example .env

Create a fine-grained GitHub token that can read public repository metadata, issues, and releases, then set GITHUB_TOKEN in .env. Never commit the token.

Fast demo

This creates three synthetic repositories, 54 releases, multiple years of issues/comments, DuckDB raw tables, dbt marts, and dashboard-ready data.

release-intel demo
streamlit run dashboard/Home.py

Production workflow

1. Audit repository candidates

release-intel audit --output reports/repository_audit.csv

Review the CSV and update config/repositories.yml. The dashboard flags a repository above 45% of current issue volume.

2. Historical and incremental extraction

release-intel extract                         # all stages
release-intel extract --stage issues          # one stage across the portfolio
release-intel extract --repository hathach/tinyusb --stage comments

The staged commands are what the Airflow DAG uses. The first run starts from extraction.issues_since. Later runs use the endpoint/repository last_successful_updated_at watermark. If a run fails after page N, the next run resumes at page N+1. Raw pages are append-only; duplicate issue/release IDs are reconciled in DuckDB.

Label snapshots are processed only for issues newer than the label watermark; empty snapshots remove labels deleted upstream. Comments are processed only for issues whose updated_at changed and are capped per repository per run. This prevents the initial comment crawl from exhausting API quota. Increase or remove the cap after the core issue history is loaded.

3. Load raw tables and build marts

release-intel load
cd dbt
dbt seed --profiles-dir .
dbt build --profiles-dir .
cd ..

4. Create the manual labeling set

python scripts/create_labeling_sample.py --size 320

Complete these fields in data/labels/manual_issue_labels.csv:

• manual_category
• labeler_notes
• is_ambiguous
• labeled_at

Freeze the taxonomy before model evaluation. Aim for at least 30 examples in each major category when feasible.

5. Train and evaluate non-LLM baselines

python scripts/train_classical.py

Outputs include baseline metrics, confusion matrices, a fixed holdout prediction file, and data/models/tfidf_logistic.joblib.

Evaluate the LLM on that same frozen holdout, then estimate a classical/LLM hybrid policy:

python scripts/evaluate_llm_holdout.py --limit 100
python scripts/evaluate_hybrid_policy.py

6. Classify new or modified issues

python scripts/classify_new_issues.py --model keyword
python scripts/classify_new_issues.py --model classical
python scripts/classify_new_issues.py --model llm --limit 100

The classifier skips an issue when the same model already processed the same issue_id and source updated_at. LLM use requires OPENAI_API_KEY. Price assumptions are intentionally not hard-coded; pass model-specific token prices in code or a project configuration before reporting cost.

After writing predictions, rerun dbt so fact_issues uses the latest prediction:

cd dbt && dbt build --profiles-dir . && cd ..

7. Forecast weekly issue-report demand

python scripts/build_forecast.py

The report compares:

• previous-week baseline
• four-week moving average
• Ridge regression
• histogram gradient boosting

Metrics include MAE, RMSE, WAPE, bias, release-week MAE, and non-release-week MAE. The selected model also produces repository-level forecasts for the next four weeks and stores both validation and future predictions in DuckDB. The target is issue-reporting workload, not latent defects or future release quality.

Generate deterministic release summaries after the marts are refreshed:

python scripts/generate_release_summaries.py

8. Update project metrics

python scripts/update_project_metrics.py

Release-cohort method

• published_at is the release timestamp.
• Each issue is assigned to the most recent preceding non-draft release.
• This prevents one issue from appearing in multiple post-release cohorts.
• Nominal windows are 14, 30, and 60 days.
• If a new release occurs before the nominal window ends, the effective window ends at the next release and has_overlapping_release_window = true.
• The 14-day pre-release period is descriptive and may overlap the previous release's post period; it is flagged rather than hidden.
• Raw and normalized values are shown. Stars are an exposure proxy, not a measure of installed devices or active users.

Main data model

Dimensions:

• marts.dim_repository
• marts.dim_release
• marts.dim_date
• marts.dim_issue_category
• marts.dim_label

Facts:

• marts.fact_issues
• marts.fact_issue_labels
• marts.fact_release_cohorts

Analytical marts:

• marts.mart_release_quality
• marts.mart_issue_resolution
• marts.mart_category_trends
• marts.mart_repository_health
• marts.mart_weekly_issue_demand
• marts.mart_classifier_evaluation

Dashboard pages

1. Portfolio Overview
2. Release Intelligence
3. Reliability Categories
4. Repository Health
5. Classification Evaluation
6. Support Forecast
7. Data Operations

Airflow

Mount the project at /opt/airflow/project, install the package in the Airflow environment, and copy or symlink airflow/dags/release_intelligence_dag.py into the configured DAG folder. The DAG runs daily, serializes repository extraction through the GitHub client's pacing/rate-limit logic, loads DuckDB, runs dbt, classifies only changed issues, builds the forecast, and publishes metrics.

Tests and CI

ruff check src tests scripts dashboard
pytest
cd dbt && dbt build --profiles-dir .

CI creates synthetic data before the dbt build, so SQL tests run without GitHub credentials.

Responsible reporting examples

Use:

Release vX.Y was followed by 86 new issues in its effective 30-day observation window, compared with 41 issues in the preceding 14 days after rate normalization.

Avoid:

Release vX.Y caused 86 defects.

Also report close release spacing, repository size, category coverage, event availability, and incomplete comment/event collection whenever they affect interpretation.

Dashboard Preview

The screenshots below show an end-to-end smoke test using the hathach/tinyusb repository, covering 18 releases and 844 GitHub issues.

Portfolio Overview

Release Intelligence

Repository Health

Current smoke-test limitations: Reopen-rate metrics are unavailable because issue-event history has not yet been collected. Forecast metrics appear only after the forecasting workflow has been run. The dashboard reports temporal associations and does not claim that releases caused subsequent issue activity.