Rush

Your office escape advisor. Rush tells you the perfect moment to flee the building — before the sky opens up, the trains break down, and everyone else has the same idea.

Full documentation and peer review manual: javihslu.github.io/rush

Overview

You finish at the office and want to know if you should leave now or wait. Rush checks the traffic that's normal for this hour on your route and the precipitation forecast, then tells you if it's going to be heavier or lighter than usual.

What it uses:

• 2 years of hourly car counts from the Stadt Zürich open data portal
• Open-Meteo archive for the matching weather, and the forecast for "right now"
• OSRM to draw the driving route between two coordinates

What you get: an expected duration and a short verdict (leave now / wait / usual conditions).

How it works

flowchart TD
    A["Stadt Zürich MIV counts (2y)"] --> B["dlt backfill"]
    W["Open-Meteo archive (2y)"] --> B
    B --> C[("raw schemas")]
    C --> D["dbt staging"]
    D --> M1["mart_traffic_baseline"]
    D --> M2["mart_weather_effect"]
    M1 --> R["recommender (on demand)"]
    M2 --> R
    F["Open-Meteo forecast"] --> R
    O["OSRM route"] --> R
    R --> V["verdict + expected minutes"]
    R --> U["streamlit UI"]

Loading

No model is trained. The marts are lookups: for each counter, hour-of-day, day-of-week and month we store the average count, and for each precipitation bucket we store a coefficient against that baseline. The recommender pulls the counters along your route, multiplies the baseline by the weather coefficient, and turns the result into a duration.

More detail in docs/use-case.md and docs/transformation.md.

The local stack ships with two Airflow DAGs: rush_backfill (manual, one-shot) and rush_daily (refreshes the current year + last week of weather, then runs dbt).

The cloud path is the same scripts in two Cloud Run Jobs (rush-ingest, rush-dbt) triggered by Cloud Scheduler, plus a Streamlit Cloud Run Service (rush-ui) for the recommender form. All of that lives in terraform/.

Quick Start

One command sets up everything on macOS, Linux, or Windows (WSL):

bash <(curl -fsSL https://raw.githubusercontent.com/javihslu/rush/main/install.sh)

This clones the repository, checks for required tools (installing anything missing), and starts the full Docker stack. If you already have the repo cloned, run ./setup.sh from inside it.

Windows

1. Open PowerShell as Administrator and install WSL 2:

   wsl --install

2. Restart your computer
3. Install Docker Desktop (enable WSL 2 backend in settings)
4. Open your WSL terminal (Ubuntu) and run the command above

What the setup does:

1. Checks for Git, Docker, gcloud CLI, and Terraform -- offers to install anything missing
2. Reads config.yaml and generates a .env file for Docker Compose
3. Builds and starts all Docker containers (docker compose up -d --build)
4. If gcloud is available, runs scripts/setup-gcp.sh for cloud onboarding
5. Prints service URLs when everything is ready

Once running:

• Airflow: http://localhost:8080 (workflow orchestration UI)
• pgAdmin: http://localhost:8085
• PostgreSQL: localhost:5432

If you have port conflicts, stop the other containers before running setup.sh.

To stop: docker compose down

To fully clean up (containers, volumes, images, generated files):

./teardown.sh

Tech Stack

• Ingestion: dlt, Python — backfill_traffic.py, backfill_weather.py, daily incremental refresh through Airflow
• Orchestration: Apache Airflow locally (rush_backfill, rush_daily); Cloud Run Jobs + Cloud Scheduler in the cloud
• Warehouse: PostgreSQL (local) / BigQuery (cloud)
• Data Lake: Google Cloud Storage
• Transformation: dbt (dual-target: PostgreSQL + BigQuery)
• UI: Streamlit on Cloud Run Service
• Routing: public OSRM demo, haversine fallback
• Infrastructure: Docker, Terraform

Data Sources

• Stadt Zürich open data — MIV hourly counts — one CSV per year, 215+ counter stations across Zurich
• Open-Meteo — archive (2y history) + forecast (next hours)
• OSRM public demo — driving route between two coordinates

Project Structure

rush/
  pipelines/
    backfill/
      backfill_traffic.py     # Stadt Zürich MIV counts (N years -> Postgres)
      backfill_weather.py     # Open-Meteo archive (N years -> Postgres)
    ingestion/
      weather.py              # Open-Meteo forecast (daily refresh)
    transformation/
      dbt/
        models/
          staging/            # raw -> cleaned (counts, counters, weather)
          marts/              # mart_traffic_baseline, mart_weather_effect
    recommend/
      route.py                # OSRM call + haversine fallback
      snap.py                 # LV95 -> WGS84, counters within 300m of route
      lookup.py               # read the marts
      weather.py              # forecast precip at a point
      recommend.py            # glue + CLI
    ui:
      app.py                  # streamlit recommender form
  dags/
    rush_pipeline.py          # rush_backfill + rush_daily
  terraform/                  # GCS, BigQuery, Cloud Run jobs + service, scheduler
  scripts/
    setup-gcp.sh
    run-ingest.sh             # entry for rush-ingest job
    run-dbt.sh                # entry for rush-dbt job
  config.yaml                 # single source of truth
  config.py                   # loader (postgres or bigquery via RUSH_TARGET)
  Dockerfile                  # local dev container
  Dockerfile.ingest           # rush-ingest cloud run job
  Dockerfile.dbt              # rush-dbt cloud run job
  Dockerfile.ui               # rush-ui cloud run service
  docker-compose.yaml
  setup.sh
  teardown.sh
  pyproject.toml

Configuration

All configuration lives in config.yaml — the single source of truth. setup.sh generates .env from it for Docker Compose. Python code reads it directly.

project:
  name: rush

database:
  user: root
  password: root
  name: rush
  host: pgdatabase
  port: 5432

pgadmin:
  email: admin@admin.com
  password: root

airflow:
  user: airflow
  password: airflow

gcp:
  region: europe-west6

To apply changes: edit config.yaml, delete .env, and re-run ./setup.sh.

In Python:

from config import cfg

db_host = cfg["database"]["host"]

Development

All development runs inside Docker. No local Python installation required.

VS Code + Dev Containers (recommended)

1. Install VS Code (free)
2. Install the Dev Containers extension
3. Open the rush folder in VS Code
4. Click "Reopen in Container" when prompted (or run Dev Containers: Reopen in Container from the command palette)

VS Code will build the container, start PostgreSQL and pgAdmin, install all dependencies, and open a terminal inside the dev environment. Python autocomplete, debugging, and Jupyter all work out of the box.

Terminal only (no VS Code)

If you prefer your own editor, just use Docker directly:

# run a script
docker compose run --rm dev uv run python pipelines/ingestion/transport.py

# open a shell inside the container
docker compose run --rm dev bash

# run jupyter
docker compose run --rm -p 8888:8888 dev uv run jupyter notebook --ip=0.0.0.0 --no-browser --allow-root

Source code is mounted from your host, so edits in any editor are reflected immediately.