railpull — Indian Railways timetable toolkit

Pull the current Indian Railways timetable straight from the source, and turn it into clean, analysis-ready data. A polite, resume-safe crawler for India's National Train Enquiry System (NTES), an exporter that flattens the result into tidy CSVs (with the running-days each train actually operates), a live train-delay poller, and optional OpenStreetMap tooling that fills in station coordinates and draws each route along the real tracks.

No API key. No paid service. Python for the core, Node for the optional map bits.

What this is for: research, analysis, hobby maps, learning. It reads a public government enquiry service through an unofficial client — please use it gently and read Responsible use before you run it.

---

Why this exists

India runs one of the largest railway networks on earth, but there's no clean, current, official open dataset of the timetable. The public data floating around is mostly a community snapshot from ~2016 — no Vande Bharat, renamed stations, changed schedules, and every train wrongly assumed to run daily. NTES has the current data, but only one train (or one station board) at a time.

railpull collects that, politely, and hands you the whole timetable as files you can actually use.

What you get

Run the crawler, then the exporter, and data/out/ fills with:

File	One row per	Key columns
trains.csv	train	number, name, type, runs_days, source, destination, distance, stops
stops.csv	stop	train, seq, station code + name, day offset, arrival, departure, halt
stations.csv	station	code, name, lat, lon (coords filled by the OSM step)
schedules.jsonl	train	the full nested record, one JSON object per line

The column you can't get anywhere else easily is runs_days — NTES returns the exact dates a train is scheduled over the coming weeks, and the exporter folds those into the weekdays it runs (Daily, or Mon,Wed,Fri). That's how you tell a daily express from a biweekly special.

Add the optional OpenStreetMap step and you also get real coordinates for every station and tracks.geojson — each station-to-station hop drawn along the actual rails, ready to drop on a map.

Quickstart

# 1. core: crawl + export  (Python)
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python ntes/crawl.py          # polite, resumable; a few hours for the full roster
python transform/export.py    # -> data/out/*.csv, schedules.jsonl

# 2. live delays (optional, Python) — one snapshot of who's late right now
python ntes/poll_delays.py    # -> data/out/delays.json

# 3. coordinates + track geometry (optional, Node + a 1.7 GB OSM download)
npm install
curl -O https://download.geofabrik.de/asia/india-latest.osm.pbf
node osm/geocode_stations.mjs india-latest.osm.pbf              # fills station lat/lon
node --max-old-space-size=4096 osm/route_tracks.mjs india-latest.osm.pbf  # -> tracks.geojson

Nothing is committed to the repo — every output is regenerated by these scripts.

How it works

The crawler (ntes/crawl.py)

NTES has no "list all trains" endpoint, so the crawler finds them itself:

1. Discovery. It searches number prefixes — 000, 001, … 999. NTES caps search results at 60, so any prefix that comes back full is drilled one digit deeper (125 -> 1250…1259). The union of everything returned is the train roster (~12,000 numbers; about a fifth turn out to be discontinued).
2. Schedules. It fetches the full stop list for each number and writes one JSON file per train. Re-runs skip files already on disk, so the crawl resumes safely after any interruption — close your laptop, run it again tomorrow.

It's deliberately slow (~1 request every ~1.2s, with jitter and backoff) and checkpoints its progress to data/raw/crawl-status.json, which you can watch:

watch -n 30 'cat data/raw/crawl-status.json'

The exporter (transform/export.py)

Reads the raw per-train JSON and writes the tidy tables above. Pure Python, no dependencies. The interesting bit is runs_days (see above) and mapping NTES's terse type codes to labels (VNDB -> Vande Bharat, SUF -> Superfast, …).

The delay poller (ntes/poll_delays.py)

One station board lists every train passing through in a time window, each with its live delay and cancellation flag. So instead of querying thousands of trains, the poller sweeps a few hundred busy junctions (seeded from ntes/major_stations.json) and merges the result into a single delays.json:

{ "updatedAt": 1783683912, "source": "ntes-station-boards",
  "trains": { "12951": {"d": 18}, "12009": {"c": 1} } }

{"d": 18} = 18 minutes late; {"c": 1} = cancelled. Run with --loop to keep it fresh every ~5 minutes.

OpenStreetMap tooling (osm/, optional)

• geocode_stations.mjs scans the OSM India extract once and fills station coordinates. OSM has ~17,000 Indian railway stations, ~11,000 tagged with the official station code, so most match directly by code; the rest fall back to a name match.
• route_tracks.mjs builds a graph of the rail lines (~600k nodes), snaps each station onto it, and runs Dijkstra between every pair of stations that are consecutive stops on some train — then simplifies and writes GeoJSON. Detours that look implausibly long are dropped rather than drawn wrong.

Field notes (things that cost time, so they don't cost you time)

• Read timeouts are mandatory. The enquiry server sometimes accepts a connection and then goes silent forever; without a socket timeout one dead read hangs the whole crawl. (Handled — but if you fork the client, keep it.)
• Search rejects queries shorter than 3 characters. Hence 3-digit prefixes.
• Station boards only accept look-ahead windows of 2, 4, or 8 hours. Other values error.
• A semantic "no" is not a network error. A bad query or a discontinued train number returns a clean error — fail fast on those, only back off on real network trouble, or your crawl crawls.
• Type codes are NTES's, not the old community ones — SUF (superfast, not SF), VNDB/VNDM/VNDS (Vande Bharat), DRNT (Duronto), GBR (Garib Rath), MEX (Mail/Express), SUB (suburban). Miscategorize these and your "superfast" bucket comes out empty.
• Board delays can be stale garbage. A board occasionally reports a train as "57:18" late — a ghost of a service long gone. The poller caps believable delays at 12 hours.
• Some station codes changed. Mughal Sarai is now DDU, Allahabad is Prayagraj. Fresh crawls use the current codes; if you mix in older data, expect a few renames.

Responsible use

This talks to a public government service through an unofficial, reverse-engineered client (ntes-client). Be a good guest:

• Keep the request rate gentle (the defaults already are — don't lower the pause).
• Run the full crawl once and cache it; it's a timetable, not a live feed. It changes on the order of weeks.
• This is for personal, research, and educational use. The schedules are public facts, but bulk collection and redistribution of the data may run against the operator's terms of use — which is exactly why this repo ships the tools and not a dataset. If you publish data you collect, that's your call and your responsibility.
• No affiliation with Indian Railways, IRCTC, CRIS, or NTES.

Data licenses & credits

• Code: MIT (see LICENSE).
• Schedules: fetched from NTES via ntes-client (MIT). Public factual data; see Responsible use.
• Station coordinates & track geometry: derived from OpenStreetMap, © OpenStreetMap contributors, ODbL.
• The major-station seed list was bootstrapped from the CC0 datameet/railways community dataset.

Repo topics

If you're browsing and this was useful, the tags that describe it: indian-railways · ntes · irctc · train-schedule · timetable · railway · india · dataset · open-data · geospatial · openstreetmap

Built by shwetank. Sibling project: a live map that renders this data as every train moving across India at once.