One Python API for Socrata, CKAN, ArcGIS and OpenDataSoft open data portals.
Nearly every government open data portal runs on one of four platforms. Each has its own client, its own pagination bugs, and its own way of losing your data quietly. portalkit detects which platform a URL is running and gives you the same API either way, returning pandas or geopandas.
pip install portalkit # add [geo] for geometry support
pip install 'portalkit[geo]'Requires Python 3.10+.
import portalkit as pk
df = pk.load("chicago/311", limit=5000) # a known dataset, by name
portal = pk.open("https://data.sfgov.org") # or any portal URL
portal.search("evictions")
gdf = portal.dataset("6z8x-wfk4").to_geopandas(where="file_date > '2026-01-01'")pk.open works out which software the portal runs by asking it. You do not have to know, and you
do not need a different library per city.
sodapy, the client most people reach for, has been unmaintained since 2022 and only speaks
Socrata. ckanapi is a faithful wrapper around CKAN's Action API with no opinion about
DataFrames. ArcGIS means either Esri's SDK or writing the pagination yourself. Most projects end
up with three half-written clients carrying the same three bugs.
Point geopandas.read_file() at an ArcGIS FeatureServer layer and you get 1000 rows back with no
error and no warning, because that is the server's maxRecordCount. Socrata's default is the
same. Analyses get published on top of that number.
Paging past it is not just a loop. Without an explicit sort, resultOffset and $offset are free
to return the same row twice and never show you another, so both need a stable order: :id on
Socrata, the object id field on ArcGIS. Some ArcGIS layers cannot page at all and have to be
fetched through returnIdsOnly and batched where clauses. OpenDataSoft caps offset at 10,000
and errors past it, so anything larger goes through its export endpoint.
portalkit does all of that and checks the result against the row count the portal itself reports.
A layer's advertised maxRecordCount describes what it will return for cheap rows. Ask for the
same number of full-resolution polygons and the server returns a 500 wrapped inside a 200 body,
which a naive client reads as "no more rows" and stops. There is no way to know the real ceiling
except to find it, so portalkit halves the page size and retries until the server copes, then
keeps the smaller size. It also requests six decimal places of coordinate precision, which is
about ten centimetres and roughly halves a polygon response.
Socrata returns every value as a string, dates included. Letting pandas infer afterwards turns ZIP
code 02134 into 2134.0. The portal already published a schema saying which column is an
identifier, so portalkit uses it. When the schema turns out to be wrong, which happens, the column
comes back untouched rather than as a column of nulls, so you can see the mess and decide.
Socrata sends GeoJSON, ArcGIS sends rings and paths, OpenDataSoft wraps a GeoJSON Feature in
one column and its centroid in another. All of it converts to shapely with the CRS set.
Esri packs every ring of a shape into one flat list and distinguishes exteriors from holes by winding direction. Treating everything after the first ring as a hole silently deletes islands: on the Census state boundaries layer, 11 of 56 states are multi-part. portalkit classifies rings by orientation and assigns holes to the shell that contains them.
When a dataset carries both a shape and its centroid, the shape wins.
| Platform | Detected by | Query language | Notes |
|---|---|---|---|
| Socrata | /api/catalog/v1 |
SoQL | Pass token= for an app token to lift the anonymous rate limit |
| CKAN | /api/3/action/site_read |
DataStore filters, SQL when where is given |
Only DataStore-backed resources are queryable |
| ArcGIS | /rest/services + f=json |
SQL-ish where |
FeatureServer and MapServer, by service or by layer URL |
| OpenDataSoft | /api/explore/v2.1/catalog/datasets |
ODSQL | Requests past 10,000 rows use the export endpoint |
| Call | Returns |
|---|---|
pk.open(url, kind=None, token=None, cache=True) |
A Portal |
pk.dataset("nyc/311") |
A Dataset from the catalog |
pk.load("nyc/311", **kwargs) |
A DataFrame, straight from the catalog |
pk.search(url, query, limit=20) |
A list of DatasetInfo |
portal.search(query, limit=20) |
A list of DatasetInfo |
portal.dataset(id) |
A Dataset |
dataset.info |
DatasetInfo, including columns and row count |
dataset.to_pandas(where=, select=, order=, limit=, cache=, progress=) |
A DataFrame |
dataset.to_geopandas(geometry=None, crs=4326, **kwargs) |
A GeoDataFrame |
where, select and order go to the portal in its own query language. Filtering server-side is
usually the difference between two seconds and five minutes.
Every tabular fetch is stored as Parquet, keyed by portal, dataset and the exact query, so re-running a notebook does not re-run the download.
| Control | Effect |
|---|---|
PORTALKIT_CACHE |
Cache directory, defaults to the platform cache dir |
cache=False |
Skip the cache for one call |
max_age=<seconds> |
Treat anything older as a miss |
portalkit clear-cache |
Empty it |
to_geopandas() does not cache. Shapely geometry does not survive a Parquet round trip, so it
would be a cache that never hits.
$ portalkit catalog chicago
reference dataset portal
------------------------ ----------------------- ------------------------------
chicago/311 311 Service Requests https://data.cityofchicago.org
chicago/building-permits Building permits https://data.cityofchicago.org
chicago/crimes Crimes, 2001 to present https://data.cityofchicago.org
$ portalkit info chicago/311
311 Service Requests
id v6vf-nfxy
portal https://data.cityofchicago.org (socrata)
rows 14,517,544
$ portalkit get chicago/311 --where "sr_type = 'Pothole in Street Complaint'" -o potholes.parquet
wrote 422,462 rows to potholes.parquet
| Command | What it does |
|---|---|
portalkit catalog [filter] |
List the built-in dataset references |
portalkit search <portal> <query> |
Search one portal |
portalkit info <ref> |
Show a dataset's columns and row count |
portalkit get <ref> -o <file> |
Download to .parquet, .csv, .json, .geojson or .gpkg |
portalkit clear-cache |
Delete everything cached |
pk.load("nyc/311") works because a small list of well-known datasets ships with the package. It
is not meant to be exhaustive, it is meant to save the five minute detour of discovering that
Chicago's 311 dataset is v6vf-nfxy. portalkit catalog lists what is there.
Adding your city is a pull request against src/portalkit/catalog.toml with a portal URL, a
dataset id and a title. For entries you would rather not publish, point PORTALKIT_CATALOG at
your own TOML file and it is merged over the built-in one.
CKAN is a catalogue first. Only resources pushed into its DataStore can be queried; everything
else is a file on disk. portalkit resolves a package to its first DataStore-backed resource, or
says plainly that there is not one and where the files are. Pass package:resource-id to pick a
specific one.
ArcGIS addressing follows the REST URLs. Give it a service and name a layer with
portal.dataset("0"), or give it the layer URL directly.
Large geometry downloads from municipal servers are genuinely slow. All 3,235 US counties at full
resolution is a few hundred megabytes. The CLI shows a running row count and to_pandas takes a
progress= callback. Filter server-side first where you can.
Rate limits belong to the portal, not to portalkit. One connection at a time, retries with jitter,
and Retry-After respected. Most of these portals are one box run by a team of two.
Bug reports and pull requests are welcome, especially catalog entries for new cities. uv sync
then uv run pytest to get started. uv run pytest -m live runs the tests that hit real portals.
MIT.