BEACONlist

A list of BEACONs scraped from https://de.wikipedia.org/wiki/Wikipedia:BEACON.

getting started

Python venv

Windows

python -m venv .venv
.\.venv\Scripts\activate
pip install -r requirements.txt

Linux/macOS

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

workflow

1) makeBEACONlist.py

• stores e.g. BEACONlist_20251203-1653_scraped.json in /data/scraped

python src/makeBEACONlist.py

2) refineBEACONlist.py

• refine scraped list (!data model work in progress!) from /data/scraped into e.g. BEACONlist_20251203-1653_refined.json in /data/refined

python src/refineBEACONlist.py

3) analyseBEACONlist.py

• get some insights, by default uses the last files found in /data/scraped and /data/refined

python src/analyseBEACONlist.py

4) optional: compareBEACONlist.py

• compares the latest 2 _refined.json from /data/refined (e.g. added/removed/modified BEACONs)
• only works/makes sense if you have done 2 scrapings (+ refined) locally

python src/compareBEACONlist.py

• if you want to pick which files to compare:

python src/compareBEACONlist.py --pick

5) latestBEACONlist.py

• copies the latest files from /data/scraped and /data/refined to...latest folder => _refined.json will be base for aggregation

python src/latestBEACONlist.py

BEACONlist_scraped.json e.g.

{
    "provider_type": "Bibliotheken, Verbünde, Archive",
    "beacon_url": "http://www.bib-bvb.de/OpenData/beacon_bvb01.txt",
    "beacon_name": "Titelaufnahmen des B3Kat",
    "beacon_description": "Titelaufnahmen des B3Kat (gemeinsamer Verbundkatalog des BVB und des KOBV) (> 3.790.000 Einträge, Beispiel: Martin Schrettinger )",
    "beacon_raw": "<li><a class=\"external text\" href=\"http://www.bib-bvb.de/OpenData/beacon_bvb01.txt\" rel=\"nofollow\">Titelaufnahmen des B3Kat</a> (gemeinsamer Verbundkatalog des BVB und des KOBV) (&gt; 3.790.000 Einträge, Beispiel: <a class=\"external text\" href=\"https://bvbat01.bib-bvb.de/TP61/start.do?Language=De&amp;Query=0100%3D%22%3FDE-588%3F117060275%22+IN+%5B1%5D\" rel=\"nofollow\">Martin Schrettinger</a>)</li>",
    "beacon_alive": true,
    "beacon_header": "#FORMAT: BEACON\n#PREFIX: http://d-nb.info/gnd/\n#TARGET:  http://www.gateway-bayern.de/opensearch?rfr_id=LinkedOpenData%3ABeacon&rft.au=%28DE-588%29{ID}\n#VERSION: 0.2\n#FEED: http://www.bib-bvb.de/OpenData/beacon_bvb01.txt\n#INSTITUTION: Bayerische Staatsbibliothek - Bibliotheksverbund Bayern - Sachgebiet Verbundnahe Dienste\n#NAME: Titelaufnahmen des B3Kat\n#ISIL: DE-604\n#TIMESTAMP: Wed Mar 6 08:34:29 CET 2019"
  }

fields/structure of BEACONlist_scraped.json

provider_type (string)

The type of the providing institution. There are different types of providers (e.g. libraries) of BEACONs.

beacon_url (string)

The BEACON's URL.

beacon_name (string)

The BEACON's name - i.e. the text value of the BEACON link.

beacon_description (string)

A description. Name, institution, number of entries/entities, examples, date of last update, etc. - has many variations.

beacon_raw (string)

The raw HTML of the scraped list item.

beacon_alive (boolean)

Whether the BEACON URL itself is alive or dead. BEWARE: No statement about whether the BEACON is still functional (i.e. that links generated from a BEACON are still alive)!

beacon_header (string)

The BEACON's header / meta data - i.e. the first lines at the top with a leading #. BEWARE: Cannot be retrieved automatically from all BEACONs!

beacon_link_examples (array: string)

Lists the first 10 links of a BEACON.

fields/structure of BEACONlist_refined.json (work in progress)

provider_type (string)

The type of the providing institution. There are different types of providers (e.g. libraries) of BEACONs.

beacon_url (string)

The BEACON's URL.

beacon_name (string)

The BEACON's name - i.e. the text value of the BEACON link.

beacon_description (string)

A description. Name, institution, number of entries/entities, examples, date of last update, etc. - has many variations.

beacon_raw (string)

The raw HTML of the scraped list item.

beacon_alive (boolean)

Whether the BEACON URL itself is alive or dead. BEWARE: No statement about whether the BEACON is still functional (i.e. that links generated from a BEACON are still alive)!

beacon_header (string)

The BEACON's header / meta data - i.e. the first lines at the top with a leading #. BEWARE: Cannot be retrieved automatically from all BEACONs!

beacon_header_fields (dict: string)

The header fields as a JSON object (i.e. a conversion of beacon_header).

beacon_link_examples (array: string)

Lists the first 10 links of a BEACON.

beacon_resolved_examples (array: string)

beaccon_link_examples resolved to URLs.

beacon_functional_examples (integer)

The number of beacon_link_examples that did not throw an error - i.e. number of links that are alive, 0-10, should ideally be 10.