Add CI to validate URLs

[lfoppiano]

We should have a CI that validates that each URL added in this repository are parseable returning not None output.

See example http:///culture.gouv.fr was added and the resulting host None is not expected:

### 0 links from living/vame.md
http:///culture.gouv.fr <--- triple slash 

Traceback (most recent call last):
  File "/Volumes/ExtremePro/sciencialab/cc/web-languages/../web-languages-code/extract_links.py", line 116, in <module>
    links = list(map(normalize_url,
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/ExtremePro/sciencialab/cc/web-languages/../web-languages-code/extract_links.py", line 57, in normalize_url
    if not h.isascii():
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'isascii'

[wumpus]

There is a script that extracts urls in web-languages-code, that's what should also be used to validate. And that script shouldn't error out when used in the crawl pipeline.

[lfoppiano]

I don't know what I'm doing wrong, I've tested right now and the script fails:

##- https://or.wikipedia.org/
### 0 links from living/budeh_stieng.md
### 0 links from living/berik.md
### 0 links from living/adnyamathanha.md
### 0 links from living/central_tagbanwa.md
### 0 links from living/ngom.md
### 0 links from living/kota_bangun_kutai_malay.md
Traceback (most recent call last):
  File "/Volumes/ExtremePro/sciencialab/cc/web-languages/../web-languages-code/extract_links.py", line 113, in <module>
    links = list(map(normalize_url,
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/ExtremePro/sciencialab/cc/web-languages/../web-languages-code/extract_links.py", line 54, in normalize_url
    if not h.isascii():
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'isascii'
(cc) ➜  web-languages git:(main) ✗

[thunderpoot]

I don't know what I'm doing wrong, I've tested right now and the script fails:

##- https://or.wikipedia.org/
### 0 links from living/budeh_stieng.md
### 0 links from living/berik.md
### 0 links from living/adnyamathanha.md
### 0 links from living/central_tagbanwa.md
### 0 links from living/ngom.md
### 0 links from living/kota_bangun_kutai_malay.md
Traceback (most recent call last):
  File "/Volumes/ExtremePro/sciencialab/cc/web-languages/../web-languages-code/extract_links.py", line 113, in <module>
    links = list(map(normalize_url,
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/ExtremePro/sciencialab/cc/web-languages/../web-languages-code/extract_links.py", line 54, in normalize_url
    if not h.isascii():
           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'isascii'
(cc) ➜  web-languages git:(main) ✗

urlparse(url).hostname returns None when the URL has no parseable host (malformed URLs, mailto:, relative links, and so on)

https://github.com/commoncrawl/web-languages-code/blob/83abb3becbbd18987c76a9435a730a3a4e151185/extract_links.py#L54

Line 54 could be:

if h and not h.isascii():

... so URLs without a host skip the IDN step instead of crashing.

[thunderpoot]

It may also be worth emitting a warning so we can go in and fix any culprit URLs.

[lfoppiano]

@thunderpoot thanks for looking at it, in fact I'm already working on a fix see commoncrawl/web-languages-code#9

I was pointing out that currently we have some invalid URL in the web-language that were not validated, so this would crash next month during the crawl.

It may also be worth emitting a warning so we can go in and fix any culprit URLs.

I though so, but the way this script is used in the crawl require to be careful when emitting warnings, because the standard output is used as input for subsequent process: https://github.com/commoncrawl/crawl-tools/blob/96c52735e9cffd633b42a69a0f3b236e6c41c0d5/server/seed/seeds_init.sh#L109