With the filenames of the GitHub Archive data we often test with, a user attempting to use from with a glob currently hits a parse error.
$ super -version &&
super -c "from 2023-02-08-*.json.gz | count()"
Version: v0.3.0-361-g7a03e7f05
parse error at line 1, column 10:
from 2023-02-08-*.json.gz | count()
=== ^ ===
While the docs disclose the reasons why this wouldn't work, super would benefit from a more straightforward way to allow glob wildcards here.
Details
Repro is with super commit 7a03e7f. These filenames were encountered "in the wild" and were not a contrived torture test designed to illustrate the challenge.
The docs on SuperSQL queries have a section on globs that does disclose why the above doesn't work. However, many users tend to hack at what they think of as intuitive UX before resorting to the docs, and if they try multiple things without luck they may simply give up.
For instance, if they go off the error message above referencing column 10 and the pointer to the first -, they might first try escaping the hyphens.
$ super -c "from 2023\-02\-08\-*.json.gz | count()"
parse error at line 1, column 10:
from 2023\-02\-08\-*.json.gz | count()
=== ^ ===
Perhaps next they'd try quoting, but of course this disables the globbing.
$ super -c "from '2023-02-08-*.json.gz' | count()"
file does not exist at line 1, column 6:
from '2023-02-08-*.json.gz' | count()
~~~~~~~~~~~~~~~~~~~~~~
"Bare" glob is ultimately the way to go, but the leading digit was foiling us, so the combination of a different prefix combined with the escaping of the hyphens does the trick.
$ super -c "from ./2023\-02\-08\-*.json.gz | count()"
4434953
Ideas
A couple ideas I've heard kicked around when this has been discussed:
-
Perhaps the parser could be relaxed (specifically for the from case?) to be more permissive of common filenames, such as these that lead with datestamps.
-
Perhaps a function could be used to wrap a quoted string but treat it as a glob once unwrapped (ike read_json() in DuckDB or file() in ClickHouse).
With the filenames of the GitHub Archive data we often test with, a user attempting to use
fromwith a glob currently hits a parse error.While the docs disclose the reasons why this wouldn't work,
superwould benefit from a more straightforward way to allow glob wildcards here.Details
Repro is with super commit 7a03e7f. These filenames were encountered "in the wild" and were not a contrived torture test designed to illustrate the challenge.
The docs on SuperSQL queries have a section on globs that does disclose why the above doesn't work. However, many users tend to hack at what they think of as intuitive UX before resorting to the docs, and if they try multiple things without luck they may simply give up.
For instance, if they go off the error message above referencing
column 10and the pointer to the first-, they might first try escaping the hyphens.Perhaps next they'd try quoting, but of course this disables the globbing.
"Bare" glob is ultimately the way to go, but the leading digit was foiling us, so the combination of a different prefix combined with the escaping of the hyphens does the trick.
Ideas
A couple ideas I've heard kicked around when this has been discussed:
Perhaps the parser could be relaxed (specifically for the
fromcase?) to be more permissive of common filenames, such as these that lead with datestamps.Perhaps a function could be used to wrap a quoted string but treat it as a glob once unwrapped (ike
read_json()in DuckDB orfile()in ClickHouse).