Skip to content

"from" with globs can be challenging with common filenames #7271

Description

@philrz

With the filenames of the GitHub Archive data we often test with, a user attempting to use from with a glob currently hits a parse error.

$ super -version &&
  super -c "from 2023-02-08-*.json.gz | count()"

Version: v0.3.0-361-g7a03e7f05

parse error at line 1, column 10:
from 2023-02-08-*.json.gz | count()
     === ^ ===

While the docs disclose the reasons why this wouldn't work, super would benefit from a more straightforward way to allow glob wildcards here.

Details

Repro is with super commit 7a03e7f. These filenames were encountered "in the wild" and were not a contrived torture test designed to illustrate the challenge.

The docs on SuperSQL queries have a section on globs that does disclose why the above doesn't work. However, many users tend to hack at what they think of as intuitive UX before resorting to the docs, and if they try multiple things without luck they may simply give up.

For instance, if they go off the error message above referencing column 10 and the pointer to the first -, they might first try escaping the hyphens.

$ super -c "from 2023\-02\-08\-*.json.gz | count()"
parse error at line 1, column 10:
from 2023\-02\-08\-*.json.gz | count()
     === ^ ===

Perhaps next they'd try quoting, but of course this disables the globbing.

$ super -c "from '2023-02-08-*.json.gz' | count()"
file does not exist at line 1, column 6:
from '2023-02-08-*.json.gz' | count()
     ~~~~~~~~~~~~~~~~~~~~~~

"Bare" glob is ultimately the way to go, but the leading digit was foiling us, so the combination of a different prefix combined with the escaping of the hyphens does the trick.

$ super -c "from ./2023\-02\-08\-*.json.gz | count()"
4434953

Ideas

A couple ideas I've heard kicked around when this has been discussed:

  1. Perhaps the parser could be relaxed (specifically for the from case?) to be more permissive of common filenames, such as these that lead with datestamps.

  2. Perhaps a function could be used to wrap a quoted string but treat it as a glob once unwrapped (ike read_json() in DuckDB or file() in ClickHouse).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions