Skip to content

Deadlock caused by subquery containing wildcard "from" #7259

Description

@philrz

The following query hangs forever.

$ printf '{a:1}\n' > n1.sup &&
  printf '{a:2}\n' > n2.sup &&
  printf '{a:1}\n{a:2}\n' > outer.sup &&
  super -version &&
  super -s -c "from './outer.sup' | values (from n*.sup | count())"
Version: v0.3.0-357-g0c71fac6a

Details

Repro is with super commit 0c71fac.

The repro above is a simplification of a hang I experienced when trying to follow the "logical equivalent" of the blend operator as described in the docs to get around the lack of spill-to-disk in the current vectorized implementation of blend. Attempting the following on the full set of GitHub Archive JSON input files triggers the same deadlock symptom:

$ super -dynamic -f parquet -o gha.parquet -c "from ./2023\-02\-08\-*.json.gz | values upcast(this, (from ./2023\-02\-08\-*.json.gz | aggregate blend(this)))"

While getting Claude's assistance with creating the simplified repro, it gave its take on the root cause and possible approach to a fix, which is written up in a Gist.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions