A plain count() on the 12 GB Parquet file located at s3://super-issues/super-7260/gha.parquet ran on my Intel Macbook for over an hour before I gave up and stopped it. It was swapping heavily the whole time. In Activity Monitor I saw its peak RSS memory consumption was in excess of 36 GB.
$ aws s3 cp s3://super-issues/super-7260/gha.parquet . &&
super -version &&
time super -c "count()" gha.parquet
Version: v0.3.0-357-g0c71fac6a
[ran for 1+ hour, consumed 36+ GB RSS memory, then I gave up and killed it]
Details
Repro is with super commit 0c71fac.
The Parquet file in question is the "blended" contents of the 24 JSON files of GitHub Archive data we test with frequently. In lieu of the genuine blend operator, it was created in a two-step process because direct use of blend has its own memory consumption problems due to lack of spill-to-disk (which is disclosed in the docs) and the "logical equivalent" also disclosed in the docs currently hits a deadlock due to #7259.
$ for num in $(seq 0 23)
do
curl -L -O "https://data.gharchive.org/2023-02-08-${num}.json.gz"
done &&
super -version &&
TYPE=$(super -s -c 'aggregate blend(this)' ./2023-02-08-*.json.gz) &&
super -dynamic -f parquet -o gha.parquet -c "values upcast(this, ${TYPE})" ./2023-02-08-*.json.gz &&
ls -l gha.parquet
-rw-r--r-- 1 phil staff 12801117818 Aug 30 16:40 gha.parquet
By comparison, attempting the same count() directly against the original JSON completes in ~1.5 minutes and the RSS memory consumption on my Macbook never exceeded 6 GB.
$ time super -c "count()" 2023-02-08-*.json.gz
4434953
real 1m32.267s
user 3m0.885s
sys 0m4.890s
I had Claude do some profiling to better understand the root cause and have captured its findings in a Gist.
A plain
count()on the 12 GB Parquet file located ats3://super-issues/super-7260/gha.parquetran on my Intel Macbook for over an hour before I gave up and stopped it. It was swapping heavily the whole time. In Activity Monitor I saw its peak RSS memory consumption was in excess of 36 GB.Details
Repro is with super commit 0c71fac.
The Parquet file in question is the "blended" contents of the 24 JSON files of GitHub Archive data we test with frequently. In lieu of the genuine
blendoperator, it was created in a two-step process because direct use ofblendhas its own memory consumption problems due to lack of spill-to-disk (which is disclosed in the docs) and the "logical equivalent" also disclosed in the docs currently hits a deadlock due to #7259.By comparison, attempting the same
count()directly against the original JSON completes in ~1.5 minutes and the RSS memory consumption on my Macbook never exceeded 6 GB.I had Claude do some profiling to better understand the root cause and have captured its findings in a Gist.