feat: parallelize theme:compile in storefront build recipe

[Joachim Rütter (mynetx)]

Motivation

Compiling themes for many active sales channels is the dominant bottleneck in sequential storefront builds. Each compile adds ~6–15 s of Node + SCSS overhead; with 10+ channels this compounds to minutes in CI.

Opt-in

The parallel path is opt-in. Default behavior is unchanged (theme:compile --active-only --sync).

Enable via flag:

bin/build-storefront.sh --parallel

or via environment variable (useful in CI pipelines where passing flags through is awkward):

SHOPWARE_THEME_COMPILE_PARALLEL=1 bin/build-storefront.sh

Worker count defaults to the number of logical CPUs; override with SHOPWARE_THEME_COMPILE_WORKERS:

SHOPWARE_THEME_COMPILE_WORKERS=4 bin/build-storefront.sh --parallel

Mechanism

The first sales channel is compiled serially so that shared theme assets (theme/<themeId>/) are written without race conditions. All remaining channels then compile in parallel with --keep-assets, which only writes the per-channel CSS path (theme/<themePrefix>/css/all.css) and has no shared write target.

This continues the direction of the --sync option added in 6.6.1.0 (changelog): --sync moved theme compilation into the foreground build process; this change parallelizes that foreground work at the process level.

Details

• Uses only existing CLI flags (--sync, --keep-assets, --only) — no core changes required.
• Falls back to standard sequential theme:compile --active-only --sync when --parallel is not set or jq is unavailable.
• Applies to both 6.6 and 6.7 recipe variants (identical change in each).

Known limitation

In setups where different sales channels use different themes, only the first channel's theme assets are refreshed. This is acceptable for the typical build scenario (fresh deployment, single shared theme). Use SHOPWARE_SKIP_THEME_COMPILE=1 plus a manual theme:compile call for heterogeneous-theme setups.

[github-actions]

Thanks for the PR 😍

How to test these changes in your application

1. Add the Shopware flex endpoint in your composer.json to https://raw.githubusercontent.com/shopware/recipes/flex/pull-254/index.json.

   # When jq is installed
   jq '.extra.symfony.endpoint |= [ "https://raw.githubusercontent.com/shopware/recipes/flex/pull-254/index.json" ] + .' composer.json > composer.tmp && mv composer.tmp composer.json

   or manually

   "endpoint": [
       "https://raw.githubusercontent.com/shopware/recipes/flex/pull-254/index.json",
       "https://raw.githubusercontent.com/shopware/recipes/flex/main/index.json",
       "flex://defaults"
   ]
   # On Unix-like (BSD, Linux and macOS)
   export SYMFONY_ENDPOINT=https://raw.githubusercontent.com/shopware/recipes/flex/pull-254/index.json
   # On Windows
   SET SYMFONY_ENDPOINT=https://raw.githubusercontent.com/shopware/recipes/flex/pull-254/index.json

2. Install the package(s) related to this recipe:

   composer req 'shopware/storefront:^6.7'

Diff between recipe versions

In order to help with the review stage, I'm in charge of computing the diff between the various versions of patched recipes.
I'm going keep this comment up to date with any updates of the attached patch.

shopware/storefront

6.4 vs 6.6

diff --git a/shopware/storefront/6.4/bin/build-storefront.sh b/shopware/storefront/6.6/bin/build-storefront.sh
index 5bf0af2..b0edd2c 100755
--- a/shopware/storefront/6.4/bin/build-storefront.sh
+++ b/shopware/storefront/6.6/bin/build-storefront.sh
@@ -7,6 +7,9 @@ set -euo pipefail
 
 export PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
 export PROJECT_ROOT="${PROJECT_ROOT:-"$(dirname "$CWD")"}"
+export NPM_CONFIG_FUND=false
+export NPM_CONFIG_AUDIT=false
+export NPM_CONFIG_UPDATE_NOTIFIER=false
 
 if [[ -e "${PROJECT_ROOT}/vendor/shopware/platform" ]]; then
     STOREFRONT_ROOT="${STOREFRONT_ROOT:-"${PROJECT_ROOT}/vendor/shopware/platform/src/Storefront"}"
@@ -24,6 +27,16 @@ if [[ ${CI:-""} ]]; then
     fi
 fi
 
+keep_cache=0
+parallel=0
+for arg in "$@"; do
+    case "$arg" in
+        --keep-cache) keep_cache=1 ;;
+        --parallel)   parallel=1 ;;
+    esac
+done
+[[ ${SHOPWARE_THEME_COMPILE_PARALLEL:-""} ]] && parallel=1
+
 # build storefront
 [[ ${SHOPWARE_SKIP_BUNDLE_DUMP:-""} ]] || "${BIN_TOOL}" bundle:dump
 [[ ${SHOPWARE_SKIP_FEATURE_DUMP:-""} ]] || "${BIN_TOOL}" feature:dump
@@ -48,7 +61,7 @@ if [[ $(command -v jq) ]]; then
         if [[ -f "$path/package.json" && ! -d "$path/node_modules" && $name != "storefront" ]]; then
             echo "=> Installing npm dependencies for ${name}"
 
-            npm install --prefix "$path" --no-audit --prefer-offline
+            npm install --prefix "$path" --prefer-offline
         fi
     done
     cd "$OLDPWD" || exit
@@ -56,8 +69,59 @@ else
     echo "Cannot check extensions for required npm installations as jq is not installed"
 fi
 
-npm --prefix "${STOREFRONT_ROOT}"/Resources/app/storefront install --no-audit --prefer-offline
+npm --prefix "${STOREFRONT_ROOT}"/Resources/app/storefront install --prefer-offline --omit=dev
 node "${STOREFRONT_ROOT}"/Resources/app/storefront/copy-to-vendor.js
 npm --prefix "${STOREFRONT_ROOT}"/Resources/app/storefront run production
 [[ ${SHOPWARE_SKIP_ASSET_COPY:-""} ]] ||"${BIN_TOOL}" assets:install
-[[ ${SHOPWARE_SKIP_THEME_COMPILE:-""} ]] || "${BIN_TOOL}" theme:compile
+if [[ -z ${SHOPWARE_SKIP_THEME_COMPILE:-""} ]]; then
+    if [[ $parallel -eq 1 ]] && command -v jq >/dev/null 2>&1; then
+        # Parallelize theme:compile across sales channels.
+        # First channel compiles with assets (writes theme/<themeId>/ serially to avoid races).
+        # Remaining channels run in parallel with --keep-assets (CSS only, no shared write path).
+        channels=$("${BIN_TOOL}" sales-channel:list --output=json \
+            | jq -r '[.[] | select(.active=="active")] | .[] | .id + "|" + .name')
+
+        if [[ -z "$channels" ]]; then
+            echo "No active sales channels, skipping theme compile."
+        else
+            first_channel=$(echo "$channels" | head -1)
+            rest_channels=$(echo "$channels" | tail -n +2)
+            rest_count=$(echo -n "$rest_channels" | grep -c . 2>/dev/null || true)
+            cpu_count=$(sysctl -n hw.logicalcpu 2>/dev/null || nproc 2>/dev/null || echo 4)
+            workers="${SHOPWARE_THEME_COMPILE_WORKERS:-$(( rest_count < cpu_count ? rest_count : cpu_count ))}"
+            [[ $workers -lt 1 ]] && workers=1
+
+            compile_channel() {
+                local pair="$1"; shift
+                local id="${pair%%|*}"
+                local name="${pair##*|}"
+                local output exit_code
+                output=$("$BIN_TOOL" theme:compile --only "$id" --sync "$@" 2>&1); exit_code=$?
+                if [[ $exit_code -eq 0 ]]; then
+                    printf "  ok  %s\n" "$name"
+                else
+                    printf "  FAIL  %s\n%s\n" "$name" "$output"
+                    return $exit_code
+                fi
+            }
+            export -f compile_channel
+            export BIN_TOOL
+
+            total=$(echo "$channels" | wc -l | tr -d ' ')
+            echo "Compiling themes: ${total} sales channel(s), ${workers} parallel worker(s)"
+
+            compile_channel "$first_channel"
+            if [[ -n "$rest_channels" ]]; then
+                # shellcheck disable=SC2016
+                echo "$rest_channels" \
+                    | xargs -P"$workers" -I{} bash -c 'compile_channel "$1" --keep-assets' _ {}
+            fi
+        fi
+    else
+        "${BIN_TOOL}" theme:compile --active-only --sync
+    fi
+fi
+
+if [[ $keep_cache -eq 0 ]]; then
+    "${BIN_TOOL}" cache:clear
+fi
diff --git a/shopware/storefront/6.4/bin/watch-storefront.sh b/shopware/storefront/6.6/bin/watch-storefront.sh
index f6c9015..735a0d2 100755
--- a/shopware/storefront/6.4/bin/watch-storefront.sh
+++ b/shopware/storefront/6.6/bin/watch-storefront.sh
@@ -4,6 +4,9 @@ CWD="$(cd -P -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)"
 
 export PROJECT_ROOT="${PROJECT_ROOT:-"$(dirname "$CWD")"}"
 export ENV_FILE=${ENV_FILE:-"${PROJECT_ROOT}/.env"}
+export NPM_CONFIG_FUND=false
+export NPM_CONFIG_AUDIT=false
+export NPM_CONFIG_UPDATE_NOTIFIER=false
 
 # shellcheck source=functions.sh
 source "${PROJECT_ROOT}/bin/functions.sh"
@@ -23,9 +26,24 @@ export PROXY_URL
 export STOREFRONT_ASSETS_PORT
 export STOREFRONT_PROXY_PORT
 
-DATABASE_URL="" "${CWD}"/console feature:dump
-"${CWD}"/console theme:compile
-"${CWD}"/console theme:dump
+if [[ -e "${PROJECT_ROOT}/vendor/shopware/platform" ]]; then
+    STOREFRONT_ROOT="${STOREFRONT_ROOT:-"${PROJECT_ROOT}/vendor/shopware/platform/src/Storefront"}"
+else
+    STOREFRONT_ROOT="${STOREFRONT_ROOT:-"${PROJECT_ROOT}/vendor/shopware/storefront"}"
+fi
+
+if [[ ! -d "${STOREFRONT_ROOT}"/Resources/app/storefront/node_modules/webpack-dev-server ]]; then
+    npm --prefix "${STOREFRONT_ROOT}"/Resources/app/storefront install --prefer-offline
+fi
+
+"${CWD}"/console bundle:dump
+"${CWD}"/console feature:dump
+"${CWD}"/console theme:compile --active-only
+if [[ -n "$1" ]]; then
+    "${CWD}"/console theme:dump --theme-name="$1"
+else
+    "${CWD}"/console theme:dump
+fi
 
 if [[ $(command -v jq) ]]; then
     OLDPWD=$(pwd)
@@ -55,4 +73,4 @@ else
     echo "Cannot check extensions for required npm installations as jq is not installed"
 fi
 
-npm --prefix vendor/shopware/storefront/Resources/app/storefront/ run-script hot-proxy
+npm --prefix "${STOREFRONT_ROOT}"/Resources/app/storefront run-script hot-proxy
diff --git a/shopware/storefront/6.4/manifest.json b/shopware/storefront/6.6/manifest.json
index 3536fe4..cb880b5 100644
--- a/shopware/storefront/6.4/manifest.json
+++ b/shopware/storefront/6.6/manifest.json
@@ -8,7 +8,7 @@
         "bin/": "%BIN_DIR%/"
     },
     "env": {
-        "STOREFRONT_PROXY_URL": "http://localhost",
+        "PROXY_URL": "http://localhost",
         "SHOPWARE_HTTP_CACHE_ENABLED": "1",
         "SHOPWARE_HTTP_DEFAULT_TTL": "7200"
     }

6.6 vs 6.7

diff --git a/shopware/storefront/6.6/bin/build-storefront.sh b/shopware/storefront/6.7/bin/build-storefront.sh
index b0edd2c..0ca94ed 100755
--- a/shopware/storefront/6.6/bin/build-storefront.sh
+++ b/shopware/storefront/6.7/bin/build-storefront.sh
@@ -44,9 +44,12 @@ done
 if [[ $(command -v jq) ]]; then
     OLDPWD=$(pwd)
     cd "$PROJECT_ROOT" || exit
+    basePathsFile=$(mktemp)
+    trap 'rm -f "$basePathsFile"' EXIT
 
     jq -c '.[]' "var/plugins.json" | while read -r config; do
         srcPath=$(echo "$config" | jq -r '(.basePath + .storefront.path)')
+        basePath=$(echo "$config" | jq -r '.basePath')
 
         # the package.json files are always one upper
         path=$(dirname "$srcPath")
@@ -58,12 +61,32 @@ if [[ $(command -v jq) ]]; then
             continue
         fi
 
+        if [[ -n $srcPath ]] && ! grep -qxF "$basePath" "$basePathsFile" 2>/dev/null; then
+            echo "$basePath" >> "$basePathsFile"
+        fi
+
         if [[ -f "$path/package.json" && ! -d "$path/node_modules" && $name != "storefront" ]]; then
             echo "=> Installing npm dependencies for ${name}"
 
-            npm install --prefix "$path" --prefer-offline
+            (cd "$path" && npm install --prefer-offline)
         fi
     done
+
+    while IFS= read -r basePath || [[ -n "$basePath" ]]; do
+        if [[ -z $basePath ]]; then
+            continue
+        fi
+        if [[ -r "${basePath}/package.json" ]]; then
+            echo "=> Installing npm dependencies for ${basePath}"
+            (cd "${basePath}" && npm ci --omit=dev --no-audit --prefer-offline)
+        fi
+
+        if [[ -r "${basePath}/../package.json" ]]; then
+            echo "=> Installing npm dependencies for ${basePath}/.."
+            (cd "${basePath}/.." && npm ci --omit=dev --no-audit --prefer-offline)
+        fi
+    done < "$basePathsFile"
+
     cd "$OLDPWD" || exit
 else
     echo "Cannot check extensions for required npm installations as jq is not installed"
diff --git a/shopware/storefront/6.6/bin/watch-storefront.sh b/shopware/storefront/6.7/bin/watch-storefront.sh
index 735a0d2..46b0f3f 100755
--- a/shopware/storefront/6.6/bin/watch-storefront.sh
+++ b/shopware/storefront/6.7/bin/watch-storefront.sh
@@ -21,10 +21,12 @@ eval "$curenv"
 set +o allexport
 
 export APP_URL
-export ESLINT_DISABLE
 export PROXY_URL
 export STOREFRONT_ASSETS_PORT
 export STOREFRONT_PROXY_PORT
+export STOREFRONT_HTTPS_KEY_FILE
+export STOREFRONT_HTTPS_CERTIFICATE_FILE
+export STOREFRONT_SKIP_SSL_CERT
 
 if [[ -e "${PROJECT_ROOT}/vendor/shopware/platform" ]]; then
     STOREFRONT_ROOT="${STOREFRONT_ROOT:-"${PROJECT_ROOT}/vendor/shopware/platform/src/Storefront"}"
@@ -65,7 +67,7 @@ if [[ $(command -v jq) ]]; then
         if [[ -f "$path/package.json" && ! -d "$path/node_modules" && $name != "storefront" ]]; then
             echo "=> Installing npm dependencies for ${name}"
 
-            npm install --prefix "$path"
+            (cd "$path" && npm install)
         fi
     done
     cd "$OLDPWD" || exit

[Joachim Rütter (mynetx)]

Renaud Hager (@renaudhager) Whom should I ask to take a look at this? :)

[Joachim Rütter (mynetx)]

Soner (@shyim) Better with these changes?