Asynchronously pre-generate selected named image thumbnails after uploads and binary changes

[Refaat-alktifan]

Affected capability

Assets

Feature description

Related to #349, pimcore/pimcore#19057 and pimcore/pimcore#19382.

Context

While working on pimcore/pimcore#19057 and its implementation PR pimcore/pimcore#19382, I investigated why image-heavy Pimcore pages can sometimes take several seconds to render or display their images, especially when remote Flysystem storage is used.

The investigation identified two separate latency problems:

1. Twig dimension resolution can read source or thumbnail files only to determine the HTML width and height attributes.
2. The first browser request for a missing thumbnail can perform the complete source download, image decode, transformation, encoding and storage write.

pimcore/pimcore#19382 is the separate fix for the first problem. It avoids physical file inspection when dimensions can be reproduced reliably.

This proposal addresses the second problem. It is intentionally separate because it introduces public configuration, a Messenger transport and new background-processing behavior, so it belongs on 2026.x.

Problem

Pimcore generates missing image thumbnails on demand. This is an important fallback, but it also means the first visitor requesting a common thumbnail can become the process that generates it.

A cold thumbnail request can involve:

browser requests missing thumbnail
→ load source from local or remote storage
→ materialize the source locally
→ decode through GD or Imagick
→ apply resize, crop and other transformations
→ encode JPEG, PNG, WebP or AVIF
→ write the derivative to thumbnail storage
→ update thumbnail status information
→ optionally dispatch image optimization
→ return the generated file

For large photographs, TIFF files, rasterized SVG files, responsive variants, automatic formats or remote storage, this can take several seconds.

Pimcore already provides pimcore:thumbnails:image for controlled catalog warming. There is currently no opt-in mechanism that automatically warms selected named thumbnail configurations after a new image upload or actual binary change.

Proposed behavior

Add an opt-in asynchronous pre-generation mechanism for explicitly selected named image thumbnail configurations.

Example configuration:

pimcore:
    assets:
        image:
            thumbnails:
                pre_generate:
                    definitions:
                        - hero
                        - card
                    max_variants_per_config: 25
                    max_definitions: 20

An empty definitions list keeps the feature disabled by default.

The first version should require explicit named configurations. Wildcards should not be accepted because they could unexpectedly generate every thumbnail definition in a project.

Processing flow

image uploaded or binary replaced
→ existing image-update processing completes
→ one message is dispatched per selected named configuration
→ dedicated thumbnail-generation workers process the messages
→ common derivatives normally exist before browsers request them

The feature should be triggered by the existing binary-change path, not by every asset save or version creation. Metadata-only changes and version-only saves should not intentionally enqueue another warming pass.

Existing on-demand generation remains the fallback:

worker unavailable
message delayed
message stale
configuration not selected
dynamic configuration used
variant limit exceeded
generation unsuccessful

→ existing on-demand generation handles the request

Dedicated Messenger transport

Pre-generation should use a dedicated asynchronous transport:

pimcore_image_thumbnail_generate

Thumbnail warming can be CPU-, memory- and temporary-disk-intensive. It should not compete with required asset processing, document processing or video work in pimcore_asset_update.

A separate transport allows independent:

• worker counts;
• CPU and memory limits;
• autoscaling;
• retry configuration;
• failed-message monitoring;
• queue-age alerts;
• graceful shutdown settings.

A synchronous transport would remain technically possible, but it would remove the intended workload isolation and should be discouraged in the documentation.

Message identity and stale-work protection

Each message should contain:

asset ID
persisted source checksum
data modification date
named thumbnail configuration

The worker reloads the image and validates the checksum and modification date before performing expensive work.

This makes old messages safe:

binary A queues warming
binary B replaces A
message for A is consumed later
→ stale message becomes a no-op

The producer should use the checksum already persisted during binary processing. It should not calculate a missing checksum through storage while dispatching the optional warming message.

If no persisted checksum is available, event-driven warming should be skipped. Normal on-demand generation and the existing command remain available.

Queue deduplication and generation coordination

Equivalent initial messages waiting for first delivery can be deduplicated using:

asset ID
+ checksum
+ data modification date
+ configuration name

This is a best-effort queue-load optimization, not the correctness boundary.

Once processing starts, Pimcore's existing deterministic thumbnail path and per-derivative generation lock remain authoritative. After acquiring the lock, the worker checks storage again before loading the source or writing the derivative.

The responsibilities remain separate:

message deduplication
→ reduce duplicate queued work

stale-message validation
→ reject obsolete source versions

existing derivative lock
→ coordinate physical generation

Variant planning

For each selected named configuration, the worker should generate a bounded, deduplicated set containing:

1. the untouched base configuration;
2. its configured high-resolution factor, including fractional values;
3. default and media-query variants;
4. integer DPR variants through max_srcset_dpi_factor;
5. enabled automatic output formats such as WebP and AVIF.

Equivalent physical outputs should be generated only once.

Work is bounded by:

max_definitions
×
max_variants_per_config

Proposed defaults:

20 selected definitions per image change
25 variants per named configuration

Installations processing large TIFF files, high-resolution photographs, rasterized SVG files, AVIF output or complex transformations can start with lower limits.

Source materialization

One asset and named-configuration message should use one lazy source snapshot.

The source should only be materialized after:

output path calculation
→ existing derivative check
→ derivative lock
→ second storage check

Therefore:

all derivatives already exist
→ no source read

first missing derivative
→ source materialized once

remaining missing variants in the message
→ reuse the same stable local source path

The snapshot should:

• verify that the local source exists;
• verify that it is readable and non-empty;
• preserve the source extension for extension-dependent image delegates;
• cache and rethrow source-wide materialization failures;
• use Pimcore-managed temporary files;
• clean temporary files before a retry.

Source reuse is scoped to one asset and one named-configuration message. Different named configurations intentionally remain separate jobs for bounded execution and retry granularity.

Each output variant still performs its own image decode and transformation. Decoded-image reuse is outside this proposal.

Failure behavior

Temporary image-processing and storage failures should be visible to Messenger.

Proposed retry configuration:

maximum retries: 5
initial delay: 2 seconds
multiplier: 2
maximum delay: 60 seconds
failure transport: pimcore_image_thumbnail_generate_failed

A temporary failure for one variant should not prevent later variants in the same message from being attempted.

After processing and temporary-file cleanup, the first retryable failure can be re-thrown so Messenger retries the message. Outputs already written during the previous attempt are found through the normal storage checks and skipped.

Source-wide failures, such as an unreadable source or an adapter being unable to load it, should stop the current batch and let Messenger retry the message. The same source failure should not be logged once for every output variant.

Known permanent configuration failures, such as unsupported output formats or maximum-scaling violations, should be logged without unnecessary retries.

Existing-catalog warming

Enabling this feature should not enqueue the complete existing asset catalog, and changing a thumbnail definition should not automatically create an unbounded deployment-time workload.

Existing assets can continue to be warmed explicitly with Pimcore's command:

bin/console pimcore:thumbnails:image \
    --thumbnails=hero,card \
    --processes=4

I propose using a shared variant resolver for the command and the asynchronous worker so both follow the same base, media, DPR and automatic-format rules.

This creates one explicit behavior decision for maintainers:

• projects with max_srcset_dpi_factor above 2 may generate additional configured DPR outputs when running the command;
• fractional high-resolution factors on the untouched base configuration are retained;
• --skip-medias, --skip-high-res, --skip-avif and --skip-webp continue to restrict the corresponding expansions.

This behavior should be approved and documented rather than introduced as an unmentioned command refactor.

Compatibility

The feature is designed to remain additive:

• disabled by default;
• no database schema change;
• no changes to thumbnail hashes or storage paths;
• no change to existing on-demand generation;
• no wildcard expansion;
• no automatic whole-catalog enqueue;
• no change to dynamic or unlisted configurations;
• no change to optimizer message formats;
• existing per-derivative locking remains in place;
• browser requests remain the fallback when warming has not completed.

Non-goals

This proposal does not attempt to solve:

• CDN cache warmth or delivery latency;
• Twig or DataObject query performance;
• low-quality-placeholder storage reads;
• image optimizer backlog;
• decoded-image reuse across variants;
• publication gating;
• every possible dynamic thumbnail configuration.

Expected result

For selected common configurations:

image binary changes
→ background generation completes
→ browser requests an existing derivative

This removes the normal first-visitor thumbnail-generation path without removing Pimcore's resilient on-demand behavior.

Together with pimcore/pimcore#19382, the two contributions address separate image latency causes:

pimcore/pimcore#19382
→ avoid unnecessary remote reads during Twig dimension resolution

this proposal
→ avoid first-request generation for selected common derivatives

The production measurement that motivated pimcore/pimcore#19057 was approximately:

79 remote storage calls
5,735 ms page-render time

compared with:

0 remote storage calls
170 ms page-render time

after eliminating unnecessary dimension-resolution I/O.

These figures relate to the #19057 dimension path. This pre-generation proposal targets the later cold-thumbnail request and does not claim the same benchmark.

Implementation

• dedicated generation and failure transports;
• bounded opt-in configuration;
• one stale-aware message per asset and named configuration;
• queue-side initial-message deduplication;
• failed-send deduplication-lock cleanup;
• shared variant planning;
• fractional DPR support;
• media and automatic-format expansion;
• lazy source snapshot reuse;
• source validation and extension preservation;
• retryable generation failures;
• permanent-error classification;
• source-wide batch termination;
• existing derivative locking;
• operational documentation.

[Refaat-alktifan]

@kingjia90 Thanks again for the detailed review of pimcore/pimcore#19382. While following the same image path, I found the separate cold-thumbnail problem described here: even after dimension resolution is fixed, the first browser request may still perform the complete generation of a missing derivative.

Since you independently traced the existing thumbnail flow, I would especially appreciate your thoughts on the dedicated transport, the one asset/configuration message boundary, and sharing the variant resolver with pimcore:thumbnails:image when you have time.

[hadl]

Wouldn’t this generate a lot of unnecessary thumbnails?

With this approach, every new image asset would get all configured thumbnail definitions, even if the asset is never used on a website or in a Document at all.
Or do i misunderstand something?

And even if the asset is used, you don’t know where it is used.
For example, an image may only be used in a teaser, it would still generate the hero, card, and other configured thumbnail variants for it.

Shouldn’t you first check the asset dependencies and, ideally, the actual usage context before generating thumbnails?
Otherwise, this process may spend a lot of CPU time and especially storage on thumbnails that will never be requested/needed.

[Refaat-alktifan]

Wouldn’t this generate a lot of unnecessary thumbnails?

With this approach, every new image asset would get all configured thumbnail definitions, even if the asset is never used on a website or in a Document at all. Or do i misunderstand something?

And even if the asset is used, you don’t know where it is used. For example, an image may only be used in a teaser, it would still generate the hero, card, and other configured thumbnail variants for it.

Shouldn’t you first check the asset dependencies and, ideally, the actual usage context before generating thumbnails? Otherwise, this process may spend a lot of CPU time and especially storage on thumbnails that will never be requested/needed.

You are correct. As proposed, this would not generate every thumbnail definition in Pimcore, because the feature would be disabled by default and limited to explicitly configured definitions. However, applying every configured definition to every uploaded or changed image would still be too broad. A teaser-only image could receive unused hero and card variants, and the limits would only bound that cost rather than make the selection relevant.

I agree that selection should happen before generation.

Dependencies can help, but they are not sufficient on their own. Pimcore’s persisted dependencies can tell us that a Document or DataObject references an asset, but they do not preserve the consuming field or editable, the Twig context, or the named thumbnail configuration. An image may also be uploaded before any dependency exists and only be assigned to content later. Dependency processing may additionally be asynchronous or disabled.

I would therefore revise the proposal to separate selection from generation:

image binary changes, known thumbnail invalidations,
or relevant asset context changes
-> enqueue one asynchronous planning message
-> reload the current image
-> select the relevant named configurations
-> filter them through a configured candidate allowlist
-> enqueue only the accepted configurations
-> generate them through dedicated workers
-> retain normal on-demand generation as fallback

The configured definitions would become candidate_definitions, meaning configurations that may be selected, rather than configurations automatically generated for every image.

A project selector could use durable information such as:

asset folders and paths;
properties, tags or metadata;
workflow state;
project-specific dependency interpretation;
a project-maintained usage mapping.

When application code already knows the exact usage context, it could schedule the required configuration directly, for example:

Article.teaserImage ->  teaser
Product.heroImage ->  hero

Dependencies would therefore remain useful as a coarse “referenced at all” input or as a reason to reevaluate an asset, but they would not be treated as the authoritative thumbnail plan.

An explicit all_candidates mode could still be available for installations that consciously prefer additional CPU and storage usage over cold thumbnail requests. It would be an intentional policy choice rather than the default behavior. The automatic modes could be:

disabled
all_candidates
service

Unknown usage, dynamic thumbnail configurations and per-editable crops would continue to use Pimcore’s existing on-demand generation.

One limitation should remain explicit: scheduling teaser when content is saved does not automatically preserve that intent after a later image-binary replacement. The project would need to re-derive the usage, keep durable intent in its own model, or rely on the normal on-demand fallback. A generic usage registry could be considered separately if needed.

Would this asynchronous selector-and-scheduler model address your concern?

[Refaat-alktifan]

The goal should be a balanced solution: avoid spending CPU and storage on thumbnails that will never be used, while also reducing cold or missing thumbnail requests so the visitor does not have to wait for generation during a page request. The selector-and-scheduler model seems like a practical way to move toward both goals without removing Pimcore’s on-demand fallback.

[hadl]

When application code already knows the exact usage context, it could schedule the required configuration directly, for example:
Article.teaserImage -> teaser

"When application code already knows the exact usage context" - this is the biggest problem.

I don't know how your are using your images in a project, but Article.teaserImage could be used as a thumbnail in a teaser, a card, a blog-list an so on.

For a website the twig template decides the thumbnail config or a DTO or an array for an ajax endpoint etc etc...

This is soooo project specific, that i honestly don't know how this could work easily and is provided as a "simple config" in the core.

I would also love to have such a feature, but i see a lot of issues - unfortunately.

For example:
In our projects we are not even using "simple named" thumbnails.
They are all dynamic depending on the used template/component/thumb-method cause we are using image src-set's.
So for us, there is no: ahh this asset is used for a hero section, i need to pre-warm the hero config. A hero-component could use maybe 6 or 8 different thumbnail configs.
And maybe (or probably) the thumbnails for JPG or WebP are never created, cause every user gets the AVIFs.

That's why i am asking. Because if you find a solution for such "dependencies", i would be happy to use that too and "speed things up".

reducing cold or missing thumbnail requests so the visitor does not have to wait for generation during a page request

Totally get that approach.
Imo, it is not "the visitor" it is "a visitor" and the latter could even be the editor checking the page after editing.
Ok, for an updated image the editor would need to open all "pages" where the asset is used and we all know that won't happen 😉

Is it really that bad for you that some - maybe even lazyloaded (what it should be) - image thumbnails takes a bit longer?
I don’t want to downplay the issue; I’m just curious

[Refaat-alktifan]

When application code already knows the exact usage context, it could schedule the required configuration directly, for example:
Article.teaserImage -> teaser

"When application code already knows the exact usage context" - this is the biggest problem.

I don't know how your are using your images in a project, but Article.teaserImage could be used as a thumbnail in a teaser, a card, a blog-list an so on.

For a website the twig template decides the thumbnail config or a DTO or an array for an ajax endpoint etc etc...

This is soooo project specific, that i honestly don't know how this could work easily and is provided as a "simple config" in the core.

I would also love to have such a feature, but i see a lot of issues - unfortunately.

For example: In our projects we are not even using "simple named" thumbnails. They are all dynamic depending on the used template/component/thumb-method cause we are using image src-set's. So for us, there is no: ahh this asset is used for a hero section, i need to pre-warm the hero config. A hero-component could use maybe 6 or 8 different thumbnail configs. And maybe (or probably) the thumbnails for JPG or WebP are never created, cause every user gets the AVIFs.

That's why i am asking. Because if you find a solution for such "dependencies", i would be happy to use that too and "speed things up".

reducing cold or missing thumbnail requests so the visitor does not have to wait for generation during a page request

Totally get that approach. Imo, it is not "the visitor" it is "a visitor" and the latter could even be the editor checking the page after editing. Ok, for an updated image the editor would need to open all "pages" where the asset is used and we all know that won't happen 😉

Is it really that bad for you that some - maybe even lazyloaded (what it should be) - image thumbnails takes a bit longer? I don’t want to downplay the issue; I’m just curious

Fair point. My Article.teaserImage -> teaser example was too simplistic.

In the setup you describe, the field identifies only the asset. The actual thumbnail plan is created later by Twig, a component, a DTO or an API endpoint. It may contain dynamic transformations, several responsive sizes and several formats, while the browser eventually requests only one candidate.

So I agree that dependencies, field mappings, asset folders or a simple selector cannot provide a generic core solution.

I am looking at this as a broader architectural problem:

How can Pimcore move expensive derivative generation away from frontend workers
without blindly generating every possible derivative?

There is an unavoidable constraint:

zero unused derivatives
+ zero first-request latency
+ no prior knowledge of usage

These cannot all be guaranteed simultaneously. Pimcore needs some form of demand signal before the request.

I therefore think generation and demand policy should be separated:

demand signals
-> bounded warming policy
-> exact asynchronous generation
-> dedicated workers
-> existing on-demand fallback

Core could provide a lower-level mechanism for asynchronously generating one exact named or dynamic thumbnail recipe. Different optional policies could then use that mechanism:

• explicitly declared critical recipes;
• candidate recipes discovered while rendering or warming pages;
• exact variants observed when the deferred endpoint generates them;
• bounded regeneration of previously observed recipes after a later source-image update.

The observed-recipe direction is not simple either. Rendering sees all picture and srcset candidates, not which candidate the browser will choose. The deferred endpoint sees the exact choice only after the first cold request. CDN or direct-storage hits may bypass Pimcore, and dynamic recipes need a safe, normalized and versioned representation with retention and cleanup rules.

So I would not claim that a simple upload-time configuration can solve this generically. A more realistic core architecture appears to be:

exact asynchronous generation primitive
+ optional explicit critical recipes
+ optional bounded observed-recipe warming
+ existing on-demand generation

Strict readiness before publication would be a separate policy and would still require the necessary recipes to be declared or discovered.

Regarding the impact, I agree that one slightly slower lazy-loaded thumbnail is not automatically a serious problem. The painful cases are usually above-the-fold or LCP images, several cold derivatives requested together, large source files, expensive AVIF processing, remote storage, or frontend PHP workers performing image processing.

This also needs a separate cold-generation benchmark. The figures from #19057 relate to dimension-resolution storage I/O, not to the generation time of a missing derivative.

And you are right that it is “a visitor”, often even the editor checking a page after an image change, rather than necessarily the first public visitor.

Does this separation between a core asynchronous generation mechanism and optional demand policies sound like a direction worth exploring, or do you think Pimcore should retain only on-demand behavior and leave proactive scheduling entirely to projects?