Skip to content

OutOfMemoryError (Out of stack space) in SpriteUpdateUtil.transitionLayouts with large modded resource pack atlas (8192x4096) #880

Description

@clayhuey77-blip

Summary: VulkanMod 0.6.1 on Minecraft 1.21.11 crashes with OutOfMemoryError: Out of stack space when connecting to a server whose resource pack produces a very large item atlas (8192x4096 in this case). The crash originates in SpriteUpdateUtil.transitionLayouts via VulkanImage.transitionLayout's MemoryStack allocation.

Steps to reproduce:

Install VulkanMod 0.6.1 on Fabric 1.21.11
Connect to a server with a large custom resource pack (item atlas builds to 8192x4096 in this case)
Game disconnects immediately after the atlas is built, citing a resource pack load failure

Tested and ruled out:

-Xss8M (JVM thread stack) — no effect
-Dorg.lwjgl.system.stackSize=8192 (LWJGL native stack) — no effect
Disabling "Texture Animations" in VulkanMod's own settings — no effect (crash occurs before this setting's code path is even reached)

Stack trace:

java.lang.OutOfMemoryError: Out of stack space.
at org.lwjgl.system.MemoryStack.nmalloc(MemoryStack.java:322)
at org.lwjgl.system.MemoryStack.ncalloc(MemoryStack.java:340)
at org.lwjgl.vulkan.VkImageMemoryBarrier.calloc(VkImageMemoryBarrier.java:426)
at net.vulkanmod.vulkan.texture.VulkanImage.transitionLayout(VulkanImage.java:368)
at net.vulkanmod.vulkan.texture.VulkanImage.transitionLayout(VulkanImage.java:361)
at net.vulkanmod.vulkan.texture.VulkanImage.transitionImageLayout(VulkanImage.java:355)
at net.vulkanmod.vulkan.texture.VulkanImage.transitionImageLayout(VulkanImage.java:290)
at net.vulkanmod.vulkan.texture.VulkanImage.readOnlyLayout(VulkanImage.java:282)
at net.vulkanmod.render.texture.SpriteUpdateUtil.lambda$transitionLayouts$0(SpriteUpdateUtil.java:35)
at java.lang.Iterable.forEach(Unknown Source)
at net.vulkanmod.render.texture.SpriteUpdateUtil.transitionLayouts(SpriteUpdateUtil.java:35)
at net.vulkanmod.render.texture.ImageUploadHelper.submitCommands(ImageUploadHelper.java:24)
at net.vulkanmod.vulkan.memory.buffer.StagingBuffer.submitUploads(StagingBuffer.java:61)
at net.vulkanmod.vulkan.memory.buffer.StagingBuffer.copyBuffer(StagingBuffer.java:38)
at net.vulkanmod.vulkan.texture.VulkanImage.uploadSubTextureAsync(VulkanImage.java:244)
at net.vulkanmod.vulkan.texture.VTextureSelector.uploadSubTexture(VTextureSelector.java:67)
at net.vulkanmod.render.engine.VkCommandEncoder.writeToTexture(VkCommandEncoder.java:439)

System info:

VulkanMod 0.6.1, Minecraft 1.21.11, Fabric Loader 0.19.3, Java 21
Windows, LWJGL 3.3.3
Atlas sizes at crash time: blocks.png-atlas 4096x2048x4, items.png-atlas 8192x4096x0

Suspected cause: transitionLayouts appears to allocate a VkImageMemoryBarrier per sprite via MemoryStack.calloc inside a single unflushed stack frame/forEach pass. With thousands of sprites in an oversized atlas, this overflows the stack regardless of -Xss or LWJGL's native stack size setting, suggesting the allocations aren't being popped/released between iterations.

Full latest.log attached.

Server attempted to connect to: OriginRealms.com

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions