This issue has evolved a lot since I filed it — final rewrite with everything I learned. Original claim was "CONFIG_CPUSETS=y hard-freezes crown boot ~9s in." That's wrong. Here's what's actually true, kept here mostly because the negative results and one gotcha are useful to anyone else instrumenting these devices. (Same tree/device as my AEC and dumpsys reports earlier this week; kernel @ 8d928c5176cc + CONFIG_CPUSETS=y/CONFIG_PROC_PID_CPUSET=y in crown_defconfig, GCC 7.5 aarch64.)
What happened
The first boot of my CPUSETS-enabled build froze ~9s in: silent, no panic in pstore, no watchdog reset (kernel alive and petting the MTK watchdog — so a userspace/init-phase wedge, not a kernel lockup). Bisecting against your crown-v0.5 boot.img — whose /proc/config.gz matches my config except the two CPUSETS lines, same source commit — looked like a clean conviction of CPUSETS.
Why CPUSETS is innocent
It never reproduced:
- 18+ consecutive clean boots on the identical CPUSETS kernel, including 3 deliberate recreations of first-boot-after-system-change conditions (flashing between builds with different package sets to force full PMS reconciliation — the original freeze was exactly such a first boot)
- A storm test run live on the booted CPUSETS kernel: ~50,000 forced hotplug cycles (toggling
/proc/hps/num_limit_custom1, which calls device_online/offline unconditionally) concurrently with cpuset cpus-mask writes, cgroup mkdir/rmdir churn, and task-attach churn — targeting the classic cpuset_write_resmask/cpuset_hotplug_work/hotplug-lock triangle — no stall, no deadlock
So 4.9.337's cpuset/hotplug paths hold up under far more pressure than boot generates, and the freeze remains a single unexplained occurrence. I'm running CPUSETS=y in production on both crown and cronos now.
The gotcha worth sharing: don't naively arm the hung-task detector on these kernels
To catch a recurrence I enabled CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y and re-armed the detector from the device rc (AOSP init.rc:228 writes hung_task_timeout_secs 0). Result: the device panic-rebooted every ~4 minutes like clockwork, each time blaming:
INFO: task GCPU:28 blocked for more than 120 seconds.
GCPU D 0 28 2 0x00000000
Call trace:
... down+0x4c/0x68
KREE_ServSemaphoreDown+0x10/0x20
tz_service_call+0x94/0xe8
KREE_TeeServiceCall+0x108/0x280
kree_thread_function+0x44/0x68
I briefly thought I'd caught a TEE hang — but the GCPU/KREE worker just idles in uninterruptible sleep by design (verified: it sits in D state on a fully healthy system a minute after boot). That's presumably exactly why AOSP zeroes hung_task_timeout_secs: vendor kernels are full of idle D-state workers.
Working recipe, if you want a boot-freeze tripwire without runtime false positives — armed only during the boot window:
# kernel: CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y (CONFIG_PANIC_TIMEOUT=1 already set)
on init
write /proc/sys/kernel/hung_task_timeout_secs 300
on property:sys.boot_completed=1
write /proc/sys/kernel/hung_task_panic 0
write /proc/sys/kernel/hung_task_timeout_secs 0
A boot wedged >5 min self-reboots with all blocked-task stacks dumped to pstore dmesg-ramoops; a healthy boot disarms the trap before the idle workers can trip it. If my one-off freeze ever recurs, this will capture real stacks and I'll report back.
Feel free to close this — leaving the writeup here in case anyone else chases the same ghosts.
This issue has evolved a lot since I filed it — final rewrite with everything I learned. Original claim was "CONFIG_CPUSETS=y hard-freezes crown boot ~9s in." That's wrong. Here's what's actually true, kept here mostly because the negative results and one gotcha are useful to anyone else instrumenting these devices. (Same tree/device as my AEC and dumpsys reports earlier this week; kernel @
8d928c5176cc+CONFIG_CPUSETS=y/CONFIG_PROC_PID_CPUSET=yin crown_defconfig, GCC 7.5 aarch64.)What happened
The first boot of my CPUSETS-enabled build froze ~9s in: silent, no panic in pstore, no watchdog reset (kernel alive and petting the MTK watchdog — so a userspace/init-phase wedge, not a kernel lockup). Bisecting against your crown-v0.5 boot.img — whose
/proc/config.gzmatches my config except the two CPUSETS lines, same source commit — looked like a clean conviction of CPUSETS.Why CPUSETS is innocent
It never reproduced:
/proc/hps/num_limit_custom1, which callsdevice_online/offlineunconditionally) concurrently with cpusetcpus-mask writes, cgroup mkdir/rmdir churn, and task-attach churn — targeting the classiccpuset_write_resmask/cpuset_hotplug_work/hotplug-lock triangle — no stall, no deadlockSo 4.9.337's cpuset/hotplug paths hold up under far more pressure than boot generates, and the freeze remains a single unexplained occurrence. I'm running CPUSETS=y in production on both crown and cronos now.
The gotcha worth sharing: don't naively arm the hung-task detector on these kernels
To catch a recurrence I enabled
CONFIG_BOOTPARAM_HUNG_TASK_PANIC=yand re-armed the detector from the device rc (AOSPinit.rc:228writeshung_task_timeout_secs 0). Result: the device panic-rebooted every ~4 minutes like clockwork, each time blaming:I briefly thought I'd caught a TEE hang — but the GCPU/KREE worker just idles in uninterruptible sleep by design (verified: it sits in D state on a fully healthy system a minute after boot). That's presumably exactly why AOSP zeroes
hung_task_timeout_secs: vendor kernels are full of idle D-state workers.Working recipe, if you want a boot-freeze tripwire without runtime false positives — armed only during the boot window:
A boot wedged >5 min self-reboots with all blocked-task stacks dumped to pstore
dmesg-ramoops; a healthy boot disarms the trap before the idle workers can trip it. If my one-off freeze ever recurs, this will capture real stacks and I'll report back.Feel free to close this — leaving the writeup here in case anyone else chases the same ghosts.