Hi,
I'm using blockCV for spatial CV in a workflow with an added wrinkle, and I'd like guidance on the "right" way to apply cv_spatial_autocor().
Setup: I'm fitting quantile regression forests (ranger) to predict a continuous response from several predictors, with the goal of finding which quantile τ (e.g., 0.1, 0.25, 0.5, 0.75, 0.9) gives the best out-of-sample R² under spatial CV. So τ is being treated as a tunable hyperparameter, not just used for uncertainty bands around a single central model.
Question: Per Roberts et al. (2017) and the cv_spatial_autocor() documentation, block size should be based on the autocorrelation range of model residuals rather than the raw response or predictors. But in my case, residual structure could plausibly differ by τ — a model fit at τ=0.1 may have different spatial residual autocorrelation than one fit at τ=0.9.
Since the fold structure needs to be fixed and identical across the entire tuning grid (to keep CV R² comparable across candidate τ values and hyperparameter combinations), I can't simply use a different block size per τ.
My current approach: fit a naive default-hyperparameter QRF separately at each candidate τ, compute residuals for each, run cv_spatial_autocor() on each residual set, and then take the median estimated range across all τ as the fixed block size for the whole tuning grid.
Is this the recommended approach, or is there a more standard way of handling block-size selection when the hyperparameter being tuned can itself affect residual spatial structure? Is taking the median defensible, or would you recommend a different consolidation rule (e.g., max), or perhaps ignoring per-τ differences entirely and just using τ=0.5 as a reasonable approximation?
Thanks for any input — happy to share more detail on the workflow if useful.
Hi,
I'm using blockCV for spatial CV in a workflow with an added wrinkle, and I'd like guidance on the "right" way to apply
cv_spatial_autocor().Setup: I'm fitting quantile regression forests (ranger) to predict a continuous response from several predictors, with the goal of finding which quantile
τ(e.g., 0.1, 0.25, 0.5, 0.75, 0.9) gives the best out-of-sample R² under spatial CV. Soτis being treated as a tunable hyperparameter, not just used for uncertainty bands around a single central model.Question: Per Roberts et al. (2017) and the
cv_spatial_autocor()documentation, block size should be based on the autocorrelation range of model residuals rather than the raw response or predictors. But in my case, residual structure could plausibly differ byτ— a model fit atτ=0.1 may have different spatial residual autocorrelation than one fit atτ=0.9.Since the fold structure needs to be fixed and identical across the entire tuning grid (to keep CV R² comparable across candidate
τvalues and hyperparameter combinations), I can't simply use a different block size perτ.My current approach: fit a naive default-hyperparameter QRF separately at each candidate
τ, compute residuals for each, runcv_spatial_autocor()on each residual set, and then take the median estimated range across allτas the fixed block size for the whole tuning grid.Is this the recommended approach, or is there a more standard way of handling block-size selection when the hyperparameter being tuned can itself affect residual spatial structure? Is taking the median defensible, or would you recommend a different consolidation rule (e.g., max), or perhaps ignoring per-
τdifferences entirely and just usingτ=0.5 as a reasonable approximation?Thanks for any input — happy to share more detail on the workflow if useful.