It was observed that the quality of the high-resolution Matlab retrievals seamed to be better than the ones from PicassoPy. In other words the Matlab retrievals showed smoother and more data than that of PicassoPy, especially at higher altitudes.
After investigation it was found that this was due to the different SNR used in estimating the quality mask of the two processing chains. In fact, in the Matlab version, the SNR is calculated twice. Once in the pre-processing form the raw background and background corrected signal, and then again before estimating the quality mask from smoothed versions of the background and background corrected signal to artificially improve the SNR. In the second one, the background and background corrected signal are smoothed with a moving average filter and then multiplied with the smoothing window parameters Nr and Nc to get back to the Photon count domain. Thus practically using a moving sum filter. This has the effect of increasing the grid size we calculate the SNR on, thus increasing the amount of signal per grid cell, while keeping the resolution intact. Therefore, artificially increasing the SNR which again lowers the amount of cells categorized with low SNR and increases the amount of good data in the quality mask.
The option for a similar solution should be included in PicassoPy.
Psudo code:
sigBGCor_sm = smooth(sigBGCor, Nr, Nc) * Nr * Nc
BG_sm = smooth(BG, Nr, Nc) * Nr * Nc
SNR_quasi = calc_snr(sigBGCor_sm, BG_sm)
lowSNRMask_quasi = (SNR_quasi < lowSNR)
qulityMask = estQualityMask(lowSNRMask_quasi, ...)
Quality mask with raw SNR
Quality mask with improved SNR
It was observed that the quality of the high-resolution Matlab retrievals seamed to be better than the ones from PicassoPy. In other words the Matlab retrievals showed smoother and more data than that of PicassoPy, especially at higher altitudes.
After investigation it was found that this was due to the different SNR used in estimating the quality mask of the two processing chains. In fact, in the Matlab version, the SNR is calculated twice. Once in the pre-processing form the raw background and background corrected signal, and then again before estimating the quality mask from smoothed versions of the background and background corrected signal to artificially improve the SNR. In the second one, the background and background corrected signal are smoothed with a moving average filter and then multiplied with the smoothing window parameters
NrandNcto get back to the Photon count domain. Thus practically using a moving sum filter. This has the effect of increasing the grid size we calculate the SNR on, thus increasing the amount of signal per grid cell, while keeping the resolution intact. Therefore, artificially increasing the SNR which again lowers the amount of cells categorized with low SNR and increases the amount of good data in the quality mask.The option for a similar solution should be included in PicassoPy.
Psudo code:
sigBGCor_sm = smooth(sigBGCor, Nr, Nc) * Nr * NcBG_sm = smooth(BG, Nr, Nc) * Nr * NcSNR_quasi = calc_snr(sigBGCor_sm, BG_sm)lowSNRMask_quasi = (SNR_quasi < lowSNR)qulityMask = estQualityMask(lowSNRMask_quasi, ...)Quality mask with raw SNR
Quality mask with improved SNR