Uncaught error in MOPITT data collection

[camden-opfer]

ADF run type

Model vs. Model

What happened?

I accidentally had MOPITT turned on in a configuration file, and got the following error.

  File "/path/to/ADF//scripts/plotting/MOPITT.py", line 70, in collect_mopitt_data
    return data_array
           ^^^^^^^^^^
UnboundLocalError: cannot access local variable 'data_array' where it is not associated with a value

This is because the main MOPITT function collects input files by searching in a hard-coded path in glade.

files = sorted(glob.glob('/glade/campaign/acom/acom-da/buchholz/MOPITT_v9_climo/*2002_2021.he5', recursive=False))

data_array = collect_mopitt_data(files, "column")

and then collect_mopitt_data only creates the object data_array if the list files has at least one element.

I have ported ADF to a non-NCAR machine, so I naturally do not have any of the related files, and the list created by glob is empty. While I don't personally intend to use MOPITT comparisons, it could be helpful to have '/glade/campaign/acom/acom-da/buchholz/MOPITT_v9_climo/*2002_2021.he5' and the related '/glade/campaign/acom/acom-da/buchholz/MOPITT_v9_climo/*2002_2021_SD.he5' be configurable parameters for other users on systems outside NCAR in case the data gets shared with them. Then again, I don't know if there's enough demand to necessitate that big of a change.

Regardless, it would be good to catch this error more directly. If the files on glade move or have permissions changed, the same error I encountered would arise, and it's not very descriptive. One suggestion for this would be adding

if not files:
    raise FileNotFoundError('No MOPITT data files found. Ensure you have access to the glade archive of observations.')

in between the two lines of code pasted in above. Or, since this is really a problem when fetching the data, collect_mopitt_data could be changed from

def collect_mopitt_data(files, varname):
    count = 0
    for filename in files:
        data = load_and_extract_grid_hdf(filename, varname)
        if count == 0:
            data_array = data
            count += 1
        else:
            data_array = xr.concat([data_array, data], 'time')

    return data_array

to

def collect_mopitt_data(files, varname):
    data_array_initialized = False
    for filename in files:
        data = load_and_extract_grid_hdf(filename, varname)
        if not data_array_initialized :
            data_array = data
            data_array_initialized  = True
        else:
            data_array = xr.concat([data_array, data], 'time')

    if not data_array_initialized :
        raise FileNotFoundError('No MOPITT data files found. Ensure you have access to the glade archive of observations.')
            
    return data_array

ADF Hash you are using

6aff7b6

What machine were you running the ADF on?

Other (please explain below)

What python environment were you using?

Other (please explain below)

Extra info

I have ported ADF to an HPC cluster run by the Digital Research Alliance of Canada, and am using a virtual environment from venv rather than Conda per the admins' requirements. This is mainly relevant to this issue because I don't have access to data on NCAR computers, which is why this error arose. Generally, ADF runs as expected in my environment.