[Bug]: CMOR Log For MPAS Variables Saves In PWD, Not In Output Directory Under `cmor_logs`

by ADMIN 91 views

What Happened?

The mpas_vars modules have their own cmorize function, which saves the cmor_logs under the PWD. However, after merging #289, the cmor_logs output is now saved under the -o/--output directory.

What Did You Expect to Happen? Are There Any Possible Answers You Came Across?

The mpas_vars modules should use the same specified cmor_logs directory as all other variables.

Minimal Complete Verifiable Example (MVCE)

"""
This script is a minimal working example (MWE) to reproduce the issue reported
in issue #297 of the `e3sm_to_cmip` GitHub repository.

Source: https://github.com/E3SM-Project/e3sm_to_cmip/issues/297#
Original Command: e3sm_to_cmip -v mlotst -u /lcrc/group/e3sm2/DSM/Ops/DSM_Manager/tmp/v3.LR.historical_0051/metadata/historical_r1i1p1f1.json -t /lcrc/group/e3sm2/DSM/Staging/Resource/cmor/cmip6-cmor-tables/Tables -o YOUR_OUTPUT_DIR -i /lcrc/group/e3sm2/DSM/Ops/test/e2c_input -s --realm mpaso --map /lcrc/group/e3sm2/DSM/Staging/Resource/maps/map_IcoswISC30E3r5_to_cmip6_180x360_traave.20240221.nc
"""

import os

from e3sm_to_cmip.__main__ import main

OUTPUT_PATH = "/lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails"

args = [
    "--var-list",
    "mlotst",
    "--user-metadata",
    "/lcrc/group/e3sm2/DSM/Ops/DSM_Manager/tmp/v3.LR.historical_0051/metadata/historical_r1i1p1f1.json",
    "--tables-path",
    "/lcrc/group/e3sm2/DSM/Staging/Resource/cmor/cmip6-cmor-tables/Tables",
    "--output-path",
    f"{OUTPUT_PATH}",
    "--input-path",
    "/lcrc/group/e3sm2/DSM/Ops/test/e2c_input",
    "--serial",
    "--realm",
    "mpaso",
    "--map",
    "/lcrc/group/e3sm2/DSM/Staging/Resource/maps/map_IcoswISC30E3r5_to_cmip6_180x360_traave.20240221.nc",
]

main(args)


# Ensure the path and its contents have the correct permissions recursively
for root, dirs, files in os.walk(OUTPUT_PATH):
    os.chmod(root, 0o505)  # o=rx (read and execute for others)
    for d in dirs:
        os.chmod(os.path.join(root, d), 0o505)
    for f in files:
        os.chmod(os.path.join(root, f), 0o404)  # o=r (read for others)

Relevant Log Output

2025-05-22 12:49:05.607233 [INFO]: __main__.py(__init__:163) >> --------------------------------------
2025-05-22 12:49:05.608262 [INFO]: __main__.py(__init__:164) >> | E3SM to CMIP Configuration
2025-05-22 12:49:05.608943 [INFO]: __main__.py(__init__:165) >> --------------------------------------
2025-05-22 12:49:05.624792 [INFO]: __main__.py(__init__:189) >>     * Timestamp: 20250522_174905_575700
2025-05-22 12:49:05.625317 [INFO]: __main__.py(__init__:189) >>     * Version Info: branch bugfix/297-v3-fails with commit 01890f837b353fdb47622785a32c6e221e4b00e5
2025-05-22 12:49:05.625760 [INFO]: __main__.py(__init__:189) >>     * Mode: Serial
2025-05-22 12:49:05.626092 [INFO]: __main__.py(__init__:189) >>     * Variable List: ['mlotst']
2025-05-22 12:49:05.626372 [INFO]: __main__.py(__init__:189) >>     * Input Path: /lcrc/group/e3sm2/DSM/Ops/test/e2c_input
2025-05-22 12:49:05.626631 [INFO]: __main__.py(__init__:189) >>     * Output Path: /lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails
2025-05-22 12:49:05.626942 [INFO]: __main__.py(__init__:189) >>     * Precheck Path: None
2025-05-22 12:49:05.627202 [INFO]: __main__.py(__init__:189) >>     * Log Path: /lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails/20250522_174905_575700.log
2025-05-22 12:49:05.627468 [INFO]: __main__.py(__init__:189) >>     * CMOR Log Path: /lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails/cmor_logs
2025-05-22 12:49:05.627743 [INFO]: __main__.py(__init__:189) >>     * Temp Path for Processing MPAS Files: /lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails/tmp
2025-05-22 12:49:05.628015 [INFO]: __main__.py(__init__:189) >>     * Frequency: mon
2025-05-22 12:49:05.628279 [INFO]: __main__.py(__init__:189) >>     * Realm: mpaso
2025-05-22 12:49:06.016511 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
  ds = xr.open_dataset(path)

2025-05-22 12:49:06.193088 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
  ds = xr.open_dataset(path)

2025-05-22 12:49:06.320772 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
  ds = xr.open_dataset(path)

2025-05-22 12:49:06.520533 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
  ds = xr.open_dataset(path)

2025-05-22 12:49:06.832609 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
  ds = xr.open_dataset(path)

2025-05-22 12:49:06.969639 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta<br/>
**Q&A: Bug in CMOR Log for MPAS Variables Saves in PWD, Not in Output Directory under `cmor_logs`**
==============================================================================================

**Q: What is the issue with the CMOR log for MPAS variables?**
---------------------------------------------------------

A: The issue is that the CMOR log for MPAS variables is saved in the PWD (current working directory) instead of the specified output directory under `cmor_logs`.

**Q: What is the expected behavior?**
--------------------------------------

A: The expected behavior is that the CMOR log for MPAS variables should be saved in the specified output directory under `cmor_logs`, just like all other variables.

**Q: What is the cause of the issue?**
--------------------------------------

A: The cause of the issue is that the `mpas_vars` modules have their own `cmorize` function, which saves the `cmor_logs` under the PWD. This is in conflict with the new behavior introduced in #289, where the `cmor_logs` output is saved under the `-o/--output` directory.

**Q: How can I reproduce the issue?**
--------------------------------------

A: You can reproduce the issue by running the minimal complete verifiable example (MVCE) provided in the issue report. This will create a new directory with the CMOR log for MPAS variables saved in the PWD instead of the specified output directory.

**Q: What are the relevant log messages?**
-----------------------------------------

A: The relevant log messages are the ones that indicate the CMOR log is being saved in the PWD instead of the specified output directory. These messages are:

* `2025-05-22 12:49:05.607233 [INFO]: __main__.py(__init__:163) >> --------------------------------------`
* `2025-05-22 12:49:05.608262 [INFO]: __main__.py(__init__:164) >> | E3SM to CMIP Configuration`
* `2025-05-22 12:49:05.608943 [INFO]: __main__.py(__init__:165) >> --------------------------------------`
* `2025-05-22 12:49:05.624792 [INFO]: __main__.py(__init__:189) >>     * CMOR Log Path: /lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails/cmor_logs`

**Q: What are the relevant warnings?**
--------------------------------------

A: The relevant warnings are the ones that indicate a future warning about the `decode_timedelta` parameter in xarray. These warnings are:

* `2025-05-22 12:49:06.016511 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.`
* `ds = xr.open_dataset(path)`

**Q: What is the error message?**
------------------------------

A: The error message is a `TypeError` that occurs when trying to write the CMOR log to the PWD instead of the specified output directory. The error message is:

`TypeError: handle() takes 3 positional arguments but 5 were given`

**Q: What is the solution to the issue?**
-----------------------------------------

A: The solution to the issue is to update the `mpas_vars` modules to use the new behavior introduced in #289, where the `cmor_logs` output is saved under the `-o/--output` directory. This can be done by modifying the `cmorize` function in the `mpas_vars` modules to use the specified output directory instead of the PWD.