slow on NERSC compute nodes about e3sm-unified HOT 7 CLOSED

I'm wondering it might not be an e3sm-unified problem. I just tried e3sm-diags from e3sm-unified through interactive jobs on haswell. It ran well. However knl has been problematic, it's a known issue: E3SM-Project/e3sm_diags#314. @wagmanbe would you try it again on haswell? If it still gives trouble, could you share your run script and I will try reproduce.

from e3sm-unified.

Are you not seeing these problems on knl when you use an E3SM_Diags development environment? I have always found python packages to run slowly on knl, so I would be surprised if this is specific to E3SM-Unified but can investigate if it appears to be. But I agree that haswell is the recommended option for all python codes.

from e3sm-unified.

It's affecting both knl and haswell. Maybe it's a NERSC issue?
salloc --nodes=1 --partition=debug --time=00:20:00 -C haswell
source /global/cfs/cdirs/e3sm/software/anaconda_envs/load_latest_e3sm_unified.sh <-- Hangs for minutes.
python <--slow
import os
from acme_diags.parameter.core_parameter import CoreParameter <--hangs for minutes.

from e3sm-unified.

NERSC was having problems yesterday afternoon/evening with very slow compute node performance that a few of us experienced, and was posted on their status page: https://www.nersc.gov/live-status/motd/
There's no notice now, so I'd recommend trying again.

from e3sm-unified.

Thank you, but this problem is occurring just the same today.

from e3sm-unified.

In this case, I suspect that the compute node problem is still there. I tried similar commands as below yesterday afternoon and got the same behavior. But tried again much later yesterday, everything looked fine...

It's affecting both knl and haswell. Maybe it's a NERSC issue?
salloc --nodes=1 --partition=debug --time=00:20:00 -C haswell
source /global/cfs/cdirs/e3sm/software/anaconda_envs/load_latest_e3sm_unified.sh <-- Hangs for minutes.
python <--slow
import os
from acme_diags.parameter.core_parameter import CoreParameter <--hangs for minutes.

from e3sm-unified.

It's at least 10x faster this afternoon.

from e3sm-unified.