We have discovered an issue affecting pods running in CA-MTL-1 when using volume disk or network storage. When executing commands, the process may hang, although the file is still created successfully.

So far, this issue primarily impacts most H100 GPUs and a few A40 GPUs. Our team is actively investigating and will provide updates here as we learn more.

We have identify the root cause of the issue, team is pushing the updates to machine.

All machines have been updated, and the issue is now resolved.