Back to overview
Degraded

Elevated network packetloss in US-OR-1 DC

Sep 3, 2024 at 5:02pm UTC
Affected services
graphql: api.runpod.io

Resolved
Sep 4, 2024 at 1:59am UTC

We have received confirmation from the upstream network provider and we have validated that this issue is resolved.

The root cause was a network protection ruleset which engaged in a false-positive manner to drop a selection of packets. This resulted in failure to establish connections and impact to bandwidth over TCP/QUIC connections.

We will provide an RCA once we receive the report from the upstream provider.

Updated
Sep 3, 2024 at 9:16pm UTC

Packetloss has returned to nominal levels. We are still monitoring closely.

Updated
Sep 3, 2024 at 8:19pm UTC

The network provider is still in the process of mitigating the issue. We will provide regular updates as they make progress.

Updated
Sep 3, 2024 at 7:32pm UTC

The network provider is still in the process of mitigating the issue. We will provide regular updates as they make progress.

Updated
Sep 3, 2024 at 6:49pm UTC

The network provider is still in the process of mitigating the issue.

Updated
Sep 3, 2024 at 6:21pm UTC

The network provider is diagnosing the issue and has isolated it to exist within a specific network segment.

Created
Sep 3, 2024 at 5:02pm UTC

We are currently observing elevated packet loss within the US-OR-1 DC. This is resulting in increased connection resets and failures. We are engaging with the network provider to determine the root cause.