Back to overview
Degraded

Elevated network packetloss in US-OR-1 DC

Sep 03 at 10:02am PDT
Affected services
GraphQL API

Status Report Update State Resolved
Sep 03 at 06:59pm PDT

We have received confirmation from the upstream network provider and we have validated that this issue is resolved.

The root cause was a network protection ruleset which engaged in a false-positive manner to drop a selection of packets. This resulted in failure to establish connections and impact to bandwidth over TCP/QUIC connections.

We will provide an RCA once we receive the report from the upstream provider.

Status Report Update State Updated
Sep 03 at 02:16pm PDT

Packetloss has returned to nominal levels. We are still monitoring closely.

Status Report Update State Updated
Sep 03 at 01:19pm PDT

The network provider is still in the process of mitigating the issue. We will provide regular updates as they make progress.

Status Report Update State Updated
Sep 03 at 12:32pm PDT

The network provider is still in the process of mitigating the issue. We will provide regular updates as they make progress.

Status Report Update State Updated
Sep 03 at 11:49am PDT

The network provider is still in the process of mitigating the issue.

Status Report Update State Updated
Sep 03 at 11:21am PDT

The network provider is diagnosing the issue and has isolated it to exist within a specific network segment.

Status Report Update State Created
Sep 03 at 10:02am PDT

We are currently observing elevated packet loss within the US-OR-1 DC. This is resulting in increased connection resets and failures. We are engaging with the network provider to determine the root cause.