Back to overview
Degraded

Elevated network packetloss in US-OR-1 DC

Sep 03 at 10:02am PDT
Affected services
GraphQL API

Resolved
Sep 03 at 06:59pm PDT

We have received confirmation from the upstream network provider and we have validated that this issue is resolved.

The root cause was a network protection ruleset which engaged in a false-positive manner to drop a selection of packets. This resulted in failure to establish connections and impact to bandwidth over TCP/QUIC connections.

We will provide an RCA once we receive the report from the upstream provider.

Updated
Sep 03 at 02:16pm PDT

Packetloss has returned to nominal levels. We are still monitoring closely.

Updated
Sep 03 at 01:19pm PDT

The network provider is still in the process of mitigating the issue. We will provide regular updates as they make progress.

Updated
Sep 03 at 12:32pm PDT

The network provider is still in the process of mitigating the issue. We will provide regular updates as they make progress.

Updated
Sep 03 at 11:49am PDT

The network provider is still in the process of mitigating the issue.

Updated
Sep 03 at 11:21am PDT

The network provider is diagnosing the issue and has isolated it to exist within a specific network segment.

Created
Sep 03 at 10:02am PDT

We are currently observing elevated packet loss within the US-OR-1 DC. This is resulting in increased connection resets and failures. We are engaging with the network provider to determine the root cause.