Elevated network packetloss in US-OR-1 DC
Status Report Update State Resolved
Sep 03 at 06:59pm PDT
We have received confirmation from the upstream network provider and we have validated that this issue is resolved.
The root cause was a network protection ruleset which engaged in a false-positive manner to drop a selection of packets. This resulted in failure to establish connections and impact to bandwidth over TCP/QUIC connections.
We will provide an RCA once we receive the report from the upstream provider.
Affected services
GraphQL API
Status Report Update State Updated
Sep 03 at 02:16pm PDT
Packetloss has returned to nominal levels. We are still monitoring closely.
Affected services
GraphQL API
Status Report Update State Updated
Sep 03 at 01:19pm PDT
The network provider is still in the process of mitigating the issue. We will provide regular updates as they make progress.
Affected services
GraphQL API
Status Report Update State Updated
Sep 03 at 12:32pm PDT
The network provider is still in the process of mitigating the issue. We will provide regular updates as they make progress.
Affected services
GraphQL API
Status Report Update State Updated
Sep 03 at 11:49am PDT
The network provider is still in the process of mitigating the issue.
Affected services
GraphQL API
Status Report Update State Updated
Sep 03 at 11:21am PDT
The network provider is diagnosing the issue and has isolated it to exist within a specific network segment.
Affected services
GraphQL API
Status Report Update State Created
Sep 03 at 10:02am PDT
We are currently observing elevated packet loss within the US-OR-1 DC. This is resulting in increased connection resets and failures. We are engaging with the network provider to determine the root cause.
Affected services
GraphQL API