Previous incidents

Our engineering team has identified a network disruption at our US-TX-4 datacenter, caused by a required firmware update for our router.

To resolve this, we will deploy an emergency fix at 21:00 UTC on March 11, 2025, with a maximum expected downtime of 10-15 minutes.

The update was successfully completed.

Our primary ISP circuit for the US-NC-1 data center experienced an outage. The secondary router failed to take over due to a known firmware issue that was scheduled for a later patch. We’ve now upgraded the router to the latest patched version and are running on the secondary circuit.

The issue has been resolved.

We have discovered an issue affecting pods running in CA-MTL-1 when using volume disk or network storage. When executing commands, the process may hang, although the file is still created successfully.

So far, this issue primarily impacts most H100 GPUs and a few A40 GPUs. Our team is actively investigating and will provide updates here as we learn more.

We have identify the root cause of the issue, team is pushing the updates to machine.

All machines have been updated, and...

We are currently upgrading the EU-CZ-1 data center, and all machines are offline during this process. Services hosted in this region are temporarily unavailable during this period.

We’ve successfully brought most of the machines online. However, due to some technical issues, we need a bit more time to restore the remaining ones. Thanks for your patience, we’ll keep you posted!

All machines in the EU-CZ-1 data center are now fully online. The data center upgrade is complete, t...

We experienced an issue affecting serverless requests from 10:00 PM to 10:23 PM UTC. This was due to an update made to improve system capacity in the NYC region, which led to temporary request issues.

The issue has been identified and resolved, and we’ve taken steps to minimize future risks.

We are still seeing issues, and our team is actively investigating. We’ll provide further updates as soon as we have more information.

We have identified the issue and will be rolling out a...

We’re currently experiencing performance issues with network volumes in the CA-MTL-1 data center. Our team is investigating the issue, and we’ll provide updates as soon as possible.

We detected a performance issue with one of the chunk servers and have isolated the affected server.

The issue has been resolved

We are currently experiencing issues accessing the Main UI Console Page. Our team is actively investigating the cause, and we will provide updates as soon as we have more information.

Our authentication provider, Clerk, is experiencing issues and is currently down. We are closely monitoring the situation and will provide updates as soon as we have more information.

Workaround:

1. Our GraphQL API and serverless endpoints are unaffected.
2. Users can still call the GraphQL ...

CA-MTL-3 is suffering a network disruption due to an upstream provider issue. We are in contact with the provider and are working to restore network availability now.

the network is restored

US-TX-4 is suffering a network disruption due to an upstream provider issue. We are in contact with the provider and are working to restore network availability now.

The US-TX-4 region will experience a short network disruption at approximately 01/23/2025 5:30 PM CST for about 10 minutes due to an emergency firewall update.

We apologize for any inconvenience and appreciate your understanding as we perform this critical update.

The issue affecting US-TX-4 has been resolve...

The network issue at the data center has been resolved. Thank you for your patience.