Networking issue with a backend change

Incident Report for CIMcloud

Resolved

We have put in place procedures and checklists to prevent this particular outage from occurring again. Sites have been stable since, with no more outages or ramifications found.

Posted Apr 16, 2021 - 12:30 EDT

Monitoring

We have resolved the networking issues on the various nodes. We are monitoring sites.

Posted Apr 15, 2021 - 19:57 EDT

Identified

We have identified an issue with the networking on our backend network. This was caused by a change made for building new servers. The fact that it caused running nodes to apply the change and begin restarting to apply the change was an unintended consequence. We are rolling this change back and addressing outages.

Posted Apr 15, 2021 - 17:42 EDT