Incidents reported on status page for Ramnode https://nyc.status.ramnode.com/ https://d1lppblt9t2x15.cloudfront.net/logos/15575d430dd7fb13453a74703f1eaf4e.png https://nyc.status.ramnode.com/ en https://nyc.status.ramnode.com/ Mon, 06 Oct 2025 06:58:12 +0000 https://nyc.status.ramnode.com/#e7cd18a2797ae8e172ba6abf4cd814b629a29a8a750820ba5c53a99e129477a3 nyc-vds-5 recovered https://nyc.status.ramnode.com/ Mon, 06 Oct 2025 06:55:40 +0000 https://nyc.status.ramnode.com/#e7cd18a2797ae8e172ba6abf4cd814b629a29a8a750820ba5c53a99e129477a3 nyc-vds-5 went down https://nyc.status.ramnode.com/ Thu, 18 Sep 2025 23:26:04 +0000 https://nyc.status.ramnode.com/#2e2faec7ac939a8a69dbe906547a2e131cb5a6170beb3f83be8c141e9731be8f nyc-vds-5 recovered https://nyc.status.ramnode.com/ Thu, 18 Sep 2025 23:23:33 +0000 https://nyc.status.ramnode.com/#2e2faec7ac939a8a69dbe906547a2e131cb5a6170beb3f83be8c141e9731be8f nyc-vds-5 went down https://nyc.status.ramnode.com/incident/555498 Fri, 02 May 2025 19:31:00 -0000 https://nyc.status.ramnode.com/incident/555498#c7f73b88e2b6d28056b68059c97ef2c96f9a5577327522908dfbd948e77aa1b3 This has been resolved now. https://nyc.status.ramnode.com/incident/555465 Fri, 02 May 2025 15:30:00 -0000 https://nyc.status.ramnode.com/incident/555465#20b1ead53c1561e18f0ddf8aff713007c88a400804b786372205c7b991228e65 This has been resolved now. https://nyc.status.ramnode.com/incident/555498 Fri, 02 May 2025 13:27:00 -0000 https://nyc.status.ramnode.com/incident/555498#b067a8babc24b3744fb4edc6d9ad471b658996d7a5b6a2f5fbd090709f5332da We're observing a connectivity issue with instances running on NYC-MKVM-1. Our engineers are working on it. https://nyc.status.ramnode.com/incident/555465 Fri, 02 May 2025 12:08:00 -0000 https://nyc.status.ramnode.com/incident/555465#1b646fec19df822584e8b9921b5ae42ded403bc323842f94e730d6d7a847c1e4 We're currently experiencing an issue with nyc-mvkm-2 which is preventing instances from booting. Our engineers are actively working on it, and we will update you when this is resolved. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 11:37:00 -0000 https://nyc.status.ramnode.com/incident/554978#0ca048d865a50f335f9e9ec7a329fd76fe17bf89b6d95702a461f905df3f25d2 We have an "all clear" from our engineering team now. If you continue experiencing any issue with your server, don't hesitate to contact our support team. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Fri, 02 May 2025 08:28:00 -0000 https://nyc.status.ramnode.com/incident/554978#25a939f5c76fc1fae2251241c508d3969a9010520c6291c96a6e573cd0fe96b1 We’ve made significant progress toward recovery. The previously unreachable storage node is back online. Our team has begun reintegrating drives into the CEPH cluster and is closely monitoring the process. This is a critical step in restoring full functionality, and we’re proceeding carefully to ensure cluster stability and data integrity. Instance operations are working again, and we expect to see all services gradually recover as the cluster health improves. Thank you again for your patience — we’ll continue to keep you updated as we move forward. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 22:17:00 -0000 https://nyc.status.ramnode.com/incident/554978#9e6b8ba5307483f630070863b030c463a203ed5d69926bc13c261ce7d24ec549 This remains our highest priority internally. We are continuing recovery work on the backend storage system and are taking every precaution to ensure data integrity throughout the process. Following extensive discussions with our engineering team and storage vendor, we now have a more concrete estimate: we expect full restoration of functionality within the next 6 to 12 hours, barring any unforeseen complications. This estimate reflects the time required to complete filesystem recovery and verify the health of the affected Ceph cluster components. We understand how disruptive this incident has been and we deeply appreciate your patience as we work toward a safe and stable resolution. We will provide a final update once recovery is complete or if there is any significant change in the timeline. Thank you again for bearing with us. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 20:01:00 -0000 https://nyc.status.ramnode.com/incident/554978#b0ff65dcbea9b516c1c9f9320244b2553703de3003a5f375d8fca845da59a5ed Recovery work is still in progress. While no major change has occurred yet, our team is engaged and working through several layers of the system to safely bring services back online. Thank you again for bearing with us — we will keep these updates coming regularly. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 18:30:00 -0000 https://nyc.status.ramnode.com/incident/554978#975db6caf1d3baba7f29e5eb92f7de860bf5c23e6305e3a7a25d714310bed595 We’re still working through the ongoing storage issue impacting instance management in this region. Our team is in continuous contact with the datacenter team and openstack engineers, ensuring all recovery efforts remain active. We’ll continue to share updates as we make progress. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 16:44:00 -0000 https://nyc.status.ramnode.com/incident/554978#e4d30cfbd661fd1617d762d38fa5170997b16e46a95364e2526d379b3a8269f5 Our engineering team continues to work on restoring full functionality in the affected region. We remain focused on resolving the underlying storage issue and are closely monitoring system behavior as we proceed through recovery steps. We understand how disruptive this is and sincerely appreciate your ongoing patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/554978 Thu, 01 May 2025 14:19:00 -0000 https://nyc.status.ramnode.com/incident/554978#effe2bfab96f4499f23283531bf1c2134d97a9bf54e7e906d31a95fc12e3f922 We are currently experiencing a disruption in our NYC/EWR cloud region due to a failure in the storage backend following our ongoing datacenter migration (NYC to EWR). A portion of our CEPH storage cluster was physically relocated ahead of schedule, while another portion remains at the old location. One critical node is also currently offline, resulting in a degraded cluster state. As a result, the CephFS (used to access base images and volumes) is in a hung state and blocking all instance operations. This means that starting, stopping, or rebooting instances is currently not possible, although running instances remain online. Additionally, instances with attached volumes are also unable to boot. Our engineering team is actively working to recover the filesystem and restore the cluster’s health. Due to the complexity of this failure and the nature of Ceph’s consistency mechanisms, this process is delicate and time-consuming. At this time, we cannot provide a precise ETA for full restoration. We will continue to post updates as we progress. We sincerely apologize for the inconvenience and understand how disruptive this is. Thank you for your continued patience. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 04:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#fe3c3fcdf1ecf65b98c57a8c3c551fb047a48bc58b656b92dae6503ddc6736a1 Maintenance completed https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 04:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#fe3c3fcdf1ecf65b98c57a8c3c551fb047a48bc58b656b92dae6503ddc6736a1 Maintenance completed https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 04:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#fe3c3fcdf1ecf65b98c57a8c3c551fb047a48bc58b656b92dae6503ddc6736a1 Maintenance completed https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 04:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#fe3c3fcdf1ecf65b98c57a8c3c551fb047a48bc58b656b92dae6503ddc6736a1 Maintenance completed https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 04:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#fe3c3fcdf1ecf65b98c57a8c3c551fb047a48bc58b656b92dae6503ddc6736a1 Maintenance completed https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 04:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#fe3c3fcdf1ecf65b98c57a8c3c551fb047a48bc58b656b92dae6503ddc6736a1 Maintenance completed https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 04:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#fe3c3fcdf1ecf65b98c57a8c3c551fb047a48bc58b656b92dae6503ddc6736a1 Maintenance completed https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 04:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#fe3c3fcdf1ecf65b98c57a8c3c551fb047a48bc58b656b92dae6503ddc6736a1 Maintenance completed https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 04:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#fe3c3fcdf1ecf65b98c57a8c3c551fb047a48bc58b656b92dae6503ddc6736a1 Maintenance completed https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable. https://nyc.status.ramnode.com/incident/348422 Sun, 31 Mar 2024 01:00:00 -0000 https://nyc.status.ramnode.com/incident/348422#dfe23d7be7bf3188a13a01812d92f14ee4370562112890e23c35dceaaf292b78 We are performing emergency maintenance on a cabinet in our NYC facility. Instances on indicated servers may be intermittently unavailable.