GitHub Availability Report: November 2023
source link: https://github.blog/2023-12-13-github-availability-report-november-2023/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
In November, we experienced one incident that resulted in degraded performance across GitHub services.
November 3 18:42 UTC (lasting 38 minutes)
Between 18:42 and 19:20 UTC on November 3, the GitHub authorization service experienced excessive application memory use, leading to failed authorization requests and users getting 404 or error responses on most page and API requests.
A performance and resilience optimization to the authorization microservice contained a memory leak that was exposed under high traffic. Testing did not expose the service to sufficient traffic to discover the leak, allowing it to graduate to production at 18:37 UTC. The memory leak under high load caused pods to crash repeatedly starting at 18:42 UTC, failing authorization checks in their default closed state. These failures started triggering alerts at 18:44 UTC. Rolling back the authorization service change was delayed as parts of the deployment infrastructure relied on the authorization service and required manual intervention to complete. Rollback completed at 19:08 UTC and all impacted GitHub features recovered after pods came back online.
To reduce the risk of future deployments, we implemented changes to our rollout strategy by including additional monitoring and checks, which automatically block a deployment from proceeding if key metrics are not satisfactory. To reduce our time to recover in the future, we have removed dependencies between the authorization service and the tools needed to roll back changes.
Please follow our status page for real-time updates on status changes and post-incident recaps. To learn more about what we’re working on, check out the GitHub Engineering Blog.
The GitHub Insider Newsletter
Discover tips, technical guides, and best practices in our monthly newsletter for developers.
SubscribeMore on GitHub Availability Report
GitHub Availability Report: October 2023
In October, we experienced two incidents that resulted in degraded performance across GitHub services.
GitHub Availability Report: September 2023
In September, we experienced two incidents that resulted in degraded performance across GitHub services.
GitHub Availability Report: August 2023
In August, we experienced two incidents that resulted in degraded performance across GitHub services.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK