We continue to share monthly updates on the work behind GitHub’s availability and reliability.

The short version of October: we added substantial independently scalable Azure capacity and removed several ways that failures could escalate. However, write outages, monitoring gaps during data migrations, and a regional network disruption reinforced the need for capacity growth to be paired with safer operations, stronger isolation, and validated regional recovery.

Recent findings suggest that as of November 9, 140 of 151 production services and 32 of 50 MySQL primaries were running in Azure Central US. Azure served 63% of migrated-service reads, 66% of monolith reads, and 52% of Git reads. We also completed a compute expansion across all nine Central US Kubernetes clusters, more than doubling capacity.

According to recent reports, that capacity is already carrying real growth. Migrated-service traffic has increased 82% since March, with Central US absorbing the growth beyond the highest weekly-average load previously served from our existing datacenter. This gives GitHub more room to grow without pushing the original environment beyond its prior operating range.

It has been revealed that we also made several targeted reliability improvements. Actions changes increased capacity for critical Azure control-plane requests. A separate NAT-gateway mitigation reduced usage by 85%; four production jobs failed during the change, compared with approximately 800 expected without the mitigation. Connection limits and other client improvements now also reduce the chance that a dependency interruption becomes a broader reconnect storm.

We removed concentrated sources of database load as well. A high-traffic GitHub Actions endpoint handling more than 250 million requests per day moved to a more efficient service path, reducing peak database reads by 85%. Requiring at least three characters before searching repository topics reduced slow or terminated queries for that pattern by approximately 98%.

Reliability was still not where it needed to be. The October 7 write outages exposed an unsafe interaction between database orchestration and production routing. An October 29 Azure network disruption affected all three Central US availability zones and at least 34,128 signed-in customers. More details on these and other incidents follow.

The East US 3 buildout is about one week behind strategy. The next phase starts with its first validated traffic ramp in the first week of December, while we continue hardening migration safety, dependency isolation, and regional recovery.

The principle remains the same: availability, then capacity, then features.

Duration: approximately 3 hours and 45 minutes.

Customers may have seen GitHub Actions jobs on GitHub-hosted runners stay queued much longer than usual, and some workflow runs failed to start. During the worst stretch, which lasted about 90 minutes, most jobs on GitHub-hosted runners did not start within five minutes. Impact eased as we failed over to healthy regions and added capacity. Self-hosted runners were not affected. GitHub Pages sites that build with actions were slow to update, though existing sites continued to serve normally. Some customers also saw errors in repository lists, licensing and billing pages, and some Copilot features.

Actions jobs stayed queued or failed to start because the service that assigns jobs to GitHub-hosted runners stopped working reliably. A partial networking failure on our cloud provider’s side disrupted some of the connections between one of our datacenter regions and the cloud-hosted databases that service depends on. Instead of failing fast, the service waited for those calls to time out. The waiting calls piled up until the service ran out of memory and restarted.

It has been revealed that jobs then shifted to the same service in a healthy region, which did not have enough capacity for the extra demand and began failing too. At the same time, runners that went several minutes without receiving a job shut down and were slow to be replaced, so fewer runners were available to pick up jobs.

How are we making incidents like this less likely or less impactful?

Duration: approximately 6 minutes of failed or slow requests across github.com, followed by approximately 3 hours and 30 minutes during which enterprise migrations were paused.

Recent findings suggest that customers may have seen server errors or slow responses when signing in, working with repositories and pull requests, calling the API, or using GitHub Codespaces. Retrying a short time later would have succeeded, and delayed webhook deliveries caught up soon after. At the peak, about 1.3% of all requests failed or timed out, and about 6.4K organizations saw at least one failed or slow request.

Migrations were paused for customers using GitHub Enterprise Importer, and were resumed once a fix was deployed.

A surge in migration activity caused GitHub Enterprise Importer to send a burst of very large queries to a shared database. These queries took much longer than usual and, together with normal traffic, used all of the database’s available connections, preventing other github.com features relying on the same database from completing their requests.

The importer did not reduce its load when the database became overloaded. The impact ended within minutes, once the importer finished its queued queries and the database’s connections were released.

To prevent a recurrence, we paused migrations until we deployed a mitigation that runs these queries in smaller batches.

How are we making incidents like this less likely or less impactful?

Duration: two separate write outages lasted approximately 10 minutes and 9 minutes, totaling about 19 minutes. The second was a recurrence of the first, with the same cause.

Industry observers note that customers may have seen errors when pushing code, signing in, updating pull requests or issues, or making changes through the API. Webhooks, GitHub Apps and OAuth, actions, code scanning, codespaces, and Copilot were also affected. Viewing content was largely unaffected. Data that was already saved was not lost, and actions that failed during the outages succeeded when retried after recovery.

In a recent development, web and API request failures peaked at 8.9% in the first outage and 9.3% in the second, affecting more than 12,000 and more than 11,000 organizations, respectively. Some organizations may have been affected by both outages.

Recent findings suggest that during regular maintenance, we replaced machines that manage where database writes are sent. One newly introduced machine started with outdated records that still listed retired database servers. Once it was automatically elected leader, it published routing settings that sent writes for several core databases to those retired servers. Incorrect settings passed our existing validation and took effect immediately. Later, an automated maintenance task did not honor a pause on maintenance and triggered another election, and because the new machine had not yet been removed, it became leader again.

How are we making incidents like this less likely or less impactful?

Duration: approximately 1 hour and 28 minutes.

Customers may have seen delays in GitHub Actions workflow status updates, logs, and checks. About 5% of jobs on GitHub-hosted runners experienced delayed starts, a very small share of workflow runs failed, and some requests to the actions jobs REST API returned errors. GitHub Pages deployments and workflow and check webhook events generated by actions runs were also delayed. For part of this time, live updates on github.com and code search were degraded.

An internal actions service that stores workflow logs and run status lost connections to some of its cache servers in one cloud region. Every server running that service then retried those connections at the same time. Together, those retries saturated the network connections between GitHub’s data center and that region, disrupting other services that use the same link.

How are we making incidents like this less likely or less impactful?

Duration: approximately 6 hours and 46 minutes.

Sources indicate that customers may have seen delays in pull request updates, GitHub Actions workflow status updates, and webhook deliveries. After a push, latest commits were slow to appear on pull requests. Workflows triggered by the push started late. Status checks from actions workflows could look stuck on pull requests, which held up merges that require those checks to pass. These features recovered within about 2 to 3.5 hours.

It has been revealed that audit log events were affected the longest. Recent activity took longer to appear when customers searched their audit logs, with delays of up to 1 hour and 50 minutes. Audit log events were fully caught up after about 6 hours and 46 minutes.

Some webhook deliveries from this period may be missing from webhook delivery history, though the webhooks themselves were delivered.

It has been revealed that some servers in our event streaming platform were close to their memory limits, and memory use on these servers grows with the number of client connections. When requests to these servers failed, fresh connections were opened immediately, which increased the number of connections and the memory they used. The increase in memory use slowed down and overloaded the servers.

We took the overloaded servers out of service, and their traffic moved to other servers. Those servers then became overloaded too, and the problem spread to more servers over the following hour. We began rebalancing overloaded partitions onto brokers that had available capacity and this mitigated the concern.

In a recent development, audit logs took the longest to recover. The service that writes audit log events kept restarting on errors from the platform for more than two hours and fell further behind. Once stabilized, it could only write events at a limited rate, so the backlog took several more hours to clear.

How are we making incidents like this less likely or less impactful?

Duration: approximately 1 hour and 32 minutes.

Sources indicate that customers may have seen errors when using several GPT-family models in Copilot. Requests using these models in Copilot Chat in IDEs and on github.com, the Copilot CLI, Copilot code review, and the Copilot coding agent returned errors or did not complete. About one in 10 requests to the affected models failed. Other models continued to work, and choosing another model avoided the errors. Auto model selection reduced the errors by sending some requests to an unaffected model.

Code scanning autofix and code quality, which are built on one specific model without an alternative model to switch to, were affected more heavily.

Copilot reaches these models through an upstream model provider and uses a credential to authenticate with that provider. That credential became invalid, the provider began rejecting our requests as unauthenticated, and our service returned those rejections to clients as authentication errors. Copilot features that use these models send their requests through that provider, so they began failing at the same time.

How are we making incidents like this less likely or less impactful?

Duration: approximately 9 minutes.

Customers may have seen some requests to github.com hang for up to about 10 seconds and then fail with a server error. Pull request merges and Git operations over SSH and Git LFS were hit hardest. At the worst point, about 43% of merge attempts failed, compared with about 3.5% of API calls. Complications, GitHub Pages, notifications, and sign-in for GitHub Apps and OAuth apps also saw intermittent errors.

A database connection pool used by many github.com services reached its limit, and for several minutes no connections were available. Upcoming requests that needed a connection waited until the connection timeout expired and then failed.

It has been revealed that the pool was already running near capacity when some upgrade requests started holding their connections far longer than expected. Each one began writing a change, then ran a separate lookup that could take several seconds, keeping its connection reserved but unused the whole time. With little headroom, those held connections helped tip the pool over.

How are we making incidents like this less likely or less impactful?

Recent findings suggest that follow our status page for real-time updates on status changes and post-incident recaps. To learn more about what we’re working on, check out the engineering section on the GitHub Blog.

Today, we laid out our vision for GitHub as the most comprehensive, end-to-end software delivery system in the world.

See how a record year of growth reached every industry, every field, and every kind of builder in the 2026 Octoverse report.

AI, agents, voice, and physical controls are shifting software development from operating tools to expressing intent—and helping developers stay focused on building.

Discover tips, technical guides, and best practices in our biweekly newsletter just for devs.

→ Read original article