Securing Deployments with eBPF: GitHub's Approach to Circular Dependencies

At GitHub, we host all our code on github.com, which creates a fascinating challenge: if the platform goes down, we can't access the source code needed to fix it. This is a classic circular dependency. To address this, we've turned to eBPF (extended Berkeley Packet Filter), a powerful kernel technology that allows us to monitor and restrict network calls made by deployment scripts. Below, we answer common questions about how eBPF helps us break these cycles and keep deployments safe.

What is the circular dependency problem GitHub faced?

GitHub relies on its own platform to host its source code and release artifacts. This creates a circular dependency: to deploy updates to github.com, we need access to github.com. If an outage occurs, we lose the very tools required to fix it. For example, a MySQL outage might prevent us from fetching a configuration script from a repository. While we maintain offline mirrors for emergencies, this doesn't solve all cases. Deployment scripts themselves can introduce new circular dependencies—for instance, by attempting to download a binary from GitHub during a recovery operation. The core challenge is ensuring that any code executed during an incident does not inadvertently depend on services that are currently unavailable.

Why are circular dependencies so dangerous during outages?

During an outage, every second counts. Circular dependencies can turn a simple fix into a prolonged incident because they create a deadlock where the recovery process itself requires the very service that is broken. For example, if a deploy script tries to pull a tool from GitHub releases but GitHub's release API is down, the script fails, blocking the fix. Even worse, hidden dependencies may cause scripts to hang or fail silently, as when a tool checks for updates online and cannot reach GitHub. Transient dependencies can propagate failures across services, magnifying the impact. In all cases, the result is the same: delayed recovery and prolonged user impact, which is why GitHub prioritized breaking these cycles with eBPF.

What are the three types of circular dependencies GitHub identified?

GitHub categorized circular dependencies into three types:

• Direct dependencies: The deployment script explicitly tries to fetch something from GitHub (e.g., a binary, a configuration file). If GitHub is down, the script cannot proceed.
• Hidden dependencies: A tool already on the machine, when executed, attempts to check for updates online. This behavior is not obvious from the script itself and can cause failures or hangs even if the primary script logic is sound.
• Transient dependencies: The script calls an internal service via API, which in turn tries to reach GitHub or another external resource. The failure propagates back, making it hard to trace the root cause.

How did GitHub traditionally mitigate these dependencies?

Historically, the responsibility fell on each team that manages stateful hosts. Engineers had to manually review every deployment script to identify potential circular dependencies—a time-consuming and error-prone process. While best practices like using local mirrors and offline binaries helped, they didn't guarantee safety. A tool's update check, for instance, could still slip through. Moreover, as services evolved and scripts grew complex, it became nearly impossible to anticipate every possible circular reference. GitHub needed a systematic, automated approach that could catch these issues at runtime without requiring changes to existing scripts.

How does eBPF help solve circular dependencies in deployments?

eBPF allows GitHub to insert custom monitoring and control logic directly into the Linux kernel, without modifying applications. For deployment safety, eBPF programs can intercept every network call made by a deploy script or its child processes. They examine the destination (IP address, port, hostname) and compare it against a whitelist of allowed services (like local mirrors or internal APIs). If a call attempts to reach a prohibited endpoint—for example, the production github.com—the eBPF program can block it or alert the operator. This provides real-time enforcement of dependency policies, ensuring that no script accidentally introduces a circular dependency during an incident. eBPF works transparently, so existing tools and scripts don't need to be rewritten.

Can you give an example of a direct dependency and how eBPF stops it?

Consider a MySQL deploy script that attempts to download a new release of an open-source tool from https://github.com/example/tool/releases/latest. During a GitHub outage, this request would fail, halting the deployment. With eBPF, we attach a program that observes all outgoing HTTP connections. When the script tries to connect to a known hostname pattern like *.github.com, the eBPF program checks if that host is on the allowed list. In incident mode, GitHub's public endpoints are blocked. The eBPF program can either drop the packet (causing the script to fail fast with a clear error) or reroute it to a local mirror. This prevents the script from ever sending the request to the external service, breaking the circular dependency instantly.

How does eBPF detect hidden dependencies that are not obvious in the script?

Hidden dependencies are the most insidious because they are not visible in the deployment script's main logic. For example, a servicing tool like apt-get or a custom agent may automatically check for updates when invoked. eBPF solves this by monitoring the entire process tree. When the deploy script spawns child processes, eBPF programs attached to those processes also inspect all network activity. So if a child tool tries to contact a remote repository to check for an update, the eBPF program sees that call just as it would for the parent. This gives us complete visibility into the full chain of execution. We can then enforce the same whitelist-based blocking, ensuring that even the most incidental network calls cannot create a circular dependency.