devsecops

Kubernetes in the Classified Crowd: Orchestrating Containers Behind the Government Firewall (Part 1)

AB Engineering

• 7 min read
Kubernetes in the Classified Crowd: Orchestrating Containers Behind the Government Firewall (Part 1)

In a world where "moving to the cloud" has become the default corporate strategy, government agencies face a unique set of challenges that make their cloud migration journeys more akin to "moving to a highly-regulated, FIPS-compliant, ATO-approved, thoroughly-inspected cloud while filling out paperwork in triplicate." As the saying goes in government IT circles, "We want innovation at the speed of relevance, just after completing the proper risk assessment forms."

Kubernetes has emerged as the de facto standard for container orchestration across the commercial sector, but its adoption in Department of Defense (DoD) and Federal Government environments introduces layers of complexity that would make even the most seasoned DevOps engineers wince. This post kicks off our series on navigating these choppy waters, where compliance requirements and classified workloads meet cutting-edge container orchestration.

The Orchestration Situation: Why Kubernetes Matters to Uncle Sam

Kubernetes, often abbreviated as K8s (because government loves acronyms almost as much as Silicon Valley), is an open-source platform that automates the deployment, scaling, and management of containerized applications. For federal agencies undertaking application modernization initiatives, Kubernetes represents more than just a trendy technology – it offers a strategic foundation for achieving critical objectives like accelerating innovation, optimizing costs, and enhancing security posture.

The benefits are compelling: containers enable agencies to modernize legacy applications and build new ones that leverage evolving cloud services. This containerized approach allows government entities to develop applications and deploy valuable resources with unprecedented speed. As Brian Douglas, Director of Public Sector at Kubecost, notes: "Kubecost partners with many federal and government agencies that are actively transitioning to Kubernetes. The benefits of a modernization effort range from improved security, faster application deployment, agility to scale, and the ability—when done correctly—to optimize resources".

However, as government agencies embrace this technology, they quickly discover that Kubernetes in classified environments isn't just about mastering YAML files and understanding pod lifecycles – it's about navigating a complex web of security frameworks, compliance requirements, and authorization processes.

Welcome to the Security Labyrinth: The Government IT Landscape

Government IT environments operate under stringent security requirements that make commercial compliance frameworks look like mere suggestions. The stakes are considerably higher – we're talking about systems that may handle national security information, control critical infrastructure, or process sensitive citizen data.

The DoD classifies cloud security by Impact Levels (IL) based on data sensitivity and potential impact if compromised. These levels range from IL2 for publicly releasable information to IL6 for classified data, with each level requiring progressively more stringent security controls. Understanding these impact levels is essential, as they dictate the security requirements your Kubernetes implementation must satisfy.

Beyond impact levels, government Kubernetes deployments must navigate a complex ecosystem of security frameworks:

  • NIST Special Publication 800-53, which details security and privacy controls for federal information systems- Department of Defense Risk Management Framework (RMF)
  • DISA Security Technical Implementation Guides (STIGs) for Kubernetes- NSA and CISA Kubernetes Hardening Guidance
  • Center for Internet Security (CIS) Kubernetes Benchmarks

If your eyes haven't glazed over yet, congratulations – you might have what it takes to be a government security architect.

Challenges: Where DevOps Dreams Meet Compliance Realities

The Compliance Conundrum

Validating compliance with frameworks like NIST 800-53 is the number one blocker for faster application delivery in government settings. Kubernetes presents a particular challenge due to its dynamic nature – it's difficult to detect when assets fall out of compliance in such a fluid environment.

Studies have shown that the annual cost of non-compliance to businesses runs an average of $14.8 million, while the cost of compliance averages $5.5 million. In government environments, the stakes are even higher, as non-compliance can delay Authorization to Operate (ATO) approvals, halt project timelines, and potentially expose sensitive data to security risks.

Container Security: Trust Issues and Supply Chain Concerns

The NSA and CISA identify three common sources of compromise in Kubernetes: supply chain risks, malicious threat actors, and insider threats. Supply chain risks are particularly challenging to mitigate and can arise during the container build cycle or infrastructure acquisition.

For government agencies, these concerns are amplified by requirements to validate the provenance of all software components. As one DoD guidance document notes, "In order to prevent any combination of human errors, supply chain interdictions, unintended code, and support the creation of a software bill of materials (SBOM), the adoption of an approved software platform and development pipeline(s) are critical".

The Open Source Dilemma

Government agencies have historically had a complicated relationship with open-source software. One ISO-27001 auditor famously "flagged" a company's use of open-source software as a security risk, causing management to nearly ban all open-source tools. This misconception persists in some government circles, despite the fact that the U.S. Department of Defense itself has clarified that it does not consider open-source software to be inherently risky.

For Kubernetes – an open-source system – this means government deployments must include rigorous evaluation processes. As one Reddit commenter advised, organizations should develop "a framework to reduce the risks" by assessing factors like how long a package has been around, the quality of its code, how often it's updated, and what professional support options exist.

NSA and CISA Weigh In: Hardening Guidance from the Experts

In August 2021, the National Security Agency (NSA) and the Cybersecurity and Infrastructure Security Agency (CISA) released "Kubernetes Hardening Guidance," which was subsequently updated with clarifications and additions to logging and threat detection.

The guidance identifies common motivations for targeting Kubernetes clusters: data theft, computational power theft, or denial of service. It provides detailed recommendations for hardening Kubernetes systems, including:

  1. Scanning containers and Pods for vulnerabilities or misconfigurations
  2. Running containers and Pods with the least privileges possible
  3. Using network separation to control the amount of damage a compromise can cause
  4. Implementing firewalls to limit unneeded network connectivity and encryption to protect confidentiality
  5. Using strong authentication and authorization to limit user and administrator access

These recommendations form the baseline for secure Kubernetes deployments in government environments, but they're just the beginning of a comprehensive security strategy.

The Role-Based Reality: Access Control in Government Kubernetes

In government environments, Role-Based Access Control (RBAC) isn't just a best practice – it's a critical security requirement. RBAC in Kubernetes presents unique challenges, especially in multi-tenant setups where robust access separation and efficient permission management are essential.

Implementing RBAC effectively requires attention to role hierarchies, namespace isolation, and scalable access management. However, even well-designed RBAC systems can face challenges in government settings due to the "separation of duties" principle. As one expert explains: "You shouldn't manage the system and be responsible for the security of the system by rule. InfoSec should write policies and rules for others to follow and then audit their work. Basically treat K8s as an OS not a piece of software".

This creates a new operational model where Kubernetes administrators and security teams must collaborate closely while maintaining clear boundaries of responsibility – a dance that requires both technical skill and organizational finesse.

Building for Authority to Operate (ATO): The Government Deployment Pathway

For many government projects, securing an Authority to Operate (ATO) represents a critical milestone. This formal declaration that a system meets security requirements can make or break a Kubernetes deployment.

A Reddit user described their experience joining a project that needed to secure an ATO for an application deployed in a DoD environment: "Our team is quite small, and we don't have anyone dedicated to navigating the ATO process... My primary concern is how to secure the Docker runtime that will host our app, and I find myself a bit uncertain about where to begin".

This uncertainty is common among teams new to government deployments. For Kubernetes, the ATO process typically requires demonstrating compliance with applicable STIGs, implementing continuous monitoring capabilities, and establishing an active cyber defense posture.

An evolution in this space is the concept of Continuous Authorization (cATO), which requires three core competencies:

  • Continuous Monitoring of RMF controls
  • Active Cyber Defense
  • Use of an approved DevSecOps Reference Design

Choosing Your Kubernetes Distribution: Not All K8s Are Created Equal

When deploying Kubernetes in government environments, the choice of distribution becomes critical. Several options have emerged that cater specifically to government security requirements:

  • Rancher RKE2, which has "reasonably easy configuration compared to kubeadm, and it is designed with the explicit goal of US Government security compliance"
  • Enterprise Kubernetes distributions with FedRAMP certifications
  • Platform One's Kubernetes offerings, which come pre-configured for DoD environments

However, as one expert cautions: "Don't assume Rancher/RKE2, EKS anywhere, or any 'out-of-the-box solution' is going to provide 100% of what you need. K8s is not impossibly complex, but it is more complex than wrangling hosts and a hypervisor. You will need to know the intimate details of your chosen CNI and CSI at a minimum".

The Data-Sharing Dilemma: Balancing Security with Function

One of the most significant challenges in government Kubernetes deployments involves data sharing between components. A recent study explored efficient and secure methods for sharing data between co-located containers on the same physical host for workloads deployed on Kubernetes.

The research demonstrated that using CSI shared local volume mounts can enable secure file sharing and even create shared memory regions, providing viable approaches to data sharing for layered storage systems in Kubernetes. These findings are particularly relevant for data-intensive workloads like distributed deep learning that generate significant temporary data between pipeline stages.

For government agencies dealing with sensitive information, these secure data-sharing mechanisms represent a crucial capability, enabling complex workflows while maintaining appropriate security boundaries.

As we've explored, deploying Kubernetes in government environments presents unique challenges that require specialized knowledge and careful planning. The intersection of container orchestration technology with strict compliance requirements creates a complex landscape that few commercial deployments encounter.

Yet, the benefits are compelling enough that agencies across the federal government continue to invest in Kubernetes capabilities. From improved application deployment speed to enhanced security through immutable infrastructure patterns, Kubernetes offers agencies a path to modernization that aligns with both mission needs and security requirements.

In our next post, "Planning a Kubernetes Deployment: Aligning with DoD Security Policies", we'll dive deeper into the practical aspects of designing a Kubernetes architecture that satisfies DoD security requirements from day one. We'll explore reference architectures approved for government use, examine the crucial role of the software supply chain in secure deployments, and provide actionable guidance for teams preparing to embark on their government Kubernetes journey.

Until then, remember that in government IT, we don't just "shift left" on security – we start with security, continue with security, and finish with even more security, all while maintaining detailed documentation of our security. Because nothing says "mission accomplished" like a fully compliant, properly hardened Kubernetes cluster that passes its authorization review on the first try.