Container images constitute the standard application delivery format in cloud-native environments. The wide distribution and deployment of these container images requires a new set of best practices for ensuring their integrity. While performing image scans to check for known vulnerabilities in operating systems and language packages remains a cornerstone of image security, it is only part of a larger set of security initiatives you need to employ to protect your environments. Understanding the risks at each stage of a container’s lifecycle will inform decisions around image infrastructure and handling to enhance and maintain your organization’s security posture.
Build
Build/CI Infrastructure
If a container image already contains malicious software, it poses an immediate threat at runtime. Security in your CI build infrastructure is every bit as important as in your production environment to prevent the introduction of external vulnerabilities into your images.
Take these important actions to help secure your build infrastructure and pipelines:
- Limit administrative access to the build infrastructure.
- Allow only required network ingress.
- Manage any necessary secrets carefully and grant only minimal required permissions.
- Carefully vet any third-party sites from which source or other files get pulled, using network firewalls to allow access only from trusted sites.
- While using a vulnerability scanner on the resulting images should be standard practice, you also need to ensure the scanner’s security and reliability to get trustworthy results.
Base Image
If you are building images on an existing, third-party base image, you want to select for a few factors:
- Trustworthiness of the base image’s source and host. Does the image come from a known company or open-source group? Is it hosted on a reputable registry? Are the Dockerfile and source code for all components in the image available?
- Update frequency. Avoid images that get updated only infrequently, especially if they do not respond to relevant vulnerability disclosures.
- Default installed software. Starting with a minimal base image and selectively installing exactly the tools and libraries your application requires tends to be more reliable than figuring out what packages you can safely remove from an existing image.
You can choose from an increasing number of secure, minimalist base images. For a given CPU architecture, any recent Linux base image should run on any Docker installation or Kubernetes container runtime, regardless of the cloud provider or platform. A few options among many are:
- Google Distroless - Extremely minimal base images, prebuilt for a few common languages. They lack a package installer, so if you need additional software, you can copy files into the image.
- Red Hat Universal Base Image (UBI) - RHEL-based image available without a Red Hat subscription. This set of images comes in three tiers - minimal, standard platform, and multi-service. It also has images for multiple languages.
Alternatively, you can build a base image from scratch.
Remove Exploitable and Non-Essential Software
If malicious attackers do manage to get access to your running container, you want to give them as few tools as possible to exploit once they get in. Using a minimal base image offers a good starting place, but if your Dockerfiles all install a standard set of multi-purpose tools, you lose the advantage. Keeping the image as minimal as possible has the added bonus of reducing the probability of encountering zero-day vulnerabilities – which will need patching – and keeping the image smaller, which makes storing and pulling it faster.
Restricting the image to required binaries, libraries, and configuration files provides the best protection. In particular, avoid installing the following tools (or remove them if present):
- Package managers: e.g., apt, yum, apk
- Network tools and clients: e.g., wget, curl, netcat, ssh. If you normally use curl to download, say, a configuration file, make the contents into a Kubernetes ConfigMap instead.
- Unix shells: e.g., sh, bash. Note that removing shells also prevents the use of shell scripts at runtime. Instead, use a compiled language when possible.
- Compilers and debuggers. These should be used only in build and development containers, but never in production containers.
If you install these tools in your production images to perform application debugging, you may want to reconsider your workflows and practices. At the very least, consider creating temporary debugging images when issues that cannot otherwise be diagnosed arise. Kubernetes also currently (as of version 1.16) has alpha support for ephemeral containers, which can be placed in an existing pod to facilitate debugging.
Build vs. Runtime Containers
The build tools you use to generate and compile your applications can be exploited when run on production systems. Remember, containers should be treated as temporary, ephemeral entities. Never plan on “patching” or altering a running container. Build a new image and replace the outdated container deployments. Use multi-stage Dockerfiles to keep software compilation out of runtime images.
FROM build-image as build
# Build my stuff
# ...
FROM base-image
# Install my packages
# ...
COPY --from=build /out/my-app /bin/my-app
Secrets
Never bake any secrets into your images, even if the images are for internal use. Secrets include TLS certificate keys, cloud provider credentials, SSH private keys, and database passwords. Anyone who can pull the image can extract the secret. Supplying sensitive data only at runtime also enables you to use the same image in different runtime environments, which should use different credentials. It also simplifies updating expired or revoked secrets without rebuilding the image.
As an alternative to baked-in secrets, supply secrets to Kubernetes pods as Kubernetes secrets, or use another secret management system.
Image Scanning
Images that contain software with security vulnerabilities become vulnerable at runtime. When building an image in your CI pipeline, image scanning must be a requirement for a passing build run. Unsafe images should never get pushed to your production-accessible container registry.
While a number of open-source and proprietary image scanners, in addition to cloud scanning services, are available, they do not all provide the same level of coverage. Many scanners check only installed operating system packages. Others may also scan installed runtime libraries for some programming languages. Some may also provide additional binary fingerprinting or other testing of file contents.
When choosing a scanner for your CI pipeline, make sure it provides the coverage you need and supports the base image’s package installer database and the programming languages your applications use. You will also need to determine acceptable levels of risk for allowing a build to pass such as any vulnerability below a certain severity - or alternatively, failing builds with fixable vulnerabilities above a certain severity. Make sure your scanner offers a compatible API or tool that you can plug into your CI pipeline and provides the data you need to evaluate your criteria for failing a build.
Store
Choosing a Registry
Once you have built your secure container image, you need to store it somewhere. Using a private, internal registry affords the greatest potential for security and configuration, but it requires managing the registry’s infrastructure and access controls carefully. Most cloud providers also offer managed registry services which use the cloud’s access management service.
Registries offered as a service exist and support private repositories, and they do ease a good deal of the administrative overhead. Security engineers and build engineers will need to choose the best solution for their organization based on their security requirements and infrastructure resources.
Image Controls
Some registries support additional controls concerning image identity.
Registries which support using immutable tags on images, preventing the same tag from being reused on multiple versions of a repository’s image, enforce deterministic image runtimes. Many audited certifications and site reliability teams require the ability to know exactly which version of a given image, and therefore of an application, is deployed at a given time, a situation which is impossible when every image pull uses the latest
tag.
Image signing support provides even greater protections. With image signing, the registry generates a checksum of a tagged image’s contents and then uses a private cryptographic key to create an encrypted signature with the image metadata. Clients could still pull and run the image without verifying the signature, but runtimes in secure environments should support image verification requirements. Image verification uses the public key counterpart of the signing key to decrypt the contents of the image signature, which can then be compared to the pulled image to ensure the image’s contents have not been modified.
Run
Runtime Image Scanning
While you should scan your images as part of your standard CI process, build-time scanning does not make runtime scanning unnecessary. On the contrary, runtime scanning is more important, both for any third-party image you may use and for your own images, which may contain newly discovered security vulnerabilities. You can use custom or third-party admission controllers in Kubernetes clusters to prevent the scheduling of insecure container images. While some scanners support storing scan output in a database or cache, users will have to weigh their tolerance for outdated information against the latency introduced by performing a real-time scan for each image pull.
See the Image Scanning section in the Build section for guidelines on choosing and using an image scanner.
Registry and Image Trust
In the Store section, we discussed criteria for choosing a registry and some additional security features supported by some registries. While configuring and using a secure registry is important, those protections are diminished or lost when they are not enforced on the client side.
Kubernetes does not offer native support for using secure image pull options. You will need to deploy a Kubernetes admission controller that can verify that pods use trusted registries. For signed image support, the controller would need to be able to verify the image’s signature.
Maintain
Vulnerability Management
Scanning an image throughout its lifecycle is crucial, as is the need for weighing your organization’s risk tolerance against maintaining velocity. Your organization will need to generate its own policies and procedures for handling image security and vulnerability management.
Start by defining your criteria for what constitutes an unsafe image, using metrics such as:
- vulnerability severity
- number of vulnerabilities
- whether those vulnerabilities have patches or fixes available
- whether the vulnerabilities impact misconfigured deployments
Next you will need to define service level objectives and procedures for handling these images. Do you want to set a deadline for building replacement images and deploying those to production? Should the deadlines vary by vulnerability severity? Do you want to block the scheduling of containers from existing images when a new vulnerability is discovered? You will also need to define procedures to handle containers with these vulnerable images that are already running in production.
Use these tips and best practices to craft a robust approach to image scanning, across the life cycle of containers, as part of your larger approach to securing your Kubernetes and container environments.
Sobre o autor
Navegue por canal
Automação
Últimas novidades em automação de TI para empresas de tecnologia, equipes e ambientes
Inteligência artificial
Descubra as atualizações nas plataformas que proporcionam aos clientes executar suas cargas de trabalho de IA em qualquer ambiente
Nuvem híbrida aberta
Veja como construímos um futuro mais flexível com a nuvem híbrida
Segurança
Veja as últimas novidades sobre como reduzimos riscos em ambientes e tecnologias
Edge computing
Saiba quais são as atualizações nas plataformas que simplificam as operações na borda
Infraestrutura
Saiba o que há de mais recente na plataforma Linux empresarial líder mundial
Aplicações
Conheça nossas soluções desenvolvidas para ajudar você a superar os desafios mais complexos de aplicações
Programas originais
Veja as histórias divertidas de criadores e líderes em tecnologia empresarial
Produtos
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Red Hat Cloud Services
- Veja todos os produtos
Ferramentas
- Treinamento e certificação
- Minha conta
- Suporte ao cliente
- Recursos para desenvolvedores
- Encontre um parceiro
- Red Hat Ecosystem Catalog
- Calculadora de valor Red Hat
- Documentação
Experimente, compre, venda
Comunicação
- Contate o setor de vendas
- Fale com o Atendimento ao Cliente
- Contate o setor de treinamento
- Redes sociais
Sobre a Red Hat
A Red Hat é a líder mundial em soluções empresariais open source como Linux, nuvem, containers e Kubernetes. Fornecemos soluções robustas que facilitam o trabalho em diversas plataformas e ambientes, do datacenter principal até a borda da rede.
Selecione um idioma
Red Hat legal and privacy links
- Sobre a Red Hat
- Oportunidades de emprego
- Eventos
- Escritórios
- Fale com a Red Hat
- Blog da Red Hat
- Diversidade, equidade e inclusão
- Cool Stuff Store
- Red Hat Summit