In your session, you focus on Kubernetes security. On a high level, what are the main points to consider?:

Michael Hausenblas: So, I’m happy to be able to share with you that security in Kubernetes is not an afterthought. In fact, it’s front and center, from connectivity to access control to policies. According to research that The New Stack carried out it seems that security takes the number one place in terms of production-relevant concerns users have.

For some of the areas you can arguably more or less opt in, for example under which UID you launch an app, but many are enforced no matter what. On a high level, the following laundry list is a good starting point:

• Building secure and performant container images
• Running containers securely, including the choice of the container runtime
• Securely using the Kubernetes control plane (API server, etcd, Controller Manager)
• What authentication options to use
• Configuring proper access control by using RBAC
• Using secrets, network policies and other security policies correctly
• Thinking about attack vectors in your cluster
• Monitoring, alerting, and auditing

In my session I’ll take a breadth-over-depth approach, providing you with a ton of pointers for above areas to dive deeper where and when you need it. In this context it’s worth mentioning that Liz Rice of Aqua Security and myself are writing on an O’Reilly book titled Kubernetes Security which provides deeper guidance on the above topics (expect it by end of 2018).

What things should Kubernetes users consider when building container images?

Michael Hausenblas: There are many things to consider, but in a nutshell, the two most important ones are to not use the root user in the container image unless you really, really have to. Second, try to minimize the attack surface. For example, rather than having a fat image that includes the entire development tooling and libraries, try to only include the bare minimum you need to run the container. There’s a tradeoff here, though: keep it as simple as possible but not simpler. For example, if you don’t have any tooling in the image that allows you to do debugging (like curl) you might run into issues later on, in production.

Check out the Track Container Technologies at DevOpsCon 2018

Can you share some best practices around running containers with Kubernetes in prod and what should people avoid doing?

Michael Hausenblas: I suppose it starts with proper housekeeping: use the built-in functionality Kubernetes offers, such as namespaces, service accounts, quotas, resource limits, etc. to cleanly separate applications and/or tenants. Make sure you’re using RBAC everywhere, that is, have a clear strategy how your organizational structure (people and groups, for example taken from an LDAP server) is projected onto the cluster.

Ask yourself, which roles (end user, app operator, namespace admin, cluster operator) do you define? Do you provide for secure identity management within your cluster, for example via Secure Production Identity Framework for Everyone (SPIFFE), a project that recently became part of CNCF?

What are the application-level policies, for example leveraging low-level Kubernetes primitives such as network policies and security policies or maybe going even further with service meshes—Istio and Conduit come to mind? Do you know where your artefacts (container images, etc.) come from and can you control who gets to deploy which artefact where, that is, have you looked into adopting provenance-aware policy frameworks such as Grafeas? Do you plan to leverage policy-enforcing tooling like the CNCF project Open Policy Agent (OPA)?

Now, while some of the above-mentioned projects are still early days and not exactly considered production-ready, it’s worth it to at least be aware of them and keep an eye on them and consider adopting those rather than to reinvent the wheel.

Can you recommend us some tools that help DevOps to keep their containers secure?

Michael Hausenblas: While I don’t want to suggest that there are any clear winners yet I’ll highlight a few tools in the following, in no particular order, which I’ve either used myself, contributed to, or seen in use at customers:

• coreos/clair … provides static analysis of vulnerabilities in application containers
• jetstack/cert-manager/ … allows you to automatically provision and manage TLS certificates
• aquasecurity/kube-bench … checks whether Kubernetes is deployed according to security best practices
• neuvector/kubernetes-cis-benchmark …  inspired by CIS Kubernetes Benchmark, checks best-practices of Kubernetes installations
• liggitt/audit2rbac … automatically generates RBAC policies based on Kubernetes audit logs
• projectatomic/buildah … securely build OCI container images
• nicholasjackson/cnitch …  checks running processes under Docker runtime and alerts if any are found to be running as root
• kubernetes-incubator/kube-arbitrator … a policy based resource sharing system
• k8guard.github.io … an auditing system for Kubernetes

What’s the takeaway message of your talk?

Michael Hausenblas: One can use Kubernetes and apps running on it in a secure way. This requires that developers and operations folks work together, in the very sense of the DevOps mantra. Being aware of the relevant abstractions Kubernetes offers, from namespaces to security contexts is the first step. Exercising discipline, from building secure images to running the containers securely, to encrypt secrets is the second step.

Finally, offerings—both projects and products—in the container security space have been rapidly evolving and growing in the past three to four quarters. Many of the traditional vendors are jumping on the container bandwagon and a series of startups are trying their luck as well, in this space. The result is that while there’s plenty to choose from, you need to constantly keep an eye on the ball, so to speak, and evaluate new players as they come in and (ideally) participate in the relevant communities (such as SIG Auth or the Policy Working Group) to understand where the whole thing is heading.