Keynote: Learn by Hacking: How to Run a 2,500 Node Kubernetes CTF - Andrew Martin & Andrés Vega

[0:00] Hello and welcome. To our talk, Learn by Hacking. We're infinitely grateful that our arc saw the cloud native enlightenment, and we extend a heartfelt thank you. To you, the attendees, volunteers, contributors and the Cloud Native Computing Foundation involved in the vision and delivery of Kubernetes. Documentation and bug fixes do not write themselves, and the incredible selfless contributions that drive open source communities have never been more freely given or gracefully received.

[0:36] security controls are generally more difficult. to get right in complex orchestrations with the functionality that Kubernetes is known for. I'm sorry. To these security teams especially, We thank you for your hard work. The tax security CTFs are a reflection of the pioneering voyage of the good ship of Kubernetes. out in the choppy and dangerous free seas of the Internet. We have a free book available to download written by myself and my veritable co-author, Mr. Michael Hassenblass. There may be useful tips and tricks for individuals looking to assail the cloud-native infrastructure. If you're intrigued enough by this talk, to join us on Thursday.

[1:21] and play our new scenarios. So we're going to talk about how we run CTFs at Cloud Native Security Con over the past years. what we did, what we hope to achieve for you, the attendees, and how we maintained our fragile sanity. while provisioning thousands of notes on demand. per day. I Over the years, we have run CTFs in person, virtually. Planning for these events began in the ethereal and misty nights of 2019.

[1:55] As we looked to determine what it would take to put together a CTF running on Kubernetes for Kubernetes. without having the entire infrastructure come crumbling down around us. Nothing is more terrifying than building a self-hosted security game and inviting all and sundry to have a blast at it. Our goals are to communicate how best to secure cloud native infrastructure, They are red team scenarios with full demonstrations, teardowns and remediations.

[2:32] The best form, as Sun Tzu has told us, of defence is attack. Know thyself. And we look to give something back to the community that has so generously shared so much with us. here stands the nefarious #PirateCaptain leading the charge with his eponymous bag of Bitcoin. and the Untitled Goose that sparked so much security enthusiasm across the last few years.

[3:04] This is a learning experience for all skill levels, starting with an introductory scenario at each event, building up in difficulty, progressing as a game that gets more and more difficult throughout the journey to captivate and intrigue all skill levels and help nurture and seat the interest in and around security. With dedicated infrastructure, we look to emulate as much of the real-world systems at some cost of complexity. that we were going to delve slightly further into the presentation. The goal is not to find the ultimate security ninja but there rising tide that will lift all boats, and that players have fast feedback and support to enhance their learning experience.

[3:45] But first, here is a digital reconstruction by the street artist Banksy. of what a bunch of pirates would look like playing a cloud-native CTF. with a splash of stable diffusion for effect as well. So there is no such thing as secure software. As fast as we can possibly paint over the colourful selection of vulnerabilities in our infrastructure, new features are shipped. Software moves quickly.

[4:18] - and those new features potentially feature untested code paths. I'm not sure. And any of those untested code paths with security side effects are by default vulnerabilities. We prioritize shipping features over fixing vulnerabilities at our peril, of course. And this is the reality of modern software that we balance in our day-to-day every day. Resemblance to the friction we see between red teams and blue teams Any resemblance in that last animation is not a coincidence.

[4:57] Organizations not only evolve for competitive advantage, they must do so in order to survive because their adversaries are also evolving. So how do we represent that complex puzzle in a safe learning environment? As scenario authors, we look to construct the most realistic scenarios that we can manage. That includes building infrastructure diagrams and thread modeling scenarios. It helps us develop with the greatest degree of realism and is born from our experience consulting with regulated industries and startups alike.

[5:28] Here we can see a supply chain attack in progress. Captain Hashjack has attacked systems through various network systems in the past. including packing repositories, container registries, and the notable hypertext coffee pod control protocol. with no stone left unhindered, Running the platform on real infrastructure gives us the opportunity to emulate simulations we see in real life. Coffee pots and IoT attacks are not the exception.

[6:02] And we are looking to educate as many people as possible as effectively as we can, here we see a pod Security treasure map from the book, "Hacking Kubernetes," These scenarios are intentionally increasingly complex and difficult. This helps to crystallize players' understanding of a wide and complex security landscape and, as with all CTFs, Attendees and players must enumerate the visible horizon, escalate their privilege, reverse back from dead ends, and intentional misdirections.

[6:40] Bye. This pod treasure map details many of the routes that an attacker might explore within a Kubernetes cluster. But with each scenario, there is only one intentional route for a player to find, with the notes and caveats that players have found, multiple bypasses, unintended shortcuts and wily workarounds as noble routes to completion. So how is the CTF played? With a terminal, IP address, and optional authentication credentials for an SSH session, a Kubernetes cluster, or other mystery piece of infrastructure.

[7:20] Here we see the first view of a cluster that a player has when placed in a root-enabled container running ncluster. They must enumerate the visible, uncover the invisible, and start to explore potential routes of compromise in the cluster. This may include hijacking sessions and credentials, attacking APIs and data stores routable from the starting point, and escalating privilege through any and all identity mechanisms, a cloud-native system may have. My personal preference is to bring a portable set of bash scripts to speed up enumeration and exploitation.

[7:57] However, there are no rules what tooling you can bring with you. You just need to find somewhere to execute it from. Over the years, we have looked at a whole host of vulnerabilities and misconfigurations, from container image file system foobars and runaway runtimes to secure config, malicious mounts, unsequestered secrets, privileged pod security policies, sensitive service accounts, no firewall networking, awful admission control, exposed etcds, terrible TLS implementations, federation failures, chronic control plane control, and so on.

[8:34] configurations and unlimited user exposures. And there's more from Ingress to the kernel. We played this remotely. We're playing it in person this time around. adapting to the pandemic and running from virtual events, online gatherings, running the infrastructure over Slack, Peace was never an option. So how did we go about this high pressure live deployment of 2500 nodes across each day we play the game? We start with the open source breach simulator tool built to help control planes clients secure their clouds, One five-node cluster is provisioned for each scenario.

[9:18] I And cluster reuse is not enabled. We add a double dose of super massive provisioning engines backed by Knative, scaling to zero. These are not centralized in a unified engine. This sits on the back of a Pub/Sub queue and reentrant state machine, which tries to ensure that even half provisioned clusters are forced into life and reinvigorated when half deployed, which just means distributed systems, as we know, are hard.

[9:51] Then, the unique credentials are distributed to each player for each scenario cluster pairing, and the games begin. Our trusty Taskmasters are on hand in person and on Slack, to hand out hints, offer suggestions and guide people through the path of the challenges. All this adds up. We spin up the clusters as needed and we hit the lofty heights of two and a half thousand nodes spun up per day with 80 to 100 participants.

[10:23] We hope to beat this record. on Thursday, What has gone wrong? Would you really believe us if we said nothing? There was that. one time And AWS internal-- Ubuntu Mirror went down. As you can imagine, nothing completed provisioning. simulator was just Host, dead in the water. Health checks are decidedly unhealthy. to showcase that. Typical distributed system problems when EC2 wouldn't provision VMs, a very red dashboard, a rising sense of panic, came from it that required us reprovisioning into a different region post-haste.

[11:05] I'M NOT SURE. That time somebody figured out how to bypass the initial SH tunnel sandboxing, which was also interesting. securely provisioning and secure infrastructure is hard. We don't want it too insecure. When the Cluster Admin RBACs the RBAC credentials disappeared, gone. found their way into a bastion node when the test suite only cover the happy paths also. And through it all, we seem to have hit our goal We're thankful to everybody who has played the game, given us feedback and given us the opportunity to iterate on this wonderful, thoroughly enjoyable platform that we love to play.

[11:50] CTFs can often be big. Can't feel daunting? So we hope that you focus on the learning experience. Walk through the scenarios. and If you feel like smashing machines, Channel that, please. And our clusters and scenarios today are of course, open source running on this cloud native breach simulation. Underneath the Kubernetes simulator project, there is a hosted version and Control Plane are running a private beta if you would like to be involved.

[12:28] As you can see, Nothing is perfect, there's an out of bounds exception on the end of the slide, if you would like to pop some shells with us please come and play tomorrow. Thank you.