1.SESSIONID:ASD-W10 Practical Approaches to Cloud Native Security Karthik Gaekwad Principal Engineer Oracle Inc @iteration1 #RSAC
2.#RSAC Hello • I’m Karthik Gaekwad • NOT a DBA •https://cloudnative.oracle.com/• Cloud Native Evangelist at Oracle Cloud •Past:Developer on the Oracle Managed Kubernetes Team @iteration1
3.Hello Been in Industry 15 years. In general, I like building stuff with friends. – Maintainer for Gauntlt- Open source security scanner. Love Teaching and building community. – Run Devopsdays Austin, Container Days, Cloud Austin. – Chair All Day Devops Cloud Native track. – LinkedIn Learning Author for Learning Kubernetes (and more). #RSAC
4.#RSAC I mustache you a question…
5.The Cloud Native Journey #RSAC Core to Edge Docker Speed Efficiency Agility Kubernetes Phase I Developer Focus Container Adoption Focus Applications Automation Community Developer adoption Dev/Test apps Simple orchestration Individual developers Phase II DevOps Focus Application Deployment DevOps deployment Production apps Advanced orchestration Teams & lines of business 5 Phase III Business Focus (end-to-end) Intelligent Operations End-to-end integration Digital business apps Serverless, DevSecOps, & ML Cloud native enterprises
6.#RSAC CNCFSurvey:August 2018 How Does Your Company Use Containers and Where? Lots of adoption on dev/staging Continued production increase
7.#RSAC CNCFSurvey:August 2018 How Does Your Company Use Containers and Where? Adoption over public and on-prem
8.Kubernetes Dominates Container Management Your company/organization manages containerswith:Kubernetes #RSAC
9.#RSAC Good News, Bad News… Many Projects… Good usage in dev/prod But...
10.Top 5 challenges to cloud native adoption… Monitoring Security Lack of Training Cultural Challenges Complexity 0 5 10 15 Percentages 20 25 30 35 40 45 #RSAC
11.#RSAC Kubernetes & Cloud Native Challenges Managing, maintaining, upgrading Kubernetes Control Plane – API Server, etcd, scheduler etc…. Managing, maintaining, upgrading Kubernetes Data Plane – In place upgrades, deploy parallel cluster etc…. Figuring out container networking & storage – Overlays, persistent storage etc… - it should just work Managing Teams – How do I manage & control team access to my clusters? Security, security, securitySource:Oracle Customer Survey 2018 11
12.How Are Teams Addressing Complexity, Training Issues? Customer Managed Fully-Managed App Management App Management App Deployment Scaling App Deployment Scaling YOU High Availability High Availability Platform Backup & Recovery Platform Backup & Recovery Upgrades & Patching Upgrades & Patching Software Installation Server Provisioning #RSAC Software Installation Server Provisioning Rack and Stack Rack and Stack Power, HVAC Power, HVAC Benefits Faster Time to Deploy Lower Risk Accelerate Innovation
13.#RSAC Which brings us to security…
14.#RSAC Where no news, is good news!
15.#RSAC Unsecured K8s dashboards Unsecured Kubernetes Dashboard with account creds. Used this to mine cryptocurrency. 2017: Aviva 2018: Tesla, Weight Watchershttps://redlock.io/blog/cryptojacking-tesla
16.#RSAC Kubelet credentials hackShopify:Server Side request Forgery Get kubelet certs/private key Root access to any container in part of infrastructure.https://hackerone.com/reports/341876
17.#RSAC
18.#RSAC
19.#RSAC
20.#RSAC
21.#RSAC
22.#RSAC How did we get here?
23.#RSAC “Kubernetes is too complicated”
24.#RSAC “Kubernetes is too complicated” “We hope it’ll get easier”
25.#RSAC What is your strate
26.#RSAC Let’s lookat:Attack Surface – More importantly, how to limit damage Security related features in K8s – The more you know, the better you build Opensource Tooling to help – Because we all need help
27.#RSAC Attack Surface
28.Attack SurfaceGoal:Reduce the attack surface Analysisfor:–Host(s) –Container (Images and running) –Kubernetes Cluster #RSAC
29.AttackSurface:Host These are the machines you’re running Kubernetes on. Age old principles of Linux stillapply:– Enable SELinux – AppArmor – Seccomp – Hardened ImagesGoal:Minimize privilege to applications running on the host Goodnews:Already a wealth of information on this subject! –http://lmgtfy.com/?q=how+to+reduce+attack+surface+linux#RSAC
30.AttackSurface:Container ImagesGOAL:Know your base image when building containers #RSAC
31.AttackSurface:Container ImagesGOAL:Know your base image when building containers **BTW, this is just a ruby helloworld app #RSAC
32.AttackSurface:Container ImagesGOAL:Know your base image when building containers **BTW, this is just a ruby helloworld app #RSAC
33.AttackSurface:Container ImagesGOAL:Know your base image when building containers Fulldisclosure:I’m karthequian; I created this as a ruby 101 container for learning purposes only #RSAC
34.AttackSurface:Container ImagesGOAL:Know your base image when building containers When in doubt, stick to an official images! Or start from a sane base image (example:alpine linux) #RSAC
35.AttackSurface:Container ImagesGOAL:Smaller the image, the better Less things for an attacker to exploit. Quicker to push, quicker to pull. #RSAC
36.AttackSurface:Container ImagesGOAL:Don’t rely on :latest tag :latest image yesterday might not be :latest image tomorrow Instead, you’d want to know what specific version you’re operating with. Sidebenefit:If there is a new vulnerability announced for OS version x.y.z, you know immediately whether you’re running that version! #RSAC
37.AttackSurface:Container Images #RSACGOAL:Check for vulnerabilities periodically Plenty of ways to do this in registries. We’ll cover more in the tooling section
38.AttackSurface:Running ContainersGOAL:Don’t run as root Containers running as root might be completely unnecessary for the actual application. If compromised, attacker can do a lot more things.. Pod security policies can help (we’ll see how later). #RSAC
39.AttackSurface:Running Containers #RSACGOAL:Limit host mounts Be wary of images that require broad access to paths on the host Limit your host mount to a smaller subset of directories Reduces blast radius on compromise
40.#RSAC AttackSurface:Kubernetes Cluster
41.Kubernetes Cluster- TLS TLS ALL THE THINGS #RSAC
42.Kubernetes Cluster- TLS TLSChecklist:1. Nodes and Master 2. User and Master 3. Everything etcd 4. Kubelet to API Server #RSAC
43.#RSAC Kubernetes Cluster- TLS
44.Kubernetes Cluster- TLS TLSChecklist:1. User and Master 2. Nodes and Master 3. Everything etcd 4. Kubelet to API Server #RSAC
45.CVE’s Happen… #RSAC
46.CVE’s Happen… #RSAC
47.#RSAC CVE’sGOAL:Have an upgrade strategy Because…CVE’s are fixed in new minor versions. Don’t treat K8s as “install once, run all the time”. Make your K8s install repeatable for different versions. ..Or use a Managed Provider. – Either automatically patch for you, or tell you what to do. 47
48.We’re a little better off now. #RSAC But what else to do?
49.#RSAC K8s Features How can the platform help me make secure choices?
50.#RSAC K8s Features Authentication Authorization Audit Logging Network Policies Pod security policies Kubernetes Secrets
51.Authentication and Authorization Do you know how you are authenticating with Kubernetes? Many ways to Authenticate – Client Certs – Static token file – Service Account tokens – OpenID – Webhook Mode – And more (https://kubernetes.io/docs/reference/access-authn-authz/authentication/)#RSAC
52.#RSACGoal:Pick a strategy that fits your use case Whatever you do, DO NOT YOLO!
53.#RSAC If you DO NOT YOLO… You can pick an authz strategy..
54.Authentication and Authorizationhttps://kubernetes.io/docs/reference/access-authn-authz/authorization/ #RSAC
55.Authentication and Authorization Protip:Nobody uses ABAC anymore. Don’t be that guy…. RBAC is the defacto standard – Based on roles and role bindings – Good set ofdefaults:https://github.com/uruddarraju/kubernetes-rbac-policiesCan use multiple authorizers together, but can get confusing. – 1st authorizer to authorize passes authz #RSAC
56.Kubernetes Cluster- Audit Logs #RSAC Wat? “Kubernetes auditing provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators or other components of the system.”Answers:What/when/who/where information on security events. Yourjob:Periodically watch Kubernetes Audit logshttps://kubernetes.io/docs/tasks/debug-application-cluster/audit/
57.#RSAC
58.Kubernetes Cluster- Network Policies Consider adding a network policy to the cluster… DefaultPolicy:All pods can talk to all other pods. Consider limiting this with a Network Policyhttps://kubernetes.io/docs/concepts/services-networking/network-policies/#RSAC
59.Kubernetes Cluster- Pod Security Policies Consider adding Pod Security policiesPodSecurityPolicy:A Defined set of conditions a pod must run with. Think of this as authorization for pods. #RSAC
60.KubernetesCluster:Pod Security Policies Capability for an admin to control specific actionshttps://kubernetes.io/docs/concepts/policy/pod-security-policy/#what-is-a-pod-security-policy#RSAC
61.Kubernetes Secrets #RSACGOAL:Use Kubernetes secrets to store sensitive data instead of config maps. Also lookat:secrets encryption provider. – Controls how etcd encrypts API data – --experimental-encryption-provider-confighttps://kubernetes.io/docs/tasks/administer-cluster/encryptdata/
62.Tooling Opensource Tooling #RSAC
63.Keep tabs on the CNCF Security landscape #RSAChttps://landscape.cncf.io/landscape=security-complia
64.#RSAC CNCF Projects “The Update Framework” Is a project. Is a framework or a methodology. Based on TUF. Used for secure software updates. Based on ideas surrounding trust and integrity. A solution to secure software updates and distribution. Used in Docker Trusted Registry.
65.Clair Open source project for the static analysis of vulnerabilities in containers. Find vulnerable images in your repo. Built into quay.io, but you can add to your own repo.https://github.com/coreos/clair#RSAC
66.#RSAC
67.#RSAC Harbor Newer! CNCF Project Registry product Supports vulnerability scanning, image signing and identity control Scope is larger than clair
68.Harbor #RSAC
69.Kube-bench Checks whether a Kubernetes cluster is deployed according to security best practices. Run this after creating your K8s cluster.https://github.com/aquasecurity/kube-benchDefined by the CIS BenchmarksDocs:https://www.cisecurity.org/cis-benchmarks/Run it against your Kubernetes Master, or Kubernetes node. #RSAC
70.Kube-bench example #RSAC
71.Kubesec Helps you quantify risk for Kubernetes resources. Run against your K8s applications (deployments/pods/daemonsets etc)https://kubesec.io/from controlplane Can be used standalone, or as a kubectl plugin (https://github.com/stefanprodan/kubectl-kubesec)#RSAC
72.Kubesec example #RSAC
73.Kubeaudit Opensourced from Shopify. Auditing your applications in your K8s cluster.https://github.com/Shopify/kubeauditLittle more targeted than Kubesec. #RSAC
74.#RSAC
75.Kubeaudit example #RSAC
76.Couple more resources to lookat:11 ways not to gethacked:https://kubernetes.io/blog/2018/07/18/11-ways-not-to-gethackedK8s security (from Image Hygiene to Network Policy):https://speakerdeck.com/mhausenblas/kubernetes-securityfrom-image-hygiene-to-network-policies#RSAC
77.Apply It! 77
78.#RSAC Apply It! Day 1: Know what version of Docker and Kubernetes you use. Understand if your control and data plane nodes are hardened. Understand how your Docker containers are built. Find out how you authenticate and authorize for your clusters. 78
79.#RSAC Apply It! Week 1: Build an AutomationPipeline:– To build Docker images on code pushes – Versioning strategy for code – To build your Kubernetes clusters 79
80.#RSAC Apply It! 1st Month Sanitize yourcode:– Know your base images – Implement versioning for your containers – Invest in a registry (or tooling) that does vulnerability scanningKubernetes:– Have an upgrade strategy in place – Analyze secrets/sensitive cluster data – Turn on audit logging 80
81.#RSAC Apply It! 3Months:Continuously Monitor – Tooling like Kubesec/Kube-audit Plan how to address vulnerabilities/CVE’sK8s:– Strategy for Pod Security Policies – Strategy for Network Policies – Run scans (like kube-bench) on cluster creation 81
82.#RSAC Apply It! 6Months:Re-ask day 1 questions. Review strategies- is it working? What needs tweaking? Review tooling- are there new tools that help? Review CVE’s 82
83.KEEP CALM AND KUBE ON @iteration1 #RSAC