Kubernetes Security: Detecting Lateral Movement and Defending Against Attackers

Hello and welcome to our Kubernetes Security webinar, sponsored by Tigera. My name’s John Armstrong, head of Product Marketing at Tigera. And today’s topic of discussion is about detecting lateral movement and defending against attackers. As Kubernetes matures, gone are the days when we can fully compromise a cluster by just taking over a pod and sending commands to the Kubernetes API service. Hardback and other Kubernetes security features force attackers to pivot at least once to find the right vulnerable pod or service account with the right privileges to take over a cluster. The attack surface grows as the cluster gets bigger, and more third-party applications are deployed. By understanding the attacker’s workflow and gaining visibility into the relevant connections, we’re able to identify our clusters’ weak points, and limit the attacker’s reach. It’s my pleasure to introduce Garwood Pang, who is leading today’s session. Garwood is a security researcher with Tigera. Garwood’s cyber security experience includes three years at GE Digital, as well as a vulnerability researcher for industrial control systems, as well as three years at Fortinet writing signatures for intrusion detection and intrusion prevention systems. Garwood’s presentation will be followed by a Q&A session. You can submit your questions to us online at any time during the webinar using the BrightTALK questions interface on your screen, and we will address all of the questions at the end of the presentation. And now I’ll pass things over to Garwood. Thank you, John. Welcome to my webinar. So for today’s agenda, we’re going to examine some common techniques that an attacker will use to gain intels of your cluster, and we’ll discuss the ways that we can limit what they do. I will show a demo of an attacker escalating their privilege using a vulnerable Kubernetes Custom Resource Definition API call. And lastly, I’ll visualize some reconnaissance technique using Calico Enterprise. Let’s first talk about the Reconnaissance Techniques. We’re talking about when the attacker has already compromised a pod. This might be using some sort of vulnerabilities that they found on the user’s website and stuff like that. We’re going to assume that they now have shell access in the pod. In the attacker’s view, they know that they’re in some sort of node or in some sort of pod. They know that there’s probably going to be more nodes that the cluster has, and it probably knows that there’s some sort of master, but it doesn’t know where it is or how to reach those. The most common areas that the attacker would use to look for more information about the cluster is via the cluster provider’s API Metadata Service. Next is the Pod Networking and Filesystem, so this is talking about the image that is actually running the pod. And lastly, the Kubernetes API Service. What’s the Cloud Provider Metadata Service? That’s used by the DevOps or SREs in your company that is using this service to manage your cloud. Let’s say you’re on GCP or AWS. These are all reachable by this specific IP address: 169.254.169.254, or the metadata.google.internal. Accessing this endpoint will then go to the cloud provider’s metadata service. From that, it will provide you with host name, IP, IAM role credentials and other useful data, so that you can help [inaudible 00:04:24]. In other view, what does that mean, right? You can see that there is your hostname, which gives you a good amount of information, because a lot of times users or DevOps will like to have their owner’s name, what the node is, or which node it is on. It will also tell you the project name. This is all good information for the attacker to then use to deduce what’s in the project or the cluster. Next, you will get to see the network. You can get the external IP and internal IP of the node that you’re in, as well as the DNS server. This will all be useful later on when you want to probe the other nodes that are in the system. There are times where the DevOps or SREs will add some custom attributes. This is just so that they have maybe such a configuration they want, and the attacker will see those and use it to their advantage. Lastly, is the IAM Role Credentials. This is basically a service account that was used to create the instance or node, and sometimes if you need to provide your node, let’s say S3 access or bucket access. Then you want to add a road so they can read those, but also in this case if the attacker was able to reach the metadata service they’re able to then also look at all your access buckets because they also have the credentials. So now that we have this kind of information or the attacker has this information they can now update their nodes. Now they know that in the compromised pod’s node, they know the hostname [inaudible 00:06:24]. It now has a rough sense of what the other nodes are. In those stages it is in the same subnet and same VPC and will have the same project name. That’s going to be the same for the master node also. So how do we defend against this? You just need to review your service accounts permissions when those used to create the instance. And you also want to have a network policy to block access to this special IP address on your pods. There’s really currently no reasons why you need it in your pods, they can just block it or monitor it. There’s a google link that you can look into or you can just google it. I will show you a quick picture of what it looks like. So you need to go to your DCP VM instance page and you want to look for the service account name. [inaudible 00:07:24] your IAM page and then just need to look at the permissions and remove anything that you don’t need. With that the attacker will then not be able to use the IAM role credentials to do more reconnaissance but now you also want to have a network policy to block it from being reached. So what we do in Calico Enterprise or Calico is to create a global network set. We want to label the IP address 169.254.169.254 as the color red and then we want to create a global network policy that will then block. We’ll set it to block egress to that colors or the label. So next the attacker will want to look at the pod networking and Filesystem. So here what the attacker is trying to do, he’s trying to get as much permission in that pod. The more the better in their view and try to find all the interesting secrets or data that they can retrieve in that pod. And once they’re done, they will then start looking around nearby pods or any other targets that they can reach. And then they’re going to do it over again when they get in. This is going to happen until they find what they need or they found enough that they can then either escalate into accessing the node or just having [inaudible 00:08:59] access to the cluster. So what they do is they go from the bottom of this list up. They first want to look around for the networking and see what’s nearby. They want to then go to look at the Filesystem and see what the permissions are and what they can read. I don’t really want to have a root user in this pod, if they have the root then they can always just add it without permissions. Next they want to look to see if there’s any special capabilities host granted, so we’re talking about the host network, host IPC or host ID. This will give them information on what the nodes running also. And lastly, if they can they want the part to be the privileged part because that even will give you a root access and it’s very trivial then to escape into the node that was running the compromised pod. So what are they looking for? They’re looking for secret data such as secrets, host-mount files, certificates and sockets. What these are is basically secrets that maybe the application has some sort of passwords. Host name files like the CLD configurations, certificates like [inaudible 00:10:21] CA he can then steal, then generally own certificates to impersonate the permission. Or sockets, like docker sockets where you can then start talking directly into the docker sockets and maybe create new images or still images. The attacker will then also want to look at the IP address or any service names that they can find, so then they can reach the other services and find the next targets. So we’re going to go over some commands. These are all very, very basic Linux commands that everyone should already know but in the sense of Kubernetes they want to give you more information than you expect. So first were going to look the env which is the environment service command. You can do it two ways in the all cat/proc PID and then your environ file. This interesting because then now you can see here Kubernetes will always add whatever [inaudible 00:11:25] you’re going to add stuff that is specific to Kubernetes. So for example, the host name and you will always have the Kubernetes API IP address. Being able to reach it might be a different story if there are network policies but it does tell you where it is. So you can maybe try to probe into that. Any other services that’s running on the same namespace will also be here. It also depends on if there’s network policy that’s blocking it but something that the attacker will then write down and maybe target later. So next is capsh. That’s showing you all the capabilities of the pod. If you look at it on those top and bottom, the bottom one is the privilege. You can see that you get a lot more capabilities when you have privileged true. The main one is capsh env but that gives you root access. And we’re talking about the real root access, not just fake root access. So this makes it very trivial for you to then jump to the node which is running your compromised pod. So next is ps aux, very simple or is/proc. This tells you what’s currently running on your pod but if you have the host PID granted on this pod for some reason, then you actually just see all your running pods or other running processes in the node. So then if you have more privilege or capabilities, what you can do is you can attach yourself into one of these other pods or inject stuff into other pods and then you can get access that way. So now mount is actually pretty interesting in the Kubernetes process. What happens is that you would actually see, as my arrow shows, other stuff that was mounted by Kubernetes. So anything that you did, for example like config maps, or secrets, or any volume mount, or host mounts, you will actually see it here. And this makes your space a lot easier for the attacker to then just use mount and see what’s specially mounted and then go and look at those instead of having them maybe going from directory to look for anything interesting. And this already tells them what they need to look at and if they can reach it. Next is the cat/etc/resolv. That’s very simple. It’s just telling you where the BMS server is. With the BMS server, I’ll know where it is, the attacker then can then start probing and do a lookup to figure out the service names and they can get a IP address or service IP address from that they can figure out where to go next or what [inaudible 00:14:39]. So next is the /var/run/. This is where usually all the sockets or the service accounts secrets are located. As I said before if you, for some reason, have a docker socket in your compromised pod then they can talk to the docker engine where then they can pull all the images out or spin out more images. And that’s going to be really bad for you. The attacker will also want to look at the service accounts secrets. That’s going to be useful when we want to target the Kubernetes API. So next is IP address, that’s also very simple but it does tell you what the compromised pods IP address is. With that then they can then [inaudible 00:15:25]. If, for example, it’s for some reason the host network was granted to this compromised pod, they would then be able to see all the interfaces that is on the node. And with that they can then start directly interact with these interfaces and probe the network. So with that what do they have? The attacker now has the pod IP address and the subnet that it’s in. It will also have the host name of what the compromised pod is to know what the pod is about. The hacker then also now knows what the application that’s running on this compromised pod and if there is any other vulnerable application that they can then use also. The attacker does not know what the capabilities that this pod has and if there’s anything wrong with the pod, they can then always pivot. And lastly all the secrets that is on the container is now probably being read by the attacker so then now they have that too. So with that we can update our nodes or our picture of what they have. Now that they look at their compromised pods they have now the stuff that I just talked about and now they may want to look at what’s nearby. So the main stuff that they are trying to access or trying to get is the service names. Basically what happens is you have your DNS so when you create a new service a new DNS entry is created so then you can access it with that domain name instead of having the service IP address. A lot of time what this is, is the servicename.servicenamespace. So it might be not [inaudible 00:17:25] and probe and maybe try to figure out what it is. And if they get it right they will get the service IP. With the service IP then they can access the exposed application that was intended to expose and they can start probing on that way. [inaudible 00:17:45] or find what the [inaudible 00:17:49]. [inaudible 00:17:55]. Just wanted to let our live audience know that we’re having a little problem with Garwood’s phone connection. All these events are managed remotely and run remotely so Garwood’s going to be back on the line shortly. I think he’s probably going to dial in again and see if he can get a better connection for the webinar presentation so, please hold on for us and we’ll just pause for a moment. Thank you. Hi. Can you hear me now? Can you hear me? Can hear you fine, Garwood. You sound great. Thank you. Is this better? Sorry about that. I guess I’ll continue on then. So I will talk about this slide again then. So in enumeration we’re just going to use the nmap-sn on the pod IP address or the pod subnet. If you run this command on the pod IP address or pod subnet, what this does is then it would do a ping scan on the host subnet of the node and then you will know of the compromised pod. While this is happening you also get the DNS lookup and what that will give you is the service names, as you can see on the right. If you do it on the service IP address it will also take a long time but you will also get your service names that way too. The issue for you running nmap is that since you’re actually paying a lot [inaudible 00:21:04]. You will generate a lot of noise and what you can do is you can just use the network policy to stop this from happening. Next is the port scan. It’s very similar to enumeration. You just do nmap-Pn and then the IP address. I’ll recommend you to actually run it on the IP address and not the subnet because it will take a long time. So what this does is it will find all the exposed ports and services that’s running on the pod or the service. And with that you can then stop probing into it. So interesting about this is that even if you have a pod that is running with the apache for example, if your user did not expose, do it again. Create a service port for this application. You can actually still reach it using the pod IP address even though there’s no service IP address just because you didn’t create a service. So we can also see, right? What they also means is that if you have a service and you have a pod IP address you can actually reach the same service using the pod IP and service IP. Typing will generate a lot of noise because if you’re doing a pod scan and lastly network policy won’t always build a block. So with that in mind, now the picture of what the attacker can see is quite a bit now. The attacker now knows what a nearby pod and what kind of service they’re running on for the target. And for pods that are running on the same nodes, you will even get a pod IP address and more [inaudible 00:22:52] running. So how do we defend against this? So what you want to do is have a minimal base image. This reduces the amount of binary application that the attacker can use. So we’re talking about the IP address on mount and stuff like that. If you use the minimal image then you will require the attacker to basically either download the image or somehow choose that they want actually choose what they want. They have to then have to get it on the [inaudible 00:23:24]. And then what we want to do is then we want to get the whole image to be root-only. With that then the attacker can’t even create files so then he can’t even bring in the binary’s or applications. We also want to set the uses for the pods to be non-root. What this does is then it prevents the attacker from being able to change the pod permissions of the image. So then they can’t change from read only to rewrite. Next, we also want to drop and remove the shell of the image. There’s really no point nowadays that you need a shell to run your applications. What you can do now is have the Kubernetes run to your application directly so it doesn’t need shell. And this makes it harder for the attacker to then somehow inject a shell so then they have shared access. We also want to drop the capabilities of your pods if you don’t need them using the pod security policy. And lastly, you just want to block the egress or internet access using network policy if, for some reason, you need internet. So the last thing we will talk about is the Kubernetes resources or the Kubernetes API. This can be done internally, externally depending on how the cluster was set up. And this you will have figured out the external IP address and stuff like that maybe for some reason the DevOps person exposed the Kubernetes API. This allows the attacker to then probe it from their side. What the attacker thinks is truly based on what service account was applied to the compromised pod. So, if there is nothing, then the attacker won’t be able to do anything. So if the attacker was able to compromise a pod that has access to the custom API’s or CRD’s, they are going to go a lot of stuff because CRD requires you to have Kubernetes permissions so then we know there is some sort of no style applied to this service account on the pod. So then they can look around that way. So what the attacker needs when they have the compromised pod is to make sure that they can access the Kubernetes API and they will be able to find this using the environment variable. You want to see that there is a service account token that’s mounted on the pod at /var/run/secret/kubernetes/serviceaccount. And ideally if they have internet access they will just download kubectl or if not they can always just manually curl. And as I said before if for some reason the DevOps person was exposed to Kubernetes API, they can then just scrap the service account token back to their laptop and then hurry or use kubectl on their end so then reach the exposed external kubernetes API service. Special cases happen when you are in a managed Kubernetes cluster or GKE and EKS. Since it is Google or AWSS managing the [inaudible 00:27:05] points there is an IP address so if for some reason the DevOps person has not used a popular PPC and have to hide it, this will allow the attacker to then reach it. So the main command we use is kubectl auth can-I –list. This will give you all the permissions of what the service account has. You can also check it per namespace because a lot of times what people like to do is have specific permissions or roles per namespace. So on the picture you can see that on the left is basically a default permissions where it has nothing on the service account and on the right you can see the wildcard [inaudible 00:27:52]. And this is basically telling you can do anything you want on the cluster. You don’t really want that to happen. The other defense against this, you can actually un-mount the service account token if you don’t need it. If you also don’t set a specific service account, it will not be a default service account with no permissions. If you do need these permissions then you have to review your cluster role and roles to make sure that they’re namespace specific and that there’s nothing that’s not needed, but you added in to it. We can add network policies to block external, internal requests to Kubernetes API service. So with that we have now basically completed or the attacker has now completed their [inaudible 00:28:46] or have a good idea what is running on the cluster. Also now know if they can access the community API or not. With that the attacker can then look at the different services or pods and find the next target if they need to. So I’ll now go over a demo of what I just talked about. In this demo I’m using a Kubernetes cluster on GCP [inaudible 00:29:17] in Kubeadm. Our start off within a compromised pod, what that is I’m just going to [inaudible 00:29:24] into a pod. I’ll look around and see if I can find any interesting information. I will then find a vulnerable CustomResourceDefinition and we’re going to steal its token. And then we’re going to pivot and impersonate that CRD. So what is a CRD? It one of the ways you can add custom functionalities to your Kubernetes cluster. This is what you see when you see your manifest with the word “kind”, like the kind global network policies or kind pods or deployment. What happens is then you can create these manifests and send it to Kubernetes API. And then what happens on the back end is that you will have a controller that’s running and it’s going to monitor the specific object that you’re creating. And once you see that you have a new manifest, it will then do whatever it needs to do to develop a new pod or deployment or stuff like that. So this is how all these third party Kubernetes applications are used. And how they are implemented. So I’m going to move over to my screen share. And now start showing how it works. So here is a pod that, except until, this is all initial access pod. We are running on [inaudible 00:30:51]. So we’re going to look around and see what we have in our system. So I’m just doing a basic [inaudible 00:30:59] the files or where the files are on the pod. We went to do the EMV files before we can see that there is a microservice username that the attacker will download that and maybe probe later. We also have the Kubernetes API service IP address. So then we won’t be able to talk to the Kubernetes API. And about the hostname, so at the moment this isn’t an interactive pod, there is nothing interesting for now. We also want to do several ps aux and we can see that ps aux will not actually start install in this pod so what we can just do is go prop and we can see that this only here on the last corner, only four different PIDS. This tells me that what we don’t have the host PID capabilities granted so it’s just stuff that is running locally. Next, we want to do IP address. With that you can then grab the IP address. What we can then do is download nmap and then start doing the enumeration and port scan to figure out what other pods is running or service is running. We also want to run mount and we can see here there isn’t anything interesting other than the four special files that the Kubernetes has mastered. So your host name, your resistance comp for DNS and we can also see that Kubernetes also mounted a service account. We want to look to see what this service account is about. Before we go there we can always just see where curl and we can look outside. What I did was I curl google.com. We see a lot of data that’s coming in so we can safely say we have internet access or egress access. So with that what we want to do is then look at the service account. So if our one secret [inaudible 00:33:23]. And you can see that there’s a token. So we’re going to use this token to then talk to the Kubernetes API if we can. So since we have egress access, you don’t have a script or command that you can use to download kubectl so I’m just going to do that now. So we’re going to change the permissions to that to something else so that we can run it. And we’re going to strike at pods. So when you’re in a pod and you have to kubectl watch all these or figure out that there’s a service account in the [inaudible 00:34:06] so then you don’t actually need to tell it where to find the credentials. It will actually automatically find it for you. This also works with the [inaudible 00:34:16], that’s why I can just do to kubectl the pods. We can see that the office user does not have permissions in this pod. So what that also means is you can always try to see can I, yes. And with that we can see that there is a bunch of useless, at the moment, permissions. We can see it here but a lot of times what it means you’re actually not able to access it because you don’t have the public share. So we do see this [inaudible 00:34:53] as vulnerable.com. So that’s probably the target that we want to look at. So we’re going to try this, see what it has. So we’re going to do kubectl [inaudible 00:35:04]. We see that there’s nothing. That doesn’t mean that we have access to it. So I’m going to do -ASG on the namespaces. And we can see that in the default namespaces actually a granted example that’s running. So we’re going to take a look at that. Let me run you [inaudible 00:35:29]. We can see that there is some sort of back here where this, at the time, is using some sort of token and different URL. So what this looks like is maybe this service application actually already does is monitoring this IP address on the microservice and it’s not using a token as a header. It’s more for authorization. So for us we can try to see if we have access to secrets which are highly docked. So we can’t really figure out, we can go and get microservice token. So we can’t really access the token but there’s a good chance that we can edit this batch feed. Let’s try that. And at the moment it looks like I actually have permissions to edit the batch feed. And what we want to do now is modify what we have here so that we can maybe steal that local token. So what we’re going to use is something called ngrok and what this is is a application that would then, if I run a command, so ngrok future VP and then enter the location. And what this does is going to expose my laptop, the file that I just showed you or the directory I showed you to this ngrok website. So If I access this ngrok website on anywhere it will then redirect to my laptop and we cannot be certain it made a mixture web server. So what we’re going to do is then see if we know that these token, this pod the header that this access goes in. We’re going to replace that URL with this ngrok website and see if we can steal the token that way. So I’m going to edit the URL to the ngrok to D-E dot ngrok.io. If I saved and this works we can also see on the left first, we can see that it was edited so we know we had edit permissions. And on the right we also see a gate request. So now we go to our ngrok web interface and we just now see what’s happening. So it’s 20, 40 and you can see that I replied to it. It will just cost us [inaudible 00:38:19] of the file that I created. But we look at the headers. We can see there’s actually the token header that we’re talking about. And we can see that this is the token that they were using for application. I’m going to copy it because there’s a good chance that this might be also a Kubernetes service account login. What I usually see is you already start that shows [inaudible 00:38:43] to me and restart as I survey Kubernetes service account tokens. So now that we have the token we’re going to try to run our kubectl with this [inaudible 00:38:54] and to see what we have. So we want to do kubectl back in our pod. Just have the token and then we’re just going to paste the token. And then we want to try get pods again. And here we can see that we actually now have the stuff that’s running on the servicenamespace. So I want the initial access that we had and the microservice that was running. What this tells us is that the credentials that we just stole has way more capabilities than the one that we had. We can always just confirm this by doing auth.can-I. And you can see right there that a lot more permissions are granted. Looks like mostly it’s related to Calico so that also tells us that we can then start probing into routing network policy [inaudible 00:39:50], maybe turn it off and stuff like that. And we also have the pod and namespace access. So with that now the attacker will be able to compromise the pod. They will be able to look at the CRVs and find a vulnerable API that they can use and then they were able to steal the token of the CRD. So what I’m talking about the token, was the token of the control that was running the CRD. So the token that we stole was actually a token that was used by the [inaudible 00:40:27] controller. So then now we’ll just do that and now we’re impersonate of that. So back to our picture. We’re going to now show what it looks like in the Calico Enterprise site. So in Calico Enterprise we have a metadata dashboard that shows all the flows. So here I’m going to show you the other page, not this one. And I have already set it I can always reset it. So we want to set the first name to the initial access pod. So now we can see the other stuff that the attacker was doing. So we’re in the initial access pod, [inaudible 00:41:24] the DNS server. The pod remotely to curl google and download the kubectl. They also talk to the Kubernetes API service which is dtf.kubernetes. It was also talking to the microservice [inaudible 00:41:40] this one. So we now know what they were trying to access. I should have also tried to access the metadata service. 169.254. And we can also see that we actually had access to the cloud metadata service. With that we can also show you the other site. We’re going to look at the DNS logs so we can see… Yup. So with the DNS logs we can actually see the DPNS request to ngrok. There’s a good chance that this is not updated yet so we might have to wait a little bit. It actually takes a little bit of time for it to consolidate all the data and then upload it to [inaudible 00:42:42]. So we’ll leave it at that. I’ll go back to our slides and finish it up. Oh, sorry. Before that we can also look at the alerts that we have. So we do already have some alerts, called lateral movements so that is what I am using to generate alerts whenever a pod in that namespace talk to the Kubernetes API. This will then generate alert. There’re alerts for the cloud metadata API that we accessed. So let’s go back to the slides. So that was what I was talking about detect and visualization. So how do we detect about lateral movements? So unless they have a container escape, this is all going to be done by the network. And since we have flow logs we’re able to look at all the attackers, what they’re doing, all their probes and lateral movements. And as I kind of showed you, you are able to use something called a GlobalAlerts to create alerts based on the logs. So we can use the GlobalNetworkSet as I showed you, one looks at when the attacker is probing the cloud provider metadata service. Next we can also see the pod networking and Filesystem. I will show you the attacker reaching the Kubernetes API, DNS and other stuff that we were talking. So then you can either create network policies to block it or just create global alerts to choose an alert. I wasn’t able to show you the DNS request because, maybe later, but you should able to see the DNS request in ngrok and then figure out the IP address that the attacker was trying to reach. So we also have the global alerts. This even then generally different alerts for stuff that you want. All the different roles that you’re interested in. And that’s basically it. I didn’t quite look here to see if I gained anything from my genius mount. Nope, not yet. So that’s basically it for my presentation. I’m happy to answer any questions. Thank you, Garwood, for an excellent presentation and demo. That was on detecting lateral movement and defending against attackers. And if you noted, it’s time for our Q&A session and we see a couple of questions coming up from the audience. We’ll just give you another minute or so to do that. Before we do the Q&A, I’d just like to remind everyone that every week Tigera is presenting live webinars and live training events to help you expand your Kubernetes networking and security knowledge. And you’re invited to attend all of these. And please visit our website at wwww.www.tigera.io for more information and a schedule of events. So now we’re going to go back to Garwood here and the Q&A. And Garwood, I have some questions here for you. So the first question is can you please explain some more about global alerts? Yeah, sure. So what this feature does is you can basically label or specify the interesting flows that you want. So in my case it was the cloud metadata API’s address. Or accessing the crib API and stuff like that. And what happens is then when the controller sees these flows coming in it will generate alert and you will see it on the alerts page of the Calico Enterprise. We actually have a really good blog about this on our blog webpage. It’s called Five Ways to Quickly Uncover Malicious Activity and Protect your Kubernetes Workloads. And you can take a look at that. All right, thank you, Garwood. Next question is, is there a way to block an attacker from using kubectl auth can-i –list command? Yup. So for that command to work it needs the cell subject, cell review, something like that. I forgot the full name. Let me double check. The cell subject access reviews. So what you want to do is you want to basically disable that role or access to that. And then they won’t be able to use that kubectl auth can-i –list. Then the attacker will then have to manually try to get pod service one by one instead of having that. Very good. Another question here for you. Is there away to test pods for container escapes automatically? Yeah. So all of the techniques that I showed you, there’s actually a [inaudible 00:48:31] that has a script that runs. Its called BOTB. You just google BOTB, on get help you’ll find this thing by [inaudible 00:48:44] pony. What it does is if you run all the well known [inaudible 00:48:49] for continuous days or try to access the metadata service or the cloud metadata service or do all of the stuff that I was trying to do today. Excellent, okay. We don’t have any more questions right now, Garwood, so I’d like to thank our audience for joining us today. We hope you found the information that we shared today to be of value and we certainly invite you to come back again for one of our future events. Garwood, thank you again for a great presentation and everyone, hope you have a great rest of your day. Thank you, John. Thanks for everyone listening.