27. Amazon ES – Hands On
So let’s follow the handson to send data from Cloud Watch logs directly into Amazon Elasticsearch Service. And because there is streaming, it talks about real time. Okay, so let’s go first into the Management console. And we’re going to type Elasticsearch. And here we go. We are into Amazon Es and we’re going to create a new domain. Okay. And it’s going to be a development and testing because we want just one easy and it’s going to be within the free tier. We’ll use the latest version that we have currently available, 7. 1 and click on Next. I will configure the domain name. I’ll call it demo. Yes. And the instance type. And we’re going to choose a T two microbe because this is going to be our free tier. So let’s find it.
T two small. And T Two Small is within the free tier, not micro. So we get up to one month every month of T two micro otu small and up to ten gigs of EBS storage. For elasticsearch service, we’ll use one instance. And then if we wanted to have a solid Es cluster, we need to have a dedicated master. But for now, we’ll just have the master and the data instances being one T Two Small because we’re doing development and we want to remain within the free tier. But obviously in the production instance, you need to have many different data instances and also different dedicated master instances to make sure that your domain is stable. We’ll keep it as ten of EBS and then we could have node to node encryption and encryption of data arrest if we wanted to.
Okay, there’s also some snapshot that will happen every day at midnight. For advanced options, we’re able to set some stuff for Elasticsearch, but for now, we don’t need to know them. Okay, let’s click on next. And we need to have a vpc access. We can use a vpc access so that everything remains within your vpc. Or we could have public access if we wanted to have access to our Elasticsearch cluster from the public. So I’m going to have public access because I want to show you how that works. But if you were to do everything within your corporate network, maybe vpc access is something you would like. Okay, kibana authentication is whether or not we want to have cognito in front of Cabana, but for now, we don’t need this.
And then an access policy, which is what do you want to allow access to your cluster? And for that, I’ll just choose to access to access from my specific IP. And so I need to enter an IP, and so I’ll just do that in the background. Then you click on Next and you review everything. So one az, T two Small, one instance, ten gigabyte EBS volume, and we’re good to go. Then I’ll click on Confirm and it will take a little bit of time before our cluster is successfully configured so for now, what I’m going to do is just wait. It will take about 15 minutes, 1015 minutes before it’s ready. So my elasticsearch cluster is not created and I’m able to access it directly through this url. So it’s not a very pretty ui, but this is just a Json Rest api right here that we have.
And then also I can go into Cabana and have a look at the ui for Kibana itself. As we do not have any data in elasticsearch yet, it’s saying that, let’s get started. You can try our sample data and it shows you the kind of capability they are around Cabana. So we can look at sample ecommerce orders, we could look at sample flight data, or we can look at sample web blogs as well. And these give us different kind of dashboards into our cluster and understand the kind of capability that Kibana has, for example. So here perfect. Everything has been installed so we can look at the three different keybana dashboards and look at the kind of capability it has against, for example, cloudwatch dashboards.
As we can see, we get some more different kind of graphs. This is a pie chart. We’re getting some information with the pie as well, and a number within. We have stacked graphs, we have total revenue, we have a map, we’re able to map stuff, we’re able to do some word cloud, view some tables as well, and so on. So, as you can see, Cabana is something that we have to create as part of the elasticsearch service, but it definitely has more capability than Cloud Watch dashboards. You also have filters, so you can search for a specific type of data, and the entire dashboard will be refreshed based on the filter you set. So you can have a play and look at Global Flights dashboard as well.
And you can look at web traffic logs. So this is based on log data, and it generates a lot of information based on that log data around the type of files that are requested and how long it took and when it took, and so on. So very, very interesting capability. And this is just cabana. So what we want to do now is how do we get data into elasticsearch? All of this was sample data, obviously. So for this we can go back into Cloud Watch, and this is the whole purpose of this, hands on. So if we go into Cloud Watch, we can look at one of our streams. So we can look, for example, at the log stream, cloud trail, and this cloud stream, we’re going to do action, and we can stream it to Amazon elasticsearch service.
Now, remember, this is exactly the same as streaming it to lambda and having Amazon create a lambda function for us that will send it to the elasticsearch service. So let’s click on this and we’re saying, okay, this account is what we want to do.And the cluster I want to target is this cluster right here. And now we have to create a new iam role for our lambda. So it takes us straight into a prompt to create that role for us. So what I’ll do is just I will approve and if you look at the policy documents, we can see that it can allow for any es http post to any elastic Search cluster. So I’ll click on allow and perfect. So this lambda function will be created by Amazon itself and will be used by Cloud Watch logs to stream logs lines to it and then it will deliver data into Amazon es.
So this looks good. Let’s click on next. And the log format now is going to be it is Cloud Trail. So you can define a type of format if you wanted to. So it could be vpc flow logs, lambda, common log formats, pastel Limited, Json or other and other would allow you to specify a pattern. But for now, because we are using Cloud Trail, let’s just use aws Cloud Trail in here. We could say a subscription filter pattern if you wanted to test and just get a little bit of data. But we want to get everything. So I’ll just have the quote unquote and this should give us or just nothing. And this should give us everything. So this gives us 15 matches out of 50 events and that means that every single log line from Cloud Trail will go into Elasticsearch. So next everything looks good.
We’ll review every single setting. Everything looks good and we confirm and we’ll say okay, start streaming. So here it needs to say, for example, if you have an access policy for your service that allows access from your browser, we can directly start interacting with it. So let’s click on Start streaming. And now the subscription filter has been created. And so now in here within the subscriptions, we can see that lambda is the target for this log group and the subscription itself is called Logs to Elasticsearch demo es. And let’s go in the Lambda service, let’s see if we have control over that function.
So we are in lambda and as we can see, this one is Logs to Elasticsearch demo es. So this lambda function was created by Amazon itself to send data into the Elasticsearch service and the source of it is Cloud Watch logs. And we have the subscription in here. So it’s quite interesting because now you can look at the code itself and see what it does and it’s quite long. But at least this is a function that we don’t have to touch ourselves and we could modify it if we wanted to, so we could do into monitoring and look at what is happening for this function. How many times is being invoked and so on. And so we probably need to wait a little bit before data gets into Elasticsearch.
So now we are in cabana at the home of Cabana on the top left. And in here, I’m able to use our elasticsearch data to create a cabana dashboard so I can connect to my elasticsearch index. And as we can see, this index was created automatically for us. So if we use cwl, and this is cloud watch logs minus and then the star, that means that any new index for cloud watch logs and there will be one new index per day will be used for this pattern. So perfect cwl minus and then star, then next steps, and then the time filter is going to be okay, what is the time going to be used? And so for this, we’ll use at timestamp, and this will be great. And then we create the index pattern, and the index pattern is being created.
And now the data that is being streamed by cloud Watch into Elasticsearch can be used by Cabana to create a dashboard. So we get a lot of information. We have about 224 fields available to us, so we get an idea of what we can build with this. And so in here, now you can just go ahead and create your own dashboard if you wanted to. Now, I’m not an expert at cabana, but this was just showing that everything was joining together. So if we go in here and select this pattern filter, then we can see all the event source and the event names, how many appeared per second, and so on. And so for this one, okay, this happened to have these json in here. So this is a json that came straight out of cloud trail into elasticsearch and in cabana.
And I’m able to look at all this information within the event, so I’m able to extract the event name, the event source, and so on, and build any kind of crazy dashboards that I want to. Okay, so that’s it for this, hands on. Now, if you go up to the lambda console and refresh this, we should start seeing a lot more metrics. And here we go. We have three invocations that have been done, we can see how long they took, we can look at the error rates and so on. And that completes the entire demo of streaming logs through a subscription filter into lambda. That lambda function is actually a managed lambda function, and so it’s by aws, and that lambda function is directly sending data into elasticsearch that we visualize through using cabana.
And the only thing we haven’t demoed is log stash. But Luxtache, just remember that it’s just an agent that runs on an EC two instance, for example, and send the log lines directly into elasticsearch, from which we can use cabinet again to visualize these log lines. So that’s it for this lecture, I hope you enjoyed it. Just make sure to remove your elasticsearch cluster when you’re done by clicking on delete domain and then delete. And that’ll be it for this lecture. Also, remember to remove the subscription. So go into here and remove the subscription filter. And this will stop that lambda function from working. All right, that’s it. I will see you in the next lecture.
28. Tagging in AWS
So the last thing you need to understand as part of domain number three in the DevOps exam is that you should have a tagging strategy for your resources in your accounts. So the tags are metadata that you assign to your AIDS resources and they’re key value pairs and they’re very easy and they will be able to help you manage, search for and filter resources. And so if you go, for example, for easy two instances and we go to tag tags, we know and we’ve seen from many different ways that tags are assigned or could be assigned to our EC two instances. For example, cloud formation will automatically assign the cloud formation stack ID, stack name and logical ID to any resource it creates.
If the resource is part of an auto scaling group, the auto scaling group itself will add its name onto the instance. If we look at the name in here, it corresponds to the name tag and the description could be a tag we want as well, but we are able to add our own tags. For example, when we had our web servers, we had our environment being development and other web servers have the environment being production. So we’re pretty free to tag anything we want. Not just EC two instances, but we could tag security groups in here, we could tag load balancers if we wanted to, we could tag crooked groups, auto scaling groups, EBS volumes, even lambda function.
If you go to a lambda function in here and you click on this one and you scroll down, you will see that there are some tags that you can set for your lambda function. So I think pretty much almost everything in aws could be tagged. So the idea is how do we have best practices and how do we implement a tagging strategy and for what? So we want a tagging strategy for many things. The first one is to do cost accounting. So if you want to track cost according to departments, to environments, to resources and so on, it is great to have a consistent set of tags because when we go to the billing report, we’ll be able to get a report that will track and divide based on the tags we’ve defined.
So the first one, the second one is for example, when we have could deploy for deployment groups, this is something that by using these tags could deploy is able to understand to which instances you want to deploy to. Another one would be for security. There is a way to use tags and implement this and combine this with iam policies and condition statements through saying that some users only have access to the EC two instances that are tagged development. This is definitely something you can do. It’s called tbac or Tag based Access Control. And so this document in here retells you exactly how you should tag.
So this is not a go to resource and implement one by one everything they say, but more around best practices and things that other people do in aws that you could get inspired from. For example, use a standardized case sensitive format for tags and implemented consistently across all resource types. So that means, for example, do you want to have lower case, snake case and so on. Also there could be tag dimensions to manage resource access control, cost tracking, automation and organization. Do you want automated tool to manage resource tags? And for example, one of these tools again is cloud formation error on the side of too many tags rather than too few tags.
So definitely how do we implement all these tags, what can we think of? And let’s implement all of them because tags are cheap, they’re free, so use them as many as possible, okay? And requirements change. So it’s possible for you and we’ll see this using a device config to track and define tag rules and ensure that everything is compliant within your organization. Now, in terms of tagging categories, what can you in Cloud as a tag? There could be some technical tags, name, application, ID, application role, cluster, environment version.
These are great. Some tags for automation, for example, daytime to see when it should be started, deleted, stopped or rotated, opt in, opt out for some features, for example, and security to understand if encryption is enabled and whatnot business tags, who is the owner, what’s the cost center or business unit that has this tag, the customer, the project. So this is really helpful when we have some custom accounting and once you understand how our infrastructure is costing us based on the clients we have and finding some security tags, for example, confidentiality or compliance if we’re using iam policies, using tag based access control. So it shows you some common tagging strategy.
There’s some tags for console organization, tags for cost allocation and tax for automation, and finally access control, as I said. And tagging governments will see how to use those using interbisconfig rules. We’ll look at service catalogs and confirmation is a great one because cloudformation automatically tags all the resources within your accounts if you do specify those. Okay, so there is a pdf version, and I’m not going to read this out to you, but I do suggest strongly that you go through this document, read it all, understand tagging strategies in aws because this is part of domain three at the exam. All right, that’s it for this lecture. I will see you in the next lecture.
