With the growing popularity of cloud computing, security incidents related to it have been on the rise. Logs are indispensable resources for countering these threats, and they can be utilized for alerting, taking remedial action, and even preventing future attacks. In this post, we will examine ways to better secure cloud-native applications using logs.

Monitoring requires a multi-faceted approach if DevOps teams want end-to-end visibility and deep insight into issues. This is especially true in the case of modern microservices applications, which are essentially collections of distributed services that talk to each other over a service mesh.

AWS Lambda is the leading serverless computing solution and is one of AWS’ most successful products, to date. Its popularity is in large part because of the way it makes development easier and faster. Lambda completely abstracts away the maintenance of underlying infrastructure including compute, storage, memory, and networking.

When using and troubleshooting IIS web server performance, logs are an essential resource. Let’s look at the top metrics on IIS logs and how to get the most out of logging when working with IIS servers.

AWS is a vast universe with over 100 cloud services, all geared to provide you with the most robust, reliable, and cost-effective cloud computing experience. Many of these services are closely related, and almost seem to overlap. There are multiple database services, logging services, resource creation services, compute services, notification services and even multiple configuration and management tools.

IT has always given importance to security and compliance. As infrastructure becomes more and more complex, IT needs to evolve toward more mature ways of monitoring applications and infrastructure. This means letting go of strategies, models, and products such as SIEM that used to work a decade ago, but are unable to serve the needs of today’s cloud-based microservice applications.

Log analysis involves large volume of data. Startups can generate a couple of gigabytes of data per day, and the largest web companies like Facebook log many terabytes of log data every day. (Back in 2009, Facebook was already logging a full 25TB daily.) How can you make sense of all that data? If you try to analyze it by manually without any tools, good luck. Today, the only way to stay on top of the massive amounts of data that your organization produces— whether you’re a small startup or a Fortune 500 company—is to leverage machine learning to help interpret log data in an automated fashion.

Mesosphere DC/OS (Data Center Operating System) lets you manage your data center as if it were a single powerful computer. It separates the infrastructure logic from the application logic, and makes it easy to manage distributed systems. Considering DC/OS is meant for highly scalable, distributed systems, logging and monitoring plays a key role in day-to-day operations with DC/OS. In this post, we’ll take a look at the goals of logging with DC/OS, and how you can set up DC/OS logging with Sumo Logic.Why Mesosphere DC/OS Clusters Need LoggingWhen you work with Mesosphere DC/OS, you typically have hundreds, if not thousands of nodes that are grouped in clusters. Tasks are executed on these nodes by Mesos “agent” instances, which are controlled by “master” instances. By grouping the nodes in clusters, DC/OS ensures high availability so that if any node or cluster fails, its workload is automatically routed to the other clusters. DC/OS uses two scheduling tools—Chronos for scheduled tasks like ETL jobs, and Marathon for long-running tasks like running a web server. Additionally, it includes app services like Docker, Cassandra, and Spark. DC/OS supports hybrid infrastructure, allowing you to manage bare metal servers, VMs on-premises, or cloud instances, all from a single pane of glass. Together, all of these components make for a complex system that needs close monitoring.There are two key purposes for collecting and analyzing DC/OS logs. The first is debugging. As new tasks are executed, DC/OS makes decisions in real time on how to schedule these tasks. While this is automated, it needs supervision. Failover needs logging so you can detect abnormal behavior early on. Also, as you troubleshoot operational issues on a day-to-day basis, you need to monitor resource usage at a granular level, and that requires a robust logging tool.Second, for certain apps in enterprises, compliance is a key reason to store historic logs over a long period of time. You may need to comply to HIPAA or PCI DSS standards.Viewing raw logs in DC/OSDC/OS services and tasks write stdout and stderr files in their sandboxes by default. You can access logs via the DC/OS CLI or the console. You can also SSH into a node and run the following command to view its logs:$ journalctl -u "dcos-*" -bWhile this is fine if you’re running just a couple of nodes, once you scale to tens or hundreds of nodes, you need a more robust logging tool. That’s where a log analysis tool like Sumo Logic comes in.Sharing DC/OS logs with Sumo LogicDC/OS shares log data via a HTTP endpoint which acts as a source. The first step to share Mesosphere DC/OS logs with Sumo Logic is to configure a HTTP source in Sumo Logic. You can do this from the Sumo Logic console by following these steps. You can edit settings like timestamp, and allow multi-line messages like stack traces.Your data is uploaded to a unique source URL. Once uploaded, the data is sent to a Sumo Logic collector. This collector is hosted and managed by Sumo Logic, which makes setup easy, and reduces maintenance later. The collector compresses the log data, encrypts it, and sends it to the Sumo Logic cloud, in real time.During this setup process, you can optionally create Processing rules to filter data sent to Sumo Logic. Here are some actions you can take on the logs being shared:Exclude messagesInclude messagesHash messagesMask messagesForward messagesThese processing rules apply only to data sent to Sumo Logic, not the raw logs in DC/OS.It may take a few minutes for data to start showing in the Sumo Logic dashboard, and once it does, you’re off to the races with state-of-the-art predictive analytics for your log data. You gain deep visibility into DC/OS cluster health. You can setup alerts based on the log data and get notifications when failed nodes reach a certain number, or when a high priority task is running too slow, or if there is any suspicious user behavior. Whether it’s an overview, or a deep dive to resolve issues, Sumo Logic provides advanced data analysis that builds on the default metrics of DC/OS. It also has options to archive historic log data for years so you can comply with various security standards like HIPAA or PCI DSS.DC/OS is changing the way we view data centers. It transforms the data center from hardware- centric to software-defined. A comprehensive package, it encourages hybrid infrastructure, prevents vendor lock-in, and provides support for container orchestration. DC/OS is built for modern web scale apps. However, it comes with a new set of challenges with infrastructure and application monitoring. This is where you need a tool like Sumo Logic so that you not only view raw log data, but are also able to analyze it and derive insights before incidents happen.About the AuthorTwain began his career at Google, where, among other things, he was involved in technical support for the AdWords team. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications.Mesosphere DC/OS Logging with Sumo Logic is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

AWS Lambda usage is exploding, and at last week’s re:invent conference, it was one of the key priorities for AWS. Lambda simplifies infrastructure by removing host management, and giving you the advantage of paying for just the compute and storage you use. This means it’s important to monitor your usage carefully to ensure you are managing your spend well. This post outlines the various options and approaches to AWS Lambda monitoring so you can make an informed decision about how you monitor your Lambda applications. AWS Lambda Monitoring Basics AWS provides vital stats for Lambda in a couple of ways. You can view metrics from your Lambda console, the CloudWatch console, via the Lambda or CloudWatch command line, or even through the CloudWatch API. The Lambda console gives you an overview of just the vital stats of your Lambda app. For a more detailed view, you can click through to the CloudWatch console. Here are the key terms you’ll notice in your CloudWatch console: Invocations: Invocations are the number of times a Lambda function is triggered in response to an event or API call. Errors: The number of failed invocations. Duration times: The time it takes to execute a Lambda function. This is measured in milliseconds. Throttles: The number of invocation attempts that were not executed because they exceed concurrency limits. Dead letter errors: Asynchronous errors are sent to a dead letter queue for further troubleshooting. If these errors are not written to a DLQ, they are counted as dead letter errors. Within CloudWatch you can set alarms to notify you of issues, or identify unused resources. You can also view all logs generated by your code in CloudWatch Logs. Archiving these logs will incur additional cost, but you can decide how far back you’d like to store log data. Troubleshooting Common Errors in Lambda With AWS lambda, you don’t have underlying infrastructure to monitor. This means that most of the errors can be resolved by troubleshooting your code. Here are the common errors that occur with Lambda: IAM roles & permissions: For your code to access other AWS services, you need to configure IAM roles correctly. If not done, this could result in a permissions denied error. Timeout exceeded: Some functions can take longer to execute than others, and will need a longer timeout setting. Memory exceeded: Some jobs like database operations require less memory. However, jobs that involve large files, like images, for example, will need more memory. You will need to adjust the MemorySize setting in your code if you see this error. Advanced Lambda Monitoring As you start out using Lambda for part of your app, you can get by with AWS’ default monitoring options. Once you start expanding Lambda usage in your applications, and even run entire apps in Lambda, you’ll need the power of a more robust monitoring tool. The Sumo Logic App for AWS Lambda is great for monitoring your Lambda functions and gaining deeper visibility into performance and usage. Here are benefits: Track compute & memory usage: The Sumo Logic app tracks compute performance of individual Lambda functions and lets you drill down to the details. This is important because you configure resources in your code, and you need visibility at the individual function level to ensure all processes have adequate resources. Cost correlation: The Sumo Logic app translates granular performance data to actual billed costs. It lets you track usage of memory, and even excess memory in functions, and prevents overspending. It enables you to split expense across multiple teams if you need to. Based on past behavior, the app can even predict future performance and spend. Integrated reporting: If you already use Sumo Logic to monitor and manage the rest of your infrastructure stack, you can bring it all in one place, and have realistic benchmarks for your Lambda usage. Real-time stats: The data in your Sumo Logic dashboards streams in real-time from AWS, so you never have to worry about seeing outdated information. Advanced visualizations: Sumo Logic lets you slice and dice your data with advanced analytics tools. The dashboards contain various chart types, including advanced types like the box plot chart. This kind of analysis is simply not possible within AWS. As you scale your usage of Lambda, you need deep monitoring that can correlate compute usage with costs. You’re used to having control and visibility over your server-driven apps using traditional monitoring tools. With Lambda, the approach to monitoring has changed, but your goals are the same—to gain visibility and efficiency into your app and the abstracted Lambda infrastructure. You need the right tools to enable this kind of monitoring. About the Author Twain began his career at Google, where, among other things, he was involved in technical support for the AdWords team. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications. Overview of AWS Lambda Monitoring is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

When people talk about containers, they usually mean application containers. Docker is automatically associated with application containers and is widely used to package applications and services. But there is another type of container: system containers. Let us look at the differences between application containers vs. system containers and see how each type of container is used: Application Containers System Containers Images Application/service centric Growing tool ecosystem Machine-centric Limited tool ecosystem Infrastructure Security concerns Networking challenges Hampered by base OS limitations Datacenter-centric Isolated & secure Optimized networking The Low-Down on Application Containers Application containers are used to package applications without launching a virtual machine for each app or each service within an app. They are especially beneficial when making the move to a microservices architecture, as they allow you to create a separate container for each application component and provide greater control, security and process restriction. Ultimately, what you get from application containers is easier distribution. The risks of inconsistency, unreliability and compatibility issues are reduced significantly if an application is placed and shipped inside a container. Docker is currently the most widely adopted container service provider with a focus on application containers. However, there are other container technologies like CoreOS’s Rocket. Rocket promises better security, portability and flexibility of image sharing. Docker already enjoys the advantage of mass adoption, and Rocket might just be too late to the container party. Even with its differences, Docker is still the unofficial standard for application containers today. System Containers: How They’re Used System containers play a similar role to virtual machines, as they share the kernel of the host operating system and provide user space isolation. However, system containers do not use hypervisors. (Any container that runs an OS is a system container.) They also allow you to install different libraries, languages, and databases. Services running in each container use resources that are assigned to just that container. System containers let you run multiple processes at the same time, all under the same OS and not a separate guest OS. This lowers the performance impact, and provides the benefits of VMs, like running multiple processes, along with the new benefits of containers like better portability and quick startup times. Useful System Container Tools Joyent’s Triton is a Container as a Service that implements its proprietary OS called SmartOS. It not only focuses on packing apps into containers but also provides the benefits of added security, networking and storage, while keeping things lightweight, with very little performance impact. The key differentiator is that Triton delivers bare-metal performance. With Samsung’s recent acquisition of Joyent, it’s left to be seen how Triton progresses. Giant Swarm is a hosted cloud platform that offers a Docker-based virtualization system that is configured for microservices. It helps businesses manage their development stack, spend less time on operations setup, and more time on active development. LXD is a fairly new OS container that was released in 2016 by Canonical, the creators of Ubuntu. It combines the speed and efficiency of containers with the famed security of virtual machines. Since Docker and LXD share the same kernels, it is easy to run Docker containers inside LXD containers. Ultimately, understanding the differences and values of each type of container is important. Using both to provide solutions for different scenarios can’t be ruled out, either, as different teams have different uses. The development of containers, just like any other technology, is quickly advancing and changing based on newer demands and the changing needs of users. Monitoring Your Containers Whatever the type of container, monitoring and log analysis is always needed. Even with all of the advantages that containers offer as compared to virtual machines, things will go wrong. That is why it is important to have a reliable log-analysis solution like Sumo Logic. One of the biggest challenges of Docker adoption is scalability, and monitoring containerized apps. Sumo Logic addresses this issue with its container-native monitoring solution. The Docker Log Analysis app from Sumo Logic can visualize your entire Docker ecosystem, from development to deployment. It uses advanced machine learning algorithms to detect outliers and anomalies when troubleshooting issues in distributed container-based applications. Sumo Logic’s focus on containers means it can provide more comprehensive and vital log analysis than traditional Linux-based monitoring tools. About the Author Twain began his career at Google, where, among other things, he was involved in technical support for the AdWords team. His work involved reviewing stack traces, and resolving issues affecting both customers and the Support team, and handling escalations. Later, he built branded social media applications, and automation scripts to help startups better manage their marketing operations. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications. Application Containers vs. System Containers: Understanding the Difference is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.