When DevSecOps is supported by sufficient log analysis, organizations can maintain application security without slowing the speed of delivery. Learn how log analysis supports DevSecOps.

AWS Lambda lets a developer create a Lambda Java function, which can be executed in the AWS Cloud. Learn how as we walk through the process of uploading and testing your function through the AWS Console.

New legislation in Australia means that companies need to prioritize detection. Read what you need to know about the law and how to improve your security posture.

Since we launched reliability management to help developers manage SLOs, we've learned how practitioners like you use it. Read more to discover best practices and common approaches.

Cloud database security is always evolving. Securing data in the cloud is only the beginning. Learn how security architectures have changed, what the modern app stack looks like, and cloud database security best practices.

How would you know it was the end of the year if companies like Sumo Logic didn't publish our end of year round-up? Read up on blogs you missed or be amazed to see how time flew by this year.

Read this blog about cloud-native. We offer some answers and insights with experts from a variety of organizations about making the switch to cloud-native, or not.

The path to success in tech can take many different forms. Finding inspiration and mentorship along the way can really make a difference in the road you take. Learn how five female leaders found their way in this recap from the Women of Sumo session at Illuminate 2022.

Whether you have plenty of developers to handle security automation or need a low or no-code platform, there's an appropriate solution for you. Learn which approach best fits your team and requirements.

Continuous security is a challenge in Kubernetes. Learn about the main types of security risks and how to maximize your Kubernetes container security with best practices.

Learn about the challenges of securing a microservices architecture and get tips on building out a microservices security program.

Learn more about database security and the challenges of cybersecurity. Discover the best practices to help you protect your data from internal and external threats.

Learn what IT tool consolidation is, why it's critical to act now, and the best practices to help you tackle IT tool and data sprawl.

This Sumo employee spotlight features Bryan Liberator, senior director of investor relations who talks about his passion for working with the investor community and how Sumo has given him exposure to multiple different paths in his career.

As AWS services expand, so do Sumo Logic’s collection and visualization capabilities. Learn how Sumo Logic has expanded AWS offerings in 2022 for improved monitoring and troubleshooting.

How can your IT systems be ready for the holiday season rush? Read about insights and best practices from Ulta Beauty’s IT director of e-commerce.

Microsoft Office Applications store credential material in memory. Learn how attackers are pivoting to these processes for sensitive authentication material theft.

Moving to the cloud can sound simple enough, but if you're going from a monolith to microservices, you'll need to redesign your architecture. Learn how Docker containers can help.

Discover insights and actionable advice from SecOps and DevOps experts about the role of DevSecOps. Learn how Sumo Logic helps DevSecOps thrive.

Concepts of simultaneous and parallel work streams are core to the evolving understanding of DevOps. Read how to start by building an Agile culture.

Every level of the application stack emits logs, metrics, traces, and events. Collecting all these signals is the first step in having an observable system. See how Open Telemetry makes this possible.

This Sumo employee spotlight features Colin Fernandes, Senior Director of Global Market Development who talks about the opportunity for Sumo Logic in the EMEA region and why Beyonce inspires him.

Gain greater visibility into AWS Lambda with the new Telemetry API integration, which makes it easier to get Lambda logs, metrics and traces into Sumo Logic.

We recently evaluated Amazon Managed Workflows for Apache Airflow (MWAA), ArgoCD and Kubernetes for our Apache Airflow. Read how we decided which one was best for our use case.

Learn how to optimize Amazon S3 with top tips and best practices. Bucket limits, transfer speeds, storage costs, and more – get answers to these S3 FAQs.

Discover insights and actionable advice from chief information security offers about how the enterprise can protect its modern applications. Learn how Sumo Logic provides full-stack application monitoring and observability.

Organizations tout using data to make key decisions, but the return to office challenge is often met with emotion and surveys. Learn how Sumo Logic uses data to address this situation.

This Sumo employee spotlight features Oliver Gompf, Sales Director based in Germany, who shares why the opportunity for Sumo Logic in the DACH market is significant and ripe for the taking.

Get ahead of AWS cost surprises with the AWS Cost Explorer app for Sumo Logic. Build budgets with confidence when you gain visibility into usage & costs.

For the second year in a row, Sumo Logic is thrilled to be named a Visionary in the Gartner Magic Quadrant for SIEM. Read why we believe our cloud-native security platform is so well regarded.

Sumo Logic is adopting open source docs, allowing customers to join developers in contributing to our documentation. Join our community, building the best documentation for all Sumos.

Looking for potential alternatives to Datadog? Learn about other tools to monitor and secure your applications at scale. Review alternatives.

Efficiently resolve user-impacting issues with these best practices. Learn how to map your infrastructure behavior to end-user experience to improve your digital customer experience.

Increased cyberattacks and compliance requirements highlight supply chain vulnerabilities. Learn why they’re vulnerable and how to protect your organization.

Sumo Logic CEO, Ramin Sayar, reflects on the two-year journey the company has taken since IPO. Read his interpretation of how the world has changed, and what this means for Sumo Logic.

How your cloud credentials are stored on Windows endpoints could lead to unforeseen risks. Learn how Sumo Logic helps you hunt for and monitor malicious activity.

While the CERT-In Directions 2022 are disputed and controversial, your organisation needs a plan to comply. Get started with Sumo Logic Cloud SIEM and Cloud SOAR to get compliant by the deadline.

If you're considering a FedRAMP authorized solution these best practices can help. Learn from HackerOne and Sumo Logic why cloud migration is a unique challenge for federal agencies.

Real User Monitoring (RUM) now provides even more value. Learn about our upgrades including Single Page Apps support, collecting browser errors and more.

Forward-looking companies are embracing reliability management. Learn how they're measuring business outcomes rather than system performance.

Every organization requires a different level of reliability depending on the domain, product and use case. Learn how to make the most of your SLOs.

Efficiently monitoring modern applications stacks to get notified about production issues is hard and complex. Improve alerting and managing as your applications grow.

Gartner’s 2022 Security Orchestration, Automation and Response (SOAR) market guide is out. Read our blog post for a recap.

How to get your VPC Flow Logs directly into Sumo Logic via Amazon Kinesis Data Firehose. This simplified process can help reduce your operational overhead and lower your total cost of ownership.

At a recent offsite in Goa, this Sumo Logic team forged connections, pride, insights and elevation. Learn how this event is shaping local company culture.

If you’re looking for inspiration on how to harness the power of your data to deliver reliable and secure digital experiences, check out the top five reasons you need to attend Sumo Logic’s Illuminate virtual user community event.

As your organization accelerates digital transformation, you need confidence in your cloud migration. Learn best practices for a successful migration.

This Sumo employee spotlight features Libby Christensen, a principal program manager who shares the lessons learned from driving product launches, why diversity makes for better team collaboration and more.

Automation is an important component of DevOps but how do you put automation into practice in order to advance the goals of DevOps? Let's answer that question.

Learn powerful insights on the business impact of digital customer experience in this DevOps Insight Forum panel discussion. Hear what e-Commerce and DevOps leaders have to say about measuring and optimizing CX.

Learn how Rick Jury, a Sr. Technical Account Manager for Sumo Logic went from being a happy customer to an employee in a quest to delight other customers.

Sumo Logic is continuing its live East Coast tour. Next stop: AWS re:Inforce Boston July 26-27, 2022. Come by our booth to see our AWS security solutions live and in action!

Enterprise-level research into SRE practices reveals current trends. Read on to go beyond the technology to understand the employee impact of SRE.

As usual the Monitorama talk selection team did a bang-up job. Every talk was interesting, but a few jumped out at me for some very specific reasons. I’ll broadly categorize these as the best talks (talks that gave me hope and inspiration), the worst talks (talks that terrified or depressed me), and the beautiful talks (talks that captured something beautiful).

Sumo Logic's integrations with Atlassian DevOps tools give better visibility into software delivery performance. Integrate with Jira, Bitbucket, and Opsgenie.

Entity Groups for Sumo Logic Cloud SIEM CSE help security analysts simplify threat detection and investigation. Read more about how Entity Groups can improve your threat response today.

While the world is largely returning to a semblance of comfort with larger public events, COVID is still a real and present threat, something we keep in the backs of our minds all the time. But having a sense of community is also important and so, we weigh the risks — go another year without collaboration and connection — or take sensible precautions, cross our fingers, and accept the risks of a large indoor gathering?

Learn how to gain comprehensive visibility into your Kubernetes applications. Discover how Sumo Logic helps with native integrations, open-source data sources for Kubernetes and data collection.

This blog is about how to communicate changes in your application monitoring process as your operations, environments and services evolve. Keep your business moving forward by tracking changes to application monitoring.

When Sumo Logic bought DFLabs it inherited not only a powerful SOAR platform but a great team of employees. Learn how one software engineer in Macedonia found himself at home at Sumo Logic.

Learn the differences and similarities between SOAR and SIEM security solutions and how both can benefit SecOps within modern SOCs. See which is right for you.

Machine intelligence is weaker than what the media and TV portray. Is Artificial Intelligence Phony? Learn About Artificial Intelligence and its applicability to IT Operations.

This blog highlights some of the ways Sumo Logic is honoring Pride and offers idea for how to become a better ally within the LGBTQIA+ community

Read about a new Remote Code Execution vulnerability with the Microsoft Support Diagnostic Tool (MSDT) in Windows from Sumo Logic Threat Labs.

Sumo Logic is making its first entry into the 2022 Gartner Magic Quadrant for APM and Observability. Learn more about our journey to get here today.

It's hard to secure applications across cloud services when developers release code fast. See how Sumo Logic secures apps in a serverless world.

Sumo Logic expanded Cloud SIEM coverage for Microsoft Azure, following the MITRE ATT&CK framework. See what's been added to further secure Azure.

Sumo Logic invited Dr. Sarah-SoonLing Blackburn to discuss the Asian experience at work and how to foster greater inclusion. Learn more!

This RSA Conference is back! In a world of modern digital business where reliability and security are critical, Sumo Logic will be there to discuss the security landscape at large.

Learn best practices for Node.js logging to maintain, manage and ensure consistency within your applications and systems.

Sumo Spotlight highlights Sumo Logic employees from around the globe. This spotlight features employee John Merideth, a technical trainer based in Bedford, NH.

Weak login portals are irresistible to hackers. Learn how to protect your login portals from common exploits. Use Sumo Logic to quickly detect bad actors and secure your customer data.

Learn about Sumo Logic’s new OpenSLO contributions. Dig into a production example of how our SREs manage SLOs as Code.

Sumo Spotlight highlights Sumo Logic employees from around the globe. This Sumo Spotlight features Supriya Desai, senior director of people, based in Redwood City, CA.

I recently had the opportunity to discuss state-of-the-art technologies to support security operations with industry analysts. I asked questions and confirmed that the current view of SOAR (security orchestration, automation and response) and SIEM (security information and event management) goes well beyond the security operation center (SOC).

Tools and specific engineering practices are key components to helping organizations evolve from a traditional IT mindset to a culture of DevOps. But this dramatic change requires taking intentional steps to influence a culture open to new approaches, deeper collaboration and service ownership.

Introducing the Sensu Integration Catalog — a marketplace-like UX for simplifying new user onboarding, and deploying production-ready monitoring in a matter of minutes. The Sensu Integration Catalog is also an open marketplace that new and existing users can contribute to by sharing Sensu configurations.

Sumo Logic is excited to join in celebrating the 52nd Earth Day! Earth Day is a global event for environmental awareness, and sustainability, and as well as honoring the anniversary of the birth of the modern environmentalist movement. We believe it's essential now more than ever to move toward greener practices as a business and as people.

Nowadays, it’s mostly a foregone conclusion that companies need a security program and centralized log aggregation and correlation platform. Unfortunately, the conversation all too often turns toward tactics for data collection and detection of specific threat actors or common vulnerabilities and exposures (CVEs).

Security professionals involved in the IT and cybersecurity industry for the last 10+ years have most likely come across the terms SIEM and recently SOAR, but there is still much confusion about what the specific use cases and purposes are.

Sumo Logic's senior security engineer Stefan Tapp shares his love for working with customers to modernize their security programs, and why he likes working at Sumo Logic.

Today I’m going to show you how to use a sample Java application called PetClinic to capture application traces using OpenTelemetry for Java auto-instrumentation and send them to Sumo Logic. Ask any Java developer: Spring is the most widely used web application framework and PetClinic is the most popular project in Spring. Thus the choice of the application as our demo subject.

The news that hacking group Lapsus$ gained unauthorized access to Single Sign-On (SSO) provider Okta through a third-party support account sent chills through information security professionals everywhere.

With so many simultaneous events going on, heightened awareness in response to state actors, US President Biden’s cybersecurity call-to-action, and the Microsoft Event, all of us need to remain aware and vigilant as supply chain attacks continue. We highly recommend taking a proactive approach to secure your environment with a defense-in-depth strategy and appropriate monitoring.

If you are performing SRE work add your voice to this important research.

Since our inception, Sumo Logic has been laser-focused on delivering real-time machine data analytics to accelerate digital transformation, while helping businesses effectively build, run, and secure their modern applications and infrastructure—in the cloud or in hybrid environments.

As organizations have moved toward a microservices design pattern, the need for reliable and performant solutions that enable decoupled services to communicate with one another has grown. RabbitMQ is an open-source message broker designed for this purpose. We’ll discuss what RabbitMQ is, how it works, why it needs to be monitored and how Sumo Logic can effectively do this.

In honor of International Women’s Day, we are revisiting our Women of Sumo panel discussion from our Illuminate user conference back in October. The topic? Practical advice for how women can advance their careers in science, technology, engineering, and math (STEM).

Learn how war room capability boost cyber team's collaboration and SOC efficiency

ActiveMQ is a message-oriented middleware, which means that it is a piece of software that handles messages across applications. It acts as a broker that can help facilitate asynchronous communication patterns like publish-subscribe and message queues. The main goal of those servers is to create a scalable and reliable message bus that different components can use to communicate with each other.

We know customers and users today demand new features to be frequently released to their favorite apps. Plus they expect any bugs or issues hindering a great user experience to be fixed—and fast. Here we're going to cover new capabilities built to help you keep up with the business by measuring how well your team works in small batches and identifying previously invisible cross-team dependencies in your development and delivery processes.

Cybersecurity presents an ever-escalating challenge for most C-level executives. As the average cost of a data breach continues to grow, the sheer volume of attacks threatens to overwhelm resource-strapped IT organizations.

It is safe to say that customers and enterprises have come to expect their digital experiences to be near instantaneous. Fifty three percent of consumers will wait no more than three seconds for a web page to render before abandoning the site. But new technologies, like connected vehicles, AR/VR, and industrial automation, are pushing the limits of what traditional architecture can handle when it comes to delivering ultra-low latency.

Public cloud hosts like Amazon Web Services (AWS) provide platforms with a wide array of capabilities that quickly scale based on demand. As a result, we’ve seen an explosion of new applications and services that continue to change our daily lives for the better.

The growth of site reliability engineering (SRE) has demonstrated the need for SRE implementations is here to stay for the foreseeable future. LinkedIn voted SRE jobs as the second most promising positions in the US in 2019, and now as we head into 2022, you can be sure to see the evolution of SRE continue to grow and expand.

Learn how to build the modern SOC with Cloud SOAR and Cloud SIEM.

Machine data analytics is the process of parsing data generated by software from a wide variety of sources including servers, networks, applications and financial records. These, and many other similar sources, produce massive amounts of data including from local operating systems, identity/access management tools, cloud consoles and their associated log files, alerts, scripts and profiles. The goal of machine data analytics is to make this data more understandable by converting it into useful insights to monitor security events.

As a DevOps, SecOps, or IT operations manager, you're surrounded by all the technology for the systems running the entire organization. This means legacy infrastructure, multi-cloud environments, services, tools, and applications.

AWS spot instances are a great way to optimize AWS costs, however, you risk an increase in service disruptions. See how we tested and monitored AWS spot instances to get the best of both worlds.

Managing the security of your Amazon Web Services (AWS) environment requires constant vigilance. Your strategy should include identifying potential threats to your environment and proactively monitoring for vulnerabilities and system weaknesses that malicious actors might exploit. In a complex environment – such as your AWS account with a multitude of services, coupled with various architectures and applications – the ideal solution should be both comprehensive and straightforward.

Delivering value to customers quickly and efficiently is critical to the success of modern businesses. Understanding the process and timeframe during which an organization generates new ideas and then designs, develops and deploys them is vital to success.

We are living in a data world. Data describes and controls almost every aspect of our life, from the president's elections to everyday grocery shopping. Data grows exponentially and so does the complexity of applications that manage that data.

Over the past decade, the software development landscape has shifted dramatically. Long development cycles resulting in big releases are now a thing of the past. Instead, the focus is on delivering value to the customer quickly and efficiently through small releases delivered at a high frequency.

Consumers and users of applications expect near 100% availability and reliability to work, transact, collaborate, etc. There’s a lot of talk about monitoring the performance of the application itself, but what about the underlying systems and components supporting the app, and in particular the infrastructure it sits on?

Sumo Logic SpecOps Situational Brief on the Log4Shell CVE-2021-44228 vulnerability and attempted exploitation observations.

While we are a long way from implementing Skynet, using machine learning combined with automation to make real-time decisions is here. In a recent talk at Sumo Logic Illuminate, Dave Frampton, General Manager of Cloud SIEM and Security Analytics, discusses the future of security with Vijaya Kaza, Head of Engineering and Data Science for Trust & Safety and Chief Security Officer at Airbnb. Kaza describes how automation, machine learning and AI can strengthen a company's overall security posture.

Technology environments are rapidly evolving as organizations look to remain competitive, accelerate innovation and make themselves more agile. But in the process, many of the observers, i.e., stakeholders who track infrastructure and application metrics, are falling behind, unable to monitor and manage modern, cloud-native apps and multi-cloud environments due to the complexity that comes with them.

We’re excited to announce updates to Sumo Logic AWS Quick Start Integrations that enable customers to automate the integration of AWS Security Reference Architecture within Sumo Logic Cloud SIEM powered by AWS. The new integrations automate the collection, ingestion, and analysis of applications, infrastructure, security, and IoT data to derive actionable insights for security engineering teams.

Cloud SOAR helps modern SOCs achieve greater levels of SecOps productivity and takes cloud workflow processes to the next level.

Let’s take a look into why and how you should be closely monitoring your Windows server environments from a security perspective. We’ll investigate the types of logs, events and other actions that you should consider. Finally, we’ll look at how you centralize monitoring into a central dashboard, and automate many of the tedious aspects of Windows security monitoring.

Back in 2018, AWS first released its Graviton processor—their 64-bit Arm processor—and followed that with the release of Graviton2—Version 2 — just a year later. Now customers running ECS and EKS on EC2 can choose between X86 and ARM64 depending on which processor best fits their application workload.

Enterprise SOCs are becoming a crucial part of most organizations’ management departments due to the increase in digitization and interconnectivity. SOCs play a major role in monitoring, managing, and responding to security alerts within a company's daily operations. Since cyber attacks have become more sophisticated, the requirements for SOCs have changed due to increased volumes of data, the complexity of security ecosystem tools, and increased data sources and attack vectors. When it comes to efficiency, SOCs need to expand their focus beyond log management and data analytics to include more advanced functionalities such as automation, leveraging big data and AI for intelligent decision support, and increasing visibility into their product through observability.

SDO helps you track siloed data across the DevOps toolchain. It normalizes and correlates data, provides you with DORA’s 4 key metrics and gives you deep insights into the velocity and quality of delivery across services and teams.

Amazon Elastic Kubernetes Service, or EKS, is a managed Kubernetes service. That means that Amazon Web Services (AWS) handles some of the deployment and management tasks for users. But the fact that EKS is a managed service doesn’t mean that AWS manages all administrative tasks.

At Sumo, the metric we use at the base of unit economics is the number of logs ingested by our customers. Our data pipeline continuously reports its ingestion statistics in a form of loglines fed into an internal organization in one of our deployments, making it easy to query them later on.

As Sumo Logic’s new President of Worldwide Field Operations, Lynne Doherty shares her thoughts on joining Sumo Logic, excitement about the journey ahead, and re-iterating the continued commitment that Sumo Logic has to our customers and partners success.

Application performance management (APM) and distributed tracing are practices that many teams have been using for years to help detect and mitigate performance issues within applications—while the first one was born in the era of big single-host monoliths, the latter is especially useful for distributed applications that use a microservices architecture, in which tracing is critical for pinpointing the source of performance issues.

Sumo Logic is constantly improving our Cloud SIEM solution to meet the needs and demands of our current and future customers and help them modernize their security operations.

When you’re planning an application performance monitoring (APM) strategy, collecting metrics from storage services like Amazon S3 may not seem like a priority. After all, part of the point of object storage is that applications can read and write from storage buckets seamlessly, with minimal configuration and overhead.

In today's digital transformation journey, observability has emerged as the hot topic. The ability to capture, analyze and correlate all of your machine data is critical to ensure you can meet your reliability objectives. Every level of the application stack emits logs, metrics, traces, and events. The collection of all of these signals is the first step of having an observable system.

Everyone here at Sumo Logic is thrilled to announce our inclusion as a Leader and Outperformer in the 2021 GigaOm Radar Report for Security Orchestration, Automation, and Response (SOAR).

The level of engagement taking place during presentations, workshops and executive meetings showed the high level of interest in open telemetry, unified analytics and full-stack observability to solve the challenges inherent in application modernization and cloud migration.

The cybersecurity skill gap remains one of the top problems in the industry. Here’s how your SOC can deal with the scarcity of qualified security professionals.

Year over year, organizations onboard new technologies to improve their cybersecurity posture, yet data breaches are still on the rise. The root cause of data breaches is people. In fact, 85% of data breaches today are caused by human error.

Think of the software supply chain as every software element in your organization—from software development of internal systems to open source or third-party enterprise software to vendors, partners, and even past suppliers who still hold access to company data or IT systems.

Cloud native technologies like SaaS, microservices, serverless and APIs, coupled with DevOps methodology enable companies and their development teams to scale efficiently while deploying frequent updates to the software and infrastructure that customers want.

As part of its recent acquisition of Sensu, Sensu Go is now part of the Sumo Logic Continuous Intelligence Platform, empowering enterprises and developers to quickly get real-time insights from unstructured data for troubleshooting, performance improvement, and security across dynamic multi-cloud infrastructure.

Imagine you are an on-call engineer, who receives an alert at 3 AM in the morning informing you that customers are experiencing high latency on your website, and are unable to shop.

With so many overlapping and self-serving definitions of XDR (Extended Detection and Response), we thought we would provide a perspective from some of us on the front lines of trying to embrace technology innovation while filtering vendor marketing noise.

We know organizations today rely on software applications to drive their digital transformation, providing customers with the tools, features and experience end-users have come to expect when doing things such as transact, work and communicate, to name a few.

Sumo Logic is excited to host our 5th user conference Illuminate 2021 from September 28-29, 2021. The security track during the two-day conference has interactive sessions covering: Modernizing security Get started and do more with Sumo Logic security Hands-on workshops Security certifications

Can you really build a cloud-native SOC? As more organizations adopt cloud-native and cloud-hosted technologies, what do security teams and security practitioners need to consider when it comes to the SOC and cloud security?

The countdown is on to our 5th annual Illuminate user conference September 28-29, 2021! This year we are once again offering a virtual experience to keep everyone healthy and safe.

We’ve all heard about cyberattacks on corporations, but when those attacks go after critical infrastructure, such as the energy grid, it can affect every person in the country. At the Modern SOC Summit, Pete Tseronis, former CTO at the Department of Energy (DoE), had a lively discussion with Adrienne Lotto, Chief Risk and Resilience Officer at the New York Power Authority (NYPA), and David Wells, Senior Advisor at the Department of Energy Office of Electricity, about current and future challenges of securing critical infrastructure.

Security automation is becoming increasingly prevalent in modern SecOps. Find out the five reasons why automation augments, instead of replacing humans.

Supervised active intelligence plays a crucial role in helping security teams elevate their efficiency and productivity.

In these uncertain times, getting your cybersecurity budget approved is a daunting task. Read this article and learn how to optimize your cybersecurity budget proposal.

Learn why Cloud SOAR’s Open Integration Framework offers unmatched capabilities.

Learn how combining the MITRE ATT&CK framework with SOAR drastically improves your SecOps efficiency.

Learn how to improve the MTTD and MTTR of your SOC team through the powers of SOAR.

Whether it’s because we’ve watched one too many sci-fi movies, or we’re just plain scared of the potential of AI, automation, and machine learning, many suggest that the dawn of automation is going to make humans obsolete in security operations.

HAProxy is one of the fastest and most widely-used load balancing solutions available today. If you’re already using HAProxy, or if you’re considering using HAProxy in your environment, then this is a great place to start. On this page, we discuss HAProxy logging and why logging is such a vital component of the load balancer implementation. We then take a deep dive into the logging offered by HAProxy. Finally, you’ll read about working with the HAProxy logging format and how you can configure the logging to suit your needs better.

Sumo Logic is excited to announce that it has achieved the New York State Department of Financial Services Attestation to support our customers’ abilities to meet privacy and security requirements for critical data.

Cybersecurity automation is becoming a necessity, rather than a luxury. Find out how to easily and safely automate your security operations.

As part of the move to the cloud, everything in the enterprise is being transformed—from infrastructure all the way to modern application development. Nowhere is this change more important than in the role of the CISO.

In this article, we will explain the basic characteristics of Redis, MongoDB, and Cassandra databases. In addition, we will show you how to collect and monitor the logs that these databases produce using Sumo Logic’s unified logs and metrics (ULM) offering and related apps.

Learn how to practically use Cloud SOAR’s Search Query Bar and drastically boost your SecOps.

Sumo Logic’s industry-leading machine data analytics platform and Cloud SIEM, combined with a new suite of native integrations for ServiceNow, solves this challenge by bringing full coverage to these IT and security workflows by automatically identifying critical events and potential threats with enriched context from assets, leading to improved service outcomes, application reliability, risk management and ROI.

Implementing a SOAR security strategy will do wonders for your SOC. What kind of wonders? Read on to find out.

Find out the top 5 cybersecurity trends to watch out for in 2021.

The cost of cyber attacks compared to the cost of cybersecurity in 2021 unfolded.

Proactive threat hunting is going to be crucial in 2021. Find out how SOAR takes your threat hunting to the next level.

Ransomware attacks are more numerous and more sophisticated than ever before. Here’s how SOAR helps you prevent them.

2020 was challenging for cybersecurity teams. Read this blog to find out what 2021 has in store for the cyber world and learn more about the role of SOAR in the upcoming year.

HAProxy is one of the world’s most innovative and highest-performing load balancing solutions. The load balancer is critical for enabling high availability and supporting the dynamic scaling of infrastructure within modern applications. Because of its importance, engineers need tools that can quickly and effectively diagnose any problems with the load balancer if they arise.

This year’s keynote at the Sumo Logic Modern SOC Summit addressed these questions and more as Greg Martin, Sumo Logic’s General Manager and VP of Security, talked with Byron Acohido, a Pulitzer-prize winning cybersecurity journalist, about some of the challenges and changes facing the modern SOC.

Red Hat OpenShift is an open source container application platform that incorporates a collection of software that enables developers the ability to run an entire Kubernetes environment. It includes streamlined workflows to help teams get to production faster and is tested with dozens of technologies while providing a robust tightly-integrated platform supported over a 9-year lifecycle.

Does the SOC really need to be disrupted? In an EY survey, 59% of enterprises admitted experiencing a material or significant breach. Despite the fact that SOC spend dominates an organization’s cybersecurity budget, more than half of these SOCs were actually ineffective in protecting their organizations from attacks.

We have all heard about Key Performance Indicators (KPIs) and how critical they can be for your security program, but confusion remains surrounding what KPIs are important to track and how they can be used to measure and improve the organization’s security program.

One of the buzzwords when talking about Incident Response is playbooks or advanced workflows with specifically tailored actions. Learn more.

Our mission is to create a force multiplier for SOC teams and security analysts so they can reduce the time to verdict or judgment while triaging new Insights. At Sumo Logic, we take a different approach than other SIEM solutions. We don’t just create alerts and leave the analyst to gather other artifacts to gain context. We associate and group alerts, or what we call Signals, to an Entity (IP, User, Hostname, etc...).

After a couple of hard-working months full of exciting strategic discussions following the acquisition of DFLabs by Sumo Logic that was concluded this May, we are surely moving forward and laying the groundwork for the future of our Cloud SOAR as a part of Sumo Logic’s Modern SOC Strategy.

We all know how our life has changed in the last two decades in terms of how we interact with our vendors for day to day activities, regardless if we shop for a birthday present or read a newspaper. This digital transformation of everything we do increased even more during the pandemic, when it appeared that a malfunctioning digital access channel can often mean lack of possibility to enter certain places like even the country or a plane to get back home.

Given today’s evolving multi-cloud dynamics and increasingly active threat landscapes, security teams have a greater need for integrated and scalable monitoring that provides meaningful real-time insights into the state of organizational security posture.

When it first burst onto the cyber security scene back in 2015, SOAR was dubbed by Gartner as a ground-breaking, revolutionary technology in the cyber security industry. Fast-forward 6 years, Security Orchestration, Automation and Response has lived up to those expectations and is rapidly growing its presence rapidly, with the SOAR market estimated to exceed $550 million by 2023.

What better way is there to get insight into your app performance than by leveraging the traffic management tools connecting your apps to your customers? With Sumo Logic and NGINX, you can easily get information on the health of your apps and save valuable time.

In this blog, we will show you some simple steps for deploying and running a PostgreSQL database on Kubernetes. We will explore a simple use case in which a developer wants to have a single PostgreSQL instance for testing, and then we’ll introduce an advanced use case in which there are a few options for deploying a more configurable instance of PostgreSQL.

With Sumo Logic’s cloud-native Continuous Intelligence Platform and Cloud SIEM, it’s now easier than ever to integrate with Zscaler’s Internet and Private Access platforms to gain visibility across your cloud apps with out-of-the-box content, respond in real time to correlated security incidents and monitor your ZTNA deployment.

In the era of data abundance, there exists a significant need for database systems that can effectively manage large quantities of data. For certain types of applications, an oft-considered option is Apache Cassandra. Like any other piece of software, however, Cassandra has issues that could potentially impact performance. When this happens, it’s critical to know where to look and what to look for in the effort to quickly restore service to an acceptable level.

It’s not every day that you get four CTOs of leading Cloud companies in a discussion about security, the changing role of the security operations center (SOC), and how best to manage data, artificial intelligence(AI), and service providers in these challenging times. To close out the 2021 Modern SOC Summit, Christian Beedgen, Sumo Logic’s CTO, hosted a discussion with Peter Silberman, CTO at Expel.io, Scott Lundgren, CTO at Carbon Black, and Todd Weber, the CTO at Optiv.

In today's environment, security teams face a pervasive threat landscape, with the expectation that some threat actors will be successful in bypassing perimeter defenses. To deal with this, security teams must learn how to actively hunt down threats, both outside and inside the perimeter, using solutions, such as Sumo Logic’s Cloud SIEM Enterprise and Continuous Intelligence Platform.

With a multitude of digital options available in almost every industry, it’s become increasingly critical that applications and services provide a positive user experience. Doing so requires a high level of availability, made possible (in part) by efficiently identifying and resolving issues with the system, when they occur.

The evolving threat landscape puts pressure on traditional SOCs, igniting the genesis of the next-gen, modern SOC.

In one of my previous blogs I explained how important it is for a modern observability platform to provide “the observers” full, flexible access to all raw telemetry. Observability’s promise to find unknown unknowns relied directly on the ability of fast, powerful and multidimensional high-cardinality analysis of raw data, to uncover previously unknown patterns that have not yet been visualized as a metric, dashboard panel or an alert or anomaly event.

In this article we look at how to monitor Cassandra database clusters. We start with the basic architecture of a Cassandra cluster, and mention the most important metrics to gather. Next, we advance step-by-step into configuring and setting up a monitoring stack with Jolokia, Telegraf and Sumo Logic collectors and dashboards – everything you need to monitor Cassandra databases.

Many of our customers today leverage Office 365 GCC High, including organizations looking to meet evolving requirements for working with the United States Department of Defense. Sumo Logic enables customers to leverage our out-of-the-box monitoring and analytics capabilities to analyze Office 365 GCC High data to offer security engineers and security analysts stronger situational awareness of internal employee data.

A common scenario for log analytics is that many log events are high value for real time analytics, but there are also events that are low value for analytics, but account for a very large percentage of overall log volume.

Apache Kafka® is one of the most popular streaming and messaging platforms, commonly used in a pub-sub (publish-subscribe) model, where consumer software applications send data via messages that producer software applications can consume. Teams use Kafka for a variety of use cases, including monitoring user activity, sending notifications, and concurrently processing streams of incoming data such as financial transactions.

It has been off to the races for the Sumo Logic and AWS teams since the general availability of the Sumo Logic Cloud SIEM powered by AWS solution on June 1, 2021. We are excited for the overwhelming response from customers from across all segments, industries and geographies.

The Sumo Logic team is excited to announce that it has been named a Visionary in the Gartner 2021 Magic Quadrant for Security Information Event Management (SIEM). We believe our placement in the Visionary quadrant reflects the value and success our customers have realized by using our cloud-native security platform and the innovative ways in which it solves SIEM and modern security operations use cases.

In this article, we're going to focus on pillar number five: measuring everything. We'll discuss five key metrics for DevOps software pipelines and explain how each of them can help measure software delivery performance.

Security and IT teams may be loath to admit it, but security has historically been mostly a reactive affair. Security engineers monitored for threats and responded when they detected one. They may have also taken steps to harden their systems against breaches, but they didn’t proactively fight the threats themselves.

Platform architects, SREs, developers and DevOps staff for mission-critical modern apps know being notified in real-time when or before critical conditions occur can make a massive difference in end-user digital experiences and in meeting a 99.99% availability objective.

Threat hunting is emerging as a must-have addition to cybersecurity strategies. By enabling organizations to find and mitigate threats before they ever touch their networks or systems, threat hunting provides the basis for a more proactive security posture – and one that delivers higher ROI on security tools and processes.

Companies generate data at an exponential rate, and the task of analyzing data to produce relevant security insights can be overwhelming. With evolving market dynamics and threat landscapes, security teams have a greater need for integrated and scalable monitoring that provides real-time and meaningful insights into the state of organizational security posture.

Organizations that want to do business with the Department of Defense are required to achieve CMMC compliance by the end of this year. Sumo Logic can serve as a tool to assist your organization in achieving CMMC compliance.

In the beginning, there was the Log – or to be a bit more precise, there were application logs. At least that's how it was in the early days of application development, when raw log data itself was more often than not the point where troubleshooting began.

Over the past decade, the way we build and deploy applications has changed dramatically. The explosion of public cloud providers enables us to deploy software without engaging in a drawn-out process to procure and set up infrastructure. Agile, DevOps, Continuous Integration, Continuous Deployment, and other changes to how we work have dramatically accelerated the speed with which we can get new applications and updates in front of our users.

Since 2010, it has been Sumo Logic’s mission to democratize machine data. Naturally, we tend to focus on the outcomes: reliable and secure applications and systems that are the engines of successful modern businesses. But to drive these outcomes, and before the spotlight-hogging analytics kick in, algorithms require data. And this is where the magic starts! Sensu has been working on championing a monitoring as code approach to building observability pipelines for a decade now.

The Sumo Logic team is pleased to introduce the general availability of Sumo Logic Cloud SIEM powered by AWS. This joint solution will empower CIOs, CISOs, security and IT leaders to solve modern and legacy security operations use cases for Enterprises of all sizes and maturity with deep and contextualized insights to reduce the time to detect and respond to threats.

I’m pleased to announce that Sumo Logic has finalized its acquisition of DFLabs S.p.A., a provider of security orchestration, automation and response (SOAR) software.

Sumo Logic is excited to host “The Modern SOC Summit” from June 8–9, 2021. This summit is a two-day virtual event hosted by Sumo Logic that is bringing together CISOs, security and operations practitioners, innovators and business leaders to debate, discuss and share best practices on modernizing security operations for the rapidly evolving threat landscape, growing adoption of cloud-native solutions and future state of security.

Application monitoring is a well-established discipline that dates back decades and remains a pillar of software management strategies today. However, as software environments and architectures have evolved, monitoring techniques have needed to evolve along with them. That’s why many teams today rely on distributed tracing to glean insights that they can’t gather from application monitoring alone. Distributed tracing provides a deeper level of visibility into complex distributed environments than application monitoring can achieve.

It's one thing to detect a cyber attack. It's another to know what the attackers are trying to do, which tactics they are using, and what their next move is likely to be. Without that additional information, it's difficult to defend effectively against an attack. You can't reliably stop an attack if you are unable to put yourself in the mindset of the attackers. This is why threat intelligence plays a critical role in modern cybersecurity operations. Threat intelligence delivers the context about attackers' motives and methods that teams need to react as effectively as possible against threats to their IT resources. Keep reading for a primer on what threat intelligence means, why it's important, and what to consider when implementing a threat intelligence strategy.

As more companies migrate workloads to the public cloud, more security operations teams face the challenge of securing those environments. Although cloud providers make accessing the logging very easy, it is not always easy to digest the mountains of data they provide. One example of this is AWS CloudTrail logging. This service is extremely robust which can lead to quite a bit of noise with basic detections.

Security operations is now a critical business function tasked with securing digital transformation initiatives, to effectively mitigate evolving attacks and expanding attack surfaces, handle complexity and tool proliferation while teams are continuing to be virtual and distributed.

Telegraf is a server-based agent for collecting all kinds of metrics for further processing. It’s a piece of software that you can install anywhere in your infrastructure and it will read metrics from specified sources – typically application logs, events, or data outputs.

In a recent experiment with my colleagues, I polled them about the following: “What would they do if the lights went out as you worked at night?” Besides identifying the funny and who-you-want-in-case-of-an-emergency responses, most of my colleagues checked to see if the problem might be broader than their own home.

When migrating to Kubernetes and re-architecting your applications into containers, logging is a critical piece to consider. The twelve-factor app methodology has a section dedicated to logging and outlines the importance of not worrying about routing and storage of your logs. As a best practice, applications running in containers should rely 100% on standard output (STDOUT). Unfortunately, getting logs from applications that do not write to STDOUT is non-trivial and has many things to consider.

It’s essential to choose the right tool for the job. I have an old, sturdy screwdriver that I use for lots of odd DIY jobs around my house, like cleaning gutters, opening paint cans, and general maintenance on my lawnmower. However, when I’m performing an upgrade on my computer, a large, rusty screwdriver isn’t the best tool to remove the screws anchoring my motherboard.

If your team falls into the majority of organizations that use NGINX – which remains the world’s most popular Web server – to host websites and Web applications, monitoring NGINX usage, performance, and transactions is critical for maintaining a positive end-user experience.

We are excited to join AWS for the launch of Amazon CloudWatch Metric Streams; a fully managed, scalable, and low latency service that streams Amazon CloudWatch metrics to partners via Amazon Kinesis Data Firehose. AWS and Sumo Logic customers can now leverage AWS Kinesis Firehose for Metrics Source for streaming CloudWatch metrics into their Sumo Logic accounts, to help simplify the monitoring and troubleshooting of AWS infrastructure, services, and applications.

Modern systems look very different than they did years ago. For the most part, development organizations have moved away from building traditional monoliths towards the development of containerized applications running across a highly-distributed infrastructure. While this change has made systems inherently more resilient, the increase in overall complexity has made it more important (and more challenging) to effectively identify and address problems at their root cause when issues occur.

Microservices and serverless are both important topics in the world of cloud-native computing. Yet, although serverless functions and microservices architectures often go hand-in-hand, they’re distinct technologies that fill different roles in modern software environments.

Cloud-native and serverless come hand in hand. One of the initial motivations to move business workflows to the cloud was related to cutting costs related to provisioning infrastructure and elasticity that on-demand allocation of resources is offering. The serverless approach takes this to the next level, where infrastructure is provisioned only for the time of code execution, and the whole stack below the executed code, including application components, OS, and hardware (of course) is provided by the cloud vendor. No surprise this approach takes more and more traction, although it’s nothing new.

Over a year ago we decided to invest heavily in Application Observability, understanding the modern observability platform must unite logs, metrics, and traces in one analytics layer to better serve reliability use cases. We have also advocated a modern trend to acquire tracing data via open source industry standards like OpenTelemetry without vendor lock-in.

At Sumo Logic, our belief is that security operations is no longer a human scale problem. We need tools and technologies to aid our defenders and responders to be able to process, investigate and respond at machine speed. Our vision for modernizing security operations to deal with threats at machine scale has always encompassed more than just SIEM.

Monitoring Redis, the open source in-memory data platform, is complicated enough when you are hosting your Redis instance on just a single server. It gets even more complex when you build a Redis cluster that consists of multiple nodes and distribute your data across them.

In a recent press release entitled ‘Sumo Logic Achieves FedRAMP Moderate Authorization’, dated Feb 2, 2021, the pioneer in continuous intelligence announced its Continuous Intelligence Platform™ has achieved Federal Risk and Authorization Management Program (FedRAMP™) Authorization at a Moderate impact level enabling the company to help public sector organizations get real-time insights into their complex on-premises and cloud environments.

We are seeing a renewed focus on accelerating digital transformation projects across business ecosystems and workflows within our customer base. These projects are enabling key business outcomes and this organizational transformation has given security and IT leaders the catalyst and opportunity to modernize security operations while eliminating on-premises debt. Throughout the current challenging environment, Sumo Logic has been helping customers with their digital transformation journey securing cloud-first, cloud migration, cloud rebuild projects, and app modernization efforts. Critical to our customers’ successes has been the adoption of cloud-native solutions such as Sumo Logic’s Cloud SIEM.

With almost every blog you read about monitoring, troubleshooting, or more recently, the observability of modern application stacks, you’ve probably read a statement saying that complexity is growing as a demand for more elasticity increases which makes management of these applications increasingly difficult.

One of the more delicate debates in the DevOps world is what observability has to do with monitoring. Is observability just a trendy buzzword that means the same thing as monitoring? Is observability an improved version of monitoring? Are monitoring and observability different types of processes that solve different problems? The answer to those questions depends in part on your perspective. Let’s take a look at the different ways of thinking about observability and monitoring, and what they have to do with each other.

It’s been almost a year since I shared some thoughts about distributed tracing adoption strategies on this blog. We have discussed how different approaches between log vendors and application performance management (APM) vendors exist in the market and how important that is to allow users to analyze the data, including custom telemetry, the way they want.

Java Management Extensions, or JMX, was first added to J2EE, and it has been part of J2SE since the 5.0 release. The JMX API aims to provide a standard for monitoring and managing Java-enabled applications and services. In this article, we will explain the JMX architecture and show you how to pull the metrics that it generates into your Sumo Logic account in order to gain unique insights and a more thorough understanding of the health of your application and services.

While the COVID-19 pandemic has disrupted business models around the world, the adoption of modern application and cloud technologies continues to grow. This year’s Continuous Intelligence Report by Sumo Logic provides an inside look into the state of the modern application technology stack, including changing trends in cloud and application adoption and usage by customers, and the impact of COVID-19 as an accelerant for digital transformation efforts.

Daemons as scheduled processes are gaining more importance in security operations. Find out what their role is in modern-day SOCs.

Whether it’s Cyber Monday deals or streaming the Super Bowl to millions of users, website’s traffic management & SRE teams often struggle to deliver a reliable user experience.

We are pleased to share with you that the Sumo Logic Continuous Intelligence Platform™ was granted authority to operate at a Federal Risk and Authorization Management Program (FedRAMP) Moderate impact level.

We take an API-first approach to our product at Sumo Logic—everything you can do in the UI can be done via API. I talked about several use cases for automation with Sumo Logic APIs in my presentation at Illuminate 2020. Here’s a recap of what I covered.

At Illuminate, Genesys Cloud principal architect Kal Patel shared how they use Sumo Logic, how they manage it at scale, and some of the lessons they’ve learned in the last year.

At Sumo Logic, we’re dealing with a large amount of data. To help our customers explore the data quickly and effectively, our product lets them write Logs, Metrics, and Tracing queries. One of the challenges we dealt with recently was improving the query building experience in our new, revamped Metrics UI.

Digital business transformation requires a fast-moving, collaborative culture. As companies on this fast track focus on innovation and speed to market, they inherently introduce more risk from the inside. Furthermore, in 2020, remote work became the norm, requiring increased adoption of cloud collaboration technologies. This shift caused a sudden acceleration of insider risk like we’ve never seen before.

If you’re a product on the web, you have less than a second to impress your customers. Web application performance is crucial to the success of online products. Using a Content Delivery Network (CDN) like Cloudflare can provide just the edge you need.

Dashboard (New) offers a host of new visualization types, like honeycombs, time series, combination graphs, maps and more. With customizable templated dashboards, observing your system is easier than ever before. Dark theme is designed to help you do all of these actions comfortably by reducing strain to your eyes in low light environments. So you can continue to monitor and troubleshoot your systems without hurting your eyes.

At Illuminate 2020, Kashif Iqbal, Head of Corporate Technology and Cyber Security at SEGA Europe, shared their data security challenges and how they currently leverage Sumo Logic for their security needs.

One of Amazon Web Services’ (AWS) most well-known services is AWS DynamoDB. Some of AWS’s most notable customers use DynamoDB for their database needs – companies such as Netflix, The Pokemon Company, and Snapchat.

Open source has come a long way. One of my favorite reports on the subject is Red Hat’s State of Enterprise Open Source. For 2020, 95% of respondents said that open source is strategically important to their business needs. Here, I will be recapping my recent Illuminate presentation about embracing open source data collection and I thought it’s important to first talk about how open source has changed.

In the wake of the widely publicized FireEye breach and the alarming SolarWinds supply chain attack, this presents an ideal opportunity for reflection on the broader shift taking place across the world—the transition from legacy on-prem infrastructures to the cloud. When a sophisticated nation-state obtains utilities intended for Red Team activities-- all but assuring nefarious intent-- it should give our community pause as to the collective state of security readiness should an attacker leverage such tools.

At Sumo Logic Illuminate 2020, The Energy Authority’s IT Director for Service Delivery and Support Scott Follick spoke about the TEA, how they do things, what they have learned in the process of searching for a SIEM solution, and why they chose Sumo.

During Sumo Logic’s Illuminate user conference, Heath Hendrickson, senior security architect at the Clorox company, and Gary Conner, senior threat protection lead, presented how they are leveraging Sumo Logic across security operations, threat hunting, IT operations, and more. 

Aurora, a hosted relational database service available on the Amazon cloud, is a popular solution for teams that want to be able to work with tooling that is compatible with MySQL and PostgreSQL without running an actual MySQL or PostgreSQL database.

At the recent Illuminate conference, Dave Sudia from GoSpotCheck shared the process that led them to choosing Sumo Logic for their observability needs.

The global security community recently learned of a supply chain attack against SolarWinds via their Orion® Platform. In this blog we are providing recommendations for Sumo Logic customers to gain a deeper understanding of how to utilize available Indicators of Compromise (IOCs) within our Cloud SIEM offerings to determine your exposure to the attack. Additionally, we’re sharing targeted search recommendations from our Sumo Logic Special Operations (or SpecOps) threat hunting team.

Sumo Logic has reviewed the announced breach on December 8, 2020 by FireEye and their subsequent public release of over 300 countermeasure rules. We are continuing to analyze the available information and would like to share this update to all existing and prospective customers interested in how our Sumo Logic services can assist with this development.

Distributed tracing allows you to track the execution of your user's transactions by following them between applications’ microservices. It provides easy to understand visualizations of transactional lifespan with the ability to pinpoint any slowdowns and errors in response to microservices.

Amazon Redshift is a cloud-based data warehousing solution that makes it easy to collect and analyze large quantities of data within the cloud. Cloud data warehouse services like Redshift can remove some of the performance and availability pain-points associated with on-premises data warehousing, but they are not a silver bullet. Getting the most out of Redshift requires carefully monitoring Redshift clusters in order to identify stability issues and performance bottlenecks.

SIEM has traditionally earned itself a bad reputation as an unwieldy and unmanageable tool that really never lived up to its promises. In my presentation during Illuminate, I talked about what Sumo Logic is doing to modernize log analytics and SIEM as a whole.

Observability is arguably the tech buzzword of the year. Whether or not you believe the hype, observability is all about how to ensure overall system health and deliver reliable customer experiences. This is done by observing the system, and when a problem arises, using real-time analytics to quickly help identify the what, where, and why of the problem.

For decades, Microsoft SQL has been a leading relational database solution within Windows-based environments. The extension of Microsoft SQL support to Linux servers in 2017 made the platform even more popular. There’s a good chance that, no matter which types of infrastructure or servers you manage, there are Microsoft SQL databases residing somewhere on them.

We at Sumo Logic believe in an open, flexible, community-driven approach to collecting observability data. Those reasons are outlined in one of my recent blogs. In that publication, I share the belief that an application’s observability gains traction from the fact that telemetry signals are designed, composed, and produced by an application developer/vendor in compliance with industry standards, and are not a proprietary, black box component of the monitoring vendor.

Modern applications are often built, deployed, and managed as container images. At the same time, however, developers are drawn to serverless technologies for running their code without having to worry about infrastructure. AWS is now helping you bridge the gap between these two paradigms by allowing you to build and deploy serverless AWS Lambda functions with container images. This allows you to harness the power of AWS Lambda without having to rewrite or modify your development workflows.

Today Amazon announced Amazon EKS Distro, a distribution for Kubernetes based on and used by Amazon EKS. Amazon EKS Distro enables you to create secure and reliable Kubernetes clusters using the same versions of Kubernetes and its dependencies deployed by Amazon EKS. Each Amazon EKS Distro release follows the EKS process, verifying new Kubernetes versions for compatibility. The Amazon EKS Distro source code, open source tooling, binaries, and container images as well as configuration are provided for reproducible builds via public Git and S3 storage locations.

Are you ready for Security Orchestration, Automation and Response? Find out the clear signs that’ll tell you if you are mature for SOAR.

As cloud applications and services become more and more common amongst organizations, adversaries will continue to evolve their toolset to target and penetrate cloud networks. With the rise in remote employees and teleconferencing, cloud computing for organizations has never been so important.

For five years, Sumo Logic has created the Continuous Intelligence Report on the state of modern apps and DevSecOps. New to this year’s report are unique insights into how enterprises are adapting to the COVID-19 pandemic—particularly in terms of security threats.

We’re happy to partner with AWS on their launch of AWS Network Firewall by providing a cloud-native integration that gives customers real-time visibility into network traffic and automated correlated events surfaced by AWS. Too often, virtual private cloud (VPC) traffic is a black box leaving many security operations teams unable to connect potential threats to their broader infrastructure. This lack of visibility only serves to increase the time it takes security analysts to investigate and respond to potential threats, thereby reducing the organization’s security posture.

Some of the largest and most prominent websites, worldwide, partner with Akamai to provide their customers with a premium web experience. What benefits could Akamai offer to you and your online product offerings? In this article, we will introduce you to Akamai and the services they provide.

Data security in healthcare institutions is a major concern. SOAR may be the way to prevent data breaches for healthcare organizations.

At Sumo Logic’s 4th annual Illuminate user conference, our CCO Aaron Feigin sat down with Lisa Hammit, an expert in artificial intelligence, big data and executive leadership, to discuss how fast things are changing within boardrooms of global business.

Cathy O’Neil found refuge in mathematics, despite being told by teachers (many of whom were women) that she didn’t need to do math because she was a girl.

MongoDB is a leading NoSQL database that has proven itself through time as a stable and easy-to-use system of great flexibility. There are numerous tutorials and real-world success stories that show how MongoDB has helped with supporting the database needs of certain organizations and projects. It’s fair to say that using MongoDB proves to be a solid choice for integrating with web-based applications and microservices serving as a database system or even as a monitoring aggregator itself.

What happens when your Medicaid is revoked while you’re in a hospital being treated for cancer? Or what if the state government constantly threatens to take your kids away because you can’t afford antibiotics? Or what if you lose your job and your home and can’t qualify for public assistance?

Cybersecurity, specifically IoT security, is becoming a critical component for every healthcare institution.

Building an effective security strategy in organizations today requires the right combination of experts, processes, tools and technologies. Luckily, there are many different ways in which you can organize them to fit your company’s needs, like SOCs and CSIRTs.

SOAR is the future of cybersecurity, but in order to make the most out of your SOAR platform, you need to learn how to maximize your SOAR ROI.

Log analysis helps organizations determine the best way to optimize application functionality while giving development teams a leg up in root cause analysis. With that said, it’s not feasible to scroll through thousands of lines of log entries in a text editor. Instead, development teams need modern tools that enable them to centralize, filter, and analyze their logs in a way that allows them to glean valuable insights in a time-efficient manner.

This week Sumo Logic announced our new Observability Suite, which included the general availability of our distributed tracing capabilities as part of our Microservices Observability solution. This new solution provides end-to-end visibility into user transactions across services, as well as seamless integration into performance metrics and logs to accelerate issue resolution and root-cause analysis. In this blog, we’ll explore the new solution in detail.

The terms security automation and security orchestration are often used almost interchangeably nowadays in the IT ecosystem. But it’s very important to note that these terms have completely different meanings and purposes. The aim of this blog is to discuss the core differences by explaining what these terms mean exactly, what their functions are, and how they can be used within an IT context.

As digital transformation accelerates across companies of all sizes and industries, knowing in real-time the what, where and why of the issues that affect your digital customer experiences will make or break whether they remain reliable, performant and secure.

The National Cyber Security Awareness month in 2020 has arrived, and we prepared a special blog post that puts special attention on one particularly important technology - SOAR.

Platform architects, SREs, developers and DevOps staff for mission-critical modern apps know shaving 15 minutes off service incidents that take 20 minutes to resolve, four times a year, is the difference between meeting a 99.99% availability objective and missing it. After a successful worldwide preview, Sumo Logic Observability is now ready for site reliability, DevOps, developers and platform engineers to resolve incidents faster, maximize availability, and optimize their cloud infrastructure, microservices, and application operations for reliability objectives.

As digital transformation accelerates across all industries, it has become increasingly important for businesses to get better, much better at the development and delivery of new software to drive revenue and engage customers in new ways. Ironically, most businesses are flying blind in how they track, benchmark and optimize their software development.

We are in the midst of an unprecedented convergence of events that are forcing enterprises to dramatically change how they secure their modern businesses. With the acceleration of digital transformation from COVID-19, work-from-home initiatives, the continued growth of SaaS and the increasing adoption of microservices-based applications, the modern enterprise threat landscaping is transforming rapidly. In fact, according to Gartner 69% of Board of Directors (BoDs) have accelerated their digital business initiatives as a result of the pandemic.

Since its inception last year, the Sumo Logic App Intelligence Partner Program has seen fantastic growth via new, innovative integrations built by our growing partner ecosystem that enable joint customers to extract greater visibility, intelligence and value from their data.

Despite all of the hype about NoSQL databases, MySQL and PostgreSQL are two database management systems that have truly stood the test of time. These systems were developed back in the mid 90’s, and they still consistently rank among the top 5 most popular databases across the internet. So why are they so widely used and how do they compare? Read on to find out!

Gartner’s 2020 SOAR market guide is out. Read our blog post for a quick recap.

Kubernetes, as a platform, is a comprehensive set of tools for orchestrating containers at scale. It consists of a modular architecture of specific components with a defined purpose. For example, the scheduler finds the ideal match for a particular pod and the kube-proxy manages the networking between the nodes and the master.

Log analyzers are software that helps contextualize massive amounts of log data. Log analyzers reduce the amount of time that it takes to perform root cause analysis, and they provide development organizations with valuable insights that help them improve their products for their end users.

When used properly, SOAR can do wonders for an organization’s cybersecurity and take their SecOps to the next level.

With the growing popularity of cloud computing, security incidents related to it have been on the rise. Logs are indispensable resources for countering these threats, and they can be utilized for alerting, taking remedial action, and even preventing future attacks. In this post, we will examine ways to better secure cloud-native applications using logs.

We’ve been witnessing the continual transformation of the cybersecurity ecosystem in the past few years. With cyber attacks becoming ever-more sophisticated, organizations have been forced to spend huge amounts of their budgets on improving their security programs in an attempt to protect their infrastructure, corporate assets, and their brand reputation from potential hackers.

Kubernetes is first and foremost an orchestration engine that has well-defined interfaces that allows for a wide variety of plugins and integrations to make it the industry leading platform in the battle to run the world’s workloads. From machine learning to running the applications a restaurant needs, Kubernetes has proven it can run things.

Brief introduction to understanding Kubernetes basics. Kubernetes is a broad platform that consists of more than a dozen different tools and components.

“Americans have their minds wrapped around a two-party system. It is hard to get people to envision something different — despite the fact that there have been tectonic changes in the American political parties at many different junctures in our history. Building a new political party from scratch feels daunting and naïve.”

Founded in 2007, Goibibo is an online travel organization focused on getting the best deals for its customers on all online travel bookings—including hotels, flights, buses, trains, and cabs.

SOAR is one of the hottest cybersecurity prospects, but many still struggle to understand the essence of the SOAR definition.

The OpenTelemetry Collector is a new, vendor-agnostic agent that can receive and send metrics and traces of many formats. It is a powerful tool in a cloud-native observability stack, especially when you have apps using multiple distributed tracing formats, like Zipkin and Jaeger; or, you want to send data to multiple backends like an in-house solution and a vendor. This article will walk you through configuring and deploying the OpenTelemetry Collector for such scenarios.

The SOAR industry is becoming more dense and diverse than ever before, and choosing the best SOAR platform out there comes down to understanding what to look for in a top SOAR solution.

Data increasingly means “people,” down to the preferences of each consumer and voter. That’s why data is so valuable to political campaigns. With good data, you can predict things that haven’t happened yet.

Integrating lessons learned into existing and future incident response playbooks ensures that the proper technologies and processes are deployed.

“Kubernetes vs. Docker” is a phrase that you hear more and more these days as Kubernetes becomes ever more popular as a container orchestration solution.

Creating a modern security operations center (SOC) requires so much more than getting the right equipment and hiring experts. In fact, it’s a constant process of keeping up with countless security threats, staying up to date with developing trends and technologies, recruiting and keeping skilled professionals, while also providing them with an environment they can thrive in. All this, while aiming to improve the overall organization’s security posture and performance.

Security incidents and digital forensics investigations are complex events with many facets, all of which must be managed in parallel to ensure efficiency and effectiveness. When investigations are not managed and documented properly, processes fail, critical items are overlooked, inefficiencies develop, and key indicators are missed, all leading to increased potential risks and losses. Investigation management can be broken down into a number of key components and it is important that an organization is able to carry out all of these elements collectively and seamlessly in order to properly handle and manage any incident they may potentially face.

Cloud SOAR is an invaluable asset for every university and college, allowing them to boost their cybersecurity and eliminate even the most sophisticated cyber threats.

SANS recently released their 2018 SOC Survey and many of their findings were of no surprise to anyone who has been responsible for maintaining their organization’s security posture. Many respondents reported a continued breakdown in communication between NOC and SOC operations, lack of dynamic asset discovery procedures, and event correlation continues to be a manual process even though SOC staffing is being worn thin by the surmounting responsibilities they have to take on.

Higher education institutions are often targeted by vicious cyber attacks. In this blog post, we will discuss the main cyber challenges universities and colleges face in today’s digital world.

The countdown is on to our 4th annual Illuminate user conference October 6-7, 2020! This year we are going virtual to keep everyone healthy and safe, and while we will miss seeing all of our customers and partners, we are excited to host the premier education platform for machine data analytics to help businesses accelerate digital transformation and customer experiences.

When it comes to Security Orchestration, Automation and Response (SOAR), the use cases will vary depending on a number of factors, such as the enterprise-specific internal environment, the industry or vertical the enterprises serve, and even the legal and regulatory compliance that need to be met.

Security automation is slowly making strides in the cybersecurity industry. Read on to find out how progressive security automation is going to affect the cyber world.

Logging as a Service is a simple, highly scalable solution for managing all of your logs in a central, systematic way. With Logging as a Service, a cloud-based log aggregation tool collects log data from across your infrastructure, consolidates it in a central location, and provides the analytics and visualization tools you need to understand the data in your logs.

Persistence is effectively the ability of the attacker to maintain access to a compromised host through intermittent network access, system reboots, and (to a certain degree) remediation activities. The ability of an attacker to compromise a system or network and successfully carry out their objectives typically relies on their ability to maintain some sort of persistence on the target system/network.

Many MSSPs are failing to respond to the evolving threat landscape, which makes their traditional security services redundant. Learn how SOAR helps MSSPs boost the efficiency of their security services.

As more and more applications move to the cloud, the complexity of application architectures inevitably increases. It is a burden we willingly take on because the benefits—flexible deployment, technology diversity, independent scaling, and much more— tend to far outweigh the costs. But along this transition, most organizations face a dilemma, to divert resources to the necessary tooling for effective monitoring and troubleshooting of these systems – i.e. observability – or slow the rate of migration to the cloud.

Today, we made an important announcement that builds on and extends our category leadership in Continuous Intelligence. This blog is to put that announcement into context for our customers, partners and the broader industry.

Automation is a key component in the management of the entire software release lifecycle. While we know it is critical to the Continuous Integration/Continuous Delivery process, it is now becoming equally essential to the underlying infrastructure you depend on. As automation has increased, a new principle for managing infrastructure has emerged to prevent environment drift and ensure your infrastructure is consistently and reliably provisioned.

Sumo Logic is taking the next step towards Moderate authorization under the Federal Risk and Authorization Management Program (FedRAMP). Today Sumo moved from “Ready” to “In Process” on the FedRAMP Marketplace, an important step toward expanding our ability to serve U.S. governmental agencies.

In our previous two blogs, we looked at some of the most common problems a Security Orchestration, Automation and Response (SOAR) Technology is designed to solve and the three pillars of a SOAR solution. We will round out this three-part series by taking a more detailed look at some of the most critical SOAR Technology components any SOAR solution should possess. While some of these components may be more critical than others to individual organizations, each plays an important role in the overall function of a SOAR solution and should be considered when evaluating different platforms.

I am spending a considerable amount of time recently on distributed tracing topics. In my previous blog, I discussed different pros and cons of various approaches to collecting distributed tracing data. Right now I would like to draw your attention to the analysis back-end: what does it take to be good at analyzing transaction traces? As mentioned in the blog above, one of the most important outcomes of adopting open source tracing standards is a freedom to choose the right analysis backend, as long as it supports these standards. So, what is the requirement list for a distributed tracing backend? What should it do and what are absolute must-haves? We have looked at many free, open source and commercial offerings on the market and found a few tools that are good here or there, but nothing would fully match a complete list.

There has been increasing buzz in the past decade about the benefits of using a microservice architecture. Let’s explore what microservices are and are not, as well as contrast them with traditional monolithic applications. We’ll discuss the benefits of using a microservices-based architecture and the effort and planning that are required to transition from a monolithic architecture to a microservices architecture.

DFLabs (now Sumo Logic) will be included in the Gartner Market Guide for the second year in a row. Read this blog to find out what to expect in the Gartner Magic Quadrant and in the Gartner Market Guide for SOAR 2020.

Today’s organizations have the challenge of managing several different applications and software within their technology stack. The more public-facing platforms an organization utilizes, the greater their public attack surface risks. Without proper protection, they and their community can become an easy target for malicious actors.

In our first blog in this series, we looked at some of the key drivers for Security Orchestration, Automation and Response (SOAR) adoption and what problems SOAR technology can help solve. Now, let’s look at the 3 core pillars which define what a SOAR solution is: Orchestration, Automation and Measurement.

We’re excited to announce the first version release of our new dashboard framework: Dashboard (New). Built on top of a scalable, flexible, and extensible charting system, the new dashboards provide customers with deep control over their visuals, enable metadata rich workflows, and create dashboards in a dashboard first GUI.

Compared to even just a few years ago, the tools available for data scientists and machine learning engineers today are of remarkable variety and ease of use. However, the availability and sophistication of such tools belies the ongoing challenges in implementing end-to-end data analytics use cases in the enterprise and in production.

Gartner dubs SOAR a revolutionary technology with the potential of changing the entire trajectory of the cybersecurity industry. Read this blog post to find out more about the future of SOAR, as predicted by Gartner.

Many of the organizations use AWS as their cloud infrastructure, and in general they have multiple AWS accounts for production, staging, and development. Inevitably, this would result in losing track of various experimental AWS resources instantiated by your developers. Eventually, you will be paying AWS bills for resources which could have been identified and deleted in time.

Customers regularly ask me what types of data sources they should be sending to their SIEMs to get the most value out of the solution. The driver for these conversations is often because the customers have been locked into a SIEM product where they have to pay more for consumption. More log data equals more money and, as a result, enterprises have to make a difficult choice around what log sources and data are what they guess is the most important. This often leads to blind spots from a logging perspective and requires that your analysts pivot to other tools and consoles to get any additional context and detail they can during an investigation.

Ever since JASK was founded, we have heavily integrated with threat intelligence platforms to gain context into attacker activity through indicators of compromise (IOCs). Now that we have joined Sumo Logic, our customers have the ability to pull in more data than ever making this feature even more powerful.

SOC teams are struggling to balance too many tools and too much data which cripples their ability to properly communicate. In this blog post, we explain how SOAR improves the collaboration of the entire security platform.

It has never been a more challenging (or better) time to be a service provider for managed security services. With an estimated 1,200+ vendors selling a variety of security solutions today, businesses are looking for help to manage the complexities of supporting these technologies while protecting critical data. According to Gartner, the managed security service (MSS) market is expected to grow to nearly $50 billion by 2023, and last year 32% of organizations increased their use of outside services due to shortages in available resources.

As more and more enterprises shift to the cloud, the pressure on SOC teams to protect them against threats rises exponentially. They are the very first line of defense against data breaches and cyber threats that become more frequent and more sophisticated. Increased investment of security tools results in unprecedented volumes of security data and alerts, and while SOCs do what they can to decipher the meaningful from the meaningless, they often become the bottlenecks of the enterprise’s security architecture.

Monitoring requires a multi-faceted approach if DevOps teams want end-to-end visibility and deep insight into issues. This is especially true in the case of modern microservices applications, which are essentially collections of distributed services that talk to each other over a service mesh.

Unless you’ve been living under a rock you are probably familiar with the recent Shadow Brokers data dump of the Equation Group tools. In that release a precision SMB backdoor was included called Double Pulsar. This backdoor is implemented by exploiting the recently patched Windows vulnerability: CVE-2017-0143.

Read our latest blog and find out how SOAR extends the limits of what your SOC team previously thought was capable of achieving.

Throughout the history of software development, one statement has remained true: no application is perfect. Due to that fact, development organizations must work with all resources at their disposal to limit the impact that application problems have on the end-user.

A number of domain “forgeries” or tricky, translated look-alikes have been observed recently. These attack campaigns cleverly abuse International Domain Names (IDN) which, once translated into ASCII in a standard browser, result in the appearance of a corporate or organization name that allows the targeting of such organization’s domains for impersonation or hijacking. This attack has been researched and defined in past campaigns as an IDN homograph attack.

Today’s modern deployment pipeline is arguably one of the most important aspects of an organization’s infrastructure. The ability to take source code and turn it into a production application that’s scalable, reliable and highly available has become an enormous undertaking due to the pervasiveness of modern application architectures, multi- or hybrid-cloud deployment strategies, container orchestration and the leftward movement of security into the pipeline.

The ability of an actor to remain undiscovered or obfuscating its doings when driving a malicious campaign usually affects the gains of such campaigns. These gains can be measured in different items such as time to allow completion of operations (exfiltration, movement of compromised data), ability to remain operative before take down notices are issued, or ability to obtain gains based on for-profit driven crimeware (DDoS for hire, Crypto mining).

I have often talked about the benefits of employing flexible playbooks to deal with evolving cyber incidents and unique threat scenarios, and in these series of blogs, I am going to explore some of the points of emphasis when creating a new playbook.

Edge computing is likely the most interesting section of the broader world of IoT. If IoT is about connecting all the devices to the Internet, edge computing is about giving more processing power to devices at the edge. Edge computing views these edge devices as mini clouds or mini data centers. They each have their own mini servers, mini networking, mini storage, apps running on top of this infrastructure, and endpoint devices. Rather than sending data to the cloud for processing and receiving already-processed data from a central hub in the cloud, in edge computing all the processing happens on the edge device itself, or close to the edge device.

Our digital surface is expanding rapidly and threats are becoming more sophisticated day by day. This is putting enormous strain on security teams, which have already been stretched to the limits. Nonetheless, organizations are skeptical of relieving this cybersecurity strain with AI and automation. Why does this situation persist when it’s simply against the logic?

Implementing standard operating procedures is a crucial aspect of running successful cybersecurity operations. Read this blog post and find out how to implement SOAR to enhance your SOP.

Implementing Standard Operating Procedures (SOP) is one of the essential steps towards ensuring a more streamlined and effective incident response process.

A type of credential reuse attack known as credential stuffing has been recently observed in higher numbers towards industry verticals. Credential stuffing is the process of automated probing of and access to online services using credentials usually coming from data breaches, or bought in the criminal underground.

An ever-increasing number of organizations are working in the cloud. It depends on their business model what cloud delivery model they use. The three most common deployment models for cloud services are software-as-a-service (SaaS), platform-as-a-service (PaaS) and infrastructure-as-a-Service (IaaS).

Machine Learning (ML) and Artificial Intelligence (AI) have been buzzwords in the cybersecurity industry for quite some time now, and many still confuse them as being the same thing. This blog post aims to clear up the confusion around these two terms, as well as to provide some insights into their importance in the cybersecurity world.

SOAR is an exciting technology that plays a critical role in improving security operations, and in that regard, security operating procedures as well.

At Sumo Logic, we manage petabytes of unstructured log data as part of our core log search and analytics offering. Multiple terabytes of data are indexed every day and stored persistently in AWS S3. When a query is executed against this data via UI, API, scheduled search or pre-installed apps, the indexed files are retrieved from S3 and cached in a custom read-through cache for these AWS S3 objects.

A question we often receive from users new to Sumo Logic’s Cloud SOAR solution is “What is the difference between a playbook and a runbook?” Many professionals within the cybersecurity industry use these terms interchangeably which often leads to confusion when both are being used.

Amazon EC2 offers a flexible and convenient way to run virtual machines in the cloud. With dozens of EC2 instance types available, as well as multiple pricing options, it’s easy to use EC2 to configure the best cloud-based virtual machines for your needs and budget.

Implementing and operationalizing the best practices and capabilities of DevOps into an organization is a key predictor for increased customer satisfaction, organizational productivity and profitability. Doing so successfully can be a challenging endeavour. Implementing DevOps can be particularly difficult because it oftentimes requires technology changes, process changes and a drastic change in mindset. Overcoming all three of these obstacles in a way that knocks down traditional barriers across development and operations teams in each stage of the software delivery lifecycle raises the bar even further.

MySQL has been one of the leading open source databases for the last couple of decades, and it underpins potentially millions of applications, from tiny prototypes to internet-scale ecommerce solutions. The beauty of MySQL is that it can be tuned as the application grows. For example, you can add higher availability options like clustering without having to refactor the application.

SOAR is continuously climbing higher on the ladder of cybersecurity, and while many are already familiar with the endless potential of this technology, there are those that are still skeptical about the revolutionary capabilities offered by SOAR.

Logs are valuable. Logs generated by a major backend resource that provides clients with access to crucial data are more than just valuable; knowing where they are and being able to manage and understand the information that they contain can mean the difference between smooth, secure operation and degraded performance or even catastrophic failure for your application.

As the shift to cloud, modern app architectures and technology stacks continue to accelerate, the demand for real-time analytics to monitor, troubleshoot, secure and speed new innovations to these environments is also accelerating. So, we're not surprised to see demand for continuous intelligence—what we define as: real-time analytics from a cloud-native platform, supporting multiple use cases—is also accelerating.

As service architectures have transitioned from the monolith to microservices, one of the tougher problems that organizations have had to solve is service discovery and load balancing. The advent of service mesh technologies seeks to solve these and other problems exacerbated by the number of hosts that has grown exponentially.

In this post, we continue our discussion of use cases involving account take over and credential access in enterprise data sets. In the first part of this series, we introduced the definition of a VIP account as any account that has privileged or root level access to systems/services. These VIP accounts are important to monitor for changes in behavior, particularly because they have critical access to key parts of the enterprise. As a follow up to our first post, this blog will describe a real-time approach for automatically profiling VIP accounts and detecting when they are potentially being misused.

Incident response has become a crucial component in the information security management process of every well-prepared organization. Performing effective incident response is a complex operation that requires resources and planning. Forensically sound preservation of evidence can also easily be overlooked by an incident responder due to the rapid and changing nature of the incident being investigated, which can potentially have a huge impact on the effective post-incident analysis and potential legal position.

At its core, DevOps is a set of principles. There are certain types of tools and processes that reinforce those principles and put them into practice. But the tools and processes are just a means to an end. The end itself is DevOps principles.

DevOps is an approach to software development and delivery that emphasizes collaboration between different stakeholders. DevOps also places priority on automation and the integration of different tools in a single, well-oiled pipeline.

The OT-IT convergence has a lot of benefits, but it also creates cybersecurity gaps. Read this blog post and find out how to manage IT and OT and strengthen your cybersecurity with Cloud SOAR.

NoSQL technology has become more popular in recent years thanks to the development of new open-source NoSQL databases that are relatively easy to install, use and integrate with web frameworks. An example of one of those popular frameworks on the internet is known as MEAN (MongoDb, Express.js, Angular.js, Node.js).

A new kind of spam is being observed in the field that uses the browser notification feature to trick users into subscribing to sites that will in turn bombard users with notifications usually related to click or add profit schemes.

Here at Sumo Logic, we share a lot of thoughts about managing data at scale, and the innovative ways we help customers address their unique use cases. It’s not just about analysis of logs.

Detecting malicious activity is rarely easy, but some attacker methods are more challenging to detect than others. One of the most vexing techniques to counter is credential theft. Attackers that gain control over a user account have access to the assets of that user.

In the fast-moving and demanding world of software development, it is critical for organizations to utilize solutions that help streamline server setup in a scalable and affordable manner. Serverless computing solutions were developed in response to these requirements.

EDR offers enhanced endpoint protection, but the technology itself has limitations. Find how to boost EDR’s significance in SOC operations by combining SOAR with EDR.

The energy industry used to operate on a simple hub-and-spoke model, in which large power plants would produce energy in a centralized location and distribute it out to consumers. Yet as solar, wind, and other small-scale renewable energy sources take hold in the market, that hub-and-spoke model is being replaced by a complex grid of interconnected devices.

AWS Lambda is the leading serverless computing solution and is one of AWS’ most successful products, to date. Its popularity is in large part because of the way it makes development easier and faster. Lambda completely abstracts away the maintenance of underlying infrastructure including compute, storage, memory, and networking.

The cybercrime threatscape is constantly changing as hackers adapt and repurpose the use of many different types of tools and attack vectors, and a recent report by Kaspersky Lab indicates that the use of remote administration tools (RATs) has increased during 2018.

When we talk about cybersecurity, having clear goals and objectives in place is key in determining the success of the tools, processes, and management techniques used to combat threat actors. With the right cybersecurity solutions in place, your organization may be able to detect and respond to an incident, or even prevent a cyber attack before it takes place, but let’s not forget, in today’s continually evolving threat landscape there is no such thing as 100% security.

PSPs often struggle to comply with the regulations imposed by PSD2. That’s why they must implement SOAR to improve the effectiveness of their SOCs.

Since various versions of lockdown around the world kicked in, we’ve all been forced to learn how to live as if we’ve just awoke from a deep stasis into a new reality in which everything on earth works differently. It was nearly that abrupt. And if there was ever evidence of how business adaptability is related to business success, it is now.

After StatsD and Graphite weren’t able to meet their needs for metrics and monitoring, engineers at SoundCloud developed the open source event monitoring and alerting tool, Prometheus. Because it’s easy to deploy and get started with -- and on the surface seems free -- it’s become a popular part of many DevOps teams' observability stack.

Introducing our new Cloud Flex Credits licensing model and Infrequent data tier to provide customers unparalleled flexibility and value for their Continuous Intelligence solution.

In this article I’m going to show you three quick and easy ways to enrich your AWS log data in Sumo Logic using fields.

The financial sector remains a highly targeted area for fraudsters and hackers. Learn how to set a proper anti-fraud strategy by implementing a SOAR solution.

The increase in the number and complexity of cybersecurity threats and attacks in the last several years is continuing to heavily influence enterprise security decisions. As well as seeing the growing business need, the significant benefit that Security Orchestration, Automation and Response (SOAR) technology can offer security operations and incident response teams is now truly being realized.

We’re excited to share that our Sumo Logic Continuous Intelligence Platform™ was recently recognized as the Data Analytics Solution of the Year by Data Breakthrough. We join an impressive list of innovative solutions and companies that are solving complex and critical problems and disrupting new markets and industries.

Learn how to optimally secure a remote working environment and prevent cyber threats by implementing a SOAR solution.

Data classification can be broadly defined as the process of organizing and tagging data by categories so that collected data may be used and protected in the most efficient way possible. Sumo Logic is an analytics platform that can ingest almost any type of machine data.

The coronavirus pandemic is one tough test for business leaders. It disrupted all routines and scrapped most of the business plans for the coming months, or even years. Our lives have changed dramatically and no longer is it business as usual. Businesses have to adapt to the change and they need to do it with agility.

PowerShell had its beginnings as a way to enable administrators to perform their tasks both locally and remotely with unprecedented access to underlying Windows components, such as COM objects and WMI. Since being included in every major Windows Operating System since Windows 7, PowerShell based tooling is well proliferated for both legitimate and malicious use and includes common tooling such as SharpSploit, PowerSploit, PowerShell Empire, Nishang and Invoke-Obfuscation.

Observability has become one of the most important areas of your application and infrastructure landscape, and the market has an abundance of tools available that seem to do what you need. In reality, however, most products – especially leading open-source based products – were created to solve a single problem extremely well, and have added additional supporting functionality to become a more robust solution; but the non-core functionality is rarely best of breed. Examples of these are Prometheus and Grafana.

We are excited to announce a new solution for our customers to monitor the performance, availability, and security of their Zoom video conferencing service. The Sumo Logic for Zoom app is available today in our app catalog.

As a system administrator, you will need to monitor your containers for a wide array of reasons. Between analyzing the health of your containers, avoiding resource constraints, and collecting, parsing, and visualizing data, one could easily get lost.

Every year security operations teams continue to report that one of the biggest obstacles they face when trying to protect their organization is the lack of qualified security staff. With the security threats targeting these organizations continuing to grow in sophistication, this deficiency has evolved from an inconvenience to a full-blown epidemic.

As more than 90% of Americans were forced to work from home in March and started to increasingly rely on Zoom for work-related meetings, it has been pointed out that the communication platform didn’t use end-to-end encryption. Read the main security concerns and reliability tips with Zoom.

What a world! In February, everyone was busy minding their own business, but since March, the entire globe suddenly focused on the same challenge. The COVID-19 pandemic has taken our businesses and private lives by storm.

Although the exact features and capabilities of Security Orchestration Automation and Response (SOAR) platforms on the market will vary from vendor to vendor, there are certain core features and capabilities that they should all inherently possess. Depending on the unique set of challenges and pain points that your organization is facing, some functions might prove more important than others in trying to achieve your overall security program objectives. This blog post will briefly explain the four core functions of SOAR technology with the aim of helping you to align them with your organization’s security goals.

SOAR has established itself as a vital solution in the cybersecurity world, but what does the future hold for SOAR? Read this blog post to find out.

Cien años de Soledad? Do not want! And yet, for all of us, the world has changed pretty dramatically over the last few weeks. We could debate over whether this should, in fact, have been the last few months instead, but looking backward at this point isn’t really pragmatic.

In today’s digital economy, the pressure to develop and release new software has become paramount to deliver great customer experiences and create a competitive advantage. Unfortunately, the process by which to deliver this software is increasingly complex and highly fragmented across various tools and teams. While this complexity increases, there is also no easy way to centrally measure, monitor and secure the overall performance of the software delivery pipeline in real-time.

How do SOAR, SOCs, and SecOps work together? In our latest blog post, we unravel the bond between SOAR, SOCs, and SecOps in security operations.

Alcide recently introduced Alcide kAudit, an automatic tool for analyzing Kubernetes Audit logs. This tool focuses on detecting non-compliant and anomalous behavior of users, automated service accounts and suspicious administration operations.

Today, we are facing an unprecedented situation. The COVID-19 pandemic is affecting everything we know -- our families, our businesses, our communities, and our way of life.

The tectonic shift happening within the public sector is seeing more and more federal organizations transitioning from legacy, on-premises systems to more scalable and secure cloud-based architectures. Sumo Logic’s cloud-first approach is a perfect fit for this so we’re excited to announce Sumo Logic has been prioritized by FedRAMP to work with the Joint Authorization Board (JAB) to achieve a Provisional Authority to Operate (P-ATO).

As technology becomes more sophisticated, organizations find it more difficult to fully protect sensitive data, as cyber fraudsters use evolving technology to penetrate security platforms. Read this blog post to find out how SOAR strengthens security teams and prevents cyber fraud.

My role as a Chief Security Officer (CSO) has dramatically changed as we work to understand and adapt to COVID-19. It’s hard to believe that just a few weeks ago, my mind was focused on things such as FedRamp and the California Privacy Act (CCPA), now the majority of my time is focused on ensuring our employees safety and productivity, so they can continue to deliver products and support our customers and partners.

Not so long ago we used to hear about a cyber-attack or a new form of vulnerability in the news perhaps on a quarterly or monthly basis. Today, they are becoming increasingly more frequent and I don’t think a day goes by that we don’t read in the headlines about the consequences an organization is facing due to another attack.

As the number of vendors offering SOAR rises, it becomes particularly hard to choose the right SOAR solution. Read this article, and find out how to choose the perfect SOAR solution that aligns with your needs.

The first means to collect security-relevant information at Cloud SIEM Enterprise (CSE) was our Network Sensor. It was built to analyze network traffic and provide visibility beyond traditional SIEM's down to the network-level. Beyond organizing packets into flows, the sensor supports more advanced features such as decoding of common protocols, file carving, SSL certificate validation, OS fingerprinting, clustered deployment and more.

Since automation is becoming increasingly prevalent in the cybersecurity world, we dug a little deeper and unraveled the secrets of cybersecurity automation.

Many companies are moving their infrastructure and web applications to the cloud. Along with moving to the cloud, organizations are finding new ways to remain agile and nimble, especially when it comes to software. By utilizing Amazon AWS and Amazon ECS, companies are decoupling their monolithic applications and taking advantage of microservice architecture.

Today’s businesses spend more money on SaaS tools than on laptops. On average, today’s employees use a minimum of eight different SaaS tools. The security implications of this robust cloud landscape cannot be neglected and we trust you are fully aware of it already.

Finding out which key performance indicators to set for security operations is not an easy task. In this article, we’ll deconstruct the process of creating the perfect KPIs with your SOAR solution.

As businesses transform their traditional business models into new digital ones, and aggressively compete for turf within the digital economy, their constant pursuit of competitive edge drives technology, process, and architectural innovations. As a result, it seems that every 18 months a technology paradigm shift comes about that enables better agility, lower cost, improved quality of service, better intelligence and more.

Even though it is becoming increasingly popular, many still don’t exactly know what is SOAR. In this guide, you’ll find out what SOAR is, and how it can enhance your cyber security program.

The main theme for this year’s RSA event focused on the human element in addressing the behaviors and activities of users and analysts. This is something that was echoed in our Cloud SIEM Enterprise announcement the previous week and we demonstrated in our booth with our truly modernized security analyst experience. Actually, when attendees spotted our Cloud SIEM Enterprise user interface they immediately requested to see a live demo and witness this new security analyst experience.

There are clear benefits that explain why SOAR is a virtually indispensable asset for MSSPs. Read this article to find out what benefits MSSPs can expect from using SOAR.

As the cloud continues to expand with no end in sight, it’s only wise to invest in it. Infrastructure-as-a-Service, Platform-as-a-Service and Software-as-a-Service bring significant cost savings (personnel and ownership), improved performance, better reliability, freedom to scale and - above all - significant security benefits. It’s no wonder that so many businesses have already adopted all three of these models.

When thinking about many of the worst data breaches we’ve seen so far, there was one common element: The attacks were not detected while they were active on the internal networks.

In advance of the RSA Conference 2020, we wanted to get a pulse of attendees’ perceptions on a few topics, specifically challenges facing modern SOCs (security operations centers) and the value they are getting from technologies such as analytics, automation, and their SIEM tools.

Incident response automation brings many benefits to SOCs, but it must be done in a balanced and well-organized manner.

The tongue-in-cheek named malware detection tool, Yet Another Recursive Acronym (YARA) is described as “the pattern-matching Swiss Army knife for malware researchers (and everyone else)”. The Sumo Logic Cloud SIEM Enterprise platform is one of the first SIEM solutions to incorporate it as a built-in feature.

SaaS adoption is continuously on the rise and so is the number of companies migrating their email services to Microsoft Office365. It’s the most popular SaaS service and while over 90% of enterprises use it, only less than a quarter of them have already migrated to the cloud-based suite. Nonetheless, this number is growing steadily, as cloud adoption rates are increasing.

I have a confession to make. I was going to use this blog to wax poetically about the flaw inherent in the well-established notion of the Three Pillars of Observability. I was going to argue that a fourth pillar was badly needed: topological metadata.

In 2010, cloud computing just started to lead the IT revolution. It’s 2020 and the cloud is already mainstream. If you’re not running your business in virtual yet, you’re missing out on huge profit opportunities and capabilities that the cloud has to offer. Adopting a cloud strategy brings better security, increased stability and overall greater flexibility for your organization.

In the first post of our Amazon EKS series, we went deep into what EKS is and how organizations that run Kubernetes can benefit from it. In this second installment, we’ll learn why it’s essential for organizations to monitor EKS logs, along with how to do it.

We enable our customers to monitor, troubleshoot and resolve operational issues and security threats to their cloud or hybrid environments with our machine data analytics suite. Our users already know that Sumo Logic can help them dramatically improve the security and economy of their organization.

Continuing Diagnostics and Mitigation (CDM) is a program of the Department of Homeland Security designed to enhance cybersecurity across the Federal government. By deploying a standardized stack of pre-approved security tools, CDM ensures that small and large agencies alike can protect their networks from common threats.

System administrators hold many key responsibilities within an IT organization. Most importantly, they must ensure that all systems, services, and applications are up, running, and performing as expected. When a system starts to lag or an application is down, the system administrators are called upon to troubleshoot and resolve the issue as quickly as possible to limit the impact on customers.

As any developer or system administrator will tell you, log files are an extremely useful tool for debugging issues within a web application. In fact, log files are typically utilized as the primary source of information when a website is malfunctioning.

At this year’s AWS reInvent, we heard Andy Jassy go on stage to announce a bunch of new services to help companies unleash the power of cloud. 27 new services to be exact - everything from Machine learning IDE , to code review tools to contact center offerings (see the full list here); last year, AWS announced another 30 new services ranging from machine learning to VR/AR to satellite data.

A few blogs posts ago I wrote about new BI for digital companies and in that blog I alluded that quite a bit of that BI is based on log data. I wanted to follow up on the topic of logs, why they exist and why they contain so much data that is relevant to BI.

In this three-part series, we will take a hard look at Amazon’s Elastic Kubernetes Service: how it helps organizations run Kubernetes on AWS, what insights can be gained in EKS and how it’s monitored, and finally, how organizations can get the most out of EKS with the help of Sumo Logic. For the first installment, we’ll learn how it works and how organizations can get started with Amazon’s Elastic Kubernetes Service.

With 2020 around the corner, we’re entering a new decade in the cybersecurity landscape. What does the future hold for security professionals?

In a perfect world, computers would function properly on the network at all times. There would be no issues with the operating system and no problems with the applications. Unfortunately, this isn’t a perfect world. System failures can and will occur, and when they do, it is the responsibility of system administrators to diagnose and resolve the issues. But where can system administrators begin the search for solutions when problems arise? The answer is Windows event logs.

Kubernetes has become the de-facto solution for container orchestration. While it has, in some ways, simplified the management and deployment of your distributed applications and services, it has also introduced new levels of complexity.

If you are reading this, I don’t have to convince you any further of the powerful intelligence we can derive from logs and machine data. If you are anything like the many, many users, customers and prospects we have been talking to over the years, you might, however, have some level of that pesky modern condition commonly known as volume anxiety.

The last fifteen years have seen huge increases in developer productivity for several reasons, including the arrival of open source into the mainstream and the ability to better emulate target environments. In addition, the process of resetting a development environment back to the last known stable version has been vastly improved by Vagrant and then Docker.

Today's IT and DevOps teams have not one, but two, feature-rich open source Web servers to choose from: NGINX and Apache HTTP Server (which is often called simply "Apache"). At a high level, both platforms do the same core thing: Host and serve Web content. Both also offer comparable levels of performance and security.

For nearly 10 years, AWS and Sumo Logic have been the perfect pairing for businesses going through their digital transformation journey. AWS provides the best technology to help companies with their digital transformation, while Sumo Logic provides continuous intelligence and insights to monitor, run and secure those applications on AWS.

Security is a top concern for any enterprise to move their applications and workloads to the public cloud. AWS offers a broad selection of native security tools and as our Continuous Intelligence Report noted, AWS customers are using several of these to improve the security of their AWS environment. However, it can be overwhelming to know where to start and how to deploy best practices for detecting security misconfigurations caused by human errors and attacks from external sources.

AWS offers more than 150 discrete services, spanning compute, storage, database, network, and identity management to name a few. Earlier this year we published our Continuous Intelligence Report in which we surveyed Sumo Logic customers on how broadly they used the various AWS services. We found that the median number of different services most orgs use was 15.

We are excited to announce the beta release of Sumo Logic’s Archive Intelligence Service, which enables customers to forward logs directly from Sumo Logic’s installed collector to their own, self-managed AWS S3 buckets.

An overview of how to collect Cloud Run container logs and metrics, and send them to Sumo Logic for monitoring and troubleshooting.

In the second installment of our Amazon Redshift series, we covered the different ways you can monitor the performance and disk space of your Redshift servers using tools in AWS. In this final post, we will discuss how you can take your monitoring and logging efforts up a couple of notches by using Sumo Logic with Amazon Redshift.

In part 2 of our AWS S3 Monitoring series, we covered the basics of AWS S3 logging, why it’s important to log all the information in your cloud environment, and also the benefits of monitoring those logs.

Nowadays, it’s not uncommon to see enterprise IT leaders in a situation that seems like a catch 22. Oftentimes, they are expected to be involved in making data-driven decisions for augmenting productivity and profitability. Paradoxically, they are preoccupied with what they consider as their core responsibilities – applying best practices to safeguard the IT infrastructure and expediting investigations when incidents occur. As practitioners of IT, we must admit that it rings a bell and also chip in with our knowhow.

First time this year, multi-cloud enterprises, as a customer segment of Sumo Logic, have grown faster than any other segment: 50% Y/Y. What took so long? In my conversations with enterprises over the last 5 years, there was only one strategy for public cloud and it was multi-cloud. But evidence of multi-cloud usage was sparse at best. Data from our Continuous Intelligence Report in previous years didn’t find much to support that the strategy for multi-cloud was being implemented.

Never before in history has the concept of identity been so vital. To a large extent, everything we rely on to live our lives depends on who we are… or perhaps more accurately, who we can prove ourselves to be. Our data has come to be the standard by which we define ourselves. Because this identity-defining data is online, the protection of our data is of paramount importance.

If there is one thing that all Software as a Service (SaaS) companies understand, it is the pressure of “being fired”, as SPS Commerce’s Andy Domeier puts it. SPS Commerce is a cloud-based supply chain management software company and Andy is a Senior Director of Technology there - so he knows what he is talking about.

Amazon Simple Storage Service, widely known as Amazon S3, is a highly scalable, fast, and durable solution for object-level storage of any data type. Unlike the operating systems we are all used to, Amazon S3 does not store files in a file system, instead it stores files as objects. Object Storage allows users to upload files, videos, and documents like you were to upload files, videos, and documents to popular cloud storage products like Dropbox and Google Drive. This makes Amazon S3 very flexible and platform agnostic.

How do you migrate a production system to Kubernetes with confidence? Lior Mechlovich is an SRE for a cloud platform made up of dozens of microservices spanning 10+ teams and 5+ countries. Migration is difficult and risky. In this talk Lior shares his experience and lessons learned migrating to Kubernetes; how they trained teams, gained visibility, and triple checked each phase of the migration.

Learn about one engineer's – Ben Abrams, Supreme Unicorn Hunter of Planet Earth and the Entire Galaxy Besides – journey migrating to Sumo Logic from an AWS-managed ELK solution in order to free up operational resources and get engineers focused on extracting value from logs rather than spending time managing the solution.

This document is meant for executives (CIO/CISO/CxO) and IT professionals who are seeking to understand the business use case for Kubernetes.

Amazon Redshift is a fully-managed petabyte-scale cloud-based data warehouse service designed for large scale data set storage and analysis. Part of the larger Amazon Web Services (AWS) platform, Redshift is built to handle massive amounts of data for processing, analysis, and migration.

Let's take a look at a few strategies you can put in place to mitigate your risk of becoming part of the next ransomware statistic.

What a pleasure it was to see many of our customers at our Illuminate user conference, September 11-12. We had record attendance from customers, influencers, and partners. Our time was packed with keynotes, customer presentations (35 customer breakout sessions), certifications, sharing best practices, and time networking and having fun together.

Virtually every organization is a victim of cybercrime today. As the threat landscape evolves and proliferates, it’s necessary to prioritize the protection of data, customers’ privacy and brand reputation. Security directors must be prepared and equipped with the necessary tools to detect security events and address them accordingly at all times. This cannot be achieved without security investigation and correlation, and with the latest technologies, these can be performed at cloud-scale with ease.

Everything old is new again–that’s a universal law! In our industry, we have talked about Service Level Agreements (SLAs) for a long time. At Sumo, we are proud to continue to be the only Machine Data Analytics Platform as a Service that commits to a query performance SLA.

Serverless computing is the latest, greatest thing in the technology world. Although the serverless concept has been around in one form or another for more than a decade, the introduction of serverless platforms from major cloud providers—starting with AWS Lambda in 2014—has brought serverless mainstream for the first time.

Kubernetes, commonly called K8, is an open-source container management system developed by Google. Containers and tools like Kubernetes enable automation of many aspects of application deployment, which provides tremendous benefits to businesses.

Serverless computing is becoming more popular as organizations look for new ways to deploy their applications in the cloud. With higher levels of abstraction, easier maintenance, a focus on high performance, and ephemeral workloads, serverless computing solutions like Lambda are finding a permanent place in the mix of cloud infrastructure options.

Not all security information and event management (SIEM) use cases are equally important. Depending on the nature of your business, some will be more useful than others. How to know which ones are crucial for your business? Read on to find out.

Whether to enable hybrid and multi-cloud, promote deeper specialization among development teams, enhance reliability, or simply stay ahead of the curve, organizations are reaping the varied benefits of this technology investment— but it comes at a cost. With each optimization, there are tradeoffs.

Sumo Logic continues to innovate and provide industry-leading product features including content management APIs, enterprise audit index, and ingest budgets to better control your log analysis.

Slack is a popular cloud-based set of software tools and online services that provides for secure collaboration across teams, departments, offices, and countries. We are happy to announce support for monitoring Slack workspaces with the new Sumo Logic app for Slack.

We're happy to announce that the MongoDB Atlas app is now available in the Sumo Logic app catalog. MongoDB Atlas is a globally distributed cloud database service and the easiest way to run MongoDB in the cloud.

Sumo Logic extends their Kubernetes monitoring solution by supporting Istio allowing users to get a comprehensive view of their Kubernetes clusters, quickly troubleshoot Istio components, and gain full stack visibility.

Sumo Logic strengthens their partnership and ecosystem footprint by opening a global presence with strategic partnerships in Japan, United Kingdom, Australia, New Zealand, Germany, Hong Kong, Singapore, Spain and the Nordics.

The fourth annual Continuous Intelligence Report benchmarks and analyzes key technology adoption trends and tools including multi-cloud usage, Kubernetes adoption, serverless trends, and more. Read The Report

Does your business accept credit card payments? Read this article to find out what the challenges to the Payment Card Industry Data Security Standard (PCI DSS) are, and the current best practices to ensure that you are in compliance with this legal requirement.

Today we speak about observability explicitly in terms of DevOps/SRE. So, what is the objective of observability for DevOps engineers or SREs? The answer to that one hides behind the acronym itself … Site Reliability.

We live in a containerized world, and traditional monitoring and logging are being forever changed. The dynamic and ephemeral nature of containers creates new logging challenges.

As DevSecOps continues to redefine the IT security landscape, security is becoming everyone’s responsibility. That means that staying ahead of the latest cybersecurity threats—such as IoT botnets—should be a priority for every DevOps professional.

In a recent post, we talked about AWS CloudTrail and saw how CloudTrail can capture histories of every API call made to any resource or service in an AWS account. These event logs can be invaluable for auditing, compliance, and governance. We also saw where CloudTrail logs are saved and how they are structured.

Containers and serverless computing are two of today’s hottest technologies for application deployment. When used the right way, they both help DevOps teams to deploy applications faster and more cost-effectively.

If you are reading this article, you’re probably familiar with syslog, a logging tool that has been around since the 1980s. It is a daemon present in most Linux-based operating systems.

Today’s web server ecosystem has three big players: IIS, Apache and NGINX. Although only two of them (Apache and, to a lesser extent, NGINX) are cross-platform, it’s increasingly important to be able to work with all three of these servers, because you never know which type of operating system and web server platform you’ll be asked to support.

As multi-cloud architectures grow in popularity, more and more organizations will start asking how to secure multi-cloud environments. Some will conclude that a multi-cloud architecture requires a fundamentally different approach to cloud security.

Amazon Redshift is a fully-managed petabyte-scale cloud based data warehouse product designed for large scale data set storage and analysis. It is also used to perform large scale database migrations.

User Conferences are powerful. I remember well the previous startup-turned-grown-up- company that I was blessed to be part of. User conferences there were always rewarding experiences. Preparations were stressful and until the first day of the actual conference, I could feel as if everything was going to fall apart.

As the worldwide spending on SaaS spending will make up more than half of all public cloud services spending through 2019, it is critical to have end-to-end visibility into threats across your SaaS and on-premise applications.

In our recent article, we outlined the benefits of Security Information and Event Management (SIEM) systems, and why it is a must-have for every organization that operates in today’s cyberspace.

Recently, we announced a stellar line-up of speakers for our third-annual user conference, Illuminate taking place on Sept 11-12 in Burlingame, CA. At this year’s conference we want our attendees to “See Business Differently..See the World Differently.”

Fastly CDN improves the performance of your web properties by bringing the content closer to your users without exhausting your resources. With the sheer amount of traffic coursing through Fastly’s data centers, there is a wealth of log data you can analyze to give you in-depth insight into the performance of your web properties.

If you’re just joining us, I highly advise you to go back and check out our first two parts of this three part series regarding NGINX and Sumo Logic where we go over a basic Introduction to NGNIX and also Touch Up On NGINX, Logs, and Why Logs Are Important. If you’ve been following along, then great, let’s jump right into it.

Not investing in Security Incident and Event Management solutions means you’re missing out on significant business benefits. SIEM detects and responds to security incidents in real time, which reduces the risk of noncompliance.

The principles of data protection are the same whether your data sits in a traditional on-prem data center or in a cloud environment. The way you apply those principles, however, are quite different when it comes to cloud security vs. traditional security. Moving data to the cloud introduces new attack-surfaces, threats, and challenges, so you need to approach security in a new way.

Fastly serves enormous amounts of traffic on behalf of their customers. Because of this, collecting, analyzing, and monitoring log data are crucial in making sure that web properties are optimally configured, secure, and performing at their best.

As many of you know, Gartner is recognized as one of the premier analyst firms by most enterprise IT organizations. Given the broad and diverse set of customers Gartner serves -- many risk averse and conservative towards new technologies, Gartner generally tends to recommend incremental and measured changes.

In part one of our introduction to NGINX “What is NGINX” , we went over the basic history of NGINX, the difference between Apache and NGINX, and why you would use NGINX over Apache in certain environments and web applications. Today we’ll be diving deeper into NGINX and going over topics such as web server performance, monitoring said performance, how to obtain and archive logs for deeper analysis, and how to even tell which web server you’re running on your environment.

Enterprises are increasingly adopting a cloud-first approach and migrating their workloads, data and applications to the Cloud. Amazon Web Services continues to lead the Public Cloud industry with more than 30% of the market. As digital transformation progresses and the digital space expands, so does the attack surface that exposes the ongoing proliferation of security risks.

Learn the basics of AWS CloudTrail, see how to create and enable custom trails, and see where the trail logs are saved.

On AWS, your workloads will be as secure as you make them. The Shared Responsibility Model in which AWS operates ensures the security of the cloud, but what’s in the cloud needs to be secured by the user.

Software has eaten the world and every company today is a software company. This is because every company today is more and more serving its customers digitally.

Today, we are announcing the general availability of our new module within our Global Intelligence Service with a benchmarking capability on AWS security by baselining the Amazon GuardDuty findings. If you are one of the 100,000 users of Sumo, go to your App catalog and install the Amazon GuardDuty benchmark app with one click and see your threats against the global threats that we gather from hundreds of Sumo customers.

Gartner has been a thought leader in the SIEM space for the last few years. Gartner’s Magic Quadrant is considered one of the top market research reports on SIEM’s capabilities and vendors. Very recently, I attended the 2019 Gartner Security & Risk Management Conference, and based on thousands of conversations Gartner has had with their clients, they have a good vantage point on the SIEM space this year.

G Suite is Google’s integrated suite of secure, cloud-native collaboration and productivity apps. Some of the most popular apps from the suite are Gmail, Docs, Calendar, and Drive.

Today we will be covering NGINX, an open source software web server and main alternative/competitor to Apache HTTP Server. NGINX has been gaining in popularity since its inception and is used across a wide spectrum of applications for web serving, reverse proxying, caching, load balancing, media streaming, and much more.

In this blog series, we will cover how Fastly and Sumo Logic empower organizations to deliver best-in-class user experience. For the first installment, we take a closer look at Fastly CDN--what it is and how it works.

In the new report, “Analytics is making its security operations mark ahead of schedule,” analyst firm 451 Research details the accelerating transition happening in the security information and event management (SIEM) space. The report underscores how new cloud-native analytics solutions are displacing traditional SIEMs at the heart of the defense.

As we continue to adopt a digital-first mentality globally, there’s a massive shift to the cloud happening within federal agencies. While the sector has traditionally been slower to adopt new technologies, these agencies are understanding the urgent need to transition from legacy on-premise systems to more scalable and secure, cloud-based architectures.

On February 3, 2019, the Sumo Logic platform experienced its biggest ever spike in incoming data and analytics usage in the company’s history. On this day, close to everybody in the U.S., and many more people across the world, experienced a massive sports event: Super Bowl LIII.

Organizations that deploy SIEM systems know this uphill climb well. Deployment typically takes 18 months, and more than half of these SIEM deployments fail. A major friction point is the big hardware refresh every three to four years, requiring all configurations & customizations be saved. With the refresh, you must update hardware, software, rules, patches, among others, and that usually renders the SIEM unusable. If you magically get the SIEM working on the first cycle, it may fail on the next. Or when you stop pedaling.

If you have had any exposure to cloud computing or app development in recent years, you likely have heard the term “cloud native” thrown around. But you might be wondering what exactly that term means, and how it differs from concepts such as “cloud ready” or “cloud enabled.”

As more enterprises migrate their infrastructure to the cloud, it's important to baseline and benchmark important metrics to properly understand the performance improvements and ROI of the migration. Read our key metrics here.

Hooks are a recent addition to React that enable more of your components to be written as functions by providing less complex alternatives to class features. One significant advantage is that typing function components in TypeScript is simpler and more direct.

Internet security, in general, is a challenge that we have been dealing with for decades. It is a regular topic of discussion and concern, but a relatively new segment of internet security is getting the lion’s share of attention—internet of things (IoT). So why is internet of things security… a thing?

Recycling is an important part of my family’s weekly chore patterns. Our Amazon deliveries alone generate copious amounts of cardboard for our weekly pickup in the giant blue can. I also find myself trying to think about the longevity of the stuff I buy and try not to be wasteful. I feel a sense of pride. But there is one area where I just don’t think recycling makes sense -- and that’s with software.

In order to effectively manage and monitor your infrastructure, a web admin needs clear and transparent information about the types of activity going on within their servers. Server logs provide a documented footprint of all traffic and errors that occur within an environment. Apache has two main log files, Error Logs, and Access Logs.

Microsoft Windows Internet Information Services (IIS) log files provide valuable information about the use and state of applications running on the web. However, it’s not always easy to find where those files are to determine important aspects of app usage like when requests for servers were made, by whom, and other user traffic concerns.

Software is eating the world. How we spend time, what we eat, who we meet, how we communicate, where we travel... is defined by the code. Increasingly, software is calling the shots and telling humans what to do. With deep learning, this trend is just going to accelerate.

Apache HTTP Server is a free and open-source web server that delivers web content through the internet. It is commonly referred to as Apache and after development, it quickly became the most popular HTTP client on the web.

These days we spend a lot of time talking about modernizing our stack, modernizing our architectures, using new application components, modern application life cycles, etc. So, what is this all about and why do we spend so much time talking about it?

We are pleased to announce the release of Ingest Budgets, a new feature which enables our users to track and control how much data is ingested into Sumo Logic and avoid overages in environments where data ingestion can spike unexpectedly. With Ingest Budgets, users can create budgets with ingestion thresholds that either cap data ingestion to a daily limit or simply alert whenever the threshold is exceeded.

Site reliability engineers (SREs) rely on monitoring and analytics tools like Sumo Logic to guarantee uptime and performance of their applications and various components or services in production. The ability to visually monitor, automatically generate alerts and efficiently troubleshoot an issue in real time has become table stakes for any modern SRE team.

There’s really nothing cooler than reaching that moment in education when you’re ready to translate. Success can be measured many ways but one of the most effective is that you’re ready to transcend your native language and start educating customers in another language. The key phrases and stock analogies you’ve used so far are about to be put to the test. Can you help others in another language?

Cloud networks are popular targets for cybercriminals and organizations will inevitably face them. If you’ve ever administered a network of any type, you know that DDoS (distributed denial of service) attack attempts are really frequent, and there’s loads of malware out there too.

Democratizing data is one of our key product goals, and we share a similar approach to content. With over half a million words, our Sumo Logic documentation set is a substantial amount of information to provide to our users on the various ways you can collect logs and metrics, query that information, and turn it into meaningful visualizations. But the real trick is making sure that people can find what they need quickly.

Dealing with IT outages and downtime is one of the biggest technical challenges of the modern era, costing North American businesses an estimated $700 billion per year. Today's world of interconnected cloud services and microservice architectures has created infinitely more opportunities for something to go wrong and disrupt service. When that happens, there's an urgent need to alert the right people or teams to fix things.

GuardDuty is a threat detection service which constantly monitors the activity in your AWS network for anomalous behavior which could indicate cyber attacks or other unauthorized uses.

All businesses today are built on layers of platforms. The app running your business is built on top of the Kubernetes application deployment platform, running on the AWS cloud platform. AWS is built on top of platforms such as the Linux operating system and the Intel X86 processor architecture. Smartly managed, a good product evolves into a platform for users to extract value and for developers to create new products and platforms. We all stand on the shoulders of giants.

What is Serverless? Serverless computing is a cloud-based application architecture where the application’s infrastructure and support services layer is completely abstracted from the software layer.

A few years ago, our UX team created personas for Sumo Logic. The intention with the personas was to capture the mindset of our different users and to create a common vocabulary throughout our organization.

Collaboration with Carbon Black allows joint customers to use Sumo Logic to correlate, validate and investigate Carbon Black Response and Defense alerts.

It’s RSA Conference 2019 and Sumo Logic’s VP of Global Partner Sales & Alliances Jabari Norton caught up with Dave Cliffe, head of strategy at PagerDuty, one of our valued partners, to share some news about our joint integration that empowers customers to holistically manager their modern security operations. Watch our latest partner cam video below for the details!

What is AWS? Discover the services Amazon offers for making IT design and management simple with AWS.

The way we all experience and interact with apps, devices, and data is changing dramatically. End users demand that apps are responsive, stable, and offer the same user experience no matter which platform they are using. To build these well, many developers consider creating cross-platform apps. Although building a separate native app per platform is a preferred approach for mass market consumer apps, there are still a lot of situations where it makes more sense to go cross-platform. In this post I’ll look at the most popular strategies a developer faces when it comes to building a mobile app, and some tools that help you to build well. Mobile Web Apps This is probably the most easiest way onto a mobile device. Mobile web apps are hosted on a remote server and built using identical technologies to desktop web apps: HTML5, JavaScript and CSS. The primary difference is that it will be accessed via mobile device’s built-in web browser, which may require you to apply responsive web design principles to ensure that the user experience is not degraded by the limited screen size on mobile and that would be costly to build and maintain. The cost of applying responsive design principles to a web site may be a significant fraction of developing a mobile app. Native Mobile Apps Native apps are mainly developed using the device’s out-of-the-box SDK. This is a huge advantage as you have full access to the device’s API, features, and inter-app integration. However it also means you need to learn Java to build Apps for Android, Objective-C for iOS, and C# for Windows phones. Whether you are a single developer or working with a company and multi team skills, learning to code in multiple languages is costly and time-consuming. And most of the time, all features will not be available on every platform. Cross-Platform Mobile Apps Cross-platform apps have somewhat of a reputation of not being competitive against native apps, but we continue to see more and more world class apps using this strategy. Developers only have to maintain a single code base for all platforms. They can reuse the same components within different platforms, and most importantly, developers can still access the native API via native modules. Below are some tools that support building cross-platform apps: PhoneGap Owned by Adobe, PhoneGap is a free resource and handy to translate HTML5, CSS and JavaScript code. Once the app is ready, the community will help in reviewing the app and it is supported all major platforms including BlackBerry. Xamarin.Forms With a free starter option, Xamarin.Forms is great tool for C# and Ruby developers to build an app cross-platform with the option of having access to native platform’s API. The a wide store of component to help achieve the goal faster. Xamarin has created a robust cross platform mobile development platform that’s been adopted by big names like Microsoft, Foursquare, IBM, and Dow Jones. Unity 3D This tool is mainly focused on building game apps, and very useful when graphics is most important detail in it. This cross platform mobile development tool goes beyond simple translation. After developing your code in UnityScriptor or C#, you can export your games to 17 different platforms, including iOS, Android, Windows, Web, Playstation, Xbox, Wii and Linux. When it comes to building an app, whether cross-platform or not, views and thoughts always differ. My preference is cross-platform for one main reason; it is less time-consuming - that is critical because I can then focus on adding new features to the app, or building another one. About the Author Mohamed Hasni is a Software Engineer focusing on end-to-end web and mobile development and delivery. He has deep experience in building line of business applications for large-scale enterprise deployments.

In October of last year, I joined Sumo Logic to lead sales and go-to-market functions with the goal of successfully launching our newly established Japan region in Tokyo. The launch was highly received by our customers, partners, prospects and peers in the Japanese market and everyone walked away from the event optimistic about the future and hungry for more!It certainly was an exciting time not only for the company, but for me, personally, and as I reflect over the past few months here, I wanted to share a little bit about why the company’s launch in Japan came at a very strategic and opportune time as well as why Sumo Logic is a great market fit.Market OpportunityIn terms of overall IT spend and market size, Japan remains the second largest market in the world behind U.S. in enterprise technology. A large part of that is because of service spending versus traditional hardware and software.For years, Japan had been a half step or more behind in cloud technology innovation and adoption, but that has since changed. Now, Japan is experiencing a tsunami of cloud adoption with major influence from Amazon Web Services (AWS) who has aggressively invested in building data centers in Japan the past several years.The fact that AWS began heavily investing in the Japanese technology market was a strong indication to us at Sumo Logic that as we continue to expand our global footprint, the time was finally right to capitalize on this market opportunity.Sumo Logic OpportunityHowever, market opportunity aside, the nature of our SaaS machine data analytics platform and the services we provide across operations, security and the business, was a perfect fit for the needs of innovating Japanese enterprises. I’ve been here in Tokyo for over 30 years so I feel (with confidence) that it was our moment to shine in Japan. From a sales perspective, we’re very successful with a land and expand approach where we start with only a small subset of the business, and then gradually, we grow to other areas, operations, security and business as we continue to deliver great customer experiences that demonstrate long term value and impact. That level of trust building and attentiveness we provide to our global customer base is very typical of how Japanese enterprises like to conduct business. In other words, the core business model and approach of Sumo Logic are immediately applicable to the Japanese market. Anyone with experience in global IT will understand the simple, but powerful meaning of this; Sumo Logic`s native affinity with Japan is an enormous anomaly.And, Japan can be a very unforgiving market. It’s not a place where you want to come with a half-baked products or a smoke and mirrors approach. Solid products, solutions and hard work are, on the other hand, highly respected.Vertical FocusAs I’ve mentioned above, the Japan market is mostly enterprise, which is a sweet spot for Sumo Logic, and there’s also a heavy influence of automotive and internet of things (IoT) companies here. In fact, four of the world’s largest automotive companies are headquartered in Japan and their emerging autonomous driving needs, in particular, align directly with the real-time monitoring, troubleshooting and security analytics capabilities that are crucial for modern innovations around connected cars and IoT, which both generate massive amounts of data. Customers like Samsung SmartThings, Sharp and Panasonic leverage our platform for DevSecOps teams that want to visualize, build, run and secure that data. The connected car today has become less about the engine and more about the driver experience, which is 100 percent internet-enabled.Japan is also one of the two major cryptocurrency exchange centers in the world, which is why financial services, especially fintech, bitcoin and cryptocurrencies companies, is another focus vertical for Sumo Logic Japan. Our DevSecOps approach and cloud-native multi-tenant platform provides massive mission-critical operations and security analytics capabilities for crypto companies. Most of these financial services companies are struggling to stay on top of increasingly stringent regulatory and data requirements, and one of the biggest use cases for these industries is compliance monitoring. Japan is regulatory purgatory, and so our customers look for us to help automate parts of their compliance checks and security audits.Strong Partner EcosystemHaving a strong partner ecosystem was another very important piece of our overall GTM strategy in Japan. We were very fortunate to have forged an early partnership with AWS Japan that led to an introduction to one of their premium consulting partners, Classmethod, the first regional partnership with AWS APN. The partnership is already starting to help Japanese customers maximize their investment in the Sumo Logic platform by providing local guided deployment, support and storage in AWS. In addition, Sumo Logic provides the backbone for Classmethod’s AWS infrastructure to provide the continuous intelligence needed to serve and expand their portfolio of customers.Going forward, we’ll continue to grow our partner ecosystem with the addition of service providers, telecoms, MSPs and MSSPs for security to meet our customer’s evolving needs.Trusted AdvisorAt the end of the day, our mission is to help our customers continue to innovate and provide support in areas where they most need it — economically visualizing data across their modern application stacks and cloud infrastructures. We’re in a position to help all kinds of Japanese customers across varying industries modernize their architectures. Japanese customers know that they need to move to the cloud and continue to adopt modern technologies.We’re in the business of empowering our customers to focus on their core competencies while they leave the data piece component to us. By centralizing all of this disparate data into one platform, they can better understand their business operations, be more strategic and focus on growing their business. We’ve gone beyond selling a service to becoming both “data steward” as well as trusted data advisors for our customers. Japanese business is famous for its organic partnering model — think of supply chain management, and so on. Sumo Logic’s core strategy of pioneering machine data stewardship is a perfect extension of this to meet the rapidly evolving needs of the digital economy in Japan.Now that we have a local presence with a ground office and support team, we can deliver a better and more comprehensive experience to new and existing customers, like Gree and OGIS-RI, and look forward to continued growth and success in this important global market.Additional ResourcesRead the press release for more on Sumo Logic's expansion into Japan Download the white paper on cloud migration Download the ‘State of Modern Apps & DevSecOps in the Cloud’ report

As discussed in our previous post, we recently had the opportunity to present some interesting challenges and proposed directions for data science and machine learning (ML) at the 2018 Scale By the Bay conference. While the excellent talks and panels at the conference were too numerous to cover here, I wanted to briefly summarize three talks in particular that I found to represent some really interesting (to me) directions for ML on the java virtual machine (JVM). Talk 1: High-performance Functional Bayesian Inference in Scala By Avi Bryant (Stripe) | Full Video Available Here Probabilistic programming lies at the intersection of machine learning and programming languages, where the user directly defines a probabilistic model of their data. This formal representation has the advantage of neatly separating conceptual model specification from the mechanics of inference and estimation, with the intention that this separation will make modeling more accessible to subject matter experts while allowing researchers and engineers to focus on optimizing the underlying infrastructure. (image source) Rainier in an open-source library in Scala that allows the user to define their model and do inference in terms of monadic APIs over distributions and random variables. Some key design decisions are that Rainier is “pure JVM” (ie, no FFI) for ease of deployment, and that the library targets single-machine (ie, not distributed) use cases but achieves high performance via the nifty technical trick of inlining training data directly into dynamically generated JVM bytecode using ASM. Talk 2: Structured Deep Learning with Probabilistic Neural Programs By Jayant Krishnamurthy (Semantic Machines) | Full Video Available Here Machine learning examples and tutorials often focus on relatively simple output spaces: Is an email spam or not? Binary outputs: Yes/No, 1/0, +/-, … What is the expected sale price of a home? Numerical outputs – $1M, $2M, $5M, … (this is the Bay Area, after all!) However, what happens when we want our model to output a more richly structured object? Say that we want to convert a natural language description of an arithmetic formula into a formal binary tree representation that can then be evaluated, for example “three times four minus one” would map to the binary expression tree “(- (* 3 4) 1)”. The associated combinatorial explosion in the size of the output space makes “brute-force” enumeration and scoring infeasible. The key idea of this approach is to define the model outputs in terms of a probabilistic program (which allows us to concisely define structured outputs), but with the probability distributions of the random variables in that program being parameterized in terms of neural networks (which are very expressive and can be efficiently trained). This talk consisted mostly of live-coding, using an open-source Scala implementation which implements a monadic API for a function from neural network weights to a probability distribution over outputs. Talk 3: Towards Typesafe Deep Learning in Scala By Tongfei Chen (Johns Hopkins University) | Full Video Available Here (image source) For a variety of reasons, the most popular deep learning libraries such as TensorFlow & PyTorch are primarily oriented around the Python programming language. Code using these libraries consists primarily of various transformation or processing steps applied to n-dimensional arrays (ndarrays). It can be easy to accidentally introduce bugs by confusing which of the n axes you intended to aggregate over, mis-matching the dimensionalities of two ndarrays you are combining, and so on. These errors will occur at run time, and can be painful to debug. This talk proposes a collection of techniques for catching these issues at compile time via type safety in Scala, and walks through an example implementation in an open-source library. The mechanics of the approach are largely based on typelevel programming constructs and ideas from the shapeless library, although you don’t need to be a shapeless wizard yourself to simply use the library, and the corresponding paper demonstrates how some famously opaque compiler error messages can be made more meaningful for end-users of the library. Conclusion Aside from being great, well-delivered talks, several factors made these presentations particularly interesting to me. First, all three had associated open-source Scala libraries. There is of course no substitute for actual code when it comes to exploring the implementation details and trying out the approach on your own test data sets. Second, these talks shared a common theme of using the type system and API design to supply a higher-level mechanism for specifying modeling choices and program behaviors. This can both make end-user code easier to understand as well as unlock opportunities for having the underlying machinery automatically do work on your behalf in terms of error-checking and optimization. Finally, all three talks illustrated some interesting connections between statistical machine learning and functional programming patterns, which I found interesting as a longer-term direction for trying to build practical machine learning systems. Additional Resources Learn how to analyze Killer Queen game data with machine learning and data science with Sumo Logic Notebooks Interested in working with the Sumo Logic engineering team? We’re hiring! Check out our open positions here Sign up for a free trial of Sumo Logic

In this latest SnapSecChat video series, our CSO, George Gerchow, talks about the imperative for building the next-gen security operations center (SOC) of the future. Why? Because today's IT landscape is becoming increasingly complex and cloud-based. Everything has changed, from applications being built on microservices, to the CI/CD pipeline, to new and merging toolsets that help us manage and secure our cloud-based infrastructure via APIs. That means we must completely reimagine how we manage security operations if we are to keep pace with the rate of technological innovation. This includes a new level of rigor, adaptive processes, and industry and team collaboration to ensure we take a proactive security approach to continue to stay ahead of attackers.  For Sumo Logic, this means working with customers like The Pokemon Co. International and partners like Nordcloud to share industry best practices, processes and key learnings on how to effectively operate a modern day global SOC. To learn more about how Sumo Logic is working with Pokemon on joint SOC initiatives, check out this recent blog as well as the customer press release. If you enjoyed this video, then be sure to stay tuned for another one coming to a website near you soon! And don’t forget to follow George on Twitter at @GeorgeGerchow, and use the hashtag #SnapSecChat to join the security conversation!

This year, at Sumo Logic’s third annual user conference, Illuminate 2018, we presented Sumo Logic Notebooks as a way to do data science within the Sumo Logic platform. Sumo Logic Notebooks integrate Sumo Logic data, data science notebooks and common machine learning frameworks.

What are Sumo Cert Jams? Sumo Logic Cert Jams are one and two-day training events held in major cities all over the world to help you ramp up your product knowledge, improve your skills and walk away with a certification confirming your product mastery. We started doing Cert Jams about a year ago to help educate our users around the world on what Sumo can really do and give you a chance to network and share use cases with other Sumo Logic users. Not to mention, you get a t-shirt. So far, we’ve had over 4,700 certifications from 2,700+ unique users across 650+ organizations worldwide. And we only launched the Sumo Cert Jam program in April! If you’re still undecided, check out this short video where our very own Mario Sanchez, Director of the Sumo Logic Learn team, shares why you should get the credit and recognition you deserve! Currently there are four certifications for Sumo Logic: Pro User Power User Power Admin Security User And these are offered in a choose-your-own-adventure format. While everyone starts out with the Pro User certification to learn the fundamentals, you can take any of the remaining exams depending on your interest in DevOps (Power User), Security, or Admin. Once you complete Sumo Pro User, you can choose your own path to Certification success. For a more detailed breakdown on the different certification levels, check out our web page, or our Top Reasons to Get Sumo Certified blog. What’s the Value? Often customers ask me in one-on-one situations what is the value of certification, and I tell them that we have seen significant gains in user understanding, operator usage and search performance once we get users certified. Our first Cert Jam in Delhi, India with members from the Bed, Bath and Beyond team showing their certification swag! First, there’s the ability to rise above “Mere Mortals” (those who haven’t been certified) and write better and more complex queries. From parsing to correlation, there’s a significant increase by certified users taking Pro (Level 1), Power User (Level 2), Admin (Level 3) and Security. Certified users are taking advantage of more Sumo Logic features, not only getting more value out of their investment, but also creating more efficient/performant queries. And from a more general perspective, once you know how to write better queries and dashboards, you can create the kind of custom content that you want. When it comes to monitoring and alerting, certified users are more likely to create dashboards and alerts to stay on top of what’s important to their organizations, further benefiting from Sumo Logic as a part of their daily workload. Here we can see that certified users show an increase in the creation of searches, dashboards and alerts, as well as key optimization features such as Field Extraction Rules (FERs), scheduled views and partitions: Join Us If you’re looking to host a Cert Jam at your company, and have classroom space for 50, reach out to our team. We are happy to work with you and see if we can host one in your area. If you’re looking for ways to get certified, or know someone who would benefit, check out our list of upcoming Cert Jams we’re offering. Don’t have Sumo Logic, but want to get started? Sign up for Sumo Logic for free! Our Cert Jam hosted by Tealium in May. Everyone was so enthusiastic to be certified.

It finally happened. At the start of DockerCon Europe and a week before KubeCon was set to take place in the U.S., researchers discovered the first major vulnerability within Kubernetes, the popular cloud container orchestration system.

This post gives a brief overview of some ideas we presented at the recent Scale By the Bay conference in San Francisco, for more details you can see a video of the talk or take a look at the slides. The Problems of Sensitive Data and Leakage Data science and machine learning have gotten a lot of attention recently, and the ecosystem around these topics is moving fast. One significant trend has been the rise of data science notebooks (including our own here at Sumo Logic): interactive computing environments that allow individuals to rapidly explore, analyze, and prototype against datasets. However, this ease and speed can compound existing risks. Governments, companies, and the general public are increasingly alert to the potential issues around sensitive or personal data (see, for example, GDPR). Data scientists and engineers need to continuously balance the benefits of data-driven features and products against these concerns. Ideally, we’d like a technological assistance that makes it easier for engineers to do the right thing and avoid unintended data processing or revelation. Furthermore, there is also a subtle technical problem known in the data mining community as “leakage”. Kaufman et al won the best paper award at KDD 2011 for Leakage in Data Mining: Formulation, Detection, and Avoidance, which describes how it is possible to (completely by accident) allow your machine learning model to “cheat” because of unintended information leaks in the training data contaminating the results. This can lead machine learning systems which work well on sample datasets but whose performance is significantly degraded in the real world. As this can be a major problem, especially in systems that pull data from disparate sources to make important predictions. Oscar Boykin of Stripe presented an approach to this problem at Scale By the Bay 2017 using functional-reactive feature generation from time-based event streams. Information Flow Control (IFC) for Data Science My talk at Scale By the Bay 2018 discussed how we might use Scala to encode notions of data sensitivity, privacy, or contamination, thereby helping engineers and scientists avoid these problems. The idea is based on programming languages (PL) research by Russo et al, where sensitive data (“x” below) is put in a container data type (the “box” below) which is associated with some security level. Other code can apply transformations or analyses to the data in-place (known as Functor “map” operation in functional programming), but only specially trusted code with an equal or greater security level can “unbox” the data. To encode the levels, Russo et al propose using the Lattice model of secure information flow developed by Dorothy E. Denning. In this model, the security levels form a partially ordered set with the guarantee that any given pair of levels will have a unique greatest lower bound and least upper bound. This allows for a clear and principled mechanism for determining the appropriate level when combining two pieces of information. In the Russo paper and our Scale By the Bay presentation, we use two levels for simplicity: High for sensitive data, and Low for non-sensitive data. To map this research to our problem domain, recall that we want data scientists and engineers to be able to quickly experiment and iterate when working with data. However, when data may be from sensitive sources or be contaminated with prediction target information, we want only certain, specially-audited or reviewed code to be able to directly access or export the results. For example, we may want to lift this restriction only after data has been suitably anonymized or aggregated, perhaps according to some quantitative standard like differential privacy. Another use case might be that we are constructing data pipelines or workflows and we want the code itself to track the provenance and sensitivity of different pieces of data to prevent unintended or inappropriate usage. Note that, unlike much of the research in this area, we are not aiming to prevent truly malicious actors (internal or external) from accessing sensitive data – we simply want to provide automatic support in order to assist engineers in handling data appropriately. Implementation and Beyond Depending on how exactly we want to adapt the ideas from Russo et al, there are a few different ways to implement our secure data wrapper layer in Scala. Here we demonstrate one approach using typeclass instances and implicit scoping (similar to the paper) as well as two versions where we modify the formulation slightly to allow changing the security level as a monadic effect (ie, with flatMap) having last-write-wins (LWW) semantics, and create a new Neutral security level that always “defers” to the other security levels High and Low. Implicit scoping Most similar to the original Russo paper, we can create special “security level” object instances, and require one of them to be in implicit scope when de-classifying data. (Thanks to Sergei Winitzki of Workday who suggested this at the conference!) Value encoding For LWW flatMap, we can encode the levels as values. In this case, the security level is dynamically determined at runtime by the type of the associated level argument, and the de-classify method reveal() returns a type Option[T] where it is None if the level is High. This implementation uses Scala’s pattern-matching functionality. Type encoding For LWW flatMap, we can encode the levels as types. In this case, the compiler itself will statically determine if reveal() calls are valid (ie, against the Low security level type), and simply fail to compile code which accesses sensitive data illegally. This implementation relies on some tricks derived from Stefan Zeiger’s excellent Type-Level Computations in Scala presentation. Data science and machine learning workflows can be complex, and in particular there are often potential problems lurking in the data handling aspects. Existing research in security and PL can be a rich source of tools and ideas to help navigate these challenges, and my goal for the talk was to give people some examples and starting points in this direction. Finally, it must be emphasized that a single software library can in no way replace a thorough organization-wide commitment to responsible data handling. By encoding notions of data sensitivity in software, we can automate some best practices and safeguards, but it will necessarily only be a part of a complete solution. Watch the Full Presentation at Scale by the Bay Learn More

To gain a better understanding of the adoption and usage of machine data in Europe, Sumo Logic commissioned 451 Research to survey 250 executives across the UK, Sweden, the Netherlands and Germany, and to compare this data with a previous survey of U.S. respondents that were asked the same questions. The research set out to answer a number of questions, including: Is machine data in fact an important source of fuel in the analytics economy? Do businesses recognize the role machine data can play in driving business intelligence? Are businesses that recognize the power of machine data leaders in their fields? The report, “Using Machine Data Analytics to Gain Advantage in the Analytics Economy, the European Edition,” released at DockerCon Europe in Barcelona this week, reveals that companies in the U.S. are currently more likely to use and understand the value of machine data analytics than their European counterparts, but that Europeans lead the U.S. in using machine data for security use cases. Europeans Trail US in Recognizing Value of Machine Data Analytics Let’s dig deeper into the stats regarding U.S. respondents that stated they were more likely to use and understand the value of machine data analytics. For instance, 36 percent of U.S. respondents have more than 100 users interacting with machine data at least once a week, while in Europe, only 21 percent of respondents have that many users. Likewise, 64 percent of U.S. respondents said that machine data is extremely important to their company’s ability to meet its goals, with 54 percent of European respondents saying the same. When asked if machine data tools are deployed on-premises, only 48 percent of European correspondents responded affirmatively, compared to 74 percent of U.S. respondents. The gap might be explained by idea that U.S. businesses are more likely to have a software-centric mindset. According to the data, 64 percent of U.S. respondents said most of their company had software-centric mindsets, while only 40 percent of European respondents said the same. Software-centric businesses are more likely to recognize that machine data can deliver critical insights, from both an operational and business perspective, as they are more likely to integrate their business intelligence and machine data analytics tools. Software-centric companies are also more likely to say that a wide variety of users, including head of IT, head of security, line-of-business users, product managers and C-level executives recognize the business value of machine data. Europeans Lead US in Using Machine Data for Security At 63 percent, European companies lead the way in recognising the benefit of machine data analytics in security use cases, which is ahead of the U.S. Given strict data privacy regulations in Europe, including the new European Union (EU) General Data Protection Regulation (GDPR), it only seems natural that security is a significant driver for machine data tools in the region. Business Insight Recognized by Europeans as Valuable Beyond security, other top use cases cited for machine data in Europe are monitoring (55 percent), troubleshooting (48 percent) and business insight (48 percent). This means Europeans are clearly recognizing the value of machine data analytics beyond the typical security, monitoring and troubleshooting use-cases — they’re using it as a strategic tool to move the business forward. When IT operations teams have better insight into business performance, they are better equipped to prioritize incident response and improve their ability to support business goals. A Wide Array of European Employees in Different Roles Use Machine Data Analytics The data further show that, in addition to IT operations teams, a wide array of employees in other roles commonly use machine data analytics. Security analysts, product managers and data analysts — some of whom may serve lines of business or senior executives — all appeared at the top of the list of the roles using machine data analytics tools. The finding emphasizes that companies recognize the many ways that machine data can drive intelligence across the business. Customer Experience and Product Development Seen as Most Beneficial to Europeans Although security emerged as an important priority for users of machine data, improved customer experience and more efficient product development emerged as the top benefit of machine data analytics tools. Businesses are discovering that the machine analytic tools they use to improve their security posture can also drive value in other areas, including better end-user experiences, more efficient and smarter product development, optimized cloud and infrastructure spending, and improved sales and marketing performance. Barriers Preventing Wider Usage of Machine Data The report also provided insight into the barriers preventing wider usage of machine data analytics. The number one capability that users said was lacking in their existing tools was real-time access to data (37 percent), followed by fast, ad hoc querying (34 percent). Another notable barrier to broader usage is the lack of capabilities to effectively manage different machine data analytics tools. European respondents also stated that the adoption of modern technologies does make it harder to get the data they need for speedy decision-making (47 percent). Whilst moving to microservices and container-based architectures like Docker makes it easier to deploy at scale, it seems it is hard to effectively monitor activities over time without the right approach to logs and metrics in place. In Conclusion Europe is adopting modern tools and technologies at a slower rate than their U.S. counterparts, and fewer companies currently have a ‘software-led’ mindset in place. Software-centric businesses are doing more than their less advanced counterparts to make the most out of the intelligence available to them in machine data analytics tools. However, a desire for more continuous insights derived from machine data is there: the data show is that once European organisations start using machine data analytics to gain visibility into their security operations, they start to see the value for other use cases across operations, development and the business. The combination of customer experience and compliance with security represent strong value for European users of machine data analytics tools. Users want their machine data tools to drive even more insight into the customer experience, which is increasingly important to many businesses, and at the same time help ensure compliance. Additional Resources Download the full 451 Research report for more insights Check out the Sumo Logic DockerCon Europe press release Download the Paf customer case study Read the European GDPR competitive edge blog Sign up for a free trial of Sumo Logic

Sumo Logic provides digital businesses a powerful and complete view of modern applications and cloud infrastructures such as AWS. Today, we’re pleased to announce complete visibility into performance, health and user activity of the leading Amazon Aurora database via two new applications – the Sumo Logic MySQL ULM application and the Sumo Logic PostgreSQL ULM application. Amazon Aurora is a MySQL and PostgreSQL-compatible relational database available on the AWS RDS platform. Amazon Aurora is up to five times faster than standard MySQL databases and three times faster than standard PostgreSQL databases. By providing complete visibility across your Amazon Aurora databases with these two applications, Sumo Logic provides the following benefits via advanced visualizations: Optimize your databases by understanding query performance, bottlenecks and system utilization Detect and troubleshoot problems by identifying new errors, failed connections, database activity, warnings and system events Monitor user activity by detecting unusual logins, failed events and geo-locations In the following sections of this blog post, we discuss details how these applications provide value to customers. Amazon Aurora Logs and Metrics Sources Amazon provides a rich set of log and metrics sources for monitoring and managing Aurora databases. The Sumo Logic Aurora MySQL ULM app works on the following three log types: AWS CloudTrail event logs AWS CloudWatch metrics AWS CloudWatch logs For Aurora MySQL databases, error logs are enabled by default to be pushed to CloudWatch. Aurora MySQL also supports slow query logs, audit logs, and general logs to be pushed to CloudWatch, however, you need to select this feature on CloudWatch. The Sumo Logic Aurora PostgreSQL ULM app works on the following log types: AWS Cloud Trail event logs AWS CloudWatch metrics For more details on setting up logs, please check the documentation for the Amazon Aurora PostgreSQL app and the Amazon Aurora MySQL app. Installing the Apps for Amazon Aurora Analyzing each of the above logs in isolation to debug a problem, or understand how your database environments are performing can be a daunting and time-consuming task. With the two new Sumo applications, you can instantly get complete visibility into all aspects of running your Aurora databases. Once you have configured your log sources, the Sumo Logic apps can be installed. Navigate to the Apps Catalog in your Sumo Logic instance and add the “Aurora MySQL ULM” or “Aurora PostgreSQL ULM” apps to your library after providing references to sources configured in the previous step. Optimizing Database Performance As part of running today’s digital businesses, customer experiences is a key outcome and towards that end closely monitoring the health of your databases is critical. The following dashboards provide an instant view on how your Amazon Aurora MySQL and PostGreSQL databases are performing across various important metrics. Using the queries from these dashboards, you can build scheduled searches and real-time alerts to quickly detect common performance problems. The Aurora MySQL ULM Logs – Slow Query Dashboard allows you to view log details on slow queries, including the number of slow queries, trends, execution times, time comparisons, command types, users, and IP addresses. The Aurora MySQL ULM Metric – Resource Utilization Monitoring dashboard allows you to view analysis of resource utilization, including usage, latency, active and blocked transactions, and login failures. The Aurora PostgreSQL ULM Metric – Latency, Throughput, and IOPS Monitoring Dashboard allows you to view granular details of database latency, throughput, IOPS and disk queue depth. It is important to monitor the performance of database queries. Latency and throughput are the key performance metrics. Detect and Troubleshoot Errors To provide the best service to your customers, you need to take care of issues quickly and minimize impacts to your users. Database errors can be hard to detect and sometimes surface only after users report application errors. The following set of dashboards help quickly surface unusual or new activity across your AWS Aurora databases. The Aurora MySQL ULM Logs – Error Logs Analysis Dashboard allows you to view details for error logs, including failed authentications, error outliers, top and recent warnings, log levels, and aborted connections. Monitor user activity With cloud environments, its becoming even more critical to investigate user behavior patterns and make sure your database is being accessed by the right staff. The following set of dashboards track all user and database activity and can help prioritize and identify patterns of unusual behavior for security and compliance monitoring. The Aurora MySQL ULM Logs – Audit Log Analysis Dashboard allows you to view an analysis of events, including accessed resources, destination and source addresses, timestamps, and user login information. These logs are specifically enabled to audit activities that are of interest from an audit and compliance perspective. The Aurora MySQL Logs – Audit Log SQL Statements Dashboard allows you to view details for SQL statement events, including Top SQL commands and statements, trends, user management, and activity for various types of SQL statements. You can drill deeper into various SQL statements and commands executed by clicking on the “Top SQL Commands” panel in the dashboard. This will open up the Aurora MySQL ULM – Logs – Audit Log SQL Statements dashboard, which will help with identifying trends, specific executions, user management activities performed and dropped objects. The Aurora PostgreSQL ULM CloudTrail Event – Overview Dashboard allows you to view details for event logs, including geographical locations, trends, successful and failed events, user activity, and error codes. In case you need to drill down for details, the CloudTrail Event – Details dashboard will help you with monitoring the most recent changes made to resources in your Aurora database ecosystem, including creation, modification, deletion and , reboot of Aurora clusters and or instances. Get Started Now! The Sumo Logic apps for Amazon Aurora helps optimize, troubleshoot and secure your AWS Aurora database environments. To get started check out the the Sumo Logic MySQL ULM application and the Sumo Logic PostgreSQL ULM application. If you don’t yet have a Sumo Logic account, you can sign up for a free trial today. For more great DevOps-focused reads, check out the Sumo Logic blog.

Take a deep dive into usage data from the Sumo Logic Platform to understand which modern technologies customers use most in their AWS environments.

The multi-tenant software clearly has many advantages over on-premises or hosted software. Let me explain.

In the world of agile there’s a demand to solve grey areas throughout the design process at lightning speed. Prototypes help the scrum team test ideas and refine them. Without prototypes, we can’t test ideas until the feature or product has been built which can be a recipe for disaster. It’s like running a marathon without training. During a two week sprint, designers often need to quickly turn around prototypes in order to test. It can be hectic to juggle meetings, design and prototyping without a little planning. The guiding principles below, inspired by my time working with one our lead product designers at Sumo Logic — Rebecca Sorensen, will help you build prototypes more effectively for usability testing under a time crunch. Define the Scope From the start, it’s essential that we understand who is the audience and what is the goal of the prototype so we can determine other parts of the prototyping process like content, fidelity level and tools for the job. We can easily find out the intent by asking the stakeholder what he or she wants to learn. By defining the scope from the beginning we are able to prioritize our time more effectively throughout the prototyping process and tailor the content for the audience. For testing usually our audience are internal users or customers. The scrum team wants to know if the customer can complete a task successfully with the proposed design. Or they may also want to validate a flow to determine design direction. If we’re testing internally, we have more flexibility showing a low or mid fidelity prototype. However, when testing with customers, sometimes we have to consider more polished prototypes with real data. Set Expectations There was a time when designers made last minute changes to the prototype — sometimes while the prototype was being tested because a stakeholder added last-minute feedback — that impacted the outcome and did not provide enough time for the researcher to understand the changes. Before jumping into details, we create milestones to set delivery expectations. This helps the scrum team understand when to give feedback on the prototype and when the research team will receive the final prototype for testing. This timeline is an estimate and it might vary depending on the level of fidelity. We constantly experiment until we find our sweet spot. The best way to get started is to start from a desired end state, like debriefing the researcher on the final prototype, and work backward. The draft of the prototype doesn’t have to be completely finished and polished. It just needs to have at least structure so we can get it in front of the team for feedback. Sometimes, we don’t have to add all the feedback. Instead, we sift through the feedback and choose what makes sense given the time constraints. Tailor the Content for your Audience Content is critical to the success of a prototype. The level of details we need in the prototype depends on the phase our design process. Discovery In the exploration phase we are figuring out what are we building and why, so at this point the content tends to be more abstract. We’re trying to understand the problem space and our users so we shouldn’t be laser focused on details, only structure and navigation matter. Abstraction allows us to have a more open conversation with users that’s not solution focused. Sometimes we choose metaphors that allow us to be on the same playing field as our users to deconstruct their world more easily. We present this in the form of manipulatives — small cut outs of UI or empty UI elements the customer can draw on during a quick participatory design session. Cutting and preparing manipulatives is also a fun team activity Delivery When we move into the delivery phase of design where our focus is on how are we building the product, content needs to reflect the customer’s world. We partner closely with our Product Manager to structure a script. Context in the form of relevant copy, charts, data and labels help support the the script and various paths the user can take when interacting with the prototype. Small details like the data ink, data points along with the correct labels help us make the prototype more realistic so the user doesn’t feel he’s stepping into an unfamiliar environment. Even though a prototype is still an experiment, using real data gives us a preview of design challenges like truncation or readability. We are lucky to have real data from our product. CSVJSON helps us convert files into JSON format so we can use the data with chart plugins and CRAFT. Collaborate to Refine Prototyping is fun and playful — too much that it can be easy to forget that there are also other people who are part of the process. Prototyping is also a social way to develop ideas with non designers so when choosing which tool to present our prototype in we need to keep in mind collaboration outside the design team, not just the final output. We use InVision to quickly convey flows along with annotations but it has a downside during this collaborative process. Annotations can leave room for interpretation since every stakeholder has his own vocabulary. Recently, a couple of our Engineers in Poland started using UXPin. At first it was used to sell their ideas but for usability testing, it has also become a common area where we can work off each others’ prototypes. They like the ability to duplicate prototypes, reshuffle screens so the prototypes can be updated quickly without having to write another long document of explanations. By iterating together we are able to create a common visual representation and move fast. UXPin came to the rescue when collaborating with cross regional teams. It’s an intuitive tool for non designers that allows them to duplicate the prototype and make their own playground too. Tools will continue to change so it’s important to have an open mindset and be flexible to learning and making judgments about when to switch tools to deliver the prototype on time to research. Architect Smartly Although we are on a time crunch when prototyping for research, we can find time to experiment by adjusting the way we build our prototype. Make a playground Our lead productdesigner Rohan Singh invented the hamster playground to go wild with design explorations. The hamster playground is an experimental space which comes in handy when we may need to quickly whip something up without messing the rest of the design. It started as a separate page in our sketch files and now this is also present in our prototyping workspace. When we design something in high fidelity, we become attached to the idea right away. This can cripple experimentation. We need that sacred space detached from the main prototype that allows us to experiment with animations or dynamic elements. The hamster playground can also be a portable whiteboard or pen and paper. Embrace libraries Libraries accelerate the prototyping process exponentially! For the tool you’re commonly using to prototype invest some time (hackathons or end of quarter) to create a pattern library of the most common interactions(this is not a static UI Kit). If the prototype we’re building has some of those common elements, we will save them into the library so other team members can reuse them on another project. Building an interactive library is time consuming but it pays off because it allows the team to easily drag, drop and combine elements like legos. Consolidate the flow We try to remove non essential items from the prototype and replace them with screenshots or turn them into loops so we can focus only on the area that matters for testing. Consolidation also forces us to not overwhelm the prototype with many artboards otherwise we risk having clunky interactions during testing. The other advantage of consolidating is that you can easily map out interactions by triggers, states and animations/transitions. Prepare Researchers for Success Our job is not done until research, our partners, understand what we built. As a best practice, set up some time with the researcher to review the prototype. Present the limitations, discrepancies in different browsers and devices and any other instructions that are critical for the success of the testing session. A short guide that outlines the different paths with screenshots of what the successful interactions look like can aid researchers a lot when they are writing the testing script. Ready, Set…Prototype! Just like marathoners, who intuitively know when to move fast, adjust and change direction, great prototypers work from principles to guide their process. Throughout the design process the scrum team constantly needs answers to many questions. By becoming an effective prototyper, not the master of x tool, you can help the team find the answers right away. The principles outlined above will guide your process so you are more aware of how you spend your time and to know when you’re prototyping too much, too little or the wrong thing. Organization doesn’t kill experimentation; it makes more time for playfulness and solving the big grey areas. This post originally appeared on Medium. Check it out here. Additional Resources Check out this great article to learn how our customers influence the Sumo Logic product and how UX research is key to improving overall experiences Curious to know how I ended up at Sumo Logic doing product design/user experience? I share my journey in this employee spotlight article. Love video games and data? Then you’ll love this article from one of our DevOps engineers on how we created our own game (Sumo Smash bros) to demonstrate the power of machine data

MySQL has always been among one of the top few database management systems used worldwide, according to DB-engines, one of the leading ranking websites. And thanks to the large open source community behind MySQL, it also solves a wide variety of use cases. In this blog post, we are going to focus on how to achieve transactional behavior with MySQL and Slick. We will also discuss how these transactions resulted in one of our production outages. But before going any further into the details, let’s first define what a database transaction is. In the context of relational databases, a sequence of operations that satisfies some common properties is known as a transaction. This common set of properties, which determine the behavior of these transactions are referred to as atomic, consistent, isolated and durable (ACID) properties. These properties are intended to guarantee the validity of the underlying data in case of power failure, errors, or other real-world problems. The ACID model talks about the basic supporting principles one should think about before designing database transactions. All of these principles are important for any mission-critical applications. One of the most popular storage engines we use in MySQL is InnoDB, whereas Slick is the modern database query and access library for Scala. Slick exposes the underlying data stored in these databases as Scala collections so that data stored onto these databases is seamlessly available. Database transactions come with their own set of overhead, especially in cases when we have long running queries wrapped in a transaction. Let’s understand the transaction behavior, which we get with Slick. Slick offers ways to execute transactions on MySQL. pre{ font-family: Consolas, Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace, serif; margin-bottom: 10px; overflow: auto; width: auto; padding: 5px; background-color: #eee; width: 650px!ie7; padding-bottom: 20px!ie7; max-height: 600px; } function fetchCustomers(): Observable<Customer[]> { ... } val a = (for { ns <- coffees.filter(_.name.startsWith("ESPRESSO")).map(_.name).result _ <- DBIO.seq(ns.map(n => coffees.filter(_.name === n).delete): _*) } yield ()).transactionally These transactions are executed with the help of the Auto-Commit feature provided by the InnoDB engine. We will go into this auto-commit feature later in this article, but first, let me tell you about an outage, which happened on our production services at Sumo Logic and resulted in one of the outages. For the rest of the article, I will be talking about one of our minor outages which happened due to this lack of understanding in this transaction behavior. Whenever any user fires a query, the query follows this course of action before getting started: Query metadata i.e. user, customerID is first sent to Service A Service A asks this common Amazon MySQL RDS for the number of concurrent sessions for this user running across all the instances for this Service A If the number is greater than some threshold we throttle the request and send 429 to the user. Otherwise, we just add the metadata of the session to the table stored in RDS All of these actions are executed within the scope of a single slick transaction. Recently we started receiving lots of lock wait timeouts on this Service A. On debugging further, we saw that from the time we started getting lots of lock wait timeouts, there was also an increase in the average CPU usage across the Service A cluster. Looking into some of these particular issues of lock wait timeouts, we noticed that whenever we had an instance in the cluster going through full GC cycles, that resulted in a higher number of lock wait timeouts across the cluster. But interestingly enough, these lock wait timeouts were all across the cluster and not isolated on the single instance, which suffered from full GC cycles. Based on that, we knew that full GC cycles on one of the nodes were somewhat responsible for causing those lock wait timeouts across the cluster. As already mentioned above, we used the transaction feature provided by slick to execute all of the actions as a single command. So the next logical step was to dig deeper into understanding the question: “how does Slick implement these transactions”? We found out that Slick uses the InnoDB feature of auto-commits to execute transactions. In the auto-commit disabled mode, the transaction is kept open until the transaction is committed from the client side, which essentially means that the connection executing the current transaction holds all the locks until the transaction is committed. Auto-Commit Documentation from the InnoDB Manual In InnoDB, all user activity occurs inside a transaction. If auto-commit mode is enabled, each SQL statement forms a single transaction on its own. By default, MySQL starts the session for each new connection with auto-commit enabled, so MySQL does a commit after each SQL statement if that statement did not return an error. If a statement returns an error, the commit or rollback behavior depends on the error. See Section 14.21.4, “InnoDB Error Handling”. A session that has auto-commit enabled can perform a multiple-statement transaction by starting it with an explicit START TRANSACTION or BEGIN statement and ending it with a COMMIT or ROLLBACK statement. See Section 13.3.1, “START TRANSACTION, COMMIT, and ROLLBACK Syntax”. If auto-commit mode is disabled within a session with SET auto-commit = 0, the session always has a transaction open. A COMMIT or ROLLBACK statement ends the current transaction and a new one starts. Pay attention to the last sentence above. This means if auto-commit is disabled, then the transaction is open, which means all the locks are still with this transaction. All the locks, in this case, will be released only when we explicitly COMMIT the transaction. So in our case, our inability to execute the remaining commands within the transaction due to a high GC, meant that we were still holding onto the locks on the table and therefore would mean that other JVMs executing the transaction touching the same table (which is, in fact, the case), would also suffer from high latencies. But we needed to be sure that was the case on our production environments. So we went ahead with reproducing the production issue on the local testbed, making sure that locks were still held by the transaction on the node undergoing high GC cycles. Steps to Reproduce the High DB Latencies on One JVM Due to GC Pauses on Another JVM Step One We needed some way to know when the queries in the transactions were actually getting executed by the MySQL server. mysql> SET global general_log = 1; mysql> SET global log_output = 'table'; mysql> SELECT * from mysql.general_log; So MySQL general logs show the recent queries which were executed by the server. Step Two We needed two different transactions to execute at the same time in different JVMs to understand this lock wait timeout. Transaction One: val query = (for { ns <- userSessions.filter(_.email.startsWith(name)).length.result _ <- { println(ns) if (ns > n) DBIOAction.seq(userSessions += userSession) else DBIOAction.successful(()) } } yield ()).transactionally db.run(query) Transaction Two: db.run(userSessions.filter(_.id === id).delete) Step Three Now we needed to simulate the long GC pauses or pauses in one of the JVMs to mimic the production environments. On mimicking those long pauses, we need to monitor the mysql.general logs for finding out when did the command reached the MySQL server for asking to be executed. The below chart depicts the order of SQL statements getting executed on both JVMs: JVM 1( Adding the session of the user ) JVM 2 ( Delete the session of the user if present ) SET auto-commit = 0 ( as in false ) SELECT count(*) FROM USERS where User_id = “temp” ( LOCKS ACQUIRED ) SET auto-commit = 0 INSERT INTO USERS user_session DELETE FROM USERS where sessionID = “121” ( Started ) INTRODUCED HIGH LATENCY ON THE CLIENT SIDE FOR 40 SECONDS DELETE OPERATION IS BLOCKED DUE TO THE WAITING ON THE LOCK COMMIT DELETE FROM USERS where sessionID = “121” ( Completed ) COMMIT In the below image, you can see the SQL statements getting executed on both the JVMs: This image shows the lock wait time of around 40 seconds on JVM 2 on “Delete SQL” command: We can clearly see from the logs how pauses in one JVM causes high latencies across the different JVMs querying on MySQL servers. Handling Such Scenarios with MySQL We more than often need to handle this kind of scenario where we need to execute MySQL transactions across the JVMs. So how can we achieve low MySQL latencies for transactions even in cases of pauses in one of the JVMs? Here are some solutions: Using Stored Procedures With stored procedures, we could easily extract out this throttling logic into a function call and store it as a function on MySQL server. They can be easily called out by clients with appropriate arguments and they can be executed all at once on the server side without being afraid of the client side pauses. Along with the use of transactions in the procedures, we can ensure that they are executed atomically and results are hence consistent for the entire duration of the transaction. Delimit Multiple Queries With this, we can create transactions on the client side and execute them atomically on the server side without being afraid of the pauses. Note: You will need to enable allowMultiQueries=true because this flag allows batching multiple queries together into a single query and hence you will be able to run transactions as a single query. Better Indexes on the Table With better indices, we can ensure that while executing SELECT statements with WHERE condition we touch minimal rows and hence ensuring minimal row locks. Let’s suppose we don’t have any index on the table, then in that case for any select statement, we need to take a shared row lock on all the rows of the table, which will mean that during the execution phase of this transaction all the delete or updates would be blocked. So it’s generally advised to have WHERE condition in SELECT to be on an index. Lower Isolation Levels for executing Transactions With READ UNCOMMITTED isolation levels, we can always read the rows which still have not been committed. Additional Resources Want more articles like this? Check out the Sumo Logic blog for more technical content! Read this blog to learn how to triage test failures in a continuous delivery lifecycle Check out this article for some clear-cut strategies on how to manage long-running API queries using RxJS Visit the Sumo Logic App for MySQL page to learn about cloud-native monitoring for MySQL https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

We are a multi-cloud managed services provider (MSP) that works strictly with Amazon Web Services (AWS), Google Cloud Platform (GCP) and Microsoft Azure. We’ve been working with enterprise public cloud leaders in Europe since 2012 and our goal is to provide fully managed public cloud transformation to our customers. This comes in many forms, including infrastructure automation, and application development and managed services that take an IP-driven approach to public cloud provisioning, optimization and ongoing management.

It’s common for a modern single-page application (SPA) to fetch data from the server via a REST API call. There are, though, many challenges related to this approach, one of which is handling long-running queries. Let us explain how we rely on the RxJS library to help us with this task.

We are very excited to announce a new capability for our Amazon S3 sources. Until recently, the only method Sumo Logic used for discovering new data in an S3 bucket was periodic polling. However, with our new notification-based approach, users can now configure S3 sources such that Sumo Logic is notified immediately (via AWS SNS) whenever a new item is added to an S3 bucket, eliminating the need to wait for new objects to be discovered via periodic polling. This capability is available today for all S3 sources, either new or pre-existing.

In this SnapSecChat video, Sumo Logic CSO George Gerchow recaps his favorite highlights from the HackerOne security conference, Security@.

Puppet is a software configuration management and deployment tool that is available both as an open source tool and commercial software. It’s most commonly used on Linux and Windows to pull the strings on multiple application servers at once. It includes its own declarative language to describe system configurations. In today’s cloud environments that consist of hundreds of distributed machines, Puppet can help in reducing development time and resources by automatically applying these configurations. Just like any other DevOps tool there can be errors and configuration issues. However, with the new Sumo Logic Puppet integration and application, customers can now leverage the Sumo Logic platform to help monitor Puppet performance, configurations and errors. Puppet Architecture and Logging Puppet can apply required configurations across new and existing servers or nodes. You can configure systems with Puppet either in a client-server architecture or in stand-alone architectures. The client-server architecture is the most commonly used architecture for Puppet implementations. Puppet agents apply the required changes and send the reports to the Puppet master describing the run and details of the client resources. These reports can help answer questions like “how often are the resources modified,” “how many events were successful in the past day” and “what was the status of the most recent run?” In addition to reports, Puppet also generates an extensive set of log files. From a reporting and monitoring perspective, the two log files of interest are the Puppet server logs and the HTTP request logs. Puppet server messages and errors are logged to the file /var/log/puppetlabs/puppetserver/puppetserver.log. Logging can be configured using the /etc/puppetlabs/puppetserver/logback.xml file, which can be used to monitor the health of the server. The /var/log/puppetlabs/puppetserver/puppetserver-access.log file contains HTTP traffic being routed via your Puppet deployment. This logging can be handled using the configuration file: /etc/puppetlabs/puppetserver/request-logging.xml. Puppet agent requests to the master are logged into this file. Sumo Logic Puppet App The Sumo Logic Puppet app is designed to effectively manage and monitor Puppet metrics, events and errors across your deployments. With Sumo Logic dashboards you will be able to easily identify: Unique nodes Puppet node runs activity Service times Catalog application times Puppet execution times Resource transition (failures, out-of-sync, modifications, etc.) Error rates and causes Installation In order to get started, the app requires three data sources: Puppet server logs Puppet access logs Puppet reports The puppet server logs and puppet access logs are present in the directory var/log/puppetlabs/puppetserver/. Configure separate local file resources for both of these log files. Puppet reports are generated as yaml files. These need to be converted into JSON files before ingesting into Sumo Logic. To ingest Puppet reports, you must configure a script source. Once the log sources are configured, the Sumo Logic app can be installed. Simply navigate to the apps Catalog in your Sumo Logic instance and add the Puppet app to the library after providing the sources configured in the previous step. For more details on app configuration, please see instructions on Sumo Logic’s DocHub. Sumo Logic Puppet App Visualizations In any given Puppet deployment, there can be a large number of nodes. Some of the nodes may be faulty or others may be very active. The Puppet server manages the nodes and it may be suffering from issues itself. The Sumo Logic Puppet app consists of predefined dashboards and search queries which help you monitor the Puppet infrastructure. The Puppet Overview dashboard shown below gives you an overview of activity across nodes and servers. If a Puppet node is failing, you can quickly find out when the node made requests, what version it is running on and how much time it is taking to prepare the catalog for the node by the server. Puppet Overview Dashboard Let’s take a closer look at the Error Rate panel. The Error Rate panel displays the error rates per hour. This helps identify when error rates spiked, and by clicking on the panel, you can identify the root cause on either the node-level or the server-level via the Puppet Error Analysis dashboard. In addition, this dashboard highlights the most erroneous nodes along with the most recent errors and warnings. With this information, it will be easier to drill down into the root cause of the issues. The panel Top Erroneous Nodes helps in identifying the most unstable nodes. Drill down to view the search query by clicking on the “Show in Search” icon highlighted in the above screenshot. The node name and the errors can be easily identified and corrective actions can be performed by reviewing the messages in the search result as shown in the screenshot below: With the help of information on the Puppet – Node Puppet Run Analysis dashboard, node health can be easily determined across different deployments such as production and pre-production. The “Slowest Nodes by Catalog Application Time” panel helps you determine the slowest nodes, which can potentially be indicative of problems and issues within those nodes. From there, you can reference the Puppet Error Analysis dashboard to determine the root cause. The “Resource Status” panel helps you quickly determine the status of various resources, further details around which can be obtained by drilling down to the query behind it. By reviewing the panels on this dashboard, highest failing or out-of-sync resources can be easily determined, which may be indicative of problems on respective nodes. To compare the average catalog application times, take a look at the “Average Catalog Application Time” and “Slowest Nodes by Catalog Application Time” panels. The resources panels show resources that failed, modified, are out-of-sync and skipped. Drilling down to the queries of the panels will help in determining the exact resource list with the selected status. Note: All the panels in the Puppet Node Puppet Run Analysis dashboard and some panels of the Puppet Overview dashboard can be filtered based on the environment, such as production, pre-production, etc. as shown below: Get Started Now! The Sumo Logic app for Puppet monitors your entire Puppet infrastructure potentially spanning hundreds of nodes and helps determine the right corrective and preventative actions. To get started check out the Sumo Logic Puppet app help doc. If you don’t yet have a Sumo Logic account, you can sign up for a free trial today. For more great DevOps-focused reads, check out the Sumo Logic blog.

The world is changing. The way we do business, the way we communicate, and the way we secure the enterprise are all vastly different today than they were 20 years ago. This natural evolution of technology innovation is powered by the cloud, which has not only freed teams from on-premises security infrastructure, but has also provided them with the resources and agility needed to automate mundane tasks. The reality is that we have to automate in the enterprise if we are to remain relevant in an increasingly competitive digital world. Automation and security are a natural pairing, and when we think about the broader cybersecurity skills talent gap, we really should be thinking about how we can replace simple tasks through automation to make way for teams and security practitioners to be more innovative, focused and strategic. A Dynamic Duo That’s why Sumo Logic and our partner, The Pokemon Co. International, are all in on bringing together the tech and security innovations of today and using those tools and techniques to completely redefine how we do security operations, starting with creating a new model for how security operations center (SOC) should be structured and how it should function. So how exactly are we teaming up to build a modern day SOC, and what does it look like in terms of techniques, talent and tooling? We’ll get into that, and more, in this blog post. Three Pillars of the Modern Day SOC Adopt Military InfoSec Techniques The first pillar is all about mindset and adopting a new level of rigor and way of thinking for security. Both the Sumo Logic and Pokemon security teams are built on the backbone of a military technique called the OODA loop, which was originally coined by U.S. Air Force fighter pilot and Pentagon consultant of the late twentieth century, John Boyd. Boyd created the OODA loop to implement a change in military doctrine that focused on an air-to-air combat model. OODA stands for observe, orient, decide and act, and Boyd’s thinking was that if you followed this model and ensured that your OODA loop was faster than that of your adversary’s, then you’d win the conflict. Applying that to today’s modern security operations, all of the decisions made by your security leadership — whether it’s around the people, process or tools you’re using — should be aimed at reducing your OODA loop to a point where, when a situation happens, or when you’re preparing for a situation, you can easily follow the protocol to observe the behavior, orient yourself, make effective and efficient decisions, and then act upon those decisions. Sound familiar? This approach is almost identical to most current incident response and security protocols, because we live in an environment where every six, 12 or 24 months we’re seeing more tactics and techniques changing. That’s why the SOC of the future is going to be dependent on a security team’s ability to break down barriers and abandon older schools of thought for faster decision making models like the OODA loop. This model is also applicable across an organization to encourage teams to be more efficient and collaborative cross-departmentally, and to move faster and with greater confidence in order to achieve mutually beneficial business goals. Build and Maintain an Agile Team But it’s not enough to have the right processes in place. You also need the right people that are collectively and transparently working towards the same shared goal. Historically, security has been full of naysayers, but it’s time to shift our mindset to that of transparency and enablement, where security teams are plugged into other departments and are able to move forward with their programs as quickly and as securely as they can without creating bottlenecks. This dotted line approach is how Pokemon operates and it’s allowed the security team to share information horizontally, which empowers development, operations, finance and other cross-functional teams to also move forward in true DevSecOps spirit. One of the main reasons why this new and modern Sumo Logic security team structure has been successful is because it’s enabled each function — data protection/privacy, SOC, DevSecOps and federal — to work in unison not only with each other, but also cross-departmentally. In addition to knowing how to structure your security team, you also need to know what to look for when recruiting new talent. Here are three tips from Pokemon’s Director of Information Security and Data Protection Officer, John Visneski: Go Against the Grain. Unfortunately there are no purple security unicorns out there. Instead of finding the “ideal” security professional, go against the grain. Find people with the attitude and aptitude to succeed, regardless of direct security experience. The threat environment is changing rapidly, and burnout can happen fast, which is why it’s more important to have someone on in your team with those two qualities.Why? No one can know everything about security and sometimes you have to adapt and throw old rules and mindsets out the window. Prioritize an Operational Mindset. QAs and test engineers are good at automation and finding gaps in seams, very applicable to security. Best Security Engineers didn’t know a think about security before joining Pokemon, but he had a valuable skill set.Find talent pools that know how the sausage is made. Best and brightest security professionals didn’t even start out in security but their value add is that they are problem solvers first, and security pros secondary. Think Transparency. The goal is to get your security team to a point where they’re sharing information at a rapid enough pace and integrating themselves with the rest of the business. This allows for core functions to help solve each other’s problems and share use-cases, and it can only be successful if you create a culture that is open and transparent. The bottom line: Don’t be afraid to think outside of the box when it comes to recruiting talent. It’s more important to build a team based on want, desire and rigor, which is why bringing in folks with military experience has been vital to both Sumo Logic’s and Pokemon’s security strategies. Security skills can be learned. What delivers real value to a company are people that have a desire to be there, a thirst for knowledge and the capability to execute on the job. Build a Modern Day Security Stack Now that you have your process, and your people, you need your third pillar — tools sets. This is the Sumo Logic reference architecture that empowers us to be more secure and agile. You’ll notice that all of these providers are either born in the cloud or are open source. The Sumo Logic platform is at the core of this stack, but its these partnerships and tools that enable us to deliver our cloud-native machine data analytics as a service, and provide SIEM capabilities that easily prioritize and correlate sophisticated security threats in the most flexible way possible for our customers. We want to grow and transform with our own customer’s modern application stacks and cloud architectures as they digitally transform. Pokemon has a very similar approach to their security stack: The driving force behind Pokemon’s modern toolset is the move away from old school customer mentality of presenting a budget and asking for services. The customer-vendor relationship needs to mirror a two way partnership with mutually invested interests and clear benefits on both sides. Three vendors — AWS, CrowdStrike and Sumo Logic — comprise the core base of the Pokemon security platform, and the remainder of the stack is modular in nature. This plug and play model is key as the security and threat environments continue to evolve because it allows for flexibility in swapping in and out new vendors/tools as they come along. As long as the foundation of the platform is strong, the rest of the stack can evolve to match the current needs of the threat landscape. Our Ideal Model May Not Be Yours We’ve given you a peek inside the security kimono, but it’s important to remember that every organization is different, and what works for Pokemon or Sumo Logic may not work for every particular team dynamic. While you can use our respective approaches as a guide to implement your own modern day security operations, the biggest takeaway here is that you find a framework that is appropriate for your organization’s goals and that will help you build success and agility within your security team and across the business. The threat landscape is only going to grow more complex, technologies more advanced and attackers more sophisticated. If you truly want to stay ahead of those trends, then you’ve got to be progressive in how you think about your security stack, teams and operations. Because regardless of whether you’re an on-premises, hybrid or cloud environment, the industry and business are going to leave you no choice but to adopt a modern application stack whether you want to or not. Additional Resources Learn about Sumo Logic's security analytics capabilities in this short video. Hear how Sumo Logic has teamed up with HackerOne to take a DevSecOps approach to bug bounties in this SnapSecChat video. Learn how Pokemon leveraged Sumo Logic to manage its data privacy and GDPR compliance program and improve its security posture.

Pitney Bowes explains how they leveraged the Sumo Logic platform to successfully excute their cloud migration strategy to AWS.

At Sumo Logic’s annual user conference, Illuminate, we announced a strategic partnership with eSentire, the largest pure-play managed detection and response (MDR) provider, that will leverage security analytics from the Sumo Logic platform to deliver full spectrum visibility across the organization, eliminating common blind spots that are easily exploited by attackers. Today’s digital organizations operate on a wide range of modern applications, cloud infrastructures and methodologies such as DevSecOps, that accumulate and release massive amounts of data. If that data is managed incorrectly, it could allow malicious threats to slip through the cracks and negatively impact the business. This partnership combines the innovative MDR and cloud-based SIEM technologies from eSentire and Sumo Logic, respectively, that provide customers with improved analytics and actionable intelligence to rapidly detect and investigate machine data to identify potential threats to cloud or hybrid environments and strengthen overall security posture. Watch the video to learn more about this joint effort as well as the broader security, MDR, and cloud SIEM market outlook from Jabari Norton, VP global partner sales & alliances at Sumo Logic, Sean Blenkhorn, field CTO and VP sales engineering & advisory services at eSentire, and Dave Monahan, managing research director at analyst firm, EMA. For more details on the specifics of this partnership, read the joint press release.

https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

While deep learning, machine learning and artificial intelligence (AI) may seem to be used synonymously, there are clear differences. One school of thought is that artificial intelligence is a larger umbrella category under which machine learning falls and deep learning falls under machine learning. Therefore, while everything that is categorized as deep learning or machine learning is part of the artificial intelligence field, not everything that is machine learning will be deep learning.

Watch the video for more Illuminate 2018 highlights, and to hear Ramin talk about our mission to democratize machine data for everyone. Stay tuned for part two, which looks at some of the new product announcements and innovations we announced during the keynote sessions.

Tag-based metrics are typically used by IT operations and DevOps teams to make it easier to design and scale their systems. Tags help you to make sense of metrics by allowing you to filter on things like host, cluster, services, etc. However, knowing which tags to use, and when, can be confusing. For instance, have you ever wondered about the difference between intrinsic tags (or dimensions) and meta tags with respect to custom application metrics? If so, you’re not alone. It is pretty common to get the two confused, but don’t worry because this blog post will help explain the difference. Before We Get Started First, let’s start with some background. Metrics in Carbon 2.0 take on the following format: Note that there are two spaces between intrinsic_tags and meta_tags. If a tag is listed before the double space, then it is an intrinsic tag. If a tag is listed after the double space, then it is a meta tag. Meta_tags are also optional. If no meta_tags are provided, there must be two spaces between intrinsic_tags and value. Some examples of Carbon 2.0 metrics might be: Understanding Intrinsic Tags Intrinsic tags may also be referred to as dimensions and are metric identifiers. If you have two data points sent with same set of dimension values then they will be values in the same metric time series. In the examples above, each metric has different dimensions so they will be separate time series. Understanding Meta Tags On the other hand, meta tags are not used as metric identifiers. This means that if two data points have the same intrinsic tags or dimensions, but different meta tags, they will still be values in the same metric time series. Meta tags are meant to be used in addition to intrinsic tags so that you can more conveniently select the metrics. Let’s Look at an Example To make that more clear, let’s use another example. Let’s say that you have 100 servers in your cluster that are reporting host metrics like “metric=cpu_idle.” This would be an intrinsic tag. You may also want to track the version of your code running on that cluster. Now if you put the code version in an intrinsic tag, you’ll get a completely new set of metrics every time you upgrade to a new code version. Unless you want to maintain the metrics “history” of the old code version, you probably don’t want this behavior. However, if you put the version in a meta tag instead then you will be able to change the version without creating a new set of metrics for your cluster. To take the example even further, let’s say you have upgraded half of your cluster to a new version and want to compare the CPU idle of the old and new code version. You could do this in Sumo Logic using the query “metric = cpu_idle | avg by version.” Knowing the Difference To summarize, if you want two values of a given tag to be separate metrics at the same time then the values should be an intrinsic tag and not a meta tag. Hopefully this clears up some of the confusion regarding intrinsic versus meta tags. By tagging your metrics appropriately you will make them easier to search and ensure that you are tracking all the metrics you expect. If you already have a Sumo Logic account, then you are ready to start ingesting custom metrics. If you are new to Sumo Logic, start by signing up for a free account here. Additional Resources Learn how to accelerate data analytics with Sumo Logic’s Logs to Metrics solution in this blog Want to know how to transform Graphite data into metadata-rich metrics? Check out our Metrics Rules solution Read the case study to learn how Paf leveraged the Sumo Logic platform to derive critical insights that enabled them to analyze log and metric data, perform root-cause analysis, and monitor apps and infrastructure

Learn why decluttering complex data in legends is hard and how Sumo Logic software engineers use metadata, heuristics & query results to make it easier.

This year, at Sumo Logic’s third annual user conference, Illuminate 2018, we presented Sumo Logic Notebooks as a way to do data science in Sumo Logic. Sumo Logic Notebooks are an experimental feature that integrate Sumo Logic, notebooks and common machine learning frameworks. They are a bold attempt to go beyond what the current Sumo Logic product has to offer and enable a data science workflow leveraging our core platform. Why Notebooks? In the data science world, notebooks have emerged as an important tool to do data science. Notebooks are active documents that are created by individuals or groups to write and run code, display results, and share outcomes and insights. Like every other story, a data science notebook follows a structure that is typical for its genre. We usually have four parts. We (a) start with defining a data set, (b) continue to clean and prepare the data, (c) perform some modeling using the data, and (d) interpret the results. In essence, a notebook should record an explanation of why experiments were initiated, how they were performed, and then display the results. Anatomy of a Notebook A notebook segments a computation in individual steps called paragraphs. A paragraph contains an input and an output section. Each paragraph executes separately and modifies the global state of the notebook. State can be defined as the ensemble of all relevant variables, memories, and registers. Paragraphs must not necessarily contain computations, but also can contain text or visualizations to illustrate the workings of the code. The input section (blue) will contain the instruction to the notebook execution engine (sometimes called kernel or interpreter). The output section (green) will display a trace of the paragraph’s execution and/or an intermediate result. In addition, the notebook software will expose some controls (purple) for managing and versioning notebook content as well as operational aspects such as starting and stopping executions. Human Speed vs Machine Speed The power of the notebook roots in its ability to segment and then slow down computation. Common executions of computer programs are done at machine speed. Machine speed suggests that when a program is submitted to the processor for execution, it will run from start to end as fast as possible and only block for IO or user input. Consequently, the state of the program changes so fast that it is neither observable, nor modifiable by humans. Programmers would typically attach debuggers physically or virtually to stop programs during execution at so-called breakpoints and read out and analyze their state. Thus, they would slow down execution to human speed. Notebooks make interrogating the state more explicit. Certain paragraphs are dedicated to make progress in the computation, i.e., advance the state, whereas other paragraphs would simply serve to read out and display the state. Moreover, it is possible to rewind state during execution by overwriting certain variables. It is also simple to kill the current execution, thereby deleting the state and starting anew. Notebooks as an Enabler for Productivity Notebooks increase productivity, because they allow for incremental improvement. It is cheap to modify code and rerun only the relevant paragraph. So when developing a notebook, the user builds up state and then iterates on that state until progress is made. Running a stand-alone program on the contrary will incur more setup time and might be prone to side-effects. A notebook will most likely keep all its state in the working memory whereas every new execution of a stand-alone program will need to build up the state on every time it is run. This takes more time and the required IO operations might fail. Working off a program state in the memory and iterating on that proved to be very efficient. This is particularly true for data scientists, as their programs usually deal with a large amount of data that has to be loaded in and out of memory as well as computations that can be time-consuming. From an the organizational point of view, notebooks are a valuable tool for knowledge management. As they are designed to be self-contained, sharable units of knowledge, they amend themselves for: Knowledge transfer Auditing and validation Collaboration Notebooks at Sumo Logic At Sumo Logic, we expose notebooks as an experimental feature to empower users to build custom models and analytics pipelines on top of log metrics data sets. The notebooks provide the framework to structure a thought process. This thought process can be aimed at delivering a special kind of insight or outcome. It could be drilling down on a search. Or an analysis specific to a vertical or an organization. We provide notebooks to enable users to go beyond what Sumo Logic operators have to offer, and train and test custom machine learning (ML) algorithms on your data. Inside notebooks we deliver data using data frames as a core data structure. Data frames make it easy to integrate logs and metrics with third-party data. Moreover, we integrate with other leading data wrangling, model management and visualization tools/services to provide a blend of the best technologies to create value with data. Technology Stack Sumo Logic Notebooks are an integration of several software packages to make it easy to define data sets using the Sumo Query language and use the result data set as a data frame in common machine learning frameworks. Notebooks are delivered as a Docker container and can therefore be installed on laptops or cloud instances without much effort. The most common machine learning libraries such as Apache Spark, pandas, and TensorFlow are pre-installed, but others are easy to add through python’s pip installer, or using apt-get and other package management software from the command line. Changes can be made persistent by committing the Docker image. The key of Sumo Logic Notebooks is the integration of the Sumo Logic API data adapter with Apache Spark. After a query has been submitted, the adapter will load the data and ingest it into Spark. From there we can switch over to a python/pandas environment or continue with Spark. The notebook software provides the interface to specify data science workflows. Best Practices for Writing Notebooks #1 One notebook, one focus A notebook contains a complete record of procedures, data, and thoughts to pass on to other people. For that purpose, they need to be focused. Although it is tempting to put everything in one place, this might be confusing for users. Better write two or more notebooks than overloading a single notebook. #2 State is explicit A common source of confusion is that program state gets passed on between paragraphs through hidden variables. The set of variables that represent the interface between two subsequent paragraphs should be made explicit. Referencing variables from other paragraphs than the previous one should be avoided. #3 Push code in modules A notebook integrates code, it is not a tool for code development. That would be an Integrated Development Environment (IDE). Therefore, a notebook should one contain glue code and maybe one core algorithm. All other code should be developed in an IDE, unit tested, version controlled, and then imported via libraries in the notebook. Modularity and all other good software engineering practices are still valid in notebooks. As in practice number one too much code clutters the notebook and distracts from the original purpose or analysis goal. #4 Use speaking variables and tidy up your code Notebooks are meant to be shared and read by others. Others might not have an easy time following our thought process, if we did not come up with good, self-explaining names. Tidying up the code goes a long way, too. Notebooks impose an even higher standard than traditional code on quality. #5 Label diagrams A picture is worth a thousand words. A diagram, however, will need some words to label axes, describe lines and dots, and comprehend other important informations such sample size, etc. A reader can have a hard time to seize the proportion or importance of a diagram without that information. Also keep in mind that diagrams are easily copy-pasted from the notebook into other documents or in chats. Then they lose the context of the notebook in which they were developed. Bottom Line The segmentation of a thought process is what fuels the power of the notebook. Facilitating incremental improvements when iterating on a problem boosts productivity. Sumo Logic enables the adoption of notebooks to foster the use of data science with logs and metrics data. Additional Resources Visit our Sumo Logic Notebooks documentation page to get started Check out Sumo Logic Notebooks on DockerHub or Read the Docs Read our latest press release announcing new platform innovations, including our new Data Science Insights innovation

Sumo Logic provides a scalable, easily deployable, serverless solution to monitor Azure Services connected with Azure Monitor that supports both logs and metrics data from Azure. Use a Sumo Logic serverless solution to monitor your Azure Services, which are connected with Azure Monitor/Event Hub. This Azure-Sumo Logic pipeline helps you integrate the following types of data from your Azure Services with Sumo Logic.

Day two of Illuminate, Sumo Logic’s annual user conference, started with a security bang, hearing from our founders, investors, customers and a special guest (keep reading to see who)! If you were unable attend the keynote in person, or watch via the Facebook Livestream, we’ve recapped the highlights below for you. If you are curious about the day one keynote, check out that recap blog post, as well. #1: Dial Tones are Dead, But Reliability Is Forever Two of our founders, Christian Beedgen and Bruno Kurtic, took the stage Thursday morning to kick off the second day keynote talk, and they did not disappoint. Sumo Logic founders Bruno Kurtic (left) and Christian Beedgen (right) kicking off the day two Illuminate keynote Although the presentation was full of cat memes, penguins and friendly banter, they delivered an earnest message: reliability, availability and performance are important to our customers, and are important to us at Sumo Logic. But hiccups happen, it’s inevitable, and that’s why Sumo Logic is committed to constantly monitoring for any hiccups so that we can troubleshoot instantly when they happen. The bottom line: our aspiration at Sumo Logic is to be the dial tone for those times when you absolutely need Sumo Logic to work. And we do that through total transparency. Our entire team has spent time on building a reliable service, built on transparency and constant improvement. It really is that simple. #2: The Platform is the Key to Democratizing Machine Data (and, Penguins) We also announced a number of new platform enhancements, solutions and innovations at Illuminate, all with the goal of improving our customers’ experiences. All of that goodness can be found in a number of places (linked at the end of this article), but what was most exciting to hear from Bruno and Christian on stage was what Sumo Logic is doing to address major macro trends. The first being proliferation of users and access. What we’ve seen from our customers, is that the Sumo Logic platform is brought into a specific group, like the security team, or the development team, and then it spreads like wildfire, until the entire company (or all of the penguins) wants access to the rich data insights. That’s why we’ve taken an API-first approach to everything we do. To keep your workloads running around the globe, we now have 20 availability zones across five regions and we will continue to expand to meet customer needs. The second being cloud scale economics because Moore’s Law is, in fact, real. Data ingest trends are going up, and for years our customers have relied on Sumo Logic to manage mission-critical data in order to keep their modern applications running and secured. Not all data is created equal, and different data sets have different requirements. Sometimes, it can be a challenge to store data outside of the Sumo Logic platform, which is why our customers now will have brand new capabilities for basic and cold storage within Sumo Logic. (Christian can confirm that the basic storage is still secure — by packs of wolves). The third trend is around the unification of modern apps and machine data. While the industry is buzzing about observability, one size does not fit all. To address this challenge, the Sumo Logic team asked, what can we do to deliver on the vision of unification? The answer is in the data. For the first time ever, we will deliver the State of Modern Applications report live, where customers can push their data to dynamic dashboards, and all of this information will be accessible in new easy to read charts that are API-first, templatized and most importantly, unified. Stay tuned for more on the launch of this new site! #3: The State of Security from Greylock, AB InBev and Pokemon One of my favorite highlights of the second day keynote was the security panel, moderated by our very own CSO, George Gerchow, with guests from one of our top investors, Greylock Partners, and two of our customers, Anheuser-Busch InBev (AB InBev) and Pokemon. From left to right: George Gerchow, CSO, Sumo Logic; Sara Guo, partner, Greylock; Khelan Bhatt, global director, security architecture, AB InBev; John Visneski, director infosecurity & DPO, Pokemon Sara Guo, general partner at Greylock, spoke about three constantly changing trends, or waves, she’s tracking in security, and what she looks when her firm is considering an investment: the environment, the business and the attackers. We all know the IT environment is changing drastically, and as it moves away from on-premises protection, it’s not a simple lift and shift process, we have to actually do security differently. Keeping abreast of attacker innovation is also important for enterprises, especially as cybersecurity resources continue to be sparse. We have to be able to scale our products, automate, know where our data lives and come together as a defensive community. When you think of Anheuser-Busch, you most likely think of beer, not digital transformation or cybersecurity. But there’s actually a deep connection, said Khelan Bhatt, global director, security architecture, AB InBev. As the largest beer distributor in the world, Anheuser Busch has 500 different breweries (brands) in all corners of the world, and each one has its own industrial IoT components that are sending data back to massive enterprise data lakes. The bigger these lakes get, the bigger targets they become to attackers. Sumo Logic has played a big part in helping the AB InBev security team digitally transform their operations, and building secure enterprise data lakes to maintain their strong connection to the consumer while keeping that data secure. John Visneski, director of information security and data protection officer (DPO) for the Pokémon Company International had an interesting take on how he and his team approach security. Be a problem solver first, and a security pro second. Although John brought on Sumo Logic to help him fulfill security and General Data Protection Regulation (GDPR) requirements, our platform has become a key business intelligence tool at Pokemon. With over 300 million active users, Pokemon collects sensitive personally identifiable information (PII) from children, including names, addresses and some geolocation data. Sumo Logic has been key for helping John and his team deliver on the company’s core values: providing child and customer safety, trust (and uninterrupted fun)! #4: Being a Leader Means Being You, First and Foremost When our very special guest, former CIA Director George Tenet, took the stage, I did not expect to walk away with some inspiring leadership advice. In a fireside chat with our CEO, Ramin Sayar, George talked about how technology has changed the threat landscape, and how nation-state actors are leveraging the pervasiveness of data to get inside our networks and businesses. Data is a powerful tool that can be used for good or bad. At Sumo Logic, we’re in it for the good. George also talked about what it means to be a leader and how to remain steadfast, even in times of uncertainty. Leaders have to lead within the context of who they are as human beings. If they try to adopt a persona of someone else, it destroys their credibility. The key to leaderships is self awareness of who you are, and understanding your limitations so that you can hire smart, talented people to fill those gaps. Leaders don’t create followers, they create other leaders. And that’s a wrap for Sumo Logic’s second annual user conference. Thanks to everyone who attended and supported the event. If we didn’t see you at Illuminate over the last two days, we hope you can join us next year! Additional Resources For data-driven industry insights, check out Sumo Logic’s third annual ‘State of Modern Applications and DevSecOps in the Cloud’ report. You can read about our latest platform innovations in our press release, or check out the cloud SIEM solution and Global Intelligence Service blogs. Check out our recent blog for a recap of the day one Illuminate keynote.

Today kicked off day one of Sumo Logic’s second annual user conference, Illuminate, and there was no better way to start the day than with a keynote presentation from our CEO, Ramin Sayar, and some of our most respected and valued customers, Samsung SmartThings and Major League Baseball (MLB). The event was completely sold out and the buzz and excitement could be felt as customers, industry experts, thought leaders, peers, partners and employees made their way to the main stage. If you were unable to catch the talk in person or tune in for the Facebook livestream, then read on for the top five highlights from the day one keynote. #1: Together, We’ve Cracked the Code to Machine Data At Sumo Logic, we’re experts in all things data. But, to make sure we weren’t biased, we partnered with 451 Research earlier this year to better understand how the industry is using machine data to improve overall customer experiences in today’s digital world. We found that 60 percent of enterprises are using machine data analytics using for business and customer insights, and to help support digital initiatives, usage and app performance. These unique findings have validated what we’ve been seeing within our own customer base over the past eight years — together, we can democratize machine data to make it easily accessible, understandable and beneficial to all teams within an organization. That’s why, as Ramin shared during the keynote, we’ve committed to hosting more meet-ups and global training and certification sessions, and providing more documentation, videos, Slack channels and other resources for our growing user base — all with the goal of ‘lluminating’ machine data for the masses, and to help customers win in today’s analytics economy. #2: Ask, and You Shall Receive Continued Platform Enhancements Day one was also a big day for some pretty significant platform enhancements and new solutions centered on three core areas: development, security and operations. The Sumo Logic dev and engineering teams have been hard at work, and have over 50 significant releases to show for it, all focused on meeting our customer’s evolving needs. Some of the newer releases on the Ops analytics side include Search Templates and Logs to Metrics. Search Templates empower non-technical users like customer support and product management, to leverage Sumo Logic’s powerful analytics without learning the query language. Logs to Metrics allow users to extract business KPIs from logs and cost-effectively convert them to high performance metrics for long-term trending and analysis. We’ve been hard at work on the security side of things as well, and are happy to announce the new cloud SIEM solution that’s going to take security analytics one step further. Our customers have been shouting from the rooftop for years that their traditional on-premises SIEM tool and rules-based correlation have let them down, and so they’ve been stuck straddling the line between old and new. With this entirely new, and first of its kind cloud SIEM solution, customers have a single, unified platform in the cloud, to help them meet their modern security needs. And we’re not done yet, there’s more to come. #3: Samsung SmartThings is Changing the World of Connected IoT Scott Vlaminck, co-founder and VP of engineering at Samsung SmartThings, shared his company’s vision for SmartThings to become the definitive platform for all IoT devices, in order to deliver the best possible smart home experience for their customers. And, as Scott said on stage, Sumo Logic helps make that possible by providing continuous intelligence of all operational, security and business data flowing across the SmartThings IoT platform, which receives about 200,000 requests per second a day! Scott talked about the company’s pervasive usage of the Sumo Logic platform, in which 95 percent of employees use Sumo Logic to report on KPIs, customer service, product insights, security metrics and app usage trends, and partner health metrics to drive deeper customer satisfaction. Having a fully integrated tool available to teams outside of traditional IT and DevOps is what continuous intelligence means for SmartThings. #4: Security is Everyone’s Responsibility at MLB When Neil Boland, the chief information security officer (CISO) for Major Baseball League took the stage, he shared how he and his security team are completely redefining what enterprise security means for a digital-first sports organization that has to manage, maintain and secure over 30 different leagues (which translates to 30 unique brands and 30 different attack vectors). Neil’s mission for 2018 is to blow up the traditional SIEM and MSSP models and reinvent them for his company’s 100 percent cloud-based initiatives. Neil’s biggest takeaway is that everyone at MLB is on the cybersecurity team, even non-technical groups like the help desk, and this shared responsibility helps strengthen overall security posture and continue to deliver uninterrupted sports entertainment to their fans. And Sumo Logic has been a force multiplier that helps Neil and his team achieve that collective goal. #5: Community, Community, Community Bringing the talk full circle, Ramin ended the keynote with a word about community, and how we are not only in it for our customers, but we’re in it with them, and we want to share data trends, usages, and best practices of the Sumo Logic platform with our ecosystem to provide benchmarking capabilities. That’s why today at Illuminate, we launched a new innovation — Global Intelligence Service — that focused on three key areas: Industry Insights, Community Insights and Data Science Insights. These insights will help customers extend machine learning and insights to new teams and use cases across the enterprise, and these are only possible with Sumo Logic’s cloud-native, multi-tenant architecture. For data-driven industry insights, check out Sumo Logic’s third annual ‘State of Modern Applications and DevSecOps in the Cloud’ report. You can read about our latest platform innovations in our press release, or check out the cloud SIEM solution and Global Intelligence Service blogs. Want the Day Two Recap? If you couldn’t join us live for day two of Illuminate, or were unable to catch the Facebook livestream, check out our second day keynote recap blog for the top highlights.

In today’s hyper-connected world, a company’s differentiation is completely dependent upon delivering a better customer experience, at scale, and at a lower cost than the competition. This is no easy feat, and involves a combination of many things, particularly adopting new technologies and architectures, as well as making better use of data and analytics.Sumo Logic is committed to helping our customers excel in this challenging environment by making it easier to adopt the latest application architectures while also making the most of their precious data.The Power of the PlatformAs a multi-tenant cloud-native platform, Sumo Logic has a unique opportunity to provide context and data to our customers that is not available anywhere else. Why is this?First of all, when an enterprise wants to explore new architectures and evaluate options, it is very difficult to find broad overviews of industry trends based on real-time data rather than surveys or guesswork.Second, it is difficult to find reliable information about how exactly companies are using technologies at the implementation level, all the way down to the configurations and performance characteristics. Finally, once implemented, companies struggle to make the best use of the massive amount of machine data exhaust from their applications, particularly for non-traditional audiences like data scientists.It is with this backdrop in mind that Sumo Logic is announcing the Global Intelligence Service today during the keynote presentation at our second annual user conference, Illuminate, in Burlingame, Calif. This unprecedented initiative of data democratization is composed of three primary areas of innovation.Industry Insights — What Trends Should I be Watching?Sumo Logic is continuing to building on the success of its recently released third annual ‘State of Modern Applications and DevSecOps in the Cloud’ report to provide more real-time and actionable insights about industry trends. In order to stay on top of a constantly changing technology landscape, this report is expanding to include more frequent updates and instant-access options to help customers develop the right modern application or cloud migration strategy for their business, operational and security needs.Chart depicting clusters of AWS Services used frequently togetherCommunity Insights — What Are Companies like us, and Teams like ours, Doing?Sumo Logic is applying the power of machine learning to derive actionable insights for getting the most out of your technology investments. We have found that many engineering teams lack the right resources and education needed to make the best technology choices early on in their prototyping phases. And then, when the system is in production, it is often too late to make changes. That’s why Sumo Logic has an opportunity to save our customers pain and frustration by giving them benchmarking and comparison information when they most need it.We all like to think that our use cases are each a beautiful, unique snowflake. The reality is that, while each of us is unique, our uses of technology fall into some predictable clusters. So, looking over a customer base of thousands, Sumo Logic can infer patterns and best practices about how similar organizations are using technologies.Uses those patterns, we will be building recommendations and content for our customers that can be used to compare performance against a baseline of usage across their peers.Chart depicting how the performance behavior across customers tend to clusterData Science Insights — Data Scientists Need Love, TooData scientists are under more pressure than ever to deliver stunning results, while also getting pushback from society about the quality of their models and the biases that may or may not be there. At the end of the day, while data scientists have control over their models, they may have less control over the data.If the data is incomplete or biased in any way that can directly influence the results. To alleviate this issue, Sumo Logic is providing an open source integration with the industry standard Jupyter and Apache Zeppelin notebooks in order to make it easier for data scientists to leverage the treasure trove of knowledge currently buried in their application machine data.Empower the People who Power Modern BusinessYou may still be wondering, why does all of this matter?At the end of the day, it is all about making our customers successful by making their people successful. A business is only as effective as the people who do the work, and it is our mission at Sumo Logic to empower those users to excel in their roles, which in return contributes to overall company growth and performance.And we also want to set users outside of the traditional IT, DevOps, and security teams up for success as well by making machine data analytics more accessible for them.So, don’t forget that you heard it here first: Democratizing machine data is all about empowering the people with love (and with unique machine data analytics and insights)!Additional ResourcesDownload the 2018 ‘State of Modern Applications and DevSecOps in the Cloud’ report and/or read the press release for more detailed insights.Read the Sumo Logic platform enhancement release to learn more about our latest platform enhancements and innovationsSign up for Sumo Logic for free

Today at Sumo Logic’s annual user conference, Illuminate, we are announcing a new cloud SIEM solution to address fundamental challenges legacy security analytics tools have failed to solve.

Imagine that you are tasked with architecting your mission-critical cloud application. Or migrating your on-premises app to the cloud. And you ask yourself “how do the cloud savvy companies like Twitter, Airbnb, Adobe, SalesForce, etc. build and manage their modern applications?”

Are your on-premises analytics and security solutions failing you in today’s digital world? Don’t have the visibility you need across your full application stack? Unable to effectively monitor, troubleshoot and secure your microservices and multi-cloud architectures? If this sounds like your organization, then be sure to watch this short video explaining why a cloud-native, scalable and elastic machine data analytics platform approach is the right answer for building, running and securing your modern applications and cloud infrastructures.To learn more about how Sumo Logic is uniquely positioned to offer development, security and operations (DevSecOps) teams the right tools for their cloud environments, watch our Miles Ahead in the Cloud and DevOps Redemption videos, visit our website or sign up for Sumo Logic for free here.Video TranscriptionYou’ve decided to run your business in the cloud. You chose this to leverage all the benefits the cloud enables – speed to rapidly scale your business; elasticity to handle the buying cycles of your customers; and the ability to offload data center management headaches to someone else so you can focus your time, energy and innovation on building a great customer experience.So, when you need insights into your app to monitor, troubleshoot or learn more about your customers, why would you choose a solution that doesn’t work the same way?Why would you manage your app with a tool that locks you into a peak support contract, one that’s not designed to handle the unpredictability of your data?Sumo Logic is a cloud-native, multi-tenant service that lets you monitor, troubleshoot, and secure your application with the same standards of scalability, elasticity and security you hold yourself to.Sumo Logic is built on a modern app stack for modern app stacks.Its scalable………….elastic…………resilient cloud architecture has the agility to move as fast as your app moves, quickly scaling up for data volume.Its advanced analytics based on machine learning are designed to cope with change. So, when that data volume spikes, Sumo Logic is there with the capacity and the answers you need.Sumo Logic is built with security as a 1st principle. That means security is baked in at the code level, and that the platform has the credentials and attestations you need to manage compliance for your industry.Sumo Logic’s security analytics and integrated threat intelligence also help you detect threats and breaches faster, with no additional costs.Sumo Logic delivers all this value in a single platform solution. No more swivel chair analytics to slow you down or impede your decision-making. You have one place to see and correlate the continuum of operations, security and customer experience analytics – this is what we call continuous intelligence for modern apps.So, don’t try to support your cloud app with a tool that was designed for the old, on-premise world, or a pretend cloud-tool.Leverage the intelligence solution that fully replicates what you’re doing with your own cloud-business — Sumo Logic, the industry leading, cloud-native, machine data analytics platform delivered to you as a service.Sumo Logic. Continuous Intelligence for Modern Applications.

Article outlining the Sumo Logic multi-level certification program and how to enroll today to harness the power of Sumo Logic machine data analytics.

Sumo Logic is no different than most companies — we are in the service of our customers and we seek to build a product that they love. As we continue to refine the Sumo Logic platform, we’re also refining our feedback loops. One of those feedback loops is internal dogfooding and learning how our own internal teams such as engineering, sales engineering and customer success, experience the newest feature. However, we know that that approach can be biased. Our second feedback loop is directly from our customers, whose thoughts are then aggregated, distilled and incorporated into the product. The UX research team focuses on partnering with external customers as well as internal Sumo Logic teams that regularly use our platform, to hear their feedback and ensure that the product development team takes these insights into account as they build new capabilities. Our Product Development Process Sumo Logic is a late-stage startup, which means that we’re in the age of scaling our processes to suit larger teams and to support new functions. The processes mentioned are in various stages of maturity, and we haven’t implemented all of these to a textbook level of perfection (yet!). Currently, there are two facets to the product development process. The first is the discovery side, for capabilities that are entirely new, while the second is focused on delivery and improving capabilities that currently exist in the product. The two sides run concurrently, as opposed to sequentially, with the discovery side influencing the delivery side. The teams supporting both sides are cross-functional in nature, consisting of engineers, product managers and product designers. Adapted from Jeff Patton & Associates Now that we’ve established the two aspects to product development, we’ll discuss how customer feedback fits into this. Customer feedback is critical to all product decisions at Sumo Logic. We seek out the opinions of our customers when the product development team has questions that need answers before they can proceed. Customer feedback usually manifests in two different categories: broad and granular. Broad Customer Questions The more high level questions typically come from the discovery side. For example, we may get a question like this: “should we build a metrics product?” For the teams focused on discovery, UX research starts with a clear hypothesis and is more open-ended and high level. It may consist of whiteboarding with our customers or observing their use cases in their workspaces. The insights from this research might spawn a new scrum team to build a capability, or the insights could indicate we should focus efforts elsewhere. Granular Customer Questions By contrast, UX research for delivery teams is much more focused. The team likely has designs or prototypes to illustrate the feature that they’re building, and their questions tend to focus on discoverability and usability. For instance, they may be wondering if customers can find which filters apply to which dashboard panels. The outcomes from this research give the team the necessary data to make decisions and proceed with design and development. Occasionally, the findings from the discovery side will influence what’s going on the delivery side. The UX Research Process at Sumo Logic The diagram below describes the milestones during our current UX research process, for both discovery and delivery teams. As a customer, the most interesting pieces of this are the Research Execution and the Report Presentation, as these include your involvement as well as how your input impacts the product. UX Research Execution Research execution takes a variety of forms, from on-site observation to surveys to design research with a prototype. As a customer, you’re invited to all types of research, and we are always interested in your thoughts. Our ideal participants are willing to share how they are using the Sumo Logic platform for their unique operational, security and business needs, and to voice candid opinions. Our participants are also all over the emotional spectrum, from delighted to irritated, and we welcome all types. The immediate product development team takes part in the research execution. For example, if we’re meeting with customers via video conference, we’ll invite engineers, product management and designers to observe research sessions. There’s a certain realness for the product development team when they see and hear a customer reacting to their work, and we’ve found that it increases empathy for our customers. This is very typical for our qualitative UX research sessions, and what you can expect as a participant. In the above clip, Dan Reichert, a Sumo Logic sales engineer, discusses his vision for a Data Allocation feature to manage ingest. Research Presentation After the UX research team has executed the research, we’ll collect all data, video, photos and notes. We’ll produce a report with the key and detailed insights from the research, and we’ll present the report to the immediate product development team. These report readouts tend to be conversational, with a lengthy discussion of the results, anecdotes and recommendations from the UX researcher. I’ve found that the teams are very interested in hearing specifics of how our customers are using the product, and how their efforts will influence that. After the report readout, the product development team will meet afterward to discuss how they’ll implement the feedback from the study. The UX researcher will also circulate the report to the larger product development team for awareness. The insights are often useful for other product development teams, and occasionally fill in knowledge gaps for them. How Can I Voice My Thoughts and Get Involved in UX Research at Sumo Logic? We’d love to hear how you’re using Sumo Logic, and your feedback for improvement. We have a recruiting website to collect the basics, as well as your specific interests within the product. Our UX research team looks forward to meeting you!

The Sumo query language can be a source of joy and pain at times. Achieving mastery is no easy path and all who set on this path may suffer greatly until they see the light. The Log Operators Cheat Sheet is a valuable resource to learn syntax and semantics of the individual operators, but the bigger questions become “how can we tie them together” and “how can we write query language that matters?”

Hello there! My name is Sam and this summer I’ve been an intern at Sumo Logic. In this post I’ll share my experience working on the web UI engineering team and what I learned from it. A year ago I started my Master of Computer Science degree at Vanderbilt University and since the program is only two years long, there’s only one internship slot before graduation. So I needed to find a good one. Like other students, I wanted the internship to prepare me for my future career by teaching me about work beyond just programming skills while also adding a reputable line to my resume. So after months of researching, applying, preparing and interviewing, I officially joined the Sumo Logic team in May. The Onboarding Experience The first day was primarily meeting a lot of new people, filling out paperwork, setting up my laptop and learning which snacks are best at the office (roasted almonds take the win). The first couple of weeks were a heads-down learning period. I was learning about the Sumo Logic machine data analytics platform — everything from why it is used and how it works to what it is built on. We also had meetings with team members who explained the technologies involved in the Sumo Logic application. In general, though, the onboarding process was fairly flexible and open ended, with a ton of opportunities to ask questions and learn. Specifically, I enjoyed watching React Courses as a part of my on boarding. In school I pay to learn this, but here I am the one being paid 🙂 Culture and Work Environment The culture and work environment are super nice and relaxed. The developers are given a lot of freedom in how and what they are working on, and the internship program is very adaptable. I was able to shape my role throughout the internship to focus on tasks and projects that were interesting to me. Of course, the team was very helpful in providing direction and answering my questions, but it was mostly up to me to decide what I would like to do. The phrase that I remember best was from my manager. On my second week at Sumo Logic he said: “You don’t have to contribute anything — the most important thing is for you to learn.” The thing that surprised me the most at Sumo Logic is how nice everyone is. This is probably the highest “niceness per person” ratio I’ve ever experienced in my life. Almost every single person I’ve met here is super friendly, humble, open minded and smart. These aspects of the culture helped me greatly. Summer Outside Sumo Logic One of the important factors in choosing a company for me was its location. I am from Moscow, Russia, and am currently living in Nashville while I attend Vanderbilt, but I knew that this summer I definitely wanted to find an internship in the heart of the tech industry — Silicon Valley. Lucky for me, Sumo Logic is conveniently located right in the middle of it in Redwood City. I also enjoyed going to San Francisco on weekends to explore the city, skateboarding to Stanford from my home and visiting my friend at Apple’s Worldwide Developers Conference (WWDC) in San Jose. I liked the SF Bay Area so much that I don’t want to work anywhere else in the foreseeable future! Actual Projects: What Did I Work On? The main project that I work on is a UI component library. As the company quickly grows, we strive to make the UI components more consistent — visually and written — in standard and the code more maintainable. We also want to simplify the communication about the UI between the Dev and Design teams. I was very excited about the future impact and benefit of this project for the company, and had asked the team join this effort. A cool thing about this library is that it is a collection of fresh and independent React components that will be then used by developers in creation of all parts of the Sumo Logic app. It is a pleasure to learn the best practices while working with cutting edge libraries like React. If that sounds interesting to you, check out this blog from one of my Sumo Logic colleagues on how to evaluate and implement react table alternatives into your project. Things I Learned That I Didn’t Know Before How professional development processes are structured How companies work, grow and evolve How large projects are organized and maintained How to communicate and work on a team What a web-scale application looks from the inside And, finally, how to develop high quality React components Final Reflection Overall, I feel like spending three months at Sumo Logic was one of the most valuable and educational experiences I’ve ever had. I received a huge return on investment of time and moved much closer to my future goals of gaining relevant software development knowledge and skills to set me up for a successful career post-graduation. Additional Resources Want to stay in touch with Sumo Logic? Follow & connect with us on Twitter, LinkedIn and Facebook for updates. If you want to learn more about our machine data analytics platform visit our “how it works” page!

It’s been a busy security year, with countless twists and turns, mergers, acquisitions and IPOs, and most of that happening in the lead up to one of the biggest security conferences of the year — Black Hat U.S.A. Each year, thousands of hackers, security practitioners, analysts, architects, executives/managers and engineers from varying industries and from all over the country (and world) descend on the desert lands of the Mandalay Bay Resort & Casino in Las Vegas for more than a week of trainings, educational sessions, networking and the good kind of hacking (especially if you stayed behind for DefCon26). Every Black Hat has its own flavor, and this year was no different. So what were some of the “buzzwords” floating around the show floor, sessions and networking areas? The Sumo Logic security team pulled together a list of the hottest, newest, and some old, but good terms that we overheard and observed during our time at Black Hat last week. Read on for more, including a recap of this year’s show trends. And the Buzzword is… APT — Short for advanced persistent threat Metasploit — Provides information about security vulnerabilities and used in pen testing Pen Testing (or Pentesting) — short for penetration testing. Used to discover security vulnerabilities OSINT — Short for open source intelligence technologies XSS — Short for cross site scripting, which is a type of attack commonly launched against web sites to bypass access controls White Hat — security slang for an “ethical” hacker Black Hat — a hacker who violates computer security for little reason beyond maliciousness or personal gain Red Team — Tests the security program (Blue Team) effectiveness by using techniques that hackers would use Blue Team — The defenders against Red Team efforts and real attackers Purple Team — Responsible for ensuring the maximum effectiveness of both the Red and Blue Teams Fuzzing or Fuzz Testing — Automated software that performs invalid, unexpected or random data as inputs to a computer program that is typically looking for structured content, i.e. first name, last name, etc. Blockchain — Widely used by cryptocurrencies to distribute expanding lists of records (blocks), such as transaction data, which are virtually “chained” together by cryptography. Because of their distributed and encrypted nature the blocks are resistant to modification of the data. SOC — Short for security operations center NOC — Short for network operations center Black Hat 2018 Themes There were also some pretty clear themes that bubbled to the top of this year’s show. Let’s dig into them. The Bigger, the Better….Maybe Walking the winding labyrinth that is the Mandalay Bay, you might have overheard conference attendees complaining that this year, Black Hat was bigger than in year’s past, and to accommodate for this, the show was more spread out. The business expo hall was divided between two rooms: a bigger “main” show floor (Shoreline), and a second, smaller overflow room (Oceanside), which featured companies new to the security game, startups or those not ready to spend big bucks on flashy booths. While it may have been a bit confusing or a nuisance for some to switch between halls, the fact that the conference is outgrowing its own space is a good sign that security is an important topic and more organizations are taking a vested interest in it. Cloud is the Name, Security is the Game One of the many themes at this year’s show was definitely all things cloud. Scanning the booths, you would have noticed terms around security in the cloud, how to secure the cloud, and similar messaging. Cloud has been around for a while, but seems to be having a moment in security, especially as new, agile cloud-native security players challenge some of the legacy on-premises vendors and security solutions that don’t scale well in a modern cloud, container or serverless environment. In fact, according to recent Sumo Logic research, 93 percent of responding enterprises face challenges with security tools in the cloud, and 49 percent state that existing legacy tools aren’t effective in the cloud. Roses are Red, Violets are Blue, FUD is Gone, Let’s Converge One of the biggest criticisms of security vendors (sometimes by other security vendors) is all of the language around fear, uncertainty and doubt (FUD). This year, it seems that many vendors have ditched the fearmongering and opted for collaboration instead. Walking the expo halls, there was a lot of language around “togetherness,” “collaboration” and the general positive sentiment that bringing people together to fight malicious actors is more helpful than going at it alone in siloed work streams. Everything was more blue this year. Usually, you see the typical FUD coloring: reds, oranges , yellows and blacks, and while there was still some of that, the conference felt brighter and more uplifting this year with purples, all shades of blues, bright greens, and surprisingly… pinks! There was also a ton of signage around converging development, security and operations teams (DevSecOps or SecOps) and messaging, again, that fosters an “in this together” mentality that creates visibility across functions and departments for deeper collaboration. Many vendors, including Sumo Logic have been focusing on security education, offering and promoting their security training, certification and educational courses to make sure security is a well-understood priority for stakeholders across all lines of the business. Our recent survey findings also validate the appetite for converging workflows, with 54 percent of respondents citing a greater need for cross-team collaboration (DevSecOps) to effectively investigate, prioritize and correlate threats for faster remediation. Three cheers for that! Sugar and Socks and Everything FREE Let’s talk swag. Now this trend is not entirely specific to Black Hat, but it seems each year, the booth swag gets sweeter (literally) with vendors offering doughnut walls, chocolates, popcorn and all sorts of tasty treats to reel people into conversation (and get those badge scans). There’s no shortage of socks either! Our friends at HackerOne were giving out some serious booth swag, and you better believe we weren’t headed home without grabbing some! Side note: Read the latest HackerOne blog or watch the latest SnapSecChat video to learn how our Sumo Logic security team has taken a DevSecOps approach to bug bounties that creates transparency and collaboration between hackers, developers, and external auditors to improve security posture. Sumo swag giveaways were in full swing at our booth, as well. We even raffled off a Vento drone for one lucky Black Hat winner to take home! Parting Thoughts As we part ways with 100 degree temps and step back into our neglected cubicles or offices this week, it’s always good to remember the why. Why do we go to Black Hat, DefCon, BSides, and even RSA? It’s more than socializing and partying, it’s to connect with our community, to learn from each other and to make the world a more secure and bette place for ourselves, and for our customers. And with that, we’ll see you next year! Additional Resources For the latest Sumo Logic cloud security analytics platform updates, features and capabilities, read the latest press release. Want to learn more about Sumo Logic security analytics and threat investigation capabilities? Visit our security solutions page. Interested in attending our user conference next month, Iluminate? Visit the webpage, or check out our latest “Top Five Reasons to Attend” blog for more information. Download and read our 2018 Global Security Trends in the Cloud report or the infographic for more insights on how the security and threat landscape is evolving in today’s modern IT environment of cloud, applications, containers and serverless computing.

Last year Sumo Logic launched its first user conference, Illuminate. We hosted more than 300 fellow Sumo Logic users who spent two days getting certified, interacting with peers to share best practices and lots of mingling with Sumo’s technical experts (all while having fun). The result? Super engaged users with a new toolbox to take back to their teams to make the most of their Sumo Logic platform investment, and get the real-time operational and security insights needed to better manage and secure their modern applications and cloud infrastructures. Watch last year’s highlight reel below: This piece of feedback from one attendee sums up the true value of Illuminate: “In 48 hours I already have a roadmap of how to maximize the use of Sumo Logic at my company and got a green light from my boss to move forward.” — Sumo Logic Customer / Illuminate Attendee Power to the People This year’s theme for Illuminate is “Empowering the People Who Power Modern Business” and is expected to attract more than 500 attendees who will participate in a unique interactive experience including over 40 sessions, Ask the Expert bar, partner showcase and Birds of a Feather roundtables. Not enough to convince you to attend? Here are five more reasons: Get Certified – Back by popular demand, our multi-level certification program provides users with the knowledge, skills and competencies to harness the power of machine data analytics and maximize investments in the Sumo Logic platform. Bonus: we have a brand new Sumo Security certification available at Illuminate this year designed to teach users how to increase the velocity and accuracy of threat detection and strengthen overall security posture. Hear What Your Peers are Doing – Get inspired and learn directly from your peers like Major League Baseball, Genesys, USA TODAY NETWORK, Wag, Lending Tree, Samsung SmartThings, Informatica and more about how they implemented Sumo Logic and are using it to increase productivity, revenue, employee satisfaction, deliver the best customer experiences and more. You can read more about the keynote speaker line up in our latest press release. Technical Sessions…Lots of Them – This year we’ve broaden our breakout sessions into multiple tracks including Monitoring and Troubleshooting, Security Analytics, Customer Experience and Dev Talk covering tips, tricks and best practices for using Sumo Logic around topics including Kubernetes, DevSecOps, Metrics, Advanced Analytics, Privacy-by-Design and more. Ask the Experts – Get direct access to expert advice from Sumo Logic’s product and technical teams. Many of these folks will be presenting sessions throughout the event, but we’re also hosting an Ask the Expert bar where you can get all of your questions answered, see demos, get ideas for dashboards and queries, and see the latest Sumo Logic innovations. Explore the Modern App Ecosystem – Sumo Logic has a rich ecosystem of partners and we have a powerful set of joint integrations across the modern application stack to enhance the overall manageability and security for you. Stop by the Partner Pavilion to see how Sumo Logic works with AWS, Carbon Black, CrowdStrike, JFrog, LightStep, MongoDB, Okta, OneLogin, PagerDuty, Relus and more. By now you’re totally ready for the Illuminate experience, right? Check out the full conference agenda here. These two days will give you all of the tools you need (training, best practices, new ideas, peer-to-peer networking, access to Sumo’s technical experts and partners) so you can hit the ground running and maximize the value of the Sumo Logic platform for your organization. Register today, we look forward to seeing you there!

Managing security and compliance in the new world of cloud and modern applications giving you chills? You’re not alone.

Regardless of industry or size, all organizations need a solid security and vulnerability management plan. One of the best ways to harden your security posture is through penetration testing and inviting hackers to hit your environment to look for weak spots or holes in security. However, for today’s highly regulated, modern SaaS company, the traditional check-box compliance approach to pen testing is failing them because it’s slowing them down from innovating and scaling. That’s why Sumo Logic Chief Security Officer and his team have partnered with HackerOne to implement a modern bug bounty program that takes a DevSecOps approach. They’ve done this by building a collaborative community for developers, third-party auditors and hackers to interact and share information in an online portal that creates a transparent bug bounty program that uses compliance to strengthen security. By pushing the boundaries and breaking things, it collectively makes us stronger, and it also gives our auditors a peek inside the kimono and more confidence in our overall security posture. It also moves the rigid audit process into the DevSecOps workflow for faster and more effective results. To learn more about Sumo Logic’s modern bug bounty program, the benefits and overall positive impact it’s had on not just the security team, but all lines of the business, including external stakeholders like customers, partners and prospects, watch the latest SnapSecChat video series with Sumo Logic CSO, George Gerchow. And if you want to hear about the results of Sumo Logic’s four bounty challenge sprints, head on over to the HackerOne blog for more. If you enjoyed this video, then be sure to stay tuned for another one coming to a website near you soon! And don’t forget to follow George on Twitter at @GeorgeGerchow, and use the hashtag #SnapSecChat to join the security conversation! Stop by Sumo Logic’s booth (2009) at Black Hat this week Aug 8-9, 2018 at The Mandalay Bay in Las Vegas to chat with our experts and to learn more about our cloud security analytics and threat investigation capabilities. Happy hacking!

Building Replicated Stateful Systems using Apache Kafka

In this Sumo Logic Employee Spotlight we interview Rocio Lopez. A lover of numbers, Rocio graduated from Columbia University with a degree in economics, but certain circumstances forced her to forego a career in investment banking and instead begin freelancing until she found a new career that suited her talents and passions: product design. Intrigued? You should be! Read Rocio’s story below. She was a delight to interview! When Creativity Calls Q: So tell me, Rocio, what’s your story? Rocio Lopez (RL): I am a product designer at Sumo Logic and focus mostly on interaction design and prototyping new ideas that meet our customers’ needs. Q: Very cool! But, that’s not what you went to school for, was it? RL: No. I studied economics at Columbia. I wanted to be an investment banker. Ever since I was a little girl, I’ve been a nerd about numbers and I love math. Part of it was because I remember when the Peso was devalued and my mom could no longer afford to buy milk. I became obsessed with numbers and this inspired my college decision. But the culture and career path at Columbia was clear — you either went into consulting or investment banking. I spent a summer shadowing at Citigroup (this was during the height of the financial crisis), and although my passion was there, I had to turn down a career in finance because I was here undocumented. Q: That’s tough. So what did you do instead? RL: When I graduated in 2011, I started doing the things I knew how to do well like using Adobe Photoshop and InDesign to do marketing for a real estate company or even doing telemarketing. I eventually landed a gig designing a database for a company called Keller Williams. They hired an engineer to code the database, but there was no designer around to think through the customer experience so I jumped in. Q: So that’s the job that got you interested in product design? RL: Yes. And then I spent a few years at Cisco in the marketing organization where they needed help revamping their training platforms. I started doing product design without even knowing what it was until a lead engineer called it out. I continued doing small design projects, started freelancing and exploring on my own until I connected with my current manager, Daniel Castro. He was hiring for a senior role, and while I was not that senior, the culture of the team drew me in. Q: Can you expand on that? RL: Sure. The design team at Sumo Logic is very unique. I’ve spent about seven years total in the industry and what I’ve been most impressed by is the design culture here, and the level of trust and level-headedness the team has. I’ve never come across this before. You would think that because we’re designing an enterprise product that everyone would be very serious and buckled up, but it’s the opposite. The Life of a Dreamer Q: Let’s switch gears here. I heard you on NPR one morning, before I even started working at Sumo Logic. Tell me about being a dreamer. RL: People come to the U.S. undocumented because they don’t know of other ways to come legally or the available paths for a visa aren’t a match for them because they may not have the right skills. And those people bring their families. I fell into that category. I was born in Mexico but my parents came over to the U.S. seeking a better life after the Tequila crisis. I grew up in Silicon Valley and went to school like any other American kid. When Barack Obama was in office, he created an executive order known as the Deferred Action for Childhood Arrivals (DACA) program, since Congress has failed to passed legislative action since 2001. To qualify for the program, applicants had to have arrived in the U.S. before age 16 since June 15, 2007 and pass a rigorous background check by homeland security every two years. . I fell into this category and was able to register in this program. Because most of the immigrants are young children who were brought here at a very young age, we’ve sort of been nicknamed “dreamers” after the 2001 DREAM Act (short for Development, Relief and Education for Alien Minors Act). Q: And under DACA you’ve been able to apply for a work permit? RL: That’s right. I have a work permit, I pay income taxes, and I was able to attend college just like a U.S. citizen, although I am still considered undocumented and that comes with certain limitations. For instance, my employer cannot sponsor me and I cannot travel outside the United States. The hope was that Congress would create a path for citizenship for Dreamers, but now that future is a bit uncertain after they failed to meet the deadline to pass a bill in March. For now I have to wait until the Supreme Court rules the constitutionality of DACA to figure out my future plans. Q: I can only imagine how difficult this is to live with. What’s helped you through it? RL: At first I was a big advocate, but now I try to block it out and live in the present moment. And the opportunity to join the Sumo Logic design team came at the right time in my life. I can’t believe what I do every day is considered work. The team has a very unique way of nurturing talent and it’s something I wish more companies would do. Our team leaders make sure we have fun in addition to getting our work done. We usually do team challenges, dress up days, etc. that really bring us all together to make us feel comfortable, encourage continued growth, and inspire us to feel comfortable speaking up with new ideas. I feel like the work I am doing has value and is meaningful, and we are at the positive end of the “data conversation.” I read the news and see the conversations taking place with companies like Facebook and Airbnb that are collecting our personal data. It’s scary to think about. And it feels good to be on the other side of the conversation; on the good side of data and that’s what gets me excited and motivated. Sumo Logic is collecting data and encrypting it and because we’re not on the consumer-facing side, we can control the lens of how people see that data. We can control not only the way our customers collect data but also how they parse and visualize it. I feel we’re at the cusp of a big industry topic that’s going to break in the next few years. Q: I take it you’re not on social media? RL: No. I am completely off Facebook and other social media platforms. When I joined Sumo Logic, I became more cautious of who I was giving my personal data to. Advice for Breaking into Design & Tech? Q: Good for you! So what advice to you have for people thinking of switching careers? RL: From 2011 to now I’ve gone through big career changes. There are a lot of people out there that need to understand how the market is shifting, that some industries like manufacturing, are not coming back, and that requires an adaptive mindset. The money and opportunity is where technology and data are and if people can’t transition to these new careers in some capacity, they’re going to be left out of the economy and will continue to have problems adjusting. It’s a harsh reality, but we have to be able to make these transitions because in 15 or 20 years from now, the world will look very different. I’ve been very active in mentoring people that want to break into technology but aren’t sure how. Q: What’s some of the specific advice related to a career path in UX/design that you give your mentees? RL: Sometimes you have to breakaway from traditions like school or doing a masters program and prioritize the job experience. Design and engineering are about showing you’ve done something, showing a portfolio. If you can change your mindset to this, you will be able to make the transition more smoothly. I also want to reiterate that as people are looking for jobs or next careers, it’s important to find that place that is fun and exciting. A place where you feel comfortable and can be yourself and also continue to grow and learn. Find meaning, find value, and find the good weird that makes you successful AND happy. Stay in Touch Stay in touch with Sumo Logic & connect with us on Twitter, LinkedIn and Facebook for updates. Want to work here? We’re hiring! Check out our careers page to join the team. If you want to learn more about our machine data analytics platform visit our “how it works” page!

Outages and postmortems are a fact of life for any software engineer responsible for managing a complex system. And it can be safely said that those two words – “outage” and “postmortem,” do not carry any positive connotations in the remotest sense of the word. In fact, they are generally dreaded by most engineers. While that sentiment is understandable given the direct impact of such incidents on customers and the accompanying disruption, our individual perspective matters a lot here as well. If we are able to look beyond the damage caused by such incidents, we might just realize that outages and postmortems shouldn’t be “dreaded,” but instead, wholeheartedly embraced. One has to only try, and the negative vibes associated with these incidents may quickly give way to an appreciation of the complexity in modern big data systems. The Accidental Harmony of Layered Failures As cliche as it may sound, “beauty” indeed lies in the eyes of the beholder. And one of the most beautiful things about an outage/postmortem is the spectacular way in which modern big data applications often blow up. When they fail, there are often dozens of things that fail simultaneously, all of which collude, resulting in an outage. This accidental harmony among failures and the dissonance among the guards and defenses put in place by engineers, is a constant feature of such incidents and is always something to marvel at. It’s almost as if the resonance frequencies of various failure conditions match, thereby amplifying the overall impact. What’s even more surprising is the way in which failures at multiple layers can collude. For example, it might so happen that an outage-inducing bug is missed by unit tests due to missing test cases, or even worse, a bug in the tests! Integration tests in staging environments may have again failed to catch the bug, either due to a missing test case or disparity in the workload/configuration of staging/production environments. There could also be misses in monitoring/alerting, resulting in increased MTTIs. Similarly, there may be avoidable process gaps in the outage handling procedure itself. For example, some on-calls may have too high of an escalation timeout for pages or may have failed to update their phone numbers in the pager service when traveling abroad (yup, that happens too!). Sometimes, the tests are perfect, and they even catch the error in staging environments, but due to a lack of communication among teams, the buggy version accidentally gets upgraded to production. Outages are Like Deterministic Chaos In some sense, these outages can also be compared to “deterministic chaos” caused by an otherwise harmless trigger that manages to pierce through multiple levels of defenses. To top it off, there are always people involved at some level in managing such systems, so the possibility of a mundane human error is never too far away. All in all, every single outage can be considered as a potential case study of cascading failures and their layered harmony. An Intellectual Journey Another very deeply satisfying aspect of an outage/postmortem is the intellectual journey from “how did that happen?” to “that happened exactly because X, Y, Z.” Even at the system level, it’s necessary to disentangle the various interactions and hidden dependencies, discover unstated assumptions and dig through multiple layers of “why’s” to make sense of it all. When properly done, root cause analysis for outages of even moderately complex systems, demand a certain level of tenacity and perseverance, and the fruits of such labor can be a worthwhile pursuit in and of itself. There is a certain joy in putting the pieces of a puzzle together, and outages/postmortems present us exactly with that opportunity. Besides the above intangibles, outages and their subsequent postmortems have other very tangible benefits. They not only help develop operational knowledge, but also provide a focused path (within the scope of the outage) to learn about the nitty-gritty details of the system. At the managerial level too, they can act as road signs for course correction and help get the priorities right. Of course, none of the above is an excuse to have more outages and postmortems! We should always strive to build reliable, fault-tolerant systems to minimize such incidents, but when they do happen, we should take them in stride, and try to appreciate the complexity of the software systems all around us. Love thy outages. Love thy postmortems. Stay in Touch Want to stay in touch with Sumo Logic? Follow & connect with us on Twitter, LinkedIn and Facebook for updates. Visit our website to learn more about our machine data analytics platform and be sure to check back on the blog for more posts like this one if you enjoyed what you read!

We have released 11 brand new Google Cloud Platform (GCP) applications to the existing Sumo Logic App Catalog. We continue to see momentum and adoption of multiple cloud providers by our customers and this release reinforces our commitment to providing our customers with continuous intelligence of their business regardless of where their applications modern and cloud infrastructure data reside — from Amazon Web Services (AWS) to Microsoft Azure to Google Cloud.

As a longtime DevOps engineer with a passion for gaming and creating things, I truly believe that in order to present data correctly, you must first understand the utility of a tool without getting hung up on the output (data). To understand why this matters, I’ll use Sumo Logic’s machine data analytics platform as an example. With a better understanding of how our platform works, you’ll be able to turn seemingly disparate data into valuable security, operational or business insights that directly service your organization’s specific needs and goals. The Beginning of Something Great Last year, I was sitting with some colleagues at lunch and suggested that it would be super cool to have a video game at our trade show booth. We all agreed it was a great idea, and what started as a personal at-home project turned into a journey to extract game data, and present it in a compelling and instructive manner. The following is how this simple idea unfolded over time and what we learned as a result. Super Smash Bros Meets Sumo Logic The overall idea was solid, however, after looking at emulators and doing hours of research (outside of office hours), I concluded that it was a lot harder to extract data from an old school arcade game even working with an emulator. My only path forward would be to use a cheat engine to read memory addresses, and all the work would be done in Assembly, which is a low-level ‘80s era programming language. It’s so retro that the documentation was nearly impossible to find and I again found myself at another impasse. Another colleague of mine who is a board game aficionado, suggested I find an open source game online that I could add code to myself in order to extract data. Before I started my search, I set some parameters. What I was looking for was a game that had the following characteristics. It should be multiplayer It would ideally produce different types of data It would manifest multiple win conditions: game and social Enter Super Smash Bros (SSB), which met all of the above criteria. If you are not familiar with this game, it’s originally owned/produced by Nintendo and the appeal is that you and up to three other players battle each other in “King of the Hill” until there is an “official” game winner. It helps to damage your opponent first before throwing them off the hill. The game win condition is whoever has the most number of lives when the game ends, wins. And the game ends when either time runs out or only one player has lives left. However, this leaves holes for friends to argue who actually won. If you’ve ever played this game (which is one of strategy), there is a second kind of condition — a social win condition. You can “officially” win by the game rules but there’s context attached to “how” you won — a social win. Creating Sumo Smash Bros I found an open source clone of Super Smash Bros written in Javascript which runs entirely in a web browser. It was perfect. Javascript is a simple language and with the help of a friend to get started, we made it so we could group log messages that would go to the console where a developer could access it and then send it directly into the Sumo Logic platform. PRO TIP: If you want game controllers for an online video game like Super Smash Bros, use Xbox controllers not Nintendo! We would record certain actions in the code, such as: When a player’s animation changed What move a player performed Who hit who, when and for how much What each players’ lives were For example, an animation change would be whenever a player was punched by an opponent. Now by the game standards, very limited data determines who is the “official” winner of the game based on the predetermined rules, but with this stream of data now flowing into Sumo Logic, we could also identify the contextual “social win” and determine if and how they were different from the game rules. Here’s an example of a “social” win condition: Imagine there’s a group of four playing the game, and one of the players (player 1) hangs back avoiding brawls until two of the three opponents are out of lives. Player 1 jumps into action, gets a lucky punch on the other remaining player who has thus far dominated (and who is really damaged) and throws him from the ring to take the “official” game win. Testing the Theory When we actually played, the data showed exactly what I had predicted. First, some quick background on my opponents: Jason E. (AKA Jiggles) — He admits to having spent a good portion of his youth playing SSB, and he may have actually played in tournaments. Michael H. (AKA Killer) — He’s my partner in crime. We’re constantly cooking up crazy ideas to try out, both in and outside of work He also had plenty of experience with the game. Mikhail M. (AKA MM$$DOLLAB) — He has always been a big talker. He too played a lot, and talked a big talk. Originally I had intended for us to pseudo choreograph the game to get the data to come out “how I wanted” in order to show that while the game awarded a “winner” title to one player, the “actual winner” would be awarded by the friends to the player who “did the most damage to others” or some other parameter. It only took about three nanoseconds before the plan was out the window and we were fighting for the top. Our colleague Jason got the clear technical game win. We had recorded the game and had the additional streams of data, and when the dust had settled, a very different story emerged. For instance, Jason came in third place for our social win parameter of “damage dealt.” Watching the recording, it’s clear that Jason’s strategy was to avoid fighting until the end. When brawls happened, he was actively jumping around but rarely engaged with the other players. He instead waited for singled-out attacks. Smart, right? Maybe. We did give him the “game win,” however, based on the “damage dealt” social win rule, the order was: Michael, myself, then Jason, and Mikhail. Watch what happened for yourself: What’s the Bigger Picture? While this was a fun experiment, there’s also an important takeaway. At Sumo Logic, we ingest more than 100 terabytes of data each day — that’s the equivalent of data from about 200 Libraries of Congress per second. That data comes from all over — it’s a mix of log, event, metrics, security data coming not just from within an organization’s applications and infrastructure, but also from third party vendors. When you have more information, you can see trends and patterns, make inferences, technical and business decisions — you gain an entirely new level of understanding beyond the 1s and 0s staring back at you on a computer screen. People also appreciate the data for different reasons. For example, engineers only care that the website they served you is the exact page you clicked on. They don’t care if you searched for hats or dog food or sunscreen. But marketers care, a lot. Marketers care about your buying decisions and patterns and they use that to inform strong, effective digital marketing campaigns to serve you relevant content. At Sumo Logic, we don’t want our customers or prospects to get hung up on the data, we want them to look past that to first understand what our tool does, to understand how it can help them get the specific data they need to solve a unique problem or use case. “In the words of Sherlock Holmes, it’s a capital mistake to theorize before one has data.” — Kenneth Barry, Sumo Logic The types of data you are ingesting and analyzing only matters if you first understand your end goal, and have the proper tools in place — a means to an end. From there, you can extract and make sense of the data in ways that matter to your business, and each use case varies from one customer to another. Data powers our modern businesses and at Sumo Logic, we empower those who use this data. And we make sure to have fun along the way! Bonus: Behind the Scenes Video Q&A with Kenneth Additional Resources Visit our website to learn more about the power of machine data analytics and to download Sumo Logic for free to try it out for yourself Read our 2018 State of Modern Applications in the Cloud report Register to attend Illuminate, our annual user conference taking place Sept. 12-13, 2018 in Burlingame, Calif.

In a previous blog post, we talked about Amazon Relational Database Service (RDS). RDS is one of the most popular cloud-based database services today and extensively used by Amazon Web Services (AWS) customers for its ease of use, cost-effectiveness and simple administration. Although as a managed service, RDS doesn’t require database administrators (DBAs) to do many of the day-to-day tasks, it still needs to be monitored for performance and availability. That’s because Amazon doesn’t auto-tune any database performance — this is a shared responsibility of the customer. That’s why there should be a monitoring strategy and processes in place for DBAs and operation teams to keep an eye on their RDS fleet. In this blog post, we will talk about an overall best-practice approach for doing this. Why Database Monitoring Keeping a database monitoring regimen in place, no matter how simple, can help address potential issues proactively before they become incidents, and cost additional time and money. Most AWS infrastructure teams typically have decent monitoring in place for different types of resources like EC2, ELB, Auto Scaling Groups, Logs, etc. Database monitoring often comes at a later stage or is ignored altogether. With RDS, it’s also easy to overlook due to the low-administration nature of the service. The DBA or the infrastructure managers should therefore invest some time in formulating and implementing a database monitoring policy. Please note that designing an overall monitoring strategy is an involved process and is not just about defining database counters to monitor. It also includes areas like: Service Level Agreement Classifying incident types (Critical, Serious, Moderate, Low etc.) Creating RACI (Responsible, Accountable, Consulted, Informed) matrix Defining escalation paths etc.. A detailed discussion of all these topics is beyond the scope of this article, so we will concentrate on the technical part only. What to Monitor Database monitoring, or RDS monitoring in this case, is not about monitoring only database performance. A monitoring strategy should include the following broad categories and their components: Monitoring category Examples of what to monitor Availability Is the RDS instance or cluster endpoint accessible from client tools? Is there any instance stopping, starting, failed over or being deleted? Is there a failover of multi-AZ instances? Recoverability Is the RDS instance being backed up – both automatically and manually? Are individual databases being backed up successfully? Health and Performance What’s the CPU, memory and disk space currently in use? What’s the query latency? What’s the disk read/write latency? What’s the disk queue length? How many database connections are active? Are there any blocking and waiting tasks? Are there any errors or warnings reported in database log files? Are these related to application queries? Are they related to non-optimal configuration values? Are any of the scheduled jobs failing? Manageability Are there any changes in the RDS instances’ Tags Security groups Instance properties Parameter and option groups? Who made those changes and when? Security Which users are connecting to the database instance? What queries are they running? Cost How much each RDS instance is costing every month? While many of these things can be monitored directly in AWS, Sumo Logic can greatly help with understanding all of the logs and metrics that RDS produces. In this article, we will talk about what AWS offers for monitoring RDS. As we go along, we will point out where we think Sumo Logic can make the work easier. Monitoring Amazon CloudWatch You can start monitoring RDS using metrics from Amazon CloudWatch. Amazon RDS, like any other AWS service, exposes a number of metrics which are available through CloudWatch. There are three ways to access these metrics: From AWS Console Using AWS CLI Using REST APIs The image below shows some of these metrics from the RDS console: Amazon CloudWatch shows two types of RDS metrics: Built-in Metrics Enhanced Monitoring Metrics Built-in Metrics These metrics are available from any RDS instance. They are collected from the hypervisor of the host running the RDS virtual machine. Some of the metrics may not be available for all database engines, but the important ones are common. It is recommended the following RDS metrics are monitored from CloudWatch Metric What it means Why you should monitor it CPUUtilization % CPU load in the RDS instance. A consistent high value means one or more processes are waiting for CPU time while one or more processes are blocking it. DiskQueueDepth The number of input and output requests waiting for the disk resource. A consistent high value means disk resource contention – perhaps due to locking, long running update queries etc. DatabaseConnections The number of database connections against the RDS instance. A sudden spike should be investigated immediately. It may not mean a DDOS attack, but a possible issue with the application generating multiple connections per request. FreeableMemory The amount of RAM available in the RDS instance, expressed in bytes. A very low value means the instance is under memory pressure. FreeStorageSpace Amount of disk storage available in bytes. A small value means disk space is running out. ReadIOPS The average number of disk read operations per second. Should be monitored for sudden spikes. Can mean runaway queries. WriteIOPS The average number of disk write operations per second. Should be monitored for sudden spikes. Can mean a very large data modification ReadLatency The average time in milliseconds to perform a read operation from the disk. A higher value may mean a slow disk operation, probably caused by locking. WriteLatency The average time in milliseconds to perform a write operation to disk. A higher value may means disk contention. ReplicaLag How far in time, the read replica of MySQL, MariaDB or PostgreSQL instance is lagging behind from its master A high lag value can means read operations from replica is not serving the current data. Amazon RDS Aurora engine also exposes some extra counters which are really useful for troubleshooting. At the time of writing, Aurora supports MySQL and PostgreSQL only. We recommend monitoring these counters: Metric What it means Why you should monitor it DDLLatency The average time in milliseconds to complete Data Definition Language (DDL) commands like CREATE, DROP, ALTER etc. A high value means the database is having performance issues running DDL commands. This can be due to exclusive locks on objects. SelectLatency The average time in milliseconds to complete SELECT queries. A high value may mean disk contention, poorly written queries, missing indexes etc. InsertLatency The average time in milliseconds to complete INSERT commands. A high value may mean locking or poorly written INSERT command. DeleteLatency The average time in milliseconds to complete DELETE commands. A high value may mean locking or poorly written DELETE command. UpdateLatency The average time in milliseconds to complete UPDATE commands. A high value may mean locking or poorly written UPDATE command. Deadlocks The average number of deadlocks happening per second in the database. More than 0 should be a concern – it means the application queries are running in such a way that they are blocking each other frequently. BufferCacheHitRatio The percentage of queries that can be served by data already stored in memory It should be a high value, near 100, meaning queries are don’t have to access disk for fetching data. Queries The average number of queries executed per second This should have a steady, average value. Any sudden spike or dip should be investigated. You can use the AWS documentation for a complete list of built-in RDS metrics. Enhanced Monitoring Metrics RDS also exposes “enhanced monitoring metrics.” These are collected by agents running on the RDS instances’ operating system. Enhanced monitoring can be enabled when an instance is first created or it can be enabled later. It is recommended enabling it because it offers a better view of the database engine. Like built-in metrics, enhanced metrics are available from the RDS console. Unlike built-in metrics though, enhanced metrics are not readily accessible from CloudWatch Metrics console. When enhanced monitoring is enabled, CloudWatch creates a log group called RDSOSMetrics in CloudWatch Logs: Under this log group, there will be a log stream for each RDS instance with enhanced monitoring. Each log stream will contain a series of JSON documents as records. Each JSON document will show a series of metrics collected at regular intervals (by default every minute). Here is a sample excerpt from one such JSON document: { “engine”: “Aurora”, “instanceID”: “prodataskills-mariadb”, “instanceResourceID”: “db-W4JYUYWNNIV7T2NDKTV6WJSIXU”, “timestamp”: “2018-06-23T11:50:27Z”, “version”: 1, “uptime”: “2 days, 1:31:19”, “numVCPUs”: 2, “cpuUtilization”: { “guest”: 0, “irq”: 0.01, “system”: 1.72, “wait”: 0.27, “idle”: 95.88, “user”: 1.91, “total”: 4.11, “steal”: 0.2, “nice”: 0 },…… It’s possible to create custom CloudWatch metrics from these logs and view those metrics from CloudWatch console. This will require some extra work. However, both built-in and enhanced metrics can be streamed to Sumo Logic from where you can build your own charts and alarms. Regardless of platform, it is recommended to monitor the enhanced metrics for a more complete view of the RDS database engine. The following counters should be monitored for Amazon Aurora, MySQL, MariaDB, PostgreSQL, or Oracle: Metric Group Metric What it means and why you should monitor cpuUtilization user % of CPU used by user processes.

Everyone’s talking about blockchain these days. In fact, there is so much hype about blockchains — and there are so many grand ideas related to them — that it’s hard not to wonder whether everyone who is excited about blockchains understands what a blockchain actually is. If, amidst all this blockchain hype, you’re asking yourself “what is blockchain, anyway?” then this article is for you. It defines what blockchain is and explains what it can and can’t do. Blockchain Is a Database Architecture In the most basic sense, blockchain is a particular database architecture. In other words, like any other type of database architecture (relational databases, NoSQL and the like), a blockchain is a way to structure and store digital information. (The caveat to note here is that some blockchains now make it possible to distribute compute resources in addition to data. For more on that, see below.) What Makes Blockchain Special? If blockchain is just another type of database, why are people so excited about it? The reason is that a blockchain has special features that other types of database architectures lack. They include: Maximum data distribution. On a blockchain, data is distributed across hundreds of thousands of nodes. While other types of databases are sometimes deployed using clusters of multiple servers, this is not a strict requirement. A blockchain by definition involves a widely distributed network of nodes for hosting data. Decentralization. Each of the nodes on a blockchain is controlled by a separate party. As a result, the blockchain database as a whole is decentralized. No single person or group controls it, and no single group or person can modify it. Instead, changes to the data require network consensus. Immutability. In most cases, the protocols that define how you can read and write data to a blockchain make it impossible to erase or modify data once it has been written. As a result, data stored on a blockchain is immutable. You can add data, but you can’t change what already exists. (We should note that while data immutability is a feature of the major blockchains that have been created to date, it’s not strictly the case that blockchain data is always immutable.) Beyond Data As blockchains have evolved over the past few years, some blockchain architectures have grown to include more than a way to distribute data across a decentralized network. They also make it possible to share compute resources. The Ethereum blockchain does this, for example, although Bitcoin—the first and best-known blockchain—was designed only for recording data, not sharing compute resources. If your blockchain provides access to compute resources as well as data, it becomes possible to execute code directly on the blockchain. In that case, the blockchain starts to look more like a decentralized computer than just a decentralized database. Blockchains and Smart Contracts Another buzzword that comes up frequently when discussing what defines a blockchain is a smart contract. A smart contract is code that causes a specific action to happen automatically when a certain condition is met. The code is executed on the blockchain, and the results are recorded there. This may not sound very innovative, but there are some key benefits and use cases. Any application could incorporate code that makes a certain outcome conditional upon a certain circumstance. If-this-then-that code stanzas are not really a big deal. What makes a smart contract different from a typical software conditional statement, however, is that because the smart contract is executed on a decentralized network of computers, no one can modify its outcomes. This feature differentiates smart contracts from conditional statements in traditional applications, where the application is controlled by a single, central authority, which has the power to modify it. Smart contracts are useful for governing things like payment transactions. If you want to ensure that a seller does not receive payment for an item until the buyer receives the item, you could write a smart contract to make that happen automatically, without relying on third-party oversight. Limitations of Blockchains By enabling complete data decentralization and smart contracts, blockchains make it possible to do a lot of interesting things that you could not do with traditional infrastructure. However, it’s important to note that blockchains are not magic. Most blockchains currently have several notable limitations. Transactions are not instantaneous. Bitcoin transactions take surprisingly long to complete, for example. Access control is complicated. On most blockchains, all data is publicly accessible. There are ways to limit access control, but they are complex. In general, a blockchain is not a good solution if you require sophisticated access control for your data. Security. While blockchain is considered a secure place for transactions and storing/sending sensitive data and information, there have been a few blockchain-related security breaches. Moving your data to a blockchain does provide an inherent layer of protection because of the decentralization and encryption features, however, like most things, it does not guarantee that it won’t be hacked or exploited. Additional Resources Watch the latest SnapSecChat videos to hear what our CSO, George Gerchow, has to say about data privacy and the demand for security as a service. Read a blog on new Sumo Logic research that reveals why a new approach to security in the cloud is required for today’s modern businesses. Learn what three security dragons organizations must slay to achieve threat discovery and investigation in the cloud.

There is no question the adoption of public cloud services by businesses is growing at an aggressive rate. And Europe is quickly catching up. Despite increasing pressure from regulatory forces, research by Forrester [content gated], shows adoption growth rates in the European public cloud services market are rapidly catching up with adoption rates in the U.S.

Tables are used to present data in a tabular form and are great for looking at individual values as opposed to trends in data. Simply put, a table consists of an ordered arrangement of data into rows and columns (think of an Excel spreadsheet). The elements of a table may be grouped, segmented or arranged in many different ways. They can even be nested recursively. Tables are widely used in communication, research and data analysis.

I always look forward to attending the annual Gartner Security & Risk Management Summit in National Harbor, Maryland. This event provides the latest insights from both Gartner and industry thought leaders, and is focused on many current challenges facing organizations today with key areas including agile architectures, business continuity management (BCM), cloud security, privacy and securing internet of things (IoT).”

Quick History LessonEarly internet data communications were enabled through the use of a protocol called HyperText Transmission Protocol (HTTP) to transfer data between nodes on the internet. HTTP essentially establishes the “request-response” rules to be used between a “client” (i.e. web browser) and “server”(computer hosting a website) throughout the session. While the use of HTTP grew along with internet adoption, its lack of security protocols left internet communications vulnerable to attacks from malicious actors.In the mid-nineties, Secure Sockets Layer (SSL) was developed to close this gap. SSL is known as a “cryptographic protocol” standard established to enable the privacy and integrity of the bidirectional data being transported via HTTP. You may be familiar with HTTPS or HyperText Transmission Protocol over SSL (a.k.a. HTTP Secure). Transport Layer Security (TLS) version 1.0 (v1.0) was developed in 1999 as an enhancement to the then current SSL v3.0 protocol standard. TLS standards matured over time with TLS v1.1 [2006] and TLS v1.2 [2008].Early Security Flaws Found in HTTPSWhile both SSL and TLS protocols remained effective for some time, in October of 2014, Google’s security team discovered a vulnerability in SSL version 3.0. Skilled hackers were able to use a technique called Padding Oracle On Downgraded Legacy Encryption — widely referred to as the “POODLE” exploit to bypass the SSL security and decrypt sensitive (HTTPS) information including secret session cookies. By doing this, hackers could then hijack user accounts.In December 2014, the early versions of TLS were also found to be vulnerable from a new variant of the POODLE attack exploits, that enabled hackers to downgrade the protocol version to one that was more vulnerable.Poodle Attacks Spur Changes to PCI StandardsSo what do POODLE attacks have to do with Payment Card Industry Data Security Standards (PCI DSS) standards and compliance? PCI DSS Requirement 4.1 mandates the use of “strong cryptography and security protocols to safeguard sensitive cardholder data during transmission” and these SSL vulnerabilities (and similar variants) also meant sensitive data associated with payment card transactions was also open to these risks. And in April of 2015 the PCI Standards Security Council (SSC) issued a revised set of industry standards — PCI DSS v3.1, which stated “SSL has been removed as an example of strong cryptography in the PCI DSS, and can no longer be used as a security control after June 30, 2016.”This deadline applied to both organizations and service providers to remedy this situation in their environments by migrating from SSL to TLS v1.1 or higher. They also included an information supplement: “Migrating from SSL and Early TLS” as a guide.However, due to early industry feedback and push back, in December of 2015 the PCI SSC issued a bulletin extending the deadline to June 30, 2018 for both service providers and end users to migrate to higher, later versions of TLS standards. And in April of 2016 the PCI SSC issued PCI v3.2 to formalize the deadline extension and added an “Appendix 2” to outline the requirements for conforming with these standards.Sumo Logic Is Ready, Are You?The Sumo Logic platform was built with a security-by-design approach and we take security and compliance very seriously. As a company, we continue to lead the market in securing our own environment and providing the tools to help enable our customers to do the same.Sumo Logic complied with the the PCI DSS 3.2 service provider level one standards in accordance with the original deadline (June 30, 2016), and received validation from a third party expert, Coalfire.If your organization is still using these legacy protocols it is important to take steps immediately and migrate to the newest versions to ensure compliance by the approaching June 30, 2018 deadline.If you are unsure whether these vulnerable protocols are still in use in your PCI environment, don’t wait until it’s too late to take action. If you don’t have the resources to perform your own audit, the PCI Standards Council has provided a list of “Qualified Security Assessors” that can help you in those efforts.What About Sumo Logic Customers?If you are a current Sumo Logic customer, in addition to ensuring we comply with PCI DSS standards in our own environment, we continually make every effort to inform you if one or more of your collectors are eligible for an upgrade.If you have any collectors that might still be present in your PCI DSS environment that do not meet the new PCI DSS standards, you would have been notified through the collectors page in our UI (see image below). It’s worthwhile to note that TLS v1.1 is still considered PCI compliant, however, at Sumo Logic we are leapfrogging the PCI requirements and moving forward, we will only be supporting TLS v1.2.If needed you can follow these instructions to upgrade (or downgrade) as required.Sumo Logic Support for PCI DSS ComplianceSumo Logic provides a ton of information, tools and pre-built dashboards to our customers to help with managing PCI DSS compliance standards in many cloud and non-cloud environments. A collection of these resources can be found on our PCI Resources page.If you are a cloud user, and are required to manage PCI DSS elements in that type of environment, in April 2018 the PCI SSC Cloud Special Interest Group issued an updated version 3.0 to their previous version 2.0 that was last released in February 2013.Be looking for another related blog to provide a deeper dive on this subject.PCI SSC Cloud Computing Guidelines version 3.0 include the following changes:Updated guidance on roles and responsibilities, scoping cloud environments, and PCI DSS compliance challenges.Expanded guidance on incident response and forensic investigation.New guidance on vulnerability management, as well as additional technical security considerations on topics such as Software Defined Networks (SDN), containers, dog computing and internet of things (IoT).Standardized terminology throughout the document.Updated references to PCI SSC and external resources.Additional ResourcesFor more information on the compliance standards Sumo Logic supports visit our self-service portal. You’ll need a Sumo Logic account to access the portal.Visit our DocHub page for specifics on how Sumo Logic helps support our customer’s PCI compliance needsSign up for Sumo Logic for free to learn more

Watch this short video from Sumo Logic on the need for a cloud-native, flexible and scalable log analytics solution that helps DevOps teams be successful.

Before you roll your eyes at another “as a service” term, listen to what Sumo Logic CSO George Gerchow has to say about it in this latest SnapSecChat video series. The reason why offering solutions “as a service” has become so widespread is because that’s the way it should be done, especially with security. Ultimately, when you are in the practice of security, people are your customers and you want to provide guardrails for them, to make for an easy and transparent process. Why? Because the reality of it is that security is hard, and your customers don’t always have the time to handle it themselves — that’s why they work with vendors who are security subject matter experts. At Sumo Logic we’ve provided a security as a service portal, based on customer demand and inquiries. To hear more about George’s perspective on SECaaS, and to learn how you can take a holistic view to security that enables your customers instead of creating a bottleneck, watch the video. If you enjoyed this SnapSecChat, then be sure to stay tuned for another one coming to a website near you soon! And don’t forget to follow George (@GeorgeGerchow) on Twitter and use the hashtag #SnapSecChat to join the security conversation!

Quick RefresherEarlier this year, we showed you how to monitor Amazon Web Services Elastic Load Balancer (AWS ELB) with Cloudwatch. This piece is a follow up to that, and will focus on Classic Load Balancers. Classic Load Balancers provide basic load balancing across multiple Amazon EC2 instances and operate at both the request level and connection level. Classic Load Balancers are intended for applications that were built within the EC2-Classic network. AWS provides the ability to monitor your ELB configuration with detailed logs of all the requests made to your load balancers. There is a wealth of data in the logs generated by ELB, and it is extremely simple to set up.How to Get Started: Setting up AWS ELB LogsLogging is not enabled in AWS ELB by default. It is important to set up logging when you start using the service so you don’t miss any important details!Step 1: Create an S3 Bucket and Enable ELB LoggingNote: If you have more than one AWS account (such as ops, dev, and so on) or multiple regions that generate Elastic Load Balancing data, you’ll probably need to configure each of these separately.Here are the key steps you need to followCreate an S3 Bucket to store the logsNote: Want to learn more about S3? Look no further (link)Allow AWS ELB access to the S3 BucketEnable AWS ELB Logging in the AWS ConsoleVerify that it is workingStep 2: Allow Access to external Log Management ToolsTo add AWS ELB logs to your log management strategy, you need to give access to your log management tool! The easiest way to do that is by creating a special user and policy.Create a user in AWS Identity and Access Management (IAM) with Programmatic Access. For more information about this, refer to the appropriate section of the AWS User Guide.Note: Make sure to store the Access Key ID and Secret Access Key credentials in a secure location. You will need to provide these later to provide access to your tools!Create a Custom Policy for the new IAM user. We recommend you use the following JSON policy:{“Version”:”2012-10-17″,“Statement”:[{“Action”:[“s3:GetObject”,“s3:GetObjectVersion”,“s3:ListBucketVersions”,“s3:ListBucket”],“Effect”:”Allow”,“Resource”:[“arn:aws:s3:::your_bucketname/*”,“arn:aws:s3:::your_bucketname”]}]}Note: All of the Action parameters shown above are required. Replace the “your_bucketname” placeholders in the Resource section of the JSON policy with your actual S3 bucket name.Refer to the Access Policies section of the AWS User Guide for more info.What do the Logs look like?ELB logs are stored as .log files in the S3 buckets you specify when you enable logging.The file names of the access logs use the following format:bucket[/prefix]/AWSLogs/aws-account-id/elasticloadbalancing/region/yyyy/mm/dd/aws-account-id_elasticloadbalancing_region_load-balancer-name_end-time_ip-address_random-string.logbucketThe name of the S3 bucket.prefixThe prefix (logical hierarchy) in the bucket. If you don’t specify a prefix, the logs are placed at the root level of the bucket.aws-account-idThe AWS account ID of the owner.regionThe region for your load balancer and S3 bucket.yyyy/mm/ddThe date that the log was delivered.load-balancer-nameThe name of the load balancer.end-timeThe date and time that the logging interval ended. For example, an end time of 20140215T2340Z contains entries for requests made between 23:35 and 23:40 if the publishing interval is 5 minutes.ip-addressThe IP address of the load balancer node that handled the request. For an internal load balancer, this is a private IP address.random-stringA system-generated random string.The following is an example log file name:s3://my-loadbalancer-logs/my-app/AWSLogs/123456789012/elasticloadbalancing/us-west-2/2014/02/15/123456789012_elasticloadbalancing_us-west-2_my-loadbalancer_20140215T2340Z_172.160.001.192_20sg8hgm.logSyntaxEach log entry contains the details of a single request made to the load balancer. All fields in the log entry are delimited by spaces. Each entry in the log file has the following format:timestamp elb client:port backend:port request_processing_time backend_processing_time response_processing_time elb_status_code backend_status_code received_bytes sent_bytes “request” “user_agent” ssl_cipher ssl_protocolThe following table explains the different fields in the log file. Note: ELB can process HTTP requests and TCP requests, and the differences are noted below:FieldDescriptiontimestampThe time when the load balancer received the request from the client, in ISO 8601 format.elbThe name of the load balancerclient:portThe IP address and port of the requesting client.backend:portThe IP address and port of the registered instance that processed this request.request_processing_time[HTTP listener] The total time elapsed, in seconds, from the time the load balancer received the request until the time it sent it to a registered instance.

Graphite Metrics are one of the most common metrics formats in application monitoring today. Originally designed in 2006 by Chris Davis at Orbitz and open-sourced in 2008, Graphite itself is a monitoring tool now used by many organizations both large and small. It accepts metrics from a wide variety of sources, including popular daemons like collectd and statsd, provided that the metrics are sent in the following simple format: Where metric path is a unique identifier, specified in a dot-delimited format. Implicit in this format is also some logical hierarchy specific to each environment, for example: While this hierarchical format has been widely accepted in the industry for years, it creates challenges for usability and ultimately lengthens the time to troubleshoot application issues. Users need to carefully plan and define these hierarchies ahead of time in order to maintain consistency across systems, scale monitoring effectively in the future and reduce confusion for the end user leveraging these metrics. Fortunately, the industry is evolving towards tag-based metrics to make it easier to design and scale these systems, and Sumo Logic is excited to announce the launch of Metrics Rules to take advantage of this new model immediately. Using Metrics Rules to Bring Graphite Metrics into the New World Sumo Logic built its metrics platform to support metadata-rich metrics, but we also acknowledged that the broader industry and many of our customers have invested heavily in their Graphite architecture and naming schemas over time. Sumo Logic’s Metrics Rules solution now allows users to easily transform these Graphite metrics into the next generation, tag-based metric format, which provides three key benefits: Faster Time to Value: No need to re-instrument application metrics to take advantage of this metadata-rich, multi-dimensional format. Send Graphite-formatted metrics to Sumo immediately and enrich them with tag-based metadata later. Easy Configuration: An intuitive user interface (UI) allows you to validate and edit your transformation rules in real-time, while competitive solutions require carefully defined config files that are difficult to set up and prone to errors. Improved Usability: With rich metadata, use simple key-value pairs to discover, visualize, filter and alert on metrics without knowing the original Graphite-based hierarchy. Using the example above, we can use Metrics Rules to enrich the dot-delimited Graphite names with key-value tags, which will make it easier for us to monitor metrics by our system’s logical groupings in the future: Intuitive Metrics Rules UI for Easy Validation and Edits As Graphite monitoring systems grow, so do the complexities in maintaining these dot-delimited hierarchies across the organization. Some teams may have defined Graphite naming schemes with five different path components (e.g., app.env.host.assembly.metric), while others may have more components or a different hierarchical definition altogether. To make it easier to create tags out of these metrics, the Metrics Rules configuration interface allows you to see a preview of your rules and make sure that you’ve properly captured the different components. Simply specify a match expression (i.e., which metrics the rule will apply to), define variables for each of the extracted fields and then validate that each tag field is extracting the appropriate values. After saving the rule, Sumo Logic will go back in time and tag your metrics with this new metadata so you can take advantage of these rules for prior data points. Improved Discoverability, Filtering and Alerting with Key-Value Tags Once these metrics contain the key-value tags that we’ve applied via Metrics, you can take advantage of several usability features to make finding, visualizing and alerting on your metrics even easier. For example, Sumo Logic’s autocomplete feature makes it easier to find and group metrics based on these key-value tags: Additionally, when using our unified dashboards for logs and metrics, these new tags can be leveraged as filters for modifying visualizations. Selecting a value in one of these filters will append a key-value pair to your query and filter down to the data you’re interested in: Finally, configuring alerts becomes significantly easier when scoping and grouping your metrics with key-value pairs. In the example below, we selected metric=vcpu.user from one of our namespaces, and we’re averaging this across each node in Namespace=csteam. This means that alerts will trigger across each node, and our email and/or webhook notifications will tell us which particular node has breached the threshold: The Bigger Picture Users can now convert legacy Graphite-formatted performance metrics into the metadata-rich metrics with Sumo Logic, both in real-time and after ingestion. This allows customers to increase the usability and accessibility for their analytics users by allowing them to leverage business relevant tags, instead of relying only on obscure, technical tags. Now with the capability to extract business context (metadata) from IT-focused metrics, organizations can use this data to gain actionable insight to inform strategic business decisions. In a broader context, this is significant because as we’ve been seeing from our customers, the hard lines between IT and business are becoming blurred, and there’s a strong emphasis on using data to improve the overall end-user experience. As more organizations continue to leverage machine data analytics to improve their security, IT and business operations, the ability to map machine data insights to actionable, contextual business analytics for IT and non-core-IT users is critical. Learn More Head over to Sumo Logic DocHub for more details on how to configure Metrics Rules on your account. Additionally, see how these rules can even be used for non-Graphite metrics by parsing out values from existing key-value pairs such as _sourceCategory and _sourceHost. Are you at DockerCon 2018 at Moscone Center in San Francisco this week? We’ll be there! Stop by our booth S5 to chat with our experts, get a demo and to learn more! Additional Resources Read the press release on our latest product enhancements unveiled at DockerCon Download the report by 451 Research & Sumo Logic to learn how machine data analytics helps organizations gain an advantage in the analytics economy Check out the Logs-to-Metrics blog Sign up for Sumo Logic for free

I continue to be intrigued by the evolution of software architectures and their impact on business. In my 20+ year career, I’ve participated in four of these architecture transitions – the shift from client-server to the internet, the rise of 3-tier architectures underpinning rich internet applications, virtualization that upended the dominance of hardware providers, and now the shift to microservices-based architectures based on cloud infrastructure and software automation.

Monitoring Kubernetes - understanding what an application does and how it functions is critical to monitoring it effectively. Now it’s time to journey a bit deeper into each of those components and understand what you need to keep a close eye on.

In this Sumo Logic Employee Spotlight we interview Michael Halabi. Mike graduated from UC Santa Cruz with a bachelor’s degree in business management economics, spent some time as an auditor with PwC, joined Sumo Logic as the accounting manager in the finance department, and recently transitioned to a new role at the company as a DevSecOps engineer. [Pause here for head scratch] I know what you’re thinking, and yes that is quite the career shift, but if you stay with us, there’s a moral to this story, as well as a few lessons learned. Work Smarter, Not Harder Q: Why did you initially decide business management economics was the right educational path? Mike Halabi (MH): I fell into the “uncertain college kid” category. While I was interested in engineering, I was also an entrepreneur at heart and knew that someday, if I were to start my own business, I would need a foundational business background as well as a variety of other life experiences outside of textbook knowledge. Q: How do you approach your work? MH: Everything in life, no matter how scary it may appear up front, can be broken into a series of simpler and smaller tasks. If you learn how to think about problem solving in a certain way, you can make anything work, no matter how far beyond your skill set and core competency it may originally seem. This is especially true in the technology industry where success often depends on doing it not just better, but also faster, than the competition. Breaking down complex problems into bite size chunks allows you to tackle each piece of the problem quickly and effectively and move on to the next. Q: What’s the best way for a business to achieve that — doing it better and faster? MH: Automation. This is applicable across the board. The finance industry is full of opportunities to automate processes. Half of what a traditional finance team spends its time doing is copy/pasting the same information into the same email templates or copy/pasting a formula in excel and manually tweaking each line. In other words, a bunch of tedious outdated practices that could be easily automated thanks to modern programs and technologies. One instance I recall is someone spending a full day calculating a small subset of a massive spreadsheet line by line: eight hours to do one-tenth of the massive workbook. With a proper understanding of the problem and how to leverage the tools available, I wrote a formula to copy/paste in 30 minutes that completed the entire workbook and is still in use today. Scalable, simple, efficient — this formula removes manual error and works every time. And this was a quarterly project, so that many weeks’ worth of highly paid time is saved every quarter. Low hanging fruit like this is everywhere. Q: So how did you capture the attention of Sumo Logic’s technical team? MH: Word got out about my closet-coding (really, I annoyed everyone to death until they let me help with something fun) and soon, various people in various teams were sending side projects on troubleshooting and automation my way. I continued on like this for awhile — finance accounting manager by day, coder by night until I was approached by our CSO and asked if I’d like to transition onto his team as a DevSecOps engineer. Connect the Dots Q: Let’s back up. How did you initially get into coding? MH: I took an early liking to video game development and while I didn’t have a formal engineering or coding background, using the above methodology, I taught myself how to make simple games using C++/SDL. Then, once I started helping out with various projects at Sumo Logic, I discovered Python, C# and Go. By spending time experimenting with each language I found different use-cases for them and in trying to apply what I’d learned I was pushed into the altogether different world of infrastructure. Making solutions was easy enough, getting them to less technically inclined folks became a new challenge. In order to deploy many of my cross functional projects at Sumo Logic, I had to learn about Docker, Lambda, EC2, Dynamo, ELBs, SSL, HTTP, various exploits/security related to web-based tech, etc. I devoted more of my time to learning about the backend and underlying technologies of the modern internet because making a service scalable and robust requires a holistic skillset beyond simply writing the code. Q: Are there any interesting parallels between finance and engineering? MH: As an auditor at PwC, I worked frequently with many companies from the very early startup stage to large public companies, and the problems most all of these companies face are the same. How do we get more done without hiring more people or working longer hours, and without sacrificing work quality. In finance the problem is handled generally by hiring more at a lot of companies. Q: Can you expand on that? MH: You need to look beyond the company financials. Increased revenue to increased work can’t (or should never) be a 1:1 ratio. For a company to scale, each individual employee has to understand how his or her role will scale in the future to keep pace with corporate growth and needs. You scale an organization by using technology, not by mindlessly throwing bodies at the work. That’s what I learned from finance. You don’t need a team of 10 people to collect money and write the same email to clients multiple times a day, when you can automate and have a team of two handle it. Manual processes are slow and result in human error. In engineering I think this concept is well understood, but in finance, in my experience, many companies behave as if they’re still in the 1500s with nothing more than an abacus to help them do their job. Find a Passion Project Q:What would be your advice to those considering a major career shift? MH: Our interests and passions will shift over time, and there’s nothing wrong with that. If you decide one day to do a complete 180 degree career change, go for it. If you don’t genuinely enjoy what you do, you’ll never truly advance. I loved designing video games and automating financial processes, which led to my career shift into engineering. Did I put in long hours? Yes. Did I enjoy it? Yes. Passion may be an oversung cliche but if you aren’t invested in your work, you’ll go through the motions without reaping any of the benefits, including the satisfaction of producing meaningful work that influences others in a positive way. Q: What’s your biggest learning from this whole experience? MH: The biggest takeaway for me as a coder was that theoretical knowledge doesn’t always apply in the real world because you can’t know how to make something until you make it. Coding is an iterative process of creating, breaking and improving. So never be afraid to fail occasionally, learn from it and move on. And don’t put yourself in a box or give up on your dreams simply because you don’t have a formal education or piece of paper to prove your worth. The technology industry is hungry for engineering talent, and sometimes it can be found in unusual places. In fact, finding employees with robust skill sets and backgrounds will only positively impact your team. Our collective experiences make us holistically stronger.

As a launch parter for Amazon EKS, Sumo Logic Sumo debuts the Sumo Logic Amazon EKS App to provide organizations with complete visibility into their containerized apps, giving developers the necessary insight into all applications running in Amazon EKS, ensuring that those apps are always available to customers.

Customers are visiting your website, employees are logging into your systems and countless machines are talking to each other in an effort to deliver the perfect user experience. We’d like to believe that all of these individuals and machines are operating with the best of intentions, but how can we be so sure? One possible answer lies in the connecting device’s IP address and its respective physical location. IP geolocation is the process of determining the location of a device based on its unique IP address. It not only requires knowledge about the physical location of the computer where the IP address is assigned, but also how the device is connecting (e.g., via anonymous proxy, mobile, cable, etc.). This challenge becomes further complicated in an increasingly digital world with proliferating devices and millions of connections being established across the globe daily. That’s why we’re excited to announce that we’ve partnered with Neustar, a leading IP intelligence provider, to deliver one of the most comprehensive and precise geolocation databases in the industry. As a Sumo Logic customer, you can now leverage Neustar’s 20+ years of experience gathering and delivering IP intelligence insights, all at no additional charge. Precision Database + Weekly Updates = Higher Confidence Analytics In the pre-cellphone era (remember that?), everyone had a landline which meant area codes were fairly accurate identifiers of an end-user location. I knew that 516 meant someone was calling from Long Island, New York, while 415 was likely coming from the San Francisco Bay Area. But the invention of the cellphone complicated this matter. I might be receiving a call from someone with a 516 number, but because the caller was using a “mobile” device, he or she could be located anywhere in the U.S. IP addresses are like very complicated cellphone numbers — they can be registered in one place, used in another and then re-assigned to someone else without much notice. Keeping track of this is an enormous task. And over time, malicious actors realized that they could take advantage of this to not only mask their true location, but create false security alerts to distract security teams from identifying and prioritizing legitimate high-risk threats. That’s why partnering with a leader like Neustar, that uses a global data collection network and a team of network geography network analysts, to update their IP GeoPoint database on a daily basis, is key. This accuracy allows security teams to have full visibility into their distributed, global IT environment and when there’s an attempt to compromise a user’s credentials within an application, they can quickly flag any anomalous activity and investigate suspicious logins immediately. Proactive Geo Monitoring and Alerting in Sumo Logic With Neustar’s IP GeoPoint database, you can rest assured that your geolocation results are more trustworthy and reliable than ever before. Using Sumo Logic, you can continue to take advantage of the proactive alerting and dashboarding capabilities to make sense of IP intelligence across your security and operational teams. For example, you’ll have a high confidence in your ability to: Detect Suspicious Logins: alert on login attempts occurring outside of trusted regions. Maintain Regulatory Compliance: see where data is being sent to and downloaded from to keep information geographically isolated. Analyze End-User Behavior: determine where your users are connecting from to better understand product adoption and inform advertising campaigns. With real-time alerts, for example, you can receive an email or Slack notification if a login occurs outside of your regional offices: Configure real-time alerts to get notified when a machine or user is appearing from outside of a specific region. You can also use real-time dashboards to monitor the launch of a new feature, track customer behavior or gain visibility into AWS Console Logins from CloudTrail: Using Sumo Logic’s Applications, you can install out-of-the-box dashboards for instant geographic visibility into AWS Console Logins, for example. The Bigger Picture Born in AWS, Sumo Logic has always held a cloud-first, security-by-design approach and our vision is to create a leading cloud security analytics platform to help our customers overcome the challenges of managing their security posture in the cloud. There is a major gap in the available on-premises security tools for customers that not only need to manage security in the cloud, but also meet rigorous regulatory compliance standards, especially the European Union’s General Data Protection Regulation (GDPR) that went into effect last week on May 25, 2018. Geolocation is key for those needs which is why we’re thrilled to be rolling this out to our customers as part of a bigger strategy to provide visibility and security across the full application stack. Learn More Head over to Sumo Logic DocHub for more details on how to leverage the new database, then schedule some searches and create dashboards to take advantage of the enhanced IP geolocation. Check out our latest press announcement to learn about the additional features and to our cloud security analytics solution, including intelligent investigation workflows, privacy and GDPR dashboards, and enhanced threat intelligence.

AWS Database Services In a previous article of this series, we introduced Amazon S3 and Amazon Glacier, both suitable for storing unstructured or semi-structured data in the Amazon Web Services (AWS) cloud. In most cases, organizations have different types of databases powering their applications. AWS offers a number of robust, scalable and secured database services which makes them an ideal substitute for on-premise databases. In this article, we’ll compare some of the most popular AWS database services to help you select the right option for your organization.

In today’s world, we are surrounded by data. It’s flying through the air all around us over radio signals, wireless internet, mobile networks and more. We’re so immersed in data every day that we rarely stop to think about what the data means and why it is there. I, for one, rarely have a quiet moment where I am not directly, or indirectly, absorbing a flow of information, regardless of whether I want to or not.So, I wonder, are we going to master this all-encompassing data, or be mastered by it?Data is the new currency of our worldData has become the lifeblood of the modern business, and those who succeed in today’s competitive landscape are those who leverage data the best. Amazing applications exist that take the raw data flowing out of the exhaust pipe of the modern economy (and our lives) and enable companies to develop products we couldn’t have even conceived of a few years ago. Artificial intelligence-driven innovations are anticipating our next move. Social media products are connecting us and targeting us. Self-driving cars are swimming in data to navigate a world full of faulty humans. But, how often do we stop to talk about the good, the bad and the ugly of data?[epq-quote align="align-right"]"A single conversation across the table with a wise man is better than ten years mere study of books." - Henry Wadsworth Longfellow [/epq-quote]The discussions now about data privacy and the nonstop stream of hackers stealing our personal information, are actually elevating data from a sub-theme of our culture into a primary topic, even for non-experts. The value of data is also rising into the awareness of boardrooms, and this is a good thing. The only way to keep ourselves honest about how we use data is to talk about how we use data — the wonderful and the despicable, the innovative and the regressive.A new podcast to explore the data revolution[epq-quote align="align-left"]The only way to keep ourselves honest about how we use data is to talk about how we use data — the wonderful and the despicable, the innovative and the regressive" - Ben Newton, director of product marketing, Sumo Logic[/epq-quote]As a long-time podcast listener and fan of the spoken word, I am excited to announce a new podcast about this data revolution — Masters of Data. Each episode we interview innovators, big thinkers and provocateurs, to learn theirs views about data.We also want to meet the people behind the data, who can humanize the data by helping us understand the cultural content and intention of the data on human experience. That way we turn this from stories about widgets and gimmicks into stories about humans and the value of data, as well as the dangers of misusing it.Our first podcast is now live and features Bill Burns, the chief trust officer at Informatica. Bill and I take a journey through time and security and discuss the evolving tech landscape over the last 20 years. We talk about how he got started in a computer lab, cut his teeth at Netscape, helped change the world at Netflix, and is on his next journey at Informatica.How to listen and subscribeTo listen visit https://mastersofdata.sumologic.com or subscribe via the iTunes or Google Play app stores. Once you’ve subscribed on your favorite platform, be sure to check back for new episodes and leave us a review to let us know what you think! We will be releasing several discussions over the next few months, and we look forward your feedback!Until then, listen and enjoy.And don’t forget to join the conversation on Twitter #mastersofdata

*Authored by Kevin Keech, Director of Support at Sumo Logic, and Graham Watts, Senior Solutions Engineer at Sumo Logic Many Sumo Logic customers ask, “How can I use Sumo Logic for support and customer success teams?” If you need a better customer experience to stay ahead of the competition, Sumo Logic can help. In this post, I will describe why and how support and customer success teams use Sumo Logic, and summarize the key features to use in support and customer success use cases. Why Use Sumo Logic For Support and Customer Success Teams? Improved Customer Experience Catch deviations and performance degradation before your customers report it Using dashboards and scheduled alerts, your CS and Support teams can be notified of any service impacting issues and can then reach out and provide solutions before your customers may ever know they have a problem This helps your customers avoid experiencing any frustrations with your service, which in the past may have led them to look into competitive offerings Improve your Net Promoter Score (NPS) and Service Level Agreements (SLAs) Alert team members to reach out to a frustrated customer before they go to a competitors website or log out Efficiency and Cost Savings – Process More Tickets, Faster Sumo Logic customers report an increase in the number of support tickets each team member can handle by 2-3x or more Direct access to your data eliminates the need for your Support team to request access and wait for engineering resources to grant access This leads to a higher level of customer satisfaction, and allows you to reallocate engineering time to innovate and enhance your product offerings Your support reps can perform real-time analysis of issues as they are occurring, locate the root of a problem, and get your customers solutions quicker Customers report that using LogReduce cuts troubleshooting time down from hours/days to minutes As your teams and products grow, team members can process more tickets instead of needing to hire more staff Security Eliminate the need to directly log into servers to look at logs – you can Live Tail your logs right in Sumo Logic or via a CLI Use Role Based Access Control to allow teams to view only the data they need How to Use Sumo Logic For Support and Customer Success Teams Key features that enable your Support Team, Customer Success Team, or another technical team while troubleshooting are: Search Templates See here for a video tutorial of Search Templates Form based search experience – no need for employees to learn a query language Users type in human-friendly, easy to remember values like “Company Name” and Sumo will look up and inject complex IDs, like “Customer ID” or some other UUID, into the query, shown to the right: LogReduce Reduce 10s or 100s of thousands of log messages into a few patterns with the click of a button This reduces the time it takes to identify the root cause of an issue from hours or days to minutes In the below example, a bad certificate and related tracebacks are exposed with LogReduce Dashboards Dashboard filters – Auto-populating dashboard filters for easy troubleshooting TimeCompare – Is now ‘normal’ compared to historical trends? The example below shows production errors or exceptions today, overlaid with the last 7 days of production errors or exceptions:

Today at Sumo Logic, we’re excited to announce that registration as well as call for speaker papers are open for our annual user conference Illuminate 2018! For those that did not attend last year or are unfamiliar with Illuminate, it’s a two-day event where Sumo Logic users and ecosystem partners come together to share best practices for shining the light on continuous intelligence for modern applications. Illuminate 2018 takes place Sept. 12-13, 2018 at the Hyatt Regency San Francisco Airport Hotel in Burlingame, Calif., and registration details can be found on the conference website under the frequently asked questions (FAQ) section. Why Attend? The better question is why not? Last year, the conference brought together more than 400 customers, partners, practitioners and leaders across operations, development and security for hands-on training and certifications through the Sumo Logic Certification Program, as well as technical sessions and real-world case studies to help attendees get the most out of the Sumo Logic platform. You can register for Illuminate 2018 here: The 2017 keynote was chock-full of interesting stories from some of our most trusted and valued customers, partners and luminaries, including: Ramin Sayar, president and CEO, Sumo Logic David Hahn, CISO, Hearst Chris Stone, chief products officer, Acquia Special guest Reid Hoffman, co-founder of LinkedIn; partner of Greylock Partners You can watch the full 2017 keynote here, and check out the highlights reel below! Interested in Speaking at This Year’s Conference? If you’ve got an interesting or unique customer use-case or story that highlights real-world strategies and technical insights about how machine data analytics have improved operations, security and business needs for you and your end-users, then we’d love to hear from you! These presentations must provide a holistic overview into how today’s pioneers are pushing the boundaries of what’s possible with machine data analytics and developing powerful use cases within their industries or organizations. Still on the fence? Keep in mind that if your session is accepted, you’ll receive one complimentary registration to Illuminate as well as branding and promotional opportunities for your session. More on Topics, Requirements & Deadline To make it easier on our customers, we’ve compiled a list of desired topics to help guide you in the submission process. However, this is not an exhaustive list, so if you have another interesting technical story to tell that is relevant to Sumo Logic users and our ecosystem, we’d love to hear it. The Journey to the Cloud (Cloud Migration) Operations and Performance Management of Applications and Infrastructure Operations and Performance Management for Containers and Microservices Best Practices for using Serverless Architectures at scale Cloud Security and Compliance Best practices for implementing DevSecOps Sumo Logic to Enable Improved Customer Support/Success Unique Use Cases (I Didn’t Know You Could Use Sumo to…) Best Practices to Adopting and Leveraging Sumo Logic Effectively within your Organization Regardless of topic, all submissions MUST include: Title Brief Abstract (200-400 words) 3 Key Audience Takeaways or Learnings Speaker Bio and Links to any Previous Presentations The deadline to submit session abstracts is June 22, 2018. Please email submissions to [email protected]. Examples of Previous Customer Presentations Last year, we heard great customer stories from Samsung SmartThings, Hootsuite, Canary, Xero, and more. Here are a few customer highlight reels to give you examples of the types of stories we’d like to feature during Illuminate 2018.  Final Thoughts Even if you aren’t thinking of submitting a call for paper, we’d love to see you at Illuminate to participate in two full days of learning, certification and trainings, content sharing, networking with peers and Sumo Logic experts, and most importantly, fun! You can register here, and as a reminder, call for papers closes June 22, 2018! Together, we can bring light to dark, and democratize machine data analytics for everyone. If you have any additional questions, don’t hesitate to reach out to [email protected]. Hope you see you there!

D-Day for the General Data Protection Regulation (GDPR) is right around the corner. Many companies have had to adjust and revise the way they operate their businesses, both in and outside of the European Union to make sure they meet the necessary GDPR requirements. But what if you haven’t even begun to think about GDPR and how it will affect your organization? First, don’t panic. Second, be transparent about your progress with your customers and key stakeholders. Third, recognize the hurdles ahead and develop a road map for dealing with these challenges — starting now! For more tips on what your organization can start doing now to prepare for GDPR, watch the latest SnapSecChat video series with Sumo Logic’s Chief Security Officer George Gerchow. Remember — GDPR will be an evolving regulation, and there is no official “GDPR-readiness” seal of approval, so it’s not too late to start implementing or refining your strategy now. If you enjoyed this video, then be sure to stay tuned for another one coming to a website near you soon! And don’t forget to follow George on Twitter at @GeorgeGerchow, and use the hashtag #SnapSecChat to join the security (and GDPR) conversation!

Amazon Web Services (AWS) is the most popular cloud service provider today. Since inception, AWS has pioneered and revolutionized the way modern organizations deploy, operate and manage their IT services in a virtualized environment.

The cloud is an attractive proposition not just for enterprises, but also for startups looking to disrupt markets. Leveraging the cloud, innovative startup teams can disrupt legacy business models and markets by reducing cost and time to market. Most developers have a very good understanding of Amazon Web Services (AWS), but not too many have taken the plunge into Google Cloud Platform (GCP) and its offerings yet.

As organizations scale and grow, teams begin to emerge with areas of specialization and ownership. Dependencies develop, with individuals and teams acting as service providers to other functional areas. We’re finding information technology and DevOps teams rely on Sumo Logic for not just their own monitoring of application health and infrastructure, but also for sharing this data out to other functional areas such as customer support and business analysts, to empower them to make stronger data-driven decisions. Our top customers such as Xero, Acquia and Delta each have hundreds if not thousands of users spread across multiple teams who have unique data needs that align with varying business priorities and wide ranges of skill sets. Scaling our platform to support the data distribution and sharing needs of such a broad and diverse user base is a key driver of Sumo Logic’s platform strategy. To this end, we are excited to announce the general availability of our new Content Sharing capabilities. Underlying this updated ability to share commonly used assets such as searches and dashboards is a secure, fine-grained and flexible role-based access control (RBAC) model. Check out our intro to content sharing video for more details. Collaboration with Control The updated model allows for data visualization assets such as searches and dashboards to be shared out individually (or grouped in folders) to not just other users, but also to members of a particular role. Users can be invited to simply view the asset or participate in actively editing or managing it. This allows for individual users to collaborate on a search or dashboard before sharing it out to the rest of the team. When new information is discovered or a meaningful visualization is created, they can be easily added to a dashboard by anyone with edit access and and the change is made immediately available to other users. While ease-of-use is critical in ensuring a smooth collaboration experience, it should not come at the price of data security. It is always transparent as to who has what access to an asset and this access can be revoked at any time by users with the right privileges. Users can control the data that is visible to the viewers of a dashboard and new security measures such as a “run-as” setting have been introduced to prevent viewing or exploitation of data access settings put in place by administrators. Business Continuity Supporting business continuity is a key conversation we have with our customers as their businesses grow and their data use-cases evolve and deepen. This new set of features reflects Sumo Logic’s belief that the data and its visualizations belong to the organization and should be easily managed as such. We’ve replaced our single-user ownership model with a model where multiple users can manage an asset (ie, view, edit and if required, delete it). This ensures that a team member could, for example, step in and update an email-alert-generating scheduled search when the original author is unavailable. Even if the asset was not shared with other team members, administrators now have the ability (via a newly introduced “Administrative Mode”) to manage any asset in the library, regardless of whether it was actively shared with them or not. The user deletion workflow has been simplified and updated to support business continuity. When a user is deleted, any of their searches and dashboards that are located in a shared folder continue to exist where they are, thus providing a seamless experience for team members depending on them for their business workflows. Administrators can assign and transfer the management of any assets located in the user’s personal folder to another team member with the right context. (Previously, administrators would have had to take ownership and responsibility.) Teams as Internal Service Providers Development teams that own a specific app or run an area of infrastructure, are increasingly seen as internal service providers, required to meet SLAs and demonstrate application health and uptime. They need to make this information easy to discover and access by field and service teams who in turn, rely on this data to support end customers. Our newly introduced Admin Recommended section allows administrators to set up organizational folder structures that facilitate this easy interaction between producers and consumers. Relevant folders are pinned to the very top of our Library view, enabling easy discoverability to infrequent users. The contents of these folders can be managed and updated by individual development teams, without the administrator needing to intervene. This allows for sharing of best practices, commonly used searches and dashboards as well as enables users to create team-specific best practice guides. As data-driven decision-making becomes the norm across all job functions and industries, the Sumo Logic platform is designed to scale in performance, usability and security to meet these needs. Features like fine-grained RBAC and administrative control, coupled with intuitive design make information accessible to those who need it the most and enable the creation of workflows that democratize data and drive business growth. Upcoming Webinars To learn more about how these features can be applied at your organization, please sign up for one of our upcoming webinars on May 1st (2:00 p.m. PST) or May 3rd (9:00 a.m. PST)

That's a wrap for RSA Conference 2018, and besides being exhausted from a week full of show activities, we wanted to hear what our CSO, George Gerchow, had to say about RSA this year, and how it has or hasn't changed in the past 14 years since he's attended. What's the long and short of it? Overall -- not a bad showing with over 50 thousand attendees infiltrating the streets of San Francisco hungry to learn more about security. The show floor was buzzing and sessions were packed with eager listeners soaking up all they could about a variety of topics like GDPR, blockchain and cloud security. And the hardest part about being a vendor at RSA? Standing above the noise and cutting through the fear, uncertainty and doubt (FUD) messaging. Listen to our latest SnapSecChat video series with George to hear more about his RSA musings, and stay tuned for another video coming to a website near you soon! Don't forget to follow George on Twitter at @GeorgeGerchow, and use the hashtag #SnapSecChat to join the security conversation!

Containers are portable assets that let you design and deploy with little overhead from your development team. They take your monolithic code base and turn it into several lightweight modules that you can more easily manage and interconnect without worry of one small module taking out your entire application. This gives you more granular control of your code, but it also means that you have several moving parts as part of your platform.

Every year, the proverbial cyber gods anticipate a major data breach during tax day, and this year, they weren’t completely wrong. While we didn’t have a cyber meltdown, we did experience a major technology failure. Many U.S. taxpayers may already know what I am talking about here but for those outside of the country, let me fill you in. The U.S. Internal Revenue Service (IRS) website crashed on April 17 — the infamous tax filing day. And much like death, or rush hour traffic, no one can escape doing his or her taxes. And thanks to the good graces of the IRS, the deadline was extended a day to April 18, so that taxes could be filed by everyone as required by U.S. laws. Now you might say “ah shucks, it’s just a day of downtime. How bad can it be?” In short, bad, and I’m going to crunch some numbers to show you just how bad because here at Sumo Logic, we care a lot about customer experience and this outage was definitely low on the customer experience scale. The below will help you calculate the financial impact of the total downtime and how the delays affected the U.S. government — and more importantly, the money the government collects from all of us. To calculate the cost of the one day downtime of the IRS app, we did some digging in terms of tax filing statistics. And here’s what we found: On average over 20 million taxpayers submit their taxes in the last week, according to FiveThirtyEight. Half of these folks — 50 percent, ~10M (million) people — submitted on the last day, and therefore were affected by the site outage. How did we arrive at this number? Well, we made a conservative assumption that the folks who waited until the last week were the classic procrastinators (you know who you are!). And these procrastinators generally wait until the last day (maybe even the last hour) to file their taxes, which was also backed up by the IRS and FiveThirtyEight data points. We also make a practical assumption that most last-minute tax filers are “payees,” meaning you are waiting for the last day because you are paying money to the government. After all, if you are getting money back, there’s more incentive to file early and cash in that check! You with me so far? Now, in order to determine the amount of money the IRS collects on the last day, we need to know what the average filer pays to the IRS. This was a tricky question to answer. The IRS assumes that most folks get a refund of $2900, but does not specify the average filer payment amount. Since there exists no standard amount, we modeled a few payments ($3K, $10K, $30K) to calculate the amount that got delayed because of the last minute outage. Number of filers who pay Avg. payee amount IRS revenue ($’s) on last filing day 10M $3K $30B 10M $10K $100B 10M $30K $300B So the government delayed getting some big money, but they do eventually get it all (didn’t we say that taxes and death are inevitable?). It’s important to note here that a lot of taxes are collected via W-2 forms, so filing does not mean the payee will actually pay, and in some instances, there will be refunds granted. So with that in mind, let’s now calculate the cost of the one day delay. To do this, we use the 1.6 percent* treasury yield on the money and we calculate the actual cost of downtime. Number of filers who pay Avg. payee amount IRS revenue ($’s) on last filing day Lost revenue for one day delay 10M $3K $30B $1.3M 10M $10K $100B $4.3M 10M $30K $300B $13.1M What you see is that even one day’s cost of downtime for the U.S. government is measured in millions of dollars. And for a government that is trillions of dollars in debt (and at least in theory focused on cost/debt reduction), these numbers add up quickly. So here’s a quick PSA for the U.S. government and for the IRS in particular: Your software application matters. And the customer experience matters. Update your systems, implement the proper controls, and get your act together so that we can rest assured our money is in good hands. If you continue to run your infrastructure on legacy systems, you’ll never be able to scale, stay secure or deliver the ultimate customer experience. With the pace of technological innovation, cloud is the future and you need visibility across the full application stack. But let us not forget the second moral of this story, said best by Abraham Lincoln: “You cannot escape the responsibility of tomorrow by evading it today.” And if you want to see customer experience done right, contact Sumo Logic for more questions on how to build, secure and run your modern applications in the cloud!

Cloud computing has been around for so long now that cloud is basically a household word. Yet, despite how widespread cloud computing has become, continued adoption of the cloud is now being challenged by new types of use cases that people and companies are developing for cloud environments.

RSA Conference 2018 is in full swing and our very own Sumo Logic CSO, George Gerchow, will be live at the show all week talking to customers and partners for our CSO Corner video series. Yesterday, George caught up with one of our partners, Ernesto Tey, VP of global ISV alliances at Okta, outside of the Sumo Logic booth to discuss the importance of using a single sign on solution to minimize risk with cloud workloads, improve the end-user experience, and onboard and de-provision users. Check out the video to learn how nearly 400 joint Sumo Logic/Okta customers are taking advantage of multi-factor authentication (MFA) and feeding Okta identity data into the Sumo Logic platform to track, measure and identify usage patterns and potential malicious activity within their modern application and cloud infrastructure. Don’t forget to stop by Sumo Logic Booth 4516 in North Hall, and swing by Okta’s Booth 1421 in the South Hall for more information about our integration.

It’s day three of RSA Conference 2018, and our very own Sumo Logic CSO, George Gerchow, will be live at the show all week talking to customers and partners for our CSO Corner video series. Today, George spoke with Jackie Wadhwa, Sr. global manager, customer success at Neustar about one of the biggest buzzwords at this year’s show: General Data Protection Regulation (GDPR). As a valued customer, Sumo Logic leverages Neustar’s 20+ years of experience gathering and delivering IP intelligence insight to provide accurate geolocation services to our customers. This allows them to confidently detect suspicious logins, maintain regulator compliance such as GDPR, and analyze end-use behavior. Check out the video to hear more from Jackie on why the Neustar IP intelligence platform is naturally GDPR compliant, and why that’s a game changer for a highly secure, modern day application company born and bred in AWS like Sumo Logic. Don’t forget to stop by Sumo Logic Booth 4516 in North Hall, and swing by Neustar Booth 3221 also in the North Hall for more information!

New World of Modern Apps and Cloud Create Complex Security Challenges As the transition to the cloud and modern applications accelerates, the traditional security operations center (SOC) functions of threat correlation and investigation are under enormous pressure to adapt. These functions have always struggled with alert overload, poor signal to noise ratio in detection, complex and lengthy workflows, and acute labor churn; however, cloud and modern applications add new challenges to integrate previously siloed data and process while coping with much larger threat surface areas. To overcome these challenges, security must continuously collaborate with the rest of IT to acquire and understand essential context. In addition, cloud and application-level insight must be integrated with traditional infrastructure monitoring, and investigation workflows must accelerate at many times the current speed in order to keep pace with the exploding threat landscape. In the past 2 months we’ve formally surveyed hundreds of companies about their challenges with security for modernizing IT environments in the 2018 Global Security Trends in the Cloud report, conducted by Dimensional Research in March 2018 and sponsored by Sumo Logic. The survey included a total of 316 qualified independent sources of IT security professionals across the U.S. and Europe, the Middle East and Africa (EMEA). In addition, we’ve interviewed a broad cross-section of both current and potential future Sumo Logic customers. According to the survey results, a strong majority of respondents called out the need for a fundamentally new approach for threat assessment and investigation in the cloud, and even the laggard voices conceded these are “if not when” transitions that will redraw boundaries in traditional security tools and process. In the Customer Trenches: Why Security and IT Must Collaborate Eighty-seven percent of surveyed security pros observed that as they transition to the cloud, there is a corresponding increase in the need for security and IT operations to work together during threat detection and investigation. Customer interviews gave color to this strong majority with many use cases cited. For instance, one SaaS company security team needed end customer billing history to determine the time budget and priority for conclusion/case queuing. Another online business process firm needed close collaboration with the cloud ops teams to identify if slow application access was a security problem or not. A third company needed IT help for deeper behavioral insight from identity and access management (IAM) systems. In all of these examples the heavy dose of cloud and modern applications made it nearly impossible for the already overburdened security team to resolve the issues independently and in a timely manner. They required real-time assistance in getting data and interpreting it from a variety of teams outside the SOC. These examples are just a few of the complex workflows which can no longer be solved by siloed tools and processes that are holding organizations back from fully securing their modern IT environments. These challenges surface in the survey data as well, with 50 percent of respondents specifically looking for new tools to improve cross-team workflows for threat resolution. This group — as you would expect — had plenty of overlap with the over 50 percent of respondents who observed that on-premises security tools and traditional security information and event management (SIEM) solutions can’t effectively assimilate cloud data and threats. Unified Visibility is Key: Integrating Cloud and Application Insight Eighty-two percent of those surveyed observed that as their cloud adoption increases there is a corresponding increase in the need to investigate threats at both the application and infrastructure layers. A clear pattern in this area was best summarized by one SOC manager, who said: “I feel like 90 percent of my exposure is at the application layer but my current defense provides only 10 percent of the insight I need at that layer.” Attackers are moving up the stack as infrastructure defenses solidify for cloud environments, and the attack surface is expanding rapidly with modular software (e.g. microservices) and more externally facing customer services. “Ninety percent of my exposure is at the application layer but my current defense provides only 10 percent of the insight I need” In the survey, 63 percent of security pros reported broader technical expertise is required when trying to understand threats in the cloud. An industry veteran who spent the past 3 years consulting on incorporating cloud into SOCs noted a “three strikes you’re out” pattern for SOC teams in which they could not get cloud application data, could not understand the context in the data when they did get it, and even if they understood it could not figure out how to apply the data to their existing correlation workflows. One CISO described the process like “blind men feeling an elephant,” a metaphor with a long history describing situations in which partial understanding leads to wild divergence of opinion. Customers interviews provided several examples of this dynamic. One incident pesponse veteran described painstaking work connecting the dots from vulnerabilities identified in DevOps code scans to correlation rules to detect cross-site scripting, a workflow invisible to traditional infrastructure-focused SOCs. Another enterprise with customer facing SaaS offerings described a very complex manual mapping from each application microservice to possible IOCs, a process the traditional tools could only complete in disjointed fragments. Many reported the need to assess user activity involving applications in ways standard behavior analytics tools could not. More broadly these cloud and application blind spots create obvious holes in the security defense layer, such as missing context, lost trials, unidentified lateral movement and unsolvable cases (e.g. cross-site scripting) to name a few. Diversity of log/API formats and other challenges make moving up the stack a non-trivial integration, but these obstacles must be overcome for the defense to adapt to modern IT. New Approach Needed to Break Down Existing Silos With all of these challenges in the specific areas of threat correlation and investigation, it’s no surprise that more generally an aggregate of 93 percent of survey respondents think current security tools are ineffective for the cloud. Two-thirds of those surveyed are looking to consolidate around tools able to plug the holes. A full third say some traditional categories such as the SIEM need to be completely rethought for the cloud. At Sumo Logic we’ve lived the imperative to bridge across the traditional silos of IT vs. security, application vs. infrastructure, and cloud vs. on-premises to deliver an integrated cloud analytics platform. We’re applying that hard won insight into new data sources, ecosystems and application architectures to deliver a cloud security analytics solution that meets the demands of modern IT. Stop by the Sumo Logic booth (4516 in North Hall) this week at RSA for a demo of our new cloud security analytics platform features, including privacy and GDPR-focused dashboards, intelligent investigation workflow and enhanced threat intelligence. To read the full survey, check out the report landing page, or download the infographic for a high-level overview of the key findings.

Live at RSA Conference 2018, theCube host Jeff Frick sat down with our very own Dave Frampton, VP of security solutions at Sumo Logic, to discuss the increasing challenges organizations face around maintaining their security posture as they move from on-premises and adopt modern applications in the cloud. Dave shares the Sumo Logic vision for creating a cloud security analytics solution as well as best practices for identifying indicators of compromise across complex datasets. Watch the video for more insights and tips on why breaking down silos to converge development, security and operations (DevSecOps) teams will help us better orchestrate our security defenses in today's cloud-enabled world. Also, check out Dave's latest blog that analyzes recent survey findings to uncover common security challenges and trends facing security and operations teams as they migrate their infrastructure to cloud environments.

The Microsoft Cloud, also known as Microsoft Azure, is a comprehensive collection of cloud services available for developers and IT professionals to deploy and manage applications in data centers around the globe. Managing applications and resources can be challenging, especially when the ecosystem involves many different types of resources, and perhaps multiple instances of each. Being able to view logs from those resources and perform log analysis is critical to effective management of your environment hosted in the Microsoft Cloud. In this article, we’re going to investigate what logging services are available within the Microsoft Cloud environment, and then what tools are available to assist you in analyzing those logs. What Types of Logs are Available? The Microsoft Cloud Infrastructure supports different logs depending on the types of resources you are deploying. Let’s look at the logs that are gathered within the ecosystem and then investigate each in more depth. Activity Logs Diagnostic Logs Application logs are also gathered within the Microsoft Cloud. However, these are limited to compute resources and are dependent on the technology used within the resource, and application and services which are deployed with that technology. Activity Logs All resources report their activity within the Microsoft Cloud ecosystem in the form of Activity Logs. These logs are generated as a result of some different categories of events. Administrative – Creation, deletion and updating of the resource. Alerts – Conditions which may be cause for concern, such as elevated processing or memory usage. Autoscaling – When the number of resources is adjusted due to autoscale settings. Service Health – Related to the health of the environment in which the resource is hosted. These logs contain information related to events occurring external to the resource. Diagnostic Logs Complementary to the activity logs are the diagnostic logs. Diagnostic logs provide a detailed view into the operations of the resource itself. Some examples of actions which would be included in these logs are: Accessing a secret vault for a key Security group rule invocation Diagnostic logs are invaluable in troubleshooting problems within the resource and gaining additional insight into the interactions with external resources from within the resource being monitored. This information is also valuable in determining the overall function and performance of the resource. Providing this data to an analysis tool can offer important insights which we’ll discuss more in the next section. Moving Beyond a Single Resource Log viewing tools and included complex search filters are available from within the Microsoft Cloud console. However, these are only useful if you are interested in learning more about the current state of a specific instance. And while there are times when this level of log analysis is valuable and appropriate, sometimes it can’t accomplish the task. If you find yourself managing a vast ecosystem consisting of multiple applications and supporting resources, you will need something more powerful. Log data from the Microsoft Cloud is available for access through a Command Line Interface (CLI), REST API and PowerShell Cmdlet. The real power in the logs lies in being able to analyze them to determine trends, identify anomalies and automate monitoring so that engineers can focus on developing additional functionality, improving performance and increasing efficiencies. There are some companies which have developed tools for aggregating and analyzing logs from the Microsoft Cloud, including Sumo Logic. You can learn more about the value which Sumo Logic can provide from your log data by visiting their Microsoft Azure Management page. I’d like to touch on some of the benefits here in conclusion. Centralized aggregation of all your log data, both from the Microsoft Cloud and from other environments, makes it easier to gain a holistic view of your resources. In addition to making this easier for employees to find the information they need quickly, it also enhances your ability to ensure adherence to best practices and maintain compliance with industry and regulatory standards. Use of the Sumo Logic platform also allows you to leverage their tested and proven algorithms for anomaly detection, and allows you to segregate your data by source, user-driven events, and many other categories to gain better insight into which customers are using your services, and how they are using them.

It’s day two of RSA Conference 2018, and our very own Sumo Logic CSO, George Gerchow, will be live at the show all week talking to customers and partners for our CSO Corner video series. This afternoon, George spoke with Ed Smith, principal product manager at CloudPassage to discuss evolving trends around DevSecOps and how the migration to cloud is impacting the way we approach security. Through our integration with CloudPassage, customers can correlate CloudPassage Halo data with other data streams in the Sumo Logic platform to give IT Ops, DevOps, and security teams a comprehensive, real-time view of their security and compliance postures while rapidly detecting and containing attacks. Check out the video to hear more about what CloudPassage and Sumo Logic are up to at RSA and how our trusted partnership brings customers a security analytics and incident response solution for the new world of modern applications and cloud. Don’t forget to stop by Sumo Logic Booth 4516 in North Hall — CloudPassage will be there, too!

Sumo Logic is live at RSA Conference 2018 all week and last night we celebrated a successful first day of the show with our closest friends, colleagues, customers and partners at our Guide to the GDPR Galaxy pre-RSA party at Minna Gallery. If you weren’t able to attend and missed out on the GDPR Flash Q&A with Sumo Logic CSO, George Gerchow and Data Protection Officer (DPO), Jen Brown, moderated by Ben Newton, you can watch the full recorded session here, or check it out on our Sumo Logic Facebook page. Special thanks to everyone who came out to our party last night, and for those that couldn’t join us, stop by booth 4516 in the North Hall to meet our experts and learn more about Sumo Logic’s GDPR journey, and how we are enabling customers to achieve GDPR-readiness, too. If you want to learn more about all of the foundational work Jen is doing not just for Sumo Logic, but for our customers, around building a privacy-and-security-by-design program, and educating her peers on how best to implement a privacy program that addresses the requirements of not just GDPR, but all top compliance regulations from HIPAA to ISO2700 to PCI, follow her on Twitter, and be sure to read her ongoing GDPR series in Dark Reading.

How have monitoring tools evolved over the years? That’s a big question, and one that few people are capable of answering based on personal experience. Monitoring software has been around in one form or another since the early years of computing, and few people who are active in the profession today were working then.

https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

Configuring health checks for the AWS Elastic Load Balancer (ELB) is an important step to ensure that your cloud applications run smoothly. In this article, we’re going to look specifically at how to configure the health checks on an ELB. In addition, we’re going to look at what factors you should consider when creating and configuring efficient and useful health checks for your services. Our goal is to ensure we have an environment which operates efficiently and minimizes downtime and degraded performance due to increased load.

If you use AWS, you have two load-balancing options: ELB and ALB. And while choice is always a good thing, the ELB vs. ALB debate can be intimidating. Which load balancer is right for your application? Is ALB better than ELB, or vice versa? This article explores these questions.

Organizations are increasingly moving workloads, applications and infrastructure to Amazon Web Services (AWS) to leverage the cost, scale and agility the cloud has to offer. Moving to the cloud has become an imperative in order to keep pace with their competitors who are already taking similar steps. However, these same organizations are all finding that their legacy security and networking tools have many gaps in their abilities to provide the necessary visibility and control they need to manage regulatory and risk management requirements in these modern environments.

Microservices have undoubtedly become mainstream as the market continues to shift toward building agile and scalable modern architectures. Transitioning from traditional monolithic applications architectures to microservices can not only positively impact the productivity and collaboration between IT departments such as development, security and operations, but also deliver tangible business benefits and ROI across the entire organization. So what’s the secret to successful implementation? Our very own Christian Beegden, CTO at Sumo Logic and Stefan Zier, chief architect at Sumo Logic, sat down with Jake Lumetta, the CEO of ButterCMS to discuss the history and evolution of microservices adoption within Sumo Logic for the Microservices for Startups ebook. Interested in reading more about what Christian and Stefan had to say about the strategy behind and implementation of moving from a monolith to microservices architecture across Sumo Logic? Read the in-depth interview for more. Rapid Tech Trends are Changing Everything

AWS Config Rules allow the user to configure and respond to configuration changes in an automated manner.

For several years now, the tool of choice for collecting performance metrics in a Linux environment has been Graphite. While it is true that other monitoring tools, such as Grafana, have gained traction in the last several years, Graphite remains the go tool monitoring tool for countless organizations. But what about those organizations that run Windows, or a mixture of Windows and Linux? Because Graphite was designed for Linux, it is easy to assume that you will need a native Win32 tool for monitoring Windows systems. After all, the Windows operating system contains a built-in performance monitor, and there are countless supplementary performance monitoring tools available, such as Microsoft System Center. While using Graphite to monitor Linux systems and a different tool to monitor Windows is certainly an option, it probably isn’t the best option. After all, using two separate monitoring tools increases cost, as well as making life a little more difficult for the administrative staff. Fortunately, there is a way to use Graphite to monitor Windows systems. Bringing Graphite to Windows As you have probably already figured out, Graphite does not natively support the monitoring of Windows systems. However, you can use a tool from GitHub to bridge the gap between Windows and Graphite. In order to understand how Graphite monitoring for Windows works, you need to know a little bit about the Graphite architecture. Graphite uses a listener to listen for inbound monitoring data, which is then written to a database called Whisper. Graphite is designed to work with two different types of metrics—host metrics and application metrics. Host metrics (or server metrics) are compiled through a component called Collectd. Application metrics, on the other hand, are compiled through something called StatsD. In the Linux world, the use of Collectd and StatsD means that there is a very clear separation between host and application metrics. In the case of Windows however, Graphite monitoring is achieved through a tool called PerfTap. PerfTap does not as cleanly differentiate between host and application monitoring. Instead, the tool is designed to be compatible with StatsD listeners. Although StatsD is normally used for application monitoring, PerfTap can be used to monitor Windows operating system-level performance data, even in the absence of Collectd. The easy way of thinking of this is that StatsD is basically treating Windows as an application. As is the case with the native Windows Performance Monitor, PerfTap is based around the use of counters. These counters are grouped into five different categories, including: System Counters – Used for monitoring hardware components such as memory and CPU Dot Net Counters – Performance counters related to the .NET framework ASP Net Counters – Counters that can be used to track requests, sessions, worker processes, and errors for ASP.NET SQL Server Counters – Most of these counters are directly related to various aspects of Microsoft SQL Server, but there is a degree of overlap with the System Counters, as they relate to SQL Server. Web Service Counters – These counters are related to the native web services (IIS), and allow the monitoring of ISAPI extension requests, current connections, total method requests, and more. PerfTap allows monitoring to be enabled through the use of a relatively simple XML file. This XML file performs four main tasks. First, it sets the sampling interval. The second task performed by the XML file is to provide the location of the counter definition file. The XML file’s third task is to list the actual counters that need to be monitored. And finally, the XML file provides connectivity information to the Graphite server by listing the hostname, port number, prefix key, and format. You can find an example of the XML file here. Graphite and Sumo Logic Although Graphite can be a handy tool for analyzing performance metrics, Graphite unfortunately has trouble maintaining its efficiency as the organization’s operations scale. One possible solution to this problem is to bring your Graphite metrics into the Sumo Logic service. Sumo Logic provides a free video of a webinar in which they demonstrate their platform’s ability to natively ingest, index, and analyze Graphite data. You can find the video at: Bring your Graphite-compatible Metrics into Sumo Logic. Conclusion Although Graphite does not natively support the monitoring of Windows systems, you can use a third-party utility to send Windows monitoring data to a Graphite server. Of course, Graphite is known to have difficulty with monitoring larger environments, so adding Windows monitoring data to your existing Graphite deployment could complicate the management of monitoring data. One way of overcoming these scalability challenges is to bring your Graphite monitoring data into Sumo Logic for analysis.

Enterprises today are adopting modern IT architectures and moving to the cloud. And for good reason. These new architectures offer agility, faster application code delivery, high scalable and elastic solutions. And to manage these applications — from an operational and security standpoint — enterprises are turning to machine data analytics solutions like Sumo Logic.

The General Data Protection Regulation (GDPR) will go into effect on May 25, 2018 and this has sent many people into a panic. If you’re one of those people, and you’re wondering where you should start, have no fear, we’re here to help! In a recent Dark Reading article titled A Data Protection Officer's Guide to the GDPR Galaxy, Jen Brown, Data Protection Officer (DPO) at Sumo Logic explores the GDPR universe and shares five tips to help you study up on the regulation and sort through which of the 99 articles and 173 recitals that comprise GDPR are the most relevant to your organization. Read the full article

Amazon Web Services (AWS) first introduced Elastic Load Balancing (ELB) in May of 2009. The objective of the service was to provide a method for users to distribute traffic across multiple Amazon EC2 instances. Today, AWS offers their ELB service in three distinct flavors.

AWS security best practices are crucial in an age when AWS dominates the cloud computing market. Although moving workloads to the cloud can make them easier to deploy and manage, you’ll shoot yourself in the foot if you don’t secure cloud workloads well. Toward that end, this article outlines common AWS configuration mistakes that could lead to security vulnerabilities, then discusses strategies for addressing them. IAM Access The biggest threat that any AWS customer will face is user access control, which in AWS-speak is known as Identity and Access Management‎ (IAM). When you sign up for a brand-new AWS account, you are taken through steps that will enable you to grant privileged access to people in your company. When the wrong access control is given to a person that really doesn’t require it, things can go terribly downhill. This is what happened with GitLab, when their production database was partially deleted by mistake! Mitigation Fortunately, IAM access threats can be controlled without too much effort. One of the best ways to go about improving IAM security is to make sure you are educated about how AWS IAM works and how you can take advantage of it.When creating new identities and access policies for your company, grant the minimal set of privileges that everyone needs. Make sure you get the policies approved by your peers and let them reason out why one would need a particular level of access to your AWS account. And when absolutely needed, provide temporary access to get the job done.Granting access to someone does not just stop with the IAM access control module. You can take advantage of the VPC methods that allow administrators to create isolated networks that connect to only some of your instances. This way, you can have staging, testing and production instances. [Read More: Threat Hunting] Loose Security Group Policies Administrators sometimes create loose security group policies that expose loopholes to attackers. They do this because group policies are simpler than setting granular permissions on a per-user basis.Unfortunately, anyone with basic knowledge of AWS security policies can easily take advantage of permissive group policy settings to exploit AWS resources. They leave your AWS-hosted workloads at risk of being exploited by bots (which account for about a third of the visitors to websites, according to web security company Imperva). These bots are unmanned scripts that run on the Internet looking for basic security flaws, and misconfigured security groups on AWS servers that leave unwanted ports open are something they look for. Mitigation The easiest way to mitigate this issue is to have all the ports closed at the beginning of your account setup. One method of doing this is to make sure you allow only your IP address to connect to your servers. You can do this while setting up your security groups for your instances, to allow traffic only to your specific IP address rather than to have it open like: 0.0.0.0/0. Above all, making sure you name your security group when working in teams is always a good practice. Names that are confusing for teams to understand is also a risk. It’s also a good idea to create individual security groups for your instances. This allows you to handle all your instances separately during a threat. Separate security groups allow you to open or close ports for each machine, without having to depend on other machines’ policies. Amazon’s documentation on Security Groups can help you get tighter on your security measures. Protecting Your S3 Data One of the biggest data leaks from Verizon happened not because of a bunch of hackers trying to break their system, but from a simple misconfiguration in their AWS S3 storage bucket that contained a policy that allows anyone to read information from the bucket. This misconfiguration affected anywhere between six million and 14 million Verizon customers. This is a disaster for any business.Accidental S3 data exposure is not the only risk. A report released by Detectify identifies a vulnerability in AWS servers that allows hackers to identify the name of the S3 buckets. Using this information, an attacker can start talking to Amazon’s API. Done correctly, attackers can then read, write and update an S3 bucket without the bucket owner ever noticing. Mitigation According to Amazon, this is not actually an S3 bug. It’s simply a side effect of misconfiguring S3 access policies. This means that as long as you educate yourself about S3 configuration, and avoid careless exposure of S3 data to the public, you can avoid the S3 security risks described above. Conclusion Given AWS’s considerable market share, there is a good chance that you will deploy workloads on AWS in the future, if you do not already. The configuration mistakes described above that can lead to AWS security issues are easy to make. Fortunately, they’re also easy to avoid, as long as you educate yourself. None of these security vulnerabilities involve sophisticated attacks; they center on basic AWS configuration risks, which can be avoided by following best practices for ensuring that AWS data and access controls are secured.

Running a modern application in the cloud is a complex task which requires clear, real-time visibility across your entire application stack and infrastructure. With Sumo Logic you can fix problems before they negatively affect your customers' experience and make sure your application is running at peak performance.

Before Azure and AWS, Microsoft development teams would need a server on their local network that would manage change control and commits. Once a week, a chosen developer would compile and deploy the application to a production server. Now developers have the option of creating cloud applications in Visual Studio and connecting their projects directly to an Azure cloud server. For a Windows developer, Azure was the choice I made to develop my own software projects and manage them in the cloud due to the easy integration between my development desktop and Azure cloud instances. Azure has similar services as other cloud providers, but it has some advantages for Microsoft developers that need to deploy applications for the enterprise. The biggest advantage for an enterprise is that it reduces the amount of on-site resources needed to support a developer team. For instance, the typical Microsoft-based enterprise has a team of developers, a staging server, a development server, QA resources, Jira, and some kind of ticketing system (just to name a few). With Azure, the team can set up these resources without the real estate or the hardware on-site. It also has integrated IaaS, so you can create a seamless bond between your internal network and the cloud infrastructure. In other words, your users will never know if they are working on a local server or running applications on Azure. The Dashboard For anyone who has managed Windows servers, the dashboard (Azure calls it your portal) is intuitive. The difficult part of starting an Azure account is understanding all of the options you see in the dashboard. For developers used to local environments where you need to provision multiple resources for one application, it's important to understand that everything you need to build an application is at your fingertips. You no longer need to go out and purchase different services such as SSL and database software and install it. You just click the service that you want and click "Create" in the Azure portal and Microsoft builds it into your service agreement. If you pay as you go, then you only pay for the resources that you use. You can build APIs, services, start virtual servers, and even host WordPress sites from your portal. In the image above, I've blacked out the names of my resources for security reasons, but you can see that I have a web application, an SSL certificate, two databases (one MySQL and one MSSQL), email services and a vault (for the SSL cert) in Azure. I have an individual plan and don't use many resources from their servers, so I pay about $100 a month for a small WordPress site. When I had two web applications running, I paid about $150 a month. Integration with Visual Studio The main reason I use Azure is for it's easy integration with my Visual Studio projects. For Windows projects, integration between Visual Studio and Azure is as simple as creating a web application and copying the connection information into your projects. Once you've created the connection in Visual Studio, you can use Azure for change control and promote code directly from your development computer. For a simple web application, Microsoft offers a web app and a web app with SQL. This should be self-explanatory. If you just want to build a web application and don't need a database, then you choose the first option. The second one will set up a web app and a SQL Server. You aren't limited to a Windows environment either. When you create a web app, Azure asks if you want to use Windows or Linux. When you create your application, it sits on a subdomain named <your_app_name>.azurewebsites.net. This will be important when you set up your TLD. This feature is the downside of using Azure over traditional hosting. When you set up your TLD, you set up a CNAME and configure a custom domain in your Azure settings. When search engines crawl your site, sometimes they index this subdomain instead of only the TLD. This makes it difficult to work with when you use applications such as WordPress. When you install WordPress, you install it on the subdomain, so WordPress has the subdomain in all of its settings. This causes errors when you promote the site to your TLD, and you must do a global database search and replace to remove the it from your settings. I found this was one con to using Azure for my websites. After you create the web app, you're shown basic information to get started should you just want to upload files to the new location using FTP. The "Get publish profile" provides you with a downloadable text file that you use to connect Visual Studio. You can also connect it directly to your own GitHub repository. As a matter of fact, Microsoft will generate pre-defined settings for several repositories. After you download one of these files, you import the settings directly into your profile and you're automatically connected. When I work in Visual Studio with an Azure connection, I check out files, edit them and check them back in as I code. It feels exactly like the old-school Team Foundation Server environment except I don't have to buy the expensive equipment and licenses to host an internal Microsoft Windows server. Easy VM Creation for Testing Third-Party Software Occasionally, I get a customer that has an application that they want me to test. It's usually in beta or it's their recent stable release but it's something that I wouldn't normally use on my machine. It's dangerous for me to install third-party software on my work machine. Should my customers' software crash my computer, I lose important data. Not only could I lose data, but I don't know what type of software I'm installing on my important developer machine connected to my home network. I don't want anything to have the ability to search my home network and storage. With an Azure VM, I not only keep the software off of my local machine, but it's shielded from my local network too. I could install a VM on my local desktop, but I like to keep this machine clean from anything other than what I need to develop and code. I use Azure VMs to install the software, and then I can destroy the VM when I'm done with it. What's great about Azure is that they have several pre-installed operating systems and frameworks to choose from. Here are just a few: Some other platforms not shown above but can be installed on an Azure VM: Citrix XenDesktop Kali Linux Cisco CSR SQL Server cluster Kiteworks Suse Linux Jira Magento WordPress When I'm working with client software, I use the default Windows Server installation, which is 2016. Microsoft adds the latest operating system to its list as they are released. It's an Azure advantage over working with traditional VPS service. With traditional service, you must ask the host to upgrade your operating system and change your service. With Azure, you spin up a new VM with any operating system you choose. I then use the VM to install the software, work with it until I'm finished a project, and then destroy the VM when I'm done. Because I only use the VM for a short time and don't connect it to any resource-intensive applications, it only costs me a few extra dollars a month. But I keep third-party apps "sandboxed" from my own network without using local resources. Working with WordPress on Azure Occasionally organizations have a separate WordPress site for content even with a team of Windows developers. It's easier for marketing to publish content with WordPress. Instead of using a separate shared host, Azure has the option to create an app with WordPress pre-installed. Most people think running WordPress on a Windows server is senseless, but you get some kind of security from standard hacks that look for .htaccess or Apache vulnerabilities, and you can keep your web services all in one place. It only costs a few dollars to host the WordPress site, but you pay for traffic that hits the service so it might be much more if you have a high-traffic blog. Azure has its Insights services as well, so you can monitor WordPress unlike traditional shared or VPS services where you need a third-party application. You also get monitoring and all the available infrastructure such as load balancing, SSL, and monitoring should you need it with the WordPress site. These aren't available with simple shared or VPS hosting. While running WordPress on Windows seems counterintuitive, running it in Azure is more beneficial for the enterprise that needs to keep detailed statistics on site usage and security from script kiddies that attack any public WordPress site with poor Linux management. Is Azure Better than AWS? Most people will tell you that AWS is a better choice, but I preferred Azure's portal. I found spinning up services was more intuitive, but what sold me was the integration into Visual Studio. If you're in an enterprise, the one advantage of Azure is that you can integrate Active Directory Services so that your network expands into Azure's cloud IaaS. This would be much better than building a separate portal that you must control with individual security settings. This eliminates the possibility of accidentally exposing data from incorrect security settings. If you decide to try Azure, they give you the first 30 days free. AWS gives you 12 months, so users have longer to figure out settings. I found Azure beneficial and kept paying for a low-traffic, low-resource account until I could figure out if I wanted to permanently use it. I've had an account for two years now and don't plan to ever switch.

When using and troubleshooting IIS web server performance, logs are an essential resource. Let’s look at the top metrics on IIS logs and how to get the most out of logging when working with IIS servers.

Serverless computing has enabled developers to develop their next-generation products from ideation to production, without waiting for, or worrying about, infrastructure. AWS Lambda and AWS CloudWatch Logs, both services provided by AWS, are one of the important components in serverless architectures.

Worries over monitoring and managing the the zettabytes of data the Internet of Things will produce this year and beyond loom like dark mountains on the minds of IT decision makers. But equipped with the right tools and partnerships, those mountains can be mined for business intelligence that drives golden results.

If you are like most software developers, your code often doesn’t work the first time you write and test it—and a good portion of your job is dedicated to troubleshooting errors caused by the configuration of your application server or the code you have written. This type of troubleshooting typically involves analyzing log files.

Early on, one of the most common reasons cited for not taking advantage of public cloud computing was security concerns. In general, that is no longer true. Today, a properly managed cloud environment is just as secure as any other type of well-maintained environment.

Chinese dragon symbol. Threat correlation and prioritization (what do I pay attention to in an avalanche of highlighted threats?) and threat investigation (how do I decide what happened and what to do quickly?) are extremely challenging core functions of the security defense, resulting in many cases with less than 10% of high priority threats fully investigated. The accelerating migration to cloud and modern application deployment are making these already difficult workflows untenable in traditional models, leading to questions such as how to gather and correlate all of the new sources of data at cloud scale? How to understand and triangulate new dynamic data from many layers in the stack? How to react with the pace demanded by new models of DevSecOps deployment? And how to collaborate to connect the dots across evolving boundaries and silos? Last week a veteran of many cloud migration security projects I know described many SOCs as“groping in the dark” with these challenges and looking for a new approach despite all of the vendor claims mapped to their pains. The usual crowd of incremental enhancements (e.g. bringing cloud data into the traditional SIEM, automating manual workflows, layering more tools for specialized analytics, leveraging wisdom of crowds, etc.) leaves three dragons roaming the countryside which need to be slain for security to keep pace with the unstoppable accelerating migration to the cloud. Dragon #1 – Siloed Security and IT Ops Investigation Workflows A basic dilemma in security for the cloud is that often the knowledge needed to pursue an investigation to conclusion is split between two groups. Security analysts understand the process of investigation and the broad context, but often only IT ops understands the essential specific context – application behavior and customer content, for example – needed to interpret and hypothesize at many steps in a security investigation. A frequent comment bucket item goes something like, “The SOC understands the infrastructure, but they don’t know how to interpret app logs or new data sources like container orchestration.” This gap in understanding makes real time collaboration essential to prevent exploding backlogs, partial investigations, and bias toward more solvable on-prem alerts. Aside from needing to understand unfamiliar, new, and rapidly changing data sources in a single security investigation, cloud deployments generate more frequent “Dual Ticket” cases in which it is unknown whether a security issue or an IT issue is the root cause (ex: my customer is complaining they can’t access our app – network congestion? Cloud provider outage? Server CPU overload? DDoS attack? Malware? Customer issue?) It isn’t just that two separate investigations take more time and resources to complete and integrate, often, in cloud cases, neither side can reach conclusion without the other. Working from common data isn’t enough – analytics and workflow need to be common as well to enable the seamless collaboration required. In addition, modern cloud deployments often employ DevSecOps models in which the pace of application update, rollout, and change is measured in days or hours as opposed to months or quarters. One security threat investigation implication is that the processing of the threat resolution backlog must align so that current resources can be applied to current environments without being mired in “old” cases or chasing continuous flux in the data. This is challenge enough, but having to manage this triage across two separate backlogs in both IT and security with the usual integration taxes means operating on the scale of hours and days is extremely challenging. While separate siloes for IT ops and security investigations were feasible and logical in on-prem classic IT, modern cloud deployments and application architecture demand a seamless back and forth workflow where at each step the skills and perspective from both IT and security are needed to properly interpret the results of queries, evidence uncovered, or unfamiliar data. Asking both sides to completely subsume the knowledge of the other is unrealistic in the short term – a much better solution is to converge their workflows so they can collaborate in real time. Dragon #2 – Traditional Security Bias on Infrastructure vs. Application Insight Traditional SIEMs have long been exhorted to look up the stack to the application layer, and in several instances new product areas have sprung up when they have not. In the cloud world this application layer “nice to have” becomes a “must have.” Clould providers have taken on some of the infrastructure defense previously done by individual companies, creating harder targets that cause attackers to seek softer targets. At the same time, much of the traditional infrastructure defense from the on-prem world has not yet been replicated in the cloud, so often application layer assessment is the only investigation method available. In addition to the defensive need to incorporate the application layer, there clearly is additional insight at that layer which is unknown at the infrastructure layer (e.g. customer context, behavioral analytics, etc.). This is particularly true when it is unclear whether a security or an IT problem exists. Many point systems specialize in extracting actionable insight from this layer, but the holistic correlation and investigation of threat is more difficult, in part because of wide variations in APIs, log formats, and nomenclature. Looking forward, modern application deployment in the cloud also increases the surface area for investigation and threat assessment. For example, chained microservices create many possible transitions in variables important to investigators. For all of these reasons, adding insight from the application layer is necessary and good for cloud deployments, but integrating this insight quickly with infrastructure insight is better. Many investigation workflows jump back and forth across these layers several times in a single step, so fully integrated workflows will be essential to leverage the assimilation of new insight. Dragon #3 – Investigation Times Measured in 10s of Minutes and Hours In cloud and modern application deployment, the sheer volume of incoming data will make yesterday’s data avalanche seem like a pleasant snow dusting. Also, dynamic and transient data, entities, and nomenclature make workflows straightforward (although still slow and annoying) in the old world (e.g. track changing IP addresses for a user or machine) extremely challenging in the cloud. Finally, collaboration will require new models of distributed knowledge transfer since investigation workflows will be shared across both security and IT ops. [Read More: Threat Intelligence] Many SOCs are at the breaking point in traditional environments with growing backlogs of investigations and reactive triage. Achieving investigation times in minutes to keep pace in the cloud despite these additional challenges, will require breakthrough innovation in getting rapid insight in huge dynamic data sets and in scaling learning models across both humans and machines. Slaying these dragons will not be easy or quick – new solutions and thinking will collide with comfort zones, entrenched interests, perceived roles of people and process, and more than a few “sacred cows.” Despite these headwinds – I’m optimistic looking ahead based on two core beliefs: 1) The massive economic and technological leverage of the cloud has already led to many other transition dragons of comparable ferocity being attacked with zeal (e.g. DevSecOps, Data Privacy, Regional Regulation, etc.), and 2) unlike many other transitions a broad cross section of the individuals involved in these messy transitions on the front lines have far more to gain in the leap forward of their own skills, learning, and opportunity than they have to lose. Aside from that, the increasingly public scorecard of the attackers vs. the defenders will help keep us honest about progress along the way.

Docker is hard.Don't get me wrong. It's not the technology itself that is difficult...It's the learning curve. Committing to a Docker-based infrastructure means committing to a new way of thinking, which can be a harsh adjustment from the traditional thinking behind bare metal and virtualized servers.Because of Docker's role-based container methodology, simple things like log management can seem like a bear to integrate. Thankfully, as with most things in tech, once you wrap your head around the basics, finding the solution is simply a matter of perspective and experience.Collecting LogsWhen it comes to aggregating Docker logs in Sumo Logic, the process starts much like any other: Add a Collector. To do this, open up the Sumo Logic Collection dashboard and open up the Setup Wizard.Because we will be aggregating logs from a running Docker container, rather than uploading pre-collected logs, select the Set Up Streaming Data option in the Setup Wizard when prompted.Next up, it is time to select the data type. While Docker images can be based on just about any operating system, the most common base image—and the one used for this demonstration—is Linux-based.After selecting the Linux data type, it's time for us to get into the meat of things. At this point, the Setup Wizard will present us with a script that can be used to install a Collector on a Linux system.The DockerfileWhile copying and pasting the above script is generally all that is required for a traditional Linux server, there are some steps required to translate it into a Docker-friendly environment. To accomplish this, let's take a look at the following Dockerfile:<strong>FROM</strong> ubuntu<strong>RUN</strong> apt-get update <strong>RUN</strong> apt-get install -y wget nginx<strong>CMD</strong> /etc/init.d/nginx start && tail -f /var/log/nginx/access.logThat Dockerfile creates a new container from the Ubuntu base image, installs NGINX, and then prints the NGINX access log to stdout (which allows our Docker image to be long-running). In order to add log aggregation to this image, we need to convert the provided Linux Collector script into Docker-ese. By replacing the sudo and && directives with RUN calls, you'll end up with something like this:<strong>RUN</strong> wget "https://collectors.us2.sumologic.com/rest/download/linux/64" -O SumoCollector.sh <strong>RUN</strong> chmod +x SumoCollector.sh <strong>RUN</strong> ./SumoCollector.sh -q -Vsumo.token_and_url=b2FkZlpQSjhhcm9FMzdiaVhBTHJUQ1ZLaWhTcXVIYjhodHRwczovL2NvbGxlY3RvcnMudXMyLnN1bW9sb2dpYy5jb20=Additionally, while this installs the Sumo Logic Linux Collector, what it does not do is start up the Collector daemon. The reason for this goes back to Docker's "one process per container" methodology, which keeps containers as lightweight and targeted as possible.While this is the "proper" method in larger production environments, in most cases, starting the Collector daemon alongside the container's intended process is enough to get the job done in a straightforward way. To do this, all we have to do is prefix the /etc/init.d/nginx start command with a /etc/init.d/collector start && directive.When all put together, our Dockerfile should look like this:<strong>FROM</strong> ubuntu<strong>RUN</strong> apt-get update <strong>RUN</strong> apt-get install -y wget nginx<strong>RUN</strong> wget "https://collectors.us2.sumologic.com/rest/download/linux/64" -O SumoCollector.sh <strong>RUN</strong> chmod +x SumoCollector.sh <strong>RUN</strong> ./SumoCollector.sh -q -Vsumo.token_and_url=b2FkZlpQSjhhcm9FMzdiaVhBTHJUQ1ZLaWhTcXVIYjhodHRwczovL2NvbGxlY3RvcnMudXMyLnN1bW9sb2dpYy5jb20=<strong>CMD</strong> /etc/init.d/collector start && /etc/init.d/nginx start && tail -f /var/log/nginx/access.logBuild ItIf you've been following along in real time up to now, you may have noticed that the Set Up Collection page hasn't yet allowed you to continue on to the next page. The reason for this is that Sumo Logic is waiting for the Collector to get installed. Triggering the "installed" status is as simple as running a standard docker build command:docker build -t sumologic_demo .Run ItNext, we need to run our container. This is a crucial step because the Setup Wizard process will fail unless the Collector is running.docker run -p 8080:80 sumologic_demoConfigure the SourceWith our container running, we can now configure the logging source. In most cases, the logs for the running process are piped to stdout, so unless you take special steps to pipe container logs directly to the syslog, you can generally select any log source here. /var/log/syslog is a safe choice.Targeted CollectionNow that we have our Linux Collector set up, let's actually send some data up to Sumo Logic with it. In our current example, we've set up a basic NGINX container, so the easiest choice here is to set up an NGINX Collector using the same Setup Wizard as above. When presented with the choice to set up the Collection, choose the existing Collector we just set up in the step above.Viewing MetricsOnce the Collectors are all set up, all it takes from here is to wait for the data to start trickling in. To view your metrics, head to your Sumo Logic dashboard and click on the Collector you’ve created.This will open up a real-time graph that will display data as it comes in, allowing you to compare and reduce the data as you need in order to identify trends from within your running container.Next StepsWhile this is a relatively simplistic example, it demonstrates the potential for creating incredibly complex workflows for aggregating logs across Docker containers. As I mentioned above, the inline collector method is great for aggregating logs from fairly basic Docker containers, but it isn't the only—or best—method available. Another more stable option (that is out of the scope of this article) would be using a dedicated Sumo Logic Collector container that is available across multiple containers within a cluster. That said, this tutorial hopefully provides the tools necessary to get started with log aggregation and monitoring across existing container infrastructure.

AWS is a vast universe with over 100 cloud services, all geared to provide you with the most robust, reliable, and cost-effective cloud computing experience. Many of these services are closely related, and almost seem to overlap. There are multiple database services, logging services, resource creation services, compute services, notification services and even multiple configuration and management tools.

Last week, a security vulnerability was announced involving the exploitation of common features in microprocessor chips that power computers, tablets, smartphones and data centers. The vulnerabilities known as “Meltdown” and “Spectre” are getting lot attention in the media, and no doubt people are concerned about its impact on business, customers, partners and more. Here’s what you really need to know about these vulnerabilities. What are Meltdown and Spectre? The Meltdown vulnerability, CVE-2017-5754, can potentially allow hackers to bypass the hardware barrier between applications and kernel or host memory. A malicious application could therefore access the memory of other software, as well as the operating system. Any system running on an Intel processor manufactured since 1995 (except Intel Itanium and Intel Atom before 2013) is affected. The Spectre vulnerability has two variants: CVE-2017-5753 and CVE-2017-5715. These vulnerabilities break isolation between separate applications. An attacker could potentially gain access to data that an application would usually keep safe and inaccessible in memory. Spectre affects all computing devices with modern processors manufactured by Intel or AMD, or designed by ARM*. These vulnerabilities could potentially be exploited to steal sensitive data from your computer, such as passwords, financial details, and other information stored in applications. Here is a great primer explaining these security flaws. What can be compromised? The core system, known as the kernel, stores all types of sensitive information in memory. This means banking records, credit cards, financial data, communications, logins, passwords and secret information could which is all be at risk due to Meltdown. Spectre can be used to trick normal applications into giving up sensitive data, which potentially means anything processed by an application can be stolen, including passwords and other data. Was the Sumo Logic platform affected? Yes. Practically every computing device affected by Spectre, including laptops, desktops, tablets, smartphones and even cloud computing systems. A few lower power devices, such as certain Internet of Things gadgets, are unaffected. How is Sumo Logic handling the vulnerabilities? As of January 4th, 2018, AWS confirmed that all Sumo Logic systems were patched, rebooted and protected from the recent Meltdown/Spectre vulnerability. We worked very closely with our AWS TAM team and verified the updates. Sumo Logic started the OS patching process with the latest Ubuntu release Canonical on January 9th. Risk level now that AWS has patched is low, but we will continue to be diligent in following up and completing the remediation process. We take this vulnerability very seriously and are dedicated to ensuring that Sumo Logic platform is thoroughly patched and continuously monitored for any malicious activity. If you have questions please reach out to [email protected].

2017 has been the year for Kubernetes. The orchestrator has become the top orchestration platform, and whenever someone says that they work with Kubernetes, ears start perking up. Docker is still relevant (specifically Swarm mode for orchestration features), but Kubernetes is viewed as a stronger platform.

You’ve probably heard about logs and metrics and have been told you need to monitor them. You may be told to monitor them for compliance or because someone suggested it’d be a great way to reduce time to investigate bugs. It can be a ton to handle and understand, and there’s a great deal of information out there.

This is an exciting time to be in enterprise software. With the rise of serverless, the power of hybrid computing and the endless uses of artificial intelligence (AI), 2017 will likely go down as the most disruptive ever. But what does 2018 have in store? Sumo Logic executives weighed in in our yearly prediction series. Read on to see what they predict will influence the coming year in technology the most. Also keep these in mind and check back in mid-year to see how many come true! Demand for multi-cloud, multi-platform will drive the need for multi-choice Over the past few years, there has been much debate within enterprise IT about moving critical infrastructure to the cloud – specifically, around which cloud model is the most cost effective, secure and scalable. One thing is for certain – the cloud is the present (and future) of enterprise IT, and legacy companies continuing to predominantly or solely house their infrastructure on-premises to support existing or new modern applications will become increasingly irrelevant in a few years from now as their competitors will prevail. Moreover, this problem is further exacerbated as cloud users are demanding choice, which is going to drive massive growth in multi-cloud, multi-platform adoption in 2018. As a result, enterprises will need a unified cloud native analytics platform that can run across any vendor, whether it’s Amazon, Microsoft or Google, including what’s traditionally running on-premise. This agnostic model will serve as the backbone for the new world I refer to as the analytics economy, defined by positive disruption at every layer of the stack. –Ramin Sayar, CEO, Sumo Logic ICO creates another Wild West Next year we will begin to see the results from the growth in the initial coin offering (ICO) frenzy from this year. Investors have poured more than $300B into more than 200 ICOs this year, but it’s still a very unregulated market with a mix of players attempting to gain early entry. While legitimate companies are seeking and benefiting from crypto-token funding, there is still a lot of dubious activity from questionable characters in the space trying to make a quick buck. If crypto-token equity begins to take hold and legitimizes its worth to investors who pursued ICOs this year, then in 2018 the startup market will become the wild freaking west. AI will not transform the enterprise in the near future Previous predictions and claims about the direct impact of AI on enterprises have been overblown. There is excessive hype around how AI will lead us to new discoveries and medical breakthroughs. However, those expecting AI to be the ultimate truth conveyer are mistaken. It will be very hard to design a model that can determine unbiased truth, because human bias – whether explicitly or implicitly – will be coded into these data analytics systems and reinforce existing beliefs and prejudices. With that said, there are certain applications where systems can make better decisions in a shorter amount of time than humans, such as in the case of autonomous vehicles. In 2018 we will begin to see real use cases of the power of AI appear in our everyday lives — it just isn’t ready to be the shining star for the enterprise quite yet. When you look at the maturity of the enterprise, only half of the Global 2000 offer fully digital products. So, despite all of the buzz around digital transformation, there’s a lot of catch-up to be done before many of these companies can even consider looking at advanced developments such as AI. — Christian Beedgen, CTO, Sumo Logic GDPR regulations will turn massive tech companies into walking targets It won’t take long after the May 25 GDPR deadline before the gloves come off and the European Union cracks down on audits of big tech companies. We’re talking about Uber, Google, Apple and so forth. This will be EU’s effort to reinforce the severity of meeting GDPR regulations and to show that no business – not even the household names – will be immune to complying with GDPR stands. After the EU cracks down on the big tech companies, financial institutions and travel companies will be next, as these types of organizations are the most globalized industries, where data flows freely across geographical borders. And regardless of the EU’s efforts, the reality is that many companies won’t meet the May deadline, whether due to lack of resources, laziness or apathy. You better believe that those businesses that don’t get on board – and get caught – will be crushed, as business will come to a grinding halt. Government will continue to fall flat with security If I were a hacker, I would target the path of least resistance, and right now – and into 2018 – that path collides squarely with government agencies. What’s scary is that government organizations hold some of our most critical data, such as social security numbers, health records and financial information. It’s shocking how the government generally lags in terms of security and technology innovation. Over the past few years the government has been a prime target for bad actors. Take a look at the Office of Personnel Management breach in 2015, and more recently the hacks into the Department of Homeland Security and FBI in 2016. Next year will be no different. Even with all of the panels, hearings and legislation, such as the Modernizing IT Act and the executive order, reaffirming its commitment to updating and implementing stronger cybersecurity programs, the government is already 10-15 years behind, and I don’t see this improving over the next year. Millennials will be our security saving grace Millennials will inspire a societal shift in the way we view security and privacy. If you follow the data, it’ll make sense. For instance, Facebook is now most popular among adults age 65 and older. It’s less appealing to younger generations who’ve moved on to newer, more secure ways to express themselves, such as disappearing video chats with Snapchat. As social media evolves, privacy, user control/access and multi-factor authentication have become a natural part of protecting online identity, for both users and developers alike. My personal resolution for 2018 is to step up my mentorship to this younger generation. If we can encourage them to channel this “Security First” way of thinking in a professional capacity, we can continue to build a resilient and robust cybersecurity workforce that makes us all more secure. –George Gerchow, VP of Security and Compliance, Sumo Logic Now that you have read Sumo’s, tell us your predictions for the coming year. Tweet them to us at @SumoLogic

Memory leaks happen when programs allocate more memory than they return. Memory is beside Compute one of the critical assets of any computer system. If a machine runs out of memory, it cannot provide its service. In the worst case, the entire machine might crash and tear down all running programs. The bugs responsible for that misbehavior are often hard to find. Sumo’s collector enables monitoring memory consumption out of the box. Using some additional tooling, it is possible to collect fine-grained logs and metrics that accelerate finding and efficient debugging of memory leaks. Ready to get started? See all the ways the Sumo Logic platform helps monitor and troubleshoot—from a seamless ingestion of data, to cross-platform versatility, and more. You can even get started for free.Free Trial Memory Management and Memory Leaks Memory management is done on multiple levels: The Operating System (OS) keeps track of memory allocated by its program in kernel space. In user space, virtual machines like the JVM might implement their own memory management component. At its core, memory management follows a Producer-Consumer pattern. The OS or VM gives away (produces) chunks of memory whenever programs are requesting (consuming) memory. Since memory is a finite resource in any computer system, programs have to release the allocated memory that is then returned to the pool of available memory managed by the producer. For some applications, the programmer is responsible for releasing memory, in others like the JVM a thread called garbage collector will collect all objects that are used no more. A healthy system would run through this give-and-take in a perfect circle. In a bad system, the program fails to return unused memory. This happens for example if the programmer forgets to call the functionfree, or if some objects keep on being referenced from a global scope after usage. In that case, new operations will allocate more memory on top of the already allocated, but unused memory. This is misbehavior is called a memory leak. Depending on the size of the objects this can be as little as a few bytes, kilobytes, or even megabytes if the objects, for example, contain images. Based on the frequency the erroneous allocation is called, the free space fills up as quickly as a few microseconds or it could take months to exhaust the memory in a server. This long time-to-failure can make memory leaks very tricky to debug because it is hard to track an application running over a long period. Moreover, if the leak is just a few bytes this marginal amount gets lost in the noise of common allocation and release operations. The usual observation period might be too short to recognize a trend. This article describes a particularly interesting instance of a memory leak. This example uses the Akka actor framework, but for simplicity, you can think of an actor as an object. The specific operation in this example is downloading a file: An actor is instantiated when the user invokes a specific operation (download a file) The actor accumulates memory over its lifetime (keeps adding to the temporary file in memory) After the operation completes (file has been saved to disk), the actor is not released The root cause of the memory leak is that it can handle only one request and it is useless after saving the content of the file. There are no references to the actor in the application code, but there still is a parent-child relationship defined in the actor system that defines a global scope. From After-the-Fact Analysis to Online Memory Supervision Usually, when a program runs out of memory it terminates with an “Out of Memory” error or exception. In case of the JVM, it will create a heap dump on termination. A heap dump is an image of program’s memory at the termination instant and saved to disk. This heap dump file can then be analyzed using tools such as MemoryAnalyzer, YourKit, or VisualVM for the JVM. These tools are very helpful to identify which objects are consuming what memory. They operate, however, on a snapshot of the memory and cannot keep track of the evolution of the memory consumption. Verifying that a patch works is out of the scope of these tools. With a little scripting, we can remediate this and use Sumo to build an “Online Memory Supervisor” that stores and processes this information for us. In addition to keeping track of the memory consumption history of our application, it saves us from juggling around with heap dump files that can potentially become very large. Here’s how we do it: 1. Mechanism to interrogate JVM for current objects and their size The JVM provides an API for creating actual memory dumps during runtime, or just retrieve a histogram of all current objects and their approximate size in memory. We want to do the latter as this is much more lightweight. The jmap tool in the Java SDK makes this interface accessible from the command line: jmap -histo PID Getting the PID of the JVM is as easy as grepping for it in the process table. Note that in case the JVM runs as a server using an unprivileged user, we need to run the command as this user via su. A bash one-liner to dump the object histogram could look like: sudo su stream -c"jmap -histops -ax | grep "[0-9]* java" | awk '{print $1}' > /tmp/${HOSTID}_jmap-histo-`date +%s`.txt" 2. Turn result into metrics for Sumo or just drop it as logs As a result of the previous operation, we have now a file containing a table with object names, count, and retained memory. IN order to use it in Sumo we’ll need to submit it for ingestion. Here we got two options: (a) send the raw file as logs, or (b) convert the counts to metrics. Each object’s measurement is a part of a time series tracking the evolution of the object’s memory consumption. Sumo Metrics ingest various time series input formats, we’ll use Graphite because it’s simple. To affect the conversion of a jmap histogram to Graphite we use bash scripting. The script cuts beginning and end of the file and then parses the histogram to produce two measurements: <class name, object count, timestamp> <class name, retained size, timestamp> Sending these measurements to Sumo can be done either through Sumo’s collector, using collectd with Sumo plugin, or sending directly to the HTTP endpoint. For simplicity, we’ve used the Graphite format and target the Sumo collector. To be able to differentiate both measurements as well as different hosts we prepend this information to the classpath: <count|size>.<host>.classpath For example, a jmap histogram might contain data in tabular form like: 69: 18 1584 akka.actor.ActorCell 98: 15 720 akka.actor.RepointableActorRef 103: 21 672 akka.actor.ChildActorPath 104: 21 672 akka.actor.Props Our script turns that into Graphite format and adds some more hierarchy to the package name. In the next section, we will leverage this hierarchy to perform queries on objects counts and sizes. count.memleak1.akka.actor.ActorCell 18 123 count.memleak1.akka.actor.RepointableActorRef 15 123 count.memleak1.akka.actor.ChildActorPath 21 123 count.memleak1.akka.actor.Props 21 123 In our case, we’ll just forward these logs to the Sumo collector. Previously, we’ve defined a Graphite source for Metrics. Then, it’s as easy as cat histogram-in-graphite | nc -q0 localhost 2003. 3. Automate processing via Ansible and StackStorm So far we are now capable of creating a fine-grained measurement of an application’s memory consumption using a couple of shell commands and scripts. Using the DevOps automation tools Ansible and StackStorm, we can turn this manual workflow in an Online Memory Supervision System. Ansible helps us to automate taking the measurement of multiple hosts. For each individual host, it connects to the hosts via ssh, runs the jmap command, the python conversion script, and submits the measurement to Sumo. StackStorm manages this workflow for us. In a given period, it kicks off Ansible and logs the process. In case something goes wrong, it defines remediation steps. Of course, there are alternatives to the myriad of available tools. Ansible competes with SaltStack, Chef, and Puppet. StackStorm is event-driven automation with all bells and whistles, for this example, we could have used a shell script with sleepor a simple cron job. Using Sumo to Troubleshoot Memory Leaks Now it’s time to use Sumo to analyze our memory. In the previous steps, we have submitted and ingested our application’s fine-grained memory consumption data. After this preparation, we can leverage Sumo to query the data and build dashboards. Using queries, we can perform in-depth analysis. This is useful as part of a post-mortem analysis to track down a memory leak, or during development to check, if a memory allocation/deallocation scheme actually works. During runtime, dashboards could monitor critical components of the application. Let’s check this out on a live example. We’ll use a setup of three JVMs simulating an application and a StackStorm instance. Each is running in their own Docker container, simulating a distributed system. To make our lives easier, we orchestrate this demo setup using Vagrant: Figure 1: Memory leak demo setup and control flow A Memory Measurement node orchestrates the acquisition process. We’ve developed a short Ansible script that connects to several application nodes and retrieves a histogram dump from the JVMs running the faulty program from [1]. It converts the dumps to Graphite metrics and sends them via the collector to Sumo. StackStorm periodically triggers the Ansible workflow. Finally, we use the UI to find and debug memory leaks. Analyze memory consumption First, we want to get an overview of what’s going on in the memory. We start to look at the total memory consumption of a single host. A simple sum over all objects sizes yields the application’s memory consumption over time. The steeply increasing curve abruptly comes to an end at a total of about 800 Mb. This is the total memory that we dispatched to the JVM (java -Xmx800m -jar memleak-assembly-0.1.jar). Figure 2: Total memory consumption of host memleak3 Drilling down on top memory consumers often hints at the responsible classes for a memory leak. For that query, we parse out all objects and sum their counts and sizes. Then we display only the top 10 counts. In the size query, we filter out objects above a certain size. These objects are the root objects of the application and do not contain much information. Figure 3: Top memory consumers on a single node Figure 4: To memory top consumers by size We find out that a Red-Black Tree dominates the objects. Looking at the Scala manual tells us that HashMaps make extensive use of this data structure: Scala provides implementations of immutable sets and maps that use a red-black tree internally. Access them under the names TreeSet and TreeMap. We know that ActorSystem uses HashMaps to store and maintain actors. Parsing and aggregating queries help to monitor entire subsystems of a distributed application. We use that to find out that the ActorSystem accumulates memory not only on a single host but over a set of hosts. This leads us to believe that this increase might not be an individual error, by a systemic issue. Figure 5: Use query parsing and aggregation operations to display the ActorSystem’s memory consumption A more detailed view of the Child actor reveals the trend how it accumulates memory. The trick in this query is that in the search part we filter out the packages inakka.actor.* the search expression and then use the aggregation part to parse out the single hosts and sum the size values of their individual objects. Since all three JVMs started at the same time, their memory usage increases at a similar rate in this picture. We can also split this query into three separate queries like below. These are looking at how the Child actors on all three hosts are evolving. Figure 6: The bad Child actor accumulating memory Finally, we verify that the patch worked. The latest chart shows that allocation and deallocation are now in balance on all three hosts. Figure 7: Memory leak removed, all good now Memory Analysis for Modern Apps Traditional memory analyzers were born in the era of standalone, desktop applications. Therefore, they work on snapshots and heap dumps and cannot track the dynamicity of memory allocation and deallocation patterns. Moreover, they are also restricted to work on single images and it is not easy to adapt them to a distributed system. Modern Apps have different requirements. Digital Businesses provide service 24/7, scale out in the cloud, and compete in terms of feature velocity. To achieve feature velocity, detecting memory issues online is more useful than after-the-fact. Bugs such as memory leaks need rapid detection and bugfixes inserted frequently and without stopping services. Pulling heap dumps and starting memory analyzers just won’t work in many cases. Sumo takes memory analysis to the next level. Leveraging Sumo’s Metrics product we can track memory consumption for classes and objects within an application. We look at aggregations of their counts and sizes to pinpoint the fault. Memory leaks are often hard to find and need superior visibility into an application’s memory stack to become debuggable. Sumo achieves this not only for a single instance of an application but scales memory analysis across the cloud. Additionally, Sumo’s Unified Logs and Monitoring (ULM) enables correlating logs and metrics and facilitates understanding the root cause of a memory leak. Bottom Line In this post, we showed how to turn Sumo into a fine-grained, online memory supervision system using modern DevOps tools. The fun doesn’t stop here. The presented framework can be easily extended to include metrics for threads and other resources of an application. As a result of this integration, developers and operators gain high visibility in the execution of their application. References Always stop unused Akka actors – Blog Post Acquire object histograms from multiple hosts – Ansible Script Sumo’s Modern Apps report – BI Report

Serverless computing with AWS Lambda is an increasingly popular business solution, but keeping track of Lambda function activity isn’t always so easy. Sumo Logic’s integration with AWS CloudTrail’s logging of Lambda data events helps you continuously monitor your Lambda functions and gain insight into the AWS Lambda function activity.

Security concerns and skill shortages continue to impede cloud adoption Migration to the cloud is still being hampered by the security concerns this new frontier poses to these organizations and due to the same cybersecurity skills gaps already present in many if not most of these organizations today. This was highlighted in a 2017 survey by Forbes where 49% of respondents stated that they were delaying cloud deployment due to a cyber security skills gap. And even with adequate staffing, those organizations who have adopted some facet of cloud into their organization, express concerns in their abilities to monitor and manage these new environments. Sumo Logic and Amazon GuardDuty to the rescue Sumo Logic was founded over seven years ago, by security industry professionals, as a secure, cloud-native, machine data analytics platform, to convert machine data into real-time continuous intelligence, providing organizations with the full-stack visibility, analytics and insights they need to build, run and secure their modern applications and cloud infrastructures. The Sumo Logic platform provides security analytics and visibility across the entire AWS environment with context derived from details such as user access, platform configurations, changes, and with the ability to generate audit trails to demonstrate compliance with industry standards. Sumo Logic also correlates analytics from Crowdstrike threat intelligence to identify risks and threats in the AWS environment such as communications with malicious IPs, URLs, or Domains. At AWS’ annual re:Invent 2017 conference in Las Vegas this week, they announced the availability of Amazon GuardDuty. GuardDuty, provides AWS users with a continuous security monitoring and threat detection service. And due to Sumo Logic’s strong, and long standing relationship with AWS, Sumo Logic was provided early access to the beta version of GuardDuty, which allowed the team to develop, announce and release in parallel with Amazon, the complimentary Sumo Logic Amazon GuardDuty App. Click to enlarge The way GuardDuty works is by gathering log data from three distinct areas of the AWS cloud environment including: AWS Virtual Private Cloud (VPC) “flow logs” AWS CloudTrail “event logs” AWS Route 53 DNS “query logs” Along with the log data above, AWS provides additional sources of context (including threat intel associated with the AWS environment) to provide users with identification of potential threats in their environments. These potential threats are called “findings” by GuardDuty. Each “finding” provides users with details about each of the threats identified so that they can take any necessary action as needed. “Findings” details include to following information: Last seen – the time at which the activity took place that prompted the finding. Count – the number of times the finding was generated. Severity – the severity level (High, Medium, or Low) High – recommendation to take immediate remediation steps. Medium – investigate the implicated resource at your earliest convenience. Low – suspicious or malicious activity blocked. No immediate action needed. Finding Type – details and include the: Threat Purpose (more details available in the GuardDuty User Guide): Backdoor Behavior Cryptocurrency Pentest Recon Stealth Trojan UnauthorizedAccess Resource Type Affected: with the initial release of GuardDuty “only EC2 instances and IAM users (and their credentials) can be identified in findings as affected resources” Threat Family Name: the overall threat or potential malicious activity detected. Threat Family Variant: the specific variant of the Threat Family detected. Artifact: a specific resource owned by a tool used in the attack. Region – the region in which the finding was generated. Account ID – the ID of the AWS account in which the activity took place t Resource ID – the ID of the AWS resource against which the activity took place Target – the area of your AWS infrastructure where GuardDuty detected potentially malicious or anomalous activity Action – the activity that GuardDuty perceived to be potentially malicious or anomalous. Actor – the user that engaged in the potentially malicious or unexpected activity The Sumo Logic Amazon GuardDuty App Value-Add Pre-built Sumo Logic GuardDuty dashboards: Sumo Logic provides a single pane of glass to reduce the complexity of managing multiple environments, with pre-configured, user friendly and customizable dashboards that take GuardDuty’s linear data format and layers-on rich graphical reporting and depictions of trends over time. Click to enlarge Click to Fix: The Sumo Logic Amazon GuardDuty App allows users to rapidly, and visually identify “findings”, ranked by their severity levels (high, medium, and low), and can simply click on any of them to be automatically routed to their AWS environment to take any necessary actions for remediation. Value-added Context: The Sumo Logic Amazon GuardDuty App adds additional sources of analytics for deeper and wider visibility in the AWS environment and context across the organization including full stack visibility into application/infra logs, Application/Elastic Load Balancer (ALB/ELB) performance, and supplemental threat intel provided by Crowdstrike with no additional fees. The new Amazon GuardDuty offering along with capabilities from Sumo Logic’s tightly integrated GuardDuty App provides organizations with the tools they need to more simply and effectively manage and monitor their AWS cloud environments. And with the visibility for more rapid detection and remediation of real and potential threats to mission critical resources in those environments. Get the Sumo Logic Amazon GuardDuty App Sign up for Sumo Logic instantly and for free Watch the Sumo Logic product overview video.

Docker and containers have grown in prominence, so has k8s as the de-facto standard for container orchestration.

I travel to a lot of conferences over the course of the year. But the grand poohbah of them all (and one of my personal favorites) is AWS re:Invent. This has quickly become the must-attend tech event, and with more than 40,000 attendees expected in Las Vegas, this year will no doubt be one for the books. The Sumo Logic team will be there in full force showcasing our real-time machine data analytics platform and how we help businesses get the continuous intelligence needed to build, run and secure modern applications to accelerate digital business transformation. Here’s a rundown of some of our key activities: Sumo Logic Breakout Presentation: Making the Shift to Practical DevSecOps Agility is the cornerstone of the DevOps movement with security best practices and compliance are now the responsibility of everyone in the development lifecycle. Our VP of Security and Compliance George Gerchow will be presenting on Tuesday, Nov. 28 at 2:30 pm at the Aria Hotel. Swing by to learn best practices for making the shift to DevSecOps leveraging the CIS AWS Foundation Benchmarks. Visit us at Booth #1804 Stop by our booth to learn more about the power of real-time machine data analytics and how to centralize your data and turn analytics into business, operational, and security insights for full stack visibility of your AWS workloads. See live demos, talk to our technical experts and pick up limited edition swag! Join us at the Modern App Ecosystem Jam On Wednesday, Nov. 29 we will be hosting a party with our awesome partner ecosystem celebrating today’s new world of building, running and securing modern applications. No presentations, no pitches, just an evening networking with peers. Take a break from AWS re:Invent and join us for yummy Cuban appetizers, specialty mojitos and drinks, cigar rollers, entertainment, swag and more! Space is limited – secure your spot today! Closed-Loop Security Analytics with CloudPassage Throughout the week Sumo Logic will be co-presenting with CloudPassage to highlight our joint integration which gives users a comprehensive, real-time view of security and compliance postures while rapidly detecting and containing attacks. Stop by the CloudPassage booth #913 to learn more. Follow the Conversations on Social We will be live tweeting, posting Facebook Live videos and photos throughout the week. Twitter: @SumoLogic LinkedIn: www.linkedin.com/SumoLogic Facebook: https://www.facebook.com/Sumo.Logic/ For a full list of events and news, check out our re:Invent events page. We look forward to seeing you in Las Vegas next week!

I was able to attend my third Web Summit last week. This is the second time for me in Lisbon, as I was lucky enough to be invited to talk on the Binate.io stage again after last year. If you are interested, check out my musings on instinct, intuition, experience and data analytics. Web Summit has grown tremendously since I first attended the Dublin incarnation in 2013. This year, the event was sold out at 60,000 attendees (!) - the Portuguese came out in force, but it was very clear that this event is, while of course drawing most attendees from all across Europe, ultimately an international affair as well. With so many people attending, Web Summit can be rather overwhelming. There is a bit of everything, and an incredible crowd of curious people. Lisbon is fantastically beautiful city, off the beaten path when it comes to tech conferences mostly, so the local folks are really coming out in force to take in the spectacle. So, what is Web Summit? Originally started in Dublin in 2009, it has over the years become a massive endeavor highlighting every conceivable aspect of technology. There's four massive conference halls with multiple stages for speakers and podium discussions in each hall. Christian Beedgen on binate.io stage - Web Summit, Lisbon 2017 Then there is the main arena holding 20,000 people; this is where the most high-profile keynote speakers hit the stage. Web Summit has always brought in government officials and politicians to the show as well in an effort to promote technology. I was actually standing next to Nigel Farage at the speaker cloak room waiting for my coat. There was another guy there as well who was already berating this unfortunate character, so thankfully I didn't have to do it myself. I managed to catch a couple of the keynotes in the aforementioned large arena. Three of them left an impression. Firstly, it was great to see Max Tegmark speak. I am reading his current book, Life 3.0, right now, and it is always a bit of trip when the author suddenly appears on a stage and you realize you have to throw away your mental image of that voice in your head that has been speaking to you from the pages of the book and adopt reality. In this case however, this was not a negative, as Max came across as both deeply knowledgeable and quite relaxed. He looked a bit like he is playing in the Ramones with his black leather jacket and black jeans; this I didn't see coming. In any case, I highly recommend checking out what he has to say. In light of the current almost bombastically overblown hype around AI, he is taking a very pragmatic view, based on many years of his own research. If you can imagine a future of "beneficial AI", check out his book, Life 3.0, for why and how we have a chance to get there. I was also impressed by Margrethe Vestager. She is a Danish politician and currently the European Commissioner for Competition. She captured the audience by simply speaking off of a couple of cue cards, not PowerPoint slides at all. Being a politician, she was casting a very official appearance, of course - but she wore some sick sneakers to a conservative dress which I thought was just awesome. Gotta love the Danish! Her talk centered around the reasoning behind the anti-trust investigation she brought against Google (which eventually lead to a $2.7 billion fine!) The details are too complicate to be reasonably summarized here, but essentially centered around the fact that while nobody in the EU has issues with Google's near-monopoly on search, in the eyes of the competition watchdogs, for Google to use this position to essentially favor their own products in search results creates intolerable fairness issues for other companies. It is very interesting to see how these views are developing outside of the US. The third and last memorable session had animated AI robots dialoguing with their inventor, Einstein, Artificial General Intelligence, distributed AI and models and the blockchain. It was by and large only missing Taylor Swift. SingularityNET is a new effort to create an open, free and decentralized market place for AI technology, enabled by Smart Contracts. I frankly don't have the slightest clue how that would work, but presenter Ben Goertzel was animatedly excited about the project. The case for needing an AI marketplace for narrow AIs to compose more general intelligences was laid out in a strenuous "discussion" with "lifelike" robots from Hanson Robotics. It is lost on me why everybody thinks they need to co-opt Einstein; first Salesforce calls their machine learning features Einstein, now these robotics guys have an Einstein robot on stage. I guess the path to the future requires still more detours to the past. I guess Einstein can't fight back on this anymore and at least they are picking an exceptional individual... Now that I am back in the US for only a day, the techno-optimism that's pervasive at Web Summit feels like a distant memory already.

DevSecOps has been a hot topic among security professionals. According to DevSecOps.com, the purpose and intent of DevSecOps is “to build on the mindset that everyone is responsible for security with the goal of safely distributing security decisions at speed and scale to those who hold the highest level of context without sacrificing the safety required.” But the road to DevSecOps isn’t necessarily easy. Sumo Logic’s VP of Security and Compliance George Gerchow outlines six steps you can take to get on the path to DevSecOps. Think code analysis, change management, compliance monitoring and more.

If you’re reading this article, you probably already use Apache, and are familiar with its features and functionality. Users of Apache and any technology should also be aware of the importance of monitoring application logs to ensure that they are running optimally, and of using them to identify problems before they cause problems for the consumers of a site.

When building a new application or microservice on AWS, there are several options for handling load balancing in front of the application. In this article, I’ll explain and compare two of the most common and robust options: The built-in AWS Elastic Load Balancer (ELB) or more commonly known as AWS ELB and NGINX’s load balancer.

https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

Whether you're new to automating your image builds with Packer, new to Sumo Logic, or just new to integrating Packer and Sumo Logic, this post guides you through creating an image with Sumo Logic baked in. We'll use AWS as our cloud provider, and show how to create custom machine images in one command that allow you to centralize metrics and logs from applications, OSs, and other workloads on your machines. Overview When baking a Sumo Logic collector into any machine image, you'll need to follow three main steps: First, create your sources.json file, and add it to the machine. This file specifies what logs and metrics you'd like to collect It's usually stored at /etc/sources.json, although you can store it anywhere at point to it Next, download, rename, and make the collector file executable. Collector downloads for various operating systems and Sumo Logic deployments can be found here An example command might look like: sudo wget 'https://collectors.us2.sumologic.com/rest/download/linux/64' -O SumoCollector.sh && sudo chmod +x SumoCollector.sh Finally, run the install script and skip registration. The most important part here is to use the -VskipRegistration=true flag so that the collector doesn't register to the temporary machine you are trying to built the image with Other important flags include -q > Run the script in quiet mode -Vephemeral=true > This tells Sumo Logic to auto-remove old collectors that are no longer alive, usually applicable for autoscaling use cases where VMs are ephemeral -Vsources=/etc/sources.json > Point to the local path of your sources.json file -Vsumo.accessid=<id> -Vsumo.accesskey=<key> > This is your Sumo Logic access key pair See all installation options here An example command might look like: sudo ./SumoCollector.sh -q -VskipRegistration=true -Vephemeral=true -Vsources=/etc/sources.json -Vsumo.accessid=<id> -Vsumo.accesskey=<key> Packer and Sumo Logic - Provisioners Packer Provisioners allow you to communicate with third party software to automate whatever tasks you need to built your image. Some examples of what you'd use provisioners for are: installing packages patching the kernel creating users downloading application code In this example, we'll use the Packer Shell Provisioner, which provisions your machine image via shell scripts. The basic steps that Packer will execute are: Start up an EC2 instance in your AWS account Download your sources.json file locally, which describes the logs and metrics you'd like to collect Download the Sumo Logic collector agent Run the collector setup script to configure the collector, while skipping registration (this creates a user.properties config file locally) Create the AMI and shut down the EC2 instance Print out the Amazon Machine Image ID (AMI ID) for your image with Sumo baked in Instructions: Packer and Sumo Logic Build Before You Begin To ensure Packer can access your AWS account resources, make sure you have an AWS authentication method to allow Packer to control AWS resources: Option 1: User key pair Option 2: Set up the AWS CLI or SDKs in your local environment I have chosen option 2 here so my Packer build command will not need AWS access key pair information. After setting up your local AWS authentication method, create a Sumo Logic free trial here if you don't already have an account. Then, generate a Sumo Logic key pair inside you Sumo Logic account. Copy this key down, as the secret key will only be shown once. Step 1 - Get Your Files After downloading Packer, download the Packer+Sumo_template.json and the packer_variables.json files, and place all 3 in the same directory. Step 2 - Customize Variables and Test Your Template Use the command ./packer validate packer_sumo_template.json to validate your packer template. This template automatically finds the latest Amazon Linux image in whatever region you use, based on the source_ami_filter in the builders object:"source_ami_filter": { "filters": { "virtualization-type": "hvm", "name": "amzn-ami-hvm-????.??.?.x86_64-gp2", "root-device-type": "ebs" }, "owners": ["amazon"], "most_recent": true } Customize the Region in the packer_variables.json file to the AWS Region you want to build your image in You can also change the Sumo collector download URL if you are in a different deployment The sources.json file url can be updated to point to your own sources.json file, or you can update the template to use the Packer File Provisioner to upload your sources.json file, and any other files Step 3 - Build Your Image Use the command ./packer build -var-file=packer_variables.json -var 'sumo_access_id=<sumo_id>' -var 'sumo_access_key=<sumo_key>' packer_sumo_template.json to build your image. You should see the build start and finish like this: Image Build Start Image Build Finish Done! Now that you've integrated Packer and Sumo Logic, you can navigate to the AMI section of the EC2 AWS console and find the image for use in Autoscaling Launch Configurations, or just launch the image manually. Now What? View Streaming Logs and Metrics! Install the Sumo Logic Applications for Linux and Host Metrics to get pre-built monitoring for your EC2 Instance: What Else Can Sumo Logic Do? Sumo Logic collects AWS CloudWatch metrics, CloudTrail audit data, and much more. Sumo Logic also offers integrated Threat Intelligence powered by CrowdStrike, so that you can identify threats in your cloud infrastructure in real time. See below for more documentation: AWS CloudTrail AWS CloudWatch Metrics Integrated Threat Intelligence What's Next? In part 3 of this series (will be linked here when published), I'll cover how to deploy an Autoscaling Group behind a load balancer in AWS. We will integrate the Sumo Logic collector into each EC2 instance in the fleet, and also log the load balancer access logs to an S3 bucket, then scan that bucket with a Sumo Logic S3 source. If you have any questions or comments, please reach out via my LinkedIn profile, or via our Sumo Logic public Slack Channel: slack.sumologic.com (@grahamwatts-sumologic). Thanks for reading!

Written by Rod Trent Just a couple years ago, the cloud seemed far enough way that most organizations believed they could take their time migrating from on-premises systems to remote services and infrastructure. But, as the cloud wars heated up, the abilities of the cloud quickened so that now cloud adoption seems like a forgone conclusion. While Amazon continues to be a leader on paper, Microsoft is making some serious inroads with its “intelligent cloud,” Azure. And, we’ve got to the point in just a short period of time where there’s almost nothing keeping organizations from initiating full migrations to the cloud. Amazon, Microsoft, Google and others will continue to clash in the clouds, leapfrogging each other in function, feature, and cost savings. And, that’s good for everyone. Companies can eliminate much of the hardware and network infrastructure costs that have buoyed internal technology services for the last 20 years. A move to the cloud means delivering a freedom of work, allowing the business to expand without those historical, significant investments that dried up the annual budget just a few months into the fiscal year. It’s also a freedom of work for employees, making them happier, more confident, and more productive. For those that have now concluded that it’s time to migrate, and for those that are still wanting to stick a toe in the water to test, taking that first step isn’t as tough as it once was and there’s a “cool factor” about doing so – if for nothing but the cost savings alone. To get started, here are the three biggest hurdles to overcome. Determining How Much or How Little For most organizations shifting from on-premises to the cloud, it’s not an all or nothing scenario. Not everything can or should be run in the cloud – at least not yet. It takes a serious effort and proper due diligence to determine how much can be migrated to operate in the cloud. In a lot of cases, companies need to take an approach where they wean themselves slowly off reliance on on-premises operations and upgrade old systems before moving them. Many companies that go “all in” realize quickly that a slow and steady pace is more acceptable and that a hybrid cloud environment produces the most gains and positions the company for future success. Which leads to the next point… Locating an Experienced Partner Companies should not approach a migration to the cloud as something that is their sole responsibility. The organizations that have the most success are the ones that invested in partnerships. Experienced partners can help minimize headache and cost by identifying the company’s qualified software and solutions and leading organizations’ applications and processes into the cloud. A partner like Sumo Logic can help with application and service migrations. Their experience and solutions are designed to eliminate hassle and ensure success. We are happy to have Sumo Logic as a sponsor of IT/Dev Connections this year. Which leads to the next point… Hire or Educate? There are new skills required for operating in the cloud and not every organization has IT staff or developers that are well-versed in this modern environment. Many companies, through the process of determining the level of cloud and identifying the right partnerships will be able to determine the skills required for both the migration and for the continuing management and support. Once that has been identified, companies can take an inventory of the skills of current IT and developer staff. In some cases, hiring may be an option. However, in most cases, and because the current staff is already acclimated to the business, it makes the most sense to ensure that the current staff is educated for the new cloud economy. There are several resources available, but one in particular, IT/Dev Connections 2017, happens in just a few short weeks. IT/Dev Connections 2017 is an intimately unique conference with heavy focus on the cloud. With many of today’s top instructors onboard delivering over 200 sessions, one week of IT/Dev Connections 2017 delivers enough deep-dive instruction to provide enough opportunity to train the entire staff on the cloud and ensure the company’s migration is successful and enduring. IT/Dev Connections 2017 runs October 23-26, 2017 in San Francisco. Visit the site to register, to identify speakers you know, and to view the session catalog. Rod Trent is the Engagement, Education, and Conference Director for Penton. He has more than 25 years of IT experience, has written many books, thousands of articles, owned and sold businesses, has run and managed editorial teams, and speaks at various conferences and user groups. He’s an avid gadget fan, a die-hard old television show and cartoon buff, a health and exercise freak, and comic book aficionado. @rodtrent

Logging is one of the many ways in which virtual machines fundamentally differ from Docker containers. It’s also an aspect of migrating to Docker that can be easy to overlook—but one that you absolutely need to plan for if you intend to get the most out of Dockerized infrastructure and from Docker logs.

AWS and Azure are the top two cloud vendors. While AWS has a larger market share, Azure boasts of a larger customer base among the top Fortune companies. While AWS had the first mover advantage, Azure was quick to catch up. With every new update, they keep up with each other, and there’s little that separates these two leading cloud service providers.

Docker Monitoring: How It Works When it comes to monitoring and logging in Docker, the recommended pathway for developers has been for the container to write to its standard output, and let Docker collect the output. Then you configure Docker to either store it in files, or send it to syslog. Another option is to write to a directory, so the plain log file is the typical /var/log thing, and then you share that directory with another container. In practice, when you stop the first container, you indicate that /var/log will be a “volume,” essentially a special directory, that can then be shared with another container. Then you can run tail -f in a separate container to inspect those logs. Running tail by itself isn’t extremely exciting, but it becomes much more meaningful if you want to run a log collector that takes those logs and ships them somewhere. The reason is you shouldn’t have to synchronize between application and logging containers (for example, where the logging system needs Java or Node.js because it ships logs that way). The application and logging containers should not have to agree on specific dependencies, and risk breaking each others’ code. Docker Logging: The 12-Factor App However, this isn’t the only way to log in Docker. Remember the 12-Factor app, a methodology for building SaaS applications, recommending that you limit to one process per container as a best practice, with each running unbuffered and sending data to Stdout. There are numerous options for container logging from the pre-Docker 1.6 days forward, and are better than others. You could: Log Directly from an ApplicationInstall a File Collector in the ContainerInstall a File as a ContainerInstall a Syslog Collector as a ContainerUse Host Syslog for Local SyslogUse a Syslog Container for Local SyslogLog to Stdout and use a file collectorLog to StdOut and use LogspoutCollect from the Docker File systems (Not recommended)Inject Collector via Docker Exec Docker Logging Drivers in Docker Engine Docker 1.6 added 3 new log drivers: docker logs, syslog, and log-driver null. The driver interface was meant to support the smallest subset available for logging drivers to implement their functionality. Stdout and stderr would still be the source of logging for containers, but Docker takes the raw streams from the containers to create discrete messages delimited by writes that are then sent to the logging drivers. Version 1.7 added the ability to pass in parameters to drivers, and in Docker 1.9 tags were made available to other drivers. Importantly, Docker 1.10 allows syslog to run encrypted, thus allowing companies like Sumo Logic to send securely to the cloud. Recent proposals for Google Cloud Cloud Logging driver, and the TCP, UDP, Unix Domain Socket driver. “As part of the Docker engine, you need to go through the engine commit protocol. This is good, because there’s a lot of review stability. But it is also suboptimal because it is not really modular, and it adds more and more dependencies on third party libraries.” In fact, others have suggested the drivers be external plugins, similar to how volumes and networks work. Plugins would allow developers to write custom drivers for their specific infrastructure, and it would enable third-party developers to build drivers without having to get them merged upstream and wait for the next Docker release. A Comprehensive Approach for Docker Monitoring and Logging To get real value from machine-generated data, you need to look at “comprehensive monitoring.” There are five requirements to enable comprehensive monitoring. 5 Requirements of Comprehensive Monitoring Events Let's start with events. The Docker API makes it trivial to subscribe to the event stream. Events contain lots of interesting information. The full list is well described in the Docker API doc, but let’s just say you can track containers come and go, as well as observe containers getting killed, and other interesting stuff, such as out of memory situations. Docker has consistently added new events with every version, so this is a gift that will keep on giving in the future. Think of Docker events as nothing but logs. And they are very nicely structured—it's all just JSON. If, for example, you load this into my log aggregation solution, you can now track which container is running where. I can also track trends - for example, which images are run in the first place, and how often are they being run. Or, why are suddenly 10x more containers started in this period vs. before, and so on. This probably doesn't matter much for personal development, but once you have fleets, this is a super juicy source of insight. Lifecycle tracking for all your containers will matter a lot. Configurations Docker events, among other things, allow us to see containers come and go. What if we wanted also to track the configurations of those containers? Maybe we want to track drift of run parameters, such as volume settings, or capabilities and limits. The container image is immutable, but what about the invocation? Having detailed records of container starting configurations in my mind is another piece of the puzzle towards solving total visibility. Orchestration solutions will provide those settings, sure, but who is telling those solutions what to do? From experience, we know that deployment configurations are inevitably going to be drifting, and we have found the root cause to otherwise inscrutable problems there more than once. Docker allows us to use the inspect API to get the container configuration. Again, in my mental model, that's just a log. Send it to your aggregator. Alert on deviations, use the data after the fact for troubleshooting. Docker provides this info in a clean and convenient format. Logs Well, obviously, it would be great to have logs, right? Turns out there are many different ways to deal with logs in Docker, and new options are being enabled by the new log driver API. Not everybody is quite there yet in 12-factor land, but the again there are workarounds for when you need fat containers and you need to collect logs from files inside of containers. More and more people following the best practice of writing logs to standard out and standard error, and it is pretty straightforward to grab those logs from the logs API and forward them from there. The Logspout approach, for example, is really neat. It uses the event API to watch which containers get started, then turns around and attaches to the log endpoint, and then pumps the logs somewhere. Easy and complete, and you have all the logs in one place for troubleshooting, analytics, and alerting. Stats Since the release of Docker 1.5, container-level statistics are exposed via a new API. Now you can alert on the "throttled_data" information, for example - how about that? Again (and at this point, this is getting repetitive, perhaps), this data should be sucked into a centralized system. Ideally, this is the same system that already has the events, the configurations, and the logs! Logs can be correlated with the metrics and events. There are many pieces to the puzzle, but all of this data can be extracted from Docker pretty easily today already. Docker Daemon Logs and Hosts In all the excitement around APIs for monitoring data, let's not forget that we also need to have host level visibility. A comprehensive solution should therefore also work hard to get the Docker daemon logs, and provide a way to get any other system level logs that factor into the way Docker is being put to use on the hosts of the fleet. Add host level statistics to this and now performance issues can be understood in a holistic fashion - on a container basis, but also related to how the host is doing. Maybe there's some intricate interplay between containers based on placement that pops up on one host but not the other? Without quick access to the actual data, you will scratch your head all day. User Experience What's the desirable user experience for a comprehensive monitoring solution for Docker? Thanks to the API-based approach that allows us to get to all the data either locally or remotely, it should be easy to encapsulate all the monitoring data acquisition and forwarding into a container that can either run remotely, if the Docker daemons support remote access, or as a system container on every host. Depending on how the emerging orchestration solutions approach this, it might not even be too crazy to assume that the collection container could simply attach to a master daemon. It seems Docker Swarm might make this possible. Super simple, just add the URL to the collector config and go. Sumo Logic API and Docker Logging In its default configuration, our containerized Collector agent will use the Docker API to collect the logs and statistics (metrics) from all containers, and the events that are emitted from the Docker Engine. Unless configured otherwise, the Collector will monitor all containers that are currently active, as well as any containers that are started and stopped subsequently. Within seconds, the latest version of the Collector container will be downloaded, and all of the signals coming from your Docker environment will be pumped up to Sumo Logic’s platform. Using the API has its advantages. It allows us to get all 3 telemetry types (logs, metrics, and events), we can query for additional metadata during container startup, we don’t have to accommodate for different log file locations, and the integration is the same regardless of whether you log to files, or to journalD. The Benefits of Docker Agent-Based Collection The other advantage of this approach is the availability of a data collection agent that provides additional data processing capabilities and ensures reliable data delivery. Data processing capabilities include multiline processing, and data filtering and masking of data before leaving the host. This last capability is important when considering compliance requirements such as PCI or HIPAA. Also important from a compliance standpoint is reliability. All distributed logging systems must be able to accommodate networking issues or impedance mismatches, such as latency or endpoint throttling. These are all well covered issues when using the Sumo Logic Collector Agent. Docker Multiline Logging Lack of multiline logging support has always plagued Docker logging. The default Docker logging drivers, and the existing 3rd party logging drivers, have not supported multiline log messages, and for the most part, they still do not. One of Sumo Logic’s strengths has always been its ability to rejoin multiline log messages back into a single log message. This is an especially important issue to consider when monitoring JVM-based apps, and working with stack traces. Sumo Logic automatically infers common boundary patterns, and supports custom message boundary expressions. We ensure that our Docker Log Source and our Docker Logging Plugin maintain these same multiline processing capabilities. The ability to maintain multiline support is one of the reasons why we recommend using our custom Docker API based integration over simply reading the log files from the host. Generally speaking, reading container logs from the file system is a fine approach. However, when the logs are wrapped in JSON, and ornamented with additional metadata, it makes the multiline processing far more difficult. Other logging drivers are starting to consider this issue, no doubt based on market feedback. However, their capabilities are far less mature than Sumo’s. Instant Gratification of Docker Logging The installation of the containerized agent couldn’t be simpler. And with a simple query, you can see the data from all of the containers on your host, with all of the fields extracted and ready to explore. From there, it is easy to install our Docker App to monitor your complete Docker Environment as you scale this out to all of your hosts. Going Beyond Docker Basics When you deploy the Sumo Logic Collector container across a fleet of hosts, monitoring hundreds or thousands of containers, you will want to be a bit more sophisticated than just running with the default container settings. However, that is beyond the scope of this discussion. When you deploy our Collector Agent as a container, all of the Collector agent’s features are available, and all parameters can be configured. To read about how to dive into the advanced configuration options, check out the container’s readme on Docker Hub and read more details in our documentation . Sometimes You Gotta Go Agentless There are times when you require an agentless solution – or you may just prefer one. If you have another way to collect Docker container metrics, and you just need container logs, then a Docker Logging Plugin (earlier versions referred to as Logging Drivers) may be the perfect solution. Note: The agentless approach is an ideal solution for AWS ECS users that rely on CloudWatch for their container metrics and events. How Sumo Logic's Docker Logging Plugin Works Our Docker Logging Plugin is written in Go, and runs within the Docker Engine. It is configured on a per container basis, and sends data directly to Sumo Logic’s HTTP Endpoint, using a pre-configured “HTTP Source.” You can access our plugin on the new Docker Store , but the best place to read about how to use it is on its Github repo. Following the theme set out earlier, it is very easy to use in its default configuration, with a host of advanced options available. Follow these simple steps: Register the plugin with the Docker Engine :$ docker plugin install –grant-all-permissions store/sumologic/docker-logging-driver:<ver>(make sure you go to the Docker Store, and get the latest version number. As of this publishing, the latest version is 1.0.1 , and Docker Store does not support the ‘latest’ parameter. So, here is the corresponding command line for this version:$ docker plugin install –grant-all-permissions store/sumologic/docker-logging-driver:1.0.1 )Specify the driver when you run a container:$ docker run –log-driver=sumologic –log-opt sumo-url=<sumo_HTTP_url> Docker Logging Plugin Capabilities This plugin provides some very important capabilities: Buffering and batching. You can configure the size of each HTTP POSTCompression: Configurable gzip compression levels to minimize data transfer costsProxy support: Critical for highly secure enterprise deploymentTLS Required: This is a Sumo Logic requirement. All data transfer must meet PCI compliance requirements.Multiline Support: Multiline stitching is processed within the Sumo Logic cloud platform rather than in the logging plugin. This keeps the plugin fast and efficient. However, we made specific design considerations to ensure that the we preserved multiline support while providing rich metadata support.Configurable Metadata per Container: The Docker Logging Plugin framework supports a flexible templating system that is used by our plugin to construct dynamic Source Category metadata that varies per container. The template syntax gives you access to environment vars, docker labels, and the ability to pass in custom values when starting containers. Our Docker Logging Plugin is the first of our integrations to support this capability. A similar capability will be supported by our Docker Log and Stats Sources with our next Collector release. Integrating With Other Docker Source Agents If, for some reason, these two methods do not satisfy your needs, then one of our many other collection methods (aka “Sources”) will most likely do the trick. Sumo Logic also integrates with various other open source agents and cloud platform infrastructures, and relies on some of them for certain scenarios. Details on all of the above integrations are available in our docs. If you have been using Docker for a while, and have implemented a solution from the early days, such as syslog or logspout, we encourage you to review the approaches defined here, and migrate your solution accordingly.

IT has always given importance to security and compliance. As infrastructure becomes more and more complex, IT needs to evolve toward more mature ways of monitoring applications and infrastructure. This means letting go of strategies, models, and products such as SIEM that used to work a decade ago, but are unable to serve the needs of today’s cloud-based microservice applications.

Miguel Ortiz from L-STEAM Podcast chats with George Gerchow about cloud maturity, security, and the importance of constant learning. [audio mp3="http://www.sumologic.com/wp-content/uploads/GeorgeGerchow_buzzsprout.mp3"][/audio]

Sumo Logic: what thoughts come to your mind when you hear these two words? Some images that cross your mind may be a genius sumo wrestler solving math problems, or you might think that it’s the title of an autobiography of a Sumo wrestler, but really other than being a cool name, it can be a mystery. Well, at least that’s what came to my mind when I was told that I would be interning in the marketing department of Sumo Logic. Hello to the beautiful people reading this, my name is Danny Nguyen and when I first started interning at Sumo Logic, I was nothing more than your average 17 year old high schooler walking into a company thinking that he was way in over his head. Coming into Sumo Logic, I had the exact same panic, nervousness, and fear that anyone would have at their very first job – worried about making a good impression, worried that the work would be too much, and worried that the people he would be working with wouldn’t laugh at his jokes. Before starting my internship, I had these preconceived thoughts of the stereotypical intern. I figured that I would just be someone to give busy work to and someone that would be bossed around. I thought that I wouldn’t be taken seriously and looked down upon because I was still so young. However, as soon as my very first day at Sumo Logic came to an end, I knew that those worries that I had would be nothing but a figment of my imagination because being an intern at Sumo Logic had become and still is, one of the best experiences of my life. Sumo Logic completely redefined what I thought being an intern and working at a company meant. I was constantly learning something new and meeting new people every single week. I was treated with the same respect as if I were a full-time employee and coworker. I was able to gain a real-life experience of working in the marketing department with hands on learning. However, the greatest thing at Sumo Logic, that I will always remember, are all the people that make up the foundation and give it the amazing personality that it has. They made sure that I was getting a great experience where I made as many connections and learned as many things as I could. I was encouraged and inspired every single day to learn and to keep becoming a better version of myself than I was the previous day. But most importantly, people genuinely wanted to get to know as a person and become my friend. So when you ask me what Sumo Logic means to me today, I could type up a two page essay expressing all the different words and adjectives to describe my gratitude, love, and appreciation that I have for Sumo Logic and the people there – and it still would not be enough. – Danny From Maurina, Danny’s Manager: High school intern – I’ll let that sit with you a moment. When I agreed to take in a high school intern (gulp) here at Sumo Logic, I was worried to say the least, but our experience with Danny blew any expectations I had out of the water. A coworker here at Sumo introduced me to Genesys Works, whose mission is to transform the lives of disadvantaged high school students. I met with the Bay Area coordinator, and realized this was an amazing chance to make a difference in a young person’s life. I signed on…then I was terrified. Mentor and her pupil When Danny’s first day rolled around, I was unsure what to expect. From the minute we started talking, however, all my fears were put to rest. Our first conversation was a whirlwind of questions – “What’s your favorite type of food?” “What’s your degree in?” “Do you have any regrets in life?”…wait what? From there, I knew that Danny wasn’t your average 17 year old intern. For the nine months I managed Danny, I watched him grow from a quiet 17 year old, to a vibrant, confident, young man/professional who could turn a scribble into a perfect Salesforce report (hard to do if you know anything about Salesforce reporting), a Post-it drawing into a display ad, and a skeptical manager into a true believer in the power of drive and passion, regardless of age. Thank you Danny! <3

Security intelligence for SaaS and AWS Workloads is different than your traditional on-prem environment Based on Okta’s latest Business@Work report, organizations are using between 16-22 SaaS applications in their environment. In the report, Office 365 comes out as the top business applications followed by Box and G suite. These business-critical SaaS applications hold sensitive and valuable company information such as financial data, employee records, and customer data. While everyone understands that SaaS applications provide immediate time-to-value and are increasing in adoption at a faster pace than ever before, what many fail to consider is that these SaaS applications also create a new attack surface that represents substantial risk for the company due to the lack of visibility that security operations teams would typically have with traditional, on-prem applications. If employee credentials are compromised, it creates huge exposure for the company because the attacker is able to access all the applications just like an insider would. In this case, timely detection and containment of an insider threat become extremely important. Sumo Logic’s security threat intelligence will allow security operations to address the many challenges related to SaaS and cloud workload security. There are many challenges for incident management and security operations teams when organizations are using SaaS applications: How do you make sure that users across SaaS applications can be uniquely identified? How can you track anomalies in user behavior? The first step from the attacker after exploiting the vulnerability is to steal employee’s identity and move laterally in the organization. In that process, the attacker’s behavior will be considerably different than the normal user’s behavior. Second, it is critical that the entire incident response and management processes are automated for detection and containment of such attacks to minimize potential damage or data leakage. Most organizations moving to the cloud have legacy solutions such as Active Directory and on-prem SIEM solutions. While traditional SIEM products can integrate with Okta, they cannot integrate effectively with other SaaS applications to provide complete visibility into user activities. Considering there are no collectors to install to get logs from SaaS applications, traditional SIEM vendors will not be able to provide the required insight into the modern SaaS application and AWS workloads. In order to solve for these specific problems, Okta and Sumo Logic have partnered to provide better visibility and faster detection of insider threats. Okta ensures that every user is uniquely identified across multiple SaaS applications. Sumo Logic can ingest those authentication logs from Okta and be able to correlate with the user activities across multiple SaaS applications such as Salesforce, Box, and Office 365. Sumo Logic has machine learning operators such as multi-dimensional Outlier, LogReduce, and LogCompare to quickly surface the anomaly in the user activities by correlating identity from Okta with the user activities in Salesforce and Office 365. Once the abnormal activities have been identified, Sumo Logic can take multiple actions such as sending Slack message, creating ServiceNow tickets or disabling the user in Okta or triggering actions within a customer’s automation platform. The use case: Okta + Sumo Logic = accurate incident response for cloud workloads and SaaS applications ` How many times have you fat fingered your password and got the authentication failure? Don’t answer it. Authentication failure is a part of life. You cannot launch an investigation every time there is an authentication failure. That would result in too many false positives and an overload of wasted effort for your security operations team. Okta and Sumo Logic allows you to detect multiple authentication failures followed by a successful authentication. It is good enough to launch an investigation at this point, but we all know it could also be a user error. Caps Lock is on, key board is misbehaving or we might have just changed the password and forgotten! To ensure that security operations get more intelligent and actionable insights into such events, Sumo Logic can provide additional context by correlating such authentication failure logs from Okta with user activity across multiple SaaS application. For example, I changed my password and now I am getting authentication failure within Okta. After that I realized the mistake and corrected it, I get the successful authentication. I log into the Box application to work on few documents and signed off. Sumo Logic will take this Okta event and correlate with the Box activities. In case the attacker had logged in instead of me, then there will be anomalies in behavior. An attacker might download all documents or make ownership changes to the documents. While this is happening, Sumo Logic will be able to spot these anomalies in near real time and be able to take a variety of automated actions from creating a ServiceNow ticket to disable the user in Okta. You can start ingesting your Okta logs and correlate with the user activity logs across multiple SaaS applications now. Sign up for your free Sumo Logic trial that never expires! Co-author Matt Egan is a Partner Solutions Technical Architect in Business Development at Okta. In this role, he works closely with ISV partners, like Sumo Logic, to develop integrations and joint solutions that increase customer value. Prior to joining Okta, Matt has held roles ranging from Software Development to Information Security over an 18 years career in technology.

https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

The General Data Protection Regulation (GDPR) is one of the hottest topics in IT security around the globe. The European Union (EU) regulation gives people more say over what companies can do with their data, while making data protection rules more or less identical throughout the EU. Although this regulation originated in the EU, its impact is global; any organization that does business using EU citizens’ data must be compliant. With the May 2018 deadline looming, IT security professionals worldwide are scrambling to ensure they’re ready (and avoid the strict fines for non-compliance and security breaches). In the video below, Sumo Logic VP of Security and Compliance George Gerchow offers three ways to get you GDPR-ready in no time. 1. Establish a Privacy Program Establishing a privacy program allows you to set a baseline for privacy standards. Once you have a privacy program in place, when new regulations like GDPR are released, all you have to do is fill in the gaps between where you are and where you need to be. 2. Designate a Data Protection Officer This is a critical part of complying with GDPR—and a great way to build sound data security principles into your organization. Under the GDPR requirements, the Data Protection Officer: Must report directly to the highest level of management Can be a staff member or an external service provider Must be appointed on the basis of professional qualities, particularly expert knowledge on data protection law and practices Must be provided with appropriate resources to carry out their tasks and maintain their expert knowledge Must not carry out any other tasks that could result in a conflict of interest 3. Take Inventory of Customer Data and Protections Before GDPR compliance becomes mandatory, take a thorough inventory of where your customer data is housed and how it is protected. Make sure you understand the journey of customer data from start to finish. Keep in mind that the data is only as secure as the systems you use to manage it. As you dissect the flow of data, take note of critical systems that the data depends upon. Make sure the data is secured at every step using proper methodologies like encryption. Bonus Tip: Arrange Third-Party GDPR Validation Between now and May 2018, you still start to see contracts coming through that ask if you are GDPR-compliant. When the deadline rolls around, there will be two groups of organizations out there: Companies that have verification of GDPR compliance to share with prospective clients. Companies that say they are GDPR compliant and want clients to take their word for it. Being in the first group gives your company a head start. Conduct a thorough self-assessment (and document the results) or use a third-party auditor to provide proof of your GDPR compliance. Learn More About GDPR Compliance Ready to get started with GDPR? George Gerchow, the Sumo Logic VP of Security and Compliance, shares more tips for cutting through the vendor FUD surrounding GDPR.

If you administer web servers—or any kind of Windows server, really—there is a good chance that you work at least occasionally with IIS logs. IIS logs store data from Internet Information Services (IIS), Microsoft’s website and web application hosting product. Whether you are a seasoned IIS administrator, or just need to learn to work occasionally with IIS logs, the tips below can help you work more effectively with IIS logs.

Apache, an open source web server that dates back to the mid-1990s, is one of the oldest web servers still in widespread use today. It remains one of the most popular web server platforms out there (even though NGINX has been slowly eroding its market share for a while). This means that there is a good chance that you’ll find yourself at some point working with Apache logs, if you’re not already. Below, I explain how logging in Apache works, where to find logs, what to look for in them and which tools can help you work with them. For this post, I’ll focus on Apache error logs. Some of these tips apply to Apache error logs as well.

Log analysis involves large volume of data. Startups can generate a couple of gigabytes of data per day, and the largest web companies like Facebook log many terabytes of log data every day. (Back in 2009, Facebook was already logging a full 25TB daily.) How can you make sense of all that data? If you try to analyze it by manually without any tools, good luck. Today, the only way to stay on top of the massive amounts of data that your organization produces— whether you’re a small startup or a Fortune 500 company—is to leverage machine learning to help interpret log data in an automated fashion.

Are you using Terraform and looking for a way to easily monitor your cloud infrastructure? Whether you're new to Terraform, or you control all of your cloud infrastructure through Terraform, this post provides a few examples how to integrate Sumo Logic's monitoring platform into Terraform-scripted cloud infrastructure.*This article discusses how to integrate the Sumo Logic collector agent with your EC2 resources. To manage a hosted Sumo Logic collection (S3 sources, HTTPS sources, etc.), check out the Sumo Logic Terraform Provider here or read the blog.Collect Logs and Metrics from your Terraform InfrastructureSumo Logic's ability to Unify your Logs and Metrics can be built into your Terraform code in a few different ways. This post will show how to use a simple user data file to bootstrap an EC2 instance with the Sumo Logic collector agent. After the instance starts up, monitor local log files and overlay these events with system metrics using Sumo Logic's Host Metrics functionality:AWS CloudWatch Metrics and Graphite formatted metrics can be collected and analyzed as well.Sumo Logic integrates with Terraform to enable version control of your cloud infrastructure and monitoring the same way you version and improve your software.AWS EC2 Instance with Sumo Logic Built-InBefore we begin, if you are new to Terraform, I recommend Terraform: Up and Running. This guide originated as a blog, and was expanded to a helpful book by Yevgeniy Brikman.What We'll MakeIn this first example, we'll apply the Terraform code in my GitHub repo to launch a Linux AMI in a configurable AWS Region, with a configurable Sumo Logic deployment. The resources will be created in your default VPC and will include:One t2.micro EC2 instance One AWS Security Group A Sumo Logic collector agent and sources.json fileThe Approach - User Data vs. Terraform Provisioner vs. PackerIn this example, we'll be using a user data template file to bootstrap our EC2 instance. Terraform also offers Provisioners, which run scripts at the time of creation or destruction of an instance. HashiCorp offers Packer to build machine images, but I have selected to use user data in this example for a few reasons:User Data is viewable in the AWS console Simplicity - my next post will cover an example that uses Packer rather than user data, although user data can be included in an autoscaling group's launch configuration For more details, see the Stack Overflow discussion here If you want to build Sumo Logic collectors into your images with Packer, see my blog with instructions hereThe sources.json file will be copied to the instance upon startup, along with the Sumo Logic collector. The sources.json file instructs Sumo Logic to collect various types of logs and metrics from the EC2 instance:Linux OS Logs (Audit logs, Messages logs, Secure logs) Host Metrics (CPU, Memory, TCP, Network, Disk) Cron logs Any application log you needA Note on SecurityThis example relies on wget to bootstrap the instance with the Sumo Logic collector and sources.json file, so ports 80 and 443 are open to the world. In my next post, we'll use Packer to build the image, so these ports can be closed. We'll do this by deleting them in the Security Group resource of our main.tf file.Tutorial - Apply Terraform and Monitor Logs and Metrics InstantlyPrerequisitesFirst, you'll need a few things:Terraform - see the Terraform docs here for setup instructions A Sumo Logic account - Get a free one here Access to an AWS account with AmazonEC2FullAccess permissions - If you don't have access you can sign up for the free tier here An AWS authentication method to allow Terraform to control AWS resourcesOption 1: User key pair Option 2: Set up the AWS CLI or SDKs in your local environmentInstructions1. First, copy this repo (Example 1. Collector on Linux EC2) somewhere locally.You'll need all 3 files: main.tf, vars.tf, and user_data.sh main.tf will use user_data.sh to bootstrap your EC2 main.tf will also use vars.tf to perform lookups based on a Linux AMI map, a Sumo Logic collector endpoint map, and some other variables2. Then, test out Terraform by opening your shell and running:/path/to/terraform planYou can safely enter any string, like 'test', for the var.Sumo_Logic_Access_ID and var.Sumo_Logic_Access_Key inputs while you are testing with the plan command. After Terraform runs the plan command, you should see: "Plan: 2 to add, 0 to change, 0 to destroy." if the your environment is configured correctly.3. Next, run Terraform and create your EC2 instance, using the terraform apply commandThere are some configurable variables built in For example, the default AWS Region that this EC2 will be launched into is us-east-1, but you can pass in another region like this:path/to/terraform/terraform apply -var region=us-west-2If your Sumo Logic Deployment is in another Region, like DUB or SYD, you can run the command like this:path/to/terraform/terraform apply -var Sumo_Logic_Region=SYD5. Then, Terraform will interactively ask you for your Sumo Logic Access Key pair because there is no default value specified in the vars.tf fileGet your Sumo Logic Access Keys from your Sumo Logic account and enter them when Terraform prompts youFirst, navigate to the Sumo Logic Web Application and click your name in the left nav and open the Preferences page Next, click the blue + icon near My Access Keys to create a key pair See the official Sumo Logic documentation here for more infoYou will see this success message after Terraform creates your EC2 instance and Security Group: "Apply complete! Resources: 2 added, 0 changed, 0 destroyed."6. Now you're done!After about 3-4 minutes, check under Manage Data > Collection in the Sumo Logic UI You should see you new collector running and scanning the sources we specified in the sources.json (Linux OS logs, Cron log, and Host Metrics)CleanupMake sure to delete you resources using the Terraform destroy command. You can enter any string when you are prompted for the Sumo Logic key pair information. The -Vephemeral=true flag in our Sumo Logic user data configuration command instructs Sumo Logic to automatically clean out old collectors are no longer alive./path/to/terraform destroyNow What? View Streaming Logs and Metrics!Install the Sumo Logic Applications for Linux and Host Metrics to get pre-built monitoring for your EC2 Instance:What Else Can Sumo Logic Do?Sumo Logic collects AWS CloudWatch metrics, CloudTrail audit data, and much more. Sumo Logic also offers integrated Threat Intelligence powered by CrowdStrike, so that you can identify threats in your cloud infrastructure in real time. See below for more documentation:AWS CloudTrail AWS CloudWatch Metrics Integrated Threat IntelligenceWhat's Next?In part 2 of this post, I'll cover how to deploy an Autoscaling Group behind a load balancer in AWS. We will integrate the Sumo Logic collector into each EC2 instance in the fleet, and also log the load balancer access logs to an S3 bucket, then scan that bucket with a Sumo Logic S3 source.Thanks for reading!Graham Watts is an AWS Certified Solutions Architect and Sales Engineer at Sumo Logic

Are you a bit unsure about the difference between log aggregation and Application Performance Monitoring (APM)? If so, you’re hardly alone. These are closely related types of operations, and it can be easy to conflate them—or assume that if you are doing one of them, there’s no reason to do the other. In this post, we’ll take a look at log aggregation vs APM, and the relationship between these two data accumulation/analysis domains, and why it is important to address both of them with a suite of domain-appropriate tools, rather than a single tool.Defining APM First, let’s look at Application Performance Monitoring, or APM. Note that APM can stand for both Application Performance Monitoring and Application Performance Management, and in most of the important ways, these terms really refer to the same thing—monitoring and managing the performance of software under real-world conditions, with emphasis on the user experience, and the functional purpose of the software. Since we’ll be talking mostly about the monitoring side of cloud APM, we’ll treat the acronym as being interchangeable with Application Performance Monitoring, but with the implicit understanding that it includes the performance management functions associated with APM. What does APM monitor, and what does it manage? Most of the elements of APM fall into two key areas: user experience, and resource-related performance. While these two areas interact (resource use, for example, can have a strong effect on user experience), there are significant differences in the ways in which they are monitored (and to a lesser degree, managed):APM: User Experience The most basic way to monitor application performance in terms of user experience is to monitor response time. How long does it take after a user clicks on an application input element for the program to display a response? And more to the point, how long does it take before the program produces a complete response (i.e., a full database record displayed in the correct format, rather than a partial record or a spinning cursor)?Load is Important Response time, however, is highly dependent on load—the conditions under which the application operates, and in particular, the volume of user requests and other transactions, as well as the demand placed on resources used by the application. To be accurate and complete, user experience APM should include in-depth monitoring and reporting of response time and related metrics under expected load, under peak load (including unreasonably high peaks, since unreasonable conditions and events are rather alarmingly common on the Internet), and under continuous high load (an important but all too often neglected element of performance monitoring and stress testing). Much of the peak-level and continuous high-level load monitoring, of course, will need to be done under test conditions, since it requires application of the appropriate load, but it can also be incorporated into real-time monitoring by means of reasonably sophisticated analytics: report performance (and load) when load peaks above a specified level, or when it remains above a specified level for a given minimum period of time.APM: Resource Use Resource-based performance monitoring is the other key element of APM. How is the application using resources such as CPU, memory, storage, and I/O? When analyzing these metrics, the important numbers to look at are generally percentage of the resource used, and percentage still available. This actually falls within the realm of metrics monitoring more than APM, and requires tools dedicated to metrics monitoring. If percent used for any resource (such as compute, storage or memory usage) approaches the total available, that can (and generally should) be taken as an indication of a potential performance bottleneck. It may then become necessary to allocate a greater share of the resource in question (either on an ongoing basis, or under specified conditions) in order to avoid such bottlenecks. Remember: bottlenecks don’t just slow down the affected processes. They may also bring all actions dependent on those processes to a halt.Once Again, Load Resource use, like response time, should be monitored and analyzed not only under normal expected load, but also under peak and continuous high loads. Continuous high loads in particular are useful for identifying potential bottlenecks which might not otherwise be detected.Log Aggregation It should be obvious from the description of APM that it can make good use of logs, since the various logs associated with the deployment of a typical Internet-based application provide a considerable amount of performance-related data. Much of the monitoring that goes into APM, however, is not necessarily log-based, and many of the key functions which logs perform are distinct from those required by APM.Logs as Historical Records Logs form an ongoing record of the actions and state of the application, its components, and its environment; in many ways, they serve as a historical record for an application. As we indicated, much of this data is at least somewhat relevant to performance (load level records, for example), but much of it is focused on areas not closely connected with performance:Logs, for example, are indispensable when it comes to analyzing and tracing many security problems, including attempted break-ins. Log analysis can detect suspicious patterns of user activity, as well as unusual actions on the part of system or application resources.Logs are a key element in maintaining compliance records for applications operating in a regulated environment. They can also be important in identifying details of specific transactions and other events when they require verification, or are in dispute.Logs can be very important in tracing the history and development of functional problems, both at the application and infrastructure level—as well as in analyzing changes in the volume or nature of user activity over time. APM tools can also provide historical visibility into your environment, but they do it in a different way and at a different level. They trace performance issues to specific lines of code. This is a different kind of visibility and is not a substitute for the insight you gain from using log aggregation with historical data in order to research or analyze issues after they have occurred.The Need for Log Aggregation The two greatest problems associated with logs are the volume of data generated by logging, and the often very large number of different logs generated by the application and its associated resources and infrastructure components. Log aggregation is the process of automatically gathering logs from disparate sources and storing them in a central location. It is generally used in combination with other log management tools, as well as log-based analytics. It should be clear at this point that APM and log aggregation are not only different—It also does not make sense for a single tool to handle both tasks. It is, in fact, asking far too much of any one tool to take care of all of the key tasks required by either domain. Each of them requires a full suite of tools, including monitoring, analytics, a flexible dashboard system, and a full-featured API. A suite of tools that can fully serve both domains, such as that offered by Sumo Logic, can, on the other hand, provide you with the full stack visibility and search capability into your network, infrastructure and application logs.

Cloud Data Breaches are on the rise because organizations do not leverage best practices and make recommended configuration changes to their S3 instances in AWS. As companies transition to the cloud, they should study the shared responsibility model and settle for the default settings. If you take these three simple steps you can help protect your data and prevent breaches in S3.

Continuous Integration, or CI for short, is a development practice wherein developers can make changes to project code and have those changes automatically trigger a process which builds the project, runs any test suites and deploys the project into an environment. The process enables teams to rapidly test and develop ideas and bring innovation faster to the market. This approach allows teams to detect issues much earlier in the process than with traditional software development approaches.With its roots in Oracle’s Hudson server, Jenkins is an open source integration server written in Java. The server can be extended through the use of plugins and is highly configurable. Automated tasks are defined on the server as jobs, and can be executed manually on the server itself, or triggered by external events, such as merging a new branch of code into a repository. Jobs can also be chained together to form a pipeline, taking a project all the way from code to deployment, and even monitoring of the deployed solution in some cases.In this article, we’re going to look at how to set up a simple build job on a Jenkins server and look at some of the features available natively on the server to monitor and troubleshoot the build process. This article is intended as a primer on Jenkins for those who have not used it before, or have never leveraged it to build a complete CI pipeline.Before We Get StartedThis article assumes that you already have a Jenkins server installed on your local machine or on a server to which you have access. If you have not yet accomplished this, the Jenkins community and documentation can be an excellent source of information and resources to assist you. Jenkins is published under the MIT License and is available for download from their GitHub repository, or from the Jenkins website.Within the Jenkins documentation, you’ll find a Guided Tour, which will walk you through setting up a pipeline on your Jenkins box. One of the advantages of taking this tour is that it will show you how to create a configuration file for your pipeline, which you can store in your code repository, side-by-side with your project. The one downside of the examples presented is that they are very generic. For a different perspective on Jenkins jobs, let’s look at creating a build pipeline manually through the Jenkins console.Creating A Build JobFor this example, we’ll be using a project on GitHub that creates a Lambda function to be deployed on AWS. The project is Gradle-based and will be built with Java 8. The principles we’re using could be applied to other code repositories, build and deployment situations.Log in to your Jenkins server, and select New Item from the navigation menu.Jenkins New Item WorkflowChoose a name for your project, select Freestyle project and then scroll down and click OK. I’ll be naming mine Build Example Lambda. When the new project screen appears, follow the following steps. Not all of these steps are necessary, but they’ll make maintaining your project easier.Enter a Description for your project and describe what this pipeline will be doing with it.Check Discard old builds, and select the Log Rotation Strategy with the Max # of builds to keep set to 10. These are the settings I use, but you may select different numbers. Having this option in place prevents old builds from taking too much space on your server.We’ll add a parameter for the branch to build, and default it to master. This will allow you to build and deploy from a different branch if the need arises.Select This project is parameterized.Click on Add Parameter and select String Parameter.Name: BRANCHDefault Value: masterDescription: The branch from which to pull. Defaults to master.Scroll down to Source Code Management. Select Git.Enter the Repository URL. In my case, I entered https://github.com/echovue/Lambda_SQSMessageCreator.gitYou may also add credentials if your Git repository is secure, but setting that up is beyond the scope of this article.For the Branch Specifier, we’ll use the parameter we set up previously. Parameters are added by enclosing the parameter name in curly braces and prefixing it with a dollar sign. Update this field to read */${BRANCH}Git Configuration Using Parameterized BranchFor now, we’ll leave Build Triggers alone.Under Build Environment, select Delete workspace before build starts, to ensure that we are starting each build with a clean environment.Under Build, select Add build step, and select Invoke Gradle script.When I want to build and deploy my project locally, I’ll enter ./gradlew build fatJar on the command line. To accomplish this as part of the Jenkins job, I’ll complete the following steps.Select Use Gradle WrapperCheck From Root Build Script DirFor Tasks, enter build fatJarFinally, I want to save the Fat Jar which is created in the /build/libs folder of my project, as this is what I’ll be uploading to AWS in the next step.Under Post-build Actions, Select Add post-build action and choose Archive the artifacts.In files to archive, enter build/libs/AWSMessageCreator-all-*Finally, click on Save.Your job will now have been created. To run your job, simply click on the link to Build with Parameters. If the job completes successfully, you’ll have a jar file which can then be deployed to AWS Lambda. If the job fails, you can click on the job number, and then click on Console Output to troubleshoot your job.Next StepsIf your Jenkins server is hosting on a network that is accessible from the network which hosts the code repository you’re using, you may be able to set up a webhook to trigger the build job when changes are merged into the master branch.The next logical step is to automate the deployment of the new build to your environment if it builds successfully. Install the AWS Lambda Plugin and the Copy Artifact Plugin on your Jenkins server, and use it to create a job to deploy your Lambda to AWS, which copies the jar file we archived as part of the job we built above.When the deployment job has been successfully created, open the build job, and click on the Configure option. Add a second Post-build action to Build other projects. Enter the name of the deployment project, and select Trigger only if build is stable.At this point, the successful execution of the build job will automatically start the deployment job.Congrats! You’ve now constructed a complete CI pipeline with Jenkins.

When Sumo Logic and New Relic announced a partnership at AWS re:Invent 2016, we immediately started hearing the excitement from our joint customers. The ability to combine the strengths of two leading SaaS services that offer fast time-to-value for monitoring and troubleshooting modern applications would offer a powerful and complete view of digital businesses, from the client down to the infrastructure. Today, we’re pleased to announce another advancement in our partnership: integrated machine data analytics with application and infrastructure performance data in New Relic Insights via a custom New Relic webhook built directly into Sumo Logic. Custom New Relic webhook in Sumo Logic Unlocking Insights from Sumo Logic Scheduled searches in Sumo Logic allow you to monitor and alert on key events occurring in your application and infrastructure. The flexibility of the query language allows you to pull just the information you need while fine tuning the thresholds to trigger only when necessary. Combined with your New Relic APM and New Relic Infrastructure data in New Relic Insights, you’ll now be able to visualize information such as: Events: Service upgrades, exceptions, server restarts, for example Alerts: More than 10 errors seen in 5 minutes, for example, or failed login attempts exceeding 5 in 15 minutes KPIs: Count of errors by host, for example, or top 10 IPs by number of requests Integrating these insights into New Relic provides an integrated context for faster root cause analysis and reduced Mean Time to Resolution (MTTR), all within a single pane of glass. In just three simple steps, you’ll be able to leverage Sumo Logic webhooks to send data to New Relic. Step 1: Configure the New Relic webhook connection In New Relic Insights, you will first need to register an API key that will be used by the Sumo Logic webhook. These keys allow you to securely send custom events into New Relic from different data sources. Type in a short description to keep a record of how this API key will be used, then copy the Endpoint and Key for setup in Sumo Logic. Generate an API Key from New Relic Insights to be used in Sumo Logic In Sumo Logic, create a New Relic webhook connection and insert the Endpoint and Key into the URL and Insert Key fields. The payload field gives you the flexibility to customize the event for viewing in New Relic. In addition to the actual results, you can optionally specify metadata to provide additional context. For example, the name of the Sumo Logic search, a URL to that particular search, a description, and more. This payload can also be customized later when you schedule the search. Variables from your Sumo Logic search can be included in your payload for additional context in New Relic. Step 2: Schedule a search to send custom events After saving your New Relic webhook, you have the option to specify this as the destination for any scheduled search in Sumo Logic. The example below shows a query to look for “Invalid user” in our Linux logs every 15 minutes. To store and visualize this information in New Relic, we simply schedule a search, select the New Relic webhook that we configured in Step 1, and customize the payload with any additional information we want to include. This payload will send each result row from Sumo Logic as an individual event in New Relic. The Sumo Logic query language allows you to transfer meaningful insights from your logs to New Relic Step 3: Visualize events in New Relic Insights Once the scheduled search has been saved and triggered, we can see the data populating in New Relic Insights and use the New Relic Query Language (NRQL) to create the visualizations we need. NRQL’s flexibility lets you tailor the data to your use case, and the visualization options make it seamless to place alongside your own New Relic data. In fact, you might not even notice the difference between the data sources—can you tell which data below is coming from New Relic, and which is coming from Sumo Logic? A unified view: “Source IP’s from Failed Attempts” streams in from Sumo Logic, while “Errors by Class” comes from New Relic The ability to visualize application and infrastructure performance issues alongside insights from your logs reduces the need to pivot between tools, which can speed root cause analysis. If you’ve spotted an issue that requires a deeper analysis of your logs, you can jump right into a linked Sumo Logic dashboard or search to leverage machine learning and advanced analytics capabilities. Learn more Head over to Sumo Logic DocHub for more details on how to configure the New Relic webhook, then schedule some searches to send custom events to New Relic Insights. We’re excited to continue advancing this partnership, and we look forward to sharing more with you in the future. Stay tuned!

The power of modern applications is their ability to leverage the coming together of mobile, social, information and cloud to drive new and disruptive experiences…To enable companies to be more agile, to accelerate the pace at which they roll out new code, to adopt DevSecOps methodologies where traditional siloed walls between the teams are disappearing. But these modern applications are highly complex with new development and testing processes, new architectures, new tools (i.e. containers, micro-services and configuration management tools), SLA requirements, security in the cloud concerns, and explosion of data sources, coming from these new architectures as well as IOT. In this journey to the cloud with our 1500+ customers, we have learned a few things about their challenges: All of this complexity and volume of data is creating unprecedented challenges to enable ubiquitous user access to all this machine data to drive continuous intelligence across operational and security use cases. In this new world of modern applications and cloud infrastructures, they recognize that not all data is created equal. For example, the importance, the life expectancy, the access performance needed, the types of analytics that need to be run against that data. Think IT Operations data (high value, short life span, frequent and high performance access needs) vs. regulatory compliance data (long term storage, periodic searches, esp. at audit times, slower performance may be acceptable). Data ingest in certain verticals such as retail and travel, fluctuate widely and provisioning at maximum capacity loads – with idle capacity the majority of the year – is unacceptable in this day and age. So if we step back for a moment and look at the industry as a whole, what is hindering a company’s ability to unleash their full data potential? The root of the problem comes from two primary areas: 1. The more data we have, the higher the cost 2. The pricing models of current solutions are based on volume of data ingested and not optimized for varying use cases that we are seeing… it is like a “one size fits all” kind of approach Unfortunately, organizations are often forced to make a trade-off because of the high cost of current pricing models, something we refer to as the data tax – the cost of moving data into your data analytics solution. They have to decide: “What data do I send to my data analytics service?” as well as “Which users do I enable with access?” As organizations are building out new digital initiatives, or migrating workloads to the cloud, making these kinds of tradeoffs will not lead to ultimate success. What is needed is a model that will deliver continuous intelligence across operational and security use cases. One that leverages ALL kinds of data, without compromise. We believe there is a better option – one which leverages our cloud-native machine data analytics platform, shifting from a volume based approach – fixed, rigid, static – to a value based pricing model – flexible and dynamic – aligned with the dynamic nature of the modern apps that our customers are building. One that moves us to a place where democratization of machine data is realized! Introducing Sumo Logic Cloud Flex As this launch was being conceived, there were four primary goals we set out to accomplish: Alignment: Alignment between how we priced out service and the value customers received from it. Flexibility: Maximum flexibility in the data usage and consumption controls that best align to the various use cases Universal Access: Universal access of machine data analytics to all users, not just a select few Full Transparency: Real-time dashboards on how our service is being used, the kind of searches people are running, and the performance of the system And there were four problem areas we were trying to address: Data Segmentation: Different use cases require different retention durations Data Discrimination: Not all data sets require the same performance and analytics capabilities Not economical to store and analyze low value data sets Not economical to store data sets for long periods of time, esp. as it relates to regulatory compliance mandates Data Ubiquity: Not economical for all users to access machine data analytics Data Dynamics: Support seasonal business cycles and align revenue with opex So with this Cloud Flex launch, Sumo Logic introduces the following product capabilities to address these four pain points: Variable Data Retention Analytics Profile Unlimited Users Seasonal Pricing If increasing usage flexibility in your data analytics platform is of interest, please reach out to us. If you would like to get more information on cloud flex and the Democratizing Machine Data Analytics, please read our press release.

“In God we trust, all others must bring data.” – W. Edward DenningsOver the years, we’ve all internalized the concept that data driven decision making is key. We’ve watched as digital businesses like AirBnB, Uber, and Amazon far outpace their competitors in market-share and profitability. They do this by tapping into continuous intelligence: they use the data generated as users interact with their applications to provide customized experiences – allowing them to learn and adapt quickly to where their users want to be.I had always imagined decision-making at these companies to be kind of like the stock photo; a well-heeled executive looking at an interesting chart and experiencing that moment of brilliant insight that leads to a game-changing business decision.The reality though, is that it was never as simple as that. It was hard work. It was not one key decision made by one key executive. It was several hundreds of small every-day decisions made across all levels in the organization by all kinds of individuals, that slowly but surely inched the company across the finish-line of success and sustains them today in their continued growth. The better equipped employees were with the relevant data, the better they could execute in their roles.At most companies, the decision to be data-driven is simple, actually getting to that state, not so much. Conversations might go something like this:“We should be more data-driven!”“Yeah!”“What data do we need?”“Depends, what are we trying to solve?”“Where’s the data?”“How do we get to it?”“What do we do once we have it?”“How can we share all this goodness?”At Sumo Logic, we’ve already cracked the hard fundamental problems of getting to the data and being able to ask meaningful questions of that data. We support a vast and scalable set of collection tools and at Sumo’s core is very powerful machine data analytics platform that allows our customers to query their logs and metrics data to quickly derive insights and address impactful operational and security issues.We’re working our way up the problem chain. Now that we can easily get to the data and analyze it – how do we use it as an organization? How can our machine data tell our infrastructure engineers about the health of the service, the support team about performance against SLAs, help PMs to understand user adoption and find a way to summarize all of this into a format that can be presented to executives and key stakeholders?To solve this, we recently introduced the concept of public dashboards; data-rich dashboards that could be shared outside of Sumo and across the organization. This helped expose data to users who relied on it to make decisions, but who were far removed from the actual applications and infrastructure that generated it.Now, we’re tackling a deeper issue: how do users and teams collaborate on data analysis within Sumo? How do they learn from each other about what kind of metrics other teams collect, what are best practices and how do they learn and grow exponentially as an organization as they become empowered with this data?We plan to solve this, later this year, by allowing users to share their dashboards, log searches and metrics queries with other users and roles in their organization. Teams can collaborate with each other and control with granularity how different users and roles in the organization can edit or view a dashboard or a search. Administrators can efficiently organize and make accessible the content that’s most relevant for a particular group of people.We’ve embraced the concept of Universal Access to mean accessibility to Sumo and more importantly, the data in Sumo, to all users regardless of their skill or experience levels with Sumo. We’ve redesigned Sumo to be contextual and intuitive with the introduction of simpler navigation and workflows. Current users will appreciate the new types of content that can be opened in tabs – such as dashboards, log searches, metrics queries and live tail – and the fact that these tabs are persistent across login sessions. New users will have a smoother onboarding experience with a personalized homepage.To check out the new UI (beta) & learn more about how Sumo Logic can help your organization be more data-driven, sign-up today!

There is no denying it – the digital business transformation movement is real, and the time for this transformation is now. When, according to survey from Bain & Company, 48 of 50 Fortune Global companies have publicly announced plans to adopt public cloud, it is clear that there are no industries immune from this disruption. We are seeing traditional industries such as insurance, banking, and healthcare carving out labs and incubators that bring innovative solutions to market, and establish processes and platforms to help the rest of the organization with their evolution. For large enterprises it is critical that they manage the challenges of moving to public cloud, while satisfying the needs of a diverse set of internal customers. They need to support a variety of development languages, multiple deployment tool chains, and a mix of data centers and multiple public cloud vendors. Because these are long term strategies that involve considerable investment, they are concerned about long-term vendor lock-in, and are being proactive about developing strategies to mitigate those risks. These organizations are looking toward cloud-neutral commercial vendors to help them migrate to the cloud, and have consistency in how they deploy and manage their applications across heterogeneous environments. These enterprises are increasingly turning to Pivotal Cloud Foundry® to help them abstract their app deployments from the deployment specifics of individual cloud platforms, and maintain their ability to move apps and workloads across cloud providers when the time comes. Effective DevOps Analytics for the Modern Application The migration of enterprise workloads to the cloud, and the rise of public cloud competition, is driving the demand for Sumo Logic as a cloud-native platform for monitoring and securing modern applications. Pivotal Cloud Foundry enables users to abstract the underlying plumbing necessary to deploy, manage and scale containerized cloud native applications. This benefits developers by greatly increasing their productivity and ability to launch applications quickly. Such an environment also exposes a broader set of operational and security constructs that are useful to track, log and analyze. However it can also be more complicated to diagnose performance issues with decoupled architectures and composable micro-services. Full stack observability and the ability to trace all the apps and services together are critical to successful cloud deployments. Observability of decoupled architectures with composable services requires the ability to trace all layers of the stack With Pivotal Cloud Foundry and tools from Sumo Logic, an organization can have an observable, enterprise-class platform for application delivery, operations, and support across multiple public cloud providers and on-premises data centers. Beyond platform operations, Cloud Foundry customers want to enable their app teams to be self sufficient, and promote an agile culture of DevOps. Often, with legacy monitoring and analytics tools, the operations team will have access to the data, but they can’t scale to support the application teams. Or, the apps team may restrict access to their sensitive data, and therefore not support the needs of the security and compliance team. Sumo Logic believes in democratized analytics. This means that this massive flow of highly valuable data, from across the stack and cloud providers, should be available to everyone that can benefit from it. This requires the right level of scale, security, ubiquity of access, and economics that only Sumo Logic can provide. Sumo Logic & Pivotal Cloud Foundry Partnership Through our collaboration with Pivotal®, Sumo Logic has developed an app for Pivotal Cloud Foundry, as well as an easy-to-deploy integration with Pivotal Cloud Foundry Loggregator. A customer ready Beta of the “Sumo Logic Nozzle for PCF”, is available now as an Operations Manager Tile for Pivotal Cloud Foundry, available for download in BETA from Pivotal Network. Sumo Logic Tile Installed in the PCF Ops Manager If you are already using or evaluating Pivotal Cloud Foundry you can get started with operational and security analytics in a manner of minutes. With this integration, all of the log and metrics data collated by Cloud Foundry Loggregator will be streamed securely to the Sumo Logic Platform. For deployments with security and compliance requirements, Sumo Logic’s cloud-based service is SOC 2, HIPAA, and PCI-compliant. The Sumo Logic integration for Pivotal Cloud Foundry will be available in the App Library soon. If you would like early access, please contact your account team. Sumo Logic App for Pivotal Cloud Foundry highlight key Pivotal data and KPI’s The Sumo Logic App for Pivotal Cloud Foundry highlights key Pivotal data and KPIs. Sumo Logic’s App for Cloud Foundry operationalizes Pivotal Cloud Foundry’s monitoring best practices for you, and provides a platform for you to build upon to address your unique monitoring and diagnostic requirements.

Earlier today we announced a revolutionary set of new platform services and capabilities. As such, I wanted to provide more context around this and our strategy. While this new announcement is very exciting, we have always been pushing the boundaries to continuously innovate in order to remove the complexity and cost associated with getting the most value out of data. Whether it’s build-it-yourself open source toolkits or legacy on-premise commercial software packages, the “data tax” associated with these legacy licensing models, let alone the technology limitations have prevented universal access for all types of data sources and more importantly users. This strategy and the innovations we announced address the digital transformation taking place industry- wide, led by the mega trends of cloud computing adoption, DevSecOps and the growth of machine data. For example, IDC recently forecasted public cloud spending to reach 203.4 billion by 2020, while Bain & Company’s figure is nearly twice that at $390 billion. Whatever number you believe, the bottom line is that public cloud adoption is officially on a tear. For example, according to Bain, 48 of the 50 Fortune Global companies have publicly announced cloud adoption plans to support a variety of needs. In the world of DevSecOps, our own Modern App Report released last November substantiated the rise of a new Modern Application Stack, replete with new technologies, such as Docker, Kuburnetes, NoSQL, S3, Lambda, and CloudFront that are seriously challenging the status quo of traditional on-premise standards from Microsoft, HP, Oracle, and Akamai. However, the most significant, and arguably the most difficult digital transformation trend for businesses to get their arms around is the growth of machine data. According to Barclay’s Big Data Handbook, machine data will account for 40 percent of all data created by 2020, reaching approximately 16 zetabytes. (To put that number in perspective, 16 zetabytes is equivalent to streaming the entire Netflix catalogue 30 million times!) Since machine data is the digital blueprint of digital business, it’s rich source of actionable insights either remains locked away or difficult-to-extract at best because of expensive, outdated, disparate tooling that limits visibility, impedes collaboration and slows down the continuous processes required to build, run, secure and manage modern applications. Seven years ago, Sumo Logic made a big bet: disrupt traditional Big Data models with their lagging intelligence indicators by pursuing a different course: a real-time, continuous intelligence platform strategy better equipped to support the continuous innovation models of transformational companies. Now this need is becoming more critical than ever as the laggards make their shift and cloud computing goes mainstream, which not only will drive those market data numbers even higher, but also put the squeeze on the talent necessary to execute the shift. That’s why Sumo Logic’s vision, to “Democratize Machine Data”, now comes to the forefront. To truly enable every company to have the power of real-time, machine data analytics, we believe the current licensing, access and delivery models surrounding machine data analytics are also ripe for disruption. Our announcement today provides essential new innovations – ones that are only achievable because of our market-leading, multi-tenant, cloud-native platform – that remove economic, access and visibility barriers holding companies back from reaching their full data-insight potential. They are: Sumo Cloud Flex: a disruptive data analytics economic model that enables maximum flexibility to align data consumption and use with different use cases, and provide universal access by removing user-based licensing. While this was purpose-built and optimized for the massive untapped terabyte volume data sets, it’s also applicable to the highly variable data sets. Unified Machine Data Analytics: New, native PaaS and IaaS-level integrations to our cloud-native, machine data analytics platform to support data ingest from a variety of cloud platforms, apps and infrastructures. These additions will enable complete visibility and holistic management across the entire modern application and infrastructure stack. Universal Access: New experience capabilities such as a contextual and intuitive user interface to improve user productivity and public dashboards, and improved content sharing for faster collaboration with role-based access controls (RBAC). With this innovation, machine data insights are easier to access for non-technical, support services and business users. Over time, we predict ease-of-use initiatives like this will be one of the drivers to help close the current data scientist/security analyst talent gap. With our new innovations announced today, plus more coming later in the year, Sumo Logic is positioned to become the modern application management platform for digital transformation, delivered to our customers as a low TCO, scalable, secure service. That’s because machine data analytics will be the layer that provides complete visibility to manage the growing complexity of cloud-based, modern applications, which is sorely needed today and in the future. As the leading, cloud-native machine data analytics service on AWS, we service more than 1500 customers, from born-in-the-cloud companies like Salesforce, Twilio and AirBnB to traditional enterprises, such as Marriott, Alaska Airlines and Anheuser-Busch. Our platform system on average analyzes 100+ petabytes of data, executes more than 20 million searches, and queries 300+ trillion records every day. While these numbers seem massive, the numbers keep growing and yet we are only at the beginning of this massive opportunity. Other machine data analytics options such as cobbling a solution together with old technologies, or trying to build it on your own fall short because they don’t address the fundamental problem – machine data will just keep growing. To address this, the data layer must be re-architected – similar to the compute layer – to utilize the power of true distributed computing to address a problem that is never over – the volume, velocity and variety of machine data growth – and to do so in a way that meets the speed, agility and intelligence demands of digital business. You can’t put old, enterprise application architectures on top of the cloud and expect to be prepared. Sumo Logic’s ability to flexibly manage and maximize data and compute resources – the power of multi-tenant, distributed system architecture – across 1500+ customers means our customers have the ability to maximize their data insight potential to realize the holy grail of being real-time, data-driven businesses. We invite you to experience the power of the Sumo Logic for free. As always, I look forward to your feedback and comments.

No no no NOOOOO NOOOOOOOO… One could hear Mike almost yelling while staring at his computer screen. Suddenly Mike lost the SSH connection to one of the core billing servers. He was in the middle of the manual backup before he could upgrade the system with the latest monkey patch. Mike, gained a lot of visibility and the promotion, after his last initiative of migrating ELK Stack to SaaS-based machine data analytics platform, Sumo Logic. He improved MTTI/MTTR by 90% and uptime of the log analytics service by 70% in less than a month time. With the promotion, he was in-charge of the newly formed site reliability engineering (SRE) team. He had 4 people reporting to him. It was a big deal. This was his first major project after the promotion and he wanted to ensure that everything goes well. But just now, something happened to the billing server and Mike had a bad feeling about it. He waited for few minutes to check if the billing server will start responding again. It has happened before, where SSH client used to temporarily lose the connection to the server. The root cause of the connection loss was the firewall in the corporate headquarters. They had to upgrade the firewall to fix this issue. Mike was convinced that it’s not the firewall, but something else has happened to the billing server, and this time around there was a way to confirm his hunch. To view what happened to the billing server he runs a query on Sumo Logic. “_SourceHost=billingserver AND “shut*” He quickly realizes that server was rebooted. He broadens the search to +-5 minutes range from the above log message and identifies that disk was full. He added some more disk to the existing server to ensure that billing server does not restart because of the lack of hard drive space. However, Mike had no visibility into host metrics such as CPU, Hard Disk, and Memory usage. He needed a solution to gather host and custom metrics. He couldn’t believe how the application was managed without these metrics. He knew very well that Metrics must be captured to get visibility into system health. So he reprioritized his Metrics project over making ELK stack and entire infrastructure PCI compliant. Stage 1: Installing Graphite After a quick search, he identifies Graphite as one of his options. He had a bad taste in his mouth related to ELK, which cost him arm and a leg for just a search feature. This time though, he thought it will be different. Metrics were only 12 bytes in size! He thought how hard can it be to store 12 Bytes of data for 200 Machines? He chose Graphite as their open-source host metrics system. He downloads and installs the latest graphite on AWS t2.medium @ $0.016 USD per hour, Mike can get 4GB RAM with 2 vCPU. In less than $300 USD Mike is ready to test his new Metrics system. Graphite has three main components. Carbon, whisper and Graphite Web. Carbon listens on a TCP port and expects time series metrics. Whisper is a flat-file database while Graphite Web is a Django application that can query Carbon-cache and Whisper. He installs all of this on one single server. The logical architecture looks some like in Figure 1 below. Figure 1: Simple Graphite Logical Architecture on a Single Server Summary: At the end of stage 1, Mike had a working solution with a couple of servers on AWS. Stage 2: New Metrics stopped updating – the First issue with Graphite On a busy day, suddenly new metrics were not shown in the UI. This was the first time ever after few months of operations that Graphite was facing issues. After careful analysis, it was clear that metrics were getting written to the whisper files. Mike, thought for a second and realized that whisper pre-allocates the disk space to whisper files based on the configuration in carbon.conf file. To make it more concrete, 31.1 MB is pre-allocated by whisper for 1 metric collected every 1 second for one host and retained for 30 days. Total Metric Storage = 1 Host* 1 metric/sec* 60 sec *60 mins *24 hrs *30 days retention. He realized that he might have run out of disk space and sure enough, that was the case. He doubled the disk space, restarted the graphite server and now new data points started showing up. Mike was happy that he was able to resolve the issue before it got escalated. However, his mind started creating “What-If” scenarios. What if the application he is monitoring goes down exactly at the same time Graphite gives up? He parks that scenario in the back of his head and goes back to working on other priorities. Summary: At the end of stage 2, Mike already had incurred additional storage cost and ended up buying EBS Provisioned IoPS volume. SSD would have been better but this is the best he could do with the allocated budget. Stage 3: And Again New Metrics Stopped Updating On Saturday night 10 PM there was a marketing promotion. Suddenly it went viral and a lot of users logged into the application. Engineering had auto-scaling enabled on its front end while Mike had ensured that new images will automatically enable StatsD. Suddenly the metrics data points per minute (DPM) grew significantly and way above average DPM. Mike, had no idea about these series of events. The ticket with only information he received was “New Metrics are not showing up, AGAIN!” He quickly found out the following. MAX_UPDATES_PER_SECOND which determines how many updates you must have per second was increasing gradually also MAX_CREATES_PER_MINUTE was at its max. Mike quickly realized the underlying problem. It was the I/O problem causing the server to crash because graphite server is running out of memory. Here is how he connects the dots. Auto-scaling kicks in and suddenly 800 servers start sending the metrics to graphite. This is four times the load than the average number of hosts running at any given time. This quadruples the metrics ingested as well. Graphite configurations MAX_UPDATE_PER_SECOND and MAX_CREATES_PER_MINUTE reduces the load on disk I/O but it has an upstream impact. Suddenly carbon-cache starts using more and more memory. Considering “MAX_CACHE_SIZE” was set to infinite, Carbon-cache kept storing the metrics in the memory that was waiting to be written to whisper/disk. As carbon-cache process ran out of memory it crashed and sure enough, metrics stopped getting updated. So Mike added EBS volume with provisioned I/O and upgraded the server to M3 Medium instead of t2. Summary: At the end of stage 3, Mike has already performed two migrations. First, by changing the hard-drive he had to transfer the graphite data. Second, after changing the machine he had to reinstall and repopulate the data. Not to mention this time he has to reconfigure all the clients to send metrics to this new server. Figure 2: Single Graphite M3 Medium Server after Stage 3 Stage 4: Graphite gets resiliency, but at what cost? Mike from his earlier ELK experience learned one thing, that he cannot have any single point of failures in his data ingest pipeline at the same time he has to solve for the Carbon relay crash. Before anything happens he has to resolve the single point of failure in the above architecture and allocate more memory to carbon-relay. He decided to replicate similar graphite deployment in a different availability zone. This time he turns on the replication in the configuration file and creates the architecture as below. The architecture below ensures replication and adds more memory to carbon-relay process so that it can hold metrics in memory while whisper is busy writing them to the disk. Summary: At the end of stage 4, Mike has resiliency with replication and more memory for Carbon relay process. This change has doubled the Graphite cost from the last time. Figure 3: Two Graphite M3 Medium Server with replication after Stage 4 Stage 5: And another one bites the dust… Yet another Carbon Relay issue. Mike was standing in the line for the hot breakfast. At this deli, one has to pay first and then get their breakfast. He saw a huge line at the cashier. The cashier seemed to be a new guy. He was slow and the line was getting longer and longer. It was morning and everyone wanted to quickly get back. Suddenly Mike’s brain started drawing an analogy. He thought carbon-relay as a cashier, person serving the breakfast as a carbon-cache and chef as a whisper. The chef takes the longest time because he has to cook the breakfast. Suddenly he realizes the flaw in his earlier design. There is a line port (TCP 2003) and a Pickle port(TCP 2004) on Carbon-relay. Every host is configured to throw metrics at those ports. The moment Carbon-Relay gets saturated there is no way to scale them up without adding new servers and some network reconfigurations and hosts configuration changes. To avoid that kind of disruptions, he quickly comes up with a new design he calls it relay-sandwich. He separates out HA proxy on its dedicated server. Carbon-relay also gets its own server so that it can scale horizontally without changing the configuration at the host level. Summary: Each Graphite instance has four servers and total of 8 servers across two graphite instances. At this point, the system is resilient with headroom to scale carbon-relay. Figure 4: Adding more servers with HA Proxy and Carbon Relay Stage 6: Where is my UI? As you all must have noticed this is just the backend architecture. Mike was the only person running the show but if he wants more users to have access to this system, he must scale front end as well. He ends up installing Graphite-Web and the final architecture becomes as shown in figure 5. Summary: Graphite evolved from single server to 10 machine Graphite cluster instance managing metrics only for the fraction of their infrastructure. Figure 5: Adding more servers with HA Proxy and Carbon Relay Conclusion: It was Deja-vu for Mike. He had seen this movie before with ELK. After 20 servers in with Graphite, he was just getting started. He quickly realizes that if he enables custom metrics he has to double the size of his graphite cluster. Currently, the issue is graphite has metrics indicating “What” is wrong with the system while with Sumo Logic platform with correlated logs and metrics not only indicates “what” is wrong with the system but also indicates “why” something is wrong. Mike, turns on Sumo Logic metrics on the same collectors collecting logs and gets correlated logs and metrics on Sumo Logic platform. Best part he is not on the hook to manage the management system.

Monitoring application metrics and other telemetry from production environments is important for keeping your app stable and healthy. That you know. But app telemetry shouldn’t start and end with production. Monitoring telemetry during builds is also important for application quality. It helps you detect problems earlier on, before they reach production. It also allows you to achieve continuous, comprehensive visibility into your app. Below, we’ll take a look at why monitoring app telemetry during builds is important, then discuss the specific types of data you should collect at build time. App Telemetry During Builds By monitoring application telemetry during the build stage of your continuous delivery pipeline, you can achieve the following: Early detection of problems. Telemetry statistics collected during builds can help you to identify issues with your delivery chain early on. For example, if the number of compiler warnings is increasing, it could signal a problem with your coding process. You want to address that before your code gets into production. Environment-specific visibility. Since you usually perform builds for specific types of deployment environments, app telemetry from the builds can help you to gain insight into the way your app will perform within each type of environment. Here again, data from the builds helps you find potential problems before your code gets to production. Code-specific statistics. App telemetry data from a production environment is very different from build telemetry. That’s because the nature of the app being studied is different. Production telemetry focuses on metrics like bandwidth and active connections. Build telemetry gives you more visibility into your app itself—how many internal functions you have, how quickly your code can be compiled, and so on. Continuous visibility. Because app telemetry from builds gives you visibility that other types of telemetry can’t provide, it’s an essential ingredient for achieving continuous visibility into your delivery chain. Combined with monitoring metrics from other stages of delivery, build telemetry allows you to understand your app in a comprehensive way, rather than only monitoring it in production. Metrics to Collect If you’ve read this far, you know the why of build telemetry. Now let’s talk about the how. Specifically, let’s take a look at which types of metrics to focus on when monitoring app telemetry during the build stage of your continuous delivery pipeline. Number of environments you’re building for. This might seem so basic that it’s not worth monitoring. But in a complex continuous delivery workflow, it’s possible that the types of environments you target will change frequently. Tracking the total number of environments can help you understand the complexity of your build process. It can also help you measure your efforts to stay agile by maintaining the ability to add or subtract target environments quickly. Total lines of source code. This metric gives you a sense of how quickly your application is growing—and by extension, how many resources it will consume, and how long build times should take. The correlation between lines of source code and these factors is rough, of course. But it’s still a useful metric to track. Build times. Monitoring how long builds take, and how build times vary between different target environments is another way to get a sense of how quickly your app is growing. It’s also important for keeping your continuous delivery pipeline flowing smoothly. Code builds are often the most time-consuming process in a continuous delivery chain. If build times start increasing substantially, you should address them in order to avoid delays that could break your ability to deliver continuously. Compiler warnings and errors. Compiler issues are often an early sign of software architecture or coding issues. Even if you are able to work through the errors and warnings that your compiler throws, monitoring their frequency gives you an early warning sign of problems with your app. Build failure rate. This metric serves as another proxy for potential architecture or coding problems. Code load time. Measuring changes in the time it takes to check out code from the repository where you store it helps you prevent obstacles that could hamper continuous delivery. Monitoring telemetry during the build stage of your pipeline by focusing on the metrics outlined above helps you not only build more reliably, but also gain insights that make it easier to keep your overall continuous delivery chain operating smoothly. Most importantly, they help keep your app stable and efficient by assisting you in detecting problems early and maximizing your understanding of your application.

We’re excited to announce our biggest user experience overhaul as Sumo Logic enters its 8th year. Here’s quick list of some amazing things in the new UI. 1. An integrated workspace Everything now opens in tabs. This makes workflows like drilling down into dashboards, switching between log searches, or jumping between Metrics and Log Searching much smoother. The workspace remembers your state so when you log back into Sumo Logic, it fires up your tabs from the previous session. An Integrated Workspace 2. Quick access to your content Did you know about Sumo Logic library? It’s front and center now so you can quickly access your saved content and content shared with you. If you find yourself running the same searches and opening the same dashboards over and over again, you can now quickly launch them from the Recent tab. Quick access to content 3. Sumo Home Do you feel like a deer in headlights when you log in and see the blinking cursor on the search page? Not anymore! Sumo Home gives you a starting point full of useful content. Future improvements will let you personalize this page for your unique workflows. 4. A modern, cleaner, more consistent interface A fresh set of icons, an updated content library, and a tabbed, browser-like behavior are some of the many visual upgrades we did to get Sumo Logic ready for 2017. A modern interface 5. A beautiful App Catalog The App Catalog was redesigned from the ground up and now gives you a visual way of browsing through prebuilt content to help you get started faster and get more out of your logs. 6. Distraction-free mode Sometimes you need all the space you can get while troubleshooting. You can collapse the left navigation and pop out Dashboard tabs to give more real estate to your data. Distraction Free Mode 7. The Back Button works Hitting your browser’s back button will not log you out of Sumo Logic anymore, thanks to a smart UI routing. We solved one of biggest user pet peeves Check out the redesign! If you have any feedback on the redesign, please feel free to reach out to us at [email protected] or leave us comments directly on the Home Page.

Mesosphere DC/OS (Data Center Operating System) lets you manage your data center as if it were a single powerful computer. It separates the infrastructure logic from the application logic, and makes it easy to manage distributed systems. Considering DC/OS is meant for highly scalable, distributed systems, logging and monitoring plays a key role in day-to-day operations with DC/OS. In this post, we’ll take a look at the goals of logging with DC/OS, and how you can set up DC/OS logging with Sumo Logic.Why Mesosphere DC/OS Clusters Need LoggingWhen you work with Mesosphere DC/OS, you typically have hundreds, if not thousands of nodes that are grouped in clusters. Tasks are executed on these nodes by Mesos “agent” instances, which are controlled by “master” instances. By grouping the nodes in clusters, DC/OS ensures high availability so that if any node or cluster fails, its workload is automatically routed to the other clusters. DC/OS uses two scheduling tools—Chronos for scheduled tasks like ETL jobs, and Marathon for long-running tasks like running a web server. Additionally, it includes app services like Docker, Cassandra, and Spark. DC/OS supports hybrid infrastructure, allowing you to manage bare metal servers, VMs on-premises, or cloud instances, all from a single pane of glass. Together, all of these components make for a complex system that needs close monitoring.There are two key purposes for collecting and analyzing DC/OS logs. The first is debugging. As new tasks are executed, DC/OS makes decisions in real time on how to schedule these tasks. While this is automated, it needs supervision. Failover needs logging so you can detect abnormal behavior early on. Also, as you troubleshoot operational issues on a day-to-day basis, you need to monitor resource usage at a granular level, and that requires a robust logging tool.Second, for certain apps in enterprises, compliance is a key reason to store historic logs over a long period of time. You may need to comply to HIPAA or PCI DSS standards.Viewing raw logs in DC/OSDC/OS services and tasks write stdout and stderr files in their sandboxes by default. You can access logs via the DC/OS CLI or the console. You can also SSH into a node and run the following command to view its logs:$ journalctl -u "dcos-*" -bWhile this is fine if you’re running just a couple of nodes, once you scale to tens or hundreds of nodes, you need a more robust logging tool. That’s where a log analysis tool like Sumo Logic comes in.Sharing DC/OS logs with Sumo LogicDC/OS shares log data via a HTTP endpoint which acts as a source. The first step to share Mesosphere DC/OS logs with Sumo Logic is to configure a HTTP source in Sumo Logic. You can do this from the Sumo Logic console by following these steps. You can edit settings like timestamp, and allow multi-line messages like stack traces.Your data is uploaded to a unique source URL. Once uploaded, the data is sent to a Sumo Logic collector. This collector is hosted and managed by Sumo Logic, which makes setup easy, and reduces maintenance later. The collector compresses the log data, encrypts it, and sends it to the Sumo Logic cloud, in real time.During this setup process, you can optionally create Processing rules to filter data sent to Sumo Logic. Here are some actions you can take on the logs being shared:Exclude messagesInclude messagesHash messagesMask messagesForward messagesThese processing rules apply only to data sent to Sumo Logic, not the raw logs in DC/OS.It may take a few minutes for data to start showing in the Sumo Logic dashboard, and once it does, you’re off to the races with state-of-the-art predictive analytics for your log data. You gain deep visibility into DC/OS cluster health. You can setup alerts based on the log data and get notifications when failed nodes reach a certain number, or when a high priority task is running too slow, or if there is any suspicious user behavior. Whether it’s an overview, or a deep dive to resolve issues, Sumo Logic provides advanced data analysis that builds on the default metrics of DC/OS. It also has options to archive historic log data for years so you can comply with various security standards like HIPAA or PCI DSS.DC/OS is changing the way we view data centers. It transforms the data center from hardware- centric to software-defined. A comprehensive package, it encourages hybrid infrastructure, prevents vendor lock-in, and provides support for container orchestration. DC/OS is built for modern web scale apps. However, it comes with a new set of challenges with infrastructure and application monitoring. This is where you need a tool like Sumo Logic so that you not only view raw log data, but are also able to analyze it and derive insights before incidents happen.About the AuthorTwain began his career at Google, where, among other things, he was involved in technical support for the AdWords team. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications.Mesosphere DC/OS Logging with Sumo Logic is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

When Sumo Logic receives metrics data, we put those metrics datapoints into a Kafka queue for processing. To help us distribute the load, that Kafka queue is broken up into multiple Kafka Topic Partitions; we therefore have to decide which partition is appropriate for a given metrics datapoint. Our logic for doing that has evolved over the last year in a way that spread the decision logic out over a few different classes; I thought it was time to put it all in one place. My initial version had an interface like this: def partitionFor(metricDefinition: MetricDefinition): TopicPartition As I started filling out the implementation, though, I began to feel a little bit uncomfortable. The first twinge was when calculating which branch to go down in one of the methods: normally, when writing code, I try to focus on clarity, but when you’re working at the volumes of data that Sumo Logic has to process, you have to keep efficiency in mind when writing code that is evaluated on every single data point. And I couldn’t convince myself that one particular calculation was quite fast enough for me to want to perform it on every data point, given that the inputs for that calculation didn’t actually depend on the specific data point. So I switched over to a batch interface, pulling that potentially expensive branch calculation out to the batch level: class KafkaPartitionSelector { def partitionForBatch(metricDefinitions: Seq[MetricDefinition]): Seq[TopicPartition] = { val perMetric = calculateWhetherToPartitionPerMetric() metricDefinitions.map { metric => partitionFor(metric, perMetric) } } private def partitionFor(metricDefinition: MetricDefinition, perMetric: Boolean): TopicPartition = { if (perMetric) { ... } else { ... } } } That reduced the calculation in question from once per data point to once per batch, getting me past that first problem. But then I ran into a second such calculation that I needed, and a little after that I saw a call that could potentially translate into a network call; I didn’t want to do either of those on every data point, either! (The results of the network call are cached most of the time, but still.) I thought about adding them as arguments to partitionFor() and to methods that partitionFor() calls, but passing around three separate arguments would make the code pretty messy. To solve this, I reached a little further into my bag of tricks: this calls for a Method Object. Method Object is a design pattern that you can use when you have a method that calls a bunch of other methods and needs to pass the same values over and over down the method chain: instead of passing the values as arguments, you create a separate object whose member variables are the values that are needed in lots of places and whose methods are the original methods you want. That way, you can break your implementation up into methods with small, clean signatures, because the values that are needed everywhere are accessed transparently as member variables. In this specific instance, the object I extracted had a slightly different flavor, so I’ll call it a “Batch Method Object”: if you’re performing a calculation over a batch, if every evaluation needs the same data, and if evaluating that data is expensive, then create an object whose member variables are the data that’s shared by all batches. With that, the implementation became: class KafkaPartitionSelector { def partitionForBatch(metricDefinitions: Seq[MetricDefinition]): Seq[TopicPartition] = { val batchPartitionSelector = new BatchPartitionSelector metricDefinitions.map(batchPartitionSelector.partitionFor) } private class BatchPartitionSelector { private val perMetric = calculateWhetherToPartitionPerMetric() private val nextExpensiveCalculation = ... ... def partitionFor(metricDefinition: MetricDefinition): TopicPartition = { if (perMetric) { ... } else { ... } } ... } } One question that came up while doing this transformation was whether every single member variable in BatchPartitioner was going to be needed in every batch, no matter what the feature flag settings were. (Which was a potential concern, because they would all be initialized at BatchPartitioner creation time, every time this code processes a batch.) I looked at the paths and checked that most were used no matter the feature flag settings, but there was one that only mattered in some of the paths. This gave me a tradeoff: should I wastefully evaluate all of them anyways, or should I mark that last one as lazy? I decided to go the route of evaluating all of them, because lazy variables are a little conceptually messy and they introduce locking behind the scenes which has its own efficiency cost: those downsides seemed to me to outweigh the costs of doing the evaluation in question once per batch. If the potentially-unneeded evaluation had been more expensive (e.g. if it had involved a network call), however, then I would have made them lazy instead. The moral is: keep Method Object (and this Batch Method Object variant) in mind: it’s pretty rare that you need it, but in the right circumstances, it really can make your code a lot cleaner. Or, alternatively: don’t keep it in mind. Because you can actually deduce Method Object from more basic, more fundamental OO principles. Let’s do a thought experiment where I’ve gone down the route of performing shared calculations once at the batch level and then passing them down through various methods in the implementation: what would that look like? The code would have a bunch of methods that share the same three or four parameters (and there would, of course, be additional parameters specific to the individual methods). But whenever you see the same few pieces of data referenced or passed around together, that’s a smell that suggests that you want to introduce an object that has those pieces of data as member variables. If we follow that route, we’d apply Introduce Parameter Object to create a new class that you pass around, called something like BatchParameters. That helps, because instead of passing the same three arguments everywhere, we’re only passing one argument everywhere. (Incidentally, if you’re looking for rules of thumb: in really well factored code, methods generally only take at most two arguments. It’s not a universal rule, but if you find yourself writing methods with lots of arguments, ask yourself what you could do to shrink the argument lists.) But then that raises another smell: we’re passing the same argument everywhere! And when you have a bunch of methods called in close proximity that all take exactly the same object as one of their parameters (not just an object of the same type, but literally the same object), frequently that’s a sign that the methods in question should actually be methods on the object that’s a parameter. (Another way to think of this: you should still be passing around that same object as a parameter, but the parameter should be called this and should be hidden from you by the compiler!) And if you do that (I guess Move Method is the relevant term here?), moving the methods in question to BatchParameters, then BatchParameters becomes exactly the BatchPartitionSelector class from my example. So yeah, Method Object is great. But more fundamental principles like “group data used together into an object” and “turn repeated function calls with a shared parameter into methods on that shared parameter” are even better. And what’s even better than that is to remember Kent Beck’s four rules of simple design: those latter two principles are both themselves instances of Beck’s “No Duplication” rule. You just have to train your eyes to see duplication in its many forms.

Learn to build Java microservices that process Dropwizard metrics with this step by step example that returns the shipping cost for packages based on weight.

I’ve recently received some questions about the AWS Application Load Balancer, what advantages it provides, and how to monitor it. AWS is already calling the original Elastic Load Balancer it’s ‘Classic’ Load Balancer, so if you’re anxious to understand why so many are using it over the Classic ELB, this post is for you.

Across all of the nation-state targeted attacks, insider thefts, and criminal enterprises that CrowdStrike® has investigated, one thing is clear: logs are extremely important. Event logs from individual computers provide information on attacker lateral movement, firewall logs show the first contact of a particular command and control domain, and Active Directory authentication logs build a timeline of user accounts moving throughout the network. Sounds great, right? It is, as long as all the logs are saved and searchable.

Ahhh DockerCon, the annual convention for khaki pant enthusiasts. Oh, wait, not that Docker. Last week DockerCon kicked off with 5500 Developers, IT Ops Engineers and enterprise professionals from across the globe. With the announcement of new features like LinuxKit and the Moby project, Docker is doubling down on creating tools that enable mass innovation while simplifying and accelerating the speed of the delivery cycle. Docker is starting to turn a corner, becoming a mature platform for creating mission-critical, Enterprise class applications. Throughout all of this, monitoring and visibility into your infrastructure continues to be critical to success. Current Trends In the world of containers, there are three trends we are seeing here at Sumo Logic. First, is the rapid migration to containers. Containers provide great portability of code and easier deployments. Second is the need for visibility. While migrating to containers have simplified the deployment process, it is definitely a double-edged sword. The ability to monitor your containers health, access the container logs and monitor the cluster on which your containers run is critical to maintaining the health of your application. The last trend is the desire to consolidate tools. You may have numerous tools helping you monitor your applications. Having multiple tools introduces “swivel chair” syndrome, where you have to switch back and forth between different tools to help diagnose issues as they are happening. You may start with a tool showing you some metrics on CPU and memory, indicating something is going wrong. Metrics only give you part of the visibility you need. You need to turn to your logs to figure out why this is happening. Monitoring Your Containers and Environment Sumo Logic’s Unified Logs and Metrics are here to help give you full visibility into your applications. To effectively monitor your applications, you need the whole picture. Metrics give you insights into what is happening, and logs give you insights into why. The union of these two allow you to perform root cause analysis on production issues to quickly address the problem. Sumo Logic can quickly give you visibility into your Docker containers leveraging our Docker Logs and Docker Stats sources. Our Docker application allows you to gain immediate visibility into the performance of your containers across all of your Docker hosts. Collecting Logs and Metrics From Kubernetes At DockerCon, we saw an increased use of Kubernetes and we received many questions on how to collect data from Kubernetes clusters. We have created a demo environment that is fully monitored by Sumo Logic. This demo environment is a modern application leveraging a micro-services architecture running in containers on Kubernetes. So how do we collect that data? Well, the below diagram helps illustrate that. We created a FluentD plugin to gather the logs from the nodes in the cluster and enrich them with metadata available in Kubernetes. This metadata can be pulled into Sumo Logic giving you increased ability to search and mine your data. We run the FluentD plugin as a Daemonset which ensures we collect all the logs for every node in our cluster. For metrics, we are leveraging Heapster’s ability to output to a Graphite sink and using a Graphite Source on our collector to get the metrics into Sumo Logic. Since Heapster can monitor metrics at the cluster, container and node level, we just need to run it and the collector as a deployment to get access to all the metrics that Heapster has to offer. What's Next What if you are not running in Kubernetes? In a previous post, we discussed multiple ways to collect logs from containers. However, due to the fast-paced growth in the container community, it is time to update that and we will add a post to dive deeper into that.

Overview When logging information about your operating system, services, network, or anything else, usually there’s predefined log structures in place by the vendor. There are times when there aren’t predefined logs created by some software or you have custom application logs from your own software. Without properly planning your log syntax you’ll be using, things can get messy and your data may lose its integrity to properly tell the story. These best practices for creating custom logs can be applied to most logging solutions. The 5 W's There are 5 critical components of a good log structure*: When did it happen (timestamp) What happened (e.g., error codes, impact level, etc) Where did it happen (e.g., hostnames, gateways, etc) Who was involved (e.g., usernames) Where he, she, or it came from (e.g., source IP) Additionally, your custom logs should have a standard syntax that is easy to parse with distinct delimiters, a key-value pair, or a combination of both. An example of a good custom log is as follows: 2017-04-10 09:50:32 -0700 - dan12345 - 10.0.24.123 - GET - /checkout/flight/ - credit.payments.io - Success - 2 - 241.98 This log message shows when it was performed, what was performed, where it happened in your system, who performed it, and where that user came from. It’s also structured cleanly with a space-dash-space as the delimiters of each field. Optionally, you can also have key-value pairs to assist with parsing: timestamp: 2017-04-10 09:50:32 -0700 - username: dan12345 - source_ip: 10.0.24.123 - method: GET - resource: /checkout/flight/ - gateway: credit.payments.io - audit: Success - flights_purchased: 2 - value: 241.98 Once you have your log syntax and what will be going into the logs, be sure to document this somewhere. You can document it by adding a comment at the top of each log file. Without documentation, you may forget or someone may not know what something like “2” or “241.98” represents (for this example, it means 2 flights in the checkout at a value of $241.98). You can document our log syntax as such: Timestamp - username - user_ip - method - resource - gateway - audit - flights_purchased - value In the second part of this three part series, we'll go into deeper details around timestamps and log content. In the final part, we'll go even deeper into log syntax and documentation. *Source: Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management. Chuvakin, A., Phillips, C., & Schmidt, K. J. (2013).

Diving Deeper Now that you have an overview for custom logs and what is involved in creating a good log practice from Part I of the series, it’s time to look further into what and why you should log in your system. This will be broken up into two parts. The first will cover timestamps and content, and the second part will cover syntax and documentation. Timestamp The first and most critical component of just about any log syntax is your timestamp - the “when”. A timestamp is important as it will tell you exactly the time an event took place in the system and was logged. Without this component, you’ll be relying on your log analysis solution to stamp it based upon when it came in. Adding a timestamp at the exact point of when an entry is logged will make sure you are consistently and accurately placing the entry at the right point in time for when it occurred. RFC 3339 is used to define the standard time and date format on the internet. Your timestamp should include year, month, day, hour, minute, second, and timezone. Optionally, you’ll also want to include sub-second depending on how important and precise you’ll need your logs to get for analysis. For Sumo Logic, you can read about the different formats for timestamps that are supported here - Timestamps, Time Zones, Time Ranges, and Date Formats. Log Content To figure out what happened, this can include data such as the severity of the event (e.g., low, medium, high; or 1 through 5), success or failure, status codes, resource URI, or anything else that will help you or your organization know exactly what happened in an event. You should be able to take a single log message or entry out of a log file and know most or all critical information without depending on logs’ file name, storage locations, or automatic metadata tagging from your tool. Your logs should tell a story. If they’re complex, they should also be documented as discussed later on. Bad Logs For a bad example, you may have a log entry as such: 2017-04-10 09:50:32 -0700 Success While you know that on April 10, 2017 at 9:50am MT an event happened and it was a success, you don’t really know anything else. If you know your system inside and out, you may know exactly what was successful; however, if you handed these logs over to a peer to do some analysis, they may be completely clueless! Good Logs Once you add some more details, the picture starts coming together: 2017-04-10 09:50:32 -0700 GET /checkout/flights/ Success From these changes you know on April 10th, a GET method was successfully performed on the resource /checkout/flights/. Finally, you may need to know who was involved and where. While the previous log example can technically provide you a decent amount of information, especially if you have tiny environment, it’s always good to provide as much detail since you don’t know what you may need to know for the future. For example, usernames and user IPs are good to log: 2017-04-10 09:50:32 -0700 dan12345 10.0.24.123 GET /checkout/flights/ Success Telling the Story Now you have even more details about what happened. A username or IP may individually be enough, but sometimes (especially for security) you’ll want as much as you can learn about the user since user accounts can be hacked and/or accessed from other IPs. You have just about enough at this point to really tell a story. To make sure you know whatever you can about the event, you also want to know where things were logged. Again, while your logging tool may automatically do this for you, there’s many factors that may affect the integrity and it’s best to have your raw messages tell as much as possible. To complete this, let’s add the gateway that logged the entry: 2017-04-10 09:50:32 -0700 dan12345 10.0.24.123 GET /checkout/flights/ credit.payments.io Success Now you know that this was performed on a gateway named credit.payments.io. If you had multiple gateways or containers, you may come to a point of needing to identify which to fix. Omitting this data from your log may result in a headache trying to track down exactly where it occurred. This was just 1 example of some basics of a log. You can add as much detail to this entry to make sure you know whatever you can for any insight you need now or in the future. For example, you may want to know other info about this event. How many flights were purchased? 2017-04-10 09:50:32 -0700 dan12345 10.0.24.123 GET /checkout/flights/ credit.payments.io Success 2 Where 2 is the amount of flights. What was the total value of the flights purchased? 2017-04-10 09:50:32 -0700 dan12345 10.0.24.123 GET /checkout/flights/ credit.payments.io Success 2 241.9 Where 2 is the amount of flights, and they totalled $241.98. Now that you know what to put into your custom logs, you should also consider deciding on a standard syntax throughout your logs. This will be covered in the last part of this series on best practices for creating custom logs.

Diving Even DeeperIn Part I there was a general overview of custom logs and Part II discussed timestamps and log content. At this point, you have a log that contains a bunch of important data to help you analyze it to gather useful information about your systems. In this final part of this series, you’ll learn about how to organize the data in your logs and how to make sure you properly document it.Log SyntaxYou may have the most descriptive and helpful data in your logs, but it can be very difficult to analyze your logs if you don’t have a defined and structured syntax. There are generally 2 ways to go about structuring your logs.Key-ValueWhen it comes to log analysis and parsing your logs, a key-value pair may be the simplest and allow for the most readable format. In our previous example, it may not be the most human-readable format and it may be a little more difficult to find anchors to parse against.You can change the message to be easier to read by humans and easier to parse in a tool like Sumo Logic:timestamp: 2017-04-10 09:50:32 -0700, username: dan12345, source_ip: 10.0.24.123, method: GET, resource: /checkout/flights/, gateway: credit.payments.io, audit: Success, flights_purchased: 2, value: 241.98You can take it a step further and structure your logs in a JSON format:{ timestamp: 2017-04-10 09:50:32 -0700,username: dan12345,source_ip: 10.0.24.123,method: GET,resource: /checkout/flights/,gateway: credit.payments.io,audit: Success,flights_purchased: 2,value: 241.98,}In Sumo Logic, you have various ways to parse through this type of structure including a basic Parse operator on predictable patterns or even Parse JSON. While it is ideal to use some sort of key-value pairing, it is not always the most efficient as you’re potentially doubling the size of an entry that gets sent and ingested. If you have low log volume, this wouldn’t be an issue; however, if you are generating logs at a high rate, it can become very costly to have log entries of that size.This brings us to the other format, which are delimited logs.DelimitedDelimited logs are essentially the type of log you built in the previous examples. This means that it’s a set structure to your log format, and different content is broken up by some sort of delimiter.2017-04-10 09:50:32 -0700 dan12345 10.0.24.123 GET /checkout/flights/ credit.payments.io Success 2 241.98Because of how this example is structured, spaces are the delimiters. To an extent, this is perfectly reasonable. The problem this provides you when parsing is figuring out where fields start and end as you see with the timestamp, though it may be the most efficient and smallest size you can get for this log. If you need to stick with this format, you’ll probably be sticking to regular expressions to parse your logs. This isn’t a problem to some, but others regular expressions can understandably be a challenge.To try and reduce the need for regular expressions, you’ll want to use a unique delimiter. A space can sometimes be one, but it may require us to excessively parse the timestamp. You may want to use a delimiter such as dash, semicolon, comma, or another character (or character pattern) that you can guarantee will never be used in the data of your fields.2017-04-10 09:50:32 -0700 - dan12345 - 10.0.24.123 - GET - /checkout/flights/ - credit.payments.io - Success - 2 - 241.98A syntax like this will allow you to parse out the entire message with a space-dash-space ( - ) as your delimiter of the fields. Space-dash-space would make sure that the dashes in the timestamp are not counted as a delimiter.Finally, to make sure you don’t have an entry that can be improperly parsed, always make sure you have some sort of filler in place of any fields that may not have data.For example:2017-04-10 09:50:32 -0700 - dan12345 - 10.0.24.123 - GET - /checkout/flights/ - credit.payments.io - Failure - x - xFurthermore from the example, you know that the event was a failure. Because it failed, it didn’t have flight totals or values. To prevent needing additional parsers for not having those fields, you simply can replace those fields with something like an ‘x’. Note that if you’re running aggregates or math against a field that may typically be a number, you may require adding some additional logic to your search queries.DocumentationYou may have the greatest log structure possible, but without proper documentation it’s possible to forget why something was part of your logging structure or you may forget what certain fields represented. You should always have documented what your log syntax represents.Referring back to the previous log example:2017-04-10 09:50:32 -0700 - dan12345 - 10.0.24.123 - GET - /checkout/flights/ - credit.payments.io - Success - 2 - 241.98You can document your log syntax as such:Timestamp - username - user_ip - method - resource - gateway - audit - flights_purchased - valueThis log syntax can placed at the very start of the log file one time for future reference if necessary.ConclusionAt Sumo Logic, we regularly work with those who are new to logging and have many questions around how to get the most out of their logs. While you can start ingesting your logs and getting insights almost immediately, the information provided from the tool is only as good as the data we receive. Though most vendors do a good job in sticking to standard log structures with great data to get these insights, it’s up to you to standardize a custom created log.In this series, I set out to help you create logs that have relevant data to know as much as you can about your custom applications. As long as you stick to the “5 W’s”, you structure your logs in a standard syntax, and you document it, then you’ll be on the right track to getting the most out of Sumo Logic.Be sure to sign up for a free trial of Sumo Logic to see what you can do with your logs!

Operational and Security Tips, Tricks and Best Practices In Gartner’s Top 10 Strategic Technology Trends for 2016: Adaptive Security Architecture, they argued that “Security must be more tightly integrated into the DevOps process to deliver a DevSecOps process that builds in security from the earliest stages of application design.” We ultimately need to move to this model if we are going to be successful and continue to reduce the dwell time of cyber criminals who are intent on compromising our applications and data. But how do we get from this: To this? Easier said than done. To answer this question, I sat down with our CISO and IANS Faculty Member George Gerchow about what it means to implement and maintain a DevSecOps approach in the cloud – and what operational and security best practices should organizations follow to ensure success in their move to the cloud. Below is a transcript of the conversation. DevSecOps seems like a buzz word that everyone is using these days. What does DevSecOps really mean? George: It is really about baking security in from Day 1. When you’re starting to put new workloads in the cloud or have these green field opportunities identified, start changing your habits and your behavior to incorporate security in from the very beginning. In the past we used to have a hard shell soft center type approach to security and in the cloud there is no hardshell, and we don’t run as many internal applications anymore. Now we’re releasing these things out into the wild into a hostile environment so you gotta be secure since day 1. Your developers and engineers, you have to have people who think security first when they’re developing code, that is most important take away” What does it really mean when you say baking security in….or the term shifting left, which I am starting to hear our there? George: It is about moving security earlier into the conversation, earlier into the software development lifecycle. You need to get developers to do security training. I’m talking about code review, short sprints, understanding what libraries are safe to use, and setting up feature flags that will check code in one piece at a time. The notion of a full release is a thing of the past – individual components are released continually. There also needs to be a QA mindset of testing the code and micro services to break it and then, fix accordingly through your agile DevSecOps methodologies. Sumo Logic is a cloud native service running in AWS for over 7 years now – why did you decide to build your service in the cloud? Can you describe a bit about that journey, what was it like, what obstacles did you face, how did you overcome them? And lastly, what did you learn along the way? George: Our company founders came from HP Arcsight and new full well of the pain in managing the execution environment – the hardware and software provisioning, the large teams needed, the protracted time to roll out new services. The cloud enabled us to be agile, flexible, highly elastic, and do this all securely at scale – it is at a level that was just not possible if we chose an on-prem model. The simplicity and automation capabilities of AWS was hugely attractive. You start setting up load balancers to be able to leverage tools like Chef to be able to do manage machine patching – it gets easier – and then you can start automating things from the very beginning so I think it’s that idea of starting very simple and leveraging native services that cloud service providers give you you and then looking for the gaps. The challenge initially was that this is a whole new world out and then the bigger challenge became getting people to buy off on the fact that cloud is more secure. People just weren’t there yet. What does Sumo Logic’s footprint look like in AWS? George: 100PB+ of data that is analyzed daily, 10K EC2 instances on any given day, 10M keys under management! We have over 1,300 customers and our service is growing by leaps and bounds. At this stage, it is all about logos – you wanna bring people in and you can’t afford to have bad customer service because this is a subscription-based model. When you think about the scale that we have, it’s also the scale that we have to protect our data. Now the challenge of quadrupling that number every year is extremely difficult so you have a long term view when it comes to scalability of security. 10,000+ instances it’s a very elastic type environment and auditors really struggle with this. One of the things that i’m the most proud of…if you look at hundreds of petabytes processed and analyzed daily, thats insane…thats the value of being in the cloud. 10 million of keys under management…thats huge…really?? George: It’s a very unique way that we do encryption. It makes our customers very comfortable with the dual control models…that they have some ownership over the keys and then we have capability to vault the keys for them. We do rotate the keys every 24 hours. The customers end up with 730 unique key rings on their keys at the end of the year. It’s a very slick, manageable program. We do put it into a vault and that vault is encrypted with key encryption key (KEK). So What Tools and Technologies are you using in AWS? George: Elastic load balancers are at the heart of what we do…we set up those load balancers to make sure that nothing that’s threatening gets through…so that’s our first layer of defense and then use security groups and we use firewalls to be able to route traffic to the right places to make sure only users can access. We use file integrity monitoring and we happen to use host sec for that across every host and we manage and that gives us extreme visibility. We also leverage IDS and snort and those signatures across those boxes to detect any kind of signature based attacks. Everything we do in the cloud is agentless or on the host. When you’re baking security in you have it on ALL of your systems, spun up automatically via scripts. We also have a great partnership with Crowdstrike, where threat intelligence is baked into our platform to identify malicious indicators of compromise and match that automatically to our customers logs data – very powerful So how are you leveraging Sumo to secure your own service? Can you share some of the tips, tricks and best practices you have gleaned over the years? George: Leveraging apps like CloudTrail, now we are able to see when a event takes place who is the person behind the event, and start looking for the impact of the event. I’m constantly looking for authorization type events (looking at Sumo Dashboards). When it comes to compliance I have to gather evidence of who is in the security groups. Sumo is definitely in the center of everything that we do. We have some applications built also for PCI and some other things as well to VPC flow logs but it gives us extreme visibility. We have dashboards that we have built internally to manage the logs and data sources. It is extremely valuable once you start correlating patterns of behavior and unique forms of attack patterns across the environment. You need to be able to identify how does that change that you just made impact the network traffic and latency in my environment and pulling in things like AWS inspector…How did that change that you made have an impact on my compliance and security posture. You want to have the visibility but then measure the level of impact when someone does make a change and even more proactively I want to have the visibility when something new is added to the environment or when something is deleted from the environment. Natively in AWS, it is hard to track these things” How does the Sumo Logic technology stack you talked about earlier help you with Compliance? George: Being able to do evidence gathering and prove that you’re protecting data is difficult. We’re protecting cardholder data, healthcare data and a host of other PII from the customers we serve across dozens of industries. We pursue our own security attestations like PCI, CSA Star, ISO 27001, SOC 2 Type 2, and more. We do not live vicariously through the security attestations of AWS like too many organizations do. Also, encryption across the board. All of these controls and attestations give people a level of confidence that we are doing the right things to protect their data and there’s actual evidence gathering going on. Specifically with respect to PCI, we leverage Sumo Logic PCI apps for evidence gathering -nonstop- across CloudTrail, Windows and Linus servers. We built out those apps for internal use, but released them to the public at RSA. There are a lot of threat actors out there, from Cyber Criminals, Corporate Spies, Hacktivists and Nation States. How do you see the threat landscape changing wrt the cloud. Is the risk greater given the massive scale of the attack surface? If someone hacked into an account, could they cause more damage by pointing their attack at Amazon, from within the service, possibly affecting millions of customers? George: It all starts with password hygiene. People sacrifice security for convenience. It’s a great time for us to start leveraging single sign on and multi factor authentication and all these different things that need to be involved but at a minimum end users should use heavily encrypted passwords…they should not bring in their personal type application passwords into the business world…If you start using basic password hygiene since day 1, you’re gonna follow the best habits in the business world. The people who should be the most responsible are not…I look at admins and developers in this way…all the sudden you have a developer put their full blown credentials into a slack channel. So when you look out toward the future, wrt the DevSecOps movement, the phenomenal growth of cloud providers like AWS and Azure, Machine learning and Artificial Intelligence, the rise of security as code, ….What are your thoughts, where do you see things going, and how should companies respond? George: First off, for the organizations that aren’t moving out to the cloud, at one point or the other, you’re gonna find yourself irrelevant or out of business. Secondly, you’re going to find that that the cloud is very secure. You can do a lot using cloud-based security if you bake security in since day one and work with your developers…if you work with your team…. you can be very secure. The future will hold a lot of cloud-based attacks. User behavior analytics…I can’t no longer go through this world of security and have hard-coded rules and certain things that I’m constantly looking for with all these false positives. I have to be able to leverage machine learning algorithms to consume and crunch through that data. The world is getting more cloudy more workloads moving into the cloud, teams will be coming together…security will be getting more backed in into the process. How would you summarize everything? George: “You’re developing things, you wanna make sure you have the right hygiene and security built into it and you have visibility into that and that allows you to scale as things get more complex where things actually become more complex is when you start adding more humans into it and you have less trust but if you have that scalability and visibility from day one and a simplistic approach, it’s going to do a lot of good for you. Visibility allows you to make quick decisions and it allows you to automate the right things and ultimately you need to have visibility because it allows you to have the evidence that you need to be compliant to help people feel comfortable that you’re protecting your data in the right way. George Gerchow can be reached at https://www.linkedin.com/in/georgegerchow or @georgegerchow

Why do you need patterns for building a successful microservices architecture? Shouldn’t the same basic principles apply, whether you’re designing software for a monolithic or microservices architecture? Those principles do largely hold true at the highest and most abstract levels of design (i.e., the systems level), and at the lowest and most concrete levels (such as classes and functions). But most code design is really concerned with the broad range between those two extremes, and it is there that the very nature of microservices architecture requires not only new patterns for design, but also new patterns for reimagining existing monolithic applications. The truth is that there is nothing in monolithic architecture that inherently imposes either structure or discipline in design. Almost all programming languages currently in use are designed to enforce structure and discipline at the level of coding, of course, but at higher levels, good design still requires conscious adherence to methodologies that enforce a set of architectural best practices. Microservices architecture, on the other hand, does impose by its very nature a very definite kind of structural discipline at the level of individual resources. Just as it makes no sense to cut a basic microservice into arbitrary chunks, and separate them, it makes equally little sense to bundle an individual service with another related or unrelated service in an arbitrary package, when the level of packaging that you’re working with is typically one package per container. Microservices Architecture Requires New Patterns In other words, you really do need new patterns in order to successfully design microservices architecture. The need for patterns starts at the top. If you are refactoring a monolithic program into a microservices-based application, the first pattern that you need to consider is the one that you will use for decomposition. What pattern will you use as a guide in breaking the program down into microservices? What are the basic decomposition patterns? At the higher levels of decomposition, it makes sense to consider such functional criteria as broad areas of task-based responsibility (subdomains), or large-scale business/revenue-generating responsibilities (business capabilities). In practice, there is considerable overlap between these two general functional patterns, since a business’ internal large-scale organization of tasks is likely to closely match the organization of business responsibilities. In either case, decomposition at this level should follow the actual corporate-level breakdown of basic business activities, such as inventory, delivery, sales, order processing, etc. In the subsequent stages of decomposition, you can define groups of microservices, and ultimately individual microservices. This calls for a different and much more fine-grained pattern of decomposition—one which is based largely on interactions within the application, with individual users, or both. Decomposition Patterns for Microservices Architecture There are several ways to decompose applications at this level, depending in part on the nature of the application, as well as the pattern for deployment. You can combine decomposition patterns, and in many if not most cases, this will be the most practical and natural approach. Among the key microservice-level decomposition patterns are: Decomposition by Use Case In many respects, this pattern is the logical continuation of a large-scale decomposition pattern, since business capabilities and subdomains are both fundamentally use case-based. In this pattern, you first identify use cases: sequences of actions which a user would typically follow in order to perform a task. Note that a user (or actor) does not need to be a person; it can, in fact, be another part of the same application. A use case could be something as obvious and common as filling out an online form or retrieving and displaying a database record. It could also include tasks such as processing and saving streaming data from a real-time input device, or polling multiple devices to synchronize data. If it seems fairly natural to model a process as a unified set of interactions between actors with an identifiable purpose, it is probably a good candidate for the use case decomposition pattern. Decomposition by Resources In this pattern, you define microservices based on the resources (storage, peripherals, databases, etc.) that they access or control. This allows you to create a set of microservices which function as channels for access to individual resources (following the basic pattern of OS-based peripheral/resource drivers), so that resource-access code does not need to be duplicated in other parts of the application. Isolating resource interfaces in specific microservices has the added advantage of allowing you to accommodate changes to a resource by updating only the microservice that accesses it directly. Decomposition by Responsibilities/Functions This pattern is likely to be most useful in the case of internal operations which perform a clearly defined set of functions that are likely to be shared by more than one part of the application. Such responsibility domains might include shopping cart checkout, inventory access, or credit authorization. Other microservices could be defined in terms of relatively simple functions (as is the case with many built-in OS-based microservices) rather than more complex domains. Microservices Architecture Deployment Patterns Beyond decomposition, there are other patterns of considerable importance in building a microservices-based architecture. Among the key patterns are those for deployment. There are three underlying patterns for microservices deployment, along with a few variations: Single Host/Multiple Services In this pattern, you deploy multiple instances of a service on a single host. This reduces deployment overhead, and allows greater efficiency through the use of shared resources. It has, however, greater potential for conflict, and security problems, since services interacting with different clients may be insufficiently isolated from each other. Single Service per Host, Virtual Machine, or Container This pattern deploys each service in its own environment. Typically, this environment will be a virtual machine (VM) or container, although there are times when the host may be defined at a less abstract level. This kind of deployment provides a high degree of flexibility, with little potential for conflict over system resources. Services are either entirely isolated from those used by other clients (as is the case with single-service-per-VM deployment), or can be effectively isolated while sharing some lower-level system resources (i.e., containers with appropriate security features). Deployment overhead may be greater than in the single host/multiple services model, but in practice, this may not represent significant cost in time or resources. Serverless/Abstracted Platform In this pattern, the service runs directly on pre-configured infrastructure made available as a service (which may be priced on a per-request basis); deployment may consist of little more than uploading the code, with a small number of configuration settings on your part. The deployment system places the code in a container or VM, which it manages. All you need to make use of the microservice is its address. Among the most common serverless environments are AWS Lambda, Azure Functions, and Google Cloud Functions. Serverless deployment requires very little overhead. It does, however, impose significant limitations, since the uploaded code must be able to meet the (often strict) requirements of the underlying infrastructure. This means that you may have a limited selection of programming languages and interfaces to outside resources. Serverless deployment also typically rules out stateful services. Applying Other Patterns to Microservices Architecture There are a variety of other patterns which apply to one degree or another to microservices deployment. These include patterns for communicating with external applications and services, for managing data, for logging, for testing, and for security. In many cases, these patterns are similar for both monolithic and microservices architecture, although some patterns are more likely to be applicable to microservices than others. Fully automated parallel testing in a virtualized environment, for example, is typically the most appropriate pattern for testing VM/container-based microservices. As is so often the case in software development (as well as more traditional forms of engineering), the key to building a successful microservices architecture lies in finding the patterns that are most suitable to your application, understanding how they work, and adapting them to the particular circumstances of your deployment. Use of the appropriate patterns can provide you with a clear and accurate roadmap to successful microservices architecture refactoring and deployment. About the Author Michael Churchman is involved in the analysis of software development processes and related engineering management issues. Top Patterns for Building a Successful Microservices Architecture is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

Graphite is a complete monitoring tool for both simple and complex environments. It can be used for the monitoring of various networked systems—including sites, apps, services and servers in local and cloud environments. This range of possibilities serves companies of diverse segments and sizes. In this post all explain how graphite monitoring can help you get greater visibility into your application and infrastructure. A Quick Graphite Monitoring FAQ Graphite can support a company that has a specialized infrastructure, with a dedicated team and complex environments. It can also work well for small companies that have smaller teams and equipment. When it comes to choosing a monitoring tool, an admin usually asks several questions, such as: How long does it take to deploy? Does this tool address our issues? Will my team be able to work with and understand the essence of the tool? What reports are available? An administrator needs to ask these questions so that the choice of tool is as accurate as possible. Below are the answers to these questions for Graphite. How long does Graphite take to set up? Setting up and deploying Graphite is simple thanks to an installation script called Synthesize. And extensive technical documentation on Graphite makes it possible to gather information on the tool quickly. Essentially, Synthesize is a script that allows the installation and configuration of a range of components to automate the configuration of Graphite. Does this tool work in my environment? Graphite can support almost any type of environment you need to run it in. It works in the cloud. It runs on hybrid infrastructure. It works with on-premises servers. And it can run efficiently no matter what the size of your environment. Will my team be able to work with and understand the essence of the tool? If your team is able to read and interpret, they will understand the essence of the tool.As previously mentioned, Graphite has thorough documentation with step-by-step instructions, and scripts available to change according to your needs. What reports are available? Graphite reports are inclusive, well-crafted and easy to manipulate. This is useful because reports are often used by people who are not part of the technology team, and they need to be able to understand the information quickly. The reports will be used most often to justify requests for purchases of new equipment, as well as hardware upgrades, and for performance measurement. How Graphite Monitoring Works Now that we’ve discussed what Graphite is and where you can use it, let’s take a look at how it works. Graphite is composed of the following items: Carbon A service that will be installed on the client computer and will listen for the TCP/UDP packets to be sent over the network. Whisper The database that will store data collected from the machines. Graphite Webapp Django web app for graphics generation. To provide a sense of Graphite in action, I’ll next provide an overview of how it works on different Linux distributions. I’ll discuss my experience installing and configuring Graphite on both Ubuntu and CentOS, so that I cover both sides (the Ubuntu/Debian and CentOS/Red Hat of the Linux universe). Installing Graphite on Ubuntu Using Synthesize I installed Graphite on an Ubuntu server hosted in the cloud. Everything went smoothly, but here are a couple of special tweaks I had to perform: After logging into the cloud environment and installing Ubuntu 14.04 to test the Synthesize script, it was necessary for the cloud platform to release a port on the firewall so that the data could be sent to the dashboard (which should be done in any infrastructure, be it cloud or local). I had to be careful to release only the application port and not the full range. After that, I used a data collection dashboard recommended by Graphite, called Grafana. However, you can also stream Graphite data directly into Sumo Logic. Manual Installation of Graphite on CentOS Now let’s try a manual installation of Graphite, without using Synthesize. In a manual installation, we have to be careful about the dependencies that the operating system requires to run smoothly. To make the job a bit more complex for this example, I decided to use CentOS, and I followed the steps below. The first requirement for every operating system is to upgrade (if you haven’t already) so that it does not cause dependency problems later. sudo yum -y update Install the dependencies: sudo yum -y install httpd gcc gcc-c++ git pycairo mod_wsgi epel-release python-pip python-devel blas-devel lapack-devel libffi-devel Access the local folder to download the sources: cd /usr/local/src Clone the source of Carbon: sudo git clone https://github.com/graphite-project/carbon.git Access the Carbon folder: cd /usr/local/src/carbon/ Install Carbon: sudo python setup.py install Clone the source of Graphite Web: sudo git clone https://github.com/graphite-project/graphite-web.git Access the Graphite Web folder: cd /usr/local/src/graphite-web/ Install Graphite Web: sudo pip install -r /usr/local/src/graphite-web/requirements.txt sudo python setup.py install Copy the Carbon configuration file: sudo cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf Copy the storage schemas configuration file: sudo cp /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf Copy the storage aggregation configuration file: sudo cp /opt/graphite/conf/storage-aggregation.conf.example /opt/graphite/conf/storage-aggregation.conf Copy the relay rules configuration file: sudo cp /opt/graphite/conf/relay-rules.conf.example /opt/graphite/conf/relay-rules.conf Copy the local settings file: sudo cp /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py Copy the Graphite WSGI file: sudo cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi Copy the virtual hosts file: sudo cp /opt/graphite/examples/example-graphite-vhost.conf /etc/httpd/conf.d/graphite.conf Copy the initi files to /etc/init.d: sudo cp /usr/local/src/carbon/distro/redhat/init.d/carbon-* /etc/init.d/ Give permission to execute the init files: sudo chmod +x /etc/init.d/carbon-* Start the Carbon cache: sudo systemctl start carbon-cache Enable httpd: sudo systemctl enable httpd Start http: sudo systemctl start httpd With this configuration, we can access the Graphite web interface in https://localhost:8080, and monitor the local server running CentOS. Graphite and Sumo Logic Now that you have Graphite running, you can stream Graphite-formatted metrics directly into Sumo Logic. All you need do is set up an installed collector and connect it to a metrics source. This webinar walks you through the steps: Sources are the environments that Sumo Logic Collectors connect to to collect data from your site. Each Source is configured to collect files in a specific way, depending on the type of Collector you’re using. The Setup Wizard in Sumo Logic walks you through the process. You’ll find Linux, Mac OS and Windows instructions for installing a new Graphite collector here. Part of this process defines how your data will be tagged for _sourceCategory, the protocol and port you’ll use to stream data. Next, simply configure a Graphite source for the collector to connect to. Here are the steps for configuring your Graphite source. That’s it. Now you can use Sumo Logic’s advanced analytics to search and visualize data streaming from your application and infrastructure. Enjoy! About the Author Brena Monteiro is a software engineer with experience in the analysis and development of systems. She is a free software enthusiast and an apprentice of new technologies. Getting Started with Graphite Monitoring is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

Curious about Infrastructure as Code (IaC)? Whether you're new to AWS CloudFormation, or you control all of your cloud infrastructure through CloudFormation templates, this post demonstrates how to integrate Sumo Logic's monitoring platform into an AWS CloudFormation stack. Collect Logs and Metrics from your Stack Sumo Logic's ability to Unify your Logs and Metrics can be built into your CloudFormation Templates. Collect operating system logs, web server logs, application logs, and other logs from an EC2 instance. Additionally, Host Metrics, AWS CloudWatch Metrics, and Graphite formatted metrics can be collected and analyzed.With CloudFormation and Sumo Logic, you can achieve version control of your AWS infrastructure and your monitoring platform the same way you version and improve your software. CloudFormation Wordpress Stack with Sumo Logic Built-In Building off of the resources Adrian Cantrill provided in his Advanced CloudFormation course via A Cloud Guru, we will launch a test Wordpress stack with the following components: Linux EC2 instance - you choose the size! RDS instance - again, with a configurable size S3 bucket The Linux EC2 instance is bootstrapped with the following to create a LAMP stack: Apache MySQL PHP MySQL-PHP Libraries We also install Wordpress, and the latest version of the Sumo Logic Linux collector agent. Using the cfn-init script in our template, we rely on the file key of AWS::CloudFormation::Init metadata to install a sources.json file on the instance. This file instructs Sumo Logic to collect various types of logs and metrics from the EC2 instance: Linux OS Logs (Audit logs, Messages logs, Secure logs) Host Metrics (CPU, Memory, TCP, Network, Disk) Apache Access logs cfn-init logs Tutorial - Launch a CloudFormation Stack and Monitor Logs and Metrics Instantly First, you'll need a few things: A Sumo Logic account - Get a free one Here Access to an AWS account - If you don't have access you can sign up for the free tier here A local EC2 Key Pair - if you don't have one you can create one like this After you have access to your Sumo Logic account and an AWS account, navigate to an unused Region if you have one. This will give you a more isolated sandbox to test in so that we can more clearly see what our CloudFormation template creates. Make sure you have an EC2 key pair in that Region, you'll need to add this to the template.*Leveraging pseudo parameters, the template is portable, meaning it can be launched in any Region. First, log into AWS and navigate to CloudFormation. Choose 'Create New Stack' Then, download the example CloudFormation template from GitHub here Next, on line 87, in the EC2 Resources section, make sure to edit the value of the "KeyName" field to whatever your EC2 key is named for your current Region*Make sure the Region you choose to launch the stack in has an EC2 Key Pair, and that you update line 87 with your key's name. If you forget to do this your stack will fail to launch! Select 'Choose File' and upload the template you just downloaded and edited, then click Next Title your stack Log into Sumo Logic. and in the top-right click on your email username, then preferences, then '+' to create a Sumo Logic Access key pair Enter the Sumo Logic key pair into the stack details page. You can also select an EC2 and RDS instance size, and enter a test string that we can navigate to later when checking that we can communicate with the instance. Click 'Next', name/tag your stack if you'd like, then click 'Next' again, the select 'Create' to launch your stack! Now What? View Streaming Logs and Metrics! You've now launched your stack. In about 10-15 minutes, we can visit our Wordpress server to verify everything is working. We can also search our Apache logs and see any visitors (probably just us) that are interacting with the instance. Follow these steps to explore your new stack, and your Sumo Logic analytics: View the CloudFormation Events log. You should see four CREATE_COMPLETE statuses like so: Check your Sumo Logic account to see the collector and sources that have been automatically provisioned for you: What's Next? Sumo Logic collects AWS CloudWatch metrics, S3 Audit logs, and much more. Below is more information on the integrations for AWS RDS Metrics and also S3 Audit Logs: Amazon RDS Metrics Amazon S3 Audit Explore your logs! Try visiting your web server by navigating to your EC2 instance's public IP address This template uses the default security group of your Region's VPC, so you'll need to temporarily allow inbound HTTP traffic from either your IP, or anywhere (your IP is recommended) To do this, navigate to the EC2 console and select the Linux machine launched via the CloudFormation Template Then, scroll down to the Security Group and click 'default' as shown below Edit the inbound rules to allow HTTP traffic in, either from your IP or anywhere After you've allowed inbound HTTP traffic, navigate in your browser to <your-public-ip>/wordpress (something like 54.149.214.198/wordpress) and you'll see you're new Wordpress front end: You can also test the string we entered during setup by navigating to <your-public-ip>/index2.html Search you Sumo Logic account with _sourceCategory=test/apache and view your visits to your new Wordpress web server in the logs Finally, check out the metrics on your instance by installing the Host Metrics App: Cleanup Make sure to delete you stack as shown below, and to remove inbound HTTP rules on your default Security Group.

This is a two-part series focused on ELK stack vs. Sumo Logic. In part 1 we will highlight the journey of a DevOps engineer moving Elastic Stack from development into the production. We will also highlight typical efforts involved in managing and scaling ELK stack or Elastic stack in the production.

Not long ago I was visiting the CISO of a large agriculture biotechnology company in the Midwest – we’ll call him Ron – and he said to me “Mark these cyber terrorists are everywhere, trying to hack into our systems from Russia and China, trying to steal our intellectual property. We have the biggest and the brightest people and the most advanced systems working on it, but they are still getting through. We are really challenged in our ability to identify and resolve these cyber threats in a timely manner. Can you help us?” Business issues that CISOs and their security teams face are significant. Customers are now making different decisions based on the trust factors they have with the companies they do business with. So implementing the right levels of controls, increasing team efficiency, to rapidly identify and resolve security incidents becomes of paramount importance. But despite this big wall that Ron has built, and the SIEM technology they are currently using, threats are still permeating the infrastructure, trying to compromise their applications and data. With over 35 security technologies in play, trying to get holistic visibility was a challenge, and with a small team, managing their SIEM was onerous. Additionally, the hardware and refresh cycles over the years, as their business has grown, has been challenged by flat budget allocations. “Do more with less” was frequently what they heard back from the CIO. Like any company that wants to be relevant in this modern age, they are moving workloads to the cloud, adopting DevOps methodologies to increase the speed of application delivery, creating new and disruptive experiences for their customers, to maintain their competitive edge. But as workloads were moved to the cloud – they chose AWS- the way things were done in the past were no longer going to work. The approach to security needed to change. And it was questionable if the SIEM solution they were using was even going to run in the cloud and support native AWS services, as scale. SIEMs are technologies that were architected over 15 year ago, and they were really designed to solve a different kind of problem – traditional on prem, perimeter based, mode 1 type security applications, going after known security threats. But as organizations are starting to move to the cloud, accelerating the pace at which they roll our new code, adopting DevOps methodologies, they need something different. Something that aligns to the Mode 2 digital initiatives of modern applications. Something that is cloud native, provides elasticity on demand, and delivers rapid time to value, not constrained by fixed rule sets going after known threats but instead, leveraging machine learning algorithms to uncover anomalies, deviations and unknown threats in the environment. And lastly, something that integrates threat intelligence OOTB to increase velocity and accuracy of threat detection – so you can get a handle on threats coming at your environment trying to compromise your applications and data. Is that great big wall working for you? Likely not. To learn more about Sumo Logic’s Security Analytics capabilities, please checkout our press release, blog or landing page. Mark Bloom can be reached at https://www.linkedin.com/in/markbloom or @bloom_mark

Containers technologies like Docker enable DevOps teams to build, ship, and run distributed applications more efficiently. Docker is also an excellent infrastructure choice to build microservices. But do you know how many AWS workloads run on Docker? The Sumo Logic State of the Modern App report is the first of its kind that provides data-driven insights and best practices to help companies build, run and secure their modern applications more effectively. How did we get this this data – this report is based on insights from 1,000+ customers using Sumo Logic’s machine data analytics platform. Other insights in this report includes: How many AWS customers use Lambda? What are the common databases in AWS? What solutions are used to secure AWS applications? Click here to download State of Modern Applications in AWS Report.

You just finished building some beautiful, real-time Sumo Logic dashboards to monitor your infrastructure and application performance and now you want to show them off to your colleagues. But your boss doesn’t have a Sumo Logic account and your ops team wants this information on TVs around the office. Sound like a familiar situation? We’ve got you covered. You can now share your live dashboards in view-only mode with no login required, all while maintaining the security and transparency that your organization requires. We’ll even kick things off with a live dashboard of our own. Share Information with Colleagues and Customers This new feature enables you to share a dashboard so that anyone with the URL can view your dashboard without logging in. It reduces the friction for sharing information even further so that the right people have the right information when they need it. For example: Colleagues: Share operational and business KPIs with colleagues or executives who do not have a Sumo Logic account. Internal TVs: Display real-time information about your infrastructure and application on monitors throughout your building. Customers: Provide SLA performance or other statistics to your customers. Granular Permissions for Administrators Sharing your sensitive information to users without a login is a serious matter. With great power comes great responsibility, and no matter how much you trust your colleagues that use Sumo Logic, you may not want this power being wielded by all of your team members. If you are an administrator, you can decide which users have this permission and educate them on best practices for sharing information within and outside of your organization. By default, this capability is turned off and can only be enabled by administrators on the account. Protect Dashboard URLs with an IP / CIDR Whitelist For those who want even more protection over who can view these dashboards without logging in, you can restrict viewers to only those accessing it from specific IP addresses or CIDRs. This works great when you are placing live dashboards on TVs throughout your building and you want to make sure that this information stays in your building. Similarly, you might want to help your internal ops team troubleshoot a problem quickly without logging in. Send them the URL via email or Slack, for example, and rest assured that the information will remain in the right hands. If you decide to remove an IP address from your whitelist, any users connecting from that IP will no longer be able to view that dashboard. Complete Visibility through Audit Logs As an extra layer of transparency, you can keep track of which dashboards are shared outside of your organization and see which IPs are viewing them through your audit logs. With this information, you can: Configure real-time alerts to get notified anytime a user shares a dashboard Generate daily or weekly reports with a list of users and their shared dashboards Create dashboards of your shared dashboards – see where your dashboards are being viewed from so you can follow up on any suspicious activity. Receive alerts when someone shares a dashboard outside of your organization Use audit logs to see where your dashboards are being viewed from Learn More So go ahead – earn those bonus points with your boss and show off your dashboards today! Check out this webinar for a refresher on creating dashboards, then head over to Sumo Logic DocHub for more information on sharing these to users without an account.

The biggest risk to any organization is the end user and their password hygiene. It is an age old problem as users want to keep things easy using the same dictionary based password for all applications! This problem will continue to exist until we change end user behavior via policy enforcement and apply other layers of protection such as Single Sign On and Multi Factor Authentication (MFA). Because of this, MFA is becoming more of a must than a nice to have as companies starting to adopt a healthier security posture/program. In fact, MFA has become a full blown requirement to achieve critical compliance certifications that would provide your company with a better security reputation and demonstrate evidence of data protection. As a Cloud Security Engineer, I would love for MFA to be adopted across the board, which is part of the reason we are writing this blog, to provide our insights into the importance of implementing MFA across an enterprise. As some of you may have recently heard, Sumo Logic is now PCI 3.2 DSS compliant, which we could not have achieved without the diligence of our DevSecOps team putting some cycles together to get Multi-Factor Authentication delivered to the Sumo Logic base via the platform for another layer of password defense. When logging into the Sumo platform, you can now enable the 2-step verification for your entire organization, within the security policies section of Sumo, as seen below. When Multi Factor Authentication is enabled globally for the Org, you will be prompted with the following screen, to configure your MFA. Every login from here on out will now prompt the following screen after completed configuration. What does Multi-factor authentication provide to the end user? A low friction way to keep their credentials from being compromised and make it extremely difficult for attackers to take advantage of weak end user passwords. With the emergence of Cloud Computing, password-based security just won’t cut it anymore. Applying this extra layer of defense to credentials drastically drops the chance of your account ever being compromised. At Sumo Logic, we are glad to extend this extra layer of defense to our customers as they access our Multi Tenant Saas based offering.

In a world of shortened release cycles, agile development practices and continuous integration (CI), it is imperative teams have access to a stable tool which can manage the process of building, testing and deploying projects as changes are made. For many years, Jenkins has been the tool of choice among Java developers, but as many projects have moved into the cloud, and in particular, Amazon’s AWS ecosystem, other tools are gaining traction. One such tool is AWS CodePipeline.

OneLogin and Sumo Logic are thrilled to announce our new partnership and technology integration (app coming May 2017) between the two companies. We’re alike in many ways: we’re both cloud-first, our customers include both cloud natives and cloud migrators, and we are laser-focused on helping customers implement the best security with the least amount of effort. Today’s integration is a big step forward in making effortless security a reality. What does this integration do? OneLogin’s identity and access management solution allows for the easy enforcement of login policies across all their laptops, both Macs and Windows, SaaS applications, and SAML-enabled desktop applications. This new partnership takes things a step further by making it possible to stream application authentication and access events to over 200 application-related events. This includes over 200 application-related events, including: Who’s logged into which laptops — including stolen laptops Who’s accessed which applications — e.g., a salesperson accessing a finance app Who’s unsuccessfully logged in — indicating a potential attack in progress Who’s recently changed their password — another potential indicator of an attack Which users have lost their multi-factor authentication device — indicating a potential security weakness Which users have been suspended — to confirm that a compromised account is inactive User provision and de-provision activity – to track that users are removed from systems after leaving the company And finally, which applications are the most popular and which might be underutilized, indicating potential areas of budget waste These capabilities are critical for SecOps teams that need to centralize and correlate machine data across all applications. This, in turn, facilitates early detection of targeted attacks and data breaches, extends audit trails to device and application access, and provides a wider range of user activity monitoring. Because OneLogin has over 4000 applications in our app catalog, and automatically discover new applications and add them to its catalog, we can help you extend visibility across a wide range of unsanctioned Shadow IT apps. The integration uses streaming, not polling. This means that events flow from OneLogin into Sumo as soon as they are generated, not after a polling interval. This lets you respond more quickly to attacks in progress. How does the integration work? Since both OneLogin and Sumo Logic are cloud-based, integrating the two is a simple one-screen setup. Once integration is complete, you can use Sumo Logic to query OneLogin events, as well as view the following charts: Visitors heatmap by metro area. Suppose you don’t have any known users in Alaska — that anomaly is quite clear here, and you can investigate further. Logins by country. Suppose you don’t have any known users in China; 80 potentially malicious logins are evident here. Failed logins over time. If this number spikes, it could indicate a hacking attempt. Top users by events. If one user has many events, it could indicate a compromised account that should be deactivated in OneLogin. Events by app. If an app is utilized more than expected, it could indicate anomalous activity, such as large amounts of data downloads by an employee preparing to leave the company. All this visibility helps customers better understand how security threats could have started within their company. This is especially helpful when it comes to phishing attacks, which, according to a recent report by Gartner, are “the most common targeted method of cyberattacks, and even typical, consumer-level phishing attacks can have a significant impact on security.” Summing up: Better Threat Detection and Response Sumo Logic’s vice president of business development, Randy Streu, sums it up well: “Combining OneLogin’s critical access and user behavior data with Sumo Logic’s advanced real-time security analytics solution provides unparalleled visibility and control for both Sumo Logic and OneLogin customers.” This deep and wide visibility into laptop and application access helps SecOps teams uncover weak points within their security infrastructures so that they know exactly how to best secure data across users, applications, and devices. Get started for free Even better, OneLogin and Sumo Logic are each offering free versions of their respective products to each other’s customers to help you get started. The OneLogin for Sumo Logic Plan includes free single sign-on and directory integration, providing customers with secure access to Sumo Logic through SAML SSO and multi-factor authentication while eliminating the need for passwords. Deep visibility. Incredibly simple integration. Free editions. We’re very pleased to offer all this to our customers. Click here to learn more. *The Sumo Logic App for One Login, for out of the box visualizations and dash boarding will be available May 2017* This blog was written by John Offenhartz who is the Lead Product Owner of all of OneLogin’s integration and development programs. John’s previous experiences cover over twenty years in Cloud-based Development and Product Management with such companies as Microsoft, Netscape, Oracle and SAP. John can be reached at https://www.linkedin.com/in/johnoffenhartz

As organizations increasingly adopt DevOps as part of the digital initiative, the very nature of DevOps is rapidly changing. As DevOps as a Service, security-first design patterns, containerization and microservices come into focus, it’s clear that DevOps is becoming the de facto means by which organizations build, run and secure their modern applications.

Integrated Threat Intelligence Providing Visibility into Events that Matter to You! You’ve already invested a great deal in your security infrastructure to prevent, detect, and respond to cybersecurity attacks. Yet you may feel as if you’re still constantly putting out fires and are still uncertain about your current cybersecurity posture. You’re looking for ways to be more proactive, more effective, and more strategic about your defenses, without having to “rip and replace” all your existing defense infrastructure. You need the right cyber security intelligence, delivered at the right time, in the right way to help you stop breaches. That is exactly what Sumo Logic's integrated threat intelligence app delivers. Powered by Crowdstrike, Sumo's threat intelligence offering addresses a number requests we were hearing from customers: Help me increase the velocity & accuracy of threat detection. Enable me to correlate Sumo Logic log data with threat intelligence data to identify and visualize malicious IP addresses, domain names, email addresses, URLs and MD5 Hashes. Alert me when there is some penetration or event that maps to a known indicator of compromise (IOC) and tell me where else these IOCs exist in my infrastructure. And above all, make this simple, low friction, and integrated into your platform. And listen we did. Threat intelligence is offered as part of Sumo's Enterprise and Professional Editions, at no extra cost to the customer. Threat Intel Dashboard Supercharge your Threat Defenses: Consume threat intelligence directly into your enterprise systems in real time to increase velocity & accuracy of threat detection. Be Informed, Not Overwhelmed: Real-time visualizations of IOCs in your environment, with searchable queries via an intuitive web interface. Achieve Proactive Security: Know which adversaries may be targeting your assets and organization, thanks to strategic, operational and technical reporting and alerts. We chose to partner with CrowdStrike because they are a leader in cloud-delivered next-generation endpoint protection and adversary analysis. CrowdStrike’s Falcon Intelligence offers security professionals an in-depth and historical understanding of adversaries, their campaigns, and their motivations. CrowdStrike Falcon Intelligence reports provide real-time adversary analysis for effective defense and cybersecurity operations. To learn more about Sumo Logic's Integrated Threat Intelligence Solution, please go to http://www.sumologic.com/application/integrated-threat-intelligence.

As a principal architect at Trend Micro, focused on AWS, I get all the ‘challenging’ customer projects. Recently a neat use case has popped up with multiple customers and I found it interesting enough to share (hopefully you readers will agree). The original question came as a result of queries about Deep Security’s SIEM output via syslog and how best to do an integration with Sumo Logic. Sumo has a ton of great guidance for getting a local collector installed and syslog piped through, but I was really hoping for something: a little less heavy at install time; a little more encrypted leaving the Deep Security Manager (DSM); and a LOT more centralized. I’d skimmed an article recently about Sumo’s hosted HTTP collector which made me wonder – could I leverage Deep Security’s SNS event forwarding along with Sumo’s hosted collector configuration to get Events from Deep Security -> SNS -> Sumo? With Deep Security SNS events sending well formatted json, could I get natural language query in Sumo Logic search without defining fields or parsing text? This would be a pretty short post if the answers were no… so let’s see how it’s done. Step 1: Create an AWS IAM account This account will be allowed to submit to the SNS topic (but have no other rights or role assigned in AWS). NOTE: Grab the access and secret keys during creation as you’ll need to provide to Deep Security (DSM) later. You’ll also need the ARN of the user to give to the SNS Topic. (I’m going to guess everyone who got past the first paragraph without falling into an acronym coma has seen the IAM console so I’ll omit the usual screenshots.) Step 2: Create the Sumo Logic Hosted HTTP Collector. Go to Manage-> Collection then “Add Collector”. Choose a Hosted Collector and pick some descriptive labels. NOTE: Make note of the Category for later Pick some useful labels again, and make note of the Source Category for the Collector (or DataSource if you choose to override the collector value). We’ll need that in a little while. Tip When configuring the DataSource, most defaults are fine except for one: Enable Multiline Processing in default configuration will split each key:value from the SNS subscription into its own message. We’ll want to keep those together for parsing later, so have the DataSource use a boundary expression to detect message beginning and end, using this string (without the quotes) for the expression: (\{)(\}) Then grab the URL provided by the Sumo console for this collector, which we’ll plug into the SNS subscription shortly. Step 3: Create the SNS topic. Give it a name and grab the Topic ARN Personally I like to put some sanity around who can submit to the topic. Hit “Other Topic Actions” then “Edit topic policy”, and enter the ARN we captured for the new users above as the only AWS user allowed to publish messages to the topic. Step 4: Create the subscription for the HTTP collector. Select type HTTPS for the protocol, and enter the endpoint shown by the Sumo Console. Step 5: Go to search page in the Sumo Console and check for events from our new _sourceCategory: And click the URL in the “SubscribeURL” field to confirm the subscription. Step 6: Configure the Deep Security Manager to send events to the topic Now that we’ve got Sumo configured to accept messages from our SNS topic, the last step will be to configure the Deep Security Manager to send events to the topic. Log in to your Deep Security console and head to Administration -> System Settings -> Event Forwarding. Check the box for “Publish Events to Amazon Simple Notification Service and enter the Access and Secret key for the user we created with permission to submit to the topic then paste in the topic ARN and save. You’ll find quickly that we have a whole ton of data from SNS in each message that we really don’t need associated with our Deep Security events. So let’s put together a base query that will get us the Deep Security event fields directly accessible from our search box: _sourceCategory=Deep_Security_Events | parse “*” as jsonobject | json field=jsonobject “Message” as DSM_Log | json auto field=DSM_Log Much better. Thanks to Sumo Logic’s auto json parsing, we’ll now have access to directly filter any field included in a Deep Security event. Let your event management begin! Ping us on if you have any feedback or questions on this blog… And let us know what kind of dashboards your ops & secops teams are using this for! A big thanks to Saif Chaudhry, Principle Architect at Trend Micro who wrote this blog.

https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

So you have one or more NGINX instances happily logging to files. Now what? Consolidating all your logs from all your instances using Sumo Logic makes it quick and easy to find trends and errors.

Think you know everything there is to know about using AWS S3? In this video Sumo Logic’s Chief Architect Stefan Zier offers tips you may not know.

AWS Lambda usage is exploding, and at last week’s re:invent conference, it was one of the key priorities for AWS. Lambda simplifies infrastructure by removing host management, and giving you the advantage of paying for just the compute and storage you use. This means it’s important to monitor your usage carefully to ensure you are managing your spend well. This post outlines the various options and approaches to AWS Lambda monitoring so you can make an informed decision about how you monitor your Lambda applications. AWS Lambda Monitoring Basics AWS provides vital stats for Lambda in a couple of ways. You can view metrics from your Lambda console, the CloudWatch console, via the Lambda or CloudWatch command line, or even through the CloudWatch API. The Lambda console gives you an overview of just the vital stats of your Lambda app. For a more detailed view, you can click through to the CloudWatch console. Here are the key terms you’ll notice in your CloudWatch console: Invocations: Invocations are the number of times a Lambda function is triggered in response to an event or API call. Errors: The number of failed invocations. Duration times: The time it takes to execute a Lambda function. This is measured in milliseconds. Throttles: The number of invocation attempts that were not executed because they exceed concurrency limits. Dead letter errors: Asynchronous errors are sent to a dead letter queue for further troubleshooting. If these errors are not written to a DLQ, they are counted as dead letter errors. Within CloudWatch you can set alarms to notify you of issues, or identify unused resources. You can also view all logs generated by your code in CloudWatch Logs. Archiving these logs will incur additional cost, but you can decide how far back you’d like to store log data. Troubleshooting Common Errors in Lambda With AWS lambda, you don’t have underlying infrastructure to monitor. This means that most of the errors can be resolved by troubleshooting your code. Here are the common errors that occur with Lambda: IAM roles & permissions: For your code to access other AWS services, you need to configure IAM roles correctly. If not done, this could result in a permissions denied error. Timeout exceeded: Some functions can take longer to execute than others, and will need a longer timeout setting. Memory exceeded: Some jobs like database operations require less memory. However, jobs that involve large files, like images, for example, will need more memory. You will need to adjust the MemorySize setting in your code if you see this error. Advanced Lambda Monitoring As you start out using Lambda for part of your app, you can get by with AWS’ default monitoring options. Once you start expanding Lambda usage in your applications, and even run entire apps in Lambda, you’ll need the power of a more robust monitoring tool. The Sumo Logic App for AWS Lambda is great for monitoring your Lambda functions and gaining deeper visibility into performance and usage. Here are benefits: Track compute & memory usage: The Sumo Logic app tracks compute performance of individual Lambda functions and lets you drill down to the details. This is important because you configure resources in your code, and you need visibility at the individual function level to ensure all processes have adequate resources. Cost correlation: The Sumo Logic app translates granular performance data to actual billed costs. It lets you track usage of memory, and even excess memory in functions, and prevents overspending. It enables you to split expense across multiple teams if you need to. Based on past behavior, the app can even predict future performance and spend. Integrated reporting: If you already use Sumo Logic to monitor and manage the rest of your infrastructure stack, you can bring it all in one place, and have realistic benchmarks for your Lambda usage. Real-time stats: The data in your Sumo Logic dashboards streams in real-time from AWS, so you never have to worry about seeing outdated information. Advanced visualizations: Sumo Logic lets you slice and dice your data with advanced analytics tools. The dashboards contain various chart types, including advanced types like the box plot chart. This kind of analysis is simply not possible within AWS. As you scale your usage of Lambda, you need deep monitoring that can correlate compute usage with costs. You’re used to having control and visibility over your server-driven apps using traditional monitoring tools. With Lambda, the approach to monitoring has changed, but your goals are the same—to gain visibility and efficiency into your app and the abstracted Lambda infrastructure. You need the right tools to enable this kind of monitoring. About the Author Twain began his career at Google, where, among other things, he was involved in technical support for the AdWords team. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications. Overview of AWS Lambda Monitoring is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

Monitoring components of a system, especially for performance, is the best (and really the only) way to ensure a consistent user experience and that all service levels are being achieved. System monitoring is often left to operations to figure out after the system is built and “production ready.” This article will highlight the main areas to ensure they are monitored in MongoDB. What Is Performance Monitoring? Performance monitoring is combining all the various other types of monitoring of an application, or component, to have an overall view of system performance. Good performance monitoring also provides the ability to compare current performance levels to what has been set in service level agreements and trends. The utopia of performance monitoring is to be predictive and proactive. Find changes in performance, tell the appropriate group about it, and resolve it before it ever impacts a client. Critical Metrics to Monitor in MongoDB If this was a web or mobile application, then you’d probably be thinking response time, and you would be correct. Except in MongoDB, the only response it tracks easily is how long it is taking to write out to its peers in its replication agreements. This is a valid metric since every user transaction gets written to at least one peer. Read times aren’t tracked as their own metric inside the platform. The other critical data point to monitor for performance is disk I/O. MongoDB, like any database platform, relies heavily on the speed it can read and write to the disk. If the disk I/O gets to 100% that means everything is now waiting on disk access and the entire system will slow to a crawl. Extremely Useful MongoDB Metrics Knowing the usage pattern in your database is critical to trending performance. So tracking how many reads (selects) and writes (insert, update, delete) are being performed at a regular interval will definitely help so you can tune (and size) the system accordingly. In conjunction with tracking the actual reads and writes, MongoDB also exposes the number of clients actively doing reads and the number doing writes. Combining these two metrics will allow better cache tuning and can even decide when adding more replicas would make sense. The last extremely useful metric for performance monitoring in MongoDB, in my opinion, is capturing slow queries. This is very useful for developers to find missing indexes, and tracking the number will find ways that clients are using the system that weren’t originally envisioned. I’ve seen unexpected client behavior in the past. For instance, once a user base figured out they could do wildcard searches, they stopped typing an exact 12-digit number and just searched on the first digit followed by a %. An index brought performance back in line, but it was not expected. (Remember that people are like water. They will find the fastest way to do things and follow that route for good or bad.) Additional MongoDB Metrics to Monitor Most of the metrics in this section are solid and valuable, but are too coarse-grained to be of the same value as the metrics in the above sections. The following are good overall metrics of system health, and will definitely help in scaling the database system up and down to meet the needs of your applications. MongoDB has internal read and write queues that can be watched, and will only really be used when MongoDB can’t keep up with the number of requests that are incoming. If these are used often, then you will probably need to look into adding capacity to your MongoDB deployment. Another great metric to trend is the number of client connections, and available connections. Then, of course, there are always metrics at the machine level that are important to watch on all of your nodes. These include memory, CPU, and network performance. Monitoring Tools to Use More information on the tools that ship with MongoDB for monitoring are available at mongoDB. And there are operational support platforms, like Sumo Logic, which provide a much more visual and user-friendly way to review these metrics. Sumo Logic also has the Sumo Logic App for MongoDB that includes pre-built queries and dashboards allowing you to track overall system health, queries, logins and connections, errors and warnings, replication, and sharding. You can learn more about the app from the MongoDB documentation. If you don’t already have a Sumo Logic account, sign up for the Free Trial and take the app for a spin. About the Author Vince Power is a Solution Architect who has a focus on cloud adoption and technology implementations using open source-based technologies. He has extensive experience with core computing and networking (IaaS), identity and access management (IAM), application platforms (PaaS), and continuous delivery. Overview of MongoDB Performance Monitoring is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

The AWS Well Architected Framework's Security Pillar defines cloud security best practices with five Design Principles and five focus areas. First, I'll describe the Security Design Principles, and some Sumo Logic capabilities that will help you adhere to them. Then, I'll list the five focus areas with corresponding, investigative questions you can ask to ensure your architecture is secure. A link to the next pillar in the framework, Reliability, is soon to come.

This CISO blog post was contributed by Gary Hayslip, Deputy Director, Chief Information Security Officer (CISO) for the City of San Diego, Calif., and Co-Author of the book CISO Desk Reference Guide: A Practical Guide for CISOs As businesses today focus on the new opportunities cybersecurity programs provide them, CISOs like myself have to learn job roles they were not responsible for five years ago. These challenging roles and their required skill sets I believe demonstrate that the position of CISO is maturing. This role not only requires a strong technology background, good management skills, and the ability to mentor and lead teams; it now requires soft skills such as business acumen, risk management, innovative thinking, creating human networks, and building cross-organizational relationships. To be effective in this role, I believe the CISO must be able to define their “Vision” of cybersecurity to their organization. They must be able to explain the business value of that “Vision” and secure leadership support to execute and engage the business in implementing this “Vision.” So how does this relate to the subject of my manifesto? I am glad you asked. The reason I provided some background is because for us CISOs, a large portion of our time is spent working with third-party vendors to fix issues. We need these vendors to help us build our security programs, to implement innovative solutions for new services, or to just help us manage risk across sprawling network infrastructures. The truth of the matter is, organizations are looking to their CISO to help solve the hard technology and risk problems they face; this requires CISOs to look at technologies, workflows, new processes, and collaborative projects with peers to reduce risk and protect their enterprise assets. Of course, this isn’t easy to say the least, one of the hardest issues I believe CISOs face is time and again when they speak with their technology provider, the vendor truly doesn’t understand how the CISO does their job. The vendor doesn’t understand how the CISO views technology or really what the CISO is looking for in a solution. To provide some insight, I decided I would list ten rules that I hope technology providers will take to heart and just possibly make it better for all of us in the cyber security community. Now with these rules in mind, let’s get started. I will first start with several issues that really turn me off when I speak with a technology provider. I will end with some recommendation to help vendors understand what CISOs are thinking when they look at their technology. So here we go, let’s have some fun. Top Ten Rules for Technology Providers “Don’t pitch your competition” – I hate it when a vendor knows I have looked at some of their competitors, and then they spend their time telling me how bad the competition is and how much better they are. Honestly I don’t care, I contacted you to see how your technology works and if it fits for the issue I am trying to resolve. If you spend all of your time talking down about another vendor, that tells me you are more concerned about your competitor than my requirements. Maybe I called the wrong company for a demonstration. “Don’t tell me you solve 100% of ANY problem” – For vendors that like to make grand statements, don’t tell me that you do 100% of anything. The old adage “100% everything is 0% of anything.” In today’s threat environment, the only thing I believe that is 100% is eventually that I will have a breach. The rest is all B.S. so don’t waste my time saying you do 100% coverage, or 100% remediation, or 100% capturing of malware traffic. I don’t know of a single CISO that believes that anyone does 100% of anything so don’t waste your time trying to sell that to me.

AWS Best Practices - Using tools in Amazon Web Services and Sumo Logic to analyze logs and metrics and achieve a Well Architected Framework

Any parent of a two-year old appreciates the power of speaking a common language. There is nothing more frustrating to my two-year old son than his inability to communicate what he wants. Learning to say things like “milk” and “applesauce” has transformed the breakfast experience. On the other hand, learning to say “trash” means that any object in reach is in danger of being tossed in the trash bin for the glory of saying “twash” over and over again. I see this in my world world as well. Millions of dollars are spent in the software world translating from one “language” to another. In fact, a whole industry has popped up around shared Application Program Interfaces (APIs) to standardize how systems communicate. Despite this trend, there seems to be more emphasis on the “communication” rather than the “data” itself. Data Analytics products in particular seem happy to shove all types of data into one mold, since the output is the same. That is where we at Sumo Logic decided to take the road less travelled. We believe in the idea that data needs to be treated “natively” – in other words, we don’t want to shove a square data peg in a round systems hole. Just like speaking a language natively changes the experience of travel – speaking the native language of data transforms the analytics experience. When we decided to build our Unified Logs and Metrics (ULM) product, we also decided that it was essential that we become bi-lingual – speaking the native language of both raw logs and time-series metrics. And here is why it matters – according to my Toddler. Answer my Question – Quickly Toddlers are well acquainted with the frustration of not being understood. Everything takes too long, and they need it now. And you know what. I get it. I have had the occasion of speaking to people that don’t speak my language before, and it is hard. I once spent 15 minutes in a pharmacy in Geneva trying to order anti-bacterial cream. There was a lot of waving of hands and short, obvious words (Arm. Cut. Ow!). We would have faced the same obstacles if we used a log system to store metrics. Every query needs to be translated, from one language to another, and it takes forever. At end of the day, you can try optimize a log system – built to search for needles in haystacks – to perform the equivalent of speed reading, but eventually the laws of physics intervene. You can only make it so fast. It takes too long – and like my toddler, you will just stop asking the question. What’s the use in that? Cleaning up is Hard I am always amazed with how my two-year old can turn a nice stack of puzzles or a bucket of toys into a room-sized disaster zone – it is the same components, but vastly different results. Storage optimization is essential in the world of operational data. There is a natural assumption underneath a true log-analytics system. We assume on some level that each log is a special snowflake. There is, of course, a lot of repetition, but the key is to be flexible and optimize for finding key terms very quickly. Metrics, on the hand, are repetitive by design. Every record of a measurement is the same – except for the measurement itself. Once you know you are collecting something – say system CPU performance on some server – you don’t need to capture that reference every time. You can optimize heavily for storing and retrieving long lists of numbers. Storing time series metrics as logs, or events, is extremely wasteful. You can incur any where from 3x to 10x more storage costs – and that is without the same performance. To achieve the same performance as most metrics system can reach, you are looking at 10-20x in storage costs. This, of course, is the reason why no log-analytics companies are really used for performance metrics at scale – the immense costs involved just don’t justify the benefit of tool reduction. I Want to Play with my Cars Anywhere One of the funniest things my son does is how he plays with his toy cars. He has race tracks, roads, and other appropriate surfaces. He rarely uses them. He prefers to race his cars on tables, up walls, and on daddy’s leg. The flexibility of having wheels is essential. He has other “cars” that don’t roll – he doesn’t really play with them. It is the same core truth with data analytics. Once you have high performance with cost effective storage – uses just present themselves. Now you can perform complex analytics without fear of slowing down the system to a crawl. You can compare performance over months and years, rather than minutes and hours – because storage is so much cheaper. Innovative use cases will always fill up the new space created by platform enhancements – just as restricted platforms will always restrict the use cases as well. Choose Wisely So, it’s 2 AM. Your application is down. Your DevOps/Ops/Engineering team is trying to solve the problem. They can either be frustrated that they can’t get their questions answered, or they can breeze through their operational data to get the answers they need. I know what two-year old would tell you to do. Time to put your old approach to data analytics in the twash.

How can you get the most out of monitoring your AWS Lambda functions? In this post, we’ll take a look at the monitoring and logging data that Lambda makes available, and the value that it can bring to your AWS operations. You may be thinking, “Why should I even monitor AWS Lambda? Doesn’t AWS take care of all of the system and housekeeping stuff with Lambda? I thought that all the user had to do was write some code and run it!” A Look at AWS Lambda If that is what you’re thinking, then for the most part, you’re right. AWS Lambda is designed to be a simple plug-and-play experience from the user’s point of view. Its function is simply to run user-supplied code on request in a standardized environment. You write the code, specifying some basic configuration parameters, and upload the code, the configuration information, and any necessary dependencies to AWS Lambda. This uploaded package is called a Lambda function. To run the function, you invoke it from an application running somewhere in the AWS ecosystem (EC2, S3, or most other AWS services). When Lambda receives the invoke request, it runs your function in a container; the container pops into existence, does its job, and pops back out of existence. Lambda manages the containers—You don’t need to (and can’t) do anything with them. So there it is—Lambda. It’s simple, it’s neat, it’s clean, and it does have some metrics which can be monitored, and which are worth monitoring. Which Lambda Metrics to Monitor? So, which Lambda metrics are important, and why would you monitor them? There are two kinds of monitoring information which AWS Lambda provides: metrics displayed in the AWS CloudWatch console, and logging data, which is handled by both CloudWatch and the CloudTrail monitoring service. Both types of data are valuable to the user—the nature of that value and the best way to make use of it depend largely on the type of data. Monitoring Lambda CloudWatch Console Metrics Because AWS Lambda is strictly a standardized platform for running user-created code, the metrics that it displays in the CloudWatch console are largely concerned with the state of that code. These metrics include the number of invocation requests that a function receives, the number of failures resulting from errors in the function, the number of failures in user-configured error handling, the function’s duration, or running time, and the number of invocations that were throttled as a result of the user’s concurrency limits. These are useful metrics, and they can tell you a considerable amount about how well the code is working, how well the invocations work, and how the code operates within its environment. They are, however, largely useful in terms of functionality, debugging, and day-to-day (or millisecond-to-millisecond) operations. Monitoring and Analyzing AWS Lambda Logs With AWS Lambda, logging data is actually a much richer source of information in many ways. This is because logging provides a cumulative record of actions over time, including all API calls made in connection with AWS Lambda. Since Lambda functions exist for the most part to provide support for applications and websites running on other AWS services, Lambda log data is the main source of data about how a function is doing its job. “Logs,” you say, like Indiana Jones surrounded by hissing cobras. “Why does it always have to be logs? Digging through logs isn’t just un-fun, boring, and time-consuming. More often than not, it’s counter-productive, or just plain impractical!” And once again, you’re right. There isn’t much point in attempting to manually analyze AWS Lambda logs. in fact, you have three basic choices: either ignore the logs, write your own script for extracting and analyzing log data, or let a monitoring and analytics service do the work for you. For the majority of AWS Lambda users, the third option is by far the most practical and the most useful. Sumo Logic’s Log Analytics Dashboards for Lambda To get a clearer picture of what can be done with AWS Lambda metrics and logging data, let’s take a look at how the Sumo Logic App for AWS Lambda extracts useful information from the raw data, and how it organizes that data and presents it to the user. On the AWS side, you can use a Lambda function to collect CloudWatch logs and route them to Sumo Logic. Sumo integrates accumulated log and metric information to present a comprehensive picture of your AWS Lambda function’s behavior, condition, and use over time, using three standard dashboards: The Lambda Overview Dashboard The Overview dashboard provides a graphic representation of each function’s duration, maximum memory usage, compute usage, and errors. This allows you to quickly see how individual functions perform in comparison with each other. The Overview dashboard also breaks duration, memory, and compute usage down over time, making it possible to correlate Lambda function activity with other AWS-based operations, and it compares the actual values for all three metrics with their predicted values over time. This last set of values (actual vs. predicted) can help you pinpoint performance bottlenecks and allocate system resources more efficiently. The Lambda Duration and Memory Dashboard Sumo Logic’s AWS Lambda Duration and Memory dashboard displays duration and maximum memory use for all functions over a 24-hour period in the form of both outlier and trend charts. The Billed Duration by Hour trend chart compares actual billed duration with predicted duration on an hourly basis. In a similar manner, the Unused Memory trend chart shows used, unused, and predicted unused memory size, along with available memory. These charts, along with the Max Memory Used box plot chart, can be very useful in determining when and how to balance function invocations and avoid excessive memory over- or underuse. The Lambda Usage Dashboard The Usage dashboard breaks down requests, duration, and memory usage by function, along with requests by version alias. It includes actual request counts broken down by function and version alias. The Usage dashboard also includes detailed information on each function, including individual request ID, duration, billing, memory, and time information for each request. The breakdown into individual requests makes it easy to identify and examine specific instances of a function’s invocation, in order to analyze what is happening with that function on a case-by-case level. It is integrated, dashboard-based analytics such as those presented by the Sumo Logic App for AWS Lambda that make it not only possible but easy to extract useful data from Lambda, and truly make the most of AWS Lambda monitoring. About the Author Michael Churchman started as a scriptwriter, editor, and producer during the anything-goes early years of the game industry. He spent much of the ‘90s in the high-pressure bundled software industry, where the move from waterfall to faster release was well under way, and near-continuous release cycles and automated deployment were already de facto standards. During that time he developed a semi-automated system for managing localization in over fifteen languages. For the past ten years, he has been involved in the analysis of software development processes and related engineering management issues.

Early on, we measured data volume in gigabytes. Then we moved onto terabytes. Now, it’s petabytes. But the scale of data is not the only thing that has changed. We now deal with different types of data as well. In particular, the introduction of large volumes of machine data have created new opportunities for machine data analytics. Leveraging machine data, especially logs and metrics, is a key part of advancing the DevOps workflow. Advanced analytics based on machine data allows DevOps engineers to make sense of petabytes of data by using statistical, indexing, filtering and machine learning techniques. In this post, I explain how to use Sumo Logic’s cloud-native platform to analyze large volumes of machine data to drive actionable insights. Using Sumo Logic Unified Logs and Metrics for Machine Data Let’s start by discussing how Sumo Logic allows users to visualize machine logs and metrics. Sumo makes this information available through a single, unified interface—the Sumo Logic Application Status Dashboard. The dashboard shows the DevOps engineer a real-time visualization of the status quo regarding logs and metrics. The Sumo Logic Application Status Dashboard The image above shows the available metrics in this example: latency, customer logins, CPU usage, app log errors, and memory usage and errors. Additional metrics can be visualized in the dashboard as well, depending on which type of data is available. Examples of supported logs include error logs, binary logs, general and slow query logs, and DDL logs. In addition, since the dashboard is connected to those logs, it allows you to drill down to find more details about an issue. The Sumo Logic Dashboard and DevOps Using the available logs and metrics, a DevOps engineer can perform a quick root cause analysis on a production issue so the problem can be addressed quickly. That’s essential in DevOps because quick resolution of problems assures that pipelines can keep flowing continuously. This video demonstrates the Sumo Logic Application Dashboard in action: Notice in particular the use of filtering—one of the analytics techniques Sumo Logic uses to help a DevOps engineer tackle an issue. Other analytics methods include statistical, indexing and machine learning techniques. Machine Data, Predictive Analytics and Sumo Logic Sumo Logic lets you do more with machine data than simply find out what happened. You can also use it as a predictive analytics platform to identify trends and understand what is likely to happen next with your infrastructure or DevOps development pipeline. Predictive analytics based on machine data are valuable because the vast volume of data coming daily into an organization means that a large amount of that data turns into noise and ultimately masks the messages that are most important. With predictive analytics, DevOps teams can make the most of all data, even if they can’t react to it all in real time. Consider, for example, the case of a CPU usage spike or a memory drop. Predictive analytics techniques could help you to predict when such events will occur again so that you can prepare for them. Similarly, predictive analytics delivered via tools like Sumo Logic can help you to find patterns in a vast amount of data without having to program your own code. Sumo can identify the trends and help you make sense of them through a convenient interface. That’s a big help to DevOps professionals because it means that, using Sumo Logic, they can make sense of a large volume of information without having to be experts in statistics or data analytics programming. Instead, they can focus on what they know best—whether it is coding, testing or system administration—and rely on Sumo Logic to be the data analytics expert on their team. LogReduce: Clean Up Your Machine Data A final feature worth mentioning is LogReduce. This is a feature in Sumo Logic that, like unified logs and metrics, helps DevOps engineers to reduce the noise in their machine data. The following video shows an example of LogReduce: As you can see, a lot of calculations and analysis are done under the hood. All the DevOps engineer had to do was push the LogReduce button. This saves the DevOps engineer from having to worry about machine learning techniques, freeing him or her to focus on the problem to be solved. In my opinion, every DevOps engineer using the LogReduce button should have at least a basic understanding of machine data. Otherwise, results could be misinterpreted. Still, LogReduce is a great feature for transforming a baseline knowledge of machine data analytics into expert-level results. About the Author Cordny Nederkoorn is a software test engineer with over 10 years of experience in finance, e-commerce and web development. He is also the founder of TestingSaaS, an international community researching cloud applications with a focus on forensics, software testing and security.

Wow, what a week last week at AWS re:Invent. I spent the weekend reflecting (and recovering) on the progress we have made in the market and how our great partners have impacted Sumo Logic’s success. Sumo Logic was born in the cloud and designed to help customers build, run and secure mission critical cloud applications. If you haven’t already, I encourage you to read our “State of the Modern Applications” report. As we developed this report, we found it incredibly enlightening as have many others. The report highlights how wildly different the technology choice is for building these modern applications in the cloud versus traditional technologies used for on-premises application development. This rise and increased adoption of cloud-native applications has heavily influenced Sumo Logic’s partner ecosystem. We are very proud and honored to be partnered with the leading technologies used to build today’s modern applications. We share a common vision with our partners to help customers accelerate their adoption of cloud more quickly and more safely. So, together with MongoDB, Fastly, Pivotal, NGINX, CrowdStrike, Okta and Evident.IO, we decided to THROW A PARTY to celebrate the success we shared in 2016 and to kick off 2017 with a bang! We had well over 300 customers, partners, and people wanting to learn more at the event. It was a fantastic time, and I want to thank our partners for sponsoring and everyone who attended. As I look to 2017, I couldn’t be more excited to be at Sumo Logic and the tremendous opportunity that lies ahead for us and our incredible partners. We are just getting started together on this journey, and I am supremely optimistic about our future together. Wishing you all a very happy holiday season and New Year!

At AWS re:Invent 2016 last week, New Relic and Sumo Logic announced a partnership that brings together two leaders in application operations. The integrated solution combines machine data analytics with application and infrastructure performance data to enable joint customers to use the New Relic Digital Intelligence Platform for visualizing the impact of software performance on customer experience and business outcomes. Why is this important and what does this mean for the industry? New Relic is the leader in Application Performance Management and provides detailed performance metrics across an enterprise’s digital business – customer experience, application performance, and infrastructure. Thousands of enterprises use the New Relic solution to proactively monitor and manage application performance alerts (“latency of application is spiking, application is unavailable etc.”). However, to get to the root cause of issues, IT teams also need complete visibility into the logs associated with the application, transactions and the supporting infrastructure. And that is where the integrated Sumo Logic – New Relic solution comes in. Sumo Logic turns machine data – structured and unstructured into operational insights, helping organizations more effectively build, run and secure modern applications and infrastructure. The Sumo Logic service enables customers to ingest and analyze logs with patented machine learning algorithms. And when integrated with New Relic’s full-stack visibility provides joint customers with: Complete picture of what’s happening with users, applications, and instances Proactive identification of application and infrastructure performance issues Faster troubleshooting and root cause analysis leveraging APM and log data However, it’s not just the comprehensive and integrated APM and log analytics views that distinguishes this partnership. What makes this partnership unique is the “operating model” of the two solutions. Like Sumo Logic, New Relic also focuses on modern applications and infrastructures that are developed using agile/DevOps methodologies, use microservices style distributed architectures and typically run in highly elastic and scalable cloud environments. And like Sumo Logic, New Relic is also a SaaS service, providing fast time-to-value and low total-cost-of-ownership for its customers (incidentally, both Sumo Logic and New Relic are in the newly announced AWS SaaS subscription marketplace) At Sumo Logic, we are thrilled to partner with New Relic. We believe that application operations is undergoing fundamental transformations and companies like Sumo Logic and New Relic are bringing a new vision to this market. Stay tuned to see more integration details emerge from this partnership.

We always love hearing how customers are using Sumo Logic. Sometimes it’s a unique use case we never thought Sumo would be used for such as an IoT or billing project, and other times it’s when the lightbulb goes off for an organization on the valuable insights and intelligence that log data and analytics can provide. Such is the case for OpenX, an adtech company that develops advertising marketplaces. “Put simply, SumoLogic has saved our butts multiple times by helping us illuminate system behavior and pinpoint issues quickly. It is an indispensable tool for the people at OpenX to continue being unreasonably awesome at building and operating a top-rated technology stack.” You can’t get better validation than this and it’s what we want for all of our customers. To learn more about how OpenX is leveraging the power of Sumo Logic read their blog.

Yes, that is a “T” as in trillion. In the last 30 days, I had the pleasure of attending two events that reinforced in my mind that Sumo Logic is at the center of the largest market disruption I will most likely experience in my lifetime. First, I traveled with our Sumo Logic CTO, Christian Beedgen, to Lisbon, Portugal, for his presentation at Europe’s largest technology conference, Web Summit 2016. After watching at least 100 thought leaders representing 20+ different industries from around the world, the comment from Saul Klein of LocalGlobe and Index Ventures about the expected reallocation of $30 trillion in market value from existing Fortune 2000 companies to new disruptors and yet-to-be-born companies beautifully summed up my key takeaway from the show. However this massive transfer of wealth is not just dependent on each country represented (more than 115!) finding and funding the next Google. Klein passionately appealed to global policy makers and influencers to support the 120 million “zebras” on Facebook interested in entrepreneurship in order to have more control over their lives. Self-employed entrepreneurs already represent five percent of the global population and growing. So, imagine the synchronicity I felt when I found myself a few weeks later at what I consider to be the mecca of the zebra wave: AWS re:Invent, Las Vegas. The energy, activity and dialogue of 32,000+ attendees (or ‘builders’ in parlance of AWS CTO Werner Vogels), across a myriad of market categories sharing and demonstrating their cloud-based products, solutions and services was not only exciting, but also gave me a sense of hope. Hope, in that, as Gary Vee said at Web Summit, now with the ubiquity of mobile and the accessibility of cloud, “if you have the imagination and willingness, no one can stop you. No one.” This hope is not just in transformative technologies but also in the very way people come together to create. It’s collaboration that goes beyond the sake of inclusivity in and of itself. The market transition to a size-able, global entrepreneurial workforce will create an ocean of experts who have the opportunity to come together in a much more harmonious way to solve similar problems, providing solutions to market that benefit people and profits. For businesses adopting this philosophy, silos will dissipate in favor of collaborative structures, processes and decision-making. We see this playing out already in digital businesses that have adopted DevOps as their innovation strategy. DevOps practices scuttle linear, waterfall software development approaches in favor of agile and continuous delivery practices resembling “loops.” Silos fade away, and a cross-expertise (as opposed to “cross-function”) of development, testing, deployment, operations, security and line-of-business professionals come together to create, monitor, troubleshoot, optimize and then creation again — a continuous loop that drives continuous innovation. And these loops won’t be cookie-cutter — they will be unique to the ideas, culture and opportunities of each organization, thereby creating new sources of business competitive advantage, and new sources of hope for unimagined opportunities. We see examples of this today in organizations such as Google, Facebook, Airbnb, Uber, Spotify, Snapchat, Amazon and Netflix to name a few. So, every company faces a fundamental question — to be part of the policy of continuous innovation (hope), or to maintain a policy of entrenchment to preserve the status quo. In my mind, no group will survive a policy of entrenchment or isolation. A company might keep going for a while, but ultimately it won’t survive. The longer it waits, the more it delays the inevitable. That’s why Sumo Logic is passionate about helping companies make this transition. Our machine data analytics platform provides the visibility and confidence companies desire to transition to cloud and modern applications. We think of it as “continuous intelligence” to feed the loop of continuous innovation decision-making. And the value is applicable across the organization. For example, as a 20-year marketing & communication veteran, I’m excited to finally be in the position of being a consumer of a B2B technology that’s relevant to my expertise. Our cloud-native, multi-tenant platform service enables us to leverage customer meta data to better understand their usage patterns of our own product, so we can better support and market to them, in addition to optimizing our services. And that re-allocated $30T I mentioned earlier? Well wrap your head around this: one expected outcome will be that 75% of the S&P 500 will be replaced in the next 10 years, at the current S&P churn rate. (Source: Innosight, Richard N. Foster, Standard & Poor’s). So, there’s a sense of urgency in the air, which may explain the astonishing growth rates of Web Summit (from a couple of hundred in Dublin 2010 to more than 50,000 in six years), and AWS re:Invent (5,500 attendees in 2012 to more than 32,000 this year). And that gives me hope too because it means people are getting it. So, when I look back ten years from now at Web Summit and AWS re:Invent 2016, I’m proud to know that I was there at ground zero for a $30T market disruption. And my forecast for the next ten years? It will be a wild ride for sure, but those who go ‘all-in’ on cloud computing, focus on new ideas and delivering value continuously at speed, are likely to come out on top. And at a time when much in the world seems unknown, the future certainly seems bright to me. https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

Yesterday at AWS re:Invent 2016, Sumo Logic Co-Founder and CTO Christian Beedgen presented his vision for machine data analytics in a world where modern apps are disrupting virtually every vertical market in business. Every business is a software business, Marc Andreessen wrote more than five years ago. Today, driven by customer demand, the need to differentiate and the push for agility, digital transformation initiatives are disrupting every industry. “We are still at very the beginning of this wave of Digital Transformation,” Christian said. “By 2020 half of all businesses will have figured out digitally enhanced products and services.” The result is that modern apps are being architected differently than they were just 3 years ago. Cloud applications are being built on microservices by DevOps teams that automate to deliver new functionality faster. “It used to be that you could take the architecture and put it on a piece of paper with a couple of boxes and a couple of arrows. Our application architecture was really clean.” But with this speed and agility comes complexity, and the need for visibility has become paramount. “Today our applications look like spaghetti. Building microservices, wiring them up, integrating them so they can work with something else, foundational services, SQL databases, NoSQL databases…” You need to be able to see what’s going on, because you can’t fix what you cannot see. Modern apps require Continuous Intelligence to provide insights, continuously and in real-time, across the entire application lifecycle. Designing Your Data Analytics Strategy Ben Newton, Sumo Logic’s Principal Product Manager of the Metrics team, took the stage to look at the various types of data and what you can do with them. Designing a data analytics strategy begins by understanding the data types that are produced by machine data, then focusing on the activities that data supports. The primary activities are Monitoring where you detect and notify (or alert), and Troubleshooting where you identify, diagnose, restore and resolve. “What we often find is that users can use that same data to do what we call App Intelligence – the same logs and metrics that allows you to figure out something is broken, also tells you what your users are doing. If you know what users are doing, you can make life better for them because that’s what really matters.” So who really cares about this data? When it comes to monitoring where the focus is on user-visible functionality, it’s your DevOps and traditional IT Ops teams. Engineering and development also are responsible for monitoring their code. In troubleshooting apps where the focus is on end-to-end visibility, customer success and technical support teams also become stakeholders. For app intelligence, the focus is on user activity and visibility everyone is a stakeholder including sales, marketing, and product management. “Once you have all of this data, all of these people are going to come knocking on your door,” said Ben. Once you understand the data types you have, where it is within your stack and the use cases, you can begin to use data to solve real problems. In defining what to monitor and measure, Ben highlighted: Monitor what’s important to your business and your users. Measure and monitor user visible metrics. Build fewer, higher impact, real-time monitors. “Once you get to troubleshooting side, it gets back to you can’t fix what you can’t measure.” Ben also said: You can’t improve what you can’t measure. You need both activity metrics and detailed logs. Up to date data drives better data-driven decisions. You need data from all parts of your stack. So what types of data will you be looking at? Ben broke it down to the following categories: Infrastructure Rollups vs. Detailed What resolution makes sense? Is real-time necessary? Platform Rollups vs. Detailed Coverage of all components Detailed logs for investigations Architecture in the metadata Custom How is your service measured? What frustrates users? How does the business measure itself? “Everything you have produces data. It’s important to ensure you have all of the components covered.” Once you have all of your data, it’s important to think about the metadata. Systems are complex and the way you make sense out of it is through your metadata. You use metadata to describe or tag your data. “For the customer, this is the code you wrote yourself. You are the only people that can figure out how to monitor that. So one of the things you have to think about is the metadata. ” Cloud Cruiser – A Case Study Cloud Cruiser’s Lead DevOps Engineer, Ben Abrams, took the stage to show how the company collects data and provide some tips on tagging it with metadata. Cloud Cruiser is a SaaS app that enables you to easily collect, meter, and understand your cloud spend in AWS, Azure, and GCP. Cloud Cruiser’s customers are large enterprises and mid-market players globally distributed across all verticals, and they manage hundreds of millions of cloud spend. Cloud Cruiser had been using an Elastic (Elasticsearch, Logstash, and Kibana) stack for their log management solution. They discovered that managing their own logging solution was costly and burdensome. Ben cited the following: Operational burden was a distraction to the core business. Improved security. Ability to scale + cost. Cloud Cruiser runs on AWS (300-500 instances) and utilizes microservices written in Java using the dropwizard framework. Their front-end web app runs on Tomcat and uses Angularjs. Figure 1 shows the breadth of the technology stack: In evaluating a replacement solution, Ben said “We were spending too much time on our ELK stack.” Sumo Logic’s Unified Logs and Metrics (ULM) was also a distinguishing factor. The inclusion of metrics meant that they didn’t have to employ yet another tool that would likewise have to be managed. “Logs are what you look at when something goes wrong. But Metrics are really cool.” Ben summarized the value and benefits they achieved this way: Logs Reduced operational burden. Reduced cost. Increased confidence in log integrity. Able to reduce the number of people needing VPN. Alerting based on searches did not need ops handholding. Metrics Increased visibility in system and application health. Used in an ongoing effort with application and infrastructure changes in that we were able to reduce our monthly AWS bill by over 100%. Ben then moved into a hands on session, showing how they automate the configuration and installation of Sumo Logic collectors, and how they tag their data using source categories. Cloud Cruiser currently collects data from the following sources: Chef: automation of config and collector install Application Graphite Metrics from Dropwizard Other graphite metrics forwarded by Sensu to Sumo Logic “When I search for something I want to know what environment is it, what type of log is it, and which server role did it come from.” One of their decisions was to differentiate log data from metrics data as shown below. Using this schema allows them to search logs and metrics by environment, type of log data and corresponding Chef role. Ben walked through the Chef Cookbook they used for deploying with Chef and shared how they automate the configuration and installation of Sumo Logic collectors. For those interested, I’ll follow on this up in the DevOps Blog. A key point from Ben, though, was “Don’t log secrets.” The access ID and key should be defined elsewhere, out of scope and stored in an encrypted data bag. Ben also walked through the searches they used to construct the following dashboard. Through this one dashboard, Cloud Cruiser can utilize both metrics and log data to get an overview of the health of their production deployment. Key Takeaways Designing your data analytics strategy is highly dependent on your architecture. Ultimately it’s about the experience you provide to your users. It’s no longer just about troubleshooting issues in production environments. It’s also about understanding the experience you provide to your users. The variety of data that streams in real time comes from the application, operating environment and network layers produces an ever increasing volume of data every day. Log analytics provides the forensic data you need, and time-series based metrics give you insights into the real-time changes taking place under the hood. To understand both the health of your deployment and the behavior/experience of your customers, you need to gather machine data from all of its sources, then apply both logs and metrics to give teams from engineering to marketing the insights they need. Download the slides and view the entire presentation below:

Evident.io and Sumo Logic team up to provide seamless integrated visibility into compliance monitoring and risk attribution Analyzing and visualizing all your security data in one place can be a tricky undertaking. For any SOC, DevSecOps or DevOps team in heterogeneous environments, the number of tools in place to gain visibility into and monitor compliance can be daunting. The good news is that Evident.io and Sumo Logic have teamed up to bring you a simple-to-implement, yet effective integration that allows you to perform additional analytics and visualization of your Evident Security Platform data in the Sumo Logic Analytics platform. Evident.io ESP is an agentless, cloud-native platform focused on comprehensive continuous security assessment of the control plane for AWS cloud infrastructure services. ESP can monitor all AWS services available through the API, ensuring their configurations are in line with AWS best practices for security as well as your organization’s specific compliance requirements. Sumo Logic is a leading SaaS-native, machine data analytics service for log management and time series metrics. Sumo Logic allows you to aggregate, perform statistical analytics, report on trends, visualize and alert on all your operational, performance and security related event log data in one place from just about any data source. Why integrate with Sumo Logic? Both of these platforms are architected for the cloud from the ground up and have a solid devops pedigree. This integration allows you to aggregate all the data generated by your AWS cloud infrastructure in the same place as your application level security and performance event data which allows you to perform attribution on a number of levels. The Evident.io alert data is rich with configuration state data about your security posture with regards to AWS best practices for security and the CIS Benchmarks for AWS. As customers adopt CI/CD concepts; being able to quickly visualize, alert and remediate, in near real-time, on any vulnerabilities introduced by misconfiguration is critical. Evident.io and Sumo Logic combined can help you do this better and faster. And, best yet, it is super easy to get started with Evident.io and Sumo Logic in a matter of minutes. The Sumo Logic App for Evident.io ESP The Sumo Logic App for Evident.io ESP enables a user to easily and quickly report on some key metrics from their AWS Cloud infrastructure such as: Trend analysis of alerts over time (track improving or deteriorating posture over time) Time to resolve alerts (For SLAs – by tracking the start and end of an alert in one report) Summary of unresolved alerts/risks Number of risks found by security signatures over time Below are some screen shots from the Sumo Logic App for Evident.io ESP: Figure 1 is an overview of the the types and severity of risks, alert status and how long before a risk is resolved and marked as ended on the Evident.io side. This can be an important metric when managing to SLAs. Fig. 1 Figure 2 provides a detailed view of the risks identified by Evident.io ESP within the configured time range for each of the dashboard panels. The panels present a views into: Which Evident.io ESP signatures triggered the risks A breakdown of: risks identified by AWS region risks by AWS account number of total identified risks number of newly identified risks Fig. 2 The chart in Fig 3 below is an interesting one that shows risks identified clearly trending down over 14 days. This is indicating that the teams are remediating identified issues in the Evident.io ESP alerts, and you clearly see an improvement in the security posture of this very large AWS environment that has 1000s of instances. Note: There are almost no high severity risks in this environment. Fig. 3 Is my data secure? These two platforms do an awesome job of securing your data both in flight and in transit, with both using TLS 1.2 encryption for in flight data and customer specific 256 bit AES encryption keys for at rest data. You can be confident that this data is securely transported from the Evident Security Platform to Sumo Logic and stored in a secure fashion. How can I gain access? This integration relies on the use of AWS SNS (Simple Notification Service) and a Sumo Logic native https collector. If you are both an Evident.io and Sumo Logic customer you can enable and start to benefit from the integration using the directions here: http://help.sumologic.com/Special:Search?qid=&fpid=230&fpth=&path=&search=evident.io or http://docs.evident.io/#sumo. Note you will need to have access to both Evident.io and Sumo Logic instances. Security and compliance monitoring are no longer a bottleneck in your agile environment. You can start visualizing the data from Evident Security Platform (ESP) in Sumo Logic in a matter of minutes. This blog post was written by Hermann Hesse, Senior Solutions Architect at Evident.io. He can be reached at https://www.linkedin.com/in/hermann-hesse-a040281

Consider a situation in which you’ve developed a groundbreaking website, and you would like to share your content with the world. The problem is that your hosting provider is based in New York, and you’re concerned that a user from a different region, such as Europe or Australia, won’t be able to view your content quickly and reliably. Amazon CloudFront is a content delivery network (CDN) solution that allows your content distribution to be shared in an accelerated manner from multiple edge locations around the world.

AWS is one of the greatest disruptive forces in the entire enterprise technology market. Who would have thought when they launched in 2006, it was going to kick off perhaps the most transformative shift in the history of the $300B data center industry. Over 25,000 people (or 0.0003% of the World’s population) are descending on Vegas this week to learn more about AWS, the biggest supercomputer in the world. As we get ready to eat, drink, network and learn, I wanted to provide some responses to inquiries I often get from prospects, reporters and folks who I meet at various conferences around the country. What advice would you pass on to anyone deciding to use AWS for public cloud storage? Understand the IaaS provider’s shared security model. In Amazon’s case, AWS is responsible for the infrastructure. The customer is responsible for the security of everything that runs on that infrastructure – The applications, the workloads and the data. Make sure any additional service you use on top of that have pursue their own security certifications and attestations to protect data at rest and in motion. This will allay fears and give people comfort in sending data through a SaaS-based service. We find that organizations are making different decisions based on the trust level they have with their partners, and we at Sumo Logic take this very seriously investing millions to achieve and maintain on an ongoing basis, these competitive differentiators. Too many people try to live vicariously through the certifications AWS has and pass this on as adequate Understand the benefits you are hoping to achieve before you start (i.e. Better pricing / reduced cost; Easier budget approvals (CAPEX vs. OPEX); Increase Business Agility; Increase flexibility and choices of what programming models, OS, DB and architectures make sense for the business; Increased security; Increased workload scalability / elasticity, etc.) How can we maximize AWS’s value? Crawl, walk, run – it is a learning curve that will take time to master. Adopt increasing levels of services as your teams get up to speed and understands how to leverage APIs and automate everything through code. Compute as code is now a reality. Understand the pain points you are trying to address – this will dictate approach (i.e. Pricing / Cost / Budget; Internal Politics; Control of Data Locality; Sovereignty; Security; Compliance, etc.) Turn on logging within AWS. More specifically, activate Amazon CloudWatch to log all your systems, applications and services and activate AWS CloudTrail to log all API actions. This will provide visibility into all user actions on AWS. The lack of visibility into cloud operations and controls stands as the largest security issue we see. What cautions might there be in terms of how to end up paying more than one should or not really getting full value out of this type of storage? Understand not all data is created equal…in terms of importance, frequency of access, life expectancy of the data, retention requirements, and search performance. Compare Operational data (high importance, high frequency of access, short life expectancy, high search performance requirements) to audit data (medium importance, lower frequency of access, longer life expectancy/data retention requirements, low performance requirements) Align your storage needs to the value and urgency of the data that you are logging (S3, S3 Infrequent Access, Glacier, EBS, etc.) Look for solutions and tools that are cloud native, so you can avoid unnecessary data exfiltration costs. 10 years ago, no one was virtualizing mission critical workloads because of Security and Compliance concerns…but we ended up there anyways. This is exactly the same thing for cloud. And in this new world, speed and time to market is everything. Organizations are looking to be more flexible, more agile, capitalize on business opportunities, and how you approach security is different. And to support the rapid pace of delivery of these digital initiatives – weekly, even daily – these companies are leveraging modern, advanced IT infrastructures like AWS and Sumo Logic. In this new world, we at Sumo Logic have a tremendous opportunity to help operations and security professionals get the visibility they need as those workloads are moved out to the cloud. We help them become cloud enablers, to help drive the business forward, not being naysayers. Visibility is everything! Come stop by our booth – #604 – and say hi!

Every company – if they are going to remain relevant – is going through some form of digital transformation today and software is at the heart of this transformation. According to a report by the center for digital business transformation, the digital disruption will displace approximately 40% of incumbent companies within the next 5 years. Don’t believe it? According to Forrester Research, between 1973 and 1983, 35% of the top 20 F1000 companies were new. Now jump forward 20 years, and this number increases to 70%. According to predictions from IDC’s recent FutureScape for Digital Transformation, two-thirds of Global 2000 companies will have digital transformation at the center of their corporate strategy by next year, and by 2020, 50% of the Global 2000 will see the majority of their business depend on their ability create digitally-enhanced products, services, and experiences. So what does this all mean? Keeping pace with the evolving digital marketplace requires not only increased innovation, but also updated systems, tools, and teams. Accenture and Forrester Research reported in their Digital Transformation in the Age of the Customer study that only 26% of organizations considered themselves fully operationally ready to execute against their digital strategies. In order to deliver on the promise of digital transformation, organizations must also modernize their infrastructure to support the increased speed, scale, and change that comes with it. We see three characteristics that define these modern applications and digital initiatives: They follow a DevOps or DevSecOps culture, where the traditionally siloed walls between the Dev, Ops and Security teams are becoming blurred, or go away completely. This enables speed, flexibility and agility. They are generally running on modern infrastructure platforms like AWS (see AWS Modern Apps Report), leveraging APIs and compute as code (see AWS – The Largest Supercomputer in the World) The way you approach security needs to change. You need deep visibility & native integrations across the AWS services that are used, you need to understand your risks and security vulnerabilities, you need to connect the dots between the services used, and understand what the users are doing, where are they coming from, what are they changing, what are the relationship of those changes, how this impacts network flows and security risks. And it is important to be able to match information contained in your AWS log data – i.e. IP Address, Ports, UserIDs, etc – from services like CloudTrail and VPC Flow Logs, with known Indicators of Compromise (IOCs) that are out there in the wild from premium threat intelligence providers like Crowdstrike. Pulling in global threat intelligence into Sumo Logic’s Next Gen Cloud Security Analytics for AWS accomplishes the following: Increases velocity & accuracy of threat detection Adds additional content to log data and helps to identify and visualize malicious IP addresses, domain names, ports, email addresses, URLs, and more. Improve security and operational posture through accelerated time to identify and resolve security threats (IOC) Come stop by our booth – #604 – for a demo and say hi!

Many folks we speak to ask the question: “How do I get started in AWS?” The answer used to be simple. There was a single service for compute, storage, and a few others services in early trials. Fast forward 10+ years and AWS now offers over 50 services. Taking your first steps can be daunting. What follows is my recommended approach if you’re starting fresh in the AWS Cloud and don’t have a lot of legacy applications and deployments weighing you down. If that is you, check out the companion post to this one. Do Less To Do More Everything in AWS operates under a Shared Responsibility Model. The model simple states that for each of the areas required day-to-day operations (physical, infrastructure, virtualization, operation system, application, and data), someone is responsible. That someone is either you (the user) or AWS (the service provider). Light grey options are the responsibility of AWS, Black are the user’s The workload shifts towards the service provider as you move away from infrastructure services (like Amazon EC2) towards abstract services (like AWS Lambda). As a user, you want AWS to do more of the work. This directs your service choice as you start to build in the AWS Cloud. You want to pick more and more of the services that fall under the SaaS or abstract — which is a more accurate term when compared to SaaS — category. Computation If you need to run your own code as part of your application, you should be making your choice based on doing less work. This means starting with AWS Lambda, a service that runs your functions directly without worrying about the underlying frameworks or operating system. If Lambda doesn’t meet your needs, try using a Docker container running on the Amazon EC2 Container Service (ECS). The advantage of this service is that it configures the underlying EC2 instance (the OS, Docker host, scheduling, etc.) and lets you simply worry about the application container. If ECS can’t meet your needs, see if you’re a fit for AWS Elastic Beanstalk. This is a service that takes care of provisioning, capacity management, and application health for you (a/k/a you do less work). All of this runs on top of Amazon EC2. So does Lambda and ECS for that matter. If all else fails, it’s time to deploy your own instances directly in Ec2. The reason you should try to avoid this as much as possible is the simple fact that you’re responsible for the management of the operating system, any applications your install, and — as always — your data. This means you need to keep on top of patching your systems, hardening them, and configuring them to suit your needs. The best approach here is to automate as much of this operational work as possible (see our theme of “do less” repeating?). AWS offers a number of services and features to help in this area as well (start with EC2 AMIs, AWS CodeDeploy, AWS CodePipeline, and AWS OpsWorks). Data Storage When it comes to storing your data, the same principle applies; do less. Try to store you data in services like Amazon DynamoDB because the entire underlying infrastructure is abstracted away for you. You get to focus purely on your data. If you just need to store simple file object, Amazon S3 is the place to be. In concert with Amazon Glacier (long term storage), you get the simplest version of storage possible. Just add an object (key) to a bucket and you’re all set. Under the covers, AWS manages all of the moving parts in order to get you 11 9’s of durability. This means that about 0.000000001% of objects stored in the service may experience data corruption. That’s a level of quality that your simply cannot get on your own. If you need more control or custom configurations, other services like the Amazon Elastic File System or EBS volumes in EC2 are available. Each of these technologies comes with more operational overhead. That’s the price you pay for customization. Too Many Services Due to the shear number of services that AWS provides, it’s hard to get a handle on where to start. Now that you know your guiding principle, it might be worth looking at the AWS Application Architecture Center. This section of the AWS site contains a number of simple reference architectures that provide solutions to common problems. Designs for web application hosting, batch processing, media sharing, and others are all available. These designs give you an idea of how these design patterns are applied in AWS and the services you’ll need to become familiar with. It’s a simple way to find out which services you should start learning first. Pick a design that meets your needs and start learning the services that the design is composed of. Keep Learning AWS does a great job of providing a lot of information to help get you up to speed. Their “Getting Started with AWS” page has a few sample projects that you can try under the free tier. Once you start to get your footing, the whitepaper library is a great way to dive deeper on certain topics. In addition, all of the talks from previous Summits (one to two day free events) and AWS re:Invent (the major user conference) are available for viewing on the AWS YouTube channel. There are days and days of content for you to watch. Try to start with the most recent material as a lot of the functionality has changed over the years. But basic, 101-type talks are usually still accurate. Dive In There is so much to learn about AWS that it can be paralyzing. The best advice I can give is to simply dive in. Find a simple problem that you need to solve, do some research, and try it out. There is no better way to learn than doing. Which leads me to my last point, the community around AWS is fantastic. AWS hosts a set of very active forums where you can post a question and usually get an answer very quickly. On top of that the usual social outlets (Twitter, blogs, etc.) are a great way to engage with others in the community and to find answers to your pressing questions. While this post has provided a glimpse of where to start, be sure to read the official “Getting Started” resources provided by AWS. There’s also a great community of training providers (+ the official AWS training) to help get you up and running. Good luck and happy building! This blog post was contributed by Mark Nunnikhoven, Vice President, Cloud Research at Trend Micro. Mark can be reached at https://ca.linkedin.com/in/marknca.

Getting started in AWS used to be simple. There was a single service for compute, storage, and a few others services in early trials. Fast forward 10+ years and AWS now offers over 50 services. Taking your first steps can be daunting. What follows is my recommended approach if you already have a moderate or large set of existing applications and deployments that you have to deal with and want to migrate to the AWS Cloud. If you’re starting fresh in the AWS Cloud, check out the companion post to this one. Do Less To Do More Everything in AWS operates under a Shared Responsibility Model. The model simple states that for each of the areas required day-to-day operations (physical, infrastructure, virtualization, operation system, application, and data), someone is responsible. That someone is either you (the user) or AWS (the service provider). Light grey options are the responsibility of AWS, Black are the user’s The workload shifts towards the service provider as you move away from infrastructure services (like Amazon EC2) towards abstract services (like AWS Lambda). As a user, you want AWS to do more of the work. This should direct your service choice as you start to build in the AWS Cloud. Ideally, you want to pick more and more of the services that fall under the SaaS or abstract — which is a more accurate term when compared to SaaS — category. But given your existing constraints, that probably isn’t possible. So you need to start where you can see some immediate value, keeping in mind that future project should aim to be “cloud native”. Start Up The Forklift The simplest way to get started in AWS under legacy constraints is to forklift an existing application from your data centre into the AWS Cloud. For most applications, this means you’re going to configure a VPC, deploy a few EC2 instances, an RDS instance (ideally as a Multi-AZ deployment). To make sure you can expand this deployment, leverage a tool like AWS OpsWorks to automate the deployment of the application on to your EC2 instances. This will make it a lot easier to repeat your deployments and to manage your Amazon Machine Images (AMIs). Migrating your data is extremely simple now as well. You’re going to want to use the AWS Database Migration Service to move the data and the database configuration into RDS. Second Stage Now that your application is up and running in the AWS Cloud, it’s time to start taking advantage of some of the key features of AWS. Start exploring the Amazon CloudWatch service to monitor the health of your application. You can set alarms to warn of network bandwidth constraints, CPU usage, and when the storage space on your instances starts to get a little cramped. With monitoring in place, you can now adjust the application’s configuration to support auto scaling and to sit behind a load balancer (either the classic ELB or the new ALB). This is going to provide some much needed resiliency to your application. It’s automated so you’re going to start to realize some of the benefits of AWS and reduce the operational burden on your teams at the same time. These few simple steps have started your team down a sustainable path of building in AWS. Even though these features and services are just the tip of the iceberg, they’ve allowed you to accomplish some very real goals. Namely having a production application working well in the AWS Cloud! On top of that, auto scaling and CloudWatch are great tools to help show teams the value you get by leveraging AWS services. Keep Going With a win under your belt, it’s a lot easier to convince teams to build natively in AWS. Applications that are build from the ground up to take advantages of abstract services in AWS — like Amazon Redshift, Amazon SQS, Amazon SNS, AWS Lambda, and others — will let you do more for your users with less effort on your part. Teams with existing constraints usually have a lot of preconceived notions of how to build and deliver IT services. To truly get the most out of AWS, you have to adopt a new approach to building services. Use small wins and a lot of patience to help convince hesitant team members that this is the best way to move forward. Too Many Services Due to the shear number of services that AWS provides, it’s hard to get a handle on where to start. Now that you know your guiding principle, it might be worth looking at the AWS Application Architecture Center. This section of the AWS site contains a number of simple reference architectures that provide solutions to common problems. Designs for web application hosting, batch processing, media sharing, and others are all available. These designs give you an idea of how these design patterns are applied in AWS and the services you’ll need to become familiar with. It’s a simple way to find out which services you should start learning first. Pick a design that meets your needs and start learning the services that the design is composed of. Keep Learning AWS does a great job of providing a lot of information to help get you up to speed. Their “Getting Started with AWS” page has a few sample projects that you can try under the free tier. Once you start to get your footing, the whitepaper library is a great way to dive deeper on certain topics. In addtion, all of the talks from previous Summits (one to two day free events) and AWS re:Invent (the major user conference) are available for viewing on the AWS YouTube channel. There are days and days of content for you to watch. Try to start with the most recent material as a lot of the functionality has changed over the years. But basic, 101-type talks are usually still accurate. Dive In There is so much to learn about AWS that it can be paralyzing. The best advice I can give is to simply dive in. Find a simple problem that you need to solve, do some research, and try it out. There is no better way to learn than doing. Which leads me to my last point, the community around AWS is fantastic. AWS hosts a set of very active forums where you can post a question and usually get an answer very quickly. On top of that the usual social outlets (Twitter, blogs, etc.) are a great way to engage with others in the community and to find answers to your pressing questions. While this post has provided a glimpse of where to start, be sure to read the official “Getting Started” resources provided by AWS. There’s also a great community of training providers (+ the official AWS training) to help get you up and running. Good luck and happy building! This blog post was contributed by Mark Nunnikhoven, Vice President, Cloud Research at Trend Micro. Mark can be reached at https://ca.linkedin.com/in/marknca.

Microservices have become mainstream in building modern architectures. But how do you actually develop an effective microservices architecture? This post explains how to build an optimal microservices environment by adhering to the following five principles: Cultivate a Solid Foundation Begin With the API Ensure Separation of Concerns Production Approval Through Testing Automate Deployment and Everything Else Principle 1: Great Microservices Architecture is Based on a Solid Foundation No matter how great the architecture, if it isn’t based on a solid foundation, it won’t stand the test of time. Conway’s law states that “…organizations that design systems … are constrained to produce designs which are copies of the communication structures of these organizations…” Before you can architect and develop a successful microservice environment, it is important that your organization and corporate culture can nurture and sustain a microservice environment. We’ll come back to this at the end, once we’ve looked at how we want to design our microservices. Principle 2: The API is King Unless you’re running a single-person development shop, you’ll want to have an agreed-upon contract for each service. Actually, even with a single developer, having a specific set of determined inputs and outputs for each microservice will save you a lot of headaches in the long run. Before the first line of code is typed, determine a strategy for developing and managing API documents. Once your strategy is in place, focus your efforts on developing and agreeing on an API for each microservice you want to develop. With an approved API for each microservice, you are ready to start development, and begin reaping the benefits of your upfront investment. Principle 3: Separation of Concerns Each microservice needs to have and own a single function or purpose. You’ve probably heard of separation of concerns, and microservices are prime examples for the application of that principle. Additionally, if your microservice is data-based, ensure that it owns that data, and exists as the sole access point for that data. As additional requirements come to light, it can be very tempting to add an additional endpoint to your service that kind of does the same thing (but only kind of). Avoid this at all costs. Keep your microservices focused and pure, and you’ll avoid running into the nightmare of trying to remember which service handled that one obscure piece of functionality. Principle 4: Test-Driven Approval Back in the old days, when you were supporting a large monolithic application, you’d schedule your release weeks or months in advance, including an approval meeting which may or may not have included a thumbs up/thumbs down vote, or fists of five to convey approval or confidence in the new release. With microservice architecture, that changes. You’re going to have a significant number of much smaller applications, and if you follow the same release process, you’ll be spending a whole lot more time in meetings. Therefore, if you’re implementing test-driven development (TDD), writing comprehensive contract and integration tests as you develop each application, you’ll finish your service up with a full test suite which you can automate as part of your build pipeline. Use these tests as the basis for your production deployment approval process, rather than relying on the approval meetings of yore. Principle 5: Automate, Automate, Automate As developers and engineers, we’re all about writing code which can automate and simplify the lives of others. Yet, too often, we find ourselves trapped in a world with manual deployments, manual testing and manual approval processes and change management. Automating these processes when it comes to microservices is less a convenience and more of a necessity, especially as your code base and repertoire of microservices expands and matures. Automate your build pipelines so that they trigger as soon as code is merged into the master branch. Automate your tests, static code analysis, security scans and any other process which you run your code through, and then on condition of all checks completing successfully, automate the deployment of the updated microservice into your environment. Automate it all! Once your microservice is live, ensure that you have configured a means by which the service can be automatically configured as well. An automated configuration process has the added benefit of making it easier to both troubleshoot and remember where it was that you set that “obscure” property for that one microservice. Conclusion: Back to the Foundation At the start of this article, I mentioned Conway’s law, and said I’d come back to talking about what kind of organization you need in order to facilitate successful microservice development. This is what your organization should look like: Each team should have a specific aspect of the business on which to focus, and be assigned development of functionality within that area. Determine the interfaces between teams right up front. The key is to encourage active and engaging communication and collaboration between the teams. Doing so will help to avoid broken dependencies and help to ensure that everyone remains focused. Empower teams with the ability to make their own decisions and own their services. Common sense may dictate some basic guidelines, but empowering teams is a lot like automating processes, and it’ll both fuel innovation and make your life easier. Now, with all that said, you probably don’t have the luxury of building a brand new development department from the ground up. More likely, you’re going to be trying to implement changes to a group of folks who are used to doing things in a different way. Just as with microservices, or any software project for that matter, focus on incremental improvements. Determine an aspect of the culture or the organization that you’d like to change, determine how to implement that change, how to measure success, and test it out. When you achieve success, pick another incremental change to work on, and when you fail, try a different approach or a different aspect of the business to improve. About the Author Mike Mackrory is a Global citizen who has settled down in the Pacific Northwest – for now. By day he works as a Senior Engineer on a Quality Engineering team and by night he writes, consults on several web based projects and runs a marginally successful eBay sticker business. 5 Patterns for Better Microservices Architecture is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

The founders of Sumo Logic recognized early on when they founded the company that in order to remain competitive in an increasingly disruptive world, companies would be moving to the cloud to build what are now being called modern apps. Hence, Sumo Logic purposefully architected the Sumo Logic machine data analytics platform from the ground up on Amazon Web Services. Along the way, Sumo Logic has acquired a vast knowledge and expertise in not only log management overlaid with metrics, but in the inner workings of the services offered by Amazon. Today, more 6 years later, we are pleased to announce that Sumo Logic is one of a handful of initial AWS partners participating in the launch of SaaS Subscription products on Amazon Web Services (AWS) Marketplace, and the immediate availability of the Sumo Logic Platform in AWS Marketplace. Now, customers already using AWS can leverage Sumo Logic’s expertise in machine-data analytics to visualize and monitor workloads in real-time, identify issues and expedite root cause analysis to improve operational and security insights across AWS infrastructure and services. How it Works AWS Marketplace is an online store that allows AWS customers to procure software and services directly in the marketplace and immediately start using those services. Billing runs through your AWS account, allowing your organization to consolidate billing for all AWS services, SaaS subscriptions and software purchased through the Marketplace. To get started with Sumo Logic in the AWS Marketplace go to the Sumo Logic page. You should see a screen similar to the following. Pricing As you can see, pricing is clearly marked next to the product description. Pricing is based on several factors starting with which edition of Sumo Logic you’re using – Professional or Enterprise. Professional edition supports up to 20 users and 30 days of data retention among other features. Enterprise Edition includes support for 20+ users and multi-year data retention as part of its services. See Sumo Logic Pricing page for more information. Reserved Log Ingest Once you’ve decided which edition, you’re ready to select the plan that’s best for you based on your anticipated ingest volume. Reserved Log Ingest Volume is the amount of logs you have contracted to send each day to the Sumo Logic service. The Reserve price is how you much you pay for GB of logs ingested each day. During signup, you can select a Reserved capacity in GB’s per day (see below). There are no minimum days, and you can cancel at any time. On-Demand Log Ingest Bursting is allowed and at the end of the billing cycle, for any usage beyond the total Reserved capacity for the period, you will pay the On-demand rate. Your first 30 days of service usage are FREE. Signing up When you click Continue, you’ll be taken to the Sumo Logic signup form similar to Figure 2. Enter your email address, then click Plan to select your Reserved Log Ingest volume. At this point you will select your Reserved capacity. Plans are available for increments of 1, 3, 5, 10, 20, 30, 40 and 50 GB per day. Once you’ve selected your plan, click the signup button to be taken through the signup process. Recall, billing is managed through AWS so no credit card required. What You Get If you’re not already familiar with the Sumo Logic, the platform unifies logs, metrics and events, transforming a variety of data types into real-time continuous intelligence across the entire application lifecycle enabling organizations to build, run and secure their modern applications. Highlights of Sumo Logic include: Unified Logs and Metrics Machine learning capabilities like LogReduce and LogCompare, machine learning features to quickly identify root cause. Elasticity and bursting support without over-provisioning Data encryption at rest, PCI DSS 3.1 with log immutability, and HIPAA compliance at no additional cost. Zero log infrastructure Management overhead. Go to sumologic.com for more information Sumo Logic Apps for AWS As mentioned, Sumo Logic has tremendous expertise in AWS, and experience building and operating massively multi-tenant, highly distributed cloud systems. Sumo Logic passes that expertise along to its customers in the form of Apps for AWS services. Sumo Logic Apps for AWS contain preconfigured searches and dashboards for the most common use cases, and are designed to accelerate your time to value with Sumo Logic. Using these dashboards and searches you can quickly get an overview of your entire AWS application at the app, system and network levels. You can quickly identify operational issues, drill down using search and apply tools like LogReduce and LogCompare to quickly get at the root cause of the problem. You also gain operational, security and business insight into services that support your app like S3, CloudTrail, VPC Flow and Lambda. Apps that are generally available include: Amazon S3 Audit App Amazon VPC Flow Logs App Amazon CloudFront App AWS CloudTrail App AWS Config App AWS Elastic Load Balancing App AWS Lambda App In addition, the following Apps are in Preview for Sumo Logic Customers: Amazon CloudWatch – ELB Metrics Amazon RDS Metrics Getting Started and Next Steps Sumo Logic is committed to educating its customers using the deep knowledge and expertise it has gained in working with AWS. If you’re new to Amazon Web Services, we’ve created AWS Hub, a portal dedicated to learning AWS fundamentals. The portal includes 101’s to get you started with EC2, S3, ELB, VPC Flow and AWS Lambda. In addition you’ll find deep-dive articles and blog posts walking you through many of the AWS service offerings. Finally, if you’re planning to attend AWS re:Invent at the end of November, stop by and get your questions answered, or take a quick tour of Sumo Logic and all machine learning and data analytics has to offer.

AWS re:Invent is one of the biggest events Sumo Logic partakes in each year. If you’re not familiar, it’s one of the hottest cloud shows with over 25,000 attendees, more than 400 sessions, hands on labs, boot camps and much more! We’ve been lucky to be apart of it for the last few years and it just seems to get bigger and better. In just a few short weeks we’ll be headed back to Vegas and have a few things up our sleeves! Here are some tips, tricks and all the fun activities that you can expect. As Scar from the Lion King would say… Be Prepared! Whether you’re a veteran of re:Invent or a newbie, you can never be too prepared to survive the week. Here’s some useful tips we’ve picked up along the way: Make sure to plan your schedule and session times in advance. This year AWS has introduced “Reserved Seating” for sessions so you can secure your spot! More details here. If you’re looking for a new t-shirt wardrobe make sure to visit the expo hall. Vendors basically throw them at you. But honestly, some really cool stuff goes down in there! Check out the re:Play party on Thursday night, it’s always a different setup and they announce the musical guest that day. Wonder who it’ll be? Hydrate, hydrate, hydrate and have fun! Expo Hall Madness The expo hall is a lively and happening place with over 400 vendors showcasing the best cloud services. Last year we had half a ton of sumo wrestlers that made quite a splash there and you can be sure that we’ll bring the same excitement to re:Invent this year. Visit our booth #604, to see how machine data analytics can give you the continuous intelligence you need to build, run and secure you modern applications. We will have a bunch of activities, plenty of swag, and live demos and face time with the whole Sumo crew! Learn from the Best If you’re looking to get practical advice on how developers and operators can leverage data analytics to gain critical information about their modern application then you should check out our session on Wednesday afternoon at 11:00am. Our own CTO, Christian Beedgen and Ben Newton, Principal Product Manager, along with Cloud Cruiser will take you in depth on Effective Application Data Analytics for Modern Applications. Make sure to reserve your seat! Get Social with Sumo! Whether or not you’re attending re:Invent, you can participate and follow all of the conversations from the event! We will be live blogging from relevant keynotes and sessions, posting videos, photos and hosting a fun #SumoAfterDark photo contest (think roaming gnome). Follow us on Twitter or LinkedIn for all the latest. Check out our AWS re:Invent page for more details.

Today we published Sumo Logic’s first "The State of The Modern App in AWS" report. We wanted to explore how modern applications differ from traditional applications: how are the modern app workloads run, what application components are used, and which types of application services are leveraged? This report is based on anonymized data from about 1,000 customers running modern application workloads in AWS and it examines the composition of those applications. Sumo Logic provides our customers with the operational and security visibility of their applications. As such, multiple millions events per second of live data are sent to our service from our customers’ full application stack: infrastructure, application components, custom app code, as well as variety of application services leveraged to manage the application. Fingerprint of this data then gives us unprecedented ability to answer questions about these applications. When we founded Sumo Logic, we decided to build a multi-tenant analytics service in order to be able to leverage the cross customer visibility in order to provide more value to each customer. Our architecture and technology behind it allows us to: Analyze anonymized statistics about technology usage to help our existing and future customers learn about latest technology trends emerging from enterprises leading the way to digital transformation (e.g. this report) Prioritize roadmap items in order to help our customers gain visibility into most commonly used technologies (e.g. our integration with Docker) Improve our machine learning algorithms by exposing them to larger data sets of same data types across multiple customers Derive meanings from events occurring across same sources across multiple customers to improve operational and security outcomes I am excited to have this report as it is a very tangible response to a question I very frequently get: what is the value of having a multi-tenant architecture? It is easy to quickly respond that we benefit from the same operating model as other all-in cloud services, such as AirBnB, Netflix, Hudl, etc. But this report also highlights another, and perhaps more important and unique value of multi-tenancy - it gives us the ability to analyze trillions of anonymized events across our customer community and convert them into value for each individual customer as they build, run, and secure their modern application. With Sumo Logic, you are not alone. We are excited to continue deriving insights that will help our customers create better outcomes for their business.

https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

Last week Sumo Logic rolled out several new enhancements to Scheduled Searches that will make it easier to continuously monitor your stack and receive notifications on critical events. Scheduled searches are just standard searches that have been saved, and are executed on a schedule you set. As you create your scheduled search, you can configure several different alert types including email, Script Action, ServiceNow Connection, Webhook, Save to Index, and Real-time Alerts. Once configured, the scheduled search runs continuously in the background and triggers the alert type you’ve selected. The new enhancements give you more control over the scheduling of searches, and allow you to customize how and when the search results are presented. This post introduces you to these new features and shows how you can make best use of them. If you’d like a personal tour through the new scheduled-search features, register for our webinar, “Optimize Your Scheduled Searches is Here” to be held on November 17. Specify the Scheduled Timezone When you schedule a search within Sumo Logic, the search is by default scheduled to run within the timezone set by the user’s preferences. Sumo Logic now allows you to specify a timezone other than your preferred timezone. When used in combination with the other scheduling options, your search will now run at the time and timezone set within the search. Global teams that use a standard timezone like UTC will find the timezone option especially helpful, allowing all members to see the same results and avoid confusion. This option also provides the user creating the scheduled search with the flexibility to make the email results relevant to the recipients that may be located anywhere in the world. Weekly Scheduling Option Answering the call for popular feature request, Sumo Logic has increased the scheduled-search frequency to allow for weekly scheduling and reporting. When you select the weekly option, you can select a specific day and time of the week for your search to run. Custom CRON Scheduling Option For cases where the default scheduling options just don’t fit your needed schedule, Sumo Logic now allows you to input a custom CRON expression to fully customize the scheduled day and time of your search. With the custom CRON option, you can schedule your search to run monthly on a set day and hour, certain days of the week, specific hours within a day, or any other combination you can think of. The custom CRON scheduling utilizes the Quartz CRON format. For more information on formatting your CRON expression, see Using a CRON Expression in Sumo Logic. Note: In order make sure your scheduled searches are run on time, Sumo Logic does not support scheduling searches with CRON at an interval less than 15 minutes. If you need a schedule that is less than 15 minutes, consider using the standard scheduling options of the search. Set a Custom Email Subject Another useful feature is that you can now specify the email subject for your Scheduled Search emails using a set of predefined placeholders along with your own text. This makes it easier to recognize, organize, and filter these emails within your inbox. Don’t want to update your existing email filters or Scheduled Searches? Don’t worry. We supply a default subject that matches the previous email format. Your existing searches will continue to use that subject until the owner of the search decides to change it. Select How You Want to Receive Results Display only the information you want to see within the Scheduled Search emails by selecting which sections to include. Show or hide the results histogram image, the search query string, or the summarized results set. Additionally, you can include up to the first 1000 results from your query attached as a .CSV file. Note that there’s a 5MB limit. Learn More About Scheduled Searches The new enhancements in Scheduled Search now allows you to create highly customized alert notifications so that you can customize your notification systems and work seamlessly with global teams. If you’d like to learn more about using Scheduled Searches, join us for our upcoming webinar, or visit Sumo Logic DocHub for complete product documentation.

Keeping track of all of your settings and configuration changes on AWS can be difficult if you try to do it by hand. Fortunately, AWS offers a service to automate the process called AWS Config. This article explains what AWS Config is, and show you how to get started using AWS Config rules. What is AWS Config? The official description of AWS Config reads as follows: AWS Config provides a detailed view of the configuration of AWS resources in your AWS account. This includes how the resources are related to one another and how they were configured in the past so that you can see how the configurations and relationships change over time. Reading a description like that will really start the wheels turning in any SysOps professional’s head. In the datacenter, many attempts were made, sometimes successfully, to manage inventory and changes. With an AWS account, there is a defined boundary in which to monitor. There is tagging on almost every resource type that can be provisioned, and everything that can be deployed fits neatly into a resource type definition. AWS Config ends up being an ideal tool to provide infrastructure visibility within an AWS account. Getting Started To begin setting up AWS Config, a tutorial is provided in the AWS Management Console. This tutorial walks the user through configuration items and establishes all components needed to begin using AWS Config. The setup allows the user to select resources in the region and configure global resources. An item worth noting is that AWS Config is set up on a region by region basis. The setup will also ask for an S3 bucket in which to store the AWS Config record. At the same time, an Amazon SNS topic can be set to send notifications and an IAM role can be applied to grant permission to interact with Amazon SNS and Amazon S3. The setup provides a number of suggested AWS Config template rules to select, including: Root-account-mfa-enabled – Ensures the root AWS account has multi-factor authentication enabled Cloudtrail-enabled – Checks if AWS CloudTrail (auditing of API calls) is enabled Eip-attached – Looks for unused Elastic IP addresses Encrypted-volumes – Checks to ensure that any EBS volumes that are in use are encrypted All four, at the present time, are very useful in their own way. It is recommended that all be added to the AWS account before confirming changes. Creating Rules Once confirmed, AWS Config will provide two primary navigation items along the left side of the screen—Rules and Resources. The Rules screen is used to configure—as one might expect, rules—to monitor configuration changes to Resources in the AWS account. AWS provides a healthy number of template rules, covering a number of primary use cases for using AWS Config. Three common template rules that should be enabled immediately include: Require-tags – Require specific tags on a resource. This is especially useful in preventing orphaned EC2 instances with unknown purposes. Restrict-SSH – Ensures SSH has not been enabled for a Security Group that should have permit inbound SSH requests Restricted-common-ports – Another rule to ensure common ports are not open to instances that should not be accessible. AWS Config can also leverage custom rules that can be configured in the same menu. Custom rules can be triggered either by a configuration change on the AWS account or on a periodic basis. A resource type or key value pair can be selected as a target of the rule. For even more complex rules logic, an AWS Lambda function can be invoked as well. Reviewing Configuration AWS Config provides two primary ways of consuming the configuration state of the AWS account—the Resources tab in the AWS Config console, and SNS notification. For periodic review, the Resources tab contains a couple of selectors that allow for searching either records associated with resources, or tags. This is also the screen where announcements are made about new resource types being supported in config. The other method is to configure AWS Simple Notification Service (SNS) to send an email or text message when a rule becomes non-compliant. One downside to either method is that it does not take steps to remediate the issue. That falls to the administrator who received the notification. Sumo Logic App for AWS Config AWS Config provides a great way to monitor how an AWS account is configured. For any account that has a lot of individuals, generating a large number of changes, AWS Config is an invaluable tool to provide continuous visibility into the infrastructure. The Sumo Logic App for AWS Config adds to this by presenting notifications containing snapshots of resource configurations and information about the modifications made to a resource. The app also provides predefined Live and Interactive Dashboards and filters that give you a greater level of visibility into your environment for real-time analysis of overall usage. If you don’t already have a Sumo Logic account, sign up for a Free Trial and take the app for a test drive. About the Author Over the last 10 years, Sara Jeanes has held numerous program management, engineering, and operations roles. She has led multiple team transformations and knows first-hand the pain of traditional waterfall development. She is a vocal advocate for DevOps, microservices, and the cloud as tools to create better products and services. Sara is currently a Contributor at Fixate.io and can be found on Twitter @sarajeanes. Getting Started with AWS Config Rules is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

When AWS Lambda was introduced in 2014, the first language to be supported was Node.js. When a Node.js function is deployed as an AWS Lambda, it allows for the execution of the function in a highly scalable, event-driven environment. Because the user has no control over the hardware or the systems which execute the function, however, the need for logging Node.js Lambda functions is of utmost importance when it comes to monitoring, as well as diagnosing and troubleshooting problems within the function itself. This post will consider the options for logging within a Node.js function. It will then briefly outline the process of uploading a Node.js function into a Lambda function, configuring the function to send the logs to Cloudwatch, and viewing those logs following the execution of the Lambda. Logging with Node.js Within a Node.js function, logging is accomplished by executing the appropriate log function on the console object. Possible log statements include: console.log() console.error() console.warn() console.info() In each case, the message to be logged is included as a String argument. If for example the event passed to the function included a parameter called ‘phoneNumber’, and we validated the parameter to ensure that it contained a valid phone number, we could log the validation failure as follows: console.error(“Invalid phone number passed in: “ + event.phoneNumber); Sometimes the best way to learn something is to play with it. Let’s create an example Lambda so we can see logging in action. First, we’ll need to set up the environment to execute the function. Configuring it all in AWS You’ll need access to an AWS environment in order to complete the following. All of this should be available if you are using the free tier. See Amazon for more information on how to set up a free account. Once you have an AWS environment you can access, and you have logged in, navigate to the Identity and Access Management (IAM) home page. You’ll need to set up a role for the Lambda function to execute in the AWS environment. On the IAM home page, you should see a section similar to the one shown below which lists IAM Resources (The number next to each will be 0 if this is a new account.) Click on the Roles label, and then click on the Create New Role button on the next screen. You’ll need to select a name for the role. In my case, I chose ExecuteLambda as my name. Remember the name you choose, because you’ll need it when creating the function. Click on the Next Step button. You’ll now be shown a list of AWS Service Roles. Click the Select button next to AWS Lambda. Filter: Policy Type box, type AWSLambdaExecute. This will limit the list to the policy we need for this role. Check the checkbox next to the AWSLambdaExecute policy as shown below, and then click on Next Step. You’ll be taken to a review screen, where you can review the new policy to ensure it has what you want, and you can then click on the Create Role button to create the role. Create Your Lambda Function Navigate to the Lambda home page at https://console.aws.amazon.com/lambda/home, and click on Create Lambda Function. AWS provides a number of sample configurations you can use. We’re going to be creating a very simple function, so scroll down to the bottom of the page and click on Skip. AWS also provides the option to configure a trigger for your Lambda function. Since this is just a demo, we’re going to skip that as well. Click on the Next button on the bottom left corner of the page. Now we get to configure our function. Under Configure function, choose a Name for your function and select the Node.js 4.3 Runtime. Under Lambda function code, choose Edit code inline from the Code entry type dropdown, and then enter the following in the Code Window. exports.handler = function(event, context) { console.log("Phone Number = " + event.phoneNumber); if (event.phoneNumber.match(/\d/g).length != 10) { console.error("Invalid phone number passed in: " + event.phoneNumber); } else { console.info("Valid Phone Number.") } context.done(null, "Phone number validation complete"); } Under Lambda function handler and role, set the Handler to index.handler, the Role to Choose an Existing role, and for Existing role, select the name of the role you created previously. Click on Next, review your entries and then click on Create function. Test the Function and View the Logs Click on the Test button, and enter the following test data. Click Save and Test and you should see the test results shown on the bottom of the screen. By default, Lambdas also send their logs to CloudWatch. If you click on the Monitoring tab, you can click on the View logs in CloudWatch link to view the logs in CloudWatch. If you return to the Lambda code page, you can click on Actions, select Configure test event and then enter different data to see how the result changes. Since the validator only checks to ensure that the phone number contains 10 numbers, you can simply try it with 555-1234 as the phone number and watch it fail. Please note that this example is provided solely to illustrate logging within a Node.js function, and should not be used as an actual validation function for phone numbers. About the Author Mike Mackrory is a Global citizen who has settled down in the Pacific Northwest – for now. By day he works as a Senior Engineer on a Quality Engineering team and by night he writes, consults on several web-based projects and runs a marginally successful eBay sticker business. When he’s not tapping on the keys, he can be found hiking, fishing and exploring both the urban and rural landscape with his kids. Always happy to help out another developer, he has a definite preference for helping those who bring gifts of gourmet donuts, craft beer and/or Single-malt Scotch. Logging Node.js Lambda Functions with CloudWatch is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

If you’re working with Puppet, there’s a good chance you’ve run into problems when configuring a diverse infrastructure. It could be a problem with authorization and authentication, or perhaps with MCollective, Orchestration or Live Management. Puppet logs can provide a great deal of insight into the status of your apps and infrastructure across the data center. Knowing where to look is just the first step. Knowing what to look for is another matter. Here’s a cheat sheet that will help you identify the logs that are most useful, and show you what to look for. I’ll also explain how to connect Puppet to a log aggregation and analytics tool like Sumo Logic. Where are Puppet Logs Stored? The Puppet Enterprise platform produces a variety of log files across its software architecture. This Puppet documentation describes the file path of the following types of log files: Master Logs: Master application logs containing information such as fatal errors and reports, warnings, and compilation errors. Agent Logs: Information on client configuration retrieved from the Master. ActiveMQ Logs: Information on the ActiveMQ actions on specific nodes. MCollective service logs: Information on the MCollective actions on specific nodes. Console logs: Information around console errors, fatal errors and crash reports. Installer logs: Contains information around Puppet installations, such as errors occurred installation, last installation run and other relevant information. Database logs: Information around database modifications, errors, etc. Orchestration logs: Information around orchestration changes. The root of Puppet log storage is different depending on whether Puppet is running in a Unix-like system or in a Windows environment. For *nix-based installs, the root folder for Puppet is /etc/puppetlabs/puppet. For Windows-based installs the root folder for Puppet is: C:\ProgramData\PuppetLabs\puppet\etc for all versions of Windows server from 2008 onwards. Modifying Puppet Log Configuration The main setting that needs to be configured correctly to get the best from Puppet logs is the log_level attribute within the main Puppet configuration file. The log_level parameter can have the following values, with “notice” being the default value: debug info notice warning err alert emerg crit The Puppet Server can also be configured to process logs. This is done using the Java Virtual Machine Logback library. An .xml file is created, usually named logback.xml, which can be piped into the Puppet Server at run time. If a different filename is used, it will need to be specified in the global.conf file. The .xml file allows you to override the default root logging level of ‘info’. Possible levels are trace, debug, info and debug. For example, if you wanted to produce full debug data for Puppet, you would add the following parameter to the .xml file. <root level=”debug”> The Most Useful Puppet Logs Puppet produces a number of very useful log files, from basic platform logs to full application orchestration reports. The most commonly used Puppet logs include: Platform Master Logs These give generalized feedback on issues such as compilation errors, depreciation warnings, and crash/fatal termination. They can be found at the following locations: /var/log/puppetlabs/puppetserver/puppetserver.log /var/log/puppetlabs/puppetserver/puppetserver-daemon.log Application Orchestration Logs Application orchestration is probably the single most attractive aspect of the Puppet platform. It enables the complete end-to-end integration of the DevOps cycle into a production software application. As a result, these logs are likely to be the most critical logs of all. They include: /var/log/pe-mcollective/mcollective.log – This log file contains all of the log entries that affect the actual MCollective platform. This is a good first place to check if something has gone wrong with application orchestration. /var/lib/peadmin/.mcollective.d/client.log – a log file to be found on the client connecting to the MCollective server, the twin to the log file above, and the second place to begin troubleshooting. /var/log/pe-activemq/activemq.log – a log file that contains entries for ActiveMQ. /var/log/pe-mcollective/mcollective-audit.log – a top-level view of all MCollective requests. This could be a good place to look if you are unsure of exactly where the problem occurred so that you can highlight the specific audit event that triggered the problem. Puppet Console Logs Also valuable are Puppet logs, which include the following: /var/log/pe-console-services/console-services.log – the main console log that contains entries for top-level events and requests from all services that access the console. /var/log/pe-console-services/pe-console-services-daemon.log – low-level console event logging that occurs before the standard logback system is loaded. This is a useful log to check if the problem involves the higher level logback system itself. /var/log/pe-httpd/puppet-dashboard/access.log – a log of all HTTPS access requests made to the Puppet console. Advanced Logging Using Further Tools The inbuilt logging functions of Puppet mostly revolve around solving issues with the Puppet platform itself. However, Puppet offers some additional technologies to help visualize status date. One of these is the Puppet Services Status Check. This is both a top-level dashboard and a queryable API that provides top-level real-time status information on the entire Puppet platform. Puppet can also be configured to support Graphite. Once this has been done, a mass of useful metrics can be analyzed using either the demo Graphite dashboard provided, or using a custom dashboard. The ready-made Grafana dashboard makes a good starting point for measuring application performance that will affect end users, as it includes the following metrics by default: Active requests – a graphical measure of the current application load. Request durations – a graph of average latency/response times for application requests. Function calls – graphical representation of different functions called from the application catalogue. Potentially very useful for tweaking application performance. Function execution time – graphical data showing how fast specific application processes are executed. Using Puppet with Sumo Logic To get the very most out of Puppet log data, you can analyze the data using an external log aggregation and analytics platform, such as Sumo Logic. To work with Puppet logs on Sumo Logic, you simply use the Puppet module for installing the Sumo Logic collector. You’ll then be able to visualize and monitor all of your Puppet log data from the Sumo Logic interface, alongside logs from any other applications that you connect to Sumo Logic. You can find open source collectors for Docker, Chef, Jenkins, FluentD and many other servers at Sumo Logic Developers on Github. About the Author Ali Raza is a DevOps consultant who analyzes IT solutions, practices, trends and challenges for large enterprises and promising new startup firms. Troubleshooting Apps and Infrastructure Using Puppet Logs is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

This week we deployed our new Github pages site, Sumo Logic Developers, to showcase some of the cool open-source projects that are being built in and around Sumo Logic. The goal is to share the knowledge and many projects that are being built by our customers, partners and everything in the wild. Aside from the official Sumo Logic repositories, there’s also a lot of tribal knowledge: Our Sales Engineers and Customer Success teams work daily with our customers to solve real-world problems. Often those solutions include search queries, field extraction rules, useful regular expressions, and configuration settings that are often captured in Github and other code registries. Sumo Logic Developers aggregates this knowledge in one place under Sumo Experts. Likewise, we’re seeing more organizations creating plugins and otherwise integrating Sumo Logic into to their products using our REST APIs. So under Integrations you’ll find a Jenkins Publisher plugin, along with tools for working with Twillio, Chef, DataDog, FluentD and many others. Sumo Logic Developers also provides everything our customers need to collect and ingest data including APIs and documentation, educational articles and walkthroughs from our Dev Blog, and links to all of our supported apps and integrations. The project was an enjoyable break from daily tasks and for those considering building a Github Pages site, I’d like to share my experience in building Sumo Logic Developers. If you’d like to contribute or add your project or repo to one of the lists on Sumo Logic Developers simply reply to this thread on our community site, Sumo Dojo. Alternatively, you can ping me on Twitter @codejournalist. Deploying Sumo Logic Developers on Github Pages Sumo Logic Developers is built on Github pages using Jekyll as a static site generator. Github pages are served from a special repository associated with your User or Organization account on Github. So all the code is visible unless you make it private. Github pages supports Jekyll natively so it’s easy to run builds on your local machine and review changes on a localhost before deploying to Github. Using Bootstrap I used the Bootstrap JavaScript framework to make the site responsive and mobile friendly. In particular Bootstrap CSS uses a grid system that uses rows and up to 12 columns. The grid scales as the device or viewport size increases and decreases. It includes predefined classes for easy layout options, so I can specify a new row like this: <div class=”row”> Within the row, I can designate the entire width of the screen like this: <div class=”col-xs-12 col-sm-12 col-md-12 col-lg-12 hidden-xs”> This create a full screen area using all 12 columns. Note that Bootstrap CSS lets you specify different sizes for desktop, laptop, tablets and mobile phones. This last example hides the content in this on extra small mobile screens. The following creates an area of 4 units on large and medium screens (most desktops and laptops, 6 units on small screens (most tablets) 8 units on phone devices. <div class=”col-lg-4 col-md-4 col-sm-6 col-xs-8″> Using Jekyll and Git Jekyll is a static site generator that includes the Liquid template engine to create data-driven sites. So using Liquid’s template language I was able to populate open-source projects into the site using data read from a Yaml file. This feature allows our internal teams to quickly update the site without having to muck in the code. A typical entry looks like: - title: "Sumo Report Generator" repo_description: "Tool that allows a user to execute multiple searches, and compile the data into a single report." url: "https://github.com/SumoLogic/sumo-report-generator" We use a Github workflow, so team members can simply create a new branch off of the master, edit the title, repo_description and url, then make a pull request to merge the request. You can find all of the source code at https://github.com/SumoLogic/sumologic.github.io Contributing to the Github Pages Site At it’s heart, Sumo Logic Developers is really a wiki – it’s informative with an amazing amount of rich content, it’s based on the work of our community of developers and DevOps practitioners, and it operates on a principle of collaborative trust. It’s brimming with tribal knowledge – the lessons learned from those who’ve travelled before you. In fact, the site currently represents: 30 Official Sumo Logic repositories ~ 20 Individually maintained repos by our customers, as well as SE, Customer Success and Engineering teams. 100+ Gists containing scripts, code snippets, useful regular expressions, etc. 3rd-party integrations with Jenkins, FluentD, Chef, Puppet and others. 100+ Blogs Links to our community, API’s and documentation Our main task now is finding new projects and encouraging members like you to contribute. The goal is to empower the tribe – all of those who use Sumo Logic to build, run and secure their modern applications. If you’d like to contribute, post your project on Sumo Dojo, or ping us @sumologic. Michael is the Head of Developer Programs at Sumo Logic. You can follow him on Twitter @codejournalist. Read more of Michael’s posts on Sumo Logic’s DevOps Blog.

What Happened? On Friday October 21st, Dyn, a major DNS provider, started having trouble due to a DOS attack. Many companies including PagerDuty, Reddit, Twitter, and others suffered significant downtime. Sumo Logic had a short blip of failures, but stayed up, allowing our customers to continue to seamlessly use our service for monitoring and troubleshooting within their organizations. How did Sumo Logic bear the outage? Several months ago, we suffered a DNS outage and had a postmortem that focused on being more resilient to such incidents. We decided to create a primary-secondary setup for DNS. After reading quite a bit about how this should work in theory, we implemented a solution with two providers: Neustar and Dyn. This setup saved us during today’s outage. I hope you can learn from our setup and make your service more resilient as well. How is a primary-secondary DNS setup supposed to work? You maintain the DNS zone on the primary only. Any update to that zone gets automatically replicated to the secondary via two methods: A push notification from the primary and a periodic pull from the secondary. The two providers stay in sync and you do not have to worry about maintenance of the zone. Your registrar is configured with nameservers from both providers. Order does NOT matter. DNS Resolvers do not know which nameservers are primary and which are secondary. They just choose between all the configured nameservers. Most DNS Resolvers choose which name server to use based on latency of the prior responses. The rest of the DNS Resolvers choose at random. If you have 4 nameservers with 1 from one provider and 3 from another, the more simplistic DNS Resolvers will split traffic 1/4 to 3/4, whereas the ones that track latency will still hit the faster provider more often. When there is a problem contacting a nameserver, DNS Resolvers will pick another nameserver from the list until one works. How to set up a primary-secondary DNS? Sign up for two different companies who provide high-speed DNS services and offer primary/secondary setup. My recommendation is: NS1, Dyn, Neustar (ultradns) and Akamai. Currently Amazon’s Route53 does not provide transfer ability and therefore cannot support primary/secondary setup. ( You would have to change records in both providers and keep them in sync.) Slower providers will not take on as much traffic as fast ones, so you have to be aware of how fast the providers are for your customers. Configure one to be primary. This is the provider who you use when you make changes to your DNS. Follow the primary provider’s and secondary provider’s instructions to set up the secondary provider. This usually involves configuring whitelisting the secondary’s IPs at the primary, adding notifications to primary, and telling the secondary what IPs to use to get the transfer at the primary. Ensure that the secondary is syncing your zones with the primary. (Check on their console and try doing a dig @nameserver domain for the secondary’s nameservers.) Configure your registrar with both the primary’s and secondary’s name servers. We found out that the order does not matter at all. Our nameserver setup at the registrar: ns1.p29.dynect.net ns2.p29.dynect.net udns1.ultradns.net udns2.ultradns.net What happened during the outage? We got paged at 8:53 AM for DNS problem hitting service.sumologic.com. This was from our internal as well as external monitors. The oncalls ran a “dig” against all four of our nameservers and discovered that Dyn was down hard. We knew that we had a primary/secondary DNS setup, but neither provider had experienced any outages since we set it up. We also knew that it would take DNS Resolvers some time to decide to use Neustar nameservers as opposed to Dyn ones. Our alarms went off, so, we posted a status page telling our customers that we are experiencing an incident with our DNS and to let us know if they see a problem. Less than an hour later, our alarms stopped going off (although Dyn was still down). No Sumo Logic customers reached out to Support to let us know that they had issues. Here is a graph of the traffic decreases for one of the Sumo Logic domains during the Dyn Outage: Here is a graph of Neustar (UltraDNS) pulling in more traffic during the outage: In conclusion: This setup worked for Sumo Logic. We do not have control over DNS providers, but we can prevent their problems from affecting our customers. You can easily do the same.

The AWS Elastic Cloud Compute (EC2) service offers a simple, robust vehicle for performing real-time load balancing of applications hosted within the Amazon Cloud. Elastic Load Balancer (ELB) is designed to optimize performance and scalability, and maximize resource utilization by balancing loads across multiple AWS instances.

https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

Field extraction rules compress queries into short phrases, filter out unwanted fields and drastically speed up query times. Fifty at a time can be stored and used in what Sumo Logic calls a “parser library.” These rules are a must once you move from simple collection to correlation and dashboarding. Since they tailor searches prior to source ingestion, the rules never collect unwanted fields, which can drastically speed up query times. Correlations and dashboards require many queries to load simultaneously, so the speed impact can be significant. Setting Up Field Extraction Rules The Sumo Logic team has written some templates to help you get started with common logs like IIS and Apache. While you will need to edit them, they take a lot of the pain out of writing regex parsers from scratch (phew). And if you write your own reusable parsers, save them as a template so you can help yourself to them later. To get started, find a frequently used query snippet. The best candidates are queries that (1) are used frequently and (2) take a while to load. These might pull from dense sources (like iis) or just crawl back over long periods of time. You can also look at de facto high usage queries saved in dashboards, alerts and pinned searches. Once you have the query, first take a look at what the source pulls without any filters. This is important both to ensure that you collect what’s needed, and that you don’t include anything that will throw off the rules. Since rules are “all or nothing,” only include persistent fields. In the example below, I am pulling from a safend collector. Here’s the output from a collector on a USB: 2014-10-09T15:12:33.912408-04:00 safend.host.com [Safend Data Protection] File Logging Alert details: User: [email protected], Computer: computer.host.com, Operating System: Windows 7, Client GMT: 10/9/2014 7:12:33 PM, Client Local Time: 10/9/2014 3:12:33 PM, Server Time: 10/9/2014 7:12:33 PM, Group: , Policy: Safend for Cuomer Default Policy, Device Description: Disk drive, Device Info: SanDisk Cruzer Pattern USB Device, Port: USB, Device Type: Removable Storage Devices, Vendor: 0781, Model: 550A, Distinct ID: 3485320307908660, Details: , File Name: F:\SOME_FILE_NAME, File Type: PDF, File Size: 35607, Created: 10/9/2014 7:12:33 PM, Modified: 10/9/2014 7:12:34 PM, Action: Write There are certainly reasons to collect all of this (and note that the rule won’t limit collection on the source collector) but I only want to analyze a few parameters. To get it just right, filter it in the Field Extraction panel: Below is the simple Parse Expression I used. Note that more parsing tools are supported that can grep nearly anything that a regular query can. But in this case, I just used parse and nodrop. Nodrop tells the query to pass results along even if the query returns nothing from that field. In this case, it acts like an OR function that concatenates the first three parse functions along with the last one. So if ‘parse regex “Action…”‘ returns nothing, nodrop commands the query to “not drop”, return a blank, and in this case, continue to the next function. Remember that Field Extraction Rules are “all or nothing” with respect to fields. If you add a field that doesn’t exist, nodrop will not help since it only works within existing fields. Use Field Extraction Rules to Speed Up Dashboard Load Time The above example would be a good underlying rule for a larger profiling dashboard. It returns file information only—Action on the File, File ID, File Size, and Type. Another extraction rule might return only User and User Activities, while yet another might include only host server actions. These rules can then be surfaced as dashboard panes, combined into profiles and easily edited. They load only the fields extracted, significantly improving load time, and the modularity of the rules provides a built-in library that makes editing and sharing useful snippets much simpler. Working With Field Extraction Rules in Sumo Logic is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production. About the Author Alex Entrekin served on the executive staff of Cloudshare where he was primarily responsible for advanced analytics and monitoring systems. His work extending Splunk into actionable user profiling was featured at VMworld: “How a Cloud Computing Provider Reached the Holy Grail of Visibility.” Alex is currently an attorney, researcher and writer based in Santa Barbara, CA. He holds a J.D. from the UCLA School of Law.

The top Puppet question on ServerFault is How can the little guys effectively learn and use Puppet? Learning Puppet requires learning a DSL that’s thorny enough that the final step in many migrations is to buy Puppet training classes for team. While there is no getting around learning the Puppet DSL, the “little guys” can be more effective if they avoid extending Puppet beyond the realm of configuration management (CM). It can be tempting to extend Puppet to become a monitoring hub, a CI spoke, or many other things. After all, if it’s not in Puppet, it won’t be in your environment, so why not build on that powerful connectedness? The cons of Puppet for log analysis and monitoring Here’s one anecdote from scriptcrafty explaining some of the problems with extending beyond CM: Centralized logic where none is required, Weird DSLs and templating languages with convoluted error messages, Deployment and configuration logic disembodied from the applications that required them and written by people who have no idea what the application requires, Weird configuration dependencies that are completely untestable in a development environment, Broken secrets/token management and the heroic workarounds, Divergent and separate pipelines for development and production environments even though the whole point of these tools is to make things re-usable, and so on and so forth. Any environment complex enough to need Puppet is already too complex to be analyzed with bash and PuppetDB queries. These tools work well for spot investigation and break/fix, but do not extend easily into monitoring and analysis. I’ll use “borrow-time” as an example. To paraphrase the Puppet analytics team, “borrow-time” is the amount of time that the JRuby instances handling Puppet tasks spend on each request. If this number gets high, then there may be something unusually expensive going on. For instance, when the “borrow-timeout-count” metric is > 0, some build request has gone unfilled. It’s tempting to think that the problem is solved by setting a “borrow-timeout-count” trigger in PuppetDB for >0. After all, just about any scripting language will do, and then analysis can be done in the PuppetDB logs. Puppet even has some guides for this in Puppet Server – What’s Going on in There? Monitoring a tool with only its own suggested metrics is not just a convenience sample, but one that is also blind to the problem at hand—uptime and consistency across an inconsistent and complex environment. Knowing that some request has gone unhandled is a good starting point. A closer look at Puppet logs and metrics But look at everything else that Puppet shows when pulling metrics:is trying approach is it runs a risk so let’s look at what one “borrow-time” metrics pull brings up: In the Puppet server: pe-jruby-metrics->status->experimental->metrics "metrics": { "average-borrow-time": 75, "average-free-jrubies": 1.86, "average-lock-held-time": 0, "average-lock-wait-time": 0, "average-requested-jrubies": 1.8959058782351241, "average-wait-time": 77, "borrow-count": 10302, "borrow-retry-count": 0, "borrow-timeout-count": 0, "borrowed-instances": [ { "duration-millis": 2888, "reason": { "request": { "request-method": "post", "route-id": "puppet-v3-catalog-/*/", "uri": "/puppet/v3/catalog/foo.puppetlabs.net" } }, }, ...], "num-free-jrubies": 0, "num-jrubies": 4, "num-pool-locks": 0, "requested-count": 10305, "requested-instances": [ { "duration-millis": 134, "reason": { "request": { "request-method": "get", "route-id": "puppet-v3-file_metadata-/*/", "uri": "/puppet/v3/file_metadata/modules/catalog_zero16/catalog_zero16_impl83.txt" } }, }, ...], "return-count": 10298 } If you are lucky, you’ll have an intuitive feeling about the issue before asking whether the retry count is too high, or if it was only a problem in a certain geo. If the problem is severe, you won’t have time to check the common errors (here and here); you’ll want context. How Sumo Logic brings context to Puppet logs Adding context—such as timeseries, geo, tool, and user—is the primary reason to use Sumo for Puppet monitoring and analysis. Here is an overly simplified example Sumo Logic query where jruby borrowing is compared with the Apache log 2**/3**/4** errors: _sourceName=*jruby-metrics* AND _sourceCategory=*apache* | parse using public/apache/access | if(status_code matches "2*", 1, 0) as successes | if(status_code matches "5*", 1, 0) as server_errors | if(status_code matches "4*", 1, 0) as client_errors | if (num-free-jrubies matches “0”,1,0) as borrowrequired | timeslice by 1d | sum(successes) as successes, sum(client_errors) as client_errors, sum(server_errors) as server_errors sum(borrowrequired) as borrowed_jrubies by _timeslice Centralizing monitoring across the environment means not only querying and joining siloed data, but also allowing for smarter analysis. By appending an “outlier” query to something like the above, you can set baselines and spot trends in your environment instead of guessing and then querying. | timeslice 15d | max(borrowed_jrubies) as borrowed_jrubies by _timeslice | outlier response_time source: help.sumologic.com/Search/Search_Query_Language/Search_Operators/outlier About the Author Alex Entrekin served on the executive staff of Cloudshare where he was primarily responsible for advanced analytics and monitoring systems. His work extending Splunk into actionable user profiling was featured at VMworld: “How a Cloud Computing Provider Reached the Holy Grail of Visibility.” Alex is currently an attorney, researcher and writer based in Santa Barbara, CA. He holds a J.D. from the UCLA School of Law. Monitoring and Analyzing Puppet Logs With Sumo Logic is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

When people talk about containers, they usually mean application containers. Docker is automatically associated with application containers and is widely used to package applications and services. But there is another type of container: system containers. Let us look at the differences between application containers vs. system containers and see how each type of container is used: Application Containers System Containers Images Application/service centric Growing tool ecosystem Machine-centric Limited tool ecosystem Infrastructure Security concerns Networking challenges Hampered by base OS limitations Datacenter-centric Isolated & secure Optimized networking The Low-Down on Application Containers Application containers are used to package applications without launching a virtual machine for each app or each service within an app. They are especially beneficial when making the move to a microservices architecture, as they allow you to create a separate container for each application component and provide greater control, security and process restriction. Ultimately, what you get from application containers is easier distribution. The risks of inconsistency, unreliability and compatibility issues are reduced significantly if an application is placed and shipped inside a container. Docker is currently the most widely adopted container service provider with a focus on application containers. However, there are other container technologies like CoreOS’s Rocket. Rocket promises better security, portability and flexibility of image sharing. Docker already enjoys the advantage of mass adoption, and Rocket might just be too late to the container party. Even with its differences, Docker is still the unofficial standard for application containers today. System Containers: How They’re Used System containers play a similar role to virtual machines, as they share the kernel of the host operating system and provide user space isolation. However, system containers do not use hypervisors. (Any container that runs an OS is a system container.) They also allow you to install different libraries, languages, and databases. Services running in each container use resources that are assigned to just that container. System containers let you run multiple processes at the same time, all under the same OS and not a separate guest OS. This lowers the performance impact, and provides the benefits of VMs, like running multiple processes, along with the new benefits of containers like better portability and quick startup times. Useful System Container Tools Joyent’s Triton is a Container as a Service that implements its proprietary OS called SmartOS. It not only focuses on packing apps into containers but also provides the benefits of added security, networking and storage, while keeping things lightweight, with very little performance impact. The key differentiator is that Triton delivers bare-metal performance. With Samsung’s recent acquisition of Joyent, it’s left to be seen how Triton progresses. Giant Swarm is a hosted cloud platform that offers a Docker-based virtualization system that is configured for microservices. It helps businesses manage their development stack, spend less time on operations setup, and more time on active development. LXD is a fairly new OS container that was released in 2016 by Canonical, the creators of Ubuntu. It combines the speed and efficiency of containers with the famed security of virtual machines. Since Docker and LXD share the same kernels, it is easy to run Docker containers inside LXD containers. Ultimately, understanding the differences and values of each type of container is important. Using both to provide solutions for different scenarios can’t be ruled out, either, as different teams have different uses. The development of containers, just like any other technology, is quickly advancing and changing based on newer demands and the changing needs of users. Monitoring Your Containers Whatever the type of container, monitoring and log analysis is always needed. Even with all of the advantages that containers offer as compared to virtual machines, things will go wrong. That is why it is important to have a reliable log-analysis solution like Sumo Logic. One of the biggest challenges of Docker adoption is scalability, and monitoring containerized apps. Sumo Logic addresses this issue with its container-native monitoring solution. The Docker Log Analysis app from Sumo Logic can visualize your entire Docker ecosystem, from development to deployment. It uses advanced machine learning algorithms to detect outliers and anomalies when troubleshooting issues in distributed container-based applications. Sumo Logic’s focus on containers means it can provide more comprehensive and vital log analysis than traditional Linux-based monitoring tools. About the Author Twain began his career at Google, where, among other things, he was involved in technical support for the AdWords team. His work involved reviewing stack traces, and resolving issues affecting both customers and the Support team, and handling escalations. Later, he built branded social media applications, and automation scripts to help startups better manage their marketing operations. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications. Application Containers vs. System Containers: Understanding the Difference is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

Docker data volumes are designed to solve one of the deep paradoxes of containers, which is this: For the very same reasons that containers make apps highly portable — and, by extension, create more nimble data centers — they also make it hard to store data persistently. That’s because, by design, containerized apps are ephemeral. Once you shut down a container, everything inside it disappears. That makes your data center more flexible and secure, since it lets you spin up apps rapidly based on clean images. But it also means that data stored inside your containers disappears by default. How do you resolve this paradox? There are actually several ways. You could jerry-rig a system for loading data into a container each time it is spun up ( via SSH, for example), then exporting it somehow, but that’s messy. You could also turn to traditional distributed storage systems, like NFS, which you can access directly over the network. But that won’t work well if you have a complicated (software-defined) networking situation (and you probably do in a large data center). You’d think someone would have solved the Docker container storage challenge in a more elegant way by now — and someone has! Docker data volumes provide a much cleaner, straightforward way to provide persistent data storage for containers. That’s what I’ll cover here. Keep reading for instructions on setting up and deploying Docker data volumes (followed by brief notes on storing data persistently directly on the host). Creating a Docker Data Volume To use a data volume in Docker, you first need to create a container to host the volume. This is pretty basic. Just use a command like: docker create -v /some/directory mydatacontainer debian This command tells Docker to create a new container named mydatacontainer based on the Debian Docker image. (You could use any of Docker’s other OS images here, too.) Meanwhile, the -v flag in the command above sets up a storage container in the directory /some/directory inside the container. To repeat: That means the data is stored at /some/directory inside the container called mydatacontainer — not at /some/directory on your host system. The beauty of this, of course, is that we can now write data to /some/directory inside this container, and it will stay there as long as the container remains up. Using a Data Volume in Docker So that’s all good and well. But how do you actually get apps to use the new data volume you created? Pretty easily. The next and final step is just to start another container, using the –volumes-from flag to tell Docker that this new container should store data in the data volume we created in the first container. Our command would look something like this: docker run --volumes-from mydatacontainer --volumes-from debian Now, any data changes made inside the container debian will be saved inside mydatacontainer at the directory /some/directory. And they’ll stay there if you stop debian — which means this is a persistent data storage solution. (Of course, if you stop mycontainervolume, then you’ll also lose the data inside.) You can have as many data volumes as you want, by the way. Just specify multiple ones when you run the container that will access the volumes. Data Storage on Host instead of a container? You may be thinking, “What if I want to store my data directly on the host instead of inside another container?” There’s good news. You can do that, too. We won’t use data storage volumes for this, though. Instead, we’ll run a command like: docker run -v /host/dir:/container/dir -i image This starts a new container based on the image image and maps the directory /host/dir on the host system to the directory /container/dir inside the container. That means that any data that is written by the container to /container/dir will also appear inside /host/dir on the host, and vice versa. There you have it. You can now have your container data and eat it, too. Or something like that. About the Author Hemant Jain is the founder and owner of Rapidera Technologies, a full service software development shop. He and his team focus a lot on modern software delivery techniques and tools. Prior to Rapidera he managed large scale enterprise development projects at Autodesk and Deloitte. Managing Container Data Using Docker Data Volumes is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

SaltStack, also known simply as Salt, is a handy configuration management platform. Written in Python, it’s open source and allows ITOps teams to define “Infrastructure as Code” in order to provision and orchestrate servers. But SaltStack’s usefulness is not limited to configuration management. The platform also generates logs, and like all logs, that data can be a useful source of insight in all manner of ways. This article provides an overview of SaltStack logging, as well as a primer on how to analyze SaltStack logs with Sumo Logic. Where does SaltStack store logs? The first thing to understand is where SaltStack logs live. The answer to that question depends on where you choose to place them. You can set the log location by editing your SaltStack configuration file on the salt-master. By default, this file should be located at /etc/salt/master on most Unix-like systems. The variable you’ll want to edit is log_file. If you want to store logs locally on the salt-master, you can simply set this to any location on the local file system, such as /var/log/salt/salt_master. Storing Salt logs with rsyslogd If you want to centralize logging across a cluster, however, you will benefit by using rsyslogd, a system logging tool for Unix-like systems. With rsyslogd, you can configure SaltStack to store logs either remotely or on the local file system. For remote logging, set the log_file parameter in the salt-master configuration file according to the format: <file|udp|tcp>://<host|socketpath>:/. For example, to connect to a server named mylogserver (whose name should be resolveable on your local network DNS, of course) via UDP on port 2099, you’d use a line like this one: log_file: udp://mylogserver:2099 Colorizing and bracketing your Salt logs Another useful configuration option that SaltStack supports is custom colorization of console logs. This can make it easier to read the logs by separating high-priority events from less important ones. To set colorization, you change the log_fmt_console parameter in the Salt configuration file. The colorization options available are: '%(colorlevel)s' # log level name colorized by level '%(colorname)s' # colorized module name '%(colorprocess)s' # colorized process number '%(colormsg)s' # log message colorized by level Log files can’t be colorized. That would not be as useful, since the program you use to read the log file may not support color output, but they can be padded and bracketed to distinguish different event levels. The parameter you’ll set here is log_fmt_logfile and the options supported include: '%(bracketlevel)s' # equivalent to [%(levelname)-8s] '%(bracketname)s' # equivalent to [%(name)-17s] '%(bracketprocess)s' # equivalent to [%(process)5s] How to Analyze SaltStack logs with Sumo Logic So far, we’ve covered some handy things to know about configuring SaltStack logs. You’re likely also interested in how you can analyze the data in those logs. Here, Sumo Logic, which offers easy integration with SaltStack, is an excellent solution. Sumo Logic has an official SaltStack formula, which is available from GitHub. To install it, you can use GitFS to make the formula available to your system, but the simpler approach (for my money, at least) is simply to clone the formula repository in order to save it locally. That way, changes to the formula won’t break your configuration. (The downside, of course, is that you also won’t automatically get updates to the formula, but you can always update your local clone of the repository if you want them.) To set up the Sumo Logic formula, run these commands: mkdir -p /srv/formulas # or wherever you want to save the formula cd /srv/formulas git clone https://github.com/saltstack-formulas/sumo-logic-formula.git Then simply edit your configuration by adding the new directory to the file_roots parameter, like so: file_roots: base: - /srv/salt - /srv/formulas/sumo-logic-formula Restart your salt-master and you’re all set. You’ll now be able to analyze your SaltStack logs from Sumo Logic, along with any other logs you work with through the platform. Getting the Most Out of SaltStack Logs is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production. About the Author Chris Tozzi has worked as a journalist and Linux systems administrator. He has particular interests in open source, agile infrastructure and networking. He is Senior Editor of content and a DevOps Analyst at Fixate IO.

Looking for some logging moves that will impress your business partner? In this post, we’ll show you a few. But first, a note of caution:If you’re going to wow your business partner, make a visiting venture capitalist’s jaw drop, or knock the socks off of a few stockholders, you could always accomplish that with something that has a lot of flash, and not much more than that, or you could show them something that has real and lasting substance, and will make a difference in your company’s bottom line. We’ve all seen business presentations filled with flashy fireworks, and we’ve all seen how quickly those fireworks fade away.Around here, though, we believe in delivering value—the kind that stays with your organization, and gives it a solid foundation for growth. So, while the logging moves that we’re going to show you do look good, the important thing to keep in mind is that they provide genuine, substantial value—and discerning business partners and investors (the kind that you want to have in your corner) will recognize this value quickly.Why Is Log Monitoring Useful?What value should logs provide? Is it enough just to accumulate information so that IT staff can pick through it as required? That’s what most logs do, varying mostly in the amount of information and the level of detail. And most logs, taken as raw data, are very difficult to read and interpret; the most noticeable result of working with raw log data, in fact, is the demand that it puts on IT staff time.5 Log Monitoring Steps to SuccessMost of the value in logs is delivered by means of systems for organizing, managing, filtering, analyzing, and presenting log data. And needless to say, the best, most impressive, most valuable logging moves are those which are made possible by first-rate log management. They include:Quick, on-the-spot, easy-to-understand analytics. Pulling up instant, high-quality analytics may be the most impressive move that you can make when it comes to logging, and it is definitely one of the most valuable features that you should look for in any log management system. Raw log data is a gold mine, but you need to know how to extract and refine the gold. A high-quality analytics system will extract the data that’s valuable to you, based on your needs and interests, and present it in ways that make sense. It will also allow you to quickly recognize and understand the information that you’re looking for.Monitoring real-time data. While analysis of cumulative log data is extremely useful, there are also plenty of situations where you need to see what is going on right at the moment. Many of the processes that you most need to monitor (including customer interaction, system load, resource use, and hostile intrusion/attack) are rapid and transient, and there is no substitute for a real-time view into such events. Real-time monitoring should be accompanied by the capacity for real-time analytics. You need to be able to both see and understand events as they happen.Fully integrated logging and analytics. There may be processes in software development and operations which have a natural tendency to produce integrated output, but logging isn’t one of them. Each service or application can produce its own log, in its own format, based on its own standards, without reference to the content or format of the logs created by any other process. One of the most important and basic functions that any log management system can perform is log integration, bringing together not just standard log files, but also event-driven and real-time data. Want to really impress partners and investors? Bring up log data that comes from every part of your operation, and that is fully integrated into useful, easily-understood output.Drill-down to key data. Statistics and aggregate data are important; they give you an overall picture of how the system is operating, along with general, system-level warnings of potential trouble. But the ability to drill down to more specific levels of data—geographic regions, servers, individual accounts, specific services and processes —is what allows you to make use of much of that system-wide data. It’s one thing to see that your servers are experiencing an unusually high level of activity, and quite another to drill down and see an unusual spike in transactions centered around a group of servers in a region known for high levels of online credit card fraud. Needless to say, integrated logging and scalability are essential when it comes to drill-down capability.Logging throughout the application lifecycle. Logging integration includes integration across time, as well as across platforms. This means combining development, testing, and deployment logs with metrics and other performance-related data to provide a clear, unified, in-depth picture of the application’s entire lifecycle. This in turn makes it possible to look at development, operational, and performance-related issues in context, and see relationships which might not be visible without such cross-system, full lifecycle integration.Use Log Monitoring to Go for the GoldSo there you have it—five genuine, knock-’em-dead logging moves. They’ll look very impressive in a business presentation, and they’ll tell serious, knowledgeable investors that you understand and care about substance, and not just flash. More to the point, these are logging capabilities and strategies which will provide you with valuable (and often crucial) information about the development, deployment, and ongoing operation of your software.Logs do not need to be junkpiles of unsorted, raw data. Bring first-rate management and analytics to your logs now, and turn those junk-piles into gold.5 Log Monitoring Moves to Wow Your Business Partner is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.About the AuthorMichael Churchman started as a scriptwriter, editor, and producer during the anything-goes early years of the game industry. He spent much of the ‘90s in the high-pressure bundled software industry, where the move from waterfall to faster release was well under way, and near-continuous release cycles and automated deployment were already de facto standards. During that time he developed a semi-automated system for managing localization in over fifteen languages. For the past ten years, he has been involved in the analysis of software development processes and related engineering management issues.

As an API Engineer, you’ve probably spent hours carefully considering how API will be consumed by client software, what data you are making available at which points within particular workflows, and strategies for handling errors that bubble up when a client insists on feeding garbage to your API. You’ve written tests for the serializers and expected API behaviors, and you even thought to mock those external integrations so you can dive right into the build. As you settle in for a productive afternoon of development, you notice a glaring legacy element in your otherwise modern development setup: Latest and greatest version of your IDE: Check. Updated compiler and toolchain: Installed. Continuous Integration: Ready and waiting to put your code through its paces. That random text file containing a bunch of clumsily ordered cURL commands. …one of these things is not like the others. It turns out we’ve evolved…and so have our API tools Once upon a time, that little text file was state-of-the-art in API development. You could easily copy-paste commands into a terminal and watch your server code spring into action; however, deviating from previously built requests required careful editing. Invariably, a typo would creep into a crucial header declaration, or revisions to required parameters were inconsistently applied, or perhaps a change in HTTP method resulted in a subtly different API behavior that went unnoticed release over release. HTTP Request Builders were developed to take the sting out of developing and testing HTTP endpoints by reducing the overhead in building and maintaining test harnesses, allowing you to get better code written with higher quality. Two of the leaders in the commercial space are Postman and Paw, and they provide a number of key features that will resonate with those who either create or consume APIs: Create HTTP Requests in a visual editor: See the impact of your selected headers and request bodies on the request before you send it off to your server. Want to try an experiment? Toggle parameters on or off with ease or simply duplicate an existing request and try two different approaches! Organize requests for your own workflow…or collaborate with others: Create folders, reorder, and reorganize requests to make it painless to walk through sequential API calls. Test across multiple environments: Effortlessly switch between server environments or other variable data without having to rewrite every one of your requests. Inject dynamic data: Run your APIs as you would expect them to run in production, taking data from a previous API as the input to another API. From here, let’s explore the main features of HTTP Request Builders via Paw and show how those features can help make your development and test cycles more efficient. Although Paw will be featured in this post, many of these capabilities exist in other HTTP Builder packages such as Postman. How to Streamline your HTTP Request Pipeline Command-line interfaces are great for piping together functionality in one-off tests or when building out scripts for machines to follow, but quickly become unwieldy when you have a need to make sweeping changes to the structure or format of an API call. This is where visual editors shine, giving the human user an easily digestible view of the structure of the HTTP request, including its headers, querystring and body so that you can review and edit requests in a format that puts the human first. Paw’s editor is broken up into three areas. Working from left to right, these areas are: Request List: Each distinct request in your Paw document gets a new row in this panel and represents the collection of request data and response history associated with that specific request. HTTP Request Builder: This is the primary editor for constructing HTTP requests. Tabs within this panel allow you to quickly switch between editing headers, URL parameters, and request bodies. At the bottom of the panel is the code generator, allowing you to quickly spawn code for a variety of languages including Objective-C, Swift, Java, and even cURL! HTTP Exchange: This panel reflects the most recent request and associated response objects returned by the remote server. This panel also offers navigation controls for viewing historical requests and responses. Figure 1. Paw Document containing three sample HTTP Requests and the default panel arrangement. As you work through building up the requests that you use in your API workflows, you can easily duplicate, edit, and execute a request all in a matter of a few seconds. This allows you to easily experiment with alternate request formats or payloads while also retaining each of your previous versions. You might even score some brownie points with your QA team by providing a document with templated requests they can use to kick-start their testing of your new API! Organize Request Lists for Yourself and Others The Request List panel also doubles as the Paw document’s organization structure. As you add new requests, they will appear at the bottom of the list; however, you can customize the order by dragging and dropping requests, or a create folders to group related requests together. The order and names attached to each request help humans understand what the request does, but in no way impacts the actual requests made of the remote resource. Use these organization tools to make it easy for you to run through a series of tests or to show others exactly how to replicate a problem. If the custom sort options don’t quite cover your needs, or if your document starts to become too large, Sort and Filter bars appear at the bottom of the Request List to help you focus only on the requests you are actively working with. Group by URL or use the text filter to find only those requests that contain the URL you are working with. Figure 2. Request List panel showing saved requests, folder organization, and filtering options. Dealing with Environments and Variables Of course, many times you want to be able to test out behaviors across different environments — perhaps your local development instance, or the development instance updated by the Continuous Integration service. Or perhaps you may even want to compare functionality to what is presently available in production. It would be quite annoying to have to edit each of your requests and change the URL from one host to another. Instead, let Paw manage that with a quick switch in the UI. Figure 3. Paw’s Environment Switcher changes variables with just a couple of clicks. The Manage Environments view allows you to create different “Domains” for related kinds of variables, and add “Environments” as necessary to handle permutations of these values: Figure 4. Paw’s Environment Editor shows all Domains and gives easy access to each Environment. This allows you flexibility in adjusting the structure of a payload with a few quick clicks instead of having to handcraft an entirely new request. The Code Generator pane at the bottom of the Request Builder pane updates to show you exactly how your payload changes: Figure 5. Paw Document showing the rebuilt request based on the Server Domain’s Environment. One of the most common setups is to have a Server Domain with Environments for the different deployed versions of code. From there, you could build out a variable for the Base URL, or split it into multiple variables so that the protocol could be changed, independent of the host address — perhaps in order to quickly test whether HTTP to HTTPS redirection still works after making changes to a load balancer or routing configuration. Paw’s variables can even peer into other requests and responses and automatically rewrite successive APIs. Many APIs require some form of authentication to read or write privileged material. Perhaps the mechanism is something simple like a cookie or authentication header, or something more complex like an oAuth handshake. Either way, there is a bit of data in the response of one API that should be included in the request to a subsequent API. Paw variables can parse data from prior requests and prior responses, dynamically updating subsequent requests: Figure 6. Paw Document revealing the Response Parsed Body Variable extracting data from one request and injecting it into another. In the case shown above, we’ve set a “Response parsed body” variable as a Querystring parameter to a successive API, specifically grabbing the UserId key for the post at index 0 in the Top 100 Posts Request. Any indexable path in the response of a previous request is available in the editor. You may need to extract a session token from the sign-in API and apply it to subsequent authenticated-only requests. Setting this variable gives you the flexibility to change server environments or users, execute a sign-in API call, then proceed to hit protected endpoints in just a few moments rather than having to make sweeping edits to your requests. Request Builders: Fast Feedback, Quick Test Cycles HTTP Request Builders help give both API developers and API consumers a human-centric way of interacting with what is primarily a machine-to-machine interface. By making it easy to build and edit HTTP requests, and providing mechanisms to organize, sort, and filter requests, and allowing for fast or automatic substitution of request data, working with any API becomes much easier to digest. The next time someone hands you a bunch of cURL commands, take a few of those minutes you’ve saved from use of these tools, and help a developer join us here in the future! Using HTTP Request Builders to Create Repeatable API Workflows is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production. About the Author Bryan Musial (@BKMu) is a Full-Stack Senior Software Engineer with the San Francisco-based startup, Tally (www.meettally.com), working to make managing your credit cards automatic, simple, and secure. Previously, Bryan worked for the Blackboard Mobile team where he built and shipped iOS and Android applications used by millions of students and teachers every day.

Azure functions are event driven pieces of code that can used to integrate systems, build APIs, process data and trigger any action/reaction to events without having to worry about infrastructure to run it.

In June Sumo Dojo (our online community) launched a contest to learn more about how our customers are using Amazon Web Services like EC2, S3, ELB, and AWS Lambda. The Sumo Logic service is built on AWS and we have deep integration into Amazon Web Services. And as an AWS Technology Partner we’ve collaborated closely with AWS to build apps like the Sumo Logic App for Lambda.So we wanted to see how our customers are using Sumo Logic to do things like collecting logs from CloudWatch to gain visibility into their AWS applications. We thought you’d be interested in hearing how others are using AWS and Sumo Logic, too. So in this post I’ll share their stories along with announcing the contest winner.The contest narrowed down to two finalists – SmartThings, which is a Samsung company operates in the home automation industry and provides access to a wide range of connected devices to create smarter homes that enhance the comfort, convenience, security and energy management for the consumer.WHOmentors, Inc. our second finalist, is a publicly supported scientific, educational and charitable corporation, and fiscal sponsor of Teen Hackathon. The organization is, according to their site, “primarily engaged in interdisciplinary applied research to gain knowledge or understanding to determine the means by which a specific, recognized need may be met.”At stake was a DJI Phantom 3 Drone. All entrants were awarded a $10 Amazon gift card.AWS Logging Contest RulesThe Drone winner was selected based on the following criteria:You have to be a user of Sumo Logic and AWSTo enter the contest, a comment had to be placed on this thread in Sumo Dojo.The post could not be anonymous – you were required to log in to post and enter.Submissions closed August 15th.As noted in the Sumo Dojo posting, the winner would be selected based on our own editorial judgment and community reactions to the post (in the form of comments or “likes”) to select one that’s most interesting, useful and detailed.SmartThingsSmartThings has been working on a feature to enable Over-the-air programming (OTA) firmware updates of Zigbee Devices on user’s home networks. For the uninitiated, Zigbee is an IEEE specification for a suite of high-level communication protocols used to create personal area networks with small, low-power digital radios. See the Zigbee Alliance for more information.According to one of the firmware engineers at SmartThings, there are a lot of edge cases and potential points of failure for an OTA update including:The Cloud PlatformAn end user’s hubThe device itselfPower failuresRF inteference on the mesh networkDisaster in this scenario would be a user’s device ending up in a broken state. As Vlad Shtibin related:“Our platform is deployed across multiple geographical regions, which are hosted on AWS. Within each region we support multiple shards, furthermore within each shard we run multiple application clusters. The bulk of the services involved in the firmware update are JVM based application servers that run on AWS EC2 instances.Our goal for monitoring was to be able to identify as many of these failure points as possible and implement a recovery strategy. Identifying these points is where Sumo Logic comes into the picture. We use a key-value logger with a specific key/value for each of these failure points as well as a correlation ID for each point of the flow. Using Sumo Logic, we are able to aggregate all of these logs by passing the correlation ID when we make calls between the systems.Using Sumo Logic, we are able to aggregate all of these logs by passing the correlation ID when we make calls between the systems.We then created a search query (eventually a dashboard) to view the flow of the firmware updates as they went from our cloud down to the device and back up to the cloud to acknowledge that the firmware was updated. This query parses the log messages to retrieve the correlation ID, hub, device, status, firmware versions, etc.. These values are then fed into a Sumo Logic transaction enabling us to easily view the state of a firmware update for any user in the system at a micro level and the overall health of all OTA updates on the macro level.Depending on which part of the infrastructure the OTA update failed, engineers are then able to dig in deeper into the specific EC2 instance that had a problem. Because our application servers produce logs at the WARN and ERROR level we can see if the update failed because of a timeout from the AWS ElasticCache service, or from a problem with a query on AWS RDS. Having quick access to logs across the cluster enables us to identify issues across our platform regardless of which AWS service we are using.As Vlad noted, This feature is still being tested and hasn’t been rolled out fully in PROD yet. “The big take away is that we are much more confident in our ability identify updates, triage them when they fail and ensure that the feature is working correctly because of Sumo Logic.”WHOmentors.comWHOmentors.com, Inc. is a nonprofit scientific research organization and the 501(c)(3) fiscal sponsor of Teen Hackathon. To facilitate their training to learn languages like Java, Python, and Node.js, each individual participate begins with the Alexa Skills Kit, a collection of self-service application program interfaces (APIs), tools, documentation and code samples that make it fast and easy for teens to add capabilities for use Alexa-enabled products such as the Echo, Tap, or Dot.According WHOmentors.com CEO, Rauhmel Fox, “The easiest way to build the cloud-based service for a custom Alexa skill is by using AWS Lambda, an AWS offering that runs inline or uploaded code only when it’s needed and scales automatically, so there is no need to provision or continuously run servers.With AWS Lambda, WHOmentors.com pays only for what it uses. The corporate account is charged based on the number of requests for created functions and the time the code executes. While the AWS Lambda free tier includes one million free requests per month and 400,000 gigabyte (GB)-seconds of compute time per month, it becomes a concern when the students create complex applications that tie Lambda to other expensive services or the size of their Lambda programs are too long.Ordinarily, someone would be assigned to use Amazon CloudWatch to monitor and troubleshoot the serverless system architecture and multiple applications using existing AWS system, application, and custom log files. Unfortunately, there isn’t a central dashboard to monitor all created Lambda functions.With the integration of a single Sumo Logic collector, WHOmentors.com can automatically route all Amazon CloudWatch logs to the Sumo Logic service for advanced analytics and real-time visualization using the Sumo Logic Lambda functions on Github.”Using the Sumo Logic Lambda Functions“Instead of a “pull data” model, the “Sumo Logic Lambda function” grabs files and sends them to Sumo Logic web application immediately. Their online log analysis tool offers reporting, dashboards, and alerting as well as the ability to run specific advanced queries as needed.The real-time log analysis combination of the “SumoLogic Lambda function” assists me to quickly catch and troubleshoot performance issues such as the request rate of concurrent executions that are either stream-based event sources, or event sources that aren’t stream-based, rather than having to wait hours to identify whether there was an issue.I am most concerned about AWS Lambda limits (i.e., code storage) that are fixed and cannot be changed at this time. By default, AWS Lambda limits the total concurrent executions across all functions within a given region to 100. Why? The default limit is a safety limit that protects the corporate from costs due to potential runaway or recursive functions during initial development and testing.As a result, I can quickly determine the performance of any Lambda function and clean up the corporate account by removing Lambda functions that are no longer used or figure out how to reduce the code size of the Lambda functions that should not be removed such as apps in production.”The biggest relief for Rauhmel is he is able to encourage the trainees to focus on coding their applications instead of pressuring them to worry about the logs associated with the Lambda functions they create.And the Winner of AWS Logging Contest is…Just as at the end of an epic World-Series battle between two MLB teams, you sometimes wish both could be declared winner. Alas, there can only be one. We looked closely at the use cases, which were very different from one another. Weighing factors like the breadth in the usage of the Sumo Logic and AWS platforms added to our drama. While SmartThings uses Sumo Logic broadly to troubleshoot and prevent failure points, WHOmentors.com use case is specific to AWS Lambda. But we couldn’t ignore the cause of helping teens learn to write code in popular programming languages, and building skills that may one day lead them to a job.Congratulations to WHOmentors.com. Your Drone is on its way!

Today plenty of businesses still have real concerns about migrating applications to the cloud. Fears about network security, availability, and potential downtime swirl through the heads of chief decision makers, sometimes paralyzing organizations into standing pat on existing tech–even though it’s aging by the minute. Enter Microsoft Azure, the industry leader’s solution for going to a partially or totally cloud-based architecture. Below is a detailed look at what Azure is, the power of partnering with Microsoft for a cloud or hybrid cloud solution, and the best way to get full and actionable visibility into your aggregated logs and infrastructure metrics so your organization can react quickly to opportunities. What is Microsoft Azure? Microsoft has leveraged its constantly-expanding worldwide network of data centers to create Azure, a cloud platform for building, deploying, and managing services and applications, anywhere. Azure lets you add cloud capabilities to your existing network through its platform as a service (PaaS) model, or entrust Microsoft with all of your computing and network needs with Infrastructure as a Service (IaaS). Either option provides secure, reliable access to your cloud hosted data–one built on Microsoft’s proven architecture. Azure provides an ever expanding array of products and services designed to meet all your needs through one convenient, easy to manage platform. Below are just some of the capabilities Microsoft offers through Azure and tips for determining if the Microsoft cloud is the right choice for your organization. What can Microsoft Azure Do? Microsoft maintains a growing directory of Azure services, with more being added all the time. All the elements necessary to build a virtual network and deliver services or applications to a global audience are available, including: Virtual machines. Create Microsoft or Linux virtual machines (VMs) in just minutes from a wide selection of marketplace templates or from your own custom machine images. These cloud-based VMs will host your apps and services as if they resided in your own data center. SQL databases. Azure offers managed SQL relational databases, from one to an unlimited number, as a service. This saves you overhead and expenses on hardware, software, and the need for in-house expertise. Azure Active Directory Domain services. Built on the same proven technology as Windows Active Directory, this service for Azure lets you remotely manage group policy, authentication, and everything else. This makes moving and existing security structure partially or totally to the cloud as easy as a few clicks. Application services. With Azure it’s easier than ever to create and globally deploy applications that are compatible on all popular web and portable platforms. Reliable, scalable cloud access lets you respond quickly to your business’s ebb and flow, saving time and money. With the introduction of Azure WebApps to the Azure Marketplace, it’s easier than ever to manage production, testing and deployment of web applications that scale as quickly as your business. Prebuilt APIs for popular cloud services like Office 365, Salesforce and more greatly accelerate development. Visual Studio team services. An add-on service available under Azure, Visual Studio team services offer a complete application lifecycle management (ALM) solution in the Microsoft cloud. Developers can share and track code changes, perform load testing, and deliver applications to production while collaborating in Azure from all over the world. Visual Studio team services simplify development and delivery for large companies or new ones building a service portfolio. Storage. Count on Microsoft’s global infrastructure to provide safe, highly accessible data storage. With massive scalability and an intelligent pricing structure that lets you store infrequently accessed data at a huge savings, building a safe and cost-effective storage plan is simple in Microsoft Azure. Microsoft continues to expand its offerings in the Azure environment, making it easy to make a la carte choices for the best applications and services for your needs. Why are people trusting their workloads to Microsoft Azure? It’s been said that the on-premise data center has no future. Like mainframes and dial-up modems before them, self-hosted data centers are becoming obsolete, being replaced by increasingly available and affordable cloud solutions. Several important players have emerged in the cloud service sphere, including Amazon Web Services (AWS), perennial computing giant IBM, and Apple’s ubiquitous iCloud, which holds the picture memories and song preferences of hundreds of millions of smartphone users, among other data. With so many options, why are companies like 3M, BMW, and GE moving workloads to Microsoft Azure? Just some of the reasons: Flexibility. With Microsoft Azure you can spin up new services and geometrically scale your data storage capabilities on the fly. Compare this to a static data center, which would require new hardware and OS purchasing, provisioning, and deployment before additional power could be brought to bear against your IT challenges. This modern flexibility makes Azure a tempting solution for organizations of any size. Cost. Azure solutions don’t just make it faster and easier to add and scale infrastructure, they make it cheaper. Physical services and infrastructure devices like routers, load balancers and more quickly add up to thousands or even hundreds of thousands of dollars. Then there’s the IT expertise required to run this equipment, which amounts to major payroll overhead. By leveraging Microsoft’s massive infrastructure and expertise, Azure can trim our annual IT budget by head-turning percentages. Applications. With a la carte service offerings like Visual Studio Team Services, Visual Studio Application Insights, and Azure’s scalable, on-demand storage for both frequently accessed and ‘cold’ data, Microsoft makes developing and testing mission-critical apps a snap. Move an application from test to production mode on the fly across a globally distributed network. Microsoft also offers substantial licensing discounts for migrating their existing apps to Azure, which represents even more opportunity for savings. Disaster recovery. Sometimes the unthinkable becomes the very immediate reality. Another advantage of Microsoft Azure lay in its high-speed and geographically decentralized infrastructure, which creates limitless options for disaster recovery plans. Ensure that your critical application and data can run from redundant sites during recovery periods that last minutes or hours instead of days. Lost time is lost business, and with Azure you can guarantee continuous service delivery even when disaster strikes. The combination of Microsoft’s vast infrastructure, constant application and services development, and powerful presence in the global IT marketplace has made Microsoft Azure solutions the choice of two-thirds of the world’s Fortune 500 companies. But the infinite scalability of Azure can make it just as right for your small personal business. Logging capabilities within Microsoft Azure The secret gold mine of any infrastructure and service solution is ongoing operational and security visibility, and ultimately these comes down to extracting critical log and infrastructure metrics from the application and underlying stack. The lack of this visibility is like flying a plane blind—no one does it. Azure comes with integrated health monitoring and alert capabilities so you can know in an instant if performance issues or outages are impacting your business. Set smart alert levels for events from: Azure diagnostic infrastructure logs. Get current insights into how your cloud network is performing and take action to resolve slow downs, bottlenecks, or service failures. Windows IIS logs. View activity on your virtual web servers and respond to traffic patterns or log-in anomalies with the data Azure gathers on IIS 7. Crash dumps. Even virtual machines can ‘blue screen’ and other virtual equipment crashes can majorly disrupt your operations. With Microsoft Azure you can record crash dump data and troubleshoot to avoid repeat problems. Custom error logs. Set Azure alerts to inform you about defined error events. This is especially helpful when hosting private applications that generate internal intelligence about operations, so you can add these errors to the health checklist Azure maintains about your network. Microsoft Azure gives you the basic tools you need for error logging and monitoring, diagnostics, and troubleshooting to ensure continuous service delivery in your Azure cloud environment. Gain Full Visibility into Azure with Unified Logs and Metrics Even with Azure’s native logging and analytics tools, the vast amount of data flowing to make your network and applications operate can be overwhelming. The volume, variety and velocity of cloud data should not be underestimated. With the help of Sumo Logic, a trusted Microsoft partner, managment of that data is simple. The Sumo Logic platform unifies logs and metrics from the structured, semi-structured, and unstructured data across your entire Microsoft environment. Machine learning algorithms process vast amounts of log and metrics data, looking for anomalies and deviations from normal patterns of activity, alerting you when appropriate. With Log Reduce, Log Compare and Outlier Detection, extract continuous intelligence from your application stack and proactively respond to operational and security issues. The Sumo Logic apps for Microsoft Azure Audit, Microsoft Azure Web Apps, Microsoft Windows Server Active Directory, Microsoft Internet Information Services (IIS), and the popular Windows Performance app, make ingesting machine data in real-time and rendering it into clear, interactive visualizations for a complete picture of your applications and data. Before long the on-premise data center—along with its expensive hardware and hordes of local technicians on the payroll—may be lost to technology’s graveyard. But smart, researched investment into cloud capabilities like those provided in Microsoft Azure will make facing tomorrow’s bold technology challenges and possibilities relatively painless.

In this post, I’ll be discussing the use of monitoring as a tool to optimize the cost and performance of AWS Lambda. I’ve worked on a number of teams, and almost without exception, the need to put monitoring in place has featured prominently in early plans. Tickets are usually created and discussed, and then placed in the backlog, where they seem to enter a cycle of being important—but never quite enough to be placed ahead of the next phase of work. In reality, especially in a continuous development environment, monitoring should be a top priority, and with the tools available in AWS and organizations like Sumo Logic, setting up basic monitoring shouldn’t take more than a couple of hours, or a day at most. What exactly is AWS Lambda? AWS Lambda from Amazon Web Services (AWS) allows an organization to introduce functionality into the AWS ecosystem without the need to provision and maintain servers. A Lambda function can be uploaded and configured, and then can be executed as frequently as needed without further intervention. Contrary to a typical server environment, you don’t have any control, nor do you require insight into the data elements like CPU usage or available memory, and you don’t have to worry about scaling your functionality to meet increased demand. Once a Lambda has been deployed, the cost is based on the number of requests and a few key elements we’ll discuss shortly. How to setup AWS Lambda monitoring, and what to track Before we get into how to set up monitoring and what data elements should be tracked, it is vital that you put an effective monitoring system in place. And with that decision made, AWS helps you right from the start. Monitoring is handled automatically by AWS Lambda, which means less time configuring and more time analyzing results. Logs are automatically sent to Amazon Cloudwatch where a user can view basic metrics, or harness the power of an external reporting system, and gain key insights into how Lambda is performing. The Sumo Logic App for AWS Lambda uses the Lambda logs via CloudWatch and visualizes operational and performance trends about all the Lambda functions in your account, providing insight into executions such as memory and duration usage, broken down by function versions or aliases. The pricing model for functionality deployed using AWS Lambda is calculated by the number of requests received, and the time and resources needed to process the request. Therefore, the key metrics that need to be considered are: Number of requests and associated error counts. Resources required for execution. Time required for execution or latency. Request and error counts in Lambda The cost per request is the simplest of the three factors. In the typical business environment, the goal is to drive traffic to the business’ offerings; thus, increasing the number of requests is key. Monitoring of these metrics should compare the number of actual visitors with the number of requests being made to the function. Depending on the function and how often a user is expected to interact with it, you can quickly determine what an acceptable ratio might be—1:1 or 1:3. Variances from this should be investigated to determine what the cause might be. Lambda Resources usage and processing time When Lambda is first configured, you can specify the expected memory usage of the function. The actual runtime usage may differ, and is reported on a per-request basis. Amazon then factors the cost of the request based on how much memory is used, and for how long. The latency is based on 100ms segments. If, for example, you find that your function typically completes in a time between 290ms and 310ms, there would be a definite cost savings if the function could be optimized to perform consistently in less than 300ms. These optimizations, however, would need to be analyzed to determine whether they increase resource usage for that same time, and if that increase in performance is worth an increase in cost. For more information on how Amazon calculates costs relative to resource usage and execution time, you can visit the AWS Lambda pricing page. AWS Lambda: The big picture Finally, when implementing and considering metrics, it is important to consider the big picture. One of my very first attempts with a Lambda function yielded exceptional numbers with respect to performance and utilization of resources. The process was blazingly fast, and barely used any resources at all. It wasn’t until I looked at the request and error metrics that I realized that over 90% of my requests were being rejected immediately. While monitoring can’t make your business decisions for you, having a solid monitoring system in place will give an objective view of how your functions are performing, and data to support the decisions you need to make for your business. Optimizing AWS Lambda Cost and Performance API Workflows is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production. About the Author Mike Mackrory is a Global citizen who has settled down in the Pacific Northwest – for now. By day he works as a Senior Engineer on a Quality Engineering team and by night he writes, consults on several web based projects and runs a marginally successful eBay sticker business. When he’s not tapping on the keys, he can be found hiking, fishing and exploring both the urban and the rural landscape with his kids. Always happy to help out another developer, he has a definite preference for helping those who bring gifts of gourmet donuts, craft beer and/or Single-malt Scotch.

If your data center were a beehive, Docker Swarm would be the pheromone that keeps all the bees working efficiently together.

The Sumo Logic Web app provides a search interface that lets you parse logs. This provides a great resource for a lot of use cases — especially because you can take advantage of a rich search syntax, including wildcards and various operators (documented here), directly from the Web app. But we realize that some people need to be able to harness Sumo Logic search data from within external apps, too. That’s why Sumo Logic also provides a robust RESTful API that you can use to integrate other apps with Sumo Logic search. To provide a sense of how you can use the Sumo Logic Search Job API in the real world, this post offers a quick primer on the API, along with a couple of examples of the API in action. For more detailed information, refer to the Search Job API documentation. Sumo Logic Search Integration: The Basics Before getting started there are a few essentials you should know about the Sumo Logic Search Job API. First, the API uses the HTTP GET method. That makes it pretty straightforward to build the API into Web apps you may have (or any other type of app that uses the HTTP protocol). It also means you can run queries directly from the CLI using any tool that supports HTTP GET requests, like curl or wget. Sound easy? It is! Second, queries should be directed to https://api.sumologic.com/api/v1/logs/search. You simply append your GET requests and send them on to the server. (You also need to make sure that your HTTP request contains the parameters for connecting to your Sumo Logic account; for example, with curl, you would specify these using the -u flag, for instance, curl -u [email protected]:VeryTopSecret123 your-search-query). Third, the server delivers query responses in JSON format. That approach is used because it keeps the search result data formatting consistent, allowing you to manipulate the results easily if needed. Fourth, know that the Search Job API can return up to one million records per search query. API requests are limited to four API per second and 240 requests per minute across all API calls from a customer. If the rate is exceeded, a rate limit exceeded (429) error is returned. Sumo Logic Search API Example Queries As promised, here are some real-world examples. For starters, let’s say you want to identify incidents where a database connection failure occurred. To do this, specify “database connection error” as our query, using a command like this: curl -u [email protected]:VeryTopSecret123 "https://api.sumologic.com/api/v1/logs/search?q=database connection error" (That’s all one line, by the way.) You can take things further, too, by adding date and time parameters to the search. For example, if you wanted to find database connection errors that happened between about 1 p.m. and 3 p.m. on April 4, 2012, you would add some extra data to your query, making it look like this: curl -u [email protected]:VeryTopSecret123 "https://api.sumologic.com/api/v1/logs/search?q=database connection error&from=2012-04-04T13:01:02&to=2012-04-04T15:01:02 Another real-world situation where the search API can come in handy is to find login failures. You could locate those in the logs with a query like this: curl -u [email protected]:VeryTopSecret123 "https://api.sumologic.com/api/v1/logs/search?q=failed login" Again, you could restrict your search here to a certain time and date range, too, if you wanted. Another Way to Integrate with Sumo Logic Search: Webhooks Most users will probably find the Sumo Logic search API the most extensible method of integrating their apps with log data. But there is another way to go about this, too, which is worth mentioning before we wrap up. That’s Webhook alerts, a feature that was added to Sumo Logic last fall. Webhooks make it easy to feed Sumo Logic search data to external apps, like Slack, PagerDuty, VictorOps and Datadog. I won’t explain how to use Webhooks in this post, because that topic is already covered on our blog. Integrating Apps with the Sumo Logic Search API is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production. About the Author Dan Stevens is the founder of StickyWeb (stickyweb.biz), a custom Web Technology development company. Previously, he was the Senior Product Manager for Java Technologies at Sun Microsystems and for broadcast video technologies at Sony Electronics, Accom and Ampex.

In December 2013, AWS released Kinesis, a managed, dynamically scalable service for the processing of streaming big data in real-time. Since that time, Amazon has been steadily expanding the regions in which Kinesis is available, and as of this writing, it is possible to integrate Amazon’s Kinesis producer and client libraries into a variety of custom applications to enable real-time processing of streaming data from a variety of sources.

Bintray, JFrog’s software hosting and distribution platform, offers lots of exciting features, like CI integration and REST APIs. If you’re like me, you enjoy thinking about those features much more than you enjoy thinking about software security. Packaging and distributing software is fun; worrying about the details of Bintray security configurations and access control for your software tends to be tedious (unless security is your thing, of course). Like any other tool, however, Bintray is only effective in a production environment when it is run securely. That means that, alongside all of the other fun things you can do with Bintray, you should plan and run your deployment in a way that mitigates the risk of unauthorized access, the exposure of private data, and so on. Below, I explain the basics of Bintray security, and outline strategies for making your Bintray deployment more secure. Bintray Security Basics Bintray is a cloud service hosted by JFrog’s data center provider. JFrog promises that the service is designed for security, and hardened against attack. (The company is not very specific about how it mitigates security vulnerabilities for Bintray hosting, but I wouldn’t be either, since one does not want to give potential attackers information about the configuration.) JFrog also says that it restricts employee access to Bintray servers and uses SSH over VPN when employees do access the servers, which adds additional security. The hosted nature of Bintray means that none of the security considerations associated with on-premises software apply. That makes life considerably easier from the get-go if you’re using Bintray and are worried about security. Still, there’s more that you can do to ensure that your Bintray deployment is as robust as possible against potential intrusions. In particular, consider adopting the following policies. Set up an API key for Bintray Bintray requires users to create a username and password when they first set up an account. You’ll need those when getting started with Bintray. Once your account is created, however, you can help mitigate the risk of unauthorized access by creating an API key. This allows you to authenticate over the Bintray API without using your username or password. That means that even if a network sniffer is listening to your traffic, your account won’t be compromised. Use OAuth for Bintray Authentication Bintray also supports authentication using the OAuth protocol. That means you can log in using credentials from a GitHub, Twitter or Google+ account. Chances are that you pay closer attention to one of these accounts (and get notices from the providers about unauthorized access) than you do to your Bintray account. So, to maximize security and reduce the risk of unauthorized access, make sure your Bintray account itself has login credentials that cannot be brute-forced, then log in to Bintray via OAuth using an account from a third-party service that you monitor closely. Sign Packages with GPG Bintray supports optional GPG signing of packages. To do this, you first have to configure a key pair in your Bintray profile. For details, check out the Bintray documentation. GPG signing is another obvious way to help keep your Bintray deployment more secure. It also keeps the users of your software distributions happier, since they will know that your packages are GPG-signed, and therefore, are less likely to contain malicious content. Take Advantage of Bintray’s Access Control The professional version of Bintray offers granular control over who can download packages. (Unfortunately this feature is only available in that edition.) You can configure access on a per-user or per-organization basis. While gaining Bintray security shouldn’t be the main reason you use granular access control (the feature is primarily designed to help you fine-tune your software distribution), it doesn’t hurt to take advantage of it in order to reduce the risk that certain software becomes available to a user to whom you don’t want to give access. 5 Bintray Security Best Practices is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production. About the Author Chris Tozzi has worked as a journalist and Linux systems administrator. He has particular interests in open source, agile infrastructure and networking. He is Senior Editor of content and a DevOps Analyst at Fixate IO.

https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

Why should I benchmark microservices? The ultimate goal of benchmarking is to better understand the software, and test out the effects of various optimization techniques for microservices. In this blog, we describe our approach to benchmarking microservices here at Sumo Logic. Create a spreadsheet for tracking your benchmarking We found a convenient way to document a series of benchmarks is in a Google Spreadsheet. It allows collaboration and provides the necessary features to analyze and sum up your results. Structure your spreadsheet as follows: Title page Goals Methodology List of planned and completed experiments (evolving as you learn more) Insights Additional pages Detailed benchmark results for various experiments Be clear about your benchmark goals Before you engage in benchmarking, clearly state (and document) your goal. Examples of goals are: “I am trying to understand how input X affects metric Y” “I am running experiments A, B and C to increase/decrease metric X” Pick one key metric (Key Performance Indicator – KPI) State clearly which one metric you are concerned about and how the metric affects users of the system. If you choose to capture additional metrics for your test runs, ensure that the key metrics stands out. Think like a scientist You’re going to be performing a series of experiments to better understand which inputs affect your key metric, and how. Consider and document the variables you devise, and create a standard control set to compare against. Design your series of experiments in a fashion that leads to the understanding in the least amount of time and effort. Define, document and validate your benchmarking methodology Define a methodology for running your benchmarks. It is critical your benchmarks be: Fairly fast (several minutes, ideally) Reproducible in the exact same manner, even months later Documented well enough so another person can repeat them and get identical results Document your methodology in detail. Also document how to re-create your environment. Include all details another person needs to know: Versions used Feature flags and other configuration Instance types and any other environmental details Use load generation tools, and understand their limitations In most cases, to accomplish repeatable, rapid-fire experiments, you need a synthetic load generation tool. Find out whether one already exists. If not, you may need to write one. Understand that load generation tools are at best an approximation of what is going on in production. The better the approximation, the more relevant the results you’re going to obtain. If you find yourself drawing insights from benchmarks that do not translate into production, revisit your load generation tool. Validate your benchmarking methodology Repeat a baseline benchmark at least 10 times and calculate the standard deviation over the results. You can use the following spreadsheet formula: =STDEV(<range>)/AVERAGE(<range>) Format this number as a percentage, and you’ll see how big the relative variance in your result set is. Ideally, you want this value to be < 10%. If your benchmarks have larger variance, revisit your methodology. You may need to tweak factors like: Increase the duration of the tests. Eliminate variance from the environments. Ensure all benchmarks start in the same state (i.e. cold caches, freshly launched JVMs, etc). Consider the effects of Hotspot/JITs. Simplify/stub components and dependencies on other microservices that add variance but aren’t key to your benchmark. Don’t be shy to make hacky code changes and push binaries you’d never ship to production. Important: Determine the number of results you need to get the standard deviation below a good threshold. Run each of your actual benchmarks at least that many times. Otherwise, your results may be too random. Execute the benchmark series Now that you have developed a sound methodology, it’s time to gather data. Tips: Only vary one input/knob/configuration setting at a time. For every run of the benchmark, capture start and end time. This will help you correlate it to logs and metrics later. If you’re unsure whether the input will actually affect your metric, try extreme values to confirm it’s worth running a series. Script the execution of the benchmarks and collection of metrics. Interleave your benchmarks to make sure what you’re observing aren’t slow changes in your test environment. Instead of running AAAABBBBCCCC, run ABCABCABCABC. Create enough load to be able to measure a difference There are two different strategies for generating load. Strategy 1: Redline it! In most cases, you want to ensure you’re creating enough load to saturate your component. If you do not manage to accomplish that, how would you see that you increased it’s throughput? If your component falls apart at redline (i.e. OOMs, throughput drops, or otherwise spirals out of control), understand why, and fix the problem. Strategy 2: Measure machine resources In cases where you cannot redline the component, or you have reason to believe it behaves substantially different in less-than-100%-load situations, you may need to resort to OS metrics such as CPU utilization and IOPS to determine whether you’ve made a change. Make sure your load is large enough for changes to be visible. If your load causes 3% CPU utilization, a 50% improvement in performance will be lost in the noise. Try different amounts of load and find a sweet spot, where your OS metric measurement is sensitive enough. Add new benchmarking experiments as needed As you execute your benchmarks and develop a better understanding of the system, you are likely to discover new factors that may impact your key metric. Add new experiments to your list and prioritize them over the previous ones if needed. Hack the code In some instances, the code may not have configuration or control knobs for the inputs you want to vary. Find the fastest way to change the input, even if it means hacking the code, commenting out sections or otherwise manipulating the code in ways that wouldn’t be “kosher” for merges into master. Remember: The goal here is to get answers as quickly as possible, not to write production-quality code—that comes later, once we have our answers. Analyze the data and document your insights Once you’ve completed a series of benchmarks, take a step back and think about what the data is telling you about the system you’re benchmarking. Document your insights and how the data backs them up. It may be helpful to: Calculate the average for each series of benchmarks you ran and to use that to calculate the difference (in percent) between series — i.e. “when I doubled the number of threads, QPS increased by 23% on average.” Graph your results — is the relationship between your input and the performance metric linear? Logarithmic? Bell curve? Present your insights When presenting your insights to management and/or other engineering teams, apply the Pyramid Principle. Engineers often make the mistake of explaining methodology, results and concluding with the insights. It is preferable to reverse the order and start with the insight. Then, if needed/requested, explain methodology and how the data supports your insight. Omit nitty-gritty details of any experiments that didn’t lead to interesting insights. Avoid jargon, and if you cannot, explain it. Don’t assume your audience knows the jargon. Make sure your graphs have meaningful, human-readable units. Make sure your graphs can be read when projected onto a screen or TV.

Do you like GitHub, but don’t like having to log in to check on the status of your project or code? GitHub events are your solution. GitHub events provide a handy way to receive automated status updates from your GitHub repos concerning everything from code commits to new users joining a project. And because they are accessible via a Web API as GET requests, it’s easy to integrate them into the notification system of your choosing. Keep reading for a primer on GitHub events and how to get the most out of them. What GitHub events are, and what they are not Again, GitHub events provide an easy way to keep track of your GitHub repository without monitoring its status manually. They’re basically a notification system that offers a high level of customizability. You should keep in mind, however, that GitHub events are designed only as a way to receive notifications. They don’t allow you to interact with your GitHub repo. You can’t trigger events; you can only receive notifications when specific events occur. That means that events are not a way for you to automate the maintenance of your repository or project. You’ll need other tools for that. But if you just want to monitor changes, they’re a simple solution. How to use GitHub events GitHub event usage is pretty straightforward. You simply send GET requests to https://api.github.com. You specify the type of information you want by completing the URL information accordingly. For example, if you want information about the public events performed by a given GitHub user, you would send a GET request to this URL: https://api.github.com/users//events (If you are authenticated, this request will generate information about private events that you have performed.) Here’s a real-world example, in which we send a GET request using curl to find information about public events performed by Linus Torvalds (the original author of Git), whose username is torvalds: curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X GET https://api.github.com/users/torvalds/events Another handy request lets you list events for a particular organization. The URL to use here looks like: https://api.github.com/users/:username/events/orgs/ The full list of events, with their associated URLs, is available from the GitHub documentation. Use GitHub Webhooks for automated events reporting So far, we’ve covered how to request information about an event using a specific HTTP request. But you can take things further by using GitHub Webhooks to automate reporting about events of a certain type. Webhooks allow you to “subscribe” to particular events and receive an HTTP POST response (or, in GitHub parlance, a “payload”) to a URL of your choosing whenever that event occurs. You can create a Webhook in the GitHub Web interface that allows you to specify the URL to which GitHub should send your payload when an event is triggered. Alternatively, you can create Webhooks via the GitHub API using POST requests. However you set them up, Webhooks allow you to monitor your repositories (or any public repositories) and receive alerts in an automated fashion. Like most good things in life, Webhooks are subject to certain limitations, which are worth noting. Specifically, you can only configure up to a maximum of twenty events per each GitHub organization or repository. Authentication and GitHub events The last bit of information we should go over is how to authenticate with the GitHub API. While you can monitor public events without authentication, you’ll need to authenticate in order to keep track of private ones. Authentication via the GitHub API is detailed here, but it basically boils down to having three options. The simplest is to do HTTP authentication using a command like: curl -u "username" https://api.github.com If you want to be more sophisticated, you can also authenticate using OAuth2 via either key/secrets or tokens. For example, authenticating with a token would look something like: curl https://api.github.com/?access_token=OAUTH-TOKEN If you’re monitoring private events, you’ll want to authenticate with one of these methods before sending requests about the events. Further reading If you want to dive deeper into the details of GitHub events, the following resources are useful: Overview of event types. Event payloads according to event type. Setting up Webhooks. GitHub API authentication. A Beginner’s Guide to GitHub Events is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

Gauging the health and productivity of a software release cycle is notoriously difficult. Atomic age metrics like “man months” and LOCs may be discredited, but they are too often a reflexive response for DevOps problems. Instead of understanding the cycle itself, management may hire a “DevOps expert” or homebrew one by taking someone off their project and focusing them on “automation.” Or they might add man months and LOCs with more well-intentioned end-to-end tests. What could go wrong? Below, I’ve compiled some metrics and tips for building a release cycle health dashboard using Sumo Logic. Measuring Your Software Release Cycle Speed Jez Humble points to some evidence that delivering faster not only shortens feedback but also makes people happier, even on deployment days. Regardless, shorter feedback cycles do tend to bring in more user involvement in the release, resulting in more useful features and fewer bugs. Even if you are not pushing for only faster releases, you will still need to allocate resources between functions and services. Measuring deployment speed will help. Change lead time: Time between ticket accepted and ticket closed. Change frequency: Time between deployments. Recovery Time: Time between a severe incident and resolution. To get this data to Sumo Logic, ingest your SCM and incident management tools. While not typical log streams, the tags and timestamps are necessary to tracking the pipeline. You can return deployment data from your release management tools. Tracking Teams, Services with the Github App To avoid averaging out insights, Separately tag services and teams in each of the tests above. For example, if a user logic group works on identities and billing, track billing and identity services separately. For Github users, there is an easy solution, the Sumo Logic App for Github, which is currently available in preview. It generates pre-built dashboards in common monitoring areas like security, commit/pipeline and issues. More importantly, each panel provides queries that can be repurposed for separately tagged, team-specific panels. Reusing these queries allows you to build clear pipeline visualizations very quickly. For example, let’s build a “UI” team change frequency panel. First, create a lookup table designating UserTeams. Pin it to saved queries as it can be used across the dashboard to break out teams: "id","user","email","team", "1","Joe","[email protected]","UI" "2","John","[email protected]","UI" "3","Susan","[email protected]","UI" "4","John","[email protected]","backspace" "5","John","[email protected]","backspace" Next, copy the “Pull Requests by Repository” query from the panel: _sourceCategory=github_logs and ( "opened" or "closed" or "reopened" ) | json "action", "issue.id", "issue.number", "issue.title" , "issue.state", "issue.created_at", "issue.updated_at", "issue.closed_at", "issue.body", "issue.user.login", "issue.url", "repository.name", "repository.open_issues_count" as action, issue_ID, issue_num, issue_title, state, createdAt, updatedAt, closedAt, body, user, url, repo_name, repoOpenIssueCnt | count by action,repo_name | where action != "assigned" | transpose row repo_name column action Then, pipe in the team identifier with a lookup command: _sourceCategory=github_logs and ( "opened" or "closed" or "reopened" ) | json "action", "issue.id", "issue.number", "issue.title" , "issue.state", "issue.created_at", "issue.updated_at", "issue.closed_at", "issue.body", "issue.user.login", "issue.url", "repository.name", "repository.open_issues_count" as action, issue_ID, issue_num, issue_title, state, createdAt, updatedAt, closedAt, body, user, url, repo_name, repoOpenIssueCnt | lookup team from https://toplevelurlwithlookups.com/UserTeams.csv on user=user | count by action,repo_name, team | where action != "assigned" | transpose row repo_name team column action This resulting query tracks commits — open, closed or reopened — by team. The visualization can be controlled on the panel editor, and the lookup can be easily piped to other queries to break the pipeline by teams. Don’t Forget User Experience It may seem out of scope to measure user experience alongside a deployment schedule and recovery time, but it’s a release cycle health dashboard, and nothing is a better measure of a release cycle’s health than user satisfaction. There are two standards worth including: Apdex and Net Promoter Score. Apdex: measures application performance on a 0-1 satisfaction scale calculated by… If you want to build an Apdex solely in Sumo Logic, you could read through this blog post and use the new Metrics feature in Sumo Logic. This is a set of numeric metrics tools for performance analysis. It will allow you to set, then tune satisfaction and tolerating levels without resorting to a third party tool. Net Promoter Score: How likely is it that you would recommend our service to a friend or colleague? This one-question survey correlates with user satisfaction, is simple to embed anywhere in an application or marketing channel, and can easily be forwarded to a Sumo Logic dashboard through a webhook. When visualizing these UX metrics, do not use the single numerical callout. Take advantage of Sumo Logic’s time-series capabilities by tracking a line chart with standard deviation. Over time, this will give you an expected range of satisfaction and visual cues of spikes in dissatisfaction that sit on the same timeline as your release cycle. Controlling the Release Cycle Logging Deluge A release cycle has a few dimensions that involve multiple sources, which allow you to query endlessly. For example, speed requires ticketing, CI and deployment logs. Crawling all the logs in these sources can quickly add up to TBs of data. That’s great fun for ad hoc queries, but streams like comment text are not necessary for a process health dashboard, and their verbosity can result in slow dashboard load times and costly index overruns. To avoid this, block this and other unnecessary data by partitioning sources in Sumo Logic’s index tailoring menus. You can also speed up the dashboard by scheduling your underlying query runs for once a day. A health dashboard doesn’t send alerts, so it doesn’t need to be running in real-time. More Resources: How Do you Measure Team Success? On the Care and Feeding of Feedback Cycles Martin Fowler’s Test Pyramid Just Say No to More End to End Tests Quantifying Devops Capability: It’s Important to Keep CALMS 9 Metrics DevOps Teams Track Building Software Release Cycle Health Dashboards in Sumo Logic is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production. About the Author Alex Entrekin served on the executive staff of Cloudshare where he was primarily responsible for advanced analytics and monitoring systems. His work extending Splunk into actionable user profiling was featured at VMworld: “How a Cloud Computing Provider Reached the Holy Grail of Visibility.” Alex is currently an attorney, researcher and writer based in Santa Barbara, CA. He holds a J.D. from the UCLA School of Law.

Git bisect is a great way to help narrow down who (and what) broke something in your test—and it is incredibly easy to learn. This Git Bisect tutorial will guide you through the basic process. So, what is Git bisect? What exactly is it? Git bisect is a binary search to help find the commit that introduced a bug. Not only is it a great tool for developers, but it’s also very useful for testers (like myself). I left for the weekend with my Jenkins dashboard looking nice and green. It was a three-day weekend—time to relax! I came in Tuesday, fired up my machine, logged into Jenkins…RED. I had to do a little bit of detective work. Luckily for me, a developer friend told me about git bisect (I’m relatively new to this world), and helped me quickly track down which commit broke my tests. Getting started with Git bisect First, I had to narrow down the timeline. (Side note—this isn’t really a tool if you’re looking over the last few months, but if it’s within recent history—days—it’s handy). Looking at my build history in Jenkins, I noted the date/times I had a passing build (around 11 AM), and when it started showing up red (around 5 PM). I went into SourceTree and found a commit from around 11 AM that I thought would be good. A simple double-click of that commit and I was in. I ran some tests against that commit, and all passed, confirming I had a good build. It was time to start my bisect session! git bisect start git bisect good Narrowing down the suspects Now that I’d established my GOOD build, I had time to figure out where I thought the bad build occured. Back to SourceTree! I found a commit from around 5 PM (where I noticed the first failure), so I thought I’d check that one out. I ran some more tests. Sure enough, they failed. I marked that as my bad build. git bisect bad I had a bunch of commits between the good and bad (in my case, 15), and needed to find which one between our 11 AM and 5 PM run broke our tests. Now, without bisect, I might have had to pull down each commit, run my tests, and see which started failing between good and bad. That’s very time-consuming. But git bisect prevents you from having to do that. When I ran git bisect bad, I got a message in the following format: Bisecting: revisions left to test after this (roughly steps) [<commit number>] <Commit Description> This helped identify the middle commit between what I identified as good and bad, cutting my options in half. It told me how many revisions were between the identified commit and my bad commit (previously identified), how many more steps I should have to find the culprit, and which commit I next needed to look at. Then, I needed to test the commit that bisect came up with. So—I grabbed it, ran my tests—and they all passed. git bisect good This narrowed down my results even further and gave me a similar message. I continued to grab the commit, run tests, and set them as git good until I found my culprit—and it took me only three steps (versus running through about 15 commits)! When my tests failed, I had to identify the bad commit. git bisect bad <commit number> is the first bad commit <commit number> Author: <name> Date: <date and time of commit> Aha! I knew the specific commit that broke my test, who did it, and when. I had everything I needed to go back to the engineer (with proof!) and start getting my tests green again. Getting to the Root Cause When you are in the process of stabilizing tests, it can be fairly time-consuming to determine if a failure is a result of a test, or the result of an actual bug. Using git bisect can help reduce that time and really pinpoint what exactly went wrong. In this case, we were able to quickly go to the engineer, alert the engineer that a specific commit broke the tests, and work together to understand why and how to fix it. Of course, in my perfect world, it wouldn’t only be my team that monitors and cares about the results. But until I live in Tester’s Utopia, I’ll use git bisect. Who Broke My Test? A Git Bisect Tutorial is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production. About the Author Ashley Hunsberger is a Quality Architect at Blackboard, Inc. and co-founder of Quality Element. She’s passionate about making an impact in education and loves coaching team members in product and client-focused quality practices. Most recently, she has focused on test strategy implementation and training, development process efficiencies, and preaching Test Driven Development to anyone that will listen. In her downtime, she loves to travel, read, quilt, hike, and spend time with her family.

As cloud computing continues to become more and more accessible to businesses (and individuals), it can be a full-time job keeping up with all the tools released by the likes of Amazon, Microsoft, and others. One such tool is AWS Lambda, something that stands to revolutionize the accessibility (and affordability) of distributed computing. What Are AWS Lambda Functions? The short answer is that Lambdas are discrete blocks of code which are executed without having to manage the overhead of hardware and software involved in hosting that code. Your code can currently be written in either Python or Node.js, and can be uploaded with project-specific dependencies to facilitate a seemingly endless set of possibilities. Getting Started with AWS Lambda Sometimes the best way to understand a system is to simply start using it, so let’s dispense with the introductions and get right into some code. Since I much more frequently use JavaScript in my life, I’ll be doing my examples here in Node.js. The equivalent scripts in Python will rarely be more complicated, and Amazon provides plenty of examples in both languages to allow you the chance to learn. Once you get through the AWS Console and are creating a new Lambda function, you are first presented with a selection of templates. One of the templates available is a hello-world template in Node.js. Selecting it will bring you to a configuration screen with several options, as well as the following code: There are three important pieces of this code. First, the line of code that creates the function: exports.handler = function(event, context) { This code (and the associated closing bracket) is the heart of any Lambda. The Lambda system relies on having this exported handler to know where it will pass execution when the Lambda has been triggered. Second in line is the event argument for the handler, which receives any input provided by the trigger for the lambda. This example does a good job of showing how the following JSON would be accessible to the code: { “key3”: “value3”, “key2”: “value2”, “key1”: “value1” } The final piece that’s important is the context success/failure call: context.succeed(event.key1); That call lets the Lambda host environment know whether or not the lambda was able to complete its work. You can also include (as shown) additional information with your success/failure indication, which is important if you plan on debugging what you’ve created. Everything else in the code (in this case the console.log() calls) is superfluous as far as the Lambda environment is concerned. How to Use AWS Lambda Functions With the basics out of the way, let’s get into the nitty-gritty of showing how the system can be useful to you. Let’s consider the following situation: You are tasked with sending a notification email to a manager whenever someone visits a specific page on a website. For pre-existing reasons, you aren’t able to send email directly from the web server, so you have to create a system that can be remotely invoked to deliver emails. In the past, maybe you would create a service or cron job that constantly polled a database table for outgoing emails. Maybe you’d even create your own message queue handler to do the work on demand (much better), but you’re still stuck building and maintaining extra infrastructure. Enter Lambdas! Use Case: Our Email Handler Starting from the hello-world example provided by Amazon, I’m going to remove everything except the exported handler. I’ll then include the AWS SDK (thankfully built into the Lambda environment, so I can do this without uploading my code in a ZIP file), building a parameter object for use with my AWS SES (Simple Email Service) account, and attempt sending the email through SES before deciding if the operation was successful. Here is the completed code (and the explanation of the lines below): var AWS = require('aws-sdk'); exports.handler = function(event, context) { var params = { Destination: { ToAddresses: ['[email protected]'] }, Message: { Body: { Text: { Data: 'Someone has pinged the website!', Charset: 'utf8' } }, Subject: { Data: 'A Website Ping', Charset: 'utf8' } }, Source: '[email protected]', SourceArn: 'arn-here' }; new AWS.SES().sendEmail(params, function(err, data) { if (err) { context.fail('Failed to send email (' + err + ')'); } else { context.succeed('The website has been pinged'); } }); }; The first line accomplishes loading the AWS SDK for use later on in the script. We use our typical handler export, and inside the function build our SES parameters and attempt sending through the SDK. The parameter object for SES includes the bare minimum of options for an SES email, including the Destination, Message, Source, and SourceArn definitions. The SourceArn is part of AWS’ Identity and Access Management (IAM) system and indicates which email address the lambda is attempting to use for delivery. Lambda Tutorial Extra Credit Once this much has been done, we technically have a functioning lambda (hit ‘Test’ on the configuration page to be sure). However, getting it to work within our system would require at least creating an API endpoint to enable remote triggering of the lambda via a REST call. Additionally, there are ways to add triggers for other services within AWS, so choosing the best trigger approach will be based on your technology requirements. Other Common Lambda Uses? While utilizing the system for distributed (and thus asynchronous) email notifications/delivery is useful, it’s hardly the end of where Lambdas are useful. Some other ideas for great applications of the feature: On-demand distributed file conversion (change video file encodings, get metadata from uploads, etc) Index updates for services such as ElasticSearch Heavy computations for data analysis And these are just a few of the potential use cases where Lambdas will shine. The Brass Tacks of AWS Lambda Hopefully this has been a quick but educational introduction into the world of AWS Lambdas. We’ve shown how you can perform non-time-sensitive operations outside of the confines of your application with a small amount of effort. Lambdas can accomplish much more—so good luck, and have fun embracing distributed computing. Editor’s Note: A Brief Tutorial to Understanding, Starting, and Using AWS Lambda is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to visit the Sumo Logic Developersfor free tools, API’s and example code that will enable you to monitor and troubleshoot applications from code to production. About the Author Andrew Male (@AndyM84) is a senior engineer at an enterprise development company in Boston, MA. Andrew has been programming from a young age and is entirely self-taught; he has spent time in many corners of the programming world including game/VR work, agency work, and teaching development to students and adults alike. He spends most of his time working on architecture design and pursuing his favorite hobby—physics.

This is a brief excerpt of how Mayvenn started sending JMX metrics to Sumo Logic Unified Logs and Metrics solution. For the full blog post please visit Mayvenn’s engineering blog. We’ve been using Sumo Logic for logs and were excited to have one tool and dashboard to visualize logs and metrics! In order to wire this up, we decided to use jmxtrans to regularly pipe the JMX metrics we query with Graphite formatted output to the new Sumo Logic collectors. These collectors can essentially be thought of a hosted version of Graphite. Step 1: Upgrade/Install Sumo Logic Collectors There’s a lot of guides out there on this one, but just in case you have existing collectors, they do need to be updated to have the new Graphite source. Step 2: Add a Graphite Source for the Collector This step can either be done in the Sumo Logic dashboard or through a local file for the collector that configures the sources. Either way, you will need to decide what port to run the collector on and whether to use TCP or UDP. For our purposes, the standard port of 2003 is sufficient and we don’t have an extremely high volume of metrics with network/CPU concerns to justify UDP. For configuring this source in the dashboard, the Sumo Logic guide to adding a Graphite source does a pretty thorough walkthrough. To summarize, though, the steps are pretty simple: go to to the collector management page, select the relevant collector, click add source, choose the Graphite source and configure it with the port and TCP/UDP choices. This method is certainly fast to just try out Sumo Logic metrics.

Today, Sumo Logic is announcing a host monitoring app that provides comprehensive and native metrics visibility into a server and its resources. Now, customers using Sumo Logic can get detailed host metrics, analyze these metrics and visualize them (overlayed with other host metrics and side by side with logs) to optimize application and infrastructure performance. Sumo Logic App for Host Monitoring The Sumo Logic App for Host Metrics, helps you understand and correlate key metrics for monitoring your hosts (either Windows or Linux). The table below lists out the key metrics you can monitor out of the box with Sumo Logic: Key Host Metrics Visualized Host Metrics CPU User Time, System Time, Idle Time, Avg Load Time Disk Disk Used, Bytes Available, Reads, Writes, Read Bytes, Write Bytes Memory Total, Percentage Used, Total Free, Buffered and Cached TCP Connections Inbound, Outbound, Listen, Established, Close Wait, Time Wait Network In Packets, In Bytes, Out Packets, Out Bytes As an example of how to use the Host Metrics app, let’s drill down into the CPU dashboard. Host Metrics – CPU The CPU dashboard is structured to provide information at 2 levels of granularity: CPU metrics for an individual Host CPU metrics for the entire deployment This side-by-side view helps you compare/evaluate the performance of a single host in the context of the entire deployment. Panels for CPU user time and CPU system time help you understand where the CPU is spending most of its cycles. In most typical situations, user time is considerably higher, as that’s what’s going towards processing your applications. The system time is usually a much lower metric. An unusually high system time could indicate the kernel is over-busy executing system calls, such as I/O. On the other hand, an unusually high CPU user time could be an indication of inefficiency in the application code. To learn more about your Host Metrics, check out from the App available in the preview tab in the Sumo Logic library. Learn more Help and Documentation Customer Support Sumo Logic Community

The most common way to work with JFrog Artifactory is through the browser-based interface. But if you’re using Artifactory in conjunction with npm, the Node.js package manager, there are also handy interfaces available through both the npm command line and the Artifactory CLI.

Here at devRant, we use Sumo Logic for a number of things including log monitoring (triggering alerts based on error log trends), database query analysis, and user behavior analysis + A/B testing analysis. In this article I’m going to focus on a recent A/B test analysis Sumo Logic was extremely helpful on. About the devRant Community In March 2016, my friend and I founded devRant (iOS, Android) – a fun community for devs to vent, share, and bond over how they really feel about code, tech, and life as a programmer. devRant is an app with an audience that is demanding and technical by nature since our users are developers. With that in mind, it is one of our goals to launch high-quality features and provide experiences that our community will consistently enjoy. The format of devRant is pretty simple – members of the community post rants which get displayed in a feed-like format. Right now we have three different methods users can choose from to sort the feed: recent – most recent first top – highest rated rants first algo – an algorithm sort with a number of components (recency, score, and new components which I’ll cover later) Where Sumo Logic Comes in As I noted, we use Sumo Logic for a error monitoring (triggering alerts based on error log trends), database query analysis (seeing which of our DB queries take the longest tied back to user cohort info), and most importantly for us right now, user behavior analysis + A/B testing analysis. In the introduction I mentioned the sorting methods we offer in the devRant mobile apps. Since devRant launched, the default sorting method we’ve offered is our algo sort. This is what all new users see and one of our most popular sorting options. Originally, this sorting method was simply a decaying time algorithm combined with score – so over about 12 hours, the rants shown would change, with the highest rated for that time period showing up on top. While this algo served us well for a while, it became clear that we could do better; we also started getting a lot of requests from users. For example, one of the most frequent items of feedback we got was instead of seeing much of the same content for a few hours in the feed, users wanted the app to hide the content they had already viewed. Additionally, we had an unproven hypothesis that since developers are a picky bunch (well, we are!), it would be good to make the algo sort to the taste of each specific user and mix in some slightly older rants that we think they would enjoy. We decided to base this personalized algo on criteria like users they’ve enjoyed content from before, tags they’ve like, and foremost, rants that users who have similar tastes have enjoyed and that the user hasn’t already seen. As a small but quickly growing startup, it was important to us to make sure we didn’t cannibalize our most important user experience with a hunch, so we decided we would A/B test the new algorithm and see if the new one actually out-performs the old one. Using Sumo Logic to Effectively Analyze Our Test We decided to deploy our new algorithm to 50% of our mobile user-base so we could get a meaningful amount of data fairly quickly. We then decided the main metric we would look at was number of upvotes because a user generally upvotes content only if they are enjoying it. The first thing we looked at was the number of +1’s created by users based on what algo feed version they had. The log format for a vote looks like this: event_type=’Vote’, user_id=’id of user’, vote_type=’1 for upvote, -1 for downvote’ insert_id=’a unique id for the vote’, platform=’iOS, Android or Web’, algov=’1 if the user is assigned the old algo, 2 if the new algo’, post_type=’rant or comment’ So an example log message from an upvote on a rant from an iOS user with the new algo would look like this: event_type=’Vote’, user_id=’123’, vote_type=’1’, insert_id=’123456’, platform=’iOS’, algov=’2’, post_type=’rant’ For the initial query, I wanted to see a split of votes on rants for each algo version on iOS or Android (since the new algo is currently only available on mobile), so I wrote this simple query (it just uses some string matching, parses out the algo version, and then gets a count of votes for each version): "event_type='Vote'" "vote_type='1'" "post_type='rant'" ("platform='iOS'" OR "platform='Android'") | parse "algov='*'" as algov | count by algov I ran this over a short period of time and quickly got some insight into our A/B test. The results were promising: algolv _count 2 2,921 1 1,922 This meant that based on this query and data, the new algo was resulting in about 50% more +1’s on rants compared to the old algo. However, as I thought about this more, I saw an issue with the data. Like I touched on in the beginning of the article, we offer three different sorting methods with algo sort being just one of those. In the log message for a vote, I realized we weren’t logging what sort method the user was actually using. So even if the user had algov=2, they could have easily been using the recent sort instead of the algo sort, making their vote irrelevant for this test. Without the actual sort method they had used to find the rant they voted on in the log, I was in a bind to somehow make the data work. Luckily, Sumo Logic’s robust query language came to the rescue! While there’s no sort property on the vote message, we have another event we log called “Feed Load” that logs each time the feed gets loaded for a user, and includes what sort method they are using. A feed load log message is very close in format to a vote message and looks like this: event_type=’Feed Load’, user_id=’123’, sort=’algo’, insert_id=’1234’, platform=Android, algov=’2’ The important property here is sort. Using the powerful Sumo Logic join functionality, I was able to combine vote events with these feed load events in order to ensure the user who placed the vote was using algo sort. The query looked like this: ("event_type='Vote'" OR "event_type='Feed Load'") ("platform='iOS'" OR "platform='Android'") | join (parse "vote_type='*'" as vote_type, "user_id='*'" as user_id, "insert_id='*'" as insert_id, "algov='*'" as algo_ver, "post_type='*'" as post_type) as vote_query, (parse "event_type='Feed *'" as query, "sort='*'" as sort_type, "user_id='*'" as user_id) as feed_query on vote_query.user_id=feed_query.user_id timewindow 500s | WHERE vote_query_vote_type="1" AND vote_query_post_type="rant" AND feed_query_sort_type="algo" | count_distinct(vote_query_insert_id) group by vote_query_algo_ver Here’s a brief explanation of this join query: it gets all of our vote and feed load events that occurred on iOS or Android over the given time period. Then for both the vote and feed load event type, it parses out relevant variables to be used later in the query. The votes and feed loads are joined on the user id. Meanwhile, Timewindow 500s makes it so only events that occurred within about 8 minutes of each other are included. In our use-case this is important because someone can change their sort method at any time, so it’s important that the feed algo event occurred close to the time of the vote. We then do some more filtering (only upvotes, only votes on rants, and only votes that were done near the time of an algo sort [meaning they most likely originated from content found in that sort]). Lastly, we use count_distinct on the unique vote id to make sure each vote is only counted once (since there can be multiple feed loads around the time of a vote), and group the results by the algo version so we get a nice split of number of votes for each test group. After running the query over a decently substantial dataset, I got the following results: vote_query_algo_ver _count_distinct 2 4,513 1 2,176 Whoa!! Needless to say, this was pretty exciting. It seems to indicate that the new sort algo produced more than double the amount of upvotes compared to the old one, over the exact same time period. With this data, we are now comfortably able to say that the new algo is a very big improvement and creates a much better user experience in our app. How Sumo Logic Will Continue to Help Us Push devRant Forward Both my co-founder and I are very data-centric, so having a flexible solution like Sumo Logic on our side is great because it allows us to easily base very important decisions on the data and user behaviors we want to analyze. I hope in this article I’ve provided an example of how valuable A/B testing can be and why it’s important to have a tool that lets you query your data in a number of different ways. Though our new algorithm is performing very well, it is very possible that one of our tests in the future won’t have the same success. It’s better to use data to realize a failure quickly than to look at overall metrics in a few months and wonder why they plummeted. And for us, an unnoticed mistake might result in us becoming the subject of our own app. As we grow, we look forward to continuing to develop new features and utilizing quantifiable data to measure their success. We believe this approach gives us an opportunity, as a startup that strives for innovation, to take some risks but learn quickly whether a few feature should stay or go. About the Author David Fox is the co-founder and engineering lead of devRant. He has over 10 years of experience developing high-performance backend systems and working with a large variety of databases alongside massive datasets. You can follow him on Twitter @dfoxinator or on devRant (@dfox). How devRant uses Sumo Logic for Log Monitoring and A/B Testing is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out the Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

Are you suffering from swivel chair hell? Everyone knows that logs and metrics represent two sides of application and infrastructure “machine data”: Metrics can provide your app and infrastructure KPI’s like CPU, memory usage, latencies, SQL execution times etc. Logs provide you context into application and infrastructure execution KPI’s – errors, warnings, relevant events like configurations, etc. So it’s no wonder that integrating the two streams can provide incredible insights and help IT ensure the performance and health of their application and infrastructure. However, up until now, managing log and metrics has required disparate (or poorly integrated) solutions. The lack of common data, visualization and context creates many challenges, including: Inefficient swivel chair management High MTTI/MTTR because of poor troubleshooting context High TCO with multiple tools Limited DevOps collaboration etc. If this is your problem, your life changes today ! Say hello to Sumo Logic’s Unified Logs and Metrics solution. Sumo Logic’s Unified Logs and Metrics Solution Today, Sumo Logic, the leading cloud-native, machine data analytics service, announced the general availability of Unified Logs and Metrics, the industry’s first machine data analytics platform to natively ingest, index and analyze metrics and log data together in real-time. Powered by patented machine learning technology, Unified Logs and Metrics transforms structured and unstructured data into real-time continuous intelligence for today’s modern applications and business insights. Key capabilities of the solution include: Out of the box support for AWS, infrastructure and applications metrics: Sumo Logic offers support for Host Metrics such as CPU, memory, disk usage and AWS Cloudwatch metrics (from services like AWS Cloud Trail, Elastic Load Balancing, Amazon Kinesis, AWS Config, VPC Flow Logs and Amazon Simple Storage Service (S3)). Easily extend support for custom apps and infrastructure: The Sumo Logic platform supports the Graphite protocol support, which enables customers to easily extend collection and analysis to other app and infrastructure as well as custom metrics Powerful Real-time Analytics for Troubleshooting – through machine learning, Sumo Logic enables advanced analytics of logs data and metrics data for contextual troubleshooting and quicker root cause analysis of issues. View logs and metrics in a unified dashboard : Sumo logic enables users to view, filter and report logs and metrics in one dashboard to reduce “swivel chair” management and lower mean time to identify (MTTI) and resolve (MTTR) issues. Benefits for customers Sumo Logic customers have seen several benefits with the solution: Improved customer experience by proactive management of application health Faster performance troubleshooting with contextual analysis and dashboard of logs and metrics Lowered total cost of ownership by eliminating multiple tools for logs and metrics analytics Improved DevOps collaboration by providing a single source of truth that can be securely accessed and analyzed by IT OPs, DevOps and Dev teams. As one early customer for ULM described it: “We have many (5+) disparate tools to monitor our cloud-based modern apps and infrastructures. It takes us hours to identify and troubleshoot issues currently and ULM will cut that to minutes” Want to learn more? If you are interested in learning how real-world customers are using this solution, attend our upcoming webinar, featuring Sumo Logic customers who will discuss how they are leveraging the Sumo Logic ULM platform to improve application health and management. Register at the following linksd: https://info.sumologic.com/Sumo-Logic-Unified-Logs-Metrics-Webinar.html

Already Collecting Logs with Sumo Logic? The good news is that you can use the exact same Collector to start collecting your Host Metrics. Read more to learn… Metrics are numeric samples of data collected over time. They can measure infrastructure, such as operating system performance or disk activity; application performance; or custom business and operational data that is coded into an organization’s applications. Metrics are an effective tool for monitoring, troubleshooting, and identifying the root causes of problems. They can help your organization: Gain end-to-end visibility into application performance. Track key performance indicators (KPIs) over time. Determine if an outage has occurred and restore service. Determine why an event occurred and how it might be prevented in the future. The Sumo Logic Unified Logs and Metrics, the industry’s first machine data analytics platform to natively ingest, index and analyze metrics and log data together in real-time. In these 3 short videos. Let’s walk you through the process to ingest and analyze metrics with the Sumo Logic solution: Upgrading your existing Collector: See how easy it is to use the existing Sumo Logic collectors to collect and ingest host metrics Learning how to query your Data: Analyze the metrics data to gather insights Visualizing your Data on Dashboards: View and Report your metrics. And see them with logs – all in the same dashboard. Ready to get started? http://help.sumologic.com/Metrics/Metrics_Overview/Get_Started_with_Metrics

An often under-appreciated service on AWS is Route 53. One could make the mistake of thinking of AWS Route 53 as just another DNS service. To the contrary, using AWS Route 53 for global load balancing, you can benefit from improved latency and better availability for your application stack. How is this done? This article will give you an overview of how it can be setup, and hopefully it will provide a few tips to help you along the way.

Enterprises are running their workloads across complex, hybrid infrastructures, and need solutions that provide full-stack, 360-degree visibility to support rapid time to identify and resolve security threats. Trend Micro Deep Security offers seamless integration with Sumo Logic’s data analytics service to enable rich analysis, visualizations and reporting of critical security and system data. This enables an actionable, single view across all elements in an environment. I. SOLUTION COMPONENTS FOR INTEGRATION DEEP SECURITY MANAGER (DSM) This is the management component of the system and is responsible for sending rules and security settings to the Deep Security Agents. The DSM is controlled using the web-based management console. Using the console, the administrator can define security policies, manage deployed agents, query status of various managed instances, etc. The integration with Sumo Logic is done using this interface and no additional component or software is required. DEEP SECURITY AGENT (DSA) This component provides for all protection functionality. The nature of protection depends on the rules and security settings that each DSA receives from the Deep Security Manager. Additionally, the DSA sends a regular heartbeat to the DSM, and pushes event logs and other data points about the instance being protected to the DSM. SUMO LOGIC INSTALLED COLLECTORS AND SOURCES Sumo Logic Installed Collectors receive data from one or more Sources. Collectors collect raw log data, compress it, encrypt it, and send it to the Sumo Logic, in real time via HTTPS. The Deep Security Solution Components forward security events to Installed Collectors with a syslog source. SUMO LOGIC DATA ANALYTICS SERVICE AND WEB UI The Sumo Logic Web UI is browser-based and provides visibility and analysis of log data and security events sent by the Deep Security Platform to the Sumo Logic service and also provides administration tools for checking system status, managing your deployment, controlling user access and managing Collectors. SUMO LOGIC APP FOR TREND MICRO DEEP SECURITY The Sumo Logic App for Trend Micro Deep Security delivers out-of-the-box Dashboards, saved searches, and field extraction for for each security module in the Deep Security solution, including Anti-malware, Web Reputation, Intrusion Prevention, Host-based Firewall and File Integrity Monitoring. II. HOW THE DEEP SECURITY INTEGRATED SOLUTION WORKS Overview Trend Micro Deep Security Software and Deep Security as a Service integrates with Sumo Logic through the Installed Collector and Syslog Source. This Syslog Source operates like a syslog server listening on the designated port to receive syslog messages from Trend Micro Deep Security Solution. The Installed Collectors can be deployed in your environment either on a local machine, a dedicated server or in the cloud. The Deep Security platform sends system and security event logs to this server, which forwards them securely to the Sumo Logic Data Analytics Service. Figure 1 provides a high-level overview of the integration process. III. INSTALL DATA COLLECTOR Install Options The first thing to consider when you set up the integration is how to collect data from your Deep Security deployment and forward it to Sumo Logic. There are three basic methods available, local host data collection, centralized syslog data collection and hosted collector. Deep Security uses an installed centralized collector with syslog source. In this method, an installed Collector with Syslog Sources can be used to collect all relevant data in a centralized location before forwarding it on to Sumo Logic’s cloud-based service. Installed Collector with Syslog Sources The installation process involves the deployment of a Sumo Logic collector in your environment and then adding a Syslog Source to it. A Sumo Logic Installed Collector can be installed on any standard server and used to collect local files, remote files or to aggregate logs from network services via syslog. You can choose to install a small number of collectors to minimize maintenance or you can choose to install many Collectors on many machines to leverage existing configuration management and automation tools like Puppet or Chef. At the minimum you will need one “Installed Collector” setup for Deep Security. The number of syslog sources you need depends on the types of event logs that you are sending to Sumo logic. You will need one syslog source for each type of event. There are two types of events in Deep Security: “System Events” and “Security Events”. In the example shown below, we have configured Sumo Logic Installed Collector with two Syslog Sources using UDP protocol. In this example setup, the first syslog source is listening on UDP port 514 for System Event Log forwarding. The second syslog source below is listening on UDP port 1514 for Security modules event log forwarding. IV. INTEGRATE WITH SUMO LOGIC System Event Log Forwarding The integration of Trend Micro Deep Security for system events forwarding to Sumo Logic is done via system setting (Administration System Settings SIEM) configuration as shown below: Security Event Log Forwarding The integration of Trend Micro Deep Security for security event forwarding to Sumo Logic is done via Policy configuration and requires a Syslog Source with UDP protocol and connection information to be added to the policy. Deep Security allows Policy inheritance where child policies inherit their settings from their parent Policies. This way you can create a policy tree that begins with a top/base parent policy configured with settings and rules that will apply to all computers. When you have a single collector installed in your environment to collect logs from Deep Security it is recommended to set the integration details at the Top (root/base) policy as shown below: Additionally, you can configure individual collectors for each security protection module or have all Deep Security modules to send logs to one collector depending on your requirements. Integration Options for Security Events Logs There are two integration options available to configure Deep Security Solution to forward security events to Sumo Logic, Relay via Deep Security Manager and Direct Forward. Relay via Deep Security Manager This option sends the syslog messages from the Deep Security Manager after events are collected on heartbeats as shown below: Direct Forward from Deep Security Agents This option sends the security events/messages in real time directly from the Agents as shown below: Comparison Between the Two Integration Options When you are deciding what integration option to choose from to send security events to Sumo Logic Installed Collectors among these two integration choices, consider your deep security deployment (as a Service, AWS and Azure Marketplace AMI/VM or software), your network topology/design, your available bandwidth, and deep security policy design. The table below provides comparison between these two choices for easier decision process: V. ANALYZE EVENTS LOGS Once the install and integration steps are done, you are almost set to analyze Deep Security event data in Sumo Logic. Log into the Sumo Logic console, jump down to the preview tab section, and select “install” under Trend Micro – Deep Security. Once you define the _sourceCategory, you are set to run searches, identify anomalies and correlate events across your protected workloads. You can also leverage out-of-the-box, powerful dashboards to unify, enrich and visualize security related information across your entire physical, virtual and cloud infrastructure. Sumo Logic Dashboard The Sumo Logic dashboards are a powerful visualization tool to help accelerate the time to identify anomalies and indicators of compromise (IOC). The saved searches powering these dashboards can also be leverage for forensic investigations and to reduce the time it takes for root cause analysis and remediation. The uses for Dashboards are nearly endless. Perhaps your IT security group wants to keep an eye on who is installing virtual machines. You can edit, create and save the queries you run as a panel in a Dashboard, and watch for spikes over time in a line graph. Multiple graphical options/formats are supported. Dashboards bring additional assurance, knowing that unusual activity will be displayed real time in an easy-to-digest graphical format. The data that matters the most to you is even easier to track. How to Learn More on Security For additional learning on Trend Micro Deep Security, please visit their site. To watch a video from Infor’s CISO, Jim Hoover, on how to securely scale teams, manage AWS workloads and address budget challenges, please watch here. *A special thanks to Saif Chaudhry, Principle Architect at Trend Micro and Dwayne Hoover , Sr. Sales Engineering Manager at Sumo Logic for making this integration and App a reality!

Gain better understanding of your users by visualizing and analyzing your Auth0 event logs with the Sumo Logic extension. Auth0 is a cloud-based, extensible identity provider for applications. The Sumo Logic extension for Auth0 makes it easy to analyze and visualize your Auth0 event logs and provides insight into security and operational issues. In this tutorial, we are going to install the Sumo Logic extension and explain how the dashboards we’ve created can help you quickly get a snapshot of how users are interacting with your application. To get started, you will need an Auth0 and a Sumo Logic account. Both services offer generous free tiers to get you started. Sign up for Auth0 here, and for Sumo Logic you can create an account here. You can follow the step by step tutorial below or watch our video tutorial to learn how and why combining Auth0 and Sumo Logic will be beneficial to your app. Watch the Auth0 and Sumo Logic integration video Benefits of Sumo Logic for Auth0 Before going through the process of setting up the extension, you may be asking yourself why would I even want to do this? What are the benefits? Using Auth0 as your identity provider allows you to capture a lot of data when users attempt to authenticate with your application. A lot of this data is stored in log files and easily forgotten about. Having this data visualized allows you to stay on top of what is happening in your applications. Sumo Logic makes it easy to see the latest failed logins, find and alert on error messages, create charts to visualize trends, or even do complex statistical analysis on your data. Here are some of the log types that can be collected: Logins, both successes and failures Token exchanges, both successes and failures Login failure reasons Connection errors User signup events Password changes Rate limiting events Configuring Sumo Logic to Receive Auth0 Logs To install the Sumo Logic extension, login to your Sumo Logic account and open up the Setup Wizard from the Manage top-level menu. On the next screen, you will want to select the Setup Streaming Data option. For the data type, we will select Your Custom App. Finally, select HTTP Source as the method for collecting the data logs. The last section will have you name the source category as well as select a time zone in the event one is not provided. With the configuration complete, the next screen will display the HTTP endpoint to be used for transmitting our logs. Copy the HTTP Source URL and click the Continue button to complete the setup wizard. Next, we’ll install the Sumo Logic extension from our Auth0 management dashboard. Installing the Sumo Logic Extension within Auth0 Installing the Sumo Logic extension is a fairly straightforward process. We only need the HTTP Source URL which we got when we ran through the Setup Wizard. Let’s look at the process for installing the Sumo Logic extension. Log into your Auth0 management dashboard and navigate to the Extensions tab. Scroll to find the extension title Auth0 Logs to Sumo Logic and select it. A modal dialog will open with a variety of configuration options. We can leave all the default options enabled, we’ll just need to update the SUMOLOGIC URL with the HTTP Source URL we copied earlier. Paste it here and hit save. By default, this job will run every five minutes. After five minutes have gone by, let’s check our extension and make sure that it ran properly. To do this, we can simply click into our Auth0 Logs to Sumo Logic extension and we will see the Cron job listed. Here, we can see when the job is scheduled to run again, the result of the last time it ran and other information. We can additionally click on the job name to see an in-depth history. Now that we have our Sumo Logic extension successfully installed and sending data, let’s go ahead and setup our dashboards in Sumo Logic so we can start making sense of the data. Installing the Auth0 Dashboards in Sumo Logic To install the Auth0 Dashboards in Sumo Logic, head over to your Sumo Logic dashboard. From here, select Library from the top level menu. Next, select the last tab titled Preview and you will see the Auth0 application at the very top. Note that at present time the Auth0 app is in Preview state, in the future it may be located in the Apps section. With the Auth0 app selected, click the Install button to configure and setup the app. Here, all you will need to select is the source category which will be the name you gave to the HTTP Source when we configured it earlier. You don’t have to remember the name as you will select the source from a dropdown list. We can leave all the other settings to their default values and just click the Install button to finish installing the app. To make sure the app is successfully installed, click on Library from your top level menu and select the tab titled Personal. You should see a new folder titled Auth0 and if you select it, you’ll see the two dashboards and all the predefined queries you can run. In the next section, we’ll take a look at the two dashboards Auth0 has created for us. Learning the Auth0 Dashboards We have created two different dashboards to better help you visualize and analyze the log data. The Overview dashboard allows you to visualize general login data while the Connections and Clients dashboard focuses primarily on showing you how and from where your users are logging in. Let’s look at deeper look into each of the dashboards. 1. Overview Dashboard The Overview dashboard provides a visual summary of login activity for your application. This dashboard is useful to quickly get a pulse on popular users, login success and fail rates, MFA usage, and the like. Login Event by Location. Performs a geo lookup operation and displays user logins based on IP address on a map of the world for the last 24 hours. Logins per Hour. Displays a line chart on a timeline showing the number of failed and successful logins per hour, over the last seven days. Top 10 Users by Successful Login. Shows a table chart with the top ten users with the most successful logins, including user name and count for the last 24 hours. Top 10 Users by Failed Login. Provides a table chart with the top ten users with the most failed logins, including user name and count for the last 24 hours. Top 10 Source IPs by Failed Login. Displays a table chart with a list of ten source IP addresses causing the most failed logins, including IP and count, for the last 24 hours. Top 10 User Agents. Displays the top ten most popular user agents in a pie chart from all connections for the last seven days. Top 10 Operating Systems. Shows the top ten most popular operating systems based on user agent in a pie chart for the last seven days. Guardian MFA Activity. Displays a line chart on a timeline showing the number of each Guardian MFA event per hour for the last seven days. 2. Connections and Clients Dashboard The Connections and Clients dashboard visualizes the logs that deal with how users are logging into your applications. This dashboard contains information such as countries, clients, and amount of times users login to specific clients. Logins by Client and Country. Displays a stacked bar chart showing the number of successful logins for the last 24 hours, grouped by both client and country name. This visualizes the relative popularity of each client overall, as well as in a given country. Logins by Client per Day.mShows a stacked bar chart on a timeline showing the number of successful logins for the last seven days, grouped by client per day. This shows the popularity of each client over the past week, and the relative popularity among clients. Connection Types per Hour. Provides a line chart on a timeline of the connection types used for the past seven days. Client Version Usage. Displays a line chart on a timeline of the Auth0 library version being used by all clients for the past seven days. This is useful to detect outdated clients, as well as to track upgrades. Top 10 Clients. Shows a table chart that lists the ten most popular clients, including client name and count for the past 24 hours. Top 10 Recent Errors. Provides a table chart with a list of the ten most frequent errors, including details on client name, connection, description and count for the last 24 hours. This is useful for discovering and troubleshooting operational issues. Logins by Client per Day. Shows a stacked bar chart on a timeline showing the number of successful logins for the last seven days, grouped by client per day. This shows the popularity of each client over the past week, and the relative popularity among clients. Connection Types per Hour. Provides a line chart on a timeline of the connection types used for the past seven days. Client Version Usage. Displays a line chart on a timeline of the Auth0 library version being used by all clients for the past seven days. This is useful to detect outdated clients, as well as to track upgrades. Top 10 Clients. Shows a table chart that lists the ten most popular clients, including client name and count for the past 24 hours. Top 10 Recent Errors. Provides a table chart with a list of the ten most frequent errors, including details on client name, connection, description and count for the last 24 hours. This is useful for discovering and troubleshooting operational issues. How to Learn More For additional learning on Auth0, please visit their site. For a video on how to configure the Sumo Logic App for Auth0, please watch here

Security teams need to change their approach in order to be successful in the AWS Cloud. DevSecOps in the AWS Cloud is key. DevSecOps in the AWS Cloud Sure the controls you’re using are similar but their application is very different in a cloud environment. The same goes for how teams interact as they embrace cloud technologies and techniques. The concept of DevOps is quickly becoming DevSecOps which is leading to strong security practices built directly into the fabric of cloud workloads. When embraced, this shift can result in a lot of positive change. Teams Level Up DevSecOps in the AWS Cloud With security built into the fabric of a deployment, the integration of technologies will have a direct impact on your teams. Siloed teams are ineffective. The transition to the cloud (or to a cloud mindset) is a great opportunity to break those silos down. There’s a hidden benefit that comes with the shift in team structure as well. Working hand-in-hand with other teams instead of a “gate keeper” role means that your security team is now spending more time helping the next business initiative instead of racing to put out fires all the time. Security is always better when it’s not “bolted on” and embracing this approach typically means that the overall noise of false positives and lack of context is greatly reduced. The result is a security team that’s no longer combing through log files 24/7 and other security drudge work. The shift to a DevSecOps culture lets your teams focus on the tasks they are better at. Resiliency The changes continue to pay off as your security team can now start to focus more on information security’s ignored little brother, “availability”. Information security has three primary goals; confidentiality, integrity, and availability. The easy way to relate these goals is that security works to ensure that only the people you want (confidentiality) get the correct data (integrity) when they need it (availability). DevSecOps in the AWS Cloud And while we spend a lot of time worrying and talking about confidentiality and integrity, we often ignore availability typically letting other teams address this requirement. Now with the functionality available in the AWS Cloud we can actually use aspects of availability to increase our security. Leveraging features like Amazon SNS, AWS Lambda, and Auto Scaling, we can build automated response scenarios. This “continuous response” is one of the first steps to creating self-healing workloads. When you start to automate the security layer in an environment where everything is accessible via an API some very exciting possibilities open up. This cloud security blog was written by Mark Nunnikhoven, Vice-President of Cloud Research at Trend Micro. Mark can be reached on LinkedIn at https://ca.linkedin.com/in/marknca or on Twitter @marknca. Learn More For additional learning on AWS, please visit these video resources 1. AWS re:Invent 2015 | (DVO207) Defending Your Workloads Against the Next Zero-Day Attack https://www.youtube.com/watch?v=-HW_F1-fjUU Discussion on how you can increase the security and availability of your deployment in the AWS Cloud 2. AWS re:Invent 2015 | (DVO206) How to Securely Scale Teams, Workloads, and Budgets https://www.youtube.com/watch?v=Xa5nYcCh5MU Discussion on lessons from a CISO, featuring Jim Hoover, CISO Infor along with Matt Yanchyshyn from AWS and Adam Boyle from Trend Micro.

Developing your company’s pipeline for continuous integration is a daunting and ever-evolving effort. As software becomes more complex, a need to distribute versioned internal libraries will often arise. Tools like NuGet and Maven provide excellent platforms for distribution, but running your own server can be an unnecessary hassle. Enter Bintray.What is Bintray?Bintray bills itself as “…package hosting and download center infrastructure for automated software distribution.” Perhaps a simpler way to think of it is as a way to manage and distribute your versioned products. The system provides integration with repository systems like NuGet, Maven, and Docker, making it easy to find ways of utilizing the service within your build and delivery processes. Paid subscriptions offer you additional features such as private repositories and historical statistics, while an open-source license still offers CDN-based distribution and public repositories that support formats like NuGet, NPM, YUM, and more.An Example: NuGet LibraryI spend a lot of time working in .NET and Visual Studio. The software I help build is extraordinarily complex and is comprised of over 100 different pieces. Not all of those pieces are proprietary, and some are very stable. We’ve found that in these situations, it is often helpful to repackage those components as libraries within NuGet.Here, we’ll pretend we have a library called FileHelper that is both open- source and stable, used throughout our code. We use TeamCity at work, so once the project has been separated from our main solution, we’ll have a build project configured for the library:One of the build steps would be a Powershell script that does various housekeeping before packing the project into a properly formed NuGet package (see this link for instructions). Once the package is built through NuGet commands, I’ll again use the NuGet executable to push my package to Bintray. The following notes should help you get through:Make sure you’re using a version of NuGet that is registered with your system path on Windows.Register your Bintray repository with NuGet on your machine (before trying to run this in a build step in TeamCity) using the steps described by Bintray.After you pack your nupkg file, use the described nuget push command to actually get your version pushed to Bintray’s repository.When you’ve finished all of this, you should see your new version (in this case, mine was the first version) listed on your package’s version section:Other Bintray FeaturesBintray has plenty of useful features if you subscribe. For starters, there is the ability to restrict access to packages using access tokens, which appear to work as though they were download keys. You can also white-label your downloads with your own domain, and you can even determine what countries are able to access your downloads.Like FTP, but BetterTaking care of binary distribution is an important step in the DevOps process, allowing a codebase to be much more flexible in how it handles dependencies and deployments. In the end, Bintray is a solution that can help alleviate several painful components of your development pipeline.Editor’s Note: Using Bintray with Continuous Integration is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out the Sumo Logic Developers Open Source page for free tools, API’s and example code that will enable you to monitor and troubleshoot applications from code to production.About the AuthorAndrew Male (@AndyM84) is a senior engineer at an enterprise development company in Boston, MA. Andrew has been programming from a young age and is entirely self-taught; he has spent time in many corners of the programming world including game/VR work, agency work, and teaching development to students and adults alike. He spends most of his time working on architecture design and pursuing his favorite hobby—physics.

The Center for Internet Security (CIS) released version one of the CIS AWS Foundations Benchmark in February this year. It’s a fantastic first draft, and represents the minimum security controls that should be implemented in AWS.

Last week I introduced Sumo Logic Developers’ Thought Leadership Series where JFrog’s Co-founder and Chief Architect, Fred Simon, came together with Sumo Logic’s Chief Architect, Stefan Zier, to talk about optimizing continuous integration and delivery using advanced analytics. In Part 2 of this series, Fred and Stefan dive into Docker and Dockerizing microservices. Specifically, I asked Stefan about initiatives within Sumo Logic to Dockerize parts of its service. What I didn’t realize was the scale at which these Dockerized microservices must be delivered. Sumo Logic is in the middle of Dockerizing its architecture and is doing it incrementally. As Stefan says, “We’ve got a 747 in mid-air and we have to be cautious as to what we do to it mid-flight.” The goal in Dockerizing Sumo Logic is to gain more speed out of the deployment cycle. Stefan explains, “There’s a project right now to do a broader stroke containerization of all of our microservices. We’ve done a lot of benchmarking of Artifactory to see what happens if a thousand machines pull images from Artifactory at once. That is the type of scale that we operate at. Some of our microservices have a thousand-plus instances of the service running and when we do an upgrade we need to pull a thousand-plus in a reasonable amount of time – especially when we’re going to do continuous deployment: You can’t say ‘well we’ll roll the deployment for the next three hours then we’re ready to run the code,’ That’s not quick enough anymore. It has to be minutes at most to get the code out there.” The Sumo Logic engineering team has learned a lot in going through this process. In terms of adoption and learning curve Stefan suggests: Developer Education – Docker is a new and foreign thing and the benefits are not immediately obvious to people. Communication – Talking through why it’s important and why it’s going to help and how to use it. Workshops – Sumo Logic does hands-on workshops in-house to get its developers comfortable with using Docker. Culture – Build a culture around Docker. Plan for change – the tool chain is still evolving. You have to anticipate the evolution of the tools and plan for it. As a lesson learned, Stefan explains, “We’ve had some fun adventures on Ubuntu – in production we run automatic upgrades for all our patches so you get security upgrades automatically. It turns out when you get an upgrade to the Docker Daemon it kills all the running containers. We had one or two instances where, this wasn’t in production fortunately, but in one or two instances we experienced where across the fleet all containers went away. Eventually we traced it back to Docker Daemon and now we’re explicitly holding back Docker daemon upgrades and make it an explicit upgrade so that we are in control of the timing. We can do it machine by machine instead of the whole fleet at once.” JFrog on Dockerizing Microservices Fred likewise shared JFrog’s experiences, pointing out that JFrog’s customers asked early on for Docker support. So JFrog has been in it from the early days of Docker. Artifactory has supported Docker images for more than 2 years. To Stefan’s point, Fred says “we had to evolve with Docker. So we Dockerized our pure SaaS [product] Bintray, which is a distribution hub for all the packages around the world. It’s highly distributed across all the continents, CDN enabled, [utilizes a] MongoDB cluster, CouchDB, and all of this problematic distributed software. Today Bintray is fully Dockerized. We use Kubernetes for orchestration.” One of the win-wins for Frog developers is that the components the developer is “not” working on are delivered via Docker, the exact same containers that will run in production, on their own local workstation. ‘We use Vagrant to run Docker inside a VM with all the images so the developer can connect to microservices exactly the same way. So the developer has the immediate benefit that he doesn’t have to configure and install components developed by the other team. Fred also mentioned Xray, which was just released, is fully Dockerized. Xray analyzes any kind of package within Artifactory including Docker images, Debian, RPM, zip, jar, war files and analyzes what it contains. “That’s one of the things with Docker images, it’s getting hard to know what’s inside it. Xray is based on 12 microservices and we needed a way to put their software in the hands of our customers, because Artifactory is both SaaS and on-prem, we do both. So JFrog does fully Docker and Docker Compose delivery. So developers can get the first image and all images from Bintray.” “The big question to the community at large,” Fred says, “is how do you deliver microservices software to your end customer?” There is still some work to be done here.” More Docker Adventures – TL;DR Adventures is a way of saying, we went on this journey, not everything went as planned and here’s what we learned from our experience. If you’ve read this far, I’ve provided a good summary of the first 10 minutes, so you can jump there to learn more. Each of the topics are marked by a slide so you can quickly jump to a topic of interest. Those include: Promoting containers. Why it’s important to promote your containers at each stage in the delivery cycle rather than retag and rebuild. Docker Shortcuts. How Sumo Logic is implementing Docker incrementally and taking a hybrid approach versus doing pure Docker. Adventures Dockerizing Cassandra. Evolving Conventions for Docker Distribution. New Shifts in Microservices What are the new shifts in microservices? In the final segment of this series, Fred and Stefan dive into microservices and how they put pressure on your developers to create clean APIs. Stay tuned for more adventures building, running and deploying microservices in the cloud. https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

The Chef-Sumo integration, which can be found on the Sumo Logic Developers open source page, is a way to inch towards something like automated “infrastructure problem discovery.” By “problem”, I mean services not working as intended. That could be a recurring incident, a persistent performance problem, or a security threat. Gathering useful statistics on the interaction between your code and your “Infrastructure-as-Code” will make it easier to discover these problems without intense rote manual querying. For example, a basic problem that you might discover is that something is in the logs where nothing should be. This could be a one-off security problem, or a problem across the entire service. Monitoring and alerting from kernel to user will tell you quickly. The Chef-Sumo combination is a quick and scalable way to build this solution. Chef’s verbose Infrastructure-as-Code is powerful, allowing for service description and discovery, e.g. automated AWS discovery and deployment. Sumo pares down Chef’s verbose output into dashboardable, queryable SaaS, and correlates it with other service logs, simultaneously widening coverage and narrowing focus. To Chef, Sumo is yet another agent to provision; rollout is no more complicated than anything else in a cookbook. To Sumo, Chef is yet another log stream; once provisioned, the Chef server is parsed into sources and registered in Sumo. Types of Problems This focus is critical. Since storage is cheap and logging services want lock-in, the instinct in DevOps is to hoard information. Too often, teams suffer from a cargo cult mentality where the data’s “bigness” is all that matters. In practice, this usually means collecting TBs of data that are unorganized, poorly described and not directed towards problem-solving. It’s much easier to find needles in haystacks with magnets. With infrastructure logs, that means finding literal anomalies, like an unknown user with privileged access. Or sometimes it means finding pattern mismatches or deviations from known benchmarks, like a flood of pings from a proxy. Problem Solving on Rails Sumo has two out-of-the-box query tools that can make the problem-solving process simpler— Outlier and Anomaly. These are part of Sumo’s “log reduce” family. Outlier tracks the moving average and standard deviation of a value, allowing for alerts and reports when the difference between the value exceeds the mean by some multiple of the standard deviation. Here’s an example query for a simple AWS alert: | source=AWS_inFooTown | parse “* * *: * * * * * * * * \ “* *://*:*/* HTTP/” as server, port, backend | timeslice by 1m | avg(server) as OKserver, avg(port) as OKport, avg(backend) as OKbackend by _timeslice | (OKserver+OKport+OKbackend) as total_time_OK | fields _timeslice, total_time_OK | outlier total_time_OK In other search tools, this would require indexing and forwarding the sources, setting up stdev searches in separate summary indexes, and collecting them on a manually written average. Not only does that take a lot of time and effort, it requires knowing where to look. While you will still need to parse each service into your own simple language, not having to learn where to deploy this on every new cookbook is a huge time-saver. Anomaly is also a huge time-saver, and comes with some pre-built templates for RED/YELLOW/GREEN problems. It detects literal anomalies based on some machine learning logic. Check here to learn more about the logic’s internals. How to Look before You Leap While it’s all hyperloops and SaaS in theory, no configuration management and monitoring rollout is all that simple, especially when the question is “what should monitor what” and the rollout is of a Chef-provisioning-Sumo-monitoring-Chef process. For example, sometimes the “wrong” source is monitored when Chef is provisioning applications that each consume multiple sources. The simplest way to avoid the confusion at the source is to avoid arrays completely when defining Sumo. Stick with hashes for all sources, and Chef will merge based on the hash-defined rules. Read CodeRanger’s excellent explanation of this fix here. This is a pretty tedious solution, however, and the good folks at Chef and Sumo have come up with something that’s a lot more elegant: custom resources in Chef, with directives in the JSON configuration. This avoids source-by-source editing, and is in line with Sumo’s JSON standards. To get started with this approach, take a look at the custom resources debate on GitHub, and read the source for Kennon Kwok’s cookbook for Sumo collectors. Editor’s Note: Automated Infrastructure Problem Discovery using Sumo Logic and Chef is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out the Sumo Logic Developers Open Source page for free tools, API’s and example code that will enable you to monitor and troubleshoot applications from code to production. About the Author Alex Entrekin served on the executive staff of Cloudshare where he was primarily responsible for advanced analytics and monitoring systems. His work extending Splunk into actionable user profiling was featured at VMworld: “How a Cloud Computing Provider Reached the Holy Grail of Visibility.” Alex is currently an attorney, researcher and writer based in Santa Barbara, CA. He holds a J.D. from the UCLA School of Law. Resources: LearnChef on Custom Resources CodeRanger on Solving Attribute Merging Problems in Chef with Hashes Gist of Custom Chef Resources in Sumo by Ken Kwok More Ken Kwok on Custom Resources in Chef & Sumo Using Multiple JSON files to Configure Sumo Sources

As part of the Sumo Logic’s Thought Leadership Series, JFrog’s Co-founder and Chief Architect, Fred Simon, came together with Sumo Logic’s Chief Architect, Stefan Zier, last week to share their insights into topics ranging from continuous integration and delivery to building apps with Docker and architecting for microservices. The series launches in advance of a webinar on July 21 at 10:00 am that dives into the JFrog Sumo Logic integration.

Recently, Sumo Dojo ran a contest in the community see who is analyzing Docker logs with Sumo Logic, and how. The contest ran the month of June and was presented at DockerCon. Last week, the Sumo Dojo selected the winner, Brandon Milsom, from Australia-based company Fugro Roames. Roames uses remote sensing laser (LIDAR) technology to create interactive 3D asset models for powerline networks for energy companies in Australia and the United Kingdom. As Brandon writes: "We use Docker and Sumo Logic as a part of our deployment automation. We use Ansible scripts to automatically deploy our developer’s applications onto Amazon EC2 instances inside Docker containers as part of our cloud infrastructure. These applications are automatically configured to send tagged logs to Sumo Logic using Fluentd, which our developers use to identify their running instances for debugging and troubleshooting. Not only are the application logs sent directly to Sumo Logic, but the Docker container logs are also configured using Docker’s built in Fluentd logging driver. This forwards logs to another Docker container on the same host running a Fluentd server, which then seamlessly ships logs over to Sumo Logic. The result is developers easily access their application and container OS logs that their app is running in just by opening a browser tab." Part of our development has also been trialling drones for asset inspection, and we also have a few drone fanatics in our office. Winning a drone would also be beneficial as it would give us something to shoot at with our Nerf guns, improving office morale. Brandon's coworker, Adrian Howchin also wrote in saying" "I think one of the best things that we've gained from this setup is that it allows us to keep users from connecting (SSH) in to our instances. Given our CD setup, we don't want users connecting in to hosts where their applications are deployed (it's bad practice). However, we had no answer to the question of how they get their application/OS logs." Thanks to SumoLogic (and the Docker logging driver!), we're able to get these logs out to a centralized location, and keep the users out of the instances. Congratulations to Brandon and the team at Fugro Roames. Now you have something cool to shoot at.

Docker Shipyard is a handy tool for people who love Docker Swarm, but wish it did even more. While Swarm focuses on container orchestration through the CLI, Docker Shipyard takes things further by letting you manage app images and container registries in addition to containers themselves. Plus, Shipyard offers a Web-based graphical front-end and a rich API in addition to a CLI.

https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

Every time I go to NYC I get a sudden surge of energy along with some edginess. During my first meeting ever in NYC I had a guy tell me, “You have 15 bleep bleep and bleep minutes to show me some value or your bleep is out the door.” I love this attitude because you know EXACTLY where you stand. The city that never sleeps sets the tone and pecking order when it comes to trends. I say usually because for the last two or three years, the good people of NYC have seemed a bit behind in technology and cloud services.

Containers as a Service (CaaS) is a new term that has arisen over the last several months. It strikes a nice middle ground between Infrastructure as a Service and Platform as a Service when deploying applications infrastructure. Infrastructure as a Service requires managing individual segments of infrastructure and entire low-level configurations in one or a number of Infrastructure as a Service providers.

Developers, IT Ops engineers and enterprise professionals converged on DockerCon 2016 with a vengeance and Sumo Logic was on hand to show them how the Sumo Logic platform gives them visibility into their Docker ecosystems. We also released a new eBook for practitioners, Docker, From Code to Container that provides best practices for building, testing and deploying containers and includes hands-on exercises with Docker Compose. The Sumo Logic Community also announced a chance to win a DJI Phantom 3 Drone. The contest ends June 30, so there’s still time. With announcements of new features like Containers as a Service and tools like Docker Universal Control Plane (UCP), Docker is taking the deployment of microservices via containers to a whole new level. UCP offers automated container scanning and the ability run signed binaries. As a primarily DevOps crowd with a heavy bent toward the developer, there was a lot of interest in Docker logging, monitoring and analytics, and we received a lot of questions about the internals of the Sumo Logic approach to collecting logs. In fact, the #1 question I got was how we implemented the container, so I thought I’d answer that and other questions here. How Does Sumo Logic Operate in a Docker Ecosystem? Sumo Logic uses a container to collect and ship data from Docker. The image itself contains a collector and a script source. You can grab the image from DockerHub by just running a Docker pull. docker pull sumologic/appcollector:latest Before you run the container, you’ll need to create an access key in Sumo Logic (see documentation for details). Then run the container using the AccessID and Access_key that you created previously. docker run -d -v /var/run/docker.sock:/var/run/docker.sock --name="sumologic-docker" sumologic/appcollector:latest The container creates a collector in your Sumo Logic account, and establishes two sources: Docker Logs and a Docker Stats. That’s it. Once the image is installed and configured locally, you simply select the App for Docker from the Library in Sumo Logic, bring up the one of the dashboards and watch data begin to populate. If you’d like to try it out yourself and don’t already have an account, sign up for Sumo Logic Free. Can you monitor more than One Docker Host? Another question I got was whether you could monitor more than one host. Apparently not all monitoring tools let you do this. The answer is, you can. As you can see in this Overview Dashboard, there are two Docker Hosts in this example. The Sumo Logic collector image typically runs on the same host as the Docker host. You can collect data from multiple hosts by installing an image on each host. Note, however, that you can only run one instance at a time. A better approach is to run the Sumo Logic Collector on one host, and have containers on all other hosts log to it by setting the syslog address accordingly when running the container. Our CTO, Christian Beedgen explains more in this post on Logging Drivers. What kind of data do you capture and what analytics do you provide? To get real value from machine-generated data, Sumo Logic takes a comprehensive approach to monitoring Docker. There are five requirements to enable comprehensive monitoring: Events Configurations Logs Stats Host and daemon logs For events, you can send each event as a JSON message, which means you can use JSON as a way of logging each event. The Sumo Logic collector enumerates all running containers, then starts listening to the event stream, collecting each running container and each start event. See my post on Comprehensive Monitoring in Docker for more detail. We call the inspect API to get configurations and send that in JSON. For logs, we call the logs API to open a stream and send each log. Now you have a record of all the configurations together with your logs, making it easy search for them when troubleshooting. For statistics, we call the stats API to open a stream for each running container and each start event, and send each received JSON message as a log. For host and daemon logs, you can include a collector into host images or run a collector as a container. Do you have any Free Stuff? No conference would be complete with a new backpack stuffed with hoodies, T-shirts and may be a Red Hat (Thanks guys!) But I also believe in adding value by educating developers and ops. So, I’ve put together an eBook, Docker – From Code to Container, that I hope you’ll find interesting. Docker From Code to Container explains how containerization enables Continuous Integration and Continuous Delivery processes, shows how you can take Docker to production with confidence, walks you through the process of building applications with Docker Compose, and presents a comprehensive model for monitoring Docker both your application stack and your Docker ecosystem. Ultimately, you will learn how containers enable DevOps teams build, run and secure their Dockerized a applications. In this Webinar you will learn: How Docker enables continuous integration and delivery Best practices for delivering Docker containers to production How to build Applications with Docker Compose Best practices for securing docker containers How to gauge the health of your Docker ecosystem using analytics A comprehensive approach to monitoring and logging What’s Next I’m glad you asked. We’re featuring a Live Webinar with Jason Bloomberg, president of Intelyx and Kalyan Ramanathan, VP of Marketing for Sumo Logic to dive deeper into the use cases for Docker monitoring. The webinar is July 20 at 10:00 am PDT. Be there or be square! https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

When you think of Ulta Beauty, you most likely think of makeovers, not technology. But in fact, we’ve also been doing makeovers of our technology to bring together all the technical groups that touch our guests (application development, operations, e-commerce, plus our off-shore teams) under one organization to drive the best possible guest experience. For those of you who aren’t familiar with Ulta Beauty, we’re the fastest growing beauty retailer with both brick and mortar and an online presence. Back in 2014, we experienced some challenges with online guest order fulfillment during the holiday season (our busiest time of the year). Our business partners simply lacked the visibility into inventory levels during peak season. We identified the problem in advance, but due to time constraints weren’t able to resolve so we took a novel approach using machine data analytics for better visibility. We knew we needed a solution to streamline operations and proactively identify product trends. We selected Sumo Logic to help us get a real-time view of our order throughput so that we could manually bring down order levels if throughput went too high. In my role as VP of Guest-facing Services, I rely on deep technical knowledge and business sense to make sure our applications are running smoothly. Sumo Logic was easy enough for us to manage on our own. It’s flexible and simple but powerful, which enables me to ensure my business stakeholders are empowered with the information they need to be successful. Fast forward to holiday season 2015. We not only improved our backend systems but also expanded and operationalized Sumo Logic with our DevOps, App Dev and Business Partner teams. We created multiple dashboards and reports to identify hot selling items, what’s trending, inventory levels and more. This was huge for our business partners, who could then consciously make inventory business decisions on their own. The biggest impact of rolling out Sumo Logic has been the ability to impact the guest experience in a positive way and effectively manage the channel. I confess that I became a bit of a Sumo Logic dashboard addict. During the holiday season, if I was out and about, I would check the mobile view so frequently that I blew my cellular data plan. What’s next for Ulta and Sumo Logic? We’re expanding our use of Sumo and validating new business use cases for point-of-sale, network infrastructure and warehouse management systems. With Sumo’s assistance, we’re creating an enterprise application performance management roadmap that incorporates Sumo Logic’s machine data analytics to ensure the maximum reliability and stability of our business-critical systems. Now that’s a beautiful makeover!

Security was once the bane of Docker containers. As recently as 2014, bloggers were writing things like "using containers for security isolation is not a good idea" and that Docker is not safe "in absolute terms."

LXC and LXD are two important acronyms to know if you’re into containers. Unfortunately, they’re also acronyms that are hard to keep straight from one another. They sound alike. They refer to similar platforms, which were created in large part by the same company. And they’re deeply intertwined with one another at a technical level.

Application development for the cloud has always been challenging. Cloud applications tend to run on headless Linux machines, with little or no development tools installed. According to a recent survey, most developers either use Windows or Mac OS X as their primary platform.

If you’re not a builder or architect, you may not be familiar with Dodge Data & Analytics. We help building product manufacturers, general contractors and subcontractors, architects and engineers to size markets, prioritize prospects, strengthen market positions and optimize sales strategies. Simply put, we build the analytics engine for the builder community. In our industry, it’s important that we deliver a consistent level of operational availability in order to best serve our clients. We didn’t have a solution in place for machine data analytics and needed a way to make better use of our logs and time-series metrics data to quickly surface and remediate known and unknown issues. Sumo Logic was a great choice for helping us monitor and manage our system’s operational metrics based on ease of deployment, maintenance and support, and powerful search queries. Sumo Logic’s machine data analytics platform allows our teams to accurately correlate meaningful information to provide root cause analysis and operational behavior patterns. Literally with one click of a button we have access to our data, giving us better real-time insights into our own infrastructure, so we can in turn serve up better insights for our customers. Sumo Logic calls this continuous intelligence and it’s our approach as well. Tighter organizational collaboration is also important to us. Sumo Logic is helping to bring together various teams within Dodge, including our IT operations team, DevOps, DBAs and incident managers. It provides single version of the truth for monitoring and troubleshooting so that we work better together and solve problems faster and isn’t that what it’s all about? About the bloggers: Doug Sheley is Director of Infrastructure Systems. Jay Yeras is a system administrator at Dodge Data & Analytics, unofficially known as “Doer of Things”.

Is SIEM still a great technology? Can SIEM keep the pace with the security needs of modern enterprises?

SIEM History: Is SIEM Dead? Often times when I am presenting at conferences around the country, people will ask me “Is SIEM Dead”? Such a great question! Has the technology reached its end of life? Has SIEM really crashed and burned? I think the answer to that question is NO. SIEM is not dead, it has just evolved. History of SIEM SIEMs unfortunately have struggled to keep pace with the security needs of modern enterprises, especially as the volume, variety and velocity of data has grown. As well, SIEMs have struggled to keep pace with the sophistication of modern day threats. Malware 15 years ago was static and predictable. But today’s threats are stealthy and polymorphic. Furthermore, the reality is that few enterprises have the resources to dedicate to the upkeep of SIEM and the use of SIEM technology to address threat management has become less effective and waned. Gartner Analyst Oliver Rochford famously wrote, “Implementing SIEMs continues to be fraught with difficulties, with failed and stalled deployments common.”(1) In Greek mythology, a phoenix (Greek: φοῖνιξ phoinix; Latin: phoenix, phœnix, fenix) is a long-lived bird that is cyclically regenerated or reborn. Associated with the sun, a phoenix obtains new life by arising from the ashes of its predecessor. Phoenix rising from the SIEM ashes The SIEM ashes are omnipresent and security analytics is emerging as the primary system for detection and response. Deconstructing SIEM Although we use the term SIEM to describe this market, SIEM is really made up of two distinct areas: SIM or Security Information Management (SIM) deals with the storage, analysis and reporting of log data. SIM ingests data from host systems, applications, network and security devices. SEM on the other hand, or Security Event Management (SEM), processes event data from security devices, network devices, systems and applications in real time. This is dealing with the monitoring, correlating and notification of security events that are generated across the IT infrastructure and application stack. Folks generally do not distinguish between these two areas anymore and just use “SIEM” to describe the market category. However, it’s important to take note of what you are trying to accomplish and which problems you are trying to solve with these solutions. Why Do We Care About SIEM? One could easily dismiss these solutions outright, but the security market is huge – $21.4B in 2014 according to our friends at Gartner. And the SIEM piece alone reached $1.6B last year. According to 451 Research the security market has around 1,500-1,800 vendors broken down into a number of main categories across IAM, EPP, SIEM, SMG, SWG, DLP, Encryption, Cloud Security, etc. And within each of these main categories, there are numerous sub categories. Security Landscape And despite the billions of dollars invested, current security and SIEM solutions are struggling to keep the bad guys out. Whether cyber criminals, corporate spies, or others, these bad actors are getting through. The Executive Chairman and former CEO of Cisco Systems famously said, “There are two types of companies, those who have been hacked and those who have no clue.” Consider for a moment that the median # days before a breach is detected exceeds 6 ½ months and that the % of victims notified by external 3rd parties is almost 70% (3). People indeed have no clue! Something different is clearly needed. This is the first in a series of blogs on SIEM and Security Analytics. Stay tuned next week for our second blog titled “SIEM and Security Analytics: Head to Head.” Additional SIEM Resources Find out how Sumo Logic helps deliver advanced security analytics without the pain of SIEM Sign up for a free trial of Sumo Logic. It’s quick and easy. Within just a few clicks you can configure streaming data, and start gaining security insights into your data in seconds. Mark Bloom runs Product Marketing for Compliance & Security at Sumo Logic. You can reach him on LinkedIn or on Twitter @bloom_mark Sources (1) Gartner: Overcoming Common Causes for SIEM Deployment Failures by Oliver Rochford 21Aug2014 (2) Forrester: Evolution of SIEM graph, taken from Security Analytics is the Cornerstone of Modern Detection and Response, December 2015 (3) Mandiant mTrends Reports

As distributed computing becomes more and more available, it’s good to evaluate the different options accessible to you and your organization. At Build 2016, Microsoft announced the upcoming availability of Azure Functions, an event-driven, on-demand compute system similar to AWS Lambda Functions. Azure Functions are still a new feature in Azure, so there are bound to be growing pains while they mature. With that in mind, let’s dig deeper! Building a Twitter Aggregator with Azure Functions I’ve always lived by the mantra that learning is best done by doing. Imagine that we are building a service that generates profile information for users by aggregating their content from different websites/services. We can create a Twitter aggregator as an Azure Function. This Function (which would be part of a group of Functions for aggregating content on-demand) will accept two arguments from a web request: ID and Handle. The ID will represent a mythical unique identifier created by our system elsewhere. The Handle will be the supplied Twitter handle/screen-name of the user who has requested a profile be generated. Once these arguments are supplied, the Function should connect to Twitter and pull down the five most recent tweets for use in the profile. Creating a new Function is incredibly simple. Head over to portal.azure.com/, and search for “Function App” in the marketplace to add a new resource. It is listed underneath “Web + Mobile.” Alternatively you can go to functions.azure.com/ and get a more streamlined creation process. Either way, you are presented with the ability to name your Function and choose things like its physical location and subscription. After that, you are taken straight to an editor where you can begin configuring and coding the Function. I’ll skip through the boring parts and get to the code because there’s a bit of effort involved due to working with Twitter. Here’s a link to the completed Function: I won’t go through every line of code, but there are important elements to discuss. Line #1 is #r “Newtonsoft.Json”, which is a way of referencing assemblies. You can provide new assemblies by taking advantage of the Continuous Integration options and tying your Function to a source control repository with a bin directory that contains pre-compiled DLLs. For more information, see the documentation here. You’ll notice that I’m not creating any new classes within my Function. This is not a requirement, as this is simply a C# “script.” You are free to create classes, but as of the time of this writing, you still cannot create namespaces for organizing your classes. (This can still be achieved by building libraries that your script uses, as referenced in #1). I am making use of the HttpRequestMessage.CreateResponse() method for my Function’s responses, but as I’ll discuss in a moment, it was the “quick and dirty” approach to building a new Function. There are other options that may fit your use cases better. Feel free to explore. I opted to not use any pre-built Twitter libraries for simplicity. I don’t generally recommend this, but it also isn’t terribly complicated for app-only usage, so make a decision based on your use case. Possibilities Everywhere Although I chose to build my Function in C#, Microsoft provides numerous language options including: C# F# Powershell Batch Bash JavaScript PHP Python There’s also a wide range of options for input and output. I set up my Function as a generic webhook, meaning it provides me with an API endpoint to call for triggering the Function. If that’s not your cup of tea, though, maybe one of these satisfies: Storage Blobs (trigger whenever a file is updated/added to an Azure Storage Container) Event Hubs (trigger whenever an event is received from an Azure Event Hub) GitHub WebHook (trigger whenever a GitHub webhook request is received) HttpTrigger (trigger based off of HTTP requests) QueueTrigger (trigger whenever a message is added to an Azure Storage Queue) ServiceBusQueueTrigger (trigger whenever a message is added to a ServiceBus instance) TimerTrigger (trigger on a schedule, similar to scheduled tasks or cron jobs) There is also a full set of options included to direct request output. For my Function, I simply assumed that returning the JSON as a web response was reasonable, but there are likely much better/scalable ways of directing the output into our profiles. See the full documentation for more information. In Closing While Sumo Logic doesn’t yet provide an integration for Azure Functions, you can get visibility into other Microsoft technologies using tools like the Sumo Logic Application for Office 365 (currently in Preview) that offersPre-built dashboards, searches and reports visually highlight key activity across your O365 environment including OneDrive, Exchange Online, SharePoint, and Azure Active Directory. Check out other apps like the App for Windows Directory at sumologic.com. Hopefully this has been a simple introduction to the concept of Azure Functions. The sheer number of things that can be accomplished with them is daunting, but they are fantastic for the world of distributed computing. The documentation and community for Functions is already growing, so don’t be afraid to take them for a spin. Resources Complete Source for Example About the Author Andrew Male (@AndyM84) is a senior engineer at an enterprise development company in Boston, MA. Andrew has been programming from a young age and is entirely self-taught. He has spent time in many corners of the programming world, including game/VR work, agency work, and teaching development to students and adults alike. He spends most of his time working on architecture design and pursuing his favorite hobby—physics.

Sumo Logic CTO Christian Beedgen talks about how to build an architectural platform for unified logs and metrics.

If you are looking for application logging solutions, the “direct-to-cloud” approach is usually your best option. This approach however, is not necessarily a panacea for all of your logging and monitoring needs. Depending on your business requirements around logging and monitoring, this initial approach may be all you ever need. Or, you may find out later that your needs have changed, and you need to modify your approach to increase reliability, or to capture full stack logs. Sumo Logic is committed to making it easy to capture all of your application data. If you are running on AWS, your AWS specific logs from Cloudtrail, S3, Elastic Load Balancing and many more will most likely end up in Amazon’s Cloudwatch Logs. The solution Amazon provides for distributing these logs for analysis is Amazon Kinesis. With help from the Sumo community, Kinesis is also getting the Sumo treatment. For developers already using Kinesis to stream their logs, they can now also stream to Sumo through the Kinesis connector. If you are not yet using Kinesis and are instead using something like S3 to store your AWS logs, consider making the switch. Storing your logs in a file system like S3 increases the time it takes for other applications to collect use and analyze those logs. And let’s face it, as long as it remains un-analyzed, that treasure of data is basically a useless waste of storage space. Instead, Kinesis treats your logs as a continuous data stream which allows near real time analysis instead of forcing you to wait to see your data. Logs from the Kinesis stream are transformed into JSON by the new Sumo connector and all fields can be conveniently extracted by Sumo's JSON auto-parser. The community project can be found here on GitHub. To get the most Sumo functionality out of your technology stack, visit our GitHub page to see all open-source projects: ready and in-progress. If you would like to see more Sumo support for your technologies, come join the community help us out!

If you are looking for application logging solutions, the “direct-to-cloud” approach is usually your best option. This approach however, is not necessarily a panacea for all of your logging and monitoring needs. Depending on your business requirements around logging and monitoring, this initial approach may be all you ever need. Or, you may find out later that your needs have changed, and you need to modify your approach to increase reliability, or to capture full stack logs. Sumo Logic is committed to making it easy to capture all of your application data. If you are running on AWS, your AWS specific logs from Cloudtrail, S3, Elastic Load Balancing and many more will most likely end up in Amazon’s Cloudwatch Logs. The solution Amazon provides for distributing these logs for analysis is Amazon Kinesis. With help from the Sumo community, Kinesis is also getting the Sumo treatment. For developers already using Kinesis to stream their logs, they can now also stream to Sumo through the Kinesis connector. If you are not yet using Kinesis and are instead using something like S3 to store your AWS logs, consider making the switch. Storing your logs in a file system like S3 increases the time it takes for other applications to collect use and analyze those logs. And let’s face it, as long as it remains un-analyzed, that treasure of data is basically a useless waste of storage space. Instead, Kinesis treats your logs as a continuous data stream which allows near real time analysis instead of forcing you to wait to see your data. Logs from the Kinesis stream are transformed into JSON by the new Sumo connector and all fields can be conveniently extracted by Sumo's JSON auto-parser. The community project can be found here on GitHub. To get the most Sumo functionality out of your technology stack, visit our GitHub page to see all open-source projects: ready and in-progress. If you would like to see more Sumo support for your technologies, come join the community help us out!

If you are looking for application logging solutions, the “direct-to-cloud” approach is usually your best option. This approach however, is not necessarily a panacea for all of your logging and monitoring needs. Depending on your business requirements around logging and monitoring, this initial approach may be all you ever need. Or, you may find out later that your needs have changed, and you need to modify your approach to increase reliability, or to capture full stack logs. Sumo Logic is committed to making it easy to capture all of your application data. If you are running on AWS, your AWS specific logs from Cloudtrail, S3, Elastic Load Balancing and many more will most likely end up in Amazon’s Cloudwatch Logs. The solution Amazon provides for distributing these logs for analysis is AWS Kinesis. With help from the Sumo community, Kinesis is also getting the Sumo treatment. For developers already using Kinesis to stream their logs, they can now also stream to Sumo through the Kinesis connector. If you are not yet using Kinesis and are instead using something like S3 to store your AWS logs, consider making the switch. Storing your logs in a file system like S3 increases the time it takes for other applications to collect use and analyze those logs. And let’s face it, as long as it remains un-analyzed, that treasure of data is basically a useless waste of storage space. Instead, Kinesis treats your logs as a continuous data stream which allows near real time analysis instead of forcing you to wait to see your data. Logs from the Kinesis stream are transformed into JSON by the new Sumo connector and all fields can be conveniently extracted by Sumo's JSON auto-parser. The community project can be found here on GitHub. To get the most Sumo functionality out of your technology stack, visit our GitHub page to see all open-source projects: ready and in-progress. If you would like to see more Sumo support for your technologies, come join the community help us out!

This week, CEO Ramin Sayar offered insights into Sumo Logic’s Unified Logs and Metrics announcement, noting that Sumo Logic is now the first and foremost cloud-native, machine data analytics SaaS to handle log data and time-series metrics together. Beginning this week Sumo Logic is providing “early access” to customers that are using either Amazon CloudWatch or Graphite to gather metrics. That’s good news for practitioners from developers to DevOps and release managers, because as Ben Newton explains in his blog post you’ll now be able to view both logs and metrics data together and in context. For example, when troubleshooting an application issue, developers can start with log data to narrow a problem to a specific instance, then overlay metrics to build screens that show both logs and metrics (like CPU utilization over time) in the context of the problem. What Are you Measuring? Sumo Logic already provides log analytics at three levels: System (or machine) Network Application Unified Logs & Metrics also extends the reporting of time-series data to these three levels. So using Sumo Logic you’ll now be able to focus on application performance metrics, infrastructure metrics, custom metrics and log events. Custom Application Metrics Of the three, application metrics can be the most challenging because as your application changes, so do the metrics you need to see. Often you don’t know what you will be measuring until you encounter the problem. APM tools provide byte-code instrumentation where they load code into the JVM. That can be helpful, but results are restricted to what the APM tool is designed or configured to report on. Moreover, the cost for instrumenting code using APM tools can be expensive. So developers, who know their code better than any tool, often resort to creating their own custom metrics to get the information needed to track and troubleshoot specific application behavior. That was the motivation behind an open-source tool called StatsD. StatsD allows you to create new metrics in Graphite just by sending it data for that metric. That means there’s no management overhead for engineers to start tracking something new: simply give StatsD a data point you want to track and Graphite will create the metric. Graphite itself has become a foundational monitoring tool, and because many of our customers already use it Sumo Logic felt it important to support it. Graphite, which is written in Python and open-sourced under the Apache 2.0 license, collects, stores and displays time-series data in real time. Graphite is fairly complex, but the short story is that it’s good at graphing a lot of different things like dozens of performance metrics from thousands of servers. So typically you write an application that collects numeric time-series data and sends it to Graphite’s processing backend (Carbon), which stores the data in a Graphite database. The Carbon process listens for incoming data but does not send any response back to the client. Client applications typically publish metrics using plaintext, but can also use the pickle protocol, or Advanced Message Queueing Protocol (AMQP). The data can then be visualized through a web interface like Grafana. But as previously mentioned, your custom application can simply send data points to a StatsD server. Under the hood StatsD is a simple NodeJS daemon that listens for messages on a UDP port, then parses the messages, extracts the metrics data, and periodically (every 10 seconds) flushes the data to graphite. Sumo Logic’s Unified Logs and Metrics Getting metrics into Sumo Logic is super easy. With StatsD and Graphite, you have two options. You can point your StatsD server to a Sumo Logic hosted collector or you can install native collector within the application environment. CloudWatch CloudWatch is Amazon’s service for monitoring applications running on AWS and system resources. CloudWatch tracks metrics (data expressed over a period of time) and monitors log files for EC2 Instances and other AWS resources like EBS volumes, ELB, DynamoDB tables, and so on. For EC2 Instances, you can collect metrics on things like CPU Utilization, then apply dimensions to filter by instance ID, instance type, or image id. Pricing for AWS CloudWatch is based on Data Points. A DP = 5 minute of activity (specifically the previous minutes). A Detailed DP (DDP) = 1 minute. Unified Logs and Metrics dashboards allow you to view metrics by category, and are grouped first by namespace, and then by the various dimension combinations within each namespace. One very cool feature is you can search for meta tags across EC2 instances. Sumo Logic makes the call once to retrieve meta tags and caches them. That means you no longer have to make an API call to retrieve each meta tag, which can result in cost savings since AWS charges per API call. Use Cases Monitoring – Now you’ll be able to focus on tracking KPI behavior over time with Dashboards and Alerts. Monitoring allows you to: Track SLA adherence Watch for anomalies Respond quickly to emerging issues Compare to past behavior Troubleshooting – This about determining if there is an outage and then restoring service. With Unified Logs and Metrics you can: Identify what is failing Identify when it changed Quickly iterate on ideas “Swarm” issues Root-cause Analysis – Focuses on determining why something happened and how to prevent it.Dashboards overlayed with log data and metrics allows you to: Perform historical analysis Correlate Behavior Uncover long term fixes Improve Monitoring Correlating Logs and Metrics When you start troubleshooting you really want to start correlating multiple types of metrics and multiple sources of log data. Ultimately, you’ll be able to start with Outliers and begin overlaying metrics and log data to quickly build views and help you quickly identify issues. Now you’ll be able to overlay log and metrics from two different systems and do it in real time. If you want to see what Unified Logs and Metrics can do, Product Manager Ben Newton walks you through the steps of building on logs and overlaying metrics in this short introduction.

Two weeks ago, we announced an important and exciting milestone for Sumo Logic: more than a 1,000 customers are receiving instant value from our machine data analytics platform to build, run and secure their modern applications and cloud infrastructures. Supported by the platform’s powerful, machine-learning algorithms, our customers gain business speed and insights from uncovering application patterns and anomalies, speeding issue resolution, enabling deeper security monitoring and understanding customer usage needs – all from analyzing real-time log data streams, or what we call, “Continuous Intelligence.”

Together forever and never to part Together forever we two And don't you know I would move heaven and earth To be together forever with you Rick Astley There are lots of things in this world that when put together, prove to be more than the sum of their parts. For example, take a favorite dessert of nearby Napa Valley - vanilla ice cream, olive oil and sea salt. The first time you hear it, you might think -- “You are going to pour oil on my ice cream, and then top it off with some salt? Yuck! Thanks for ruining dinner.” Take the simple, but delicious peanut butter and jelly. Many europeans think it's disgusting (this is the same culture that finds snails or marmite delectable, so consider the source.). Yet it is now so American that I have heard on good authority it’s being considered as a requirement for U.S. citizenship (along with eating apple pie and listening to Taylor Swift, of course). So, before you start thinking this is a food blog, let’s get to the point. In the world of IT we have something similar - logs and metrics. Simply put, logs are usually records of something that happened at some point in time (e.g. application errors, warnings, application events, etc.) and metrics are typically measurements of something over time (e.g. the response time of the home page, server CPU, server memory, etc.). In the past, those types of data were often treated very differently. We at Sumo Logic believe that’s a problem, and have worked to solve it. In this blog, I want to dig a little deeper into why logs and metrics go together like peanut butter and jelly. Building the Tesla for Logs & Metrics Once a software company builds a product and starts selling it, it’s very hard to change the skeleton of the thing - the architecture. Switching analogies for a second - consider for a moment that software is similar to cars. When automotive designers are faced with a new problem or scenario, the easiest approach is to tinker with the existing design. This works if you’re adding new tires to an SUV for off-roading. It does not work if you are designing a fully electric car. Tesla completely reworked everything - the transmission, the engine - even the sales model. In the case of systems dealing with machine data, the tinkering approach has definitely not been successful. For example, some log analytics companies, having built engines to make sense of messy, unstructured logs, have also tweaked their mechanics to address highly structured time series data (e.g. metrics). The end result is a lot of hoop jumping to get their engines to perform the way users expect. On the other hand, companies that have chosen to build metrics analysis engines have tried their hand at logs - or at least they say they do. However, in reality, the “logs” passing through the metrics engines are so massaged and the messiness wrung mercilessly out of them, that you end up with something more akin to metrics data. So, this approach may improve engine performance, but breaks down when it comes to accuracy -- all for the sake of expediency. We dealt with this same conundrum at Sumo Logic. We decided to take a big risk and actually evolve our engine design, resulting in a next generation architecture. We built a purpose-built, high-performance engine for time series data, alongside, and integrated with, our high performance log analytics engine. We didn’t forget all of the lessons of the past in building a highly scalable, multi-tenant application -- we just didn’t take shortcuts. The end result is a platform that treats both types of data natively, or, what I consider to be the respect they deserve. It’s all About the Analytics The secret of a great peanut butter and jelly sandwich is not just bringing unique ingredients together, but creating it with the entire sandwich experience in mind. That applied to us as well when we unified logs and metrics -- what’s the point of it all? For example, we aren’t a data warehouse in the sky. People come to Sumo Logic to make sense of their data -- to do analytics. And, the core of analytics is the type of questions asked of the data. Because of data’s nature, some types of data answer certain types of questions better than others. In the case of logs, users often find themselves looking for the proverbial needle in the haystack, or, to be more accurate, hundreds of needles sprinkled across a field of haystacks. So, log analytics has to excel at “x-raying” those haystacks to get to the offending needles very quickly, and even better, detecting patterns in the needles - basically troubleshooting and root cause analysis. Counting the needles, haystacks, or even the hay straw itself is of secondary concern. With metrics, the goal is often to to look for behavior or trends over time. Users rely on time-series metrics to understand how the things they measure change over time, and whether it indicates something deeper beneath the surface. Fitness devices (like the Apple Watch) are a perfect example here. I can track my heart-rate, running speed, distance, etc., over time. This helps me determine if getting out of bed to run outside was remotely worth it. If my stats improve over time, then yes. If no, then I’m sleeping in! At Sumo Logic, we knew we couldn’t afford to ignore the different ways that people use their data. And, simply put, we couldn’t treat metrics like a half-brother of logs. So, we focused on tools that help uncover system and application behavior, not just draw nice looking graphs. Users can overlay related data to find anomalies and patterns in the data. And we strove to make the system as real-time as possible, shaving off microseconds wherever we could so the data is as relevant and timely as possible. Listen to the Users A platform that optimizes for the data and the analytics, but ignores the user, will end up being stiff and hard to use. In building this new functionality at Sumo Logic, we’ve strived to understand our users. That meant investing a lot of time and energy in talking to our users, and listening to their feedback -- and then being willing to change. At the end of the day, that is why we embarked on this journey in the first place. Our customers intuitively understand that having multiple tools for logs and metrics was slowing them down. They have transitioned, or are transitioning, to a world of small, collaborative teams, with cross-functional skill sets, who own segments of their applications. The implication is that the majority of our users aren’t log experts, server experts, or network specialists, but software developers, and/or DevOps teams that support modern applications. They know what they need to measure and to analyze because they write the code and support it in the wild. What they are not interested in is learning the deep intricacies of machine data or advanced analytics. So, after listening to our customers, we embarked on a journey to empower them to explore their data without learning complex search languages or getting a Ph.D. in machine learning. We want our users to be able to lose themselves in the analysis of their data without context-switching from one platform to another, and without diverting time away from their tasks. When it is 2 a.m., and the application is down, learning the theory of statistical analysis is definitely not fun. So, wrapping up, I’ll conclude with this: we are committed to providing the best possible experience for our users -- and that has meant questioning many of our own assumptions about how machine data analytics works. While it might have been easy to recognize that peanut butter and jelly belong together, it takes dedicated hard work and perseverance to get it exactly right. We didn’t do it to prove a point. We did it because our customers, in one way or another, asked us to. They have taken immense risks on their own to ride the cutting edge of innovation in their fields, and they deserve a machine data analytics tool that is right there in the thick of it with them. But we are strong, each in our purpose, and we are all more strong together. ― Van Helsing in Bram Stoker’s Dracula

“Toto, I’ve a feeling we’re not in Kansas any more.” Dorothy, The Wizard of Oz We, the software vendors of the world, love to talk about ridding the IT world of silos. So what, may you ask, is a silo anyway? According to the source of all knowledge (Wikipedia, of course), silo comes from the greek word σιρός (siros), which translates to “pit for holding grain.” For most of us, Wikipedia’s definition brings to mind large, metal containers that dot the golden landscapes of states like Idaho or Nebraska. But the term “silos” has also become a great visual analogy for many of the problems with IT and software development, so it’s worth delving beneath the surface to see what we can learn. Why do we need silos? My brother lived in Kansas for a few years and we’ve talked about the ways in which local farmers use grain elevators and silos. It’s amazing to think that farmers from hundreds of miles around bring their grain to these giant grain elevators so the fruit of their labor can be deposited into giant concrete silos. Why does this make sense for farmers? Because their corn is a commodity, i.e.,. all of the corn in that silo is basically the same and sold in bulk just like oil, copper ore, beans, etc. For a long time corporations have done and continue to do something very similar with their data. They create large scale processes to dump related types of data into purpose-built databases. These “data silos” often mirror the organizational structure of the IT organization–network monitoring data goes here, systems monitoring here, etc. Do Data Silos work? Silos serve an important, rather obvious function for the commodities industry. They enable the storage of commodities. No one would put rice in oil or even rice with corn. Silos optimize for costs and efficiency since the value of the commodity is in its volume, or bulk–not an individual kernel of corn. From a corporate perspective, data silos have the opposite effect of siloed corn–they increase costs, decrease efficiency and limit innovation. In the modern corporation, IT culture is rapidly evolving to a shared ownership model removing the organizational inertia to sharing data and ideas. When data is siloed, it actually creates unnecessary friction–hindering collaboration and problem solving across functional boundaries. Conversely, mixing related data can dramatically increase the value of the data by uncovering important patterns and correlating issues across different components. One only needs to look back at the recent past of data warehousing to see a mostly failed attempt to uncover this untapped value. Who benefits from silos anyway? So, who benefits from silos? In the world of commodities, silos empower industrial scale commodities trading and efficiencies. For the agriculture industry, in particular, siloed commodities have led to lower prices for consumers though, one might argue, some reduction in quality. Regardless, silo-based storage has mostly been a net positive. What about data silos? In the world of modern applications, data silos are definitely not a help to application developers and operators. And, they rarely help business decision makers. Most importantly, they don’t help end users or customers. So, who do they help? One clear answer–software vendors. You depend on their products to derive value from the data they house for you. These software products and services either empower you to gain the fullest possible value from your data or hinder and hobble your efforts. In the past, software vendors could sidestep this thorny issue because their interests were aligned with the interests of the traditional organizational structure. No more. The architecture of the modern application (microservices, containers, etc.) demands the free flow of data to connect the dots and form a complete picture. Cross-functional, empowered teams of the Agile/DevOps revolution don’t defer to functional boundaries when solving problems. So, does it make sense to use machine data analytics tools that operate in this same siloed way, holding your team back from delivering value to your customers? Better Together The last 10 years have brought amazing changes to the power of software to deliver business value. The cause of this shift is the symbiotic relationship between powerful cultural changes in how companies organize themselves and get work done along with the tectonic shifts in how business value is delivered over the Internet to customers at massive scale and near real-time speed. Unfortunately, many corporations are still stuck in the world of data silos and losing out on the rich value their data has to offer by treating it like corn. So, it’s time to see how valuable your organization’s data truly is. Don’t let anyone, or any product, stand in your way.

DevOps has gone mainstream. The DevOps philosophy of increasing cross-functional collaboration to quickly deploy high quality software releases that address end-customer demands makes good business sense. Case in point: a 2015 survey by the analyst firm, EMA, found that 80 percent of companies now report the formation of cross-functional teams supporting application development and delivery.More and more of these cross-functional teams (i.e., development, operations, customer success/support, SecOps, and line-of-business) are relying on log and/or machine data analytics solutions to speed their processes. However, it’s important to realize that not all machine data analytics solutions are alike, especially if they are marketed as cloud solutions. At Sumo Logic, we believe cloud-native solutions – that is, SaaS that’s built for the cloud, not merely hosted in the cloud – provide a superior value for DevOps teams looking to leverage machine data analytics to build, run and secure their applications. Below are five key reasons why:Reason 1: New Functionality Without the WaitWith a cloud-native, machine data analytics SaaS, you have the simplicity of being up and running in minutes, as opposed to weeks or months with a traditional, enterprise packaged software. And, cloud-native solutions are designed for velocity; new features can be added at a much faster rate, enabling faster time-to-value and better quality for the same price. New updates are rolled out to all users at the same time, so no one is left behind on an outdated version. So there is no need to wait 12 or 18 months for upgrades to provide access to new features (or spend time and money deploying them). The result is a machine data analytics platform that keeps pace with the shorter development cycles and agile methodologies you employ to drive your own software release velocity.Reason 2: The Elasticity You Need Without Sticker Shock In a rapid DevOps world of continuous technology and underlying infrastructure changes, problems can surface when you least expect it. Just consider the moment when your production applications or infrastructure has a serious problem, and it is “all hands on deck” to investigate and resolve. Suddenly, you have your entire team running simultaneous queries with debug mode activated, bringing in more data than usual.From a DevOps perspective, this scenario is where the elasticity of a multi-tenancy platform outshines a single tenant solution. Since only a small percentage of customers have “incidents” at the exact same time, excess capacity is always available for you and your cross-functional team to use in your hour of crisis. This elasticity also comes in handy to address the expected or unexpected seasonal demands on your application environment. Lastly, with a cloud-based metering payment model, your elasticity doe not come at an extra price. On the contrary, you pay for your “average” capacity, even though there may be times when you utilize more to handle those bursts that require 5, 10 or 100 times more capacity.Reason 3: Scalability Without Performance HitsFor DevOps engineers, machine generated log data is growing exponentially from their IT environments as their infrastructure and application stacks grow in complexity. In fact, by 2020, IT departments will have 50 times more data to index, search, store, analyze, alert and report on. To address this demand, cloud-native solutions have dedicated tiers for each log management function (ingest, index, search, etc.) that scale independently, to keep up with your data’s volume and velocity of change. And, there’s no hassle with managing search nodes or heads in order to ensure search performance. Since not all cloud-based solutions are alike, make sure to understand how much data you can store, query, analyze or report on before being locked out when considering a potential machine data data analytics vendor. Remember, when you have a problem is exactly when you’ll need to scale.Reason 4: Reliability. Reliability. Reliability.The power of aggregated computing resources in a multi-tenant, native cloud architecture provides built-in availability for peak capacity when you need it without incurring payment penalties for bursting. This architectural approach results in always-on service continuity and availability of your data. Additionally, cloud SaaS ensures enterprise-class support for you and your team. Solving customer problems quickly enables SaaS vendors to deliver value across the entire customer base, which is very motivating for the vendor.Reason 5: Never Overbuy or OverprovisionRapid development cycles will also fluctuate your machine data analytics needs over time (e.g. more usage fluctuations, more cross-functional projects that will bring more data, etc.), thereby complicating planning. Native cloud solutions that include metered billing alleviate the need to think ahead on current and future needs—just simply pay as you go for what you need. Additionally, since you will never overbuy or overprovision, your ROI improves over the long term.In summary, DevOps philosophies and agile methodologies are driving the new pace of business innovation. So, it only seems logical that your machine data analytics tools would need to do the same for DevOps, and anything different will just slow you down. Arm yourself with a machine data analytics solution built for the cloud to truly align with your DevOps needs.To learn more, I encourage you to read Ramin Sayar’s recent blog, Built for the Cloud or Hosted in the Cloud – So What’s the Difference? And, you can obtain a checklist of questions to review when considering cloud vendors from a new, IDC white paper, Why Choose Multi-Tenant Cloud-Native Services for Machine Data Analytics.

One frequent question I get is “what is Amazon Kinesis, and what can it do for me?” I also get a lot of questions around Kinesis Streams vs. Firehose? Let’s go through the key concepts and show how to get started with logging using the Kinesis Connector.

One of the current market shifts getting a lot of attention these days is the transition from traditional monolithic application architectures to microservices, the impact of which can be felt from technological innovation to business/team culture. We, at Sumo Logic, believe in the importance of this trend because the same forces driving companies to microservices are also driving them to cloud-based services for supporting capabilities like machine data analytics. Below are three reasons why the benefits of consuming machine data analytics as a service greatly outweighs the old monolithic application delivery model. Focus on the Product, not the Skillsets In recent years, the gourmet food truck has become one of my favorite food trends. The expectation of lines of food trucks all peddling some highly refined comfort food has been a welcome motivation on many family excursions. My wife can get korean-style fried chicken, my daughter a quesadilla, and I can indulge in a completely ridiculous barbeque sandwich with an egg on top. In my mind, microservices are like food trucks while monolithic applications remind me of those cafeterias my grandma dragged us to as kids. Bin after bin of unremarkable, over salted food made by general purpose cooks counting down the hours to clock-out. Sound familiar? Companies today face the same dilemma as the food industry - optimize for predictable mediocrity or allow for creativity, with the higher risk of failure. Customers have made that decision for them - the old way of doing things is not working. What this means is the the modern, digital company has had to move to an organization philosophy focused on business capabilities, not silo-ed, generalized capabilities like database architects and web operations teams. This allows those companies to rapidly develop and experiment with new functionality without toppling the entire edifice, and, most important, make a better quality product in the process. Essentially, create specialization around your product - “korean-style fried chicken”, not platform components - “warmer of canned green beans”. Shared ownership with shared standards Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure. -- Melvyn Conway, 1967 What I have found with food trucks is that their product tends to be highly unique and, at least for the successful ones, highly attuned to the tastes of their users. On the other extreme, the cafeteria is attuned to nobody, and responsive to nobody. In today’s world of opinionated consumers, restaurants have to balance these extremes in order to be successful - the quality of the food truck balanced against the efficient operations of the tightly managed cafeteria. Companies, large and small, are facing this same conundrum - moving from hierarchical decision making that produces middling applications to more dynamic shared-ownership models that can produce unique engaging experiences. For example, traditional, monolithic applications rely on shared components like databases and web services. With microservices, small, cross-functional, multi-skilled teams are expected, and empowered, to maintain stable components with well-known integrations points and APIs, built on common infrastructure components. The quality of the output and the agility of the team is valued above office politics. Consuming your data analytics as a cloud-native service delivers similar benefits to microservices.. The service accepts data from your application in well-understood ways with published methods for processing and analyzing data. And just like with microservices, this allows your team to focus on their application services, rather than having to worry about supporting data services like log or metrics analytics. Focus on Agility Remaining relevant to today’s discerning customers makes speed essential to business. My grandma’s preferred cafeteria menu remained unchanged in the 20 odd years I can remember going there. Food trucks, on the other hand, can change their menu at the drop of a hat in response to the fickle foodies everywhere. The new modern application architecture emphasizes agility and flexibility to respond to similar expectations in the Internet world. Users change loyalties as often as hipsters take food selfies, and businesses can no longer accept nail-biting code deployments every few months. Just like food trucks enable cooks to experiment at the micro-level within a standard kitchen platform, microservices have allowed engineering teams to simplify the act of making changes by pushing complexity down into the smallest box possible. Now, instead of standardizing on lumbering platform components, these companies have standardized on APIs and infrastructure automation. The appeal of data analytics as a service is driven by the same imperatives. Engineers can bake machine data spigots into their microservices components and then depend on the data analytics services’ to provide the appropriate services, as published. I have good memories of eating bland mash potatoes with my grandma, just like I have fond memories of supporting monolithic applications in my early career. But you won’t catch me sneaking into a cafeteria or a chain restaurant for plate full of highly-salted vegetables these days. I’ve moved on. The world has moved on. You can say the same with the modern application. It’s time to focus on making the best possible korean taco or falafel sandwich your business can possibly make, and let us take care of machine data analytics for you. Now, go get some lunch.