Forward-looking companies are embracing reliability management. Learn how they're measuring business outcomes rather than system performance.

Since various versions of lockdown around the world kicked in, we’ve all been forced to learn how to live as if we’ve just awoke from a deep stasis into a new reality in which everything on earth works differently. It was nearly that abrupt. And if there was ever evidence of how business adaptability is related to business success, it is now.

Today, we are facing an unprecedented situation. The COVID-19 pandemic is affecting everything we know -- our families, our businesses, our communities, and our way of life.

As businesses transform their traditional business models into new digital ones, and aggressively compete for turf within the digital economy, their constant pursuit of competitive edge drives technology, process, and architectural innovations. As a result, it seems that every 18 months a technology paradigm shift comes about that enables better agility, lower cost, improved quality of service, better intelligence and more.

A few blogs posts ago I wrote about new BI for digital companies and in that blog I alluded that quite a bit of that BI is based on log data. I wanted to follow up on the topic of logs, why they exist and why they contain so much data that is relevant to BI.

First time this year, multi-cloud enterprises, as a customer segment of Sumo Logic, have grown faster than any other segment: 50% Y/Y. What took so long? In my conversations with enterprises over the last 5 years, there was only one strategy for public cloud and it was multi-cloud. But evidence of multi-cloud usage was sparse at best. Data from our Continuous Intelligence Report in previous years didn’t find much to support that the strategy for multi-cloud was being implemented.

Today we speak about observability explicitly in terms of DevOps/SRE. So, what is the objective of observability for DevOps engineers or SREs? The answer to that one hides behind the acronym itself … Site Reliability.

Software has eaten the world and every company today is a software company. This is because every company today is more and more serving its customers digitally.

These days we spend a lot of time talking about modernizing our stack, modernizing our architectures, using new application components, modern application life cycles, etc. So, what is this all about and why do we spend so much time talking about it?

Today we published Sumo Logic’s first "The State of The Modern App in AWS" report. We wanted to explore how modern applications differ from traditional applications: how are the modern app workloads run, what application components are used, and which types of application services are leveraged? This report is based on anonymized data from about 1,000 customers running modern application workloads in AWS and it examines the composition of those applications. Sumo Logic provides our customers with the operational and security visibility of their applications. As such, multiple millions events per second of live data are sent to our service from our customers’ full application stack: infrastructure, application components, custom app code, as well as variety of application services leveraged to manage the application. Fingerprint of this data then gives us unprecedented ability to answer questions about these applications. When we founded Sumo Logic, we decided to build a multi-tenant analytics service in order to be able to leverage the cross customer visibility in order to provide more value to each customer. Our architecture and technology behind it allows us to: Analyze anonymized statistics about technology usage to help our existing and future customers learn about latest technology trends emerging from enterprises leading the way to digital transformation (e.g. this report) Prioritize roadmap items in order to help our customers gain visibility into most commonly used technologies (e.g. our integration with Docker) Improve our machine learning algorithms by exposing them to larger data sets of same data types across multiple customers Derive meanings from events occurring across same sources across multiple customers to improve operational and security outcomes I am excited to have this report as it is a very tangible response to a question I very frequently get: what is the value of having a multi-tenant architecture? It is easy to quickly respond that we benefit from the same operating model as other all-in cloud services, such as AirBnB, Netflix, Hudl, etc. But this report also highlights another, and perhaps more important and unique value of multi-tenancy - it gives us the ability to analyze trillions of anonymized events across our customer community and convert them into value for each individual customer as they build, run, and secure their modern application. With Sumo Logic, you are not alone. We are excited to continue deriving insights that will help our customers create better outcomes for their business.

https://www.sumologic.com/blog... dir="ltr">The new Sumo Logic Transaction capability allows users to analyze related sequences of machine data. The comprehensive views uncover user behavior, operational and security insights that can help organizations optimize business strategy, plans and processes. The new capability allows you to monitor transactions by a specific transaction ID (session ID, IP, user name, email, etc.) while handling data from distributed systems, where a request is passed through several different systems, each with its own transaction ID. Over the past two months, we have worked with beta customers on a variety of use cases, including: Tracking transactions in a payment processing platform Following typical user sessions, detecting anomalous checkout transactions and catching checkout drop off in e-commerce websites Tracking renewals, upgrades and new signup transactions Monitoring phone registrations failures over a specific period Tracking on-boarding of new users in SaaS products The last use case is reflective of what SaaS companies care most about: truly understanding the behavior of users on their website that drive long-term engagement. We’ve used our new transaction analytics capabilities to better understand how users find our site, the process by which they get to our Sumo Logic Free page, and how quickly they sign up. Our customer success team uses Transaction Analytics to monitor how long it takes users to create a dashboard, run a search, and perform other common actions. This enables them to provide very specific feedback to the product team for future improvements. This screenshot depicts a query with IP as the transaction ID and the various states mapped from the logs Sankey diagram visualizes the flow of the various components/states of a transaction on an e-commerce website Many of our customers are already using tools such as Google Analytics to monitor visitors flow on their website and understand customer behavior. We are not launching this new capability to replace Google Analytics (even if it’s not embraced in some countries as Germany). What we bring on top of monitoring visitors flow, is the ability to identify divergence in state sequences and understand better the transitions between the states, in terms of latency for example. You probably see updates that some companies are announcing on plugins for log management platforms to detect anomalies and monitor user behavior and sessions. The team’s product philosophy is that we would like to provide our users all-rounded capability that enables them to make smart choices without requiring external tools, all from their machine data within the Sumo product. It was a fascinating journey working on the transaction capability with our analytics team. It’s a natural evolution of our analytics strategy which now includes: 1) real-time aggregation and correlation with our Dashboards; 2) machine learning to automatically uncover anomalies and patterns; and 3) now transaction analytics to rapidly uncover relationships across distributed events. We are all excited to launch Transaction Analytics. Please share with us your feedback on the new capability and let us know if we can help with your use cases. The transaction searches and the new visualization are definitely our favorite content. https://www.sumologic.com/blog... class="at-below-post-recommended addthis_tool">

Sumo Logic app for AWS Elastic Load Balancing takes raw log data & presents it in visual dashboards for real-time insights into operations & status codes.

Security is a tricky thing and it means different things to different people. It is truly in the eye of the beholder. There is the checkbox kind, there is the “real” kind, there is the checkbox kind that holds up, and there is the “real” kind that is circumvented, and so on. Don’t kid yourself: the “absolute” kind does not exist. I want to talk about security solutions based on log data. This is the kind of security that kicks in after the perimeter security (firewalls), intrusion detection (IDS/IPS), vulnerability scanners, and dozens of other security technologies have done their thing. It ties all of these technologies together, correlates their events, reduces false positives and enables forensic investigation. Sometimes this technology is called Log Management and/or Security Information and Event Management (SIEM). I used to build these technologies years ago, but it seems like decades ago. A typical SIEM product is a hunking appliance, sharp edges, screaming colors - the kind of design that instills confidence and says “Don’t come close, I WILL SHRED YOU! GRRRRRRRRRR”. Ahhhh, SIEM, makes you feel safe doesn’t it. It should not. I proclaim this at the risk at being yet another one of those guys who wants to rag on SIEM, but I built one, and beat many, so I feel I’ve got some ragging rights. So, what’s wrong with SIEM? Where does it fall apart? SIEM does not scale It is hard enough to capture a terabyte of daily logs (40,000 Events Per Second, 3 Billion Events per Day) and store them. It is couple of orders of magnitude harder to run correlation in real time and alert when something bad happens. SIEM tools are extraordinarily difficult to run at scales above 100GB of data per day. This is because they are designed to scale by adding more CPU, memory, and fast spindles to the same box. The exponential growth of data over the two decades when those SIEM tools were designed has outpaced the ability to add CPU, memory, and fast spindles into the box. Result: Data growth outpaces capacity → Data dropped from collection → Significant data dropped from correlation → Gap in analysis → Serious gap in security SIEM normalization can’t keep pace SIEM tools depend on normalization (shoehorning) of all data into one common schema so that you can write queries across all events. That worked fifteen years ago when sources were few. These days sources and infrastructure types are expanding like never before. One enterprise might have multiple vendors and versions of network gear, many versions of operating systems, open source technologies, workloads running in infrastructure as a service (IaaS), and many custom written applications. Writing normalizers to keep pace with changing log formats is not possible. Result: Too many data types and versions → Falling behind on adding new sources → Reduced source support → Gaps in analysis → Serious gaps in security SIEM is rule-only based This is a tough one. Rules are useful, even required, but not sufficient. Rules only catch the thing you express in them, the things you know to look for. To be secure, you must be ahead of new threats. A million monkeys writing rules in real-time: not possible. Result: Your rules are stale → You hire a million monkeys → Monkeys eat all your bananas → You analyze only a subset of relevant events → Serious gap in security SIEM is too complex It is way too hard to run these things. I’ve had too many meetings and discussions with my former customers on how to keep the damned things running and too few meetings on how to get value out of the fancy features we provided. In reality most customers get to use the 20% of features because the rest of the stuff is not reachable. It is like putting your best tools on the shelf just out of reach. You can see them, you could do oh so much with them, but you can’t really use them because they are out of reach. Result: You spend a lot of money → Your team spends a lot of time running SIEM → They don’t succeed on leveraging the cool capabilities → Value is low → Gaps in analysis → Serious gaps in security So, what is an honest, forward-looking security professional who does not want to duct tape a solution to do? What you need is what we just started: Sumo Logic Enterprise Security Analytics. No, it is not absolute security, it is not checkbox security, but it is a more real security because it: Scales Processes terabytes of your data per day in real time. Evaluates rules regardless of data volume and does not restrict what you collect or analyze. Furthermore, no SIEM style normalization, just add data, a pinch of savvy, a tablespoon of massively parallel compute, and voila. Result: you add all relevant data → you analyze it all → you get better security Simple It is SaaS, there are no appliances, there are no servers, there is no storage, there is just a browser connected to an elastic cloud. Result: you don’t have to spend time on running it → you spend time on using it → you get more value → better analysis → better security Machine Learning Rules, check. What about that other unknown stuff? Answer: machine that learns from data. It detects patterns without human input. It then figures out baselines and normal behavior across sources. In real-time it compares new data to the baseline and notifies you when things are sideways. Even if “things” are things you’ve NEVER even thought about and NOBODY in the universe has EVER written a single rule to detect. Sumo Logic detects those too. Result: Skynet … nah, benevolent overlord, nah, not yet anyway. New stuff happens → machines go to work → machines notify you → you provide feedback → machines learn and get smarter → bad things are detected → better security Read more: Sumo Logic Enterprise Security Analytics

Cloud data comes to life with our Sumo Logic Application for AWS CloudTrail, helping our customers across security and compliance, operational visibility, and cost containment.

I’m very pleased to announce our strategic alliance with Akamai. Our integrated solution delivers a unified view of application availability, performance, security, and business analytics based on application log data. Customers who rely on Akamai’s globally distributed infrastructure now can get the real-time feed of all logs generated by Akamai’s infrastructure into their Sumo Logic account in order to integrate and cross-analyze them with their internally generated application data sets! What problems does the integrated solution solve? To date, there have been two machine data sets generated by applications that leverage Akamai: 1. Application logs at the origin data centers, which application owners can usually access. 2. Logs generated by Akamai as an application is distributed globally. Application owners typically have zero or limited access to these logs. Both of these data sets provide important metrics and insights for delivering highly-available, secure applications that also provide detailed view of business results. Until today there was no way to get these data sets into a single tool for real-time analysis, causing the following issues: No single view of performance. While origin performance could be monitored, but that provides little confidence that the app is performant for end users. Difficult to understand user interaction. Without data on how real users interact with an application, it was difficult to gauge how users interacted with the app, what content was served, and ultimately how the app performed for those users (and if performance had any impact on conversions). Issues impacting customer experience remained hidden. The root cause of end-user issues caused at the origin remained hidden, impacting customer experience for long periods of time. Web App Firewall (WAF) security information not readily available. Security teams were not able to detect and respond to attacks in real-time and take defensive actions to minimize exposure. The solution! Akamai Cloud Monitor and Sumo Logic provide an integrated approach to solving these problems. Sumo Logic has developed an application specifically crafted for customers to extract insights from their Akamai data, which is sent to Sumo Logic in real time. The solution has been deployed by joint customers (at terabyte scale) to address the following use cases: Real-time analytics about user behavior. Combine Akamai real-user monitoring data and internal data sets to gain granular insights into user behavior. For example, learn how users behave across different device types, geographies, or even how Akamai quality of service impacts user behavior and business results. Security information management and forensics. Security incidents and attacks on an application can be investigated by deep-diving into sessions, IP addresses, and individual URLs that attackers are attempting to exploit and breach. Application performance management from edge to origin. Quickly determine if an application’s performance issue is caused by your origin or by Akamai’s infrastructure, and which regions, user agents, or devices are impacted. Application release and quality management. Receive an alert as soon as Akamai detects that one or more origins have an elevated number of 4xx or 5xx errors that may be caused by new code push, configuration change, or another issue within your origin application infrastructure. Impact of quality of service and operational excellence. Correlate how quality of service impacts conversions or other business metrics to optimize performance and drive better results I could go on, but I’m sure you have plenty of ideas of your own. Join us for a free trial here – as always, there is nothing to install, nothing to manage, nothing to run – we do it all for you. You can also read our announcement here or read more about the Sumo Logic application for Akamai here. Take a look at the Akamai press release here.

I’m glad you ask, I just might. In fact, we started collecting data about machine data some 9 months ago when we participated at the AWS Big Data conference in Boston. Since then we continued collecting the same data at a variety of industry show and conferences such as VMworld, AWS re: Invent, Velocity, Gluecon, Cloud Slam, Defrag, DataWeek, and others. The original survey was printed on my home printer, 4 surveys per page, then inexpertly cut with the kitchen scissors the night before the conference – startup style, oh yeah! The new versions made it onto a shiny new iPad as an IOS App. The improved method, Apple caché, and a wider reach gave us more than 300 data points and, incidentally, cost us more than 300 Sumo Logic T-Shirts which we were more than happy to give up in exchange for data. (btw, if you want one come to one of our events, next one coming up will be the Strata Conference). As a data junkie, I’ve been slicing and dicing the responses and thought that end of our fiscal year could be the right moment to revisit it and reflect on my first blog post on this data set. Here is what we asked: Which business problems do you solve by using machine data? Which tools do you use to analyze machine data in order to solve those business problems? What issues do you experience solving those problems with the chosen tools? The survey was partially designed to help us to better understand the Sumo Logic’s segment of IT Operations Management or IT Management markets as defined by Gartner, Forrester, and other analysts. I think that the sample set is relatively representative. Responders come from shows with varied audiences such as developers at Velocity and GlueCon, data center operators at VMworld, and folks investigating a move to the cloud at AWS re: Invent and Cloud Slam. Answers were actually pretty consistent across the different “cohorts”. We have a statistically significant number of responses, and finally, they were not our customers or direct prospects. So let’s dive in and see what we’ve got and let’s start at the top: Which business problems do you solve by using logs and other machine data? Applications management, monitoring, and troubleshooting (46%) IT operations management, monitoring, and troubleshooting (33%) Security management, monitoring, and alerting (21%) Does anything in there surprise? I guess it depends on what your point of reference is. Let me compare it to the overall “IT Management” or “IT Operations Management” market. The consensus(if such a thing exists) is that size by segment is: IT Infrastructure (servers, networks, etc) is up to 50-60% of the total market Application (internal, external, etc.) is just north of 30-40% Security is around 10% Source: Sumo Logic analysis of aggregated data from various industry analysts who cover IT Management space. There are a few things that could explain the big difference between how much our subsegment leans more toward Applications vs. IT infrastructure. (hypothesis #1) analysts measure total product sold to derive the market size which might not be the same as effort people apply to these use cases. (hypothesis #2) there is more shelfware in IT Infrastructure which overrepresented effort. (hypothesis #3) there are more home-grown solutions in Application management which underrepresents effort. (hypothesis #4) our data is an indicator or a result of a shift in the market (e.g., when enterprises shift toward the IaaS, they spend less time managing IT Infrastructure and shift more toward the core competency, their applications). (obnoxious hypothesis #5) intuitively, it’s the software stupid – nobody buys hardware because they love it, it exists to run software (applications), and we care more about applications, and that’s why it is so. OK, ok, let’s check the data to see which hypothesis can our narrow response set help test/validate. I don’t think our data can help us validate hypothesis #1 or hypothesis #2. I’ll try to come up with additional survey questions that will, in the future, help test these two hypotheses. Hypothesis #3 on the other hand might be partially testable. If we compare responses from users who use commercial vs. who use home-grown, we are left with the following: Not a significant difference between responders who use commercial vs. responders who use home grown tools. Hypothesis #3 explains only a couple of percentage points of difference. Hypothesis #4 – I think we can use a proxy to test it. Let’s assume that responders from VMworld are focused on internal data center and the private cloud. In this case they would not be relying as much on IaaS providers for IT Infrastructure Operations. On the other hand, let’s also assume that AWS, and other cloud conference attendees are more likely to rely on IaaS for IT Infrastructure Operations. Data please: Interesting, seems to explain some shift between security and infrastructure, but not applications. So, we’re left with: hypothesis #1 – spend vs. reported effort is skewed – perhaps hypothesis #2 – there is more shelfware in IT infrastructure – unlikely obnoxious hypothesis #5 – it’s the software stupid – getting warmer That should do it for one blog post. I’ve barely scratched the surface by stopping with the responses to the first question. I will work to see if I can test the outstanding hypotheses and, if successful, will write about the findings. I will also follow-up with another post looking at the rest of the data. I welcome your comments and thoughts. While you’re at it, try Sumo Logic for free.

I recently represented Sumo Logic at the AWS Big Data conference in Boston. It was a great show, very well-attended. Sumo Logic was one of the few vendors invited to participate. During the conference I conducted a survey of the attendees to try to understand how this, emerging early-adopter segment of IT professionals, manages log data for their infrastructure and applications. Common characteristics of attendees surveyed: They run their apps and infrastructure in the cloud They deal with large data sets They came to learn how to better exploit/leverage big data and cloud technologies What I asked: Do you use logs to help you in your daily work, and if so, how? What types of tools do you use for log analysis and management? What are the specific pain points associated with your log management solutions? The findings were interesting. Taking each one in turn: No major surprises here. Enterprises buy IaaS in order to run applications, either for burst capacity or because they believe it’s the wave of the future. The fact that someone else manages the infrastructure does not change the fact that you have to manage and monitor your applications, operating systems, and virtual machines. A bit of a surprise here. In my previous analysis, some 45% of enterprises use homegrown solutions, but in this segment it’s 70%. Big difference with the big data and cloud crowd. A possible explanation for this is that existing commercial solutions are not easy to deploy and run in the cloud and don’t scale to handle big data. So, the solution = build it yourself. Hmm. Yes, yes, I know, it adds up to more than 100%. That’s because the question was stated as “select as many as apply” and many respondents have more than one problem. So, nothing terribly interesting in there. But let me dig a bit deeper into issues associated with homegrown vs. commercial. This makes a bit more sense. For the home grown, it looks like complexity is the biggest pain – which makes sense. Assembling together huge systems to support big volumes of log data is more difficult than many people anticipate. Hadoop and other similar solutions are not optimized to simply and easily deliver answers. This then leads to the next pain point: if it is not easy to use, then you don’t use it = does not deliver enough value. The responses on commercial solutions make sense as well. Today’s commercial products are expensive and hard to operate. On top of the sticker price, you have to spend precious employee time to perform frequent software upgrades and implement “duct tape” scaling. If you don’t have expertise internally you buy it from vendors’ professional services at beaucoup $$$$$. You have to get your own compute and storage, which grow as your data volume grows. So, commercial “run yourself” solutions = very high CAPEX (upfront capital expenditures) and OPEX (ongoing operational expenditures). In the end (as the second pain point highlights), commercial solutions are also complex to operate and hard to use, requiring highly skilled and hard to find personnel. Pretty bleak – what now? At Sumo Logic, we think we have a solution. The pain points associated with home-grown and commercial solutions that were architected in the last decade are exactly what we set out to solve. We started this company after building, selling and supporting the previous generation of log management and analysis solutions. We’ve incorporated our collective experience and customer feedback into Sumo Logic. Built for the cloud The Sumo Solution is fundamentally different from anything else out there. It is built for big data and is “cloud native”. All of the complexities associated with deploying, managing, upgrading, and scaling are gone – we do all that for you. Our customers get a simple-to-use web application, and we do all the rest. Elastic scalability Our architecture is true cloud, not a “cloud-washed” adaptation of on-premise single-instance software solutions that are trying to pass themselves off as cloud. Each of our services are separate and can be scaled independently. It takes us minutes to triple the capacity of our system. Insights beyond your wildest dreams Because of our architecture, we are able to build analytics at scale. Our LogReduce™ and Push Analytics™ uncover things that you didn’t even know you should be paying attention to. The whole value proposition is turned on its head – instead of having to do all the work yourself, our algorithms do the work for you while you guide them to get better over time. Come try it out and see for yourself: http://www.sumologic.com/free-trial/