Simplify infrastructure and reduce costs with VPC Flow Logs ingest via Amazon Kinesis Data Firehose into Sumo Logic

Sumo Logic is proud to announce that, in collaboration with AWS, we now fully support Virtual Private Cloud (VPC) Flow Logs ingestion via Amazon Kinesis Data Firehose. Customers can now simplify log delivery to Sumo Logic which is natively integrated with Kinesis Data Firehose. You can also simplify your toolchains for aggregating, transforming and enriching VPC Flow Logs using Kinesis Data Firehose.

What is Amazon Kinesis Data Firehose, and when do you use it?

Amazon Kinesis Data Firehose is an extract, transform, and load (ETL) service that's available as a managed streaming solution on AWS. Kinesis Data Firehose can receive, buffer and process various forms of data in real time, including video feeds, IoT data and logging events from thousands of sources. You can use Kinesis to transmit data and events to machine learning systems, data analytics, business intelligence and many other destinations.

Because Kinesis is managed and built on top of AWS infrastructure, you don’t have to worry about provisioning hardware or handling fluctuations in the volume or frequency of data.

When would you use VPC Flow Logs with Amazon Kinesis Data Firehose?

Streamlined, near real-time log delivery to analytics platforms: You can now easily and reliably stream your log data to Sumo Logic with minimal infrastructure setup and management.

Simplified log processing and delivery pipelines: Enriching, transforming or aggregating your flow logs and delivering it to multiple destinations can simplify your log enrichment and log delivery pipelines.

How will this benefit you?

Lower operational overhead: VPC Flow Logs to Kinesis Data Firehose will simplify your operations for ingesting and processing VPC Flow Logs as follows: 

1. Eliminate dependency on custom integrations: Today, many Sumo Logic users must use custom tooling to ingest logs from Amazon CloudWatch logs or Amazon S3 via Kinesis Data Firehose. With VPC Flow Logs delivered natively to Kinesis Data Firehose, you no longer need to maintain such custom integrations and can simplify your delivery of VPC Flow Logs to analytics platforms like Sumo Logic.

2. Streamline log processing: You may need to enrich VPC Flow Logs with your own metadata context. Today, you probably maintain your own pipelines to do this. With Kinesis Data Firehose’s extensible data transformation capabilities and built-in lambda functions, you can also streamline your log processing and log delivery pipelines into a single Kinesis Data Firehose delivery stream. 

Lower total cost of ownership: Currently, to send logs to partner solutions, you must stream your logs from CloudWatch to Kinesis Data Firehose and then to Sumo Logic. Adding Kinesis Data Firehose as a direct destination will eliminate an extra integration hop and may help cut down customer costs. 

Getting started

To get started with ingesting data via AWS Kinesis Data Firehose to Sumo Logic, you’ll need to create a log source for AWS Kinesis Data Firehose. Read this post for more information on how to do that.

Then you will need to create a Kinesis Data Firehose data stream. You can do this through console or CLI. When creating, you will need to select Sumo Logic as the destination and provide the HTTP endpoint url for your Sumo Logic log source, which you created previously. Give your Kinesis Data Firehose stream a name, and select whatever parameters fit your need.

Create your delivery stream and you are good to go—all that’s left is to make the VPC Flow Log subscription.


Example VPC Flow Log subscription creation:

aws ec2 create-flow-logs \
--resource-type VPC \
--resource-ids <vpc_id> \
--log-destination-type kinesis-data-firehose \
--traffic-type ALL \
--log-destination <Arn_For_created_delivery_stream> \
--max-aggregation-interval 60

Once the subscription is created, your remaining VPC Flow Logs should start getting ingested into Sumo Logic.

New to Sumo Logic and want unified visibility of your AWS services? Read more about how Sumo Logic and AWS work well together.