Kinesis

Overview

• Kinesis makes it easy to collect, process, and analyse real-time, streaming data so you can get timely insights and react quickly to new information.

• Kinesis offers key capabilities to cost-effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application.

• With Kinesis, you can ingest real-time data such as video, audio, application logs, website clickstreams, and IoT telemetry data for machine learning, analytics, and other applications.

• Kinesis enables you to process and analyse data as it arrives and respond instantly instead of having to wait until all your data is collected before the processing can begin.

Kinesis Data Streams - Collect streaming data, at scale, for real-time analytics

• Shard:

  • A uniquely identified sequence of data records in a stream.

• Kinesis Data Stream:

  • A set of shards.

  • Same partition key will go to the same shard.

• Kinesis Data Record:

  • The unit of data stored in a data stream.

  • Data records are composed of a sequence number, a partition key, and a data blob, which is an immutable sequence of bytes.

• Retention Period:

  • The length of time that data records are accessible after they are added to the stream.

  • A stream’s retention period is set to a default of 24 hours after creation.

• Producer:

  • Producers put records into Kinesis Data Streams.

    • For example, a web server sending log data to a stream is a producer.

• Consumer:

  • They get records from Kinesis Data Streams and process them.

Kinesis Data Firehose - Prepare and load near real-time data streams into data stores and analytics services

• Kinesis Data Firehose is a fully managed service for delivering near real-time streaming data (lowest buffer time is 1 minute) and send the data to destinations such as S3, Redshift, Amazon Elasticsearch, Splunk, and any custom HTTP endpoint.

• With Kinesis Data Firehose, you don't need to write applications or manage resources.

• You configure your data producers to send data to Kinesis Data Firehose, and it automatically delivers the data to the destination that you specified.

• You can also configure Kinesis Data Firehose to transform your data before delivering it.

Kinesis Data Analytics - Analyse streaming data in real-time with SQL

• With Kinesis Data Analytics for SQL Applications, you can process and analyse streaming data using standard SQL.

• The service enables you to quickly author and run powerful SQL code against streaming sources to perform time series analytics, feed real-time dashboards, and create real-time metrics.