Administrator

Amazon Kinesis Data Stream configuration

  • Updated:
    info_outline
    Created:

When setting up your Amazon Kinesis Data Stream account for Buzz's API servers to write into, you need to configure cross-account access.

This article covers how to:

  1. Create Data Stream
  2. Create an IAM Role and cross-account access
  3. Configure Buzz API Console

To configure Buzz to use Amazon Kinesis Data Firehose: Configure Data streams with Amazon Kinesis Data Stream

Amazon Kinesis Data Stream Limitations: Review to make sure Kinesis Data Stream is right for you

A Kinesis Data Stream with a single shard may not be a suitable solution for customers wanting to get large numbers of notifications.

The API and Task servers send data stream events in realtime without any buffering on our side, and Kinesis Data Streams are limited to 1,000 records and 1MB per second (writes).

Accounts are limited to 500 shards by default, so 500 shards would provide a maximum burst capability of 500,000 data stream notifications per second (assuming the notifications average less than 1KB each).

Messages are retained for 24 hours by default but up to 365 days if configured.

Kinesis Data Streams appear to be a kind of middle-ground between Firehose and SQS, with reduced throughput but longer message retention than SQS, and increased throughput but shorter retention than Firehose.

Event notifications may retry a few times (based on AWS SDK retry policies), but will not retry outside of that, so if limits are exceeded, event notifications may be lost.

Create Data Stream

In the AWS Console:

  1. Navigate to Amazon Kinesis > Data streams.
  2. Click Create data stream.
  1. Enter a Data stream name.
  2. Select your Capacity mode, On-demand or Provisioned according to your needs.
  3. Complete the rest of your setup as you normally do for internal use, following AWS instructions, and click Create data stream.

This screen summarizes the data stream you created.

Create an IAM Role and cross-account access

Next, create an IAM Role that grants access to Agilix's AWS account, so Buzz can write to your Kinesis Data Stream.

  1. Navigate to IAM > Roles.
  2. Click Create role.
  1. Under Select trusted entity, select AWS account as the Trusted entity type.
  2. Select Another AWS account and enter the Agilix AWS account number (you can get this number through your support account or sales representative).
  3. Click Next.
  1. Now, you can select or create the permissions you want associated with this role. Assuming you don't already have a role set up for this, click Create policy.
    • Note: If you already have a role set up, skip to Step 11.
  1. In the JSON editor, use the following configuration to enter your data, including your own AWS account number (because you're granting permissions to your role and delivery stream):
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "VisualEditor0",
			"Effect": "Allow",
			"Action": [
				"kinesis:DescribeStream", 
				"kinesis:PutRecord",
				"kinesis:PutRecords"
			],
			"Resource": "arn:aws:kinesis:us-east-1:____________:stream/___________"
		}
	]
}
Click to copy
  1. Click Next: Tags, and enter any tags you want on that screen, then click Next: Review
  1. Enter the policy Name and Description.
  2. Click Create policy
  1. Return to the Create role screen from Step 6 (this may still be open in another tab depending on your browser).
  2. Click the refresh button next to the Create policy button
  3. Find the policy you just created (you can search for the Name you provided in Step 9) and check its box, and click Next.
  1. Provide a Role name and Description.
  2. Review the trusted Entities, Permissions, and Tags if needed.
  3. Click Create role.
  1. This takes you to the role summary page; click Edit.
  1. Change the Maximum session duration to 12 hours.
  2. Save changes.

Configure Buzz API Console

Lastly, log into the Buzz API Console:

  1. Select the POST Method.
  2. In POST Data, enter JSON for the SetDataStreamConfiguration command as shown, using:
    • Your domain ID.
    • Your selected title.
    • The name of your Kinesis Data Stream.
    • The ARN for the cross-account role you created above.
      • Note: No ARN is needed for the data stream name because it will be looked up in the context of the specified role.
  3. Send request and review Result.
    • If the response has an error, check for mistakes made in the setup based on the error message.
    • If the response has code:OK, you can remove the line with test:true and change enabled:false to enabled:true to push the configuration and begin receiving data stream events to the data stream you've configured.

Note: Due to the nature of Kinesis Data Stream, there is a delay between Buzz posting the events and your being able to see them. This delay depends on the configured Buffer size and Buffer interval. The events become visible as soon as either one of these thresholds has been met. If you have a domain with a lot of traffic and you're set up to receive a lot of events, the events will be visible more quickly. If not, they will be visible after the specified buffer interval.

forum

Have a question or feedback? Let us know over in Discussions!

Articles in this section

See all 126 articles

Related articles