Administrator

Amazon Kinesis Data Firehose configuration

  • Updated:
    info_outline
    Created:

When setting up your Amazon Kinesis Data Firehose account for Buzz's API servers to write into, you need to configure cross-account access.

This article covers how to:

  1. Create delivery stream
  2. Create an IAM Role and cross-account access
  3. Configure Buzz API Console

To configure Buzz to use Amazon Kinesis Data Firehose: Configure Data streams with Amazon Kinesis Data Firehose

Amazon Kinesis Data Firehose Limitations: Review to make sure Firehose is right for you

A single Firehose may not be a suitable solution for customers wanting to get large numbers of notifications.

The API and Task servers send datastream events in realtime without any buffering on our side, and Firehose has a limit of 2,000 requests per second, which is not adjustable without human interaction.

If you plan to ever hit a burst of more than 2,000 data stream notifications per second, Firehose is probably not a good option for you.

Firehose is, however, able to easily send data to various persistent storage systems, so if your needs fall within it's throughput limits, and you need to store the event notifications long-term, Firehose is a good choice.

Event notifications may retry a few times (based on AWS SDK retry policies), but will not retry outside ot that, so if limits are exceeded, event notifications may be lost.

Sending different types of Data Stream event notifications to different firehoses may help this problem to a certain extent.

1. Create delivery stream

In the AWS Console:

  1. Navigate to Amazon Kinesis > Delivery streams > Create delivery stream.
  2. Select Direct PUT for the Source and choose the Destination of your choice
    • In this example, we show Amazon S3, so some of the subsequent configuration will be specific to an S3 connection.
  3. Complete the rest of your setup as you normally do for internal use, following AWS instructions, and click Create delivery stream.

After you create the delivery stream, AWS automatically creates and displays a Role for the Firehose service to use to access the selected Destination. Click the role to review the access the role was given.

This screen describes the role's access.

2. Create an IAM Role and cross-account access

Next, create an IAM Role that grants access to Agilix's AWS account, so Buzz can write to your Firehose.

  1. Navigate to IAM > Roles > Create role.
  2. Under Select trusted entity, select AWS account as the Trusted entity type.
  3. Select Another AWS account and enter the Agilix AWS account number (you can get this number through your support account or sales representative).
  4. Click Next.
  1. Now, you can select or create the permissions you want associated with this role. Assuming you don't already have a role set up for this, click Create policy.
    • Note: If you already have a role set up, skip to Step 10.
  1. In the JSON editor, use the following configuration to enter your data, including your own AWS account number (because you're granting permissions to your role and delivery stream):
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "VisualEditor0",
			"Effect": "Allow",
			"Action": [
				"firehose:DescribeDeliveryStream", 
				"iam:PassRole",
				"firehose:PutRecord",
				"firehose:PutRecordBatch"
			],
			"Resource": [
				"arn:aws:firehose:us-east-1:____________:deliverystream/buzz-api-stream",
				"arn:aws:iam::____________:role/buzz-stream-agilix-access"
			]
		}
	]
}
Click to copy
  1. Click Next: Tags, and enter any tags you want on that screen, then click Next: Review.
  1. Enter the policy Name and Description.
  2. Click Create policy.
  1. Return to the Create role screen from Step 5 (this may still be open in another tab depending on your browser).
  2. Click the refresh button next to the Create policy button
  3. Find the policy you just created (you can search for the Name you provided in Step 8) and check its box.
  4. Click Next.
  1. Provide a Role name and Description.
  2. Review the trusted Entities, Permissions, and Tags if needed.
  3. Click Create role.
  1. This takes you to the role summary page; click Edit.
  1. Change the Maximum session duration to 12 hours.
  2. Save changes.

3. Configure Buzz API Console

Lastly, log into the Buzz API Console:

  1. Select the POST Method.
  2. In POST Data, enter JSON for the SetDataStreamConfiguration command as shown, using:
    • Your domain ID.
    • Your selected title.
    • The name of your firehose delivery stream.
    • The ARN for the cross-account role you created above.
      • Note: No ARN is needed for the delivery stream name because it will be looked up in the context of the specified role.
  3. Send request and review Result.
    • If the response has an error, check for mistakes made in the setup based on the error message.
    • If the response has code:OK, you can remove the line with test:true and change enabled:false to enabled:true to push the configuration and begin receiving data stream events to the firehose you've configured.

Note: Due to the nature of Firehose, there is a delay between Buzz posting the events and your being able to see them. This delay depends on the configured Buffer size and Buffer interval. The events become visible as soon as either one of these thresholds has been met. If you have a domain with a lot of traffic and you're set up to receive a lot of events, the events will be visible more quickly. If not, they will be visible after the specified buffer interval.

forum

Have a question or feedback? Let us know over in Discussions!

Articles in this section

See all 126 articles

Related articles