Spring Boot Elastic Beanstalk with Amazon MSK

Posted by:
Narayana Rallabandi

Publish Date:
4 Feb, 2022

Apache Kafka is a distributed, scalable, fault-tolerant messaging system. It is a publish-subscribe based system that enables the parallel processing of messages. Its durability, reliability, and replicability properties mean Kafka can be broadly visualized as a platform that facilitates data storage and real-time stream processing of data. Apache Kafka-based applications can be built using Amazon MSK. Amazon MSK offers console-based setup and configuration along with continuous monitoring of the cluster to prevent server downtime in the application. This article provides an overview of how to create a Spring Boot application using Amazon MSK.

kafka

Spring for Apache Kafka

Spring is provided with the Spring template model, i.e., Spring for Apache Kafka. The Kafka support to the application can be included by adding the following Maven Dependency in the pom.xml file.

spring-kafka

Amazon MSK Setup

At the outset, we are required to create a cluster in the Amazon MSK console. During setup, the connection with AWS MSK, the zookeeper and the broker servers is essential. The details of the broker servers are required while configuring the producer and consumer properties for sending and receiving messages, while zookeeper details are required to create the topic.

Broker-details

Topic Creation:

The next step is to create a topic in Kafka. Let us create a topic “demoTopic” by executing the following command in AWS SSH.

Topic-Creation

Publishing Messages:

With this, let us configure the producer properties followed by the Kafka Template for sending messages to the Kafka Topic. To do this, we need to create a configuration class and add the following producer instances:

Publishing-Messages

To send a message to the Kafka Topic, the KafkaTemplate is used. This requires the following snippet:

kafka-template

Consuming Messages:

To consume the messages, we require the following consumer configurations. These will enable the detection of the consumers based on the @KafkaListener annotation. The group id (below) specifies the consumer group to which the consumer belongs.

Consuming-Messages

Message consumption can be from multiple topics, and multiple consumers can consume a message from the same topic as follows:

Kafka-Listener

Performance Optimization:

To further tune and optimize the performance of Kafka, there are certain factors in the producer, consumer and broker side that need to be considered. These are configured to strike a perfect balance between the high throughput and low latency features of Kafka. Improvement in the producer property would achieve increased durability, reliability, throughput, and lower latency, which can be achieved as follows:

To ensure that there is no loss of messages, producer acknowledgement is set to all. A batching and buffering approach is used to keep a check on throughput and latency. Batch size — which is set in bytes — is configured along with the linger time that specifies the maximum time the batch is provided to fill. The message is sent to whichever topic occurs earlier. Along with this, the buffer memory, equal to the batch size, is configured to store the unsent messages.

AWS Deployment:

With this, the configurations for publishing and consuming messages with Kafka is completed. To observe the working of the application, Kafka must be deployed in Elastic Beanstalk. For the deployment, the .war file of the application is required. To deploy in AWS, open the Elastic Beanstalk from the services and create an environment for its deployment. Finally, upload the .war file from the “Upload and Deploy” and your application with Amazon Managed Kafka will be up and running.

Elastic-Beanstalk

Through this article, we attempted to provide an overview on how to configure Amazon MSK for creating a Spring Boot application that makes use of Apache Kafka as its messaging middleware.

Blog Author

Narayana Rallabandi

VP and Senior Principal Architect

Related Blogs

19 Aug, 2021

Public Sector Ariba Contract Management with

Executive Summary Sempra Energy, an LSI, An Invenio Company customer, is one of...

Learn more
15 Nov, 2022

UK Police Forces Adopt Technology to Combat Crime

Learn more
10 Aug, 2021

Government Software Buyers Unite!

Government software buyers, lend me your ears! If you are a business owner who...

Learn more