Consume EMP Feeds from Cloud to On-Prem

Use Case

As an on-prem consumer application, the service should be able to consume messages from an existing topic replica.

Prerequisites

• Topic name
• Keystore and Truststore Certificates along with their passwords
• c2g replication enabled for the concerned topic
• On-prem consumer group ID

The process

1. To consume events, the following mandatory configurations need to be in place:
   • Schema Registry URL (schema.registry.url)
   • Bootstrap Servers Config (bootstrap.servers)
   • SSL Key/Value Deserializer (key.deserializer | value.deserializer)
   • SSL Key Store location and password (ssl.keystore.location | ssl.keystore.password)
   • SSL Key password (ssl.key.password)
   • Trust Store location and password (ssl.truststore.location | ssl.truststore.password)
   • SASL mechanism (sasl.mechanism)
   • Schema Registry Key Store location and password (schema.registry.ssl.keystore.location | schema.registry.ssl.keystore.password)
   • Schema Registry Trust Store location and password (schema.registry.ssl.truststore.location | schema.registry.ssl.truststore.password)
   • Schema Registry Key Password (schema.registry.ssl.key.password)
   • Schema Registry SSL protocol (security.protocol)
   • Jaas Config (sasl.jaas.config)
2. These dependencies need to be added:

<dependency>
	<groupId > org.springframework.kafka</groupId>
	<artifactId > spring-kafka</artifactId>
</dependency>

<dependency>
    <groupId > io.confluent</groupId>
    <artifactId > kafka-schema-serializer</artifactId>
    <version > 6.2.0</version>
</dependency>

<dependency>
    <groupId > io.confluent</groupId>
    <artifactId > kafka-schema-registry-client</artifactId>
    <version > 6.2.0</version>
</dependency>

<dependency>
    <groupId > org.apache.avro</groupId>
    <artifactId > avro</artifactId>
    <version > 1.10.2</version>
</dependency>

<dependency>
    <groupId > io.confluent</groupId>
    <artifactId > kafka-avro-serializer</artifactId>
    <version > 6.2.0</version>
</dependency>

<dependency>
    <groupId > org.apache.kafka</groupId>
    <artifactId > kafka-streams</artifactId>
</dependency>

3. Concurrent Kafka Listener Container Factory needs to be initialized using consumer factory.

4. With all the configurations in place messages can be consumed using @KafkaListener annotation.

5. Finally, the application needs to acknowledge the topic that message has been processed which will commit the topic offset value or else there’s a chance of receiving the same message again.

On-prem consumers cannot be tested in the local environment(reason being only SIT, Pre-production and Production environments' are whitelisted) so the application needs to be deployed in the SIT WebLogic server for testing.