Vessel EMP
Introduction
This document covers the details about the integration of SMDS-EMP platform.
Requirement statement
Maersk is planning to move to event-based integration between producers and consumers. Maersk setup a separate team to manage Event Management Platform (EMP) which will be responsible for environment setup and development activities related to EMP platform. Business will get benefit in terms of no message loss between producer and consumer, can be seamlessly integrated any new consumer or producer application with SMDS, less effort for integration activities, moving to cloud platform etc.
As part of this EPIC, SMDS VESSEL application will be integrated with EMP platform to produce vessel events. Below are the requirements and the outcome of the first phase.
- Create AVRO schema for Vessel interface
- Review the schema with MIM team
- Work with EMP team in terms of schema review, schema registry, topic configuration, connectivity checks etc.
- POC to generate and push JSON message into Kafka topic
- Identify generic events needed for customer core information edits
- Configure required events for KYC
- Work with different consumers for event consume testing if events are required
- Create vessel publish services
- End to End testing
Vessel Producer Owner Info:
| Application Owner Name | Email Id |
|---|---|
| Mahindra Waghmare | Mahendra.Waghmare@maersk.com |
Vessel Consumer Owner Info:
Vessel Configuration Details :
| Topic Name | Message Format | Schema Mapping Info | Message Header Info | Key Column Info | Sample Message Format | Producer Trigger Point | Pre-Prod Sample Producer Parameters | Prod Sample Producer Parameters | Pre-Prod Sample Consumer Parameters | Prod Sample Consumer Parameters |
|---|---|---|---|---|---|---|---|---|---|---|
| Active Topic with latest feed + Full Initial data load - Please consume for this topic - topic MSK.fleet.vessel.topic.internal.any.v1 | Json | The Message what ever we are sending and validate through Avro schema.Please find the below attached Avro schema.Vessel_ValueSchema.avsc | While producing message we are appending the below audit details against the message.EventTypeEventDescriptionEventDateTimeMessageId**Sample Data for Header Fields:**EventType : Create/UpdateEventDescription : vesselCreated / vesselDataUpdatedEventDateTime : Wed Feb 13 09:46:36 UTC 2021MessageId : 93f0d94f-c1a9-4d33-8f6c-92cf88f54846Note : As per the EMP team instruction we are appending above data in byte format to actual message. While consuming make sure convert byte to string to see actual data | VesselIDWhile Producing message to EMP we are passing Vessel ID to service and fetching data based on vessel ID and converting that data into json object | Vessel_new.json | SMDS vessel application | kafka.topic: MSK.operations.vessel.topic.internal.any.v1kafka.bootstrap-servers:rp-cde-rpx-stage.crb.apmoller.net:9093kafka.username: <Emp Team Provided User Name > kafka.password:<Emp Team Provided password > kafka.schema-registry-url:https://lb-sre-stage-cde.crb.apmoller.net:8081kafka.sasl.jaas.config:org.apache.kafka.common.security.scram.ScramLoginModule required username="<Emp Team Provided User Name>" password="<Emp Team Provided password>";kafka.ssl.truststore-location: <trust store file location path > kafka.ssl.truststore-password: <trust store password > kafka.schema-registry-ssl-protocol:TLSkafka.onprem-sasl-mechanism:SCRAM-SHA-256kafka.ssl.keystore-location:<key store file location path > kafka.ssl.keystore-password:<key store password > key-password:<key store password > kafka.onprem-login-module:org.apache.kafka.common.security.scram.ScramLoginModule**Topic Level Configuration :**ccloud_Retention Period : 72 Hrsccloud_cleanup_policy: compactcompact Mode ON, bcz of that latest message survived in topic based on retention period defined .topic_type: avro replication_flow: g2cLog Compation enabled based on VesselID | kafka.topic: MSK.operations.vessel.topic.internal.any.v1kafka.bootstrap-servers:rp-cde-rpx.crb.apmoller.net:9093kafka.username: <Emp Team Provided User Name > kafka.password:<Emp Team Provided password > kafka.schema-registry-url:https://lb-sre-cde.crb.apmoller.net:8081kafka.sasl.jaas.config:org.apache.kafka.common.security.scram.ScramLoginModule required username="<Emp Team Provided User Name>" password="<Emp Team Provided password>";kafka.ssl.truststore-location: <trust store file location path > kafka.ssl.truststore-password: <trust store password > kafka.schema-registry-ssl-protocol:TLSkafka.onprem-sasl-mechanism:SCRAM-SHA-256kafka.ssl.keystore-location:<key store file location path > kafka.ssl.keystore-password:<key store password > key-password:<key store password > kafka.onprem-login-module:org.apache.kafka.common.security.scram.ScramLoginModule | BOOTSTRAP_SERVERS_CONFIG : pkc-lq8gm.westeurope.azure.confluent.cloud:9092KEY_DESERIALIZER_CLASS_CONFIG : StringDeserializer.classVALUE_DESERIALIZER_CLASS_CONFIG : KafkaAvroDeserializer.classREQUEST_TIMEOUT_MS_CONFIG : 30000BATCH_SIZE_CONFIG : 16384SEND_BUFFER_CONFIG : 131072SECURITY_PROTOCOL : SASL_SSLSASL_MECHANISM : PLAINAUTO_OFFSET_RESET_CONFIG : latestENABLE_AUTO_COMMIT_CONFIG : falseSCHEMA_REGISTRY_URL_CONFIG : https://psrc-4kk0p.westeurope.azure.confluent.cloudBASIC_AUTH_CREDENTIALS_SOURCE : USER_INFOSCHEMA_REGISTRY_USER_INFO_CONFIG: <Emp Team Provided Key>:<password > SPECIFIC_AVRO_READER_CONFIG : true | BOOTSTRAP_SERVERS_CONFIG : pkc-lq8gm.westeurope.azure.confluent.cloud:9092KEY_DESERIALIZER_CLASS_CONFIG : StringDeserializer.classVALUE_DESERIALIZER_CLASS_CONFIG : KafkaAvroDeserializer.classREQUEST_TIMEOUT_MS_CONFIG : 30000BATCH_SIZE_CONFIG : 16384SEND_BUFFER_CONFIG : 131072SECURITY_PROTOCOL : SASL_SSLSASL_MECHANISM : PLAINAUTO_OFFSET_RESET_CONFIG : latestENABLE_AUTO_COMMIT_CONFIG : falseSCHEMA_REGISTRY_URL_CONFIG : https://psrc-4kk0p.westeurope.azure.confluent.cloudBASIC_AUTH_CREDENTIALS_SOURCE : USER_INFOSCHEMA_REGISTRY_USER_INFO_CONFIG: <Emp Team Provided Key>:<password > SPECIFIC_AVRO_READER_CONFIG : true |
Was this page helpful?