Deadline EMP
Introduction
This document covers the details about the integration of SMDS-EMP platform.
Requirement statement
Maersk is planning to move to event-based integration between producers and consumers. Maersk setup a separate team to manage Event Management Platform (EMP) which will be responsible for environment setup and development activities related to EMP platform. Business will get benefit in terms of no message loss between producer and consumer, can be seamlessly integrated any new consumer or producer application with SMDS, less effort for integration activities, moving to cloud platform etc.
As part of this EPIC, SMDS Deadline application will be integrated with EMP platform to produce Deadline events. Below are the requirements and the outcome of the first phase.
- Create AVRO schema for Deadline interface
- Review the schema with MIM team
- Work with EMP team in terms of schema review, schema registry, topic configuration, connectivity checks etc.
- POC to generate and push JSON message into Kafka topic
- Identify generic events needed for customer core information edits
- Configure required events for KYC
- Work with different consumers for event consume testing if events are required
- Create vessel publish services
- End to End testing
Deadline Producer Owner Info:
| Application Owner Name | Email Id |
|---|---|
| Ramesh Varma.KAbhishek C | Ramesh.kutcherlapati@maersk.comabhishek.chitranshi@maersk.com |
Deadline Consumer Owner Info:
| Application Name | Consumer Type | SPOC | Prod Status | Expected Release date |
|---|---|---|---|---|
| dnet | From Cloud | mohamed.ayoub.hajjem@maersk.comthor.baunsgaard1@maersk.com | Pre-Prod Testing | 1st Week Feb 2022 |
Deadline Configuration Details :
| Topic Name | Message Format | Schema Mapping Info | Message Header Info | Key Column Info | Sample Message Format | Producer Trigger Point | Pre-Prod Sample Producer Parameters | Prod Sample Producer Parameters | Pre-Prod Sample Consumer Parameters | Prod Sample Consumer Parameters |
|---|---|---|---|---|---|---|---|---|---|---|
| MSK.operations.deadline.topic.internal.any.v4Note : Full IDL is not avaible in deadline because deadlines keeps on changing based on the Ace-Dnet consumer request we did the IDL on the below logicETA > sysdate - 30.On Feb 5th we did the IDL that means Jan 5th 2022 onwards data available in the above topic | JsonDeadline_ValueSchema.avsc | The Message what ever we are sending and validate through Avro schema.Please find the below attached Avro schema. | While producing message we are appending the below audit details against the message.EventTypeEventDescriptionEventDateTimeMessageId**Sample Data for Header Fields:**EventType : Create/UpdateEventDescription : vesselCreated / vesselDataUpdatedEventDateTime : Wed Feb 13 09:46:36 UTC 2021MessageId : 93f0d94f-c1a9-4d33-8f6c-92cf88f54846Note : As per the EMP team instruction we are appending above data in byte format to actual message. While consuming make sure convert byte to string to see actual data | While Producing message to EMP we are passing gsisKey, deadlineGroupName to service and fetching data based on portId and deadline groupname and converting that data into json object. The trigger point of Message produce is from OSB Side | Deadline_json1.jsonDeadline_json2.jsonDeadline_json3.jsonDeadline_json4.jsonDeadline_json5.json | From Deadline BPEL | kafka.topic: MSK.operations.vessel.topic.internal.any.v1kafka.bootstrap-servers:rp-cde-rpx-stage.crb.apmoller.net:9093kafka.username: <Emp Team Provided User Name > kafka.password:< From Azure Vault will get the Key > kafka.schema-registry-url:https://lb-sre-stage-cde.crb.apmoller.net:8081kafka.sasl.jaas.config:org.apache.kafka.common.security.scram.ScramLoginModule required username="<Emp Team Provided User Name>" password="<From Azure Vault will get the Key>";kafka.ssl.truststore-location: <trust store file location path > kafka.ssl.truststore-password: <From Azure Vault will get the Key > kafka.schema-registry-ssl-protocol:TLSkafka.onprem-sasl-mechanism:SCRAM-SHA-256kafka.ssl.keystore-location:<key store file location path > kafka.ssl.keystore-password:<From Azure Vault will get the Key > key-password:<From Azure Vault will get the Key > kafka.onprem-login-module:org.apache.kafka.common.security.scram.ScramLoginModule**Topic Level Configuration :**ccloud_Retention Period : 72 Hrsccloud_cleanup_policy: compact , compact Mode ON, bcz of that latest message survived in topic based on retention period defined .topic_type: avro replication_flow: g2cLog Compation enabled based on deadline group and port id | kafka.topic: MSK.operations.vessel.topic.internal.any.v1kafka.bootstrap-servers:rp-cde-rpx.crb.apmoller.net:9093kafka.username: <Emp Team Provided User Name > kafka.password:<From Azure Vault will get the Key > kafka.schema-registry-url:https://lb-sre-cde.crb.apmoller.net:8081kafka.sasl.jaas.config:org.apache.kafka.common.security.scram.ScramLoginModule required username="<Emp Team Provided User Name>" password="<From Azure Vault will get the Key>";kafka.ssl.truststore-location: <trust store file location path > kafka.ssl.truststore-password: <From Azure Vault will get the Key > kafka.schema-registry-ssl-protocol:TLSkafka.onprem-sasl-mechanism:SCRAM-SHA-256kafka.ssl.keystore-location:<key store file location path > kafka.ssl.keystore-password:<From Azure Vault will get the Key > key-password:<From Azure Vault will get the Key > kafka.onprem-login-module:org.apache.kafka.common.security.scram.ScramLoginModule | BOOTSTRAP_SERVERS_CONFIG : pkc-lq8gm.westeurope.azure.confluent.cloud:9092KEY_DESERIALIZER_CLASS_CONFIG : StringDeserializer.classVALUE_DESERIALIZER_CLASS_CONFIG : KafkaAvroDeserializer.classREQUEST_TIMEOUT_MS_CONFIG : 30000BATCH_SIZE_CONFIG : 16384SEND_BUFFER_CONFIG : 131072SECURITY_PROTOCOL : SASL_SSLSASL_MECHANISM : PLAINAUTO_OFFSET_RESET_CONFIG : latestENABLE_AUTO_COMMIT_CONFIG : falseSCHEMA_REGISTRY_URL_CONFIG : https://psrc-4kk0p.westeurope.azure.confluent.cloudBASIC_AUTH_CREDENTIALS_SOURCE : USER_INFOSCHEMA_REGISTRY_USER_INFO_CONFIG: <From Azure Vault will get the Key>:<password > SPECIFIC_AVRO_READER_CONFIG : true | BOOTSTRAP_SERVERS_CONFIG : pkc-lq8gm.westeurope.azure.confluent.cloud:9092KEY_DESERIALIZER_CLASS_CONFIG : StringDeserializer.classVALUE_DESERIALIZER_CLASS_CONFIG : KafkaAvroDeserializer.classREQUEST_TIMEOUT_MS_CONFIG : 30000BATCH_SIZE_CONFIG : 16384SEND_BUFFER_CONFIG : 131072SECURITY_PROTOCOL : SASL_SSLSASL_MECHANISM : PLAINAUTO_OFFSET_RESET_CONFIG : latestENABLE_AUTO_COMMIT_CONFIG : falseSCHEMA_REGISTRY_URL_CONFIG : https://psrc-4kk0p.westeurope.azure.confluent.cloudBASIC_AUTH_CREDENTIALS_SOURCE : USER_INFOSCHEMA_REGISTRY_USER_INFO_CONFIG: <From Azure Vault will get the Key>:<password > SPECIFIC_AVRO_READER_CONFIG : true |
Was this page helpful?