Geo EMP
Introduction
This document covers the details about the integration of SMDS-EMP platform.
Requirement statement
Maersk is planning to move to event-based integration between producers and consumers. Maersk setup a separate team to manage Event Management Platform (EMP) which will be responsible for environment setup and development activities related to EMP platform. Business will get benefit in terms of no message loss between producer and consumer, can be seamlessly integrated any new consumer or producer application with SMDS, less effort for integration activities, moving to cloud platform etc.
As part of this EPIC, SMDS GEO application will be integrated with EMP platform to produce Geo events. Below are the requirements and the outcome of the first phase.
- Create AVRO schema for Geo interface
- Review the schema with MIM team
- Work with EMP team in terms of schema review, schema registry, topic configuration, connectivity checks etc.
- POC to generate and push JSON message into Kafka topic
- Identify generic events needed for customer core information edits
- Configure required events for KYC
- Work with different consumers for event consume testing if events are required
- Create vessel publish services
- End to End testing
Geo Producer Owner Info:
| Name | Email ID |
|---|---|
| Noellyn Jasa | noellyn.villar.jasa@maersk.com |
Note : Please connect "noellyn.villar.jasa@maersk.com" for any kind of approval w.r.t connecting/consuming data from Topic # "MSK.geography.gda.topic.internal.any.v2"
Geo EMP ER Diagram :
Geography EMP ERD Diagram.pptx
Geo Consumer Owner Info:
Geo Configuration Details :
| Topic Name | Message Format | Geo Avro Schema | Message Header | Key Columns to get Geo data | Sample Message | Producer Trigger | Pre-Prod Sample Producer Parameter |
|---|---|---|---|---|---|---|---|
| MSK.geography.gda.topic.internal.any.v3 | JSON | Geography_AvroSchema.avscPlease find the updated schema for GeoThe Message what ever we are sending and validate through Avro schema.Please find the Above attached Avro schema for Geo.Note: Site will publish to Facility Topic. | While producing message we are appending the below audit details against the message.EventTypeEventDescriptionEventDateTimeMessageId**Sample Data for Header Fields:**EventType : Create/UpdateEventDescription : vesselCreated / vesselDataUpdatedEventDateTime : Wed Feb 13 09:46:36 UTC 2021MessageId : 93f0d94f-c1a9-4d33-8f6c-92cf88f54846 | Geo RowidWhile Producing message to EMP we are passing GeoRowid to service and fetching data based on that and converting that data into json object | businessDefinedArea.jsonCity.jsonContinent.jsoncountry.jsonpostalcode.jsonstate.jsonsubCity.json | SMDS Geography application | kafka.topic: MSK.geography.gda.topic.internal.any.v3kafka.bootstrap-servers:rp-cde-rpx-stage.crb.apmoller.net:9093kafka.username: <Emp Team Provided User Name > kafka.password:<Emp Team Provided password > kafka.schema-registry-url:https://lb-sre-stage-cde.crb.apmoller.net:8081kafka.sasl.jaas.config:org.apache.kafka.common.security.scram.ScramLoginModule required username="<Emp Team Provided User Name>" password="<Emp Team Provided password>";kafka.ssl.truststore-location: <trust store file location path > kafka.ssl.truststore-password: <trust store password > kafka.schema-registry-ssl-protocol:TLSkafka.onprem-sasl-mechanism:SCRAM-SHA-256kafka.ssl.keystore-location:<key store file location path > kafka.ssl.keystore-password:<key store password > key-password:<key store password > kafka.onprem-login-module:org.apache.kafka.common.security.scram.ScramLoginModule**Topic Level Configuration :**ccloud_Retention Period : 72 Hrsccloud_cleanup_policy: compactcompact Mode ON, bcz of that latest message survived in topic based on retention period defined .topic_type: avro replication_flow: g2cLog Compation enabled based on GeoId |