Challenges in Modernization

Topic
Onboarding to K8S	Platform-Agnostic for Cloud Deployments Enhanced Scalability and Resilience Optimized for Cost Efficiency Centralized DevOps Management managed by perpetual team Enhanced application security Seamless Alignment with Maersk's Observability Framework Mandates Internal DevOps Competence Timeframe for Project Execution
Multi-Region Ingestion	SMDS employs a zone-redundant high availability (HA) setup for its production PostgreSQL flexible server, with an uptime SLA of 99.99%. Data ingestion is exclusively available through the West Europe region, creating a potential issue where write services become inactive in case of any disruptions. Search APIs are accessible through both the West and North Europe regions, enhancing redundancy and ensuring uninterrupted service availability. Azure flexible servers currently supports multiple read replicas, limiting write capabilities to a single server. When considering the introduction of a multi-region database server for write operations, it's crucial to address data consistency between regions and develop a plan for maintaining service in case one of the regions experiences downtime. Given the high uptime SLA requirements, it's worth considering the implementation of event sourcing for the West Europe and North Europe regions to enhance reliability and fault tolerance.
Deduplication/Matching logic	Our current approach leverages ElasticSearch to efficiently identify duplicate data within SMDS. This solution significantly simplifies the process of detecting duplicate data across multiple domains, reducing complexity and maintenance overhead. It offers a seamless integration path for all other domains, requiring minimal effort for implementation. While ElasticSearch is a valuable tool for deduplication and data search, we are actively working to optimize costs associated with its maintenance. To ensure data integrity, we commit to ongoing code and pipeline development and maintenance efforts for keeping data synchronized with our relational database. We are also exploring other potential options to enhance our duplicate data identification and management capabilities.
Azure Address	Utilized for enhancing address standardization and validation processes. Does not provide optimal result (around 30-40%)

Architectural / Solutioning Challenges

• Finding an alternative for Match feature (Duplicate Search) provided earlier by Informatica MDM product suite. Initially PostgreSQL was being called out as an option but it didn’t turn out good enough after POC. Later we found Elastic as a better and promising solution. There were some features that it couldn’t match exactly as Informatica de-duplication logic, however, it was acceptable.
• Multi-region ingestion. Real-time duplicate identification is one of the critical features in CMD to avoid data quality issues related to duplicity. CMD Portal and Maersk.com ingest (create/update) requests must go through a Match API based out of Elastic as a first step. Elastic can have some latency in re-indexing data (known challenge with elastic - it can be reduced but not completely eliminated). Moreover, cross region replication may have few seconds of latency as well. If multi-region ingestion is enabled, then Match API may incur higher probability of latency that might cause duplicate record ingestion in the event of concurrent request. Unique IDs creation for Customer, Contact, Facility, Concern, CBU would also need an added logic to ensure zero collision when multi-region ingestion is exposed. With limited time to release modern platform, it was de-scoped. Since CMD have more than 80% traffic for read APIs, and low on create/update requests, it was an accepted SLA to keep Ingestion only in West Europe region with multi-zone enablement. It has been jointly approved between CMD, Maersk.com and MAST teams.
• Address Quality Assessment: We need to explore alternatives or optimize our current address quality validation process to enhance overall address quality across all SMDS domains.
• Database Design within SMDS: Within the SMDS domain, the existence of multiple databases, each specific to its respective domain, results in a beneficial loose coupling. However, this configuration also necessitates the introduction of an additional platform, such as API or Kafka, to facilitate data transfer between different entities. While maintaining separate databases offers advantages, it does come with the drawback of increased latency, elevated maintenance requirements, and higher costs.

Stumbling blocks

• Multiple systems & portals for a single CMD application, involving external tools having custom business solutions. On-premises CMD had end user portal, IDD Portal, PowerCenter application, Bulk Upload & Reporting portal. All of these had custom logic in legacy technologies & product configurations. Build over 7-8 years, the documentation/knowledge was limited. A lot of reverse engineering was involved to create the Integrated single modern CMD portal.
• Poor consumer landscape documentation & attribute consumption knowledge in legacy systems in pub-sub model. MQs were built 5-7 years ago and enriched over time without much documentation.
• ESB governance did not have complete details of operation level consumption for the SOAP monolith service-based integration
• Final go-live planning due to tight coupling of services with Maersk.com, SFDC , HSUD. Any downtime to these systems would’ve caused major incidents.

Future Challenges

• Numerous applications continue to rely heavily on MQ for event consumption, yet their capacity to efficiently process published events remains subpar.
• Notably, SAP BI maintains a direct connection with the CMD Database, utilizing a read replica for its operations.
• Decommissioning MQ connections of migrated applications is costly since it's managed by the vendor (TCS) unless we identify internal engineers with knowledge and access

Can be discussed

• Exposure of data through API/Kafka
• RETINA Migration
• Integration with Observability Framework
• Frequent changes in Maerk Guidelines
• Automation Testing
• Internal/External API