Skip to main content

CMD Integration Pattern for Search and Ingest Services

A . API-Based and Event-Based Integration Pattern for Search Service​

1. API-Based Architecture​

The API-based approach facilitates direct interaction between the Consumer Application and the search services through a centralized traffic management system.

Workflow:​

  1. Consumer Application:
    • Authenticates via the Search APIGEE Endpoint.
    • Requests for data by making an API call.
    • The consuming application raises a Stargate request to gain access to the required APIs.
  2. Traffic Manager:
    • Acts as a load balancer, routing requests to the appropriate Search Service instance based on region.
  3. Regional Search Services:
    • Located in West Europe (WE) and North Europe (NE).
    • Each service connects to its respective Elasticsearch backend for retrieving search results.

2. Event-Based Architecture​

The event-based approach relies on a publish-subscribe model, which decouples the producer (SMDS CMD) and the consumer application.

Workflow:​

  1. SMDS CMD:
    • Publishes events to a CMD Topic hosted on a Kafka platform.
    • The topic is configured with log compaction, ensuring that only the latest updates for a given key are retained.
  2. Consumer Application:
    • Subscribes to the CMD Topic to receive event updates.
    • Processes the events and persists the relevant data into consuming application’s Database (DB).
  3. Consuming Application:
    • Queries the database through the search API built on the data persisted through kafka topic.
    • Retrieves and displays the results for the end user.

B. API-Based Integration Pattern for Ingestion Service​

Outlines the step-by-step process for integrating various APIs to manage data validation, duplication detection, and customer record creation in the Consuming Application.

Step 1: Integrate Rule Set API for Field Validation (Client-Side)​

The Rule Set API ensures field-level validation on the client side by enforcing predefined rules.

Workflow:​

  1. The Consuming Application calls the Rule Set API endpoint via APIGEE.
  2. The Ingest API (/ruleset) processes the request in the West Europe region.
  3. The ruleset data is stored in the West Europe DB and mapped based on country selection.

Key Benefits:​

  • Early detection of field-level errors.
  • Reduced invalid data submissions to the backend.

Step 2: Integrate Validate API for Field Validation (Server-Side)​

The Validate API performs comprehensive field validation on the server side, ensuring data integrity.

Workflow:​

  1. The Consuming Application interacts with the Validate API endpoint via APIGEE.
  2. The Ingest API (/validate) validates the request at the backend in the West Europe region.
  3. Data integrity is maintained by updating the West Europe DB with validated information.

Key Benefits:​

  • Server-side enforcement of validation rules.
  • Enhanced data consistency.

Step 3: Integrate Match/Duplicate API for Duplicate Detection​

The Match/Duplicate API identifies potential duplicate records in the system.

Workflow:​

  1. The Consuming Application invokes the Match/Duplicate API via APIGEE.
  2. Requests are routed through the Traffic Manager, which directs them to the appropriate Search Match Service instance in the West/North Europe region.
  3. The Search Match Service interacts with Elasticsearch to find and flag duplicate records.

Key Benefits:​

  • Efficient detection and handling of duplicate records.
  • Ensures data uniqueness and accuracy.

Step 4: Integrate Ingest API for Customer Record Creation/Update​

The Ingest API handles the creation or update of customer records in the system.

Workflow:​

  1. The Consuming Application sends requests to the Ingest API (/ingest) via APIGEE.
  2. The API writes the processed data to the West Europe DB.
  3. The updated information is published to a CMD Topic for further consumption by downstream systems.

Key Benefits:​

  • Streamlined customer record management.
  • Real-time data updates for downstream consumers via Kafka topics.

CMD API Flow Diagram​

https://maersk-tools.atlassian.net/wiki/spaces/GDASMDS/pages/181703645597

Customer Data Flow Diagram​

https://maersk-tools.atlassian.net/wiki/spaces/GDASMDS/pages/183270376077/CMD+Integration+Pattern+for+Search+and+Ingest+Services#Customer-Data-Flow-Diagram

OAS and Srargate Proxy​

Service NameGithub OAS PathStargate Proxy
Customer Searchhttps://github.com/Maersk-Global/API-JSON-Schema-Definitions/blob/master/apis/901-SMDS-CMD/v1-Bundled/CustomerSearch-API-bundled.v1.yamlhttps://admiralv2.hosting.maersk.com/stargate/api-management/apis/625
Customer Ingesthttps://github.com/Maersk-Global/API-JSON-Schema-Definitions/blob/master/apis/901-SMDS-CMD/v1-Bundled/CustomerIngest-API-bundled.v1.yamlhttps://admiralv2.hosting.maersk.com/stargate/api-management/apis/385
Was this page helpful?