{"id":160864,"date":"2023-05-02T10:55:17","date_gmt":"2023-05-02T09:55:17","guid":{"rendered":"https:\/\/www.devoteam.com\/expert-view\/anypoint-mq\/"},"modified":"2023-05-02T10:55:17","modified_gmt":"2023-05-02T09:55:17","slug":"anypoint-mq","status":"publish","type":"expert-view","link":"https:\/\/devoteam.info\/uk\/expert-view\/anypoint-mq\/","title":{"rendered":"Design Reliable, High Performing Fault-tolerant Systems with Anypoint MQ"},"content":{"rendered":"\n

Introduction<\/strong><\/h2>\n\n\n\n

Designing high-performance, reliable, and fault-tolerant systems with Anypoint MQ is essential for any organisation that wants to send and receive messages between different applications securely. Anypoint MQ<\/a> is a robust messaging service from MuleSoft that makes it easier than ever to implement this type of system.<\/p>\n\n\n\n

System design becomes complicated in a diverse system with variable traffic and minimal control of incoming requests. As part of the system’s core design, a system should be fault tolerant with zero data loss and offer high performance. The system also should be reliable, ensuring reprocessing of low-quality data in a scalable manner.<\/p>\n\n\n\n

In this blog, we will explore a case study of implementing Anypoint MQ for KYC (Know Your Customer) compliance that the government mandates for the financial services industry.<\/p>\n\n\n\n

Case Study – Know Your Customer<\/strong><\/h2>\n\n\n\n

The government mandates KYC compliance for the financial services industry. Key areas where Anypoint MQ would benefit the KYC are:<\/p>\n\n\n\n

(i) Customer Identification & Acceptance Policy<\/p>\n\n\n\n

(ii) Monitoring of Transactions and On-going Due Diligence<\/p>\n\n\n\n

(iii) Risk Management<\/p>\n\n\n\n

Financial institutes, which have a huge customer base in both commercial and retail sectors, are flooded with a high non-deterministic number of KYC requests that are initiated by various third parties, customers, and partners. Managing such an uncontrollable and non-measurable frequency of requests is a huge challenge for the financial services industry.  <\/p>\n\n\n\n

Let\u2019s first understand the key capabilities required by such systems and how Anypoint MQ features fulfil these capabilities.<\/p>\n\n\n\n

Operational Capabilities<\/strong><\/p>\n\n\n\n


Key Capabilities<\/strong><\/td>
Description<\/strong><\/td>
Anypoint MQ Feature<\/strong><\/td><\/tr>
Zero loss of data<\/td>A system should be flexible enough to accept all the requests without loss of data and should also provide a resilient and reliable way to serve these requests<\/td>Cloud messaging service that performs asynchronous messaging and ensures data is not lost once received<\/td><\/tr>
Multicast the requests for parallel processing<\/td>The message received by the system should be multicasted and can be replayed into different systems for parallel processing based on the target system\u2019s need<\/td>Message exchange binding with one or more message queues<\/td><\/tr>
Handle failure and reprocess data<\/td>If a message fails repetitively for a similar reason, it should be possible to refine it by cleansing the data from the knowledge base, correcting issues, and then re-messaging.<\/td>DLQ Messages can be reprocessed and injected into the exchange after reprocessing and re-messaging<\/td><\/tr>
Automation of MQ management<\/td>As the number of target systems expands, the MQ should be created or deleted in an automated manner instead of manually creating and managing it. <\/td>AMQ Admin API. AMQ provides APIs to create\/delete\/edit MQ<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Functional Capabilities<\/strong><\/p>\n\n\n\n


Key Capabilities<\/strong><\/td>
Description<\/strong><\/td>
Anypoint MQ Feature<\/strong><\/td><\/tr>
Support Non-deterministic requests without losing any requests<\/td>The system should accept the non-deterministic load and process it asynchronously such that all the requests are processed without loss<\/td>Asynchronous messaging with the option of FIFO or Standard Queue <\/td><\/tr>
Intelligent routing based on Request<\/td>Incoming message requests should route to a specific queue which is associated with the egress<\/td>Anypoint Exchange Message Routing Rule<\/td><\/tr>
Dispatch message to the functional queue<\/td>The message should be sent to one or more queues based on the message type<\/td>Binding exchange with one or more queue<\/td><\/tr>
Undelivered Request Retentions<\/td>The invalid request should not be lost, and there should be a possibility to retain it and replay it later<\/td>Dead-letter queue (DLQ) for undeliverable messages (Refer to Figure 2 for more details)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Non-Functional Capabilities<\/strong><\/p>\n\n\n\n


Key Capabilities<\/strong><\/td>
Description<\/strong><\/td>
Anypoint MQ Feature<\/strong><\/td><\/tr>
Allow recovery from temporary failure of the back-end system<\/td>If the request is failed due to any back-end failure, the request should not be played recursively and instead should allow to re-play again after a specific interval<\/td>Enabling Circuit breaker pattern <\/td><\/tr>
High throughput for certain load<\/td>Up to a certain load of incoming requests, the request processing should be faster. The back-end throughput determines the incoming request load.<\/td>Subscribe with Prefetch Message<\/td><\/tr>
Controlled and authorised access to a Message queue<\/td>Message queue should be secured and highly accessible, considering the message consists of PII data <\/td>Role-based and client management<\/td><\/tr>
High Availability & Disaster Recovery<\/td>Ideally, a zero downtime of the messaging system is essential, considering the incoming requests are non-deterministic, so the downtime of the MQ would result in the loss of data<\/td>Cloud-based managed queue<\/td><\/tr>
Secured message for financial data<\/td>PII data within the message should be encrypted in transit to ensure the high security of financial data<\/td>Encryption of messages<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Based on the above features of Anypoint MQ, let’s look at how we can design the system with the use case of KYC compliance in a Bank.<\/p>\n\n\n\n

KYC Compliance: Technical Design (Anypoint MQ Perspective)<\/strong><\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
    \n
  1. Consider that third-party systems send KYC compliance requests to the system. These are HTTP\/HTTPS requests with collated data of the personnels whose KYC compliance checks are required. The personnels\u2019 KYC data to the front channel are received non-deterministic and irregular in nature. These are authorised requests with the Access token to ensure the authenticity and identity of the source system. <\/li>\n\n\n\n
  2. The front channel\/experience layer accepts the requests and passes them to the KYC Compliance Receiver MQ. For simplicity purposes, Gateway and Load Balancer are omitted in the diagram. <\/li>\n\n\n\n
  3. KYC requests are validated, filtered, and tagged before being sent to the exchange for specific egress for further processing. Filtration and tagging are done by the application to enrich the message.<\/li>\n\n\n\n
  4. KYC compliance exchange has various rule sets defined which decide to which MQ the message should be sent. A typical ruleset for KYC Compliance can be: If \u2018reKYC\u2019<\/strong> is \u2018required\u2019 ? Send to KYC Processing MQ. If \u2018kycTrackingOnly<\/strong>\u2019 is \u2018true\u2019 ? Send to KYC Tracking MQ. If \u2018physicalKYC<\/strong>\u2019 is \u2018required\u2019 ? KYC Logistic MQ.<\/li>\n\n\n\n
  5. Each MQ can have one or more egress associated (for simplicity, single egress is mentioned in the diagram), which consumes messages from the respective MQ. <\/li>\n\n\n\n
  6. The target system is the consumer of the specific egress MQ, which reads messages, transforms them, and sends them to the respective third-party system.<\/li>\n<\/ol>\n\n\n\n

    This kind of dispatcher logic enables intelligent processing of the messages to the respective queue. <\/p>\n\n\n\n

    This way, we build a highly reliable, available, high-performance system using Anypoint MQ. Above design covers three business use cases of KYC..<\/p>\n\n\n\n

      \n
    1. Re-KYC – Personals whose  KYC is required again due to expiry or change in  of previous KYC.<\/li>\n\n\n\n
    2. KYC Tracking – Personals whose KYC Requests were submitted earlier and now require tracking.<\/li>\n\n\n\n
    3. Physical KYC- A new KYC request is submitted and requires logistic processing.<\/li>\n<\/ol>\n\n\n\n

      We have yet to discuss how to achieve fault tolerance in the solution. Anypoint MQ provides specific features to support this. Let\u2019s look at the \u2018KYC Compliance Receiver MQ\u2019 from the previous diagram and understand it in more detail.<\/p>\n\n\n\n

      \"\"<\/figure>\n\n\n\n
        \n
      1. A KYC Compliant Receiver Queue (MQ) is backed by a Dead Letter Queue (DLQ). Messages that can\u2019t be processed after several unsuccessful delivery attempts are placed into DLQ.<\/li>\n\n\n\n
      2. There can be various reasons for the message’s failure, varying from business validation failure to an invalid message. A message cleansing system listens to DLQ and corrects the message based on the defined business rules. In some cases, manual intervention is required.<\/li>\n\n\n\n
      3. Once the message is corrected, the system retrieves the original message, enriches, reformats and generates a new message with existing metadata.<\/li>\n\n\n\n
      4. The newly generated message is now placed into the original MQ to play it again.<\/li>\n<\/ol>\n\n\n\n

        This way, the system becomes fault-tolerant and ready to process clean requests.<\/p>\n\n\n\n

        Final Thoughts<\/strong><\/h2>\n\n\n\n

        In this blog, we explored a case study showing how Anypoint MQ could be used for KYC (Know Your Customer) compliance that the government mandates for the financial services industry.<\/p>\n\n\n\n

        From this, we can see that Anypoint MQ is a multi-tenant, cloud messaging service that enables customers to perform advanced asynchronous messaging scenarios between applications in a high-performance, reliable and fault-tolerant manner.<\/p>\n\n\n\n

        If you would like to explore how your organisation could use Anypoint MQ or if you would like to discuss how Devoteam could help you to implement solutions for your business. Please do not hesitate to contact us.<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

        Introduction Designing high-performance, reliable, and fault-tolerant systems with Anypoint MQ is essential for any organisation that wants to send and receive messages between different applications securely. Anypoint MQ is a robust messaging service from MuleSoft that makes it easier than ever to implement this type of system. System design becomes complicated in a diverse system […]<\/p>\n","protected":false},"featured_media":111993,"template":"","categories":[752],"tags":[1148],"industry":[],"class_list":["post-160864","expert-view","type-expert-view","status-publish","has-post-thumbnail","hentry","category-business-platforms","tag-uk-import-uk"],"acf":[],"cards":"\n\t

        \n\n\t\t
        \"Design<\/a><\/figure>\n\n\t\t\n\t\t
        \n\t

        Expert View<\/p>\n\n\t\t\n\t\t

        Design Reliable, High Performing Fault-tolerant Systems with Anypoint MQ<\/a><\/h3><\/div>\n\t\t\n\t<\/div>\n\n","yoast_head":"\nDesign Reliable, High Performing Fault-tolerant Systems with Anypoint MQ | Devoteam<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/devoteam.info\/uk\/expert-view\/anypoint-mq\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Design Reliable, High Performing Fault-tolerant Systems with Anypoint MQ\" \/>\n<meta property=\"og:description\" content=\"Introduction Designing high-performance, reliable, and fault-tolerant systems with Anypoint MQ is essential for any organisation that wants to send and receive messages between different applications securely. Anypoint MQ is a robust messaging service from MuleSoft that makes it easier than ever to implement this type of system. System design becomes complicated in a diverse system […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/devoteam.info\/uk\/expert-view\/anypoint-mq\/\" \/>\n<meta property=\"og:site_name\" content=\"Devoteam\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/devoteam.info\\\/uk\\\/expert-view\\\/anypoint-mq\\\/\",\"url\":\"https:\\\/\\\/devoteam.info\\\/uk\\\/expert-view\\\/anypoint-mq\\\/\",\"name\":\"Design Reliable, High Performing Fault-tolerant Systems with Anypoint MQ | Devoteam\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/devoteam.info\\\/uk\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/devoteam.info\\\/uk\\\/expert-view\\\/anypoint-mq\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/devoteam.info\\\/uk\\\/expert-view\\\/anypoint-mq\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/devoteam.info\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/shutterstock_379972717-scaled-1.jpg\",\"datePublished\":\"2023-05-02T09:55:17+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/devoteam.info\\\/uk\\\/expert-view\\\/anypoint-mq\\\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/devoteam.info\\\/uk\\\/expert-view\\\/anypoint-mq\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\\\/\\\/devoteam.info\\\/uk\\\/expert-view\\\/anypoint-mq\\\/#primaryimage\",\"url\":\"https:\\\/\\\/devoteam.info\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/shutterstock_379972717-scaled-1.jpg\",\"contentUrl\":\"https:\\\/\\\/devoteam.info\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/shutterstock_379972717-scaled-1.jpg\",\"width\":2560,\"height\":1709},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/devoteam.info\\\/uk\\\/expert-view\\\/anypoint-mq\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/devoteam.info\\\/uk\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Expert View\",\"item\":\"https:\\\/\\\/devoteam.info\\\/uk\\\/expert-view\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Design Reliable, High Performing Fault-tolerant Systems with Anypoint MQ\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/devoteam.info\\\/uk\\\/#website\",\"url\":\"https:\\\/\\\/devoteam.info\\\/uk\\\/\",\"name\":\"Devoteam\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/devoteam.info\\\/uk\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Design Reliable, High Performing Fault-tolerant Systems with Anypoint MQ | Devoteam","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/devoteam.info\/uk\/expert-view\/anypoint-mq\/","og_locale":"en_GB","og_type":"article","og_title":"Design Reliable, High Performing Fault-tolerant Systems with Anypoint MQ","og_description":"Introduction Designing high-performance, reliable, and fault-tolerant systems with Anypoint MQ is essential for any organisation that wants to send and receive messages between different applications securely. Anypoint MQ is a robust messaging service from MuleSoft that makes it easier than ever to implement this type of system. System design becomes complicated in a diverse system […]","og_url":"https:\/\/devoteam.info\/uk\/expert-view\/anypoint-mq\/","og_site_name":"Devoteam","twitter_card":"summary_large_image","twitter_misc":{"Estimated reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/devoteam.info\/uk\/expert-view\/anypoint-mq\/","url":"https:\/\/devoteam.info\/uk\/expert-view\/anypoint-mq\/","name":"Design Reliable, High Performing Fault-tolerant Systems with Anypoint MQ | Devoteam","isPartOf":{"@id":"https:\/\/devoteam.info\/uk\/#website"},"primaryImageOfPage":{"@id":"https:\/\/devoteam.info\/uk\/expert-view\/anypoint-mq\/#primaryimage"},"image":{"@id":"https:\/\/devoteam.info\/uk\/expert-view\/anypoint-mq\/#primaryimage"},"thumbnailUrl":"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/shutterstock_379972717-scaled-1.jpg","datePublished":"2023-05-02T09:55:17+00:00","breadcrumb":{"@id":"https:\/\/devoteam.info\/uk\/expert-view\/anypoint-mq\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/devoteam.info\/uk\/expert-view\/anypoint-mq\/"]}]},{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/devoteam.info\/uk\/expert-view\/anypoint-mq\/#primaryimage","url":"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/shutterstock_379972717-scaled-1.jpg","contentUrl":"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/shutterstock_379972717-scaled-1.jpg","width":2560,"height":1709},{"@type":"BreadcrumbList","@id":"https:\/\/devoteam.info\/uk\/expert-view\/anypoint-mq\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/devoteam.info\/uk\/"},{"@type":"ListItem","position":2,"name":"Expert View","item":"https:\/\/devoteam.info\/uk\/expert-view\/"},{"@type":"ListItem","position":3,"name":"Design Reliable, High Performing Fault-tolerant Systems with Anypoint MQ"}]},{"@type":"WebSite","@id":"https:\/\/devoteam.info\/uk\/#website","url":"https:\/\/devoteam.info\/uk\/","name":"Devoteam","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/devoteam.info\/uk\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"}]}},"uagb_featured_image_src":{"full":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/shutterstock_379972717-scaled-1.jpg",2560,1709,false],"thumbnail":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/shutterstock_379972717-scaled-1-150x150.jpg",150,150,true],"medium":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/shutterstock_379972717-scaled-1-300x200.jpg",300,200,true],"medium_large":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/shutterstock_379972717-scaled-1-768x513.jpg",768,513,true],"large":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/shutterstock_379972717-scaled-1-1024x684.jpg",1024,684,true],"1536x1536":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/shutterstock_379972717-scaled-1-1536x1025.jpg",1536,1025,true],"2048x2048":["https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/shutterstock_379972717-scaled-1-2048x1367.jpg",2048,1367,true]},"uagb_author_info":{"display_name":"Julien Pawlowski","author_link":"https:\/\/devoteam.info\/uk\/author\/"},"uagb_comment_info":0,"uagb_excerpt":"Introduction Designing high-performance, reliable, and fault-tolerant systems with Anypoint MQ is essential for any organisation that wants to send and receive messages between different applications securely. Anypoint MQ is a robust messaging service from MuleSoft that makes it easier than ever to implement this type of system. System design becomes complicated in a diverse system…","_links":{"self":[{"href":"https:\/\/devoteam.info\/uk\/wp-json\/wp\/v2\/expert-view\/160864","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devoteam.info\/uk\/wp-json\/wp\/v2\/expert-view"}],"about":[{"href":"https:\/\/devoteam.info\/uk\/wp-json\/wp\/v2\/types\/expert-view"}],"version-history":[{"count":0,"href":"https:\/\/devoteam.info\/uk\/wp-json\/wp\/v2\/expert-view\/160864\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devoteam.info\/uk\/wp-json\/wp\/v2\/media\/111993"}],"wp:attachment":[{"href":"https:\/\/devoteam.info\/uk\/wp-json\/wp\/v2\/media?parent=160864"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devoteam.info\/uk\/wp-json\/wp\/v2\/categories?post=160864"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devoteam.info\/uk\/wp-json\/wp\/v2\/tags?post=160864"},{"taxonomy":"industry","embeddable":true,"href":"https:\/\/devoteam.info\/uk\/wp-json\/wp\/v2\/industry?post=160864"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}