Intelligent Kafka message routing using Drools DMN Engine and Apache Camel

Intelligent Kafka message routing using Drools DMN Engine and Apache Camel

In this post I want to share an interesting use-case of Healthcare message routing, which we implemented using the Drools DMN Engine and Apache Camel, in order to route and dispatch Patient’s Admission-Discharge-Transfer message types to the required Kafka topics and therefore queuing the message to the appropriate sub-system. I believe it is both a very pragmatic and interesting combination of technologies, while it also shows a blueprint and generalised pattern which can be easily replicated to other domains, in order to achieve intelligent message routing for Apache Kafka for many other use-cases.

You can see the tech demo in action in this video and further below for additional details:

The use-case for this tech demo is about specific types of message in the Healthcare domain, which contains information about the Patient’s Admission-Discharge-Transfer (ADT); these types of messages, depending on a set of rules, need to be dispatched to the appropriate Kafka topic(s). This way, further systems and applications downstream can connect to these queues and consume only the relevant messages of interest.

Rules are formalised by the domain expert, in this case the Healthcare professional, who usually is not a software developer, but is knowledgeable and interested in the data contained in the message itself. This makes it a perfect fit for DMN, as the rules can be easily encoded using a DMN Decision Table, so that the decision about the destination queue names can easily be represented in a graphical format –one of the many key advantages of the DMN Standard.

In the example screenshot, the domain expert is interested with the first rule in all ADT messages coming only from the application called "MMS": these needs to be dispatched to the queue named "MMSAllADT" on Kafka. A second rule prescribe that ADT messages, again only from the application called "MMS", but pertaining only to a Patient’s discharge (reported as "A03" on HL7), shall also be dispatched to topic "MMSDischarges" on Kafka. For the purpose of simplicity of the demo, the table contains only 2 criteria; naturally this table can be easily extended to accomodate all the specific rules required, as the demonstration video also shows.

The Enterprise Integration Pattern diagram can be revisited, now annotated with the actual technologies used:

For the purpose of this tech demo, the incoming message format is exchanged using the HL7 v2 standard, and provided to a REST Endpoints, which may also serve as a CDS hook. Naturally this could be extended to use more modern standards in the same healthcare domain, such as HL7 FHIR.

It is important to reiterate this "Intelligent message routing" blueprint is very generic, therefore can be analogously architected on top of other business domains, such as FSI, IoT, etc.

Another advantage of this blueprint is the separation of concerns between the business domain logic and the integration code; that is the separation between the rules formalised graphically by means of a DMN Decision Table to decide on which topic name the message shall be queued into, and the actual Apache Camel code implementing the integration between the different systems. This way, the business expert can focus on just and only the decision to be modelled using DMN, while the software developer can offer a very extensible and robust intelligent routing capability.

Taking a little look under the hood, we highlight how the software developer using the Camel DSL only needs to write a minimal amount of code:

I have highlighted visually where the Camel DSL integrates the Drools DMN Engine for the evaluation of the necessary business logic for the rules, in order to decide which Kafka topic(s) the message needs to be queued into, or eventually forwarded to a catch-all queue. This allow to maintain the set of rules and decisions using just and only the DMN model, separately from integration code.

In the video, three scenarios are demonstrated:

• Scenario 1: Normal routing
  This is the standard operating mode.
• Scenario 2: Failure mode
  This is highlighting the catch-all queue for messages which are not matching any of the prescribed rules of the domain expert
• Scenario 3: Add new rule
  One of the most important aspects we highlighted is the ability to update the rules for message dispatching, without touching a single line of code, and this scenario demonstrates how this aspect is technically fully under the control of the domain expert. Naturally, in a real world scenario, several actors and procedures are involved before rolling-over the update on the Production system, going beyond the merely technical requirements; the demo focuses only on the pragmatical aspects.

Conclusions

In this technical demo, we have seen how we can easily integrate Drools DMN Engine and Apache Camel, to achieve Intelligent message routing on top of Apache Kafka. Specifically we have seen how to formalise the rules for the decision of the appropriate dispatch queue using DMN decision tables, so we can manage separately the business requirements from the actual integration code.

What do you think of this use-case and this demo? Don’t hesitate to let us know in the comments below!