About Decision Insight Messaging System

The Decision Insight Messaging System (DIMS) is a component of the Axway Decision Insight ecosystem.

The Decision Insight Messaging System is the only recommended way to send messages between two Decision Insight applications in a safe and reliable way.

Data is sent from one application to the other via messages .

Messages are the fundamental unit of processing used by Decision Insight Messaging System.

  • Messages consist of a key, a timestamp, and a value whose format (a Java Map) is suitable for Decision Insight absorption.
  • Messages are independent data payloads sent over the network.
  • Messages are stored into topics , which are logical collections of messages dedicated to a common processing.

How does it work?

Decision Insight deployments are expected to communicate with the Messaging System through data integrations routes which can act as

  • Producer – publishes messages to a topic.
  • Consumer – subscribes to topics from which it extracts messages.

High availability and durability

In order to avoid server failure or disruption, pending messages can be replicated over a  cluster of DIMS nodes.
This data redundancy is enforced by  partitions of topics over several nodes:

  • Partitions are redundantly distributed across the DIMS cluster.
  • Messages are written to one partition but then copied into another partition maintained on another DIMS node.

Horizontal scalability

In order to speed up message processing, consumers can group together and form a  consumer group .
DIMS guarantees that: 

  • A message is read only by a single consumer in the group.
  • All the messages that belong to the same topic are delivered to one of the registered consumers of a group.
  • Messages are evenly distributed to the available consumers (even if a new consumer joins the group later).
  • If multiple consumer groups subscribed to the same topic, they will process messages independently from each other.

Dynamic scaling

DIMS uses partitions as the unit of parallelism in consumer groups.
DIMS dynamically adjusts its infrastructure by balancing partitions assignment between the members of a consumer group :

  • When a new consumer joins a consumer group, it rebalances the association of partitions with the available consumers.
  • When an existing consumer goes offline, its assigned partition(s) will be reassigned to one or several consumers of its group.


That means the degree of parallelism within a consumer group is bounded by the number of partitions, but be careful with the size of the group:

  • The more partitions there are in a cluster, the higher the throughput you can achieve.
    Consequently, a consumer group can only have as many active consumers to a topic as there are partitions in that topic.
    Once the limit is met, new consumers won't receive any messages.
     
  • BUT
    • More partitions can increase end-to-end latency (the message processing duration).
    • More partitions can increase unavailability when a messaging server disappears.
    • More partitions can increase memory consumption for the producer as it needs to send messages to more partitions.
    • More partitions requires more open files handles.


Related Links