Process Mining in Realtime, what is the Use-Case?
An interesting question that I get asked quite often. What’s in for me, except a nice process view of my business system?
Why we created it
I’ve spent my whole professional life in messaging. For those who don’t know me, I’m the creator of SwiftMQ, an enterprise messaging system, more than 20 years ago, and also the inventor of Flow Director, a no-code messaging platform based on SwiftMQ.
If you are in messaging, you think along these lines: messages, queues, topics, backlogs, connectivity, scalability. You don’t think about the business system that uses messaging. And precisely, this missing link is why we created Realtime Process Mining.
When I first saw a process model, I noticed that we need this. It adds the missing semantic to the picture. You can immediately see what’s going on and where and why it sticks. Having this information always (real-time) at hand provides the ability to act further than I could before.
A simple Example
The above process view is from a warehouse simulation in Flow Director. It simulates an online store with the actual store, racks, products, customers, producers, shippers, accounting.
When an order is received, the store checks if the ordered product is available in the rack; if not, the store reorders the product, and the order waits for the delivery.
But 28 orders repeat that multiple times. Why is that?
Well, in the simulation, the reason is that always 100 products are delivered. If more products are needed, they have to wait on the next delivery. But in the real world, either the racks are too small, or the producer needs to deliver more items, right? That’s what we wanted to find out.
Automation
Consider the dynamic process model as your business application’s brain as it contains all states and data of your business process. To automate your business based on this data, you can act on events that are produced from the model.
For example:
When a business object reaches Shipment Ready, inform the delivery service to schedule a pickup. When that happens at Shipment Pickup, notify the customer when it will be delivered.
Simple, and you get that for free when you use Realtime Process Mining.
Definition: Log-based vs. Realtime Process Mining
Log-based
Most process mining tools on the market use event logs and apply specialized data mining algorithms to identify trends, patterns, and details recorded by an information system.
The input data streams used in log-based process mining are unqualified. The major challenge in using these tools is the time-consuming task of qualifying data by parsing the logs, detecting events and patterns, and assigning it to the correct business activities.
The result is a business process model used to identify weak paths to optimize the process. Log-based process mining usually works in a time-shifted manner.
While log-based process mining tools may understand various input formats, there is often an additional step required to gather this data from the original systems before processing it.
Realtime
Realtime process mining doesn’t use event logs but instead intercepts the message flows of business systems. The result is a live process model generated on the fly, used to monitor the business processes as they occur and take appropriate actions when they deviate from the optimal path (aka Happy Path).
The input data streams are qualified and intercepted from queues and topics that already have a semantic relationship to business activities, i.e., messages received on a topic order_entry can be directly assigned to a business activity Order Received. All required data fields are usually already part of the message, i.e., the unique key that identifies a business object. If not, they could collect from other sources, i.e., a database, before the messages pass into the process mining.
Therefore realtime process mining mitigates the need for time-consuming log parsing, event, and pattern detection task and can instead directly generate the process model.
Is “Mining” the proper Term for the Realtime Variant?
Sometimes I register confusion from people who come from the business side, such as business analysts. When they hear Process Mining, they expect a log-based tool but get something where the result looks like that (process diagram) but does not use logs.
Mining sounds like digging for something. This term fits quite well for log-based tools because that’s what their main task is. But to explore something is not what the realtime variant does. Instead, it intercepts. But would Realtime Process Interception reduce that confusion? I don’t know. Please comment on it! We are free to change that term if we find a better one.
Update: After gathering some community feedback, we decided to rename our apps to Realtime Process Monitoring.
Who needs this?
My opinion as a messaging guy is that this must be part of, or you must be able to add this to any messaging product — a commodity tool just like your beloved admin console.
Why do I have a massive backlog in some queue every three days? Do NOT just add resources to handle that peak but check your process diagram, contact the business, and solve the root cause.
Realtime Process Mining Solutions
We plan or already offer realtime process mining apps for any major messaging system and related service:
They are all provided as Flow Director Smart Apps, which means: Web-based, runs on the Flow Director platform, single setup on the first use, and that’s it. After that, they just work. No configuration hassle or software installation.
