Message queue blocking: when the error blocks the fix
A message rejection caused subsequent messages for the same patient to be blocked, including the messages needed to create the patient and encounter.
The problem
The inbound message processor for the hospital billing system was rejecting messages from an electronic document management system, because the required encounter did not yet exist. It then blocked subsequent messages for the same patient, including the messages needed to create the patient and encounter.
In other words, the message with the error was blocking the message needed to fix the error.
Why it was difficult
The integration crossed six components. Investigation also identified delays in the middleware path, but the precise source of that performance problem could not be conclusively established.
What I did
I was responsible for investigating the integration and recommending a fix. I traced the message flow and examined integration logs, message timing and the behaviour of the downstream systems.
I also assessed alternative solutions, including allowing the inbound message processor to create encounters, routing all messages through a common path, and introducing message-skipping capability.
The selected approach was to change the document management system message primary key from the patient identifier to the encounter number. This removed the dependency that allowed a failed message to block the messages needed to create the encounter.
Result
The permanent workaround removed the failure mechanism and there were no further incidents after implementation. A manual business as usual replay step remained for the small number of cases where the inbound message processor reported that an encounter could not be created.