In this post, we will cover the following:
- An overview of the Azure Service Bus.
- The integration of Talend jobs and camel routes with Azure service bus Queues.
- Real-time integration challenges of Talend with Azure service bus Queues and Topics.
- Our solution and approach.
What is Azure Service Bus?
Azure Service Bus is a cloud enterprise messaging service that can be used to integrate applications and systems with other applications and systems. And in our case, it acts as a messaging backbone for systems running in the cloud or other platforms.
Data is transferred between different applications and services using messages. A message is in a standard format, and the payload can contain any data structure such as JSON, XML, and others.
The Azure Service Bus offers a reliable and secure platform for the asynchronous transfer of data.
The Real-time challenges to connect Talend with Azure Service Bus Queue/Topics.
- Large messages.
Azure Service Bus is offered in three tiers, Basic, Standard, and Premium.
For Basic and Standard tier service bus queues, the message or batch size is limited to 256 KB, whilst premium tier queues allow 1 MB size messages.
In the real world, the business has some scenarios where the message size is greater than 256 KB, and the message ends up being rejected.
One possible solution approach would be:
- Use compression to reduce the message size before sending it to the queue or topic.
- Receive messages through Talend job or route and decompress it.
2. Client-side batching, a feature not available in the default Talend component.
The client-side batching is a powerful feature to increase the throughput of a queue/topic where multiple messages can be batched together before they are sent to the queue or topic.
Sending messages individually is pretty inefficient compared to batching them together.
Unfortunately, this feature is not available in the Talend connectors, and we had to step up to create a custom component to fulfil the client's requirements.
A powerful feature that we have introduced in the custom component is intelligent batching. It reads the incoming message size and calculates the bytes before storing it in the batch to utilise the maximum capacity of the batch. Once a batch reaches its full capacity, the batch is then sent to the queue or topic.
Please check out the video available at the end of the post to see the exciting work we have done here at Onepoint.
3. Unable to connect to an Azure topic from Talend Jobs
A Service Bus Topic implements publish and subscribe semantics. When you publish a message, it goes to all the interested subscribers - so zero to many subscribers will receive a copy of the message. While with a queue, a single message will be received by exactly one consumer.
The functionality to connect with Topics is now available in the Onepoint's custom component.