What is real-time streaming data?
Streaming data is generated continuously by many different data sources, it is time-sensitive data that is delivered just after the collection, and there is no delay in the timeliness of the data provided.
We can collect streaming data from various data sources such as:
- Log files generated by mobile or web applications.
- Sales data from e-commerce portals.
- Live feed from social networks.
- Data from the sensor devices.
- Trading data from financial institutions.
- Data from geospatial services.
- and much more.
Leverage real-time streaming data using an interactive real-time dashboard to make quicker decisions
An interactive real-time dashboard displays this streaming data with the help of interactive data visualisation. The real-time streaming data is quite useful to make quicker and better-informed decisions.
Here are some of the examples where we can see the power of real-time streaming data:
- The weather control system uses real-time data to predict the weather accurately.
- Geospatial analytics for driverless vehicles and fleet management.
- Marketing agencies can monitor live media campaigns and adjust the budget accordingly.
- Power grid monitoring system can generate alerts in real-time when certain thresholds have reached.
- Manage home appliances and other devices that generate data from the sensor.
- Financial institutions can adjust the settings in the customer portfolio in real-time if needed (i.e. sale and purchase when a stock value reached).
- Improve operational efficiency in retail and customer service.
- Detect anomalies and frauds in real-time for retail store and website.
- In clinical healthcare, it provides the ability to monitor patient safety, personalise patient results, and assess clinical risk as well as reduce patient readmission.
- Point of Sale (POS) data for inventory control and anomaly detection.
- and much more.
Why Microsoft Power BI real-time streaming?
Power BI allows you to create interactive dashboards using real-time data, with Power BI it is easy to stream data and update dashboard in real-time.
Streaming data source could be sensors, service usage metrics, social media source, e-commerce portal and anything else which comprises time-sensitive data and can be controlled or transmitted.
There are three types of real-time datasets in Power BI that you can use to display and get interactive reports and hidden insights:
- With Push dataset:
- Data is pushed into the Microsoft Power BI Service.
- A new database in the service is created by Power BI from that data.
- Reports can be created as soon as the dataset is created.
- With Streaming dataset:
- Data is also pushed into Power BI, but there is no underlying database service.
- It helps to quickly display real-time data using a streaming dataset source by adding a tile and setting up custom streaming data.
- This is used for cases where it is critical to minimise the latency between when data is pushed and when it is visualised.
- With PubNub streaming dataset:
- The Power BI web client uses the PubNub SDK to read an existing PubNub data stream and configure it as the source.
- There is no data stored by the Power BI service because this call is made from the web client directly.
Know more about Streaming dataset matrix:
The table below describes the three types of datasets for real-time streaming and lists the capabilities and limitations of each.
Image Source - https://tinyurl.com/y9vghcjo
In the previous section, we described the three main types of real-time datasets that can be used for real-time streaming in PowerBI. Next sections explain how to push data to them.
There are three primary ways you can push data into a dataset:
- Using the Power BI REST APIs - use Power BI REST APIs to push data into data sets.
- Using the Streaming Dataset UI - use the API approach to push data.
- Using Azure Stream Analytics - add Power BI as an output within Azure Stream Analytics (ASA), and then visualise those data streams in the Power BI service in real-time.
Let's discuss further Azure Stream Analytics.
What is Azure Stream Analytics?
Azure Stream Analytics is a serverless complex event processing engine and a real-time analytics service. It is designed to analyse and process high volumes of streaming data from multiple sources(such as devices, sensors, clickstreams, web sites, social media feed, and other applications) simultaneously.
How does Azure Stream Analytics work?
Azure Stream Analytics uses the Power BI REST APIs to create its output data stream to Power BI, with defaultMode set to push streaming, which results in a dataset that can take advantage of both push and streaming.
An Azure Stream Analytics job consists of:
- An input
- SQL-based query (can be used to filter, sort, aggregate easily, and join streaming data over a period of time)
- An output
Azure Stream Analytics ingests data from:
- Azure Event Hubs
- Azure IoT Hub
- Azure Blob Storage
Image Source - https://tinyurl.com/y78vv5jq
The above image shows how data is sent to Stream Analytics, analysed (using reference data and real-time scoring), and sent for other actions like storage or presentation.
Why choose Azure Stream Analytics for real-time data processing?
- Fully integrated - we can build robust pipelines with few clicks because it is fully integrated with the Azure ecosystem.
- Intelligent edge - This allows for lower bandwidth costs and ability of a system to function even with intermittent connectivity as most data becomes useless just seconds after it’s generated.
- Easily leverage the power of Machine Learning (ML) - Azure Stream Analytics offers built-in support for commonly used scenarios such as anomaly detection, which helps reduce the complexity associated with building and deploying an ML model.
- Lower your cost of innovation - You only pay for the number of streaming units you consume to process your data streams, and there are zero upfront costs.
- Best-in-class financially backed SLA by the minute - Stream Analytics guarantees event processing with a 99.9% availability as Stream Analytics processes millions of events every second and can deliver results with low latency.
- Scale instantly - You can instantly scale up or scale-out the processing power from one to hundreds of streaming units for any job because Stream Analytics is a fully managed serverless (PaaS) offering on Azure.
- Reliable - You never have to worry about your events getting dropped because it has built-in recovery capabilities, in any case, the delivery of an event fails.
To know more about Azure Stream Analytics -