KQL Series – Interactive Analytics with Azure Data Explorer
source link: https://hybriddbablog.com/2022/03/31/kql-series-interactive-analytics-with-azure-data-explorer/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
KQL Series – Interactive Analytics with Azure Data Explorer
This blog is about how we can do interactive analytics with Azure Data Explorer to explore data with ad hoc, interactive, and lightning fast queries over small to extremely large volumes of data. This data exploration can be done using native Azure Data Explorer tools or alternative tools of your choice.
This is one of my favourite use cases of Azure Data Explorer as it shows the integration of it with the rest of the data platform ecosystem. You know being someone who loves all things Azure and being a Data Platform MVP means this is totally in my wheelhouse….
Architecture
Dataflow
- Raw structured, semi-structured, and unstructured (free text) data such as, any type of logs, business events, and user activities can be ingested into Azure Data Explorer from various sources. Ingest the data in streaming or batch mode using various methods.
- Ingest data into Azure Data Explorer with low-latency and high-throughput using its connectors for Azure Data Factory, Azure Event Hub, Azure IoT Hub, Kafka, and so on. Instead, ingest data through Azure Storage (Blob or ADLS Gen2), which uses Azure Event Grid and triggers the ingestion pipeline to Azure Data Explorer. You can also continuously export data to Azure Storage in compressed, partitioned parquet format and seamlessly query that data as detailed in continuous data export overview.
- Run interactive queries over small to extremely large volumes of data using native Azure Data Explorer tools or alternative tools of your choice. Azure Data Explorer provides many plugins and integrations with the rest of the data platform ecosystem. Use any of the following tools and integrations:
- For interactive analytics, use Azure Data Explorer Web UI, web client for Azure Data Explorer, or Kusto.Explorer, rich windows client for Azure Data Explorer.
- To connect to your Azure Data Explorer cluster, use Jupyter notebooks, Spark connector, any TDS-compliant SQL client, as well as JDBC and ODBC connections.
- To build new apps or integrate with existing apps or frameworks, use Azure Data Explorer REST APIs and SDKs available in different languages.
- Build near real-time analytics dashboards using Azure Data Explorer dashboards, Power BI, or Grafana.
- Enrich data running federated queries by combining data from SQL database and Azure Cosmos DB using Azure Data Explorer plugins.
Components
- Azure Event Hub: Fully managed, real-time data ingestion service that’s simple, trusted, and scalable.
- Azure IoT Hub: Managed service to enable bi-directional communication between IoT devices and Azure.
- Kafka on HDInsight: Easy, cost-effective, enterprise-grade service for open-source analytics with Apache Kafka.
- Azure Data Factory: Hybrid data integration service that simplifies ETL at scale.
- Azure Data Explorer: Fast, fully managed and highly scalable data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more.
- Azure Data Explorer Dashboards: Natively export Kusto queries that were explored in the Web UI to optimized dashboards.
- Azure Cosmos DB: Fully managed fast NoSQL database service for modern app development with open APIs for any scale.
- Azure SQL DB: Build apps that scale with the pace of your business with managed and intelligent SQL in the cloud.
I really would love to convince one of my clients to try something like this.
You can read a Microsoft client story here:
https://customers.microsoft.com/en-us/story/817285-episerver-professional-services-azure-sweden
It was really exciting reading what they are doing with Azure Data Explorer and of course KQL!!
#Yip.
Recommend
-
4
Microsoft 365 Developer Podcast Jeremy Thake and Paul Schaeflei...
-
40
The string data type 02/13/2020 2 minutes to read In this article The string data type represents a Unicode string. Kusto strings are enco...
-
4
MS Learn: Introduction to Azure Data Explorer MS Learn: Introduction to Azure Data Explorer ...
-
5
Editor's note: Jack Liu is a member of SAS Analytics Explorers, a SAS community that is dedicated to exploring analytics, sharing knowledge, having fun and helping SAS users in their careers. Members were recently invited to share their a...
-
4
KQL Series – quick intro to Azure Data Explorer Azure Data Explorer is a PaaS offering from Azure providing an end-to-end solution for data exploration.
-
7
KQL Series – create an Azure Data Explorer cluster I have written a lot about the magic of KQL and the brilliance of Azure Data Explorer. Now we should create a cluster of our own that hosts Azure Data Explo...
-
4
KQL Series – Big Data Analytics with Azure Data Explorer This blog pos illustrates how Azure Data Explorer and Azure Synapse Analytics complement each other for near real-time analytics and modern data wareho...
-
6
KQL Series – Security Monitoring with Azure Data Explorer This blog post demonstrates a hybrid end-to-end monitoring solution integrated with Microsoft Sentinel and Azure Monitor for ingesting streamed and ba...
-
4
KQL Series – ingesting data via Event Grid into Azure Data Explorer This blog post is about using event grid to ingest data into Azure Data Explorer and was a method I had to use with a client.
-
4
KQL Series – ingesting data into Azure Data Explorer from IoT Hub This blog post is rare because I am dedicating it to my good mate Bryn L...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK