How to Create an AI Social Listening Tool using Azure and the Twitter API Part 1...

How to Create an AI Social Listening Tool using Azure and the Twitter API Part 1Skip to the content

This is the first in a 4-part series that will show you how to build your own social media listening tool.

Azure AI is used to surface actionable insights in Twitter data.

With millions of tweets being sent per minute, Twitter can be a valuable source of what the world is talking about.

Users of the platform frequently share what’s important to them, to catch up with the news or simply just to scroll and see what’s going on in their social graph.

From a commercial perspective, listening to conversations on Twitter can help businesses or organizations glean all sorts of real-time insights such as:

• how their product, brand or service is being perceived
• topics or hashtags a company is being associated with
• where users are located

Twitter data can also be used to identify brand advocates, potential “influencers” and help form part of your marketing strategy.

Prerequisites

To build this you’ll need:

• Azure Subscription
• Visual Studio
• Developer Account with Twitter API

You can get a trial Azure subscription here and create a Twitter developer account here.

Download a free copy of Visual Studio here.

Components and Service

Several components and services are used to form this solution:

• Azure Blob Storage – for storing data fetched from the Twitter and any data that is processed and archived by the solution
• Azure Functions – to encapsulate and orchestrate the custom logic that is needed to fetch, process an archive data
• Azure Cognitive Services for Language – AI service to processing and surface insights in text
• Twitter API – for integrating with Twitter data
• Social Opinion SDK – to accelerate development and make simple to use the Twitter API in .NET

Each of these components, tools, and services interact with each other to fetch, store, and process Twitter data in Azure.

Architecture

The following diagram illustrates each component that forms the solution:

In the above diagram, you can see two Azure functions:

• TwitterFunction
• TextAnalyticsFunction

Three blob storage containers are also used:

• raw-tweets
• processed-tweets
• archived-tweets

TwitterFunction

This leverages the Social Opinion SDK to fetch content from Twitter.  This function is invoked using a timer trigger that executes every few minutes.  Data is received in JSON format and automatically uploaded to a blob storage container in Azure called raw-tweets.

TextAnalyticsFunction

This is responsible for fetching tweets from the storage container raw-tweets and using Azure Cognitive Services for Language to surface analytics insights.

Insights generated by Cognitive Services for Language are created in the container processed-tweets.

When tweets that are processed by this function are moved to a container called archived-tweets and the original data from raw-tweets is deleted.  This ensures that records are only processed once.

Inputs and Outputs

Prior to build a solution, it’s helpful to know the format, structure, and have an example of the input and output data you’re working with.

Twitter Playground

The Twitter Playground is a free, low friction tool that lets you quickly get a feel for input parameters for a given endpoint.  You can also examine the JSON payloads directly from your web browser.

You can see this in action here:

In the above example, the RecentSearch API is being used to search for 10 tweets that contain the text Microsoft.

The JSON response contains the following:

Note: the entities and context_annotations nodes have been hidden due the length of the content.

Cognitive Services for Language

Cognitive Services for language will return the following information:

• sentiment
• key phrases
• entities

You can see an example of the JSON here:

With an overview of the architecture, components, and examples of the data defined, the first Azure Function can be created.  This will be covered in Part 2 of the series.

Summary

This is the first instalment in the How to Create a Social Listening Tool using Azure.

In this instalment we’ve covered the architecture, components, main processes flows, and looked at the data we’ll be working with.

In Part 2 we’ll build the Azure Function that fetches data using the Twitter API and Social Opinion API.