What Type Of Data Can You Get From Twitter?

With over 320 million monthly active users, Twitter continues to be a rich and vast resource for data on the Web. Twitter provides direct access to Tweets and user data in a number of ways.

What can you actually collect from Twitter and how can it be used? In this post, we focus on uncovering the exact data that is stored in a user profile and Tweet and how it can be used for analysis.

How Do You Access Twitter Data?

There are many ways to access Twitter data. The two we’re going to cover are:

1. Use Twitter’s Search API for developers
2. Purchase Firehose access through providers like GNIP

The first way to access Twitter data is through their public API. Making calls using the Twitter API is similar to using Twitter’s search feature but allows you to get this data in an automated fashion.

If you need access to data beyond what’s possible with Twitter’s search and real-time APIs, Twitter also offers Firehose. This API is near real-time, guarantees 100% uptime and provides direct access to data about individual users and profiles. This incredible data stream is only available to a limited number of distributors.

BrightPlanet has direct access to Twitter data through GNIP, Twitter’s own enterprise API platform. This comes with guaranteed delivery of 100% of tweets, and a contract in place ensuring continued access even if Twitter restricts access to their Search API.

What Data Does Twitter Provide About Users?

The Twitter API provides information about specific users based on searching specific Twitter handles or by using a keyword search to find users. For example, you could search for “Doctor” to identify potential doctors on Twitter.  This request would return results similar to searching “Doctor” in their search feature.

In its search results, Twitter gives a lot of information about individual users that you can use even if a profile has been protected (preventing other users from viewing it online or in Twitter apps). Using Twitter’s API, you get back what’s called a JSON output that includes a large amount of information on the user.

That JSON output is shown to the right. Much of the data in the JSON output from Twitter isn’t necessarily important so we’ve highlighted the most important data returned about a user below by showing the most commonly used data points of BrightPlanet Data Acquisition Engineer Jamie Martin’s Twitter Account.

 

Name: Jamie Martin
Location: Sioux Falls, SD
Created Date: December 8^th, 2010
Number of Favorites: 121
URL: jamievmartin.com
Profile Image URL: http://pbs.twimg.com/profile_images/506823466541002754/m1EjM93a.jpeg
Language: English
Protected: False
Description: Data Acquisition Engineer@BrightPlanet, Passionate about Photography, Adventures, and doing cool things with Data. 2011@AugustanaSD grad.
Verified: False
Tweet Count: 252
Time Zone: Central Time (US & Canada)

Post Data: What’s In A Tweet?

Twitter also gives you direct access to individual Tweets. Users typically access Tweets by either specifying a user to collect Tweets from, or by searching across the service for a specific keyword, returning matching results from many users. Here’s an example of what we see when looking at an individual Tweet via the Search API:

Text: We look at anonymizing #bigdata and harvesting competitive intelligence for your business: http://hubs.ly/H027K6j0 
URL: https://twitter.com/brightplanet/status/701802005866668033
Retweet Count: 3
Date: 2/22/2016 8:13AM
Source: HubSpot
Favorite Count: 1
Hash Tags: #bigdata
Mentioned Users: Null
In Reply to Screen Name: Null
Geo Data: Null

How Can Twitter Data Be Used?

We covered the type of data that you can get from Twitter so that you can better understand how it’s used and what can be asked of it. For example, since source (the device that initiated a Tweet) is included, you can see when users will be sitting at a physical computer or when they’ll be using Twitter on their handhelds. You can also get an understanding of when users switched to a different phone and understand their patterns of life.

The graph below shows Twitter data collected from an individual. Users can quickly zoom in and see that this individual most often Tweets in the evening and most commonly Tweets directly from his iPhone.

 

Tap into the Internet

The Deep Web consists of thousands of sources in addition to Twitter. Want to learn how you can begin tapping into other Web sources, including Twitter? Sign up to request to chat with a Data Acquisition Engineer below!

//