5 Twitter Batch Operations with a few Lines of Javascript

Twitter is a great tool for creating social presence. However, using the twitter app GUI for batch operations is tedious and time consuming. APIs makes it possible to write programs that manipulate data in batch. . The dstwitter module is a dstools extension that makes it easy to execute the twitter API on a collection of data elements such as twitter users or tweets. It relies heavily on the method chaining pattern, similarly to jQuery. In this tutorial we will show how to:

• Get a list of followers and friends (friends are users you follow)
• Show a report with followers count, friends count, followers you don’t follow back and users you follow that don’t follow you back
• Follow all users that follow you (that you do not already follow)
• Unfollow all users you follow that don’t follow you back
• Get word cloud for your followers bios (description part of the profile)
• Search for users tweeting about a specific topic

Setting the ground — Creating a twitter application

In order to use the twitter API you will first need to create a twitter application. Once you have a twitter application, you can use its credential to access the twitter API.

Head on to apps.twitter.com and click on the “Create new app” button.

Fill in the name, description and website (you can just type in a placeholder if you wish). In this example we are just using the app for our own use. If you are creating a twitter app handling other users, make sure the website is somewhere your users can learn about the app. Don’t forget to start the website URL with http or https. Click on the button to create the app and voila, you’ve got yourself an app.

Now we need to get the credentials required to activate the twitter API. Choose the keys and access tokens tab. This will show your apps consumer key and consumer secret. You will need them for connecting with the twitter API. We will be accessing the API on your own behalf so we will need a user access token. The easiest way to get a user token is on the app page, using the “Create my access token” button. Just click it and you will get a user access token and access token secret. These will also be required to perform API requests.

The NPM Libraries

In this tutorial we will use two npm modules: dstools and dstwitter. dstools is an npm module used to easily manipulate data collections such as twitter users and tweets. It relies on method chaining, a feature I find very useful when handling data that needs to be manipulated in several steps. This library integrates well with Jupyter notebooks. My previous post explains about using Jupyter notebooks with Javascript. However, it is possible to use the package with plain Javascript files or REPL.

The dstwitter package is an addon, providing the twitter API functionality in a convenient way. An important feature of the library is its ability to work with twitter’s API restrictions such as maximum number of results per page and rate limits. With the twitter API, each API endpoint is limited in the number of items each specific API call can return and the number of API calls the application can perform every 15 minute. The dstwitter module can break a request for many items into several API calls and waits for 15 minutes if the app reached its quota.

Here is the Code

Coders like to see the code. So without further ado, here is the code (with some annotations). The code is edited so it can be copy/pasted into node REPL. Hence the trailing dots.

Initialize the environment

First, you will need to load the library and initialize it with the credentials required for the twitter API

require('dstwitter');//initialize dstwitter modulelet start = require('dstools').Collection().context('twitter',{//load twitter API credentials to context access_token_key:'ACCESS_TOKEN_KEY', access_token_secret : 'ACCESS_TOKEN_SECRET', consumer_key: 'CONSUMER_KEY', consumer_secret: 'CONSUMER_TOKEN'});

Show a report with followers count, friends count, followers you don’t follow back and users you follow that don’t follow you back

The twitter app shows us the number of followers and the number of users you follow. The interesting part is how many of the users you follow do not follow you back and how many of your followers you do not follow.

let followers, following;start.followersIDs(ME).//load followersdo((output)=>followers=output).//set followers variablefollowingIDs(ME).//load followingdo((output)=>following=output).//set following variabledo(()=>{ console.log(‘followers:’,followers.length); console.log(‘I follow:’,following.length); console.log(“Follow me that I don’t follow back:”, Collection(followers).drop(following).count());//use Collection function to create dstools collection and use its functions such as `drop` console.log(“Don’t follow me back:”, Collection(following).drop(followers).count());});

Follow all users that follow you (that you do not already follow)

Some twitter users believe in reciprocal following where you follow anybody who follows you. The following code achieves this.

start.collection(followers).  drop(following).  //ignore users I am already following follow(); //follow them

Unfollow all users you follow that don’t follow you back

This is just the opposite side of reciprocal friendship in which you do not want to follow anybody who doesn’t follow you back. The list of exceptions are users you want to follow anyway. These can be important twitter users that you want to make sure you follow even though they do not follow you back.

exceptions = ['hackernoon','JavaScriptDaily'];following. //start with all users I follow drop(followers).  //drop users that follow me twitterUser(). //translate ids to user objects column('screen_name').//translate user objects to screen_names drop(exceptions).//remove exception screen names unfollow(); //unfollow whatever is left

Get word cloud for your followers bio (description part of the profile)

Know your audience is the #1 rule of communication. When you have thousands of followers, knowing you audience is a challenge. A possible shortcut is to view a word cloud of their bios, summarizing the bios of all your users.

start.followers(ME,10000).column('description').toLowerCase().terms().dropStopwords('term').sortDesc('count').head(50).wordCloud('term','count').save('word-cloud.html');

Search for users tweeting about a specific topic

“Who to follow” is a question any twitter user encounters. If you are interested in “vue”, which twitter users should you follow? With the following code we search for tweets about “vue”, find which users wrote the tweets, load their user profiles and show the highlights so we can choose who to follow:

start.searchTweets('javascript react',10000,{result_type:'mixed'}).filter((tweet)=>!tweet.text.startsWith('RT @')). //filter out retweetsmap((tweet)=>tweet.user). //get users of the tweetsgroupBy('screen_name').sortDesc((item)=>item.data[0].followers_count).//order by followershead(20).do((users)=>{ console.log('screen_name, tweets in search, followers, following, statuses, description'); users.forEach((user)=>console.log(  user.key + ':', //screen_name  user.data.length,  user.data[0].followers_count,  user.data[0].favourites_count,  user.data[0].statuses_count,  user.data[0].description));});

Final Words

Twitter is an endless bucket of data. Using the twitter API and marrying it with data analysis tools can generate significant insights. This post describes the basics of using the API with the dstwitter package. In future posts I will demonstrate using data tools to analyze the twitter data. Meanwhile, you can have a look at my post about data analysis using Javascript

Data Science for Javascript Developers

5 Twitter Batch Operations with a few Lines of Javascript was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.