Trying to fetch Twitter data using SearchTwitter()?
Got such a message on your console?
“500 tweets were requested but the API can only return 2″
The question is extremely centric to new R users, and constantly resurfaces.
So far, you might have been tempted and suggested to tweak internal parameters to get more results.
Many would’ve claimed that their code worked well after tweaking their code, but, you by now are having a hard time reproducing the same results.
And you still get the same error again and again,
doRppAPICall(“search/tweets”, n, params = params, retryOnRateLimit = retryOnRateLimit
Now, it gets frustrating!
But don’t worry, you probably aren’t doing anything wrong with your code. You missed one small detail.
Twitter Search API!
The Search API doesn’t provides you an exhaustive set of search results, and is quite weak when it comes to providing results.
With the Twitter Search API you get a sample, not the entire results.
To properly get Twitter search results, you can either look for 3rd party providers,
You can either use the Twitter Streaming API.
The proper way to set up a Twitter mining system using Twitter Stream API includes:
1. Creating a database to store tweets
2. Regularly update them
Another benefit of Streaming API over Search API is that the Streaming API brings almost real-time data.
I am not getting into the specifics of Twitter Streaming API right now, but, getting access to Twitter Streaming API is simple and similar Twitter Search API.
Some of the packages that you need to perform data mining operations using Twitter Streaming APIs are StreamR, ROAuth, RCurl and Rjsonio.
In a subsequent post I will walk you through fetching data using Twitter Streaming API.
Meanwhile, if you have an experience of yours working with Twitter Search API that you want to share, please feel free to drop me a comment.