Skip to content

{rtweet} helpers for automating large or time-consuming downloads

License

Unknown, MIT licenses found

Licenses found

Unknown
LICENSE
MIT
LICENSE.md
Notifications You must be signed in to change notification settings

mkearney/rtweet.download

Repository files navigation

rtweet.download

CRAN status Lifecycle: experimental

Automating the collection of Twitter data (via {rtweet})

Installation

You can install the released version of {rtweet.download} from CRAN with:

install.packages("rtweet.download")

And the development version from GitHub with:

# install.packages("remotes")
remotes::install_github("mkearney/rtweet.download")

Friends IDs

Twitter’s "friends/ids" API endpoint is rate limited to 15 requests (or the friend IDs of 15* accounts) per 15 minutes. So while a single call using rtweet::get_friends() can retrieve the friend IDs of up to 15 users, a single call using rtweet.download::get_friends_download() can retrieve the friend IDs of hundreds or even thousands of users!

API Feature Value
  Endpoint "friends/ids"
  Rate limit (per 15 min.) 15
  Friends per request 5000*
R Package Function
  {rtweet} get_friends()
  {rtweet.download} get_friends_download()

The example below uses get_friends_download() to automate the collection of friend (accounts followed by) IDs of users on @Teradata’s list of data science influencers.

## get members on data science influencers influence
data_sci_influencers <- rtweet::lists_members(
  owner_user = "Teradata", slug = "data-science-influencers"
)

## download friend IDs for each user
fds <- get_friends_download(data_sci_influencers$screen_name)

## preview data
head(fds)

Users data

Twitter’s "users/lookup" API endpoint is rate limited to 900 requests (or 90,000 users) per 15 minutes. So while a single call using rtweet::lookup_users() can retrieve data on up to 90,000 users, a single call using rtweet.download::lookup_users_download() can collect data on hundreds of thousands or even millions of users!

API Feature Value
  Endpoint "users/lookup"
  Rate limit (per 15 min.) 900
  Users per request 100
R Package Function
  {rtweet} lookup_users()
  {rtweet.download} lookup_users_download()

The example below uses lookup_users_download() to automate data collection for the previously collected accounts followed by data science influencers.

## download users data
fds_data <- lookup_users_download(fds$user_id)

## preview data
head(fds)

Notes

* The "friends/ids" endpoint returns the up to 5,000 friend IDs of a single user, so 15 requests can only return all the friend IDs of 15 users if all 15 of those users follow 5,000 or fewer accounts. To retrieve all the friend IDs for users following more than 5,000 accounts, multiple requests (friends_count / 5,000) are required.

About

{rtweet} helpers for automating large or time-consuming downloads

Topics

Resources

License

Unknown, MIT licenses found

Licenses found

Unknown
LICENSE
MIT
LICENSE.md

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages