Application for Twitter Developer Account

[atomicules]

Just creating an issue for record purposes...

I received an email from Twitter on 01-Oct-2018:

Subject: Action required: you must request continued elevated POST access

Twitter
https://twitter.com

=

A few months ago, we announced that we are making changes to increase accountability on our developer platform and reduce the distribution of spam through our APIs. Last month, we extended
the previous deadline to enforce new default app-level rate limits for common POST endpoints to October 22nd. This date is quickly approaching.
https://blog.twitter.com/developer/en_us/topics/tools/2018/new-developer-requirements-to-protect-our-platform.html

You are currently using our POST endpoints near or above the new limits. To ensure that your apps will not be impacted when we make these changes, please complete the following steps.

If you have not done so already, go to developer.twitter.com/apply and submit a developer account application. Please be as specific and detailed as possible to avoid additional requests for
information and ensure a smooth review. Be sure to confirm your email address at the end of the application flow.
https://developer.twitter.com/en/apply/

While waiting for your developer account application to be reviewed, please complete the remaining steps:

For each app that needs elevated POST access:

1. Submit a developer account application at developer.twitter.com, and
2. Request elevated access to POST endpoints
   policies with helping developers get started easily. Regards,

I'm pretty slow at getting around to things (sorry)... I have just done part 1) of this; We already had a developer account, this is the newer style developer account. Will follow-up with 2) when they've reviewed it...

...I'm actually unsure why we need "elevated POST access", but it seems we do.

[atomicules]

Ugh, Twitter, as you can well imagine, are being a big bunch of dicks awkward about this.

[funnelfiasco]

I wonder if it's worth just throwing away our old creds and create new. It could be that there are still old installs using the version that included a default application key. I haven't received these notices for the other two accounts I use with OysTTYer.

[atomicules]

For the record these are the emails

1 - My initial email - basically filling in the same info they'd asked for on the web form

Hi,

sending this information again as requested.

On 15-Oct-2018 16:38:45, developer-accounts@twitter.com wrote:
[Inline image URL : https://ea.twimg.com/email/self_serve/media/icon_twitter_blue.png] https://twitter.com

Thanks for applying for access!
 

In order to complete our review of your application, we need
additional information about your use case. The most common types of
information that can help expedite our review include:
 

The core use case, intent, or business purpose for your use of the
Twitter APIs

Oysttyer is the official fork of the now defunct TTYtter from
Floodgap: an interactive console text-based command-line Twitter
client and Perl platform (whew!) It's primary functionality is to
allow users to follow their Twitter timeline and interact with it
(Tweet, reply, retweet, like, etc) in environments where no official
client is suitable. I.e. on older machines where web-browsers don't
work or can't load the official website, on machines without a
graphical environment, etc.

If you intend to analyze Tweets, Twitter users, or their content,
share details about the analyses you plan to conduct and the methods
or techniques

Oysttyer does not analyse Tweets, users or their content.

If your use involves Tweeting, Retweeting, or liking content, share
how you will interact with Twitter users or their content

Oysttyer, being a command line client, does not independently Tweet,
Rewteet or like content. However, it does allow people to Tweet,
Retweet and like content in their timeline via itself.

If you’ll display Twitter content off of Twitter, explain how and
where Tweets and Twitter content will be displayed to users of your
product or service, including whether Tweets and Twitter content will
be displayed at row level or aggregated

As a text based command line client it displays people's timelines in
chronological order. An example from my (@atomicules) timeline is
shown below:

[EXAMPLE OF MY TIMELINE]

2 - Their response

Thank you for your response.

At this time, we still do not have enough specific information about your intended use case to complete review of your application. As a reminder, we previously requested additional information about:

The core use case, intent, or business purpose for your use of the Twitter APIs

Note that “business purpose” in this context includes uses not necessarily connected to a commercial business. We require information about the problem, user story, or overall goal your use of Twitter content is intended to address.

If you are a student, learning to code, or just getting started with the Twitter APIs, please provide details about potential projects or areas of focus.

If you intend to analyze Tweets, Twitter users, or their content, share details about the analyses you plan to conduct and the methods or techniques

Note that “analyze” in this context includes any form of processing performed on Twitter content. Please provide as detailed and exhaustive an explanation as possible of your intended use case.

If your use involves Tweeting, Retweeting, or liking content, share how you will interact with Twitter users or their content.

If you’ll display Twitter content off of Twitter, explain how and where Tweets and Twitter content will be displayed to users of your product or service, including whether Tweets and Twitter
content will be displayed at row level or aggregated

Where possible, please share links to illustrations and/or sample work products. Note that we can’t view attachments.

To provide this information to Twitter, reply to this email. Failure to provide the necessary information will result in the rejection of your application.

Thank you for your interest in building on Twitter.

3 - My reply to that response

Hi,

On 16-Oct-2018 13:42:02, developer-accounts@twitter.com wrote:

Thank you for your response.

At this time, we still do not have enough specific information about
your intended use case to complete review of your application. As a
reminder, we previously requested additional information about:

Ok, thanks. I shall try to provide enough detail this time. Sorry for
the delay in replying.

On the "elevated POST access" note: We provide Oysttyer as source code
and thus the majority of users register their own API keys to use. We
can therefore accept revocation of the existing keys/credentials if
that is of any help?

The core use case, intent, or business purpose for your use of the
Twitter APIs

Note that “business purpose” in this context includes uses not
necessarily connected to a commercial business. We require
information about the problem, user story, or overall goal your use
of Twitter content is intended to address.

The main aim of this project is to provide a text based command line
client to Twitter. I.e. for those that cannot use an official mobile
client or the website - primarily those on older hardware or running
without X11. Therefore mostly targeting *nix operating systems.

If you are a student, learning to code, or just getting started with
the Twitter APIs, please provide details about potential projects or
areas of focus.

Not a student. It's an opensource project so students may be
involved.

https://github.com/oysttyer/oysttyer

If you intend to analyze Tweets, Twitter users, or their content,
share details about the analyses you plan to conduct and the methods
or techniques

Note that “analyze” in this context includes any form of processing
performed on Twitter content. Please provide as detailed and
exhaustive an explanation as possible of your intended use case.

Ok, in this case the tweets are "analysed" - i.e. the tweets are
retrieved via the API and "processed" so they can be printed out on
the screen as text.

Here's an instance of processing of data to get media URLS from
tweets:

oysttyer/oysttyer.pl

Lines 2959 to 2992 in 74b4946

	foreach my $entities (qw(entities extended_entities)) {
	$entitiesprint = 1;
	foreach $type (qw(media urls)) {
	my $array = $hash->{$entities}->{$type};
	next if (!defined($array) || ref($array) ne 'ARRAY');
	foreach $entry (@{ $array }) {
	next if (!defined($entry) || ref($entry) ne 'HASH');
	next if (!length($entry->{'url'}) ||
	(!length($entry->{'expanded_url'}) &&
	!length($entry->{'media_url'})));
	if ($entitiesprint) {
	print $stdout "$entities:\n";
	$entitiesprint = 0;
	}
	my $u1 = &descape($entry->{'url'});
	if (defined($entry->{'video_info'})) {
	foreach $variant (@{ $entry->{'video_info'}->{'variants'} }) {
	my $videourl = &descape($variant->{'url'});
	print $stdout "$u1 => $videourl\n";
	}
	}
	else {
	my $u2 = &descape($entry->{'expanded_url'});
	my $u3 = &descape($entry->{'media_url'});
	my $u4 = &descape($entry->{'media_url_https'});
	$u2 = $u4 || $u3 || $u2;
	print $stdout "$u1 => $u2\n";
	}
	#To stay compliant with TOS we can only open the tco.
	$urlshort = $u1;
	$didprint++;
	}
	}
	}

A instance of processing of the API data to construct a "tweet"

oysttyer/oysttyer.pl

Lines 4354 to 4381 in 74b4946

	$key = substr(( join "\0", $ms, $key->{'id_str'},
	$key->{'in_reply_to_status_id_str'},
	$key->{'quoted_status_id_str'},
	$key->{'quoted_status'}->{'text'},
	$key->{'quoted_status'}->{'full_text'},
	$key->{'quoted_status'}->{'extended_tweet'}->{'full_text'},
	$key->{'retweeted_status'}->{'id_str'},
	$key->{'retweeted_status'}->{'text'},
	$key->{'retweeted_status'}->{'full_text'},
	$key->{'retweeted_status'}->{'extended_tweet'}->{'full_text'},
	$key->{'retweeted_status'}->{'quoted_status'}->{'id_str'},
	$key->{'retweeted_status'}->{'quoted_status'}->{'text'},
	$key->{'retweeted_status'}->{'quoted_status'}->{'full_text'},
	$key->{'retweeted_status'}->{'quoted_status'}->{'extended_tweet'}->{'full_text'},
	$key->{'user'}->{'geo_enabled'} || "false",
	$key->{'geo'}->{'coordinates'}->[0],
	$key->{'geo'}->{'coordinates'}->[1],
	$key->{'place'}->{'id'},
	$key->{'place'}->{'country_code'},
	$key->{'place'}->{'place_type'},
	unpack("${pack_magic}H*", $key->{'place'}->{'full_name'}),
	$key->{'tag'}->{'type'},
	unpack("${pack_magic}H*", $key->{'tag'}->{'payload'}),
	$key->{'retweet_count'} || "0",
	$key->{'user'}->{'screen_name'}, $ds, $src,
	unpack("${pack_magic}H*", $key->{'text'}).
	$space_pad), 0, $packet_length);
	print P $key;

An instance of how "quoted" tweets are displayed inline:

oysttyer/oysttyer.pl

Lines 5134 to 5149 in 74b4946

	# Inject quote tweets, but only if not already at parent level in $my_json_ref
	# This prevents /thread from displaying them twice
	# Twitter website only displays one level of quotation so no looping through, use /thread for more
	foreach $t (@{ $my_json_ref }) {
	$parent_t = $t;
	if ((length($t->{'quoted_status_id_str'})) || (length($t->{'retweeted_status'}->{'id_str'}))) {
	# If it is a retweet, get the original status and check that for quoted_status
	if (length($t->{'retweeted_status'}->{'id_str'})) {
	$t = $t->{'retweeted_status'};
	};
	$t = $t->{'quoted_status'};
	# Using smartmatch would be easier, but we are kind on older versions of Perl
	if (($t) && !exists($ids{$t->{'id_str'}})) {
	# Add reference to allow badging in standardtweet
	$t->{'oysttyer_quoted'} = 'true';
	push(@{ $injected_json_ref }, $t);

I.e. we try to replicate Twitter's display preferences as much as
possible in a text only fashion.

If your use involves Tweeting, Retweeting, or liking content, share
how you will interact with Twitter users or their content.

These actions are performed via Readline commands:

Here's a lot of detail on the user interaction required:

https://github.com/oysttyer/oysttyer/wiki/Built-in-commands

If you’ll display Twitter content off of Twitter, explain how and
where Tweets and Twitter content will be displayed to users of your
product or service, including whether Tweets and Twitter content will
be displayed at row level or aggregated

Providing some more examples of what timelines look like:

My timeline:

[EXAMPLE OF MY CURRENT TIMELINE]

[EXAMPLE OF MY REPLIES]

[atomicules]

I've obviously not included examples of my timeline here since folk here know what Oysttyer looks like.

[atomicules]

This generic email today:

In July, we announced that we are making changes to increase accountability on our developer platform and reduce the distribution of spam through our APIs. We are introducing new default
app-level rate limits for common POST endpoints.
https://blog.twitter.com/developer/en_us/topics/tools/2018/new-developer-requirements-to-protect-our-platform.html

• We’ve emailed with the following information several times in the last few months. You are currently using our POST endpoints near or above the new limits and will no longer be able to post
  beyond these levels:
• Tweets & Retweets (combined): 300 per 3 hours
• Likes: 1000 per 24 hours
• Follows: 1000 per 24 hours
• Direct Messages: 15,000 per 24 hours

To apply for elevated access, you must complete the following steps.

[...]

So doesn't seem like a huge deal for us if folk are using their own keys.

[atomicules]

Whoop! We have the developer account bit. I've invited us two to that as well.

Now to figure out the "requesting elevated post..." bit

[atomicules]

Ok, submitted that...

I wonder if it's worth just throwing away our old creds and create new. It could be that there are still old installs using the version that included a default application key. I haven't received these notices for the other two accounts I use with OysTTYer.

Can "normal" folk create their own applications any more? Maybe that's the whole issue. I'm a little bit out of the loop with things. I.e. I syndicate (some) tweets through my website and created an app for that, but that was before the whole "developer account" requirement came in.

[atomicules]

Ugh (and not at Twitter this time):

Hello,

In reviewing your app's API usage we're seeing a consistent volume of daily rate-limited calls originating from the /1.1/statuses/home_timeline/ endpoint ranging from 77,000 to over 150,000
in total.

More specifically, there are some users that are Tweeting at excessive levels (e.g.: upwards of 20,000 Tweets per day).

Could you please let us know what sorts of user limits your service has? Also, what steps are you able to take to stop this behavior in advance of any POST elevation?

[funnelfiasco]

I think the answer here, if I understand the problem, is to revoke the app keys we used to ship with. We've been suggesting BYO key for a while, so this shouldn't be too much of an issue for the user base. Heck, it's one of the few things we actually have documented

[atomicules]

That's the problem though. It's no longer straight forward to create your own keys. Every single person would need to apply for a developer account. Such a shame.

[atomicules]

Twitter really are doing their utmost to kill off Twitter.

[funnelfiasco]

Oh hell, I forgot about that. Why do I keep using this garbage website?!

[atomicules]

Sent this for now to buy some time to think things over:

More specifically, there are some users that are Tweeting at excessive
levels (e.g.: upwards of 20,000 Tweets per day).

Wow, ok. That was unexpected.

As an open-source (Perl based) application the oauthkey and secret are
publicly available. It's not ideal, but is as per the previous arrangement
with TTYtter. We have relied on goodwill of our users so it is a shame to hear
some are blatantly abusing that.

Could you please let us know what sorts of user limits your service
has? Also, what steps are you able to take to stop this behavior in
advance of any POST elevation?

We are thinking this over. The problem is we are in a bit of a catch-22
situation: We could just revoke those keys and force users to get their own,
but with the developer account requirement it's a lot harder for users to do
that than before - each person wanting to use the application would need to go
through the whole developer account application process. So that's one option,
but an extreme one.

Will be back in touch soon.

[atomicules]

Maybe we could get new keys and not include them in the source, but instead have people "apply" for them via Github issues. Doesn't stop someone posting them publicly I guess, but might deter a bit more?

Any other ideas? I'm thinking...

[ScriptTiger]

Just an idea, but perhaps a middleman proxy/firewall service that users connect to that enforces limitations. Users can be authenticated (i.e. OAuth) and limitations can be imposed per user or limitations can just be imposed per IP. If going with authentication, each authenticated user can only be associated to one Twitter account and each Twitter account can only be associated to one authenticated user. This one-to-one will further help prevent abuse from multiple accounts on either side. Maybe requiring e-mail and/or phone verification could also be implemented to further secure it.

It would essentially be a buffer service to protect the Oysttyer developer account, while users will always also have the option of, of course, creating their own developer account if they don't like the shared restrictions. And since Oysttyer is designed to talk directly to Twitter and not through a middleman, it means the functions of the middleman can be kept relatively simple and primarily just forward along the messages, or not, while gathering statistics, unless there's interest in creating a whole new API which might provide some enhancement/improvement over the current Twitter API.

An added benefit of using a middleman service is that statistics can then also be gathered, anonymously of course, to know the average volume and frequency of tweets per user so reasonable limitations based on average usage as well as overall population usage can be better calculated. The overall population usage is also important since all users are essentially in the same pool under the same Oysttyer developer account and obviously not every user is "average."