🎯 New Source Connector Request: Twitter

[ehisakhile]

Source Connector Twitter

What is a Vanus Connector

Vanus Connect allows you to skip the complex integration with external services by offering out-of-the-box connectors. Each connector acts as a proxy between outside services and your system, enabling users to have faster integrations with no codes. All data produced or processed by our connectors conform to the CloudEvents specification, which helps enterprises leverage event-driven architecture to do business in the age of events.

What is a Source Connector

A Source connector obtains data from an underlying data producer and delivers it to the target after the original data has been transformed into a CloudEvents. For example:

• MySQL Source: extracts data from the Binlog, transforms it into CloudEvents, and sends it to the target.

• GitHub Source: uses a Webhook server to receive the event from GitHub, transform them to CloudEvents and send them to the target.

• AWS S3 Source: Uses the AWS SDK to get the data, transforms it into CloudEvents, and sends them to the target.

There are two types of connectors push or pull connectors.
A push connector is passive and only does its logic when it receives an event, and a pull connector must pull the data in an interval. We use a push method when it’s possible.

Features of this Source Connector

• Receive events from Twitter when a user’s tweet is liked, retweeted, quote-tweeted, DMs, tweet comments and put the data into a CloudEvent to send it to the target
• Each event type (like, dm, comment, etc ...) Should be specified in the event type specification of the CloudEvent.
• Configuration parameters

Specific data needed

When someone likes the user's tweet.

• Tweet ID: The unique identifier for the tweet that was liked.
• User ID: The unique identifier for the user who liked the tweet.
• User screen name: The screen name (i.e., username) of the user who liked the tweet.
• User name: The full name of the user who liked the tweet.
• Tweet text: The text of the tweet that was liked.
• Like creation time: The timestamp indicating when the like was created.
• Tweet URL: The URL of the tweet that was liked.
• Number of likes: The total number of likes the tweet has received.
• Number of retweets: The total number of retweets the tweet has received.
• Number of replies: The total number of replies the tweet has received.

When someone retweets or quote retweets

• Tweet ID: The unique identifier for the tweet that was retweeted.
• User ID: The unique identifier for the user who retweeted the tweet.
• User screen name: The screen name (i.e., username) of the user who retweeted the tweet.
• User name: The full name of the user who retweeted the tweet.
• Retweet ID: The unique identifier for the retweet itself.
• Retweet text: The text of the retweet, if any.
• Retweet creation time: The timestamp indicating when the retweet was created.
• Tweet URL: The URL of the original tweet.
• Number of likes: The total number of likes the original tweet has received.
• Number of retweets: The total number of retweets the original tweet has received.
• Number of replies: The total number of replies the original tweet has received.

When someone Dms the user

• DM ID: The unique identifier for the DM.
• Sender ID: The unique identifier for the user who sent the DM.
• Sender screen name: The screen name (i.e., username) of the user who sent the DM.
• Sender name: The full name of the user who sent the DM.
• Recipient ID: The unique identifier for the user who received the DM (i.e., you).
• DM text: The text of the DM.
• DM creation time: The timestamp indicating when the DM was created.

When someone replies to the user's tweet

• Tweet ID: The unique identifier for the tweet that was replied to.
• User ID: The unique identifier for the user who replied to the tweet.
• User screen name: The screen name (i.e., username) of the user who replied to the tweet.
• User name: The full name of the user who replied to the tweet.
• Reply ID: The unique identifier for the reply itself.
• Reply text: The text of the reply.
• Reply creation time: The timestamp indicating when the reply was created.
• Tweet URL: The URL of the original tweet.
• Number of likes: The total number of likes the original tweet has received.
• Number of retweets: The total number of retweets the original tweet has received.
• Number of replies: The total number of replies the original tweet has received.

How to create a new connector

1. Fork the vanus-connect repo.
2. Create a source_twitter_proposal.md based on the source_proposal_example.md under the vanus-connect/proposals directory. The proposal will introduce how a developer tends to implement the connector in detail. The proposal PR must be approved and merged before going to the next step.
3. Copy one of the source templates (Golang template, Java template) to vanus-connect/connectors/ and rename it as source-twitter.
4. Implement your connector and submit a PR.
5. Wait for the connector to be reviewed and approved.

How to claim to implement this connector

If you want to solve this issue, please leave a comment on this issue like:

I'd like to implement the connector, please assign it to me.

Vanus community will assign the issue to you on time.

[allensuvorov]

I'd like to implement the connector, please assign it to me.

[allensuvorov]

As an update. Just rechecked Twitter API
Looks like, at this point, sourcing data requires a paid account: