Final-Year-Project

Introduction

This application was developed as part of my final year project for university.

The purpose of the application is to identify potential bots that a Twitter user is following. Accounts that a Twitter user is following are known as friends.

The application does this by generating a report which contains likelihood scores for a Twitter user's friends being a bot. The report is sent to the Twitter user via email.

The scores are provided by leveraging the Botometer API. You can find more information about Botometer here.

Note

Please note that this application is not affiliated with Twitter or the Botometer project.

Dependencies

This application is written in Python 3.9 and relies on the following dependencies:

[packages]
botometer = "1.6"
tweepy = "3.10.0"
loguru = "0.5.3"
jinja2 = "2.11.3"
pycountry = "20.7.3"
faker = "8.1.1"

[dev-packages]
black = "20.8b1"
pytest = "6.2.3"
pytest-cov = "2.11.1"
pytest-mock = "3.5.1"

[requires]
python_version = "3.9"

These dependencies can be installed using either: pipenv install or pip install -r requirements.txt.

Prerequisites

Before using the project's application, credentials for the following are required:

• The Twitter API

• The Botometer API

• An email server and account

Twitter API Credentials

The project's application uses the Twitter API to collect a Twitter user's friends.

To get access to the Twitter API, a developer account is needed. You can apply for one here and find more information about the application process here.

Once you have a developer account, go to the developer portal to create an application.

When creating the application, provide a unique name and click Complete. The application's API key and secret will be shown (aka. consumer key and consumer secret). Store them somewhere safe as they will be needed in the final step.

Botometer API Credentials

The project's application uses the Botometer API to collect bot likelihood scores for a Twitter user's friends.

To get access to the Botometer API, a RapidAPI account is needed. You can create one here.

Once you have a RapidAPI account, go to the Botometer API pricing page and subscribe to the Pro or Basic plan.

Note

The Pro plan requires a credit or debit card to be added to the RapidAPI account.

Once you have chosen a plan, go to the RapidAPI developer dashboard. Then, go to security page for the default-application. The application key (API key) will be shown. Store it somewhere safe as it will be needed in the final step.

Email Credentials

Note

The instructions below let the project's application send the report via email using Google's SMTP servers and a Google account.

If you are using another email provider, you will need to find their own instructions on how to authenticate to their email servers.

At the time of writing (21/04/2021), Google's SMTP server domain is smtp.gmail.com and SSL port is 465. For more information, see the Use the Gmail SMTP server heading in Google's documentation.

Perform the steps below to generate an app password for a Google account. This will allow the project's application to send an email from the account.

Note

To generate an app password for a Google account, 2-Step Verification must be set up on the account.

Documentation on how to do this can be found here.

1. Temporarily enable less secure app access on the Google account.

   • You may need to sign in. Turn Allow less secure apps on.

2. Go to the Google account and select Security.

3. Under Signing in to Google section, select App passwords.

   • You may be prompted to enter the account password again.

4. In the Select app dropdown, select Mail.

5. In the Select device dropdown, select Other (custom name) and enter a friendly name for the app password.

6. Select GENERATE.

The app password will be shown. Store it somewhere safe as it will be needed in the next step.

Important

When finished using the project's application, make sure the app password is revoked from the Google account and less secure app access is disabled.

Documentation on how to do this can be found here.

Exporting Credentials

Once all the credentials have been collected, export them as environment variables to be accessed by the project's application.

1. Temporarily disable terminal command logging to prevent credential exposure:

   • Linux: set +o history

   [!NOTE]

   Use the command: set -o history to re-enable terminal command logging.

   • Windows (PowerShell): Set-PSReadlineOption -HistorySaveStyle SaveNothing

   [!NOTE]

   Use the command: Set-PSReadlineOption -HistorySaveStyle SaveIncrementally to re-enable terminal command logging.

2. Export the required credentials as environment variables:

   • Linux:

     export TWITTER_API_KEY="Enter the Twitter API key here."
     export TWITTER_API_SECRET="Enter the Twitter API secret here."
     export BOTOMETER_API_KEY="Enter the Botometer API key here."
     export EMAIL_SERVER_DOMAIN="smtp.gmail.com"
     export EMAIL_SERVER_PORT="465"
     export EMAIL_SENDER_ADDRESS="Enter the account email address here."
     export EMAIL_SENDER_PASSWORD="Enter the account app password here."

   • Windows (PowerShell):

     $Env:TWITTER_API_KEY="Enter the Twitter API key here."
     $Env:TWITTER_API_SECRET="Enter the Twitter API secret here."
     $Env:BOTOMETER_API_KEY="Enter the Botometer API key here."
     $Env:EMAIL_SERVER_DOMAIN="smtp.gmail.com"
     $Env:EMAIL_SERVER_PORT="465"
     $Env:EMAIL_SENDER_ADDRESS="Enter the account email address here."
     $Env:EMAIL_SENDER_PASSWORD="Enter the account app password here."

Usage

From the root of the repository, run the project's application using the command: python ./src/main.py {Twitter Username} {Email Address}

CLI Usage

usage: main.py [-h] [-d] username email

Generates and emails a report containing bot likelihood scores for a Twitter user's friends.

positional arguments:
  username    The Twitter username to obtain friends for.
  email       The recipient email address to send the report to.

optional arguments:
  -h, --help  show this help message and exit
  -d, --demo  Flag to run the application in demo mode.

Warning

For the project's application to collect a Twitter user's friends, the account must not be private/protected (have Tweets protected).

Documentation on how to disable and enable account protections can be found here.

Important

Ensure you replace the placeholders ({}) with a Twitter username (without the "@" symbol) and an email address.

For example:

python ./src/main.py dbrenuk exampleemail@gmail.com

The above command would produce a bot likelihood report for @dbrenuk's Twitter friends and send the report to exampleemail@gmail.com.

Note

The application can be run in demo mode by providing either -d or --demo as an optional argument.

Docker

Note

These instructions assume you already have Docker installed and running on your machine.

1. From the root of the repository, build the container image from the dockerfile using the following command: docker build -t fyp-app:latest .

2. Temporarily disable terminal command logging to prevent credential exposure:

   • Linux: set +o history

   [!NOTE]

   Use the command: set -o history to re-enable terminal command logging.

   • Windows (PowerShell): Set-PSReadlineOption -HistorySaveStyle SaveNothing

   [!NOTE]

   Use the command: Set-PSReadlineOption -HistorySaveStyle SaveIncrementally to re-enable terminal command logging.

3. From the root of the repository, run the project's application in a container using the following command:

   • Linux:

     docker run --rm --name fyp-app \
         -e TWITTER_API_KEY="Enter the Twitter API key here." \
         -e TWITTER_API_SECRET="Enter the Twitter API secret here." \
         -e BOTOMETER_API_KEY="Enter the Botometer API key here." \
         -e EMAIL_SERVER_DOMAIN="smtp.gmail.com" \
         -e EMAIL_SERVER_PORT="465" \
         -e EMAIL_SENDER_ADDRESS="Enter the account email address here." \
         -e EMAIL_SENDER_PASSWORD="Enter the account app password here." \
         -t fyp-app:latest \
         python ./src/main.py {Twitter Username} {Email Address}

   • Windows (PowerShell):

     docker run --rm --name fyp-app `
         -e TWITTER_API_KEY="Enter the Twitter API key here." `
         -e TWITTER_API_SECRET="Enter the Twitter API secret here." `
         -e BOTOMETER_API_KEY="Enter the Botometer API key here." `
         -e EMAIL_SERVER_DOMAIN="smtp.gmail.com" `
         -e EMAIL_SERVER_PORT="465" `
         -e EMAIL_SENDER_ADDRESS="Enter the account email address here." `
         -e EMAIL_SENDER_PASSWORD="Enter the account app password here." `
         -t fyp-app:latest `
         python ./src/main.py {Twitter Username} {Email Address}

Unit tests

See the unit tests README file.

LICENSE

This project is licensed under the GNU General Public License v3.0 - see the LICENSE for details.