ReClo is lightweight software for small and medium business recovery. Our simple client applications backup a user's Windows server to Amazon Web Services (AWS). In the event of hardware failure, users can access their backups and recover to a new virtual instance running in AWS. Connection to recovered instances is handled securely with a VPN connection.
ReClo offers two client apps:
- Backup Manager is downloaded to a client's server. Once installed, we take regular backups of the system and store them securely in S3. We take one full backup per week, followed by incremental backups (of just the files that have changed) in-between full backups. These backups are automatically uploaded from the client into AWS.
- Recovery Manager is downloaded to a client's desktop computer. Recovery Manager shows clients a list of thier backups, allowing them to choose one to recover. These backup are compressed and imported as a new virtual instances in EC2. Recovery takes about one hour to complete. Recovery Manager provides a secure VPN connection so users can continue to access their important files once recovered.
This repository contains the following resources:
Our server-side RESTful API written in node.js. Our API interacts with our AWS resources and MySQL database.
- app.js is our main API setup and handles all request routing using the express.js routing framework.
- lib contains our core libraries, database management tools, and cron jobs.
- node_modules contain all of our node.js dependencies needed for our app.
- public contains basic stylesheets for our API's web portal.
- routes contains all of our request handlers for each set of API calls (user authentication, backup, and recovery).
- views contains basic views for our API's web portal.
Our native Windows applications that orchestrate backups and connection to restored instances from the client's computer.
- client_lib contains our core libraries for our client apps.
- powershell_backups contains our powershell scripts that orchestrate backups.
- Reclo Backup Manager contains our client Windows app for backup management.
- Reclo Recovery Manager contains our client Windows app for recovery management.
Our project proposal, project architecture, and an overview of each sprint used in the development process.
Our website with information about our service and links to download our client apps.
We borrowed our theme from BlackTie.co.
Hosted publicly on Github at:
reclo.github.io
The ReClo Team
Boston University
EC500 Cloud Computing
Spring 2015
| Carlton Duffett | cduffett@bu.edu |
| Deema Kutbi | deemak@bu.edu |
| Emilio Teran | eteran@bu.edu |
| Konstantino Sparakis | sparakis@bu.edu |
| Minhan Xiang | xmh@bu.edu |
Our main API server runs on an AWS AMI Linux instance. We use the AMI setup as it comes pre-configured with the AWS command line interface (CLI). We chose Linux instead of a Windows server for its customizability and ease of use.
After starting a free-tier instance (30GB storage), install Node.js.
You can then use the node package manager (npm) to install forever.js, a simple CLI tool that keeps a node.js script running continuously, forever. We use this to keep our API server running.
sudo npm install forever
The remaining node packages are already part of our repository. Simply download the API folder to your instance. This should be located in the directory:
~/cloud-recover/api
Lastly, create a folder at the root mkdir /backups-tmp/ to hold backups during the import process. Because backups are generally large, we mounted a separate 100GB volume for this purpose. You may need to change the permissions of this folder sudo chmod 700 /backups-tmp/.
To start the API server, run:
cd ~/cloud-recover/api
forever start -l forever.log -o ./logs/out.log -e ./logs/err.log ./bin/www
This creates several logs that you can check to mointor the server's activity. The master log is in:
~/.forever/forever.log
Output and error logs are maintained in:
api/logs/out.log api/logs/err.log
To stop the server, run forever list and note the process ID (PID) of script bin/www.
Then run forever stop <pid>.
Before you can restart the server, you need to clear the previous run's logs:
rm ~/.forever/forever.log
Several cron jobs are used to perform background tasks and to manage API resources. The crontab can easily be edited using
env EDITOR=nano crontab -e. Add the following jobs to your cron tab:
-
deactivates expired tokens once per day at 4:01am UTC
01 04 * * * /usr/bin/node ~/cloud-recover/api/lib/ManageSession.js -
terminates recovered instances after stopped for 2 weeks
02 04 * * * /usr/bin/node ~/cloud-recover/api/lib/FreeInstances.js -
fails recovery tasks that are more than 24 hours old
03 04 * * * /usr/bin/node ~/cloud-recover/api/lib/FailRecovery.js -
scans for pending recovery tasks once per minute
* * * * * /usr/bin/node ~/cloud-recover/api/lib/ManagePending.js -
scans for downloading recovery tasks once per minute
* * * * * /usr/bin/node ~/cloud-recover/api/lib/ManageDownloads.js -
scans for importing recovery tasks once per minute
* * * * * /usr/bin/node ~/cloud-recover/api/lib/ManageImports.js -
scans for finishing recovery tasks once per minute
* * * * * /usr/bin/node ~/cloud-recover/api/lib/FinishRecovery.js
Our client applications are still in the developmental stage. Download them to your Windows computer and run them using MS Visual Studio or a similar development tool. Our clients talk directly to our server at a specific IP address. Change the root of our API requests:
private static string urlhead = 'http://52.24.77.177:3000/'
to match the IP address of your server. Our API server runs on port 3000.
We use a master bucket in S3 to hold all of our client backups, reclo-client-backups. Backups for each client are grouped into folders named using each clients unique ID. We also use a bucket to hold all of our imported instances (recovered from client backups), reclo-imported-vms. Create these buckets in S3 and give your API server permission to access them.
All or our records are maintained in a single database, reclodb. This holds records for:
- users
- session tokens
- backups
- uploads
- recovery tasks
- running instances
Create the following tables in a new MySQL database:
| users | ||||||
| id | user_id | username | hash | date_created | user_status | |
| INT | VARCHAR(36) | VARCHAR(32) | VARCHAR(255) | VARCHAR(60) | DATETIME | VARCHAR(1) |
| tokens | |||||
| id | token_id | user_id | date_created | date_deactivated | token_status |
| INT | VARCHAR(32) | VARCHAR(36) | DATETIME | DATETIME | VARCHAR(1) |
| backups | |||||||
| id | backup_id | user_id | file_size | file_name | type | date_created | backup_status |
| INT | VARCHAR(32) | VARCHAR(36) | FLOAT | VARCHAR(100) | VARCHAR(11) | DATETIME | VARCHAR(1) |
| uploads | |||||||
| id | upload_id | user_id | file_size | file_name | time_started | time_completed | upload_status |
| INT | VARCHAR(32) | VARCHAR(36) | FLOAT | VARCHAR(100) | DATETIME | DATETIME | VARCHAR(1) |
| recovery | |||||||
| id | recovery_id | user_id | backup_id | instance_id | conversion_id | file_name | recovery_state |
| INT | VARCHAR(32) | VARCHAR(36) | VARCHAR(32) | VARCHAR(10) | VARCHAR(18) | VARCHAR(100) | VARCHAR(12) |
| total_progress | state_progress | no_downloads | no_completed | instance_state | date_started | date_completed | recovery_status |
| INT | INT | INT | INT | VARCHAR(12) | DATETIME | DATETIME | VARCHAR(1) |
| instances | ||||
| id | user_id | instance_id | instance_name | ip_address |
| INT | VARCHAR(36) | VARCHAR(10) | VARCHAR(32) | VARCHAR(16) |
| instance_state | availability_zone | date_created | date_last_stopped | upload_status |
| VARCHAR(23) | VARCHAR(20) | DATETIME | DATETIME | VARCHAR(1) |
The database password, our AWS access key, and AWS secret key are stored in the API server's userdata. All of our userdata is maintained as a list of comma-separated, key-value pairs. For example:
mysql=mysqlpassword1, access_key=ACCESSKEYto43AWS-23ab, secret_key=234t+3#SECRETKEY1!-vrs2