Skip to content

Commit

Permalink
Merge pull request #109 from terraframe/master
Browse files Browse the repository at this point in the history 
Support for S3 ACL's, private ip addressing on autoscaled nodes and an additional setup command
  • Loading branch information
@pierotofy
pierotofy committed Apr 4, 2023
2 parents 68b8cc8 + 43408de commit e357e44
Show file tree
Hide file tree
Showing 3 changed files with 83 additions and 30 deletions.
47 changes: 25 additions & 22 deletions docs/aws.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ In order to use ClusterODM with AWS:
* Select/Create a VPC in which the resources will operate.
* Select/Create a subnet within the VPC.
* Create a security group in this region, VPC, and subnet which allows inbound access from your ClusterODM master instance on TCP port 3000. Note the name of the security group not the ID.
* Create an S3 bucket in this region to handle results. Don't configure this bucket to block public access.
* Create an S3 bucket in this region to handle results. If you don't specify an ACL, we will default to 'public-read', which requires public read access enabled for your bucket.
* Select an AMI (machine image) to run - Ubuntu has a [handy AMI finder](https://cloud-images.ubuntu.com/locator/ec2/).
* Create an IAM account for ClusterODM to use, which has EC2 and S3 permissions.
* Create a ClusterODM configuration json file as below.
Expand All @@ -31,7 +31,8 @@ the on-demand instance cost - you'll always pay the current market price, not yo
"secretKey": "CHANGEME!",
"s3":{
"endpoint": "s3.us-west-2.amazonaws.com",
"bucket": "bucketname"
"bucket": "bucketname",
"acl": "none"
},
"vpc": "",
"subnet": "",
Expand Down Expand Up @@ -64,26 +65,28 @@ the on-demand instance cost - you'll always pay the current market price, not yo
}
```

| Field | Description |
|------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
| accessKey | AWS Access Key |
| secretKey | AWS Secret Key |
| s3 | S3 bucket configuration. Note that the bucket should *not* be configured to block public access. |
| vpc | The virtual private cloud in which the instances operate. Not providing this assumes a default setting for VPC within the AWS environment. |
| subnet | The subnet supporting the instances. Not providing this assumes a default setting for the subnet within the AWS environment. |
| securityGroup | AWS Security Group name (not ID). Must exist and allow incoming connections from your ClusterODM host on port TCP/3000. |
| createRetries | Number of attempts to create a droplet before giving up. Defaults to 1. |
| maxRuntime | Maximum number of seconds an instance is allowed to run ever. Set to -1 for no limit. |
| maxUploadTime | Maximum number of seconds an instance is allowed to receive file uploads. Set to -1 for no limit. |
| monitoring | Set to true to enable detailed Cloudwatch monitoring for the instance. |
| region | Region identifier where the instances should be created. |
| zone | Zone identifier where the instances should be created. |
| ami | The AMI (machine image) to launch this instance from. |
| tags | Comma-separated list of key,value tags to associate to the instance. |
| spot | Whether to request spot instances. If this is true, a `spotPrice` needs to be provided in the `imageSizeMapping`. |
| imageSizeMapping | Max images count to instance size mapping. (See below.) |
| addSwap | Optionally add this much swap space to the instance as a factor of total RAM (`RAM * addSwap`). A value of `1` sets a swapfile equal to the available RAM. |
| dockerImage | Docker image to launch |
| Field | Description |
|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
| accessKey | AWS Access Key |
| secretKey | AWS Secret Key |
| s3 | S3 bucket configuration. |
| vpc | The virtual private cloud in which the instances operate. Not providing this assumes a default setting for VPC within the AWS environment. |
| subnet | The subnet supporting the instances. Not providing this assumes a default setting for the subnet within the AWS environment. |
| usePrivateAddress | Set to true to use the private IP address when communicating with auto-scaled nodes. Useful if ClusterODM is on the same vpc as the auto-scaled nodes. |
| securityGroup | AWS Security Group name (not ID). Must exist and allow incoming connections from your ClusterODM host on port TCP/3000. |
| createRetries | Number of attempts to create a droplet before giving up. Defaults to 1. |
| maxRuntime | Maximum number of seconds an instance is allowed to run ever. Set to -1 for no limit. |
| maxUploadTime | Maximum number of seconds an instance is allowed to receive file uploads. Set to -1 for no limit. |
| monitoring | Set to true to enable detailed Cloudwatch monitoring for the instance. |
| region | Region identifier where the instances should be created. |
| zone | Zone identifier where the instances should be created. |
| ami | The AMI (machine image) to launch this instance from. |
| tags | Comma-separated list of key,value tags to associate to the instance. |
| spot | Whether to request spot instances. If this is true, a `spotPrice` needs to be provided in the `imageSizeMapping`. |
| imageSizeMapping | Max images count to instance size mapping. (See below.) |
| addSwap | Optionally add this much swap space to the instance as a factor of total RAM (`RAM * addSwap`). A value of `1` sets a swapfile equal to the available RAM. |
| dockerImage | Docker image to launch |
| nodeSetupCmd | Can be optionally used to run a setup command on auto-scaled nodes right before we run ODM. |

## Image Size Mapping

Expand Down
20 changes: 17 additions & 3 deletions libs/asr-providers/aws.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,12 @@ module.exports = class AWSAsrProvider extends AbstractASRProvider{
"secretKey": "CHANGEME!",
"s3":{
"endpoint": "CHANGEME!",
"bucket": "CHANGEME!"
"bucket": "CHANGEME!",
"acl": "public-read"
},
"vpc": "",
"subnet": "",
"usePrivateAddress": false,
"securityGroup": "CHANGEME!",
"maxRuntime": -1,
"maxUploadTime": -1,
Expand All @@ -50,12 +52,13 @@ module.exports = class AWSAsrProvider extends AbstractASRProvider{

"addSwap": 1,
"dockerImage": "opendronemap/nodeodm",
"iamrole": ""
"iamrole": "",
"nodeSetupCmd": ""
}, userConfig);
}

async initialize(){
this.validateConfigKeys(["accessKey", "secretKey", "s3.endpoint", "s3.bucket", "securityGroup"]);
this.validateConfigKeys(["accessKey", "secretKey", "s3.endpoint", "s3.bucket", "s3.acl", "securityGroup"]);

// Test S3
const { endpoint, bucket } = this.getConfig("s3");
Expand Down Expand Up @@ -118,12 +121,19 @@ module.exports = class AWSAsrProvider extends AbstractASRProvider{
const secretKey = this.getConfig("secretKey");
const s3 = this.getConfig("s3");
const webhook = netutils.publicAddressPath("/commit", req, token);

const setupCmd = this.getConfig("nodeSetupCmd");
if (setupCmd != null && setupCmd.length > 0)
{
await dm.ssh(setupCmd);
}

await dm.ssh([`sudo docker run -d -p 3000:3000 ${dockerImage} -q 1`,
`--s3_access_key ${accessKey}`,
`--s3_secret_key ${secretKey}`,
`--s3_endpoint ${s3.endpoint}`,
`--s3_bucket ${s3.bucket}`,
`--s3_acl ${s3.acl}`,
`--webhook ${webhook}`,
`--token ${nodeToken}`].join(" "));
}
Expand Down Expand Up @@ -178,6 +188,10 @@ module.exports = class AWSAsrProvider extends AbstractASRProvider{
args.push(this.getConfig("tags").join(","));
}

if (this.getConfig("usePrivateAddress")) {
args.push("--amazonec2-use-private-address");
}

if (this.getConfig("engineInstallUrl")){
args.push("--engine-install-url")
args.push(this.getConfig("engineInstallUrl"));
Expand Down
46 changes: 41 additions & 5 deletions libs/proxy.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ const async = require('async');
const odmOptions = require('./odmOptions');
const asrProvider = require('./asrProvider');
const floodMonitor = require('./floodMonitor');
const AWS = require('aws-sdk');

module.exports = {
initialize: async function(cloudProvider){
Expand Down Expand Up @@ -521,11 +522,46 @@ module.exports = {

const s3Url = url.parse(asrProvider.downloadsPath());
s3Url.pathname = path.join(taskId, assetPath);
res.writeHead(301, {
'Location': url.format(s3Url)
});
res.end();
return;

const s3Config = asrProvider.get().getConfig("s3");

// If URL requires authentication, fetch the object on their behalf and then stream it to them
// If our aws library gets updated to v3, then we could return a redirect to a presigned url instead
if (s3Config != null && s3Config.acl !== "public-read") {
let key = path.join(taskId, assetPath)

const s3 = new AWS.S3({
endpoint: new AWS.Endpoint(s3Config.endpoint),
signatureVersion: 'v4',
accessKeyId: asrProvider.get().getConfig("accessKey"),
secretAccessKey: asrProvider.get().getConfig("secretKey")
});

s3.getObject({ Bucket: s3Config.bucket, Key: key }, (err, data) => {
if (err) {
logger.error(`Error encountered downloading object ${err}`);
res.statusCode = 500;
res.end('Internal server error');
return;
}

// Set the content-type and content-length headers
res.setHeader('Content-Type', data.ContentType);
res.setHeader('Content-Length', data.ContentLength);

// Write the object data to the response
res.write(data.Body);
res.end();
});
return;

} else {
res.writeHead(301, {
'Location': url.format(s3Url)
});
res.end();
return;
}
}
}

Expand Down

0 comments on commit e357e44

Please sign in to comment.