Sahana's Cloud Custodian Policies

Policies in Production

Policy	Description
mailer.yml	Sends email notification via Simple Email Service (SES) using notify action
s3-bucket-versioning.yml	Rectifies and enables all suspended versioning on S3 buckets, then sends notifications.
s3-bucket-public-access.yml	Rectifies and corrects the Global grants and secured S3 buckets as private, then sends notifications.
s3-toggle-logging.yml	Configure New Buckets Settings and Standards such as enabling the default S3 AES256 bucket encryption, turns on object versioning, enables logging on the bucket, and tags the user that created the bucket.

Cloud Custodian Architecture and AWS Services

Cloud Custodian (a.k.a C7N) notifies users in real-time AWS resources behavior changes, Compliance (Security/Access Control, Encryption, Backups, etc) and drives Cost savings (Off-hours, Monitoring and Garbage Collection of unused and underutilized resources).

Getting Started

Quick Install

*** Install dependencies (with virtualenv) ***
$ sudo apt-get -y install virtualenv or sudo yum install virtualenv
$ virtualenv custodian_env
$ source custodian_env/bin/activate

*** Install AWS CLI and C7N ***
$ pip install awscli c7n

** Configure AWSCLI **
$ aws configure
(Configure with AWS Credentials and Region)

*** Verify AWSCLI Installation with any CLI command ***
$ aws ec2 describe-regions

*** To Install Cloud Custodian Mailer ***
*** Install repository***
$ git clone https://github.com/capitalone/cloud-custodian
$ cd cloud-custodian/tools/c7n_mailer
$ pip install -r requirements.txt
$ python setup.py develop

*** Verify Installation ***
$ c7n-mailer
$ custodian

For more info, check out Cloud Custodian in GitHub

Usage

Getting Started

Cloud Custodian must be run within a virtual environment.

$ cd ~
$ source custodian_env/bin/activate
$ cd cloudcustodian_scripts  (this is the folder where all the custodian policies reside)

** Execute/run the Cloud Custodian Policies **

# Validate the configuration
$ custodian validate s3-bucket-public-access.yml

# Dryrun the policies 
$ custodian run --dryrun -s check-public-access s3-bucket-public-access.yml
(Note: Make sure If you get a match (e.g. count > 0), then run the below command)

# Run the policy 
$ custodian run -s check-public-access s3-bucket-public-access.yml

** Invoking c7n Mailer **
# Validate the configuration
$ custodian validate s3-bucket-public-access.yml

# Dryrun the policies 
$ custodian run --dryrun -s check-public-access s3-bucket-public-access.yml
(Note: Make sure If you get a match (e.g. count > 0), then run the below command)

# Run the policy to invoke custodian mailer
$ c7n-mailer --config mailer.yml --update-lambda && custodian run -c s3-bucket-public-access.yml -s .

When we run this policy, Check the AWS console for a new Lambda named `cloud-custodian-mailer`. 
The mailer runs every five minutes, so wait a bit and then look for an email in your inbox. (Orelse manually, edit CWE scheduled time less than 5 mins for the quick response)



 Cloud Custodian will create a log files in the ~/cloudcustodian_scripts/check-public-access/ subdirectory IF there are any matches. 

C7N Mailer Workflow - AWS SES Sends a mail on violation occurs in S3 Bucket

Workflow

Steps for Cloud Custodian mailer to ensure S3 Governance and Compliance.
Step 1: Create Mailer file

Step 2: Create Custodian Policy for S3 Public read/write Access - Sends email notification via Simple Email Service (SES) using notify action

$ vim s3-bucket-public-access-check.yml .

Step 3: Run a Command that installs the mailer and run a policy that triggers an email to your inbox.

$ c7n-mailer --config mailer.yml --update-lambda && custodian run -c s3-bucket-public-access-check.yml -s .

Step 4: Check the AWS console for a new Lambda and CWE named "cloud-custodian-mailer" and "custodian-s3-public-access".

Lambda Functions:

CloudWatch Events:

CWE S3 Bucket Logs:

CWE Custodian mailer Logs:

Step 5: Cloud Custodian mailer deployed lambda and sends a customized mail via SES service.

Environment Settings

mailer.yml

#Which queue should we listen to for messages
queue_url: https://sqs.us-east-1.amazonaws.com/930337447539/c7n_mailer_for_s3_events
#Standard Lambda Function Config
region: us-east-1
role: arn:aws:iam::930337447539:role/lambda-s3-governance
#Default from address
from_address: sjayaramu@eplus.com

Cloud Custodian Lambda AWS Role

Note: Based on your use case, additional permissions may be needed.
Cloud Custodian will generate a msg if that is the case after invocation.
AWS IAM Role & policies plays an important role to allows Lambda functions to call AWS services. (Make a note of IAM ARN ex: arn:aws:iam::930337447539:role/S3-GovernanceForLincoln)
Trust relationship:
"Service": "lambda.amazonaws.com"

Reference:
| AWSS3CustomPolicyForLincoln.json
 | A policy defines the AWS permissions that you can assign to a user, group, or role.  |

Schemas Used

s3

(custodian_env) [root@localhost custodian_scripts]# custodian schema s3
aws.s3:
actions: [attach-encrypt, auto-tag-user, configure-lifecycle, delete, delete-bucket-notification,
delete-global-grants, encrypt-keys, encryption-policy, invoke-lambda, mark-for-op,
no-op, notify, put-metric, remove-statements, remove-website-hosting, set-bucket-encryption,
set-inventory, set-statements, tag, toggle-logging, toggle-versioning, unmark]
filters: [and, bucket-encryption, bucket-notification, cross-account, data-events,
event, global-grants, has-statement, inventory, is-log-target, marked-for-op,
metrics, missing-policy-statement, missing-statement, no-encryption-statement,
not, or, value]
[ OR ]
** For S3 Schema Filters **
(custodian_env) [root@localhost custodian_scripts]# custodian schema s3.filters
aws.s3:
filters: [and, bucket-encryption, bucket-notification, cross-account, data-events,
event, global-grants, has-statement, inventory, is-log-target, marked-for-op,
metrics, missing-policy-statement, missing-statement, no-encryption-statement,
not, or, value]
** For S3 Schema actions **
(custodian_env) [root@localhost lfg-custodian]# custodian schema s3.actions
aws.s3:
actions: [attach-encrypt, auto-tag-user, configure-lifecycle, delete, delete-bucket-notification,
delete-global-grants, encrypt-keys, encryption-policy, invoke-lambda, mark-for-op,
no-op, notify, put-metric, remove-statements, remove-website-hosting, set-bucket-encryption,
set-inventory, set-statements, tag, toggle-logging, toggle-versioning, unmark]
** To undesrtand a particular filter & action: **
(custodian_env) [root@localhost custodian_scripts]# custodian schema s3.filters.global-grants
Help

Filters for all S3 buckets that have global-grants
:example:
.. code-block:: yaml
    policies:
      - name: s3-delete-global-grants
        resource: s3
        filters:
          - type: global-grants
        actions:
          - delete-global-grants

Schema

{
"additionalProperties": false,
"required": [
"type"
],
"type": "object",
"properties": {
"allow_website": {
"type": "boolean"
},
"operator": {
"enum": [
"or",
"and"
],
"type": "string"
},
"type": {
"enum": [
"global-grants"
]
},
"permissions": {
"items": {
"enum": [
"READ",
"WRITE",
"WRITE_ACP",
"READ",
"READ_ACP"
],
"type": "string"
},
"type": "array"
}
}
}

Troubleshooting Tips

Use 'custodian validate' to find syntax errors
Check 'name' of policy doesn't contain spaces
Check SQS to see if Custodian payload is entering the queue
Check cloud-custodian-mailer lambda CloudWatch rule schedule (5 minute by default)
Check Lambda error logs (this requires CloudWatch logging)
Check role for lambda(s) have adequate permissions
Remember to update the cloud-custodian-mailer lambda when making changes to a policy that uses notifications
Clear the cache if you encounter errors due to stale information (rm ~/.cache/cloud-custodian.cache)

Lambda Code Cheatsheet

mode:
  type: cloudtrail
  role: arn:aws:iam::930337447539:role/S3-GovernanceForLincoln
  events:
    - CreateBucket

mode:
  type: periodic
  role: arn:aws:iam::930337447539:role/S3-GovernanceForLincoln
  schedule: "rate(15 minutes)"

mode:
  type: periodic
  role: arn:aws:iam::930337447539:role/S3-GovernanceForLincoln
  schedule: 'cron(0/2 * * * ? *)'

Sending Notifications via SES

actions:
 - type: notify
   template: default.html
   template_format: 'html'
   priority_header: '5'
   subject: "ALERT! - S3 : Invalid Global ACL on Bucket [AWS Account: {{ account }} - Region: {{ region }}]"
   comments: "Violation of S3 policy"
   violation_desc: <Message_Of_Mail_Body>
   action_desc: "Actions Taken: Corrects the ACLs/Policy and Notify User"
   to:
     - <your-email-address-goes-here>
   owner_absent_contact:
     - <your-emails-address-goes-here>
   transport:
     type: sqs
     queue: https://sqs.us-east-1.amazonaws.com/930337447539/c7n_mailer_for_s3_events

Reference: Schedule Expressions for Rules
Useful Tool: Quick simple editor for cron schedule expressions.

Note
Config: May run in a different region but not cross-account
Event: Only run in the same region and account
Periodic: May run in a different region and different account

Cloud Custodian Important Resources

Cloud Custodian - All Resources
Cloud Custodian - Getting Started
Cloud Custodian - Github
Cloud Custodian - Docs
Cloud Custodian - 400+ actions and 300+ filters to build policies with
Cloud Custodian - Features
Cloud Custodian - S3 Module
Blog - Using Cloud Custodian for Cloud Governance in AWS
Lambda Support
Lambda
AWS CloudWatch Schedule Rules
S3 Data Events
CloudWatch Rules Expressions
Adding Custom Fields to Reports
Custodian Mailer
C7N_Mailer