Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gitchore: refactor to call secrets manager #1676

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+64 −0
Open

gitchore: refactor to call secrets manager #1676

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions dataeng/resources/run-pipeline-acceptance-test.sh
View file
Open in desktop
Expand Up @@ -26,6 +26,7 @@ export COURSE_EXPORTER=$EXPORTER_BIN/course-exporter

# Exporter configuration destination

source secrets-manager.sh analytics-secure/analytics-exporter/task-auth.json TASK_AUTH
ROOT=${WORKSPACE}/analytics-secure/analytics-exporter
SECURE_HASH=`GIT_DIR=./analytics-secure/.git git rev-parse HEAD`
EXPORTER_CONFIG_BUCKET=$EXPORTER_BUCKET_PATH/$SECURE_HASH
Expand Down
21 changes: 21 additions & 0 deletions dataeng/resources/secrets-manager.sh
View file
Open in desktop
@@ -0,0 +1,21 @@
#!/usr/bin/env bash
secret_to_call="$1"
secret_name="$2"
set +x

SECRET_JSON=$(aws secretsmanager get-secret-value --secret-id $secret_to_call --region "us-east-1" --output json)
# Check the exit status of the AWS CLI command

echo "$SECRET_JSON"
extract_and_store_secret_value() {

value=$(echo "$SECRET_JSON" | jq -r ".SecretString | fromjson.$secret_name" 2>/dev/null)
eval "$secret_name"='$value'
}

if [ $? -eq 0 ]; then
# Use jq to extract the values from the JSON response
extract_and_store_secret_value $SECRET_JSON $secret_name
else
echo "AWS CLI command failed"
fi
6 changes: 6 additions & 0 deletions dataeng/resources/snowflake-collect-metrics.sh
View file
Open in desktop
Expand Up @@ -10,6 +10,12 @@ source "${PYTHON_VENV}/bin/activate"
cd $WORKSPACE/analytics-tools/snowflake
make requirements

source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_COLLECT_METRICS_JOB_EXTRA_VARS KEY_PATH
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_COLLECT_METRICS_JOB_EXTRA_VARS PASSPHRASE_PATH
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_COLLECT_METRICS_JOB_EXTRA_VARS USER
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_COLLECT_METRICS_JOB_EXTRA_VARS ACCOUNT
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the only var in the secret SNOWFLAKE_COLLECT_METRICS_JOB_EXTRA_VARS is SECURE_BRANCH so I am confused on this one.

source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_COLLECT_METRICS_JOB_EXTRA_VARS METRIC_NAME

python collect-metrics.py \
--metric_name $METRIC_NAME \
--key_path $WORKSPACE/analytics-secure/snowflake/rsa_key_snowflake_task_automation_user.p8 \
Expand Down
5 changes: 5 additions & 0 deletions dataeng/resources/snowflake-demographics-cleanup.sh
View file
Open in desktop
Expand Up @@ -10,6 +10,11 @@ source "${PYTHON_VENV}/bin/activate"
cd $WORKSPACE/analytics-tools/snowflake
make requirements

source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_DEMOGRAPHICS_CLEANUP_JOB_EXTRA_VARS KEY_PATH
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_DEMOGRAPHICS_CLEANUP_JOB_EXTRA_VARS PASSPHRASE_PATH
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_DEMOGRAPHICS_CLEANUP_JOB_EXTRA_VARS USER
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_DEMOGRAPHICS_CLEANUP_JOB_EXTRA_VARS ACCOUNT

python demographics_cleanup.py \
--key_path $WORKSPACE/analytics-secure/$KEY_PATH \
--passphrase_path $WORKSPACE/analytics-secure/$PASSPHRASE_PATH \
Expand Down
5 changes: 5 additions & 0 deletions dataeng/resources/snowflake-public-grants-cleaner.sh
View file
Open in desktop
Expand Up @@ -10,6 +10,11 @@ source "${PYTHON_VENV}/bin/activate"
cd $WORKSPACE/analytics-tools/snowflake
make requirements

source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_PUBLIC_GRANTS_CLEANER_JOB_EXTRA_VARS KEY_PATH
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_PUBLIC_GRANTS_CLEANER_JOB_EXTRA_VARS PASSPHRASE_PATH
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_PUBLIC_GRANTS_CLEANER_JOB_EXTRA_VARS USER
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_PUBLIC_GRANTS_CLEANER_JOB_EXTRA_VARS ACCOUNT

python snowflake_public_grants_cleaner.py \
--key_path $WORKSPACE/analytics-secure/$KEY_PATH \
--passphrase_path $WORKSPACE/analytics-secure/$PASSPHRASE_PATH \
Expand Down
11 changes: 11 additions & 0 deletions dataeng/resources/snowflake-refresh-snowpipe.sh
View file
Open in desktop
Expand Up @@ -10,6 +10,17 @@ source "${PYTHON_VENV}/bin/activate"
cd $WORKSPACE/analytics-tools/snowflake
make requirements

source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_REFRESH_SNOWPIPE_JOB_EXTRA_VARS KEY_PATH
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_REFRESH_SNOWPIPE_JOB_EXTRA_VARS PASSPHRASE_PATH
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_REFRESH_SNOWPIPE_JOB_EXTRA_VARS USER
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_REFRESH_SNOWPIPE_JOB_EXTRA_VARS ACCOUNT
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_REFRESH_SNOWPIPE_JOB_EXTRA_VARS SCHEMA
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_REFRESH_SNOWPIPE_JOB_EXTRA_VARS PIPE_NAME
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_REFRESH_SNOWPIPE_JOB_EXTRA_VARS TABLE_NAME
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_REFRESH_SNOWPIPE_JOB_EXTRA_VARS DELAY
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_REFRESH_SNOWPIPE_JOB_EXTRA_VARS LIMIT


python refresh_snowpipe.py \
--key_path $WORKSPACE/analytics-secure/$KEY_PATH \
--passphrase_path $WORKSPACE/analytics-secure/$PASSPHRASE_PATH \
Expand Down
5 changes: 5 additions & 0 deletions dataeng/resources/snowflake-user-retirement-status-cleanup.sh 100644 → 100755
View file
Open in desktop
Expand Up @@ -10,6 +10,11 @@ source "${PYTHON_VENV}/bin/activate"
cd $WORKSPACE/analytics-tools/snowflake
make requirements

source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_USER_RETIREMENT_STATUS_CLEANUP_JOB_EXTRA_VARS KEY_PATH
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_USER_RETIREMENT_STATUS_CLEANUP_JOB_EXTRA_VARS PASSPHRASE_PATH
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_USER_RETIREMENT_STATUS_CLEANUP_JOB_EXTRA_VARS USER
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_USER_RETIREMENT_STATUS_CLEANUP_JOB_EXTRA_VARS ACCOUNT

python retirement_cleanup.py \
--key_path $WORKSPACE/analytics-secure/$KEY_PATH \
--passphrase_path $WORKSPACE/analytics-secure/$PASSPHRASE_PATH \
Expand Down
5 changes: 5 additions & 0 deletions dataeng/resources/snowflake-validate-stitch.sh
View file
Open in desktop
Expand Up @@ -14,6 +14,11 @@ COMPARISON_START_TIME=$(date --utc --iso=minutes -d "${COMPARISON_END_TIME} - 15
cd $WORKSPACE/analytics-tools/snowflake
make requirements

source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_VALIDATE_STITCH_JOB_EXTRA_VARS KEY_PATH
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_VALIDATE_STITCH_JOB_EXTRA_VARS PASSPHRASE_PATH
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_VALIDATE_STITCH_JOB_EXTRA_VARS USER
source secrets-manager.sh analytics-secure/job-configs/SNOWFLAKE_VALIDATE_STITCH_JOB_EXTRA_VARS ACCOUNT
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added these here but i'm not sure they're needed/used


python stitch_vs_sqoop_validation.py \
--key_path $WORKSPACE/analytics-secure/${SNOWFLAKE_KEY_PATH} \
--passphrase_path $WORKSPACE/analytics-secure/${SNOWFLAKE_PASSPHRASE_PATH} \
Expand Down
5 changes: 5 additions & 0 deletions dataeng/resources/stitch-snowflake-lag-monitor.sh
View file
Open in desktop
Expand Up @@ -10,6 +10,11 @@ source "${PYTHON_VENV}/bin/activate"
cd $WORKSPACE/analytics-tools/snowflake
make requirements

source secrets-manager.sh analytics-secure/job-configs/STITCH_SNOWFLAKE_LAG_MONITOR_JOB_EXTRA_VARS KEY_PATH
source secrets-manager.sh analytics-secure/job-configs/STITCH_SNOWFLAKE_LAG_MONITOR_JOB_EXTRA_VARS PASSPHRASE_PATH
source secrets-manager.sh analytics-secure/job-configs/STITCH_SNOWFLAKE_LAG_MONITOR_JOB_EXTRA_VARS USER
source secrets-manager.sh analytics-secure/job-configs/STITCH_SNOWFLAKE_LAG_MONITOR_JOB_EXTRA_VARS ACCOUNT

python stitch-snowflake-monitoring.py \
--key_path $WORKSPACE/analytics-secure/$KEY_PATH \
--passphrase_path $WORKSPACE/analytics-secure/$PASSPHRASE_PATH \
Expand Down