Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: fix refresh job #1758

Merged
merged 2 commits into from May 17, 2024
Merged

chore: fix refresh job #1758

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
78adc2d
chore: fix refresh job
katebygrace Apr 16, 2024
8c2d2fe
chore: revert secrets manager changes
katebygrace May 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 0 additions & 1 deletion dataeng/jobs/analytics/SnowflakeRefreshSnowpipe.groovy
View file
Open in desktop
Expand Up @@ -82,7 +82,6 @@ class SnowflakeRefreshSnowpipe {
}
publishers common_publishers(allVars)
steps {
shell(dslFactory.readFileFromWorkspace('dataeng/resources/secrets-manager-setup.sh'))
shell(dslFactory.readFileFromWorkspace('dataeng/resources/snowflake-refresh-snowpipe.sh'))
}
}
Expand Down
2 changes: 1 addition & 1 deletion dataeng/resources/secrets-manager.sh 100755 → 100644
View file
Open in desktop
Expand Up @@ -35,4 +35,4 @@ else
exit 1
fi
fetch_specific_key "$1" "$2"
fi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd eventually remove this right ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes!

fi
22 changes: 12 additions & 10 deletions dataeng/resources/snowflake-refresh-snowpipe.sh
View file
Open in desktop
Expand Up @@ -10,16 +10,14 @@ source "${PYTHON_VENV}/bin/activate"
cd $WORKSPACE/analytics-tools/snowflake
make requirements

# Source the secrets-manager.sh script to make the function available
source $WORKSPACE/secrets-manager.sh
# Fetch the secrets from AWS
set +x


secrets-manager.sh -w analytics-secure/job-configs/SNOWFLAKE_REFRESH_SNOWPIPE_JOB_EXTRA_VARS snowflake/rsa_key_snowpipe_user.p8
secrets-manager.sh -w analytics-secure/job-configs/SNOWFLAKE_REFRESH_SNOWPIPE_JOB_EXTRA_VARS snowflake/rsa_key_passphrase_snowpipe_user
python3 secrets-manager.py -w -n analytics-secure/snowflake/rsa_key_snowpipe_user.p8 -v rsa_key_snowpipe_user
python3 secrets-manager.py -w -n analytics-secure/snowflake/rsa_key_passphrase_snowpipe_user -v rsa_key_passphrase_snowpipe_user
#set -x

set -x
unset KEY_PATH
unset PASSPHRASE_PATH

python refresh_snowpipe.py \
--user 'SNOWPIPE' \
Expand All @@ -28,6 +26,10 @@ python refresh_snowpipe.py \
--pipe_name $PIPE_NAME \
--table_name $TABLE_NAME \
--delay $DELAY \
--limit $LIMIT
--key_file $KEY_PATH \
--passphrase_file $PASSPHRASE_PATH
--limit $LIMIT \
--key_file "$(cat "rsa_key_snowpipe_user")" \
--passphrase_file "$(cat "rsa_key_passphrase_snowpipe_user")"

rm rsa_key_snowpipe_user
rm rsa_key_passphrase_snowpipe_user