Skip to content

Commit

Permalink
Merge pull request #825 from zinggAI/awsdocs
Browse files Browse the repository at this point in the history
aws-s3 coming blank on site and amazons3 not coming , so combining them
  • Loading branch information
sonalgoyal committed May 1, 2024
2 parents bb109e1 + 0f04c6c commit cb52bbd
Show file tree
Hide file tree
Showing 2 changed files with 23 additions and 25 deletions.
25 changes: 0 additions & 25 deletions docs/connectors/amazons3.md

This file was deleted.

23 changes: 23 additions & 0 deletions docs/connectors/aws-s3.md
@@ -1,2 +1,25 @@
# AWS S3

1. Set a bucket e.g. zingg28032023 and a folder inside it e.g. zingg

2. Create aws access key and export via env vars (ensure that the user with below keys has read/write access to above):

export AWS_ACCESS_KEY_ID=<access key id>
export AWS_SECRET_ACCESS_KEY=<access key>

(if mfa is enabled AWS_SESSION_TOKEN env var would also be needed )

3. Download hadoop-aws-3.1.0.jar and aws-java-sdk-bundle-1.11.271.jar via maven

4. Set above in zingg.conf :
spark.jars=/<location>/hadoop-aws-3.1.0.jar,/<location>/aws-java-sdk-bundle-1.11.271.jar

5. Run using:

./scripts/zingg.sh --phase findTrainingData --properties-file config/zingg.conf --conf examples/febrl/config.json --zinggDir s3a://zingg28032023/zingg
./scripts/zingg.sh --phase label --properties-file config/zingg.conf --conf examples/febrl/config.json --zinggDir s3a://zingg28032023/zingg
./scripts/zingg.sh --phase train --properties-file config/zingg.conf --conf examples/febrl/config.json --zinggDir s3a://zingg28032023/zingg
./scripts/zingg.sh --phase match --properties-file config/zingg.conf --conf examples/febrl/config.json --zinggDir s3a://zingg28032023/zingg

6. Models etc. would get saved in
Amazon S3 > Buckets > zingg28032023 >zingg > 100

0 comments on commit cb52bbd

Please sign in to comment.