Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
hueiyuan committed Jun 14, 2022
1 parent 4731bb6 commit 7d56693
Showing 1 changed file with 19 additions and 0 deletions.
19 changes: 19 additions & 0 deletions README.md
Expand Up @@ -11,3 +11,22 @@ Step2. Execute setup.py install to install custom helpers. (> python3.8)
```
python3 setup.py install
```

Step3. Update related s3 path
Because this example emulate data which stored on AWS S3, need to update related bucket and prefix object in pyspark_main.py

Step4. Generate corresponding great_expectation suite json
```
python3 generate_expectation_suite.py --environment develop
```

Step5. Execute Spark main script
```
python3 pyspark_main.py --environment develop
```

## What is the next?
Becuase the repo just is a example, if you need to fork or refercence this module. Please refer to related document to modify.

## Reference
* [greate_expectations document](https://docs.greatexpectations.io/docs/)

0 comments on commit 7d56693

Please sign in to comment.