Sprint 1.11 (#92)

* update gosdk * update gosdk * use multi op * set chunkNumber * update gosdk * update gosdk version * update gosdk * Adding cmd to use mc with zs3server. * add aws cli info too * minor nit * disable multi part in client-api layer. * update gosdk * go mod tidy * update gosdk * Revert go work sum * Install Herumi in base image * upgrade GoSDK to sprint-1.11 * create a tag with branch name * upgrade GoSDK to sprint-1.11 * Add multipart upload partially The files will be stored locally, the next step is to see how to process the data and forward to allocation via our SDK * Add README * pipeline for data upload * Support allocation upload * Upload chunkReadSize * Fix last chunk push * Fix data push * Use performance testing gosdk * add err check * Fix early retry * Integrate into minio gateway * fix path and chunk number * update remote name * Fix bucket path * Fix RemotePath * Fix remote path * Fix go.work * Fix go.mod * Fix multipart upload interface Should align with the interface signature * Fix more interfaces * Implement more methods: GetMultipartInfo and ListObjectParts * Use v1.mov for testing * Move seqpriorityqueue to gateway * Clean up upload temp files after uploaded * Test no object size * Update gosdk for logs * Update gosdk more logs * Init larger size * Change gosdk back * Upload after all parts are uploaded * Fix chunk writer * Fix uploading * Fix logs * Start uploading whenever last part is received * Fix chunk read size error * Fix end part * Fix duplicate channel close * Fix dup close channel for dataC * Fix multipart download * Fix already download * Log remove local path * Share download reader * Remove local temp file when all downloaded * Remove unnecessary get ref requests * Add download from blobbers log * Add upload time logs * Add more logs and increase the channel buffer size * Fix race condition for removing downloaded file * update gosdk to v1.11.0 --------- Co-authored-by: Manohar Reddy <b.manu199@gmail.com> Co-authored-by: Hitenjain14 <hitenjain146@gmail.com> Co-authored-by: Sunil Kumar <sunilkr.kumar26@gmail.com> Co-authored-by: Jayash Satolia <jayash.satolia.min19@iitbhu.ac.in> Co-authored-by: peterlimg <peterlimg@protonmail.com> Co-authored-by: boddumanohar <boddumanohar@users.noreply.github.com> Co-authored-by: service-0chain <service-0chain@users.noreply.github.com>
0chain · Dec 19, 2023 · a6f8d44 · a6f8d44
1 parent 4008772
commit a6f8d44
Show file tree

Hide file tree

Showing 23 changed files with 4,779 additions and 114 deletions.
diff --git a/.github/workflows/build-push-client-docker-image.yaml b/.github/workflows/build-push-client-docker-image.yaml
@@ -53,8 +53,10 @@ jobs:
             cd client-api
             docker build -t $MINIO_CLIENT_REGISTRY:latest .
             docker tag $MINIO_CLIENT_REGISTRY:latest $MINIO_CLIENT_REGISTRY:$TAG-$SHORT_SHA
+            docker tag $MINIO_CLIENT_REGISTRY:latest $MINIO_CLIENT_REGISTRY:$TAG
             docker push $MINIO_CLIENT_REGISTRY:$TAG-$SHORT_SHA
-        
+            docker push $MINIO_CLIENT_REGISTRY:$TAG
+
         - name: Push latest
           if: ${{ github.event.inputs.latest_tag == 'yes' || github.ref == 'refs/heads/master' }}
           run: |

diff --git a/.github/workflows/build-push-logsearchapi-docker-image.yaml b/.github/workflows/build-push-logsearchapi-docker-image.yaml
@@ -54,8 +54,10 @@ jobs:
             cd logsearchapi
             docker build -t $MINIO_LOGSEARCHAPI_REGISTRY:latest .
             docker tag $MINIO_LOGSEARCHAPI_REGISTRY:latest $MINIO_LOGSEARCHAPI_REGISTRY:$TAG-$SHORT_SHA
+            docker tag $MINIO_LOGSEARCHAPI_REGISTRY:latest $MINIO_LOGSEARCHAPI_REGISTRY:$TAG
             docker push $MINIO_LOGSEARCHAPI_REGISTRY:$TAG-$SHORT_SHA
-        
+            docker push $MINIO_LOGSEARCHAPI_REGISTRY:$TAG
+
         - name: Push latest
           if: ${{ github.event.inputs.latest_tag == 'yes' || github.ref == 'refs/heads/master' }}
           run: |

diff --git a/.github/workflows/build-push-minio-docker-image.yaml b/.github/workflows/build-push-minio-docker-image.yaml
@@ -64,7 +64,9 @@ jobs:
             docker build -t zbox_base -f docker.local/base.Dockerfile .
             docker build -t $MINIO_SERVER_REGISTRY:latest .
             docker tag $MINIO_SERVER_REGISTRY:latest $MINIO_SERVER_REGISTRY:$TAG-$SHORT_SHA
+            docker tag $MINIO_SERVER_REGISTRY:latest $MINIO_SERVER_REGISTRY:$TAG
             docker push $MINIO_SERVER_REGISTRY:$TAG-$SHORT_SHA
+            docker push $MINIO_SERVER_REGISTRY:$TAG
 
         - name: Push latest
           if: ${{ github.event.inputs.latest_tag == 'yes' || github.ref == 'refs/heads/master' }}

diff --git a/README.md b/README.md
@@ -64,16 +64,136 @@ docker-compose -f environment/docker-compose.yaml up -d
 ```
 3. Make sure allocation.txt file exist in the default folder ``~/.zcn``
 
-## Test MinIO Client 
+4. Now you can interact with the clint API follow this [doc](/client-api/README.md)
 
-MinIO client `mc` provides a modern alternative to UNIX commands such as ls, cat, cp, mirror, diff, etc. It supports filesystems and Amazon S3-compatible cloud storage services. To interact with the client API follow this [doc](/client-api/README.md). You can also interact with the log search API by following this [doc.](/logsearchapi/README.md)
+5. You can also interact with the logsearch API by following this [doc](/logsearchapi/README.md)
+
+
+## Test using AWS Client `aws`
+### Installation
+Install from here: https://aws.amazon.com/cli/
+
+### Configuration
+Fetch the access key and secret from your deployed zs3server. To configure `aws cli`, type `aws configure` and 
+specify the zs3server key information like below:
+```
+aws configure
+AWS Access Key ID [None]: miniouser
+AWS Secret Access Key [None]: miniopassword
+Default region name [None]: us-east-1
+Default output format [None]: ENTER
+```
+
+Additionally enable AWS Signature Version ‘4’ for zs3server.
+
+`aws configure set default.s3.signature_version s3v4`
+
+### Examples
+#### To list your buckets
+```
+aws --endpoint-url https://localhost:9000 s3 ls
+2016-03-27 02:06:30 deebucket
+2016-03-28 21:53:49 guestbucket
+2016-03-29 13:34:34 mbtest
+2016-03-26 22:01:36 mybucket
+2016-03-26 15:37:02 testbucket
+```
+
+#### To list contents inside bucket
+```
+aws --endpoint-url https://localhost:9000 s3 ls s3://mybucket
+2016-03-30 00:26:53      69297 argparse-1.2.1.tar.gz
+2016-03-30 00:35:37      67250 simplejson-3.3.0.tar.gz
+```
+
+#### To make a bucket
+```
+aws --endpoint-url https://localhost:9000 s3 mb s3://mybucket
+make_bucket: s3://mybucket/
+```
+
+#### To add an object to a bucket
+```
+aws --endpoint-url https://localhost:9000 s3 cp simplejson-3.3.0.tar.gz s3://mybucket
+upload: ./simplejson-3.3.0.tar.gz to s3://mybucket/simplejson-3.3.0.tar.gz
+```
+
+#### To delete an object from a bucket
+```
+aws --endpoint-url https://localhost:9000 s3 rm s3://mybucket/argparse-1.2.1.tar.gz
+delete: s3://mybucket/argparse-1.2.1.tar.gz
+```
+
+#### To remove a bucket
+```
+aws --endpoint-url https://localhost:9000 s3 rb s3://mybucket
+remove_bucket: s3://mybucket/
+```
+
+
+
+## Test using MinIO Client `mc`
+`mc` provides a modern alternative to UNIX commands such as ls, cat, cp, mirror, diff etc. It supports filesystems 
+and Amazon S3 compatible cloud storage services.
+
+### Installation
+Install from here for your os: https://min.io/docs/minio/macos/index.html
 
 ## Configure MinIO Client
 ```
 mc config host add zcn http://localhost:9000 miniouser miniopassword
-mc ls zcn //List your buckets
+```
+
+### Examples
+#### To list your buckets
+```
+mc ls zcn/
+2016-03-27 02:06:30 deebucket
+2016-03-28 21:53:49 guestbucket
+2016-03-29 13:34:34 mbtest
+2016-03-26 22:01:36 mybucket
+2016-03-26 15:37:02 testbucket
+```
+
+#### To list contents inside bucket
+```
+mc ls zcn/mybucket
+2016-03-30 00:26:53      69297 argparse-1.2.1.tar.gz
+2016-03-30 00:35:37      67250 simplejson-3.3.0.tar.gz
+```
+
+#### To make a bucket
+```
+mc mb zcn/mybucket
+make_bucket: zcn/mybucket
+```
+
+#### To add an object to a bucket
+```
+mc cp simplejson-3.3.0.tar.gz zcn/mybucket
+upload: ./simplejson-3.3.0.tar.gz to zcn/mybucket/simplejson-3.3.0.tar.gz
+```
+
+#### To delete an object from a bucket
+```
+mc rm zcn/mybucket/argparse-1.2.1.tar.gz
+delete: zcn/mybucket/argparse-1.2.1.tar.gz
+```
 
-[2017-02-22 01:50:43 PST]     0B user/
-[2017-02-26 21:43:51 PST]     0B datasets/
-[2017-02-26 22:10:11 PST]     0B assets/
+#### To remove a bucket
+```
+mc rb zcn/mybucket
+remove_bucket: zcn/mybucket/
 ```
+
+Check `mc --help` for the exhaustive list of cmds available.
+
+
+## Test using Postman Using REST APIs
+- Add the following authorization settings
+![](./assets/postman-auth.png "Authorization settings")
+- The `AccessKey` would be the MINIO_ROOT_USER which you set earlier during zs3server deployment and `SecretKey` would be the MINIO_ROOT_PASSWORD.
+- If you do not want to share the MINIO_ROOT_USER and MINIO_ROOT_PASSWORD, you can also create a user from minio console and share their access key and secret instead.
+- Use the REST APIs to interact with the server. 
+- Postman collection for the same is provided below:
+[Postman Collection](./assets/Zs3ServerCollection.postman_collection.json)