Transparent encryption MD5 checksum error

[rajivml]

HI,

I am trying to enable transparent encryption by passing in the below configuration and the put upload is always failing with "MD5 Sums don't match"

    spec:
      containers:
      - env:
        - name: S3PROXY_CORS_ALLOW_ALL
          value: "true"
        - name: JCLOUDS_PROVIDER
          value: filesystem
        - name: S3PROXY_ENDPOINT
          value: http://0.0.0.0:8000
        - name: JCLOUDS_FILESYSTEM_BASEDIR
          value: /data
        - name: LOG_LEVEL
          value: debug
        - name: S3PROXY_ENCRYPTED_BLOBSTORE
          value: "true"
        - name: S3PROXY_ENCRYPTED_BLOBSTORE_PASSWORD
          value: "1234567890123456"
        - name: S3PROXY_ENCRYPTED_BLOBSTORE_SALT
          value: "12345678"

[root@server0 s3proxy]# /usr/local/bin/s3cmd put pom.xml --no-ssl --host=${AWS_HOST} --host-bucket=  s3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom.xml
upload: 'pom.xml' -> 's3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom.xml'  [1 of 1]
 17574 of 17574   100% in    0s   191.88 KB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of pom.xml
upload: 'pom.xml' -> 's3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom.xml'  [1 of 1]
 17574 of 17574   100% in    0s   287.90 KB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of pom.xml
upload: 'pom.xml' -> 's3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom.xml'  [1 of 1]
 17574 of 17574   100% in    0s   296.49 KB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of pom.xml
upload: 'pom.xml' -> 's3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom.xml'  [1 of 1]
 17574 of 17574   100% in    0s   334.40 KB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of pom.xml
upload: 'pom.xml' -> 's3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom.xml'  [1 of 1]
 17574 of 17574   100% in    0s   315.15 KB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of pom.xml
upload: 'pom.xml' -> 's3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom.xml'  [1 of 1]
 17574 of 17574   100% in    0s   342.04 KB/s  done
WARNING: MD5 Sums don't match!
WARNING: Too many failures. Giving up on 'pom.xml'
ERROR: Upload of 'pom.xml' failed too many times (Last reason: Too many failures. Giving up on 'pom.xml')

If I disable transparent encryption, then it works completely fine. Any idea what's going wrong ?

cc @FlorinPeter

[rajivml]

even passing --no-check-md5 isn't helping

[root@server0 ceph-s3-examples]# /usr/local/bin/s3cmd --help | grep md5
                        don't have matching size and md5.  Skips files/parts
                        that do.  Note: md5sum checks are not always
  --check-md5           Check MD5 sums when comparing files for [sync].
  --no-check-md5        Do not check MD5 sums when comparing files for [sync].
  --list-md5            Include MD5 sums in bucket listings (only for 'ls'
  Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overridden by options, see below)
[root@server0 ceph-s3-examples]#
[root@server0 ceph-s3-examples]#  /usr/local/bin/s3cmd put objects.txt --no-ssl --no-check-md5 --host=${AWS_HOST} --host-bucket=  s3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom1.xml --no-check-md5
upload: 'objects.txt' -> 's3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom1.xml'  [part 1 of 4, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    17.38 MB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of objects.txt
upload: 'objects.txt' -> 's3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom1.xml'  [part 1 of 4, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    20.40 MB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of objects.txt
upload: 'objects.txt' -> 's3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom1.xml'  [part 1 of 4, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    25.62 MB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of objects.txt
upload: 'objects.txt' -> 's3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom1.xml'  [part 1 of 4, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    27.27 MB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of objects.txt
upload: 'objects.txt' -> 's3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom1.xml'  [part 1 of 4, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    27.47 MB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of objects.txt
upload: 'objects.txt' -> 's3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom1.xml'  [part 1 of 4, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    26.38 MB/s  done
WARNING: MD5 Sums don't match!
WARNING: Too many failures. Giving up on 'objects.txt'
ERROR:
Upload of 'objects.txt' part 1 failed. Use
  /usr/local/bin/s3cmd abortmp s3://test-bucket/4c56d172-6b52-409b-9308-75f1e8dd6418/9f3e2c75-782a-4735-85f0-461714cbcba0/dataset/pom1.xml 0bb9f2e7-84b8-48a1-84b8-73cbf6903f47
to abort the upload, or
  /usr/local/bin/s3cmd --upload-id 0bb9f2e7-84b8-48a1-84b8-73cbf6903f47 put ...
to continue the upload.
ERROR: Upload of 'objects.txt' failed too many times (Last reason: Too many failures. Giving up on 'objects.txt')

another issue noticed while using transparent encryption is, failed multi-parts are not getting cleaned up automatically

[rajivml]

I was able to upload the files via python boto3 but not via s3cmd