Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Elastic beanstalk provider creates truncated zip files #1180

Closed
alexjurkiewicz opened this issue Mar 9, 2020 · 3 comments · May be fixed by #1181
Closed

Elastic beanstalk provider creates truncated zip files #1180

alexjurkiewicz opened this issue Mar 9, 2020 · 3 comments · May be fixed by #1181
Labels
stale

Comments

@alexjurkiewicz
Copy link
Contributor

alexjurkiewicz commented Mar 9, 2020 •
edited

Example build: https://travis-ci.com/kounta/kounta/jobs/295835426

We're using dpl v2 (edge):

    - stage: Build
      name: Deploy to test environment Elastic Beanstalk
      script: skip
      before_deploy:
        - config/travis-deploy-prepare.sh
      deploy:
        - provider: elasticbeanstalk
          edge: true # opt in to dpl v2
          app: xxx
          env: Whatever # irrelevant, only_create_app_version is true
          only_create_app_version: true
          bucket: yyy
          bucket_path: xxx
          label: $TRAVIS_VERSION
          description: "Commit: $TRAVIS_COMMIT. Branch: $TRAVIS_BRANCH."

I ran the following command in a debug Travis CI session:

dpl elasticbeanstalk --access_key_id AAA --secret_access_key BBB --env Whatever --bucket yyy --app 'xxx' --only_create_app_version --bucket_path 'xxx' --region us-west-2 --debug

The output was as expected (~130k "Adding x" lines).

Then testing the zip file:

$ unzip -tq travis-<sha>-1583721278.zip
error:  expected central file header signature not found (file #138912).
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)
At least one error was detected in travis-<sha>-1583721278.zip.

$ ls -lh travis-<sha>-1583721278.zip
-rw-rw-r-- 1 travis travis 331M Mar  9 02:35 travis-<sha>-1583721278.zip

$ file travis-<sha>-1583721278.zip
travis-<sha>-1583721278.zip: Zip archive data, at least v2.0 to extract

Any ideas?
@alexjurkiewicz
Copy link
Contributor Author

alexjurkiewicz commented Mar 9, 2020 •
edited

It seems to be caused by generating zip files with > 65535 files. I've tested with this script in the Travis debug session

require 'pathspec'
require 'zip'

def files
  files = Dir.glob('**/*', File::FNM_DOTMATCH)
  files = filter(files, '.ebignore')
  files
end

def filter(files, spec)
  spec = PathSpec.from_filename(spec)
  files.reject { |file| spec.match(file) }
end

# files.each do |f|
  # puts f
# end

def cwd
  @cwd ||= "#{Dir.pwd}/"
end

def create_zip
  ::Zip::File.open('test.zip', ::Zip::File::CREATE) do |zip|
    files.slice(0, 65539).each do |path|
      zip.add(path.sub(cwd, ''), path)
    end
  end
end

create_zip

And this command:

rm test.zip ; time ruby test.rb ; unzip -tq test.zip

65534 works, 65539 fails.

@alexjurkiewicz
Copy link
Contributor Author

It seems like you need to enable zip64:
https://en.wikipedia.org/wiki/Zip_(file_format)#ZIP64
https://github.com/rubyzip/rubyzip#zip64-support

It's enabled for all application versions uploaded by the official AWS EB CLI:
https://github.com/aws/aws-elastic-beanstalk-cli/blob/88fd9f38d8bae7296e7ba0331297a1980fd32665/ebcli/core/fileoperations.py#L406

alexjurkiewicz added a commit to alexjurkiewicz/dpl that referenced this issue Mar 9, 2020
@alexjurkiewicz
Use ZIP64 for Elastic Beanstalk versions
4aca74b 
Fixes travis-ci#1180.
@alexjurkiewicz alexjurkiewicz mentioned this issue Mar 9, 2020
Open
@stale
Copy link

stale bot commented Jun 7, 2020

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

@stale stale bot added the stale label Jun 7, 2020
@stale stale bot closed this as completed Jun 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use ZIP64 for Elastic Beanstalk versions alexjurkiewicz/dpl
1 participant
@alexjurkiewicz