From e3512192b263e1e5ba5ed3dae10a9ada33a58e6b Mon Sep 17 00:00:00 2001 From: Travis Wichert Date: Mon, 6 Nov 2017 12:01:53 -0500 Subject: [PATCH] release-2.5.1 (#381) * Update ECS CE Docker image with ECS 3.0.0 Hotfix 2 (#312) * Reorg base files (fix my glitch) * Fixed Dockerfile so it patches everything in-place. (cherry picked from commit 867189eb92e36c8388d389273f2942b9f277380e) * update release files to use new image. * Duct tape for #301 Update to ECS 3.0.0.2 (3.0.0 HF 2) (#314) * fix a small typo * fix package install issue because EPEL is between versions again. * quick fixes for crashing HF2 to unblock clients * Docs update (#313) * Adds FAQ page * installation troubleshooting * formatting * Addition of network troubleshooting * Addition of network troubleshooting * More Troubleshooting * sidebar implementation attempt * implements important links dropdown, disables page dropdown. * whoops * Adds migration page and some small updates. * Bunch of docs updates * bugfix-hf2 (#315) * multitail and dstat are coming in handy right now * change fact cache location to /var/cache/emc/ecs-install * log the state of Docker at the end of bootstrapping for troubleshooting help (those hashes are good to see!) * [Ansible] Stop templating, start regex replacing props/confs * release prep 2.3.0 (#316) * OVA prep (#318) * Configure Jenkins pipeline to test installation process * Get repository information from Jenkins SCM config * Fix env var * Obtain TF options from Jenkins params. Moved deprovision step to post action * Allow multi-node configuration * Update checkout step in jenkinsfile * add zerofill.sh to /tools (#324) (cherry picked from commit 58f7e8eea587c5a005eff3df2c2733ac2f5e1a9c) * Add slack notifications to jenkins pipeline * [WiP] Configure Jenkins server to build PR and provide feedback (#328) Configure Jenkins server to build PR and provide feedback * Docs pass 2 (#326) * Removal of deprecated procedures * Templates * OVA install guide added * Fixed broken links * preflight remove all bootstrap packages if installed (#330) * Non-PR Jenkins jobs do not provide URL and commit author (#331) Fix variables in Slack notifications * put yum actions in retry loop with timeout (#332) Put yum actions in retry loop with timeout * Implements #205 Installer must have public key initial auth capability (#270) * ECS-CommunityEdition-205 Installer must have public key initial auth capability (cherry picked from commit 6eea10b5db3985f960d7b313d2e705a0f913ba55) * More sausage for the initial ssh key auth (cherry picked from commit 8535ccb5430e89b79d253ea1e74390a39b8b20f3) * more sausage (cherry picked from commit edf961e0765cd9a06ccea1a6d1e2406816533f46) * deploy.yml change ideas (cherry picked from commit ef48e2cc57fa6d0a57aa30bc62ba816fc167aed9) * bootstrap.sh modifications (cherry picked from commit d0b3c630f0a2004fe23534aca7e4a95986dce383) * bootstrap.sh modifications (cherry picked from commit 86f897af9395a57af5b33c2162cd85422b4e6ded) * move generic help to generic_help.j2.yml file from config.yml * include shipit.lib.sh * build install paths early add copy action for ssh PKI material * fix a couple gitopt bugs * add create_install_tree() to plugin-defaults.sh * copy ssh keys in bootstrap.sh * more longopts adjustments * add loop delay in retry_with_timeout() * stop trying to autoremove curl, it'll always error. * key_vals need basename not full path set 0700 bits on ssh/ssl stores * remove optarg debugging * more ssh pubkey sausage * update reference.deploy.yml to include feature * jenkins changes * jenkins changes * jenkins changes * jenkins changes * jenkins changes * bump versions and move OVA download links. (#335) * open-vm-tools now has a cross dependency (#337) with open-vm-tools-desktop and yum fails to install open-vm-tools on remote nodes when open-vm-tools-desktop is not installed. * ECS-CommunityEdition-317 Make `ecsconfig ping -cx` loop when dtquery fails (#344) (cherry picked from commit f5c7810a3385352a7ceb3bd9af66f9f824927dca) * Change the way ecs-install is pushed to repo (#346) * invoke zerofill via bash rather than expecting exec bit (#343) * Remove Ansible verbosity flag from Jenkinsfile (#350) * OVA QoL Improvements (#351) * add `ova-step1` and `ova-step2` macros * add `ecsdeploy noop` for some ova macros to look better * make videploy more intelligent and play nice with update_deploy * Implement Ansible global OVA flag fact (#349) * implements Ansible global ova flag fact - custom fact in /etc/ansible/facts.d/ova.fact - ova conditional flags in playbooks * misaligned `when` * skip rebooting when using the OVA. * Upgrade Ansible to 2.3 (#347) * install ansible package from @edge_main for 2.3 * Ansible changes for Ansible 2.3 * ECS-CommunityEdition-235 Bump Ansible version to 2.3 * refactor Ansible task `when:` clauses to Ansible 2.3 spec * refactor node reboot actions for Ansible 2.3 Also resolve #342 * remove unused json_file plugin * must ignore_errors: True `needs-restarting -r` * refactor port-check `when`s to Ansible 2.3 spec * cleanup * add loop_control to path permissions entries * add loop_control labels to many iterators * add loop_control labels to many iterators * break out one directive per line * add loop_control labels to many iterators * incorrect `when` * speling * Switch to Alpine 3.6 release (#359) * Switch to Alpine 3.6 release Install Python 2 from APK * Changes to Rockerfile for python:2-alpine parity * Split steps out from Ansible to get realtime console logging (#358) * Split steps out from Ansible to get realtime console logging * use /tmp? * template out a script to run command on install node via IP * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * log environment info to file log only, never to console. * Add CentOS 7.4 support (#360) (cherry picked from commit 1a960873d5d6f0b1011caabcbb3987114723dd8a) * [WiP] Misc. 2.5.0 bugfixes (#352) * update reference deploy version * bugfix typo in ed25519 private key filename * fix ova flag implementation * fix ova flag implementation round 2 * fix ova flag implementation round 3 * update entrypoint.sh * bump version to 2.5.0b1 (#364) * [WiP] Update ecs-install Python requirements (#356) * update python requirements * pin python requirements to major versions rather than patches. * add python2-dev to temporary build environment * need cryptography>=1.9 * [WiP] ECS 3.1.0.0 Reduced GA and CE Support (#353) * make 3.0.0.2 to use 100% regex (cherry picked from commit 0689ce59ec9e61beea4082b1f23222031b17f62d) * prep 3.1.0.0 RC3 (cherry picked from commit 2c3d5b9e92bcec83e01362c674a00158e6a61289) * prep 3.1.0.0 RC3 (cherry picked from commit ea12c296204442adf1e5079d778171140dede101) * ECS 3.1 templates * local facts must be fully qualified? * interface roles should be defined in deploy.yml * actually use a comma in the jinja joiner() func * use ansible_fqdn for agent strings not ansible_hostname * ECS 3.1.0.0 RC4 * joiner() needs to be the prefix not the suffix * the infamous missing comma * no trailing comma * remove redundant spaces * set host: field in testing * make object-main_network.json.j2 VDC-aware + formatting * Set georeceiver initialBufferNumOnHeap to 10 * Mount /usr instead of /usr/local to capture new install path * [WiP] ECS 3.1 Full GA and CE support (#367) * Use nodeId instead of the node IP to create data store * Fix errors getting node ID * fix 3.1 patch again * migrate cm.object.properties/'MustHaveEnoughResources=false' into Dockerfile * Run cf_client in container for new low partition count vars * Run cf_client in container for new low partition count vars * migrate cf_client variable settings into Dockerfile * update comments in Dockerfile for 3.1.0.0 * release-2.5.0-prep (#370) * Update ECS-Installation.md (cherry picked from commit f8be70f53b55bf718e2f1bb32df206484a12e7e8) * Update ECS-Installation.md (cherry picked from commit b479b0722308aaa40345af524f5e4430d29b11ed) * bump versions * Check if object user is editable before continuing to S3 credential provisioning (#376) * Fix #371 * Cosmetic change to fix output * Add retry loops around funcs to add credentials * add uniqueness to hostnames of CI VMs (#378) * Enable build-help in bootstrap.sh (#377) * enable build help in bootstrap.sh * enable build help in bootstrap.sh * Switch to upstream ECS version 3.1.0.1 (#375) * Add Dockerfile ecs-object-reduced to CE image patch * bump ecs version * bump version files for install node 2.5.1 (#379) * Develop (#372) (#380) backmerge --- bootstrap.sh | 11 ++++-- docs/source/building/development-build.md | 10 +++++ patches/3.1.0.1/Dockerfile | 23 +++++++++++ tests/lab.tf | 4 +- ui/ecsconfig.py | 48 +++++++++++++++++------ ui/etc/config.yml | 4 +- ui/etc/release.conf | 6 +-- ui/setup.py | 2 +- 8 files changed, 86 insertions(+), 22 deletions(-) create mode 100644 docs/source/building/development-build.md create mode 100644 patches/3.1.0.1/Dockerfile diff --git a/bootstrap.sh b/bootstrap.sh index 7d82eec8..0013c20c 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -102,8 +102,9 @@ cat < + Use the Alpine Linux mirror at to build the ecs-install image locally. + Mirror list: https://wiki.alpinelinux.org/wiki/Alpine_Linux:Mirrors For additional information, read the docs on GitHub. For additional help, please open an issue on GitHub. @@ -181,7 +182,7 @@ fi if ! O=$( getopt \ - -l build-from:,deploy-config:,registry-cert:,help,proxy-cert:,registry-login,centos-mirror:,override-dns:,proxy-endpoint:,registry-endpoint:,proxy-test-via:,vm-tools,zero-fill-ova,ssh-private-key:,ssh-public-key:,version,help-build,yes,no,verbose,quiet \ + -l build-from:,deploy-config:,registry-cert:,help,help-build,proxy-cert:,registry-login,centos-mirror:,override-dns:,proxy-endpoint:,registry-endpoint:,proxy-test-via:,vm-tools,zero-fill-ova,ssh-private-key:,ssh-public-key:,version,help-build,yes,no,verbose,quiet \ -o c:d:hk:lm:no:r:t:p:gyvqz \ -n "${0}" \ -- ${@} @@ -199,6 +200,10 @@ while true; do usage exit 1 ;; + --help-build) + usage_build + exit 1 + ;; --version) shift version diff --git a/docs/source/building/development-build.md b/docs/source/building/development-build.md new file mode 100644 index 00000000..8d84f631 --- /dev/null +++ b/docs/source/building/development-build.md @@ -0,0 +1,10 @@ +# ECS Community Edition Installation + +## Building from develop branch + +### Prerequisites +[same as standard install] + +### Steps + + diff --git a/patches/3.1.0.1/Dockerfile b/patches/3.1.0.1/Dockerfile new file mode 100644 index 00000000..3c77f992 --- /dev/null +++ b/patches/3.1.0.1/Dockerfile @@ -0,0 +1,23 @@ +# Fixes to the default 3.1.0.1 reduced image. + +# Build on object-reduced image (GA release) +FROM emcvipr/object:3.1.0.1-95464.df479c1-reduced + +# Increase memory for transformsvc +RUN sed -i s/-Xmx128m/-Xmx512m/ /opt/storageos/bin/transformsvc + +# Fix disk partitioning script +RUN sed -i '/VMware/ s/$/ \&\& [ ! -e \/data\/is_community_edition ]/' /opt/storageos/bin/storageserver-partition-config.sh +RUN /usr/bin/chmod +x /opt/storageos/bin/storageserver-partition-config.sh + +# Set VNets useSeperateThreadPools to True +RUN f=/opt/storageos/conf/vnest-common-conf-template.xml; grep -q "object.UseSeparateThreadPools" $f || sed -i '/properties id="serviceProperties"/a \ \ \ \ \ \ \ \ true' $f + +# Set georeceiver's initialBufferNumOnHeap to something smaller for CE +RUN f=/opt/storageos/conf/georeceiver-conf.xml; grep -q 'name="initialBufferNumOnHeap" value="10"' $f || sed -i 's/name="initialBufferNumOnHeap" value="60"/name="initialBufferNumOnHeap" value="10"/' $f + +# Configure CM Object properties: Disable minimum storage device count +RUN f=/opt/storageos/conf/cm.object.properties; grep -q 'MustHaveEnoughResources=false' $f || sed -i 's/MustHaveEnoughResources=true/MustHaveEnoughResources=false/' $f + +# Allow allocation of different blocks of a chunk to be stored on the same partition +RUN sed -i 's###g' /opt/storageos/conf/ssm-cf-conf.xml diff --git a/tests/lab.tf b/tests/lab.tf index ac0c5edb..b683ca8d 100644 --- a/tests/lab.tf +++ b/tests/lab.tf @@ -13,7 +13,7 @@ resource "vsphere_folder" "tests_folder" { } resource "vsphere_virtual_machine" "install_node" { - name = "jenkins-ecsce-install" + name = "jenkins-ecsce-install-${substr(sha1(timestamp()),0,8)}" folder = "${vsphere_folder.tests_folder.path}" vcpu = 2 memory = 4096 @@ -34,7 +34,7 @@ resource "vsphere_virtual_machine" "install_node" { resource "vsphere_virtual_machine" "ecs_node" { count = "${var.ecs_nodes}" - name = "jenkins-ecsce-ecs-${count.index}" + name = "jenkins-ecsce-ecs-${count.index}-${substr(sha1(timestamp()),0,8)}" folder = "${vsphere_folder.tests_folder.path}" vcpu = 4 memory = 16384 diff --git a/ui/ecsconfig.py b/ui/ecsconfig.py index f10023b9..f09d8cbc 100755 --- a/ui/ecsconfig.py +++ b/ui/ecsconfig.py @@ -1124,18 +1124,44 @@ def add_one(name): o("Creating '{}' in namespace '{}'".format(name, ou_namespace)) conf.api_client.object_user.create(name, namespace=ou_namespace) - o("\tAdding {}'s S3 credentials".format(name)) - conf.api_client.secret_key.create(user_id=name, - namespace=ou_namespace, - expiry_time=ou_dict['s3_expiry_time'], - secret_key=ou_dict['s3_secret_key']) - - if ou_dict['swift_password'] is not None and ou_dict['swift_groups_list'] is not None: - o("\tAdding {}'s Swift credentials".format(name)) - conf.api_client.password_group.create(user_id=name, + o("\tWaiting for '{}' to become editable".format(name)) + is_editable = False + while is_editable is False: + try: + get_one(name) + is_editable = True + o("\tOK") + except Exception as e: + is_editable = False + o("\tWaiting...") + time.sleep(5) + + creds_added = False + while creds_added is False: + try: + o("\tAdding {}'s S3 credentials".format(name)) + conf.api_client.secret_key.create(user_id=name, namespace=ou_namespace, - password=ou_dict['swift_password'], - groups_list=ou_dict['swift_groups_list']) + expiry_time=ou_dict['s3_expiry_time'], + secret_key=ou_dict['s3_secret_key']) + creds_added = True + except Exception as e: + creds_added = False + time.sleep(5) + + creds_added = False + while creds_added is False: + try: + if ou_dict['swift_password'] is not None and ou_dict['swift_groups_list'] is not None: + o("\tAdding {}'s Swift credentials".format(name)) + conf.api_client.password_group.create(user_id=name, + namespace=ou_namespace, + password=ou_dict['swift_password'], + groups_list=ou_dict['swift_groups_list']) + creds_added = True + except Exception as e: + creds_added = False + time.sleep() available_configs = list_all() if l: diff --git a/ui/etc/config.yml b/ui/etc/config.yml index bd135571..96aa33d2 100644 --- a/ui/etc/config.yml +++ b/ui/etc/config.yml @@ -13,7 +13,7 @@ --- ui: name: ECS Community Edition Install Node - version: 2.5.0 + version: 2.5.1 host_root_dir: /opt/emc/ecs-install state_file: /opt/state.yml deploy_file: /opt/deploy.yml @@ -32,7 +32,7 @@ ui: ffx_sem: /opt/ffx.sem product: name: ECS - version: 3.1.0.0 + version: 3.1.0.1 vendor: Dell EMC flavor: Community Edition slogan: Free and Frictionless diff --git a/ui/etc/release.conf b/ui/etc/release.conf index f1861a87..698f33ec 100644 --- a/ui/etc/release.conf +++ b/ui/etc/release.conf @@ -8,10 +8,10 @@ # it is provided by or on behalf of EMC. release_name="ECS Community Edition" -release_version="3.1.0.0" +release_version="3.1.0.1" release_product="ECS Software" release_artifact="emccorp/ecs-software-3.1.0" -release_tag="latest" +release_tag="hf1" release_common_name="emccorp/ecs-software:latest" docker_host_root="/opt/emc/ecs-install" @@ -33,7 +33,7 @@ image_name='ecs-install' tag='latest' ver_maj='2' ver_min='5' -ver_rev='0' +ver_rev='1' ver_tag='r' serial=0 diff --git a/ui/setup.py b/ui/setup.py index 9eaffd5d..22ae65ba 100755 --- a/ui/setup.py +++ b/ui/setup.py @@ -3,7 +3,7 @@ setup( name='ecsdeploy', - version='2.5.0', + version='2.5.1', packages=find_packages(), scripts=['ui.py', 'ecsdeploy.py',