release-2.5.1 (#381)

* Update ECS CE Docker image with ECS 3.0.0 Hotfix 2 (#312) * Reorg base files (fix my glitch) * Fixed Dockerfile so it patches everything in-place. (cherry picked from commit 867189e) * update release files to use new image. * Duct tape for #301 Update to ECS 3.0.0.2 (3.0.0 HF 2) (#314) * fix a small typo * fix package install issue because EPEL is between versions again. * quick fixes for crashing HF2 to unblock clients * Docs update (#313) * Adds FAQ page * installation troubleshooting * formatting * Addition of network troubleshooting * Addition of network troubleshooting * More Troubleshooting * sidebar implementation attempt * implements important links dropdown, disables page dropdown. * whoops * Adds migration page and some small updates. * Bunch of docs updates * bugfix-hf2 (#315) * multitail and dstat are coming in handy right now * change fact cache location to /var/cache/emc/ecs-install * log the state of Docker at the end of bootstrapping for troubleshooting help (those hashes are good to see!) * [Ansible] Stop templating, start regex replacing props/confs * release prep 2.3.0 (#316) * OVA prep (#318) * Configure Jenkins pipeline to test installation process * Get repository information from Jenkins SCM config * Fix env var * Obtain TF options from Jenkins params. Moved deprovision step to post action * Allow multi-node configuration * Update checkout step in jenkinsfile * add zerofill.sh to /tools (#324) (cherry picked from commit 58f7e8eea587c5a005eff3df2c2733ac2f5e1a9c) * Add slack notifications to jenkins pipeline * [WiP] Configure Jenkins server to build PR and provide feedback (#328) Configure Jenkins server to build PR and provide feedback * Docs pass 2 (#326) * Removal of deprecated procedures * Templates * OVA install guide added * Fixed broken links * preflight remove all bootstrap packages if installed (#330) * Non-PR Jenkins jobs do not provide URL and commit author (#331) Fix <null> variables in Slack notifications * put yum actions in retry loop with timeout (#332) Put yum actions in retry loop with timeout * Implements #205 Installer must have public key initial auth capability (#270) * ECS-CommunityEdition-205 Installer must have public key initial auth capability (cherry picked from commit 6eea10b5db3985f960d7b313d2e705a0f913ba55) * More sausage for the initial ssh key auth (cherry picked from commit 8535ccb5430e89b79d253ea1e74390a39b8b20f3) * more sausage (cherry picked from commit edf961e0765cd9a06ccea1a6d1e2406816533f46) * deploy.yml change ideas (cherry picked from commit ef48e2cc57fa6d0a57aa30bc62ba816fc167aed9) * bootstrap.sh modifications (cherry picked from commit d0b3c630f0a2004fe23534aca7e4a95986dce383) * bootstrap.sh modifications (cherry picked from commit 86f897af9395a57af5b33c2162cd85422b4e6ded) * move generic help to generic_help.j2.yml file from config.yml * include shipit.lib.sh * build install paths early add copy action for ssh PKI material * fix a couple gitopt bugs * add create_install_tree() to plugin-defaults.sh * copy ssh keys in bootstrap.sh * more longopts adjustments * add loop delay in retry_with_timeout() * stop trying to autoremove curl, it'll always error. * key_vals need basename not full path set 0700 bits on ssh/ssl stores * remove optarg debugging * more ssh pubkey sausage * update reference.deploy.yml to include feature * jenkins changes * jenkins changes * jenkins changes * jenkins changes * jenkins changes * bump versions and move OVA download links. (#335) * open-vm-tools now has a cross dependency (#337) with open-vm-tools-desktop and yum fails to install open-vm-tools on remote nodes when open-vm-tools-desktop is not installed. * ECS-CommunityEdition-317 Make `ecsconfig ping -cx` loop when dtquery fails (#344) (cherry picked from commit f5c7810) * Change the way ecs-install is pushed to repo (#346) * invoke zerofill via bash rather than expecting exec bit (#343) * Remove Ansible verbosity flag from Jenkinsfile (#350) * OVA QoL Improvements (#351) * add `ova-step1` and `ova-step2` macros * add `ecsdeploy noop` for some ova macros to look better * make videploy more intelligent and play nice with update_deploy * Implement Ansible global OVA flag fact (#349) * implements Ansible global ova flag fact - custom fact in /etc/ansible/facts.d/ova.fact - ova conditional flags in playbooks * misaligned `when` * skip rebooting when using the OVA. * Upgrade Ansible to 2.3 (#347) * install ansible package from @edge_main for 2.3 * Ansible changes for Ansible 2.3 * ECS-CommunityEdition-235 Bump Ansible version to 2.3 * refactor Ansible task `when:` clauses to Ansible 2.3 spec * refactor node reboot actions for Ansible 2.3 Also resolve #342 * remove unused json_file plugin * must ignore_errors: True `needs-restarting -r` * refactor port-check `when`s to Ansible 2.3 spec * cleanup * add loop_control to path permissions entries * add loop_control labels to many iterators * add loop_control labels to many iterators * break out one directive per line * add loop_control labels to many iterators * incorrect `when` * speling * Switch to Alpine 3.6 release (#359) * Switch to Alpine 3.6 release Install Python 2 from APK * Changes to Rockerfile for python:2-alpine parity * Split steps out from Ansible to get realtime console logging (#358) * Split steps out from Ansible to get realtime console logging * use /tmp? * template out a script to run command on install node via IP * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * Jenkinsfile sausage * log environment info to file log only, never to console. * Add CentOS 7.4 support (#360) (cherry picked from commit 1a96087) * [WiP] Misc. 2.5.0 bugfixes (#352) * update reference deploy version * bugfix typo in ed25519 private key filename * fix ova flag implementation * fix ova flag implementation round 2 * fix ova flag implementation round 3 * update entrypoint.sh * bump version to 2.5.0b1 (#364) * [WiP] Update ecs-install Python requirements (#356) * update python requirements * pin python requirements to major versions rather than patches. * add python2-dev to temporary build environment * need cryptography>=1.9 * [WiP] ECS 3.1.0.0 Reduced GA and CE Support (#353) * make 3.0.0.2 to use 100% regex (cherry picked from commit 0689ce5) * prep 3.1.0.0 RC3 (cherry picked from commit 2c3d5b9) * prep 3.1.0.0 RC3 (cherry picked from commit ea12c29) * ECS 3.1 templates * local facts must be fully qualified? * interface roles should be defined in deploy.yml * actually use a comma in the jinja joiner() func * use ansible_fqdn for agent strings not ansible_hostname * ECS 3.1.0.0 RC4 * joiner() needs to be the prefix not the suffix * the infamous missing comma * no trailing comma * remove redundant spaces * set host: field in testing * make object-main_network.json.j2 VDC-aware + formatting * Set georeceiver initialBufferNumOnHeap to 10 * Mount /usr instead of /usr/local to capture new install path * [WiP] ECS 3.1 Full GA and CE support (#367) * Use nodeId instead of the node IP to create data store * Fix errors getting node ID * fix 3.1 patch again * migrate cm.object.properties/'MustHaveEnoughResources=false' into Dockerfile * Run cf_client in container for new low partition count vars * Run cf_client in container for new low partition count vars * migrate cf_client variable settings into Dockerfile * update comments in Dockerfile for 3.1.0.0 * release-2.5.0-prep (#370) * Update ECS-Installation.md (cherry picked from commit f8be70f) * Update ECS-Installation.md (cherry picked from commit b479b07) * bump versions * Check if object user is editable before continuing to S3 credential provisioning (#376) * Fix #371 * Cosmetic change to fix output * Add retry loops around funcs to add credentials * add uniqueness to hostnames of CI VMs (#378) * Enable build-help in bootstrap.sh (#377) * enable build help in bootstrap.sh * enable build help in bootstrap.sh * Switch to upstream ECS version 3.1.0.1 (#375) * Add Dockerfile ecs-object-reduced to CE image patch * bump ecs version * bump version files for install node 2.5.1 (#379) * Develop (#372) (#380) backmerge
EMCECS · Nov 6, 2017 · e351219 · e351219
1 parent 09e1186
commit e351219
Show file tree

Hide file tree

Showing 8 changed files with 86 additions and 22 deletions.
diff --git a/bootstrap.sh b/bootstrap.sh
@@ -102,8 +102,9 @@ cat <<EOH | more
 [Build Options]
  --zero-fill-ova
     Reduce ephemera, defrag, and zerofill the instance after bootstrapping
---build-from
-
+ --build-from <URL>
+    Use the Alpine Linux mirror at <URL> to build the ecs-install image locally.
+    Mirror list: https://wiki.alpinelinux.org/wiki/Alpine_Linux:Mirrors
 
 For additional information, read the docs on GitHub.
 For additional help, please open an issue on GitHub.
@@ -181,7 +182,7 @@ fi
 
 if ! O=$(
          getopt \
-         -l build-from:,deploy-config:,registry-cert:,help,proxy-cert:,registry-login,centos-mirror:,override-dns:,proxy-endpoint:,registry-endpoint:,proxy-test-via:,vm-tools,zero-fill-ova,ssh-private-key:,ssh-public-key:,version,help-build,yes,no,verbose,quiet \
+         -l build-from:,deploy-config:,registry-cert:,help,help-build,proxy-cert:,registry-login,centos-mirror:,override-dns:,proxy-endpoint:,registry-endpoint:,proxy-test-via:,vm-tools,zero-fill-ova,ssh-private-key:,ssh-public-key:,version,help-build,yes,no,verbose,quiet \
          -o c:d:hk:lm:no:r:t:p:gyvqz \
          -n "${0}" \
          -- ${@}
@@ -199,6 +200,10 @@ while true; do
         usage
         exit 1
         ;;
+    --help-build)
+        usage_build
+        exit 1
+        ;;
     --version)
         shift
         version

diff --git a/docs/source/building/development-build.md b/docs/source/building/development-build.md
@@ -0,0 +1,10 @@
+# ECS Community Edition Installation
+
+## Building from develop branch
+
+### Prerequisites
+[same as standard install]
+
+### Steps
+
+
diff --git a/patches/3.1.0.1/Dockerfile b/patches/3.1.0.1/Dockerfile
@@ -0,0 +1,23 @@
+# Fixes to the default 3.1.0.1 reduced image.
+
+# Build on object-reduced image (GA release)
+FROM emcvipr/object:3.1.0.1-95464.df479c1-reduced
+
+# Increase memory for transformsvc
+RUN sed -i s/-Xmx128m/-Xmx512m/ /opt/storageos/bin/transformsvc
+
+# Fix disk partitioning script
+RUN sed -i '/VMware/ s/$/ \&\& [ ! -e \/data\/is_community_edition ]/' /opt/storageos/bin/storageserver-partition-config.sh
+RUN /usr/bin/chmod +x /opt/storageos/bin/storageserver-partition-config.sh
+
+# Set VNets useSeperateThreadPools to True
+RUN f=/opt/storageos/conf/vnest-common-conf-template.xml; grep -q "object.UseSeparateThreadPools" $f || sed -i '/properties id="serviceProperties"/a \ \ \ \ \ \ \ \ <prop key="object.UseSeparateThreadPools">true</prop>' $f
+
+# Set georeceiver's initialBufferNumOnHeap to something smaller for CE
+RUN f=/opt/storageos/conf/georeceiver-conf.xml; grep -q 'name="initialBufferNumOnHeap" value="10"' $f ||  sed -i 's/name="initialBufferNumOnHeap" value="60"/name="initialBufferNumOnHeap" value="10"/' $f
+
+# Configure CM Object properties: Disable minimum storage device count
+RUN f=/opt/storageos/conf/cm.object.properties; grep -q 'MustHaveEnoughResources=false' $f ||  sed -i 's/MustHaveEnoughResources=true/MustHaveEnoughResources=false/' $f
+
+# Allow allocation of different blocks of a chunk to be stored on the same partition
+RUN sed -i 's#<config:boolean name="allowAllocationOnIgnoredPartitions" value="false" description="If set to true, different blocks in one chunk may be allocated on the same partition"/>#<config:boolean name="allowAllocationOnIgnoredPartitions" value="true" description="If set to true, different blocks in one chunk may be allocated on the same partition"/>#g' /opt/storageos/conf/ssm-cf-conf.xml
diff --git a/tests/lab.tf b/tests/lab.tf
@@ -13,7 +13,7 @@ resource "vsphere_folder" "tests_folder" {
 }
 
 resource "vsphere_virtual_machine" "install_node" {
-  name   = "jenkins-ecsce-install"
+  name   = "jenkins-ecsce-install-${substr(sha1(timestamp()),0,8)}"
   folder = "${vsphere_folder.tests_folder.path}"
   vcpu   = 2
   memory = 4096
@@ -34,7 +34,7 @@ resource "vsphere_virtual_machine" "install_node" {
 
 resource "vsphere_virtual_machine" "ecs_node" {
   count  = "${var.ecs_nodes}"
-  name   = "jenkins-ecsce-ecs-${count.index}"
+  name   = "jenkins-ecsce-ecs-${count.index}-${substr(sha1(timestamp()),0,8)}"
   folder = "${vsphere_folder.tests_folder.path}"
   vcpu   = 4
   memory = 16384

diff --git a/ui/ecsconfig.py b/ui/ecsconfig.py
@@ -1124,18 +1124,44 @@ def add_one(name):
         o("Creating '{}' in namespace '{}'".format(name, ou_namespace))
         conf.api_client.object_user.create(name, namespace=ou_namespace)
 
-        o("\tAdding {}'s S3 credentials".format(name))
-        conf.api_client.secret_key.create(user_id=name,
-                                          namespace=ou_namespace,
-                                          expiry_time=ou_dict['s3_expiry_time'],
-                                          secret_key=ou_dict['s3_secret_key'])
-
-        if ou_dict['swift_password'] is not None and ou_dict['swift_groups_list'] is not None:
-            o("\tAdding {}'s Swift credentials".format(name))
-            conf.api_client.password_group.create(user_id=name,
+        o("\tWaiting for '{}' to become editable".format(name))
+        is_editable = False
+        while is_editable is False:
+            try:
+                get_one(name)
+                is_editable = True
+                o("\tOK")
+            except Exception as e:
+                is_editable = False
+                o("\tWaiting...")
+                time.sleep(5)
+
+        creds_added = False
+        while creds_added is False:
+            try:
+                o("\tAdding {}'s S3 credentials".format(name))
+                conf.api_client.secret_key.create(user_id=name,
                                                   namespace=ou_namespace,
-                                                  password=ou_dict['swift_password'],
-                                                  groups_list=ou_dict['swift_groups_list'])
+                                                  expiry_time=ou_dict['s3_expiry_time'],
+                                                  secret_key=ou_dict['s3_secret_key'])
+                creds_added = True
+            except Exception as e:
+                creds_added = False
+                time.sleep(5)
+
+        creds_added = False
+        while creds_added is False:
+            try:
+                if ou_dict['swift_password'] is not None and ou_dict['swift_groups_list'] is not None:
+                    o("\tAdding {}'s Swift credentials".format(name))
+                    conf.api_client.password_group.create(user_id=name,
+                                                          namespace=ou_namespace,
+                                                          password=ou_dict['swift_password'],
+                                                          groups_list=ou_dict['swift_groups_list'])
+                creds_added = True
+            except Exception as e:
+                creds_added = False
+                time.sleep()
 
     available_configs = list_all()
     if l:

diff --git a/ui/etc/config.yml b/ui/etc/config.yml
@@ -13,7 +13,7 @@
 ---
 ui:
   name: ECS Community Edition Install Node
-  version: 2.5.0
+  version: 2.5.1
   host_root_dir: /opt/emc/ecs-install
   state_file: /opt/state.yml
   deploy_file: /opt/deploy.yml
@@ -32,7 +32,7 @@ ui:
   ffx_sem: /opt/ffx.sem
 product:
   name: ECS
-  version: 3.1.0.0
+  version: 3.1.0.1
   vendor: Dell EMC
   flavor: Community Edition
   slogan: Free and Frictionless

diff --git a/ui/etc/release.conf b/ui/etc/release.conf
@@ -8,10 +8,10 @@
 # it is provided by or on behalf of EMC.
 
 release_name="ECS Community Edition"
-release_version="3.1.0.0"
+release_version="3.1.0.1"
 release_product="ECS Software"
 release_artifact="emccorp/ecs-software-3.1.0"
-release_tag="latest"
+release_tag="hf1"
 release_common_name="emccorp/ecs-software:latest"
 
 docker_host_root="/opt/emc/ecs-install"
@@ -33,7 +33,7 @@ image_name='ecs-install'
 tag='latest'
 ver_maj='2'
 ver_min='5'
-ver_rev='0'
+ver_rev='1'
 ver_tag='r'
 serial=0
 

diff --git a/ui/setup.py b/ui/setup.py
@@ -3,7 +3,7 @@
 
 setup(
     name='ecsdeploy',
-    version='2.5.0',
+    version='2.5.1',
     packages=find_packages(),
     scripts=['ui.py',
              'ecsdeploy.py',