ODPi v2.1.0 release

Release Notes - ODPi - Version Spec 2.1

Epic

• [ODPI-123] - Introduce Hadoop Compatible Filesystem (HCFS) support
• [ODPI-206] - Improvements to HCFS for ODPi v2.1
• [ODPI-208] - Improvements to Hadoop/Yarn for ODPi v2.1

Improvement

• [ODPI-201] - Drop strict version matching for HADOOP_DIRSTRUCT_COMMON, HADOOP_DIRSTRUCT_MAPRED and HADOOP_JARCONTENT_COMMON
• [ODPI-217] - Check for tools.jar in run_itest.sh as dependency check
• [ODPI-218] - Investigate why hcfs smoke tests require jdk tools.jar instead of jre
• [ODPI-220] - Add pre-req check in run-itest.sh for user / environment requirements
• [ODPI-222] - Relax regex for servlet-api and jsp for tomcat upgrade

Task

• [ODPI-150] - Bigtop smoke tests in ODPi runtime test suite
• [ODPI-158] - Write tests for HCFS
• [ODPI-190] - Document stack-select and conf-select
• [ODPI-191] - Document stack hooks

New Feature

• [ODPI-170] - Reduce the gap between HCFS contract definition and the contract testsuite

Bug

• [ODPI-117] - HADOOP_BINCONTENT_* does't give information about why it fails
• [ODPI-202] - Push ODPi stack definition back to Apache Ambari
• [ODPI-205] - Fix hive TestHCatalog test to pass arguments using key / value pairs
• [ODPI-213] - Testsuite usability feedback from IBM
• [ODPI-216] - Need to update the operations spec to allow custom services to install themselves

Sub-task

• [ODPI-63] - Test [HADOOP_API]
• [ODPI-153] - BigTop smoke tests assume HDFS trash should be disabled

ODPi v2.0.0 release

Release Notes - ODPi - Version Spec 2.0

Epic

• [ODPI-119] - Add Apache Hive to ODPi specification
• [ODPI-163] - Fix build/packaging issues
• [ODPI-164] - Build initial Operations spec

Improvement

• [ODPI-5] - Integrate Ambari packaging into Bigtop
• [ODPI-168] - Turn HCFS installation doc into a section for the Runtime spec on HCFS/HDFS
• [ODPI-169] - Migrate HCFS/Ambari integration section to an appropriate spec
• [ODPI-171] - Does ODPi-certified mean HCFS compatible?
• [ODPI-172] - Spec out CLI for HCFS
• [ODPI-173] - Add Hive 1.2 to ODPi reference implementation
• [ODPI-174] - Provide a summary of the services covered in the Runtime SPEC
• [ODPI-175] - Need a section documenting properties in the stack definition
• [ODPI-176] - Provide an ISV app focus around Kerberos support
• [ODPI-177] - Document service-level inheritance/extension
• [ODPI-178] - Add a recommendation of one service per service definition file
• [ODPI-193] - Add Hive 1.2 to ODPi ambari reference implementation
• [ODPI-200] - Modify run_itest.sh to include required parameters for hive runtime tests

Task

• [ODPI-148] - Revisit repo.odpi.org layout
• [ODPI-156] - Review current HCFS spec
• [ODPI-157] - Make spec additions for HCFS
• [ODPI-161] - Build spec for Hive
• [ODPI-162] - Write tests for Hive
• [ODPI-165] - Write initial spec
• [ODPI-179] - Distill Ambari wiki into an immutable ISV-focused spec we can refer to
• [ODPI-180] - Test [HIVE_SQL]
• [ODPI-181] - Test [HIVE_JDBC]
• [ODPI-182] - Test [HIVE_CLI]
• [ODPI-183] - Test [HIVE_BEELINE]
• [ODPI-184] - Test [HIVE_THRIFT]
• [ODPI-185] - Test [HIVE_HCATALOG]
• [ODPI-198] - Integrate Hive tests with ODPi reference build and Jenkins

New Feature

• [ODPI-197] - Enable Kerberos in ODPi Ambari

Bug

• [ODPI-105] - Revisit [ODPI_JRE]
• [ODPI-106] - Revisit [ODPi_SCRIPT]
• [ODPI-130] - Gradle de.undercouch.download version issue
• [ODPI-139] - Zookeeper build failing
• [ODPI-141] - java.lang.ClassNotFoundException: org.apache.hadoop.log.metrics.EventCounter
• [ODPI-143] - java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being
• [ODPI-144] - Extra INFO logging is in test output
• [ODPI-186] - ODPi reference implementation Ambari deployment fails for YARN
• [ODPI-187] - ODPi reference implementation Ambari deployment fails for extra services
• [ODPI-188] - Clean up terminology in Runtime Spec
• [ODPI-194] - Fix typo in runtime spec
• [ODPI-199] - Hive Server needs to be provisioned by Bigtop so we can test it

ODPi v1.0.1 release

Release Notes - ODPi - Version Spec 1.0.1

Task

• [ODPI-121] - Instructions for spec validation for distros are incomplete

Bug

• [ODPI-120] - HADOOP_JARCONTENT_COMMON, HADOOP_JARCONTENT_HDFS, and HADOOP_JARCONTENT_YARN have too restrictive of a regular expression when checking for some jars
• [ODPI-122] - HADOOP_DISTRUCT_MAPRED is too restrictive on joda-time
• [ODPI-126] - HADOOP_BINCONTENT and HADOOP_LIBJARSCONTENT have MUSTs instead of SHOULDs
• [ODPI-127] - Vagrant config file for VM cannot find yum repo
• [ODPI-128] - Installation of hadoop inside reference build fails
• [ODPI-129] - create-sandboxes script assumes user has no other vms active on the machine
• [ODPI-131] - Reference build fails HADOOP_BINCONTENT_COMMON test
• [ODPI-140] - warning: non-varargs call of varargs method with inexact argument type for last parameter
• [ODPI-142] - HADOOP_MAPRED_HOME not set in run_itest.sh
• [ODPI-145] - Directory structure tests are too restrictive on trailing content

ODPi v1 release

Release Notes - ODPi - Version Spec 1.0

** Epic
* [ODPI-1] - A use case articulated by DataTorrent to define the APPS log4j appender
* [ODPI-23] - User level requirements for a reference implementation for ODPi Runtime spec
* [ODPI-24] - Adopt a testing framework and build out initial tests
* [ODPI-27] - Make it clear on how people can get involved and contribute to the ODPi spec, testing framework, and reference implementation, as well as the larger technical discussion
* [ODPI-80] - Finalize enablement of applicable tests from Bigtop testsuite

** Improvement
* [ODPI-109] - Make sure our test execution environment reports back on TEST_ENVIRONMENT tags

** Task
* [ODPI-28] - Add README file to odpi/spec repo
* [ODPI-30] - Write iTest tests for validating ODPi v1 spec compliance
* [ODPI-31] - Create output report of jUnit test results
* [ODPI-32] - Develop the proposal process for new projects/additions to an ODPi spec
* [ODPI-76] - dockerhub account is needed to host ODPi images
* [ODPI-88] - Add repo and process for Hadoop distro to submit ODPi compliance test results
* [ODPI-98] - Publish distro artifacts and compliance suite to the public repo
* [ODPI-104] - Remove outdated assets from repo.odpi.org

** New Feature
* [ODPI-6] - Build slaves need to support docker containers
* [ODPI-26] - Added appendix with internal Hadoop config values

** Bug
* [ODPI-13] - "Build configuration changes" link is misconfigured on jenkins master
* [ODPI-14] - Default user domain in CI master SMTP configuration is incorrect
* [ODPI-15] - Jenkins base URL seems to be misconfigured
* [ODPI-16] - Attempt to delete a job workspace brings AccessDenied error
* [ODPI-22] - Create Softlayer ODPi VPN user
* [ODPI-84] - el6 yum repo data has invalid checksums
* [ODPI-91] - ZK build and package are broken by ODPI-10
* [ODPI-92] - run_itest script does not know where bigtop-detect-javahome lives
* [ODPI-93] - vagrant config for VMs needs to point to ODPI resources
* [ODPI-94] - HADOOP_DIRSTRUCT* tests are not uniquely named
* [ODPI-96] - Remove "enabled by default" from the HADOOP_COMPRESSION specification
* [ODPI-97] - HADOOP_GETCONF test should be using the hdfs command from users's $PATH
* [ODPI-101] - Ensure license compliance for v1 release
* [ODPI-102] - Remove VM and Vagrant images from repo.odpi.org
* [ODPI-103] - http://repo.odpi.org/ should URL rewrite to http://repo.odpi.org/repository/
* [ODPI-108] - a file introduced in ODPI-10 is failing RAT check
* [ODPI-110] - Test HADOOP_GETCONF assumes cluster user is 'hadoop'
* [ODPI-111] - Adjust [HADOOP_PLATVER] to allow _ to prefix vendor string
* [ODPI-112] - Remove log file check for [HADOOP_PLATVER]
* [ODPI-114] - HADOOP_GETCONF should not rely on specific user names
* [ODPI-115] - HADOOP_DISTRUCT_* need to allow for wildcards
* [ODPI-116] - Distribution includes both 1.7 and 1.8 commons-beanutils
* [ODPI-118] - ODPI-111 introduces incorrect regex, breaks test compilation

** Sub-task
* [ODPI-3] - Define and publish versioning schema for ODPi reference implementation artifacts
* [ODPI-4] - Deploy Bigtop-based CI pipeline to ODPi Jenkins
* [ODPI-7] - Define and start managing ODPi specific branches in GitHub mirrors of Apache repos
* [ODPI-8] - Existing Bigtop toolchain needs to be downgraded to meet Ambari reqs
* [ODPI-9] - Hadoop RPM build is broken with GH based sources
* [ODPI-10] - Publish distro artifacts to Nexus (jars) and webserver (rpm's/deb's)
* [ODPI-11] - Automatic distro deployment and smoke testing is needed
* [ODPI-12] - Distro build needs to use local ODPi Nexus to speed up the build
* [ODPI-17] - Installing reprepro & createrepo tools on all build slaves
* [ODPI-18] - ODPi repos should be used for smoking deployment
* [ODPI-19] - Release name for debian repo should be called 'odpi'
* [ODPI-20] - Docker slaves need to have vagrant installed
* [ODPI-21] - Linux artifacts should be signed before publishing
* [ODPI-25] - Add SSH keys to the slaves
* [ODPI-33] - Jenkins GPG key has @hortonworks.com email in it
* [ODPI-34] - Test [HADOOP_COMMIT]
* [ODPI-35] - Test [HADOOP_CNATIVE1]
* [ODPI-36] - Test [HADOOP_CNATIVE2]
* [ODPI-37] - Test [HADOOP_HNATIVE1]
* [ODPI-38] - Test [HADOOP_YNATIVE1]
* [ODPI-39] - Test [HADOOP_MNATIVE1]
* [ODPI-40] - Test [HADOOP_MNATIVE2]
* [ODPI-41] - Test [HADOOP_EJH1]
* [ODPI-42] - Test [HADOOP_EC1]
* [ODPI-43] - Test [HADOOP_EC2]
* [ODPI-44] - Test [HADOOP_EC3]
* [ODPI-45] - Test [HADOOP_EC4]
* [ODPI-46] - Test [HADOOP_EC5]
* [ODPI-47] - Test [HADOOP_EH1]
* [ODPI-48] - Test [HADOOP_EH2]
* [ODPI-49] - Test [HADOOP_EH3]
* [ODPI-50] - Test [HADOOP_EY1]
* [ODPI-51] - Test [HADOOP_EY2]
* [ODPI-52] - Test [HADOOP_EY3]
* [ODPI-53] - Test [HADOOP_EM1]
* [ODPI-54] - Test [HADOOP_EM2]
* [ODPI-55] - Test [HADOOP_EM3]
* [ODPI-56] - Test [HADOOP_DISTCNT]
* [ODPI-57] - Test [HADOOP_ENVVAR]
* [ODPI-58] - Test [HADOOP_EJH2]
* [ODPI-59] - Test [HADOOP_EJH2]
* [ODPI-60] - Test [HADOOP_TOOLS]
* [ODPI-61] - Test [HADOOP_SUBPROJS]
* [ODPI-62] - Test [HADOOP_BIGTOP]
* [ODPI-64] - Test [HADOOP_PLATVER]
* [ODPI-65] - Test [HADOOP_DIRSTRUCT]
* [ODPI-66] - Test [HADOOP_BINCONTENT]
* [ODPI-67] - Test [HADOOP_LIBJARSCONTENT]
* [ODPI-68] - Test [HADOOP_GETCONF]
* [ODPI-69] - Test [HADOOP_COMPRESSION]
* [ODPI-70] - Test [HADOOP_NORDMPORTS]
* [ODPI-71] - Spec out MVP for the reference implementation/sandbox
* [ODPI-72] - Build out VM for sandbox/reference implementation
* [ODPI-73] - Build out Docker container for sandbox/reference implementation
* [ODPI-74] - Build out Vagrant image for sandbox/reference implementation
* [ODPI-75] - Write guidelines for creating tests
* [ODPI-77] - Enable HDFS tests from the Bigtop test suite
* [ODPI-78] - Enable HCFS tests from the Bigtop test suite
* [ODPI-79] - Enable YARN tests from the Bigtop test suite
* [ODPI-81] - Document update-flow for odpi-specific branches.
* [ODPI-82] - Smoke tests are ran as root in a container; cleaning is problematic
* [ODPI-86] - Test [ODPi_JRE]
* [ODPI-87] - Capture details on Hadoop system under test
* [ODPI-89] - Test [ODPi_SCRIPT]
* [ODPI-90] - CI build triggering should only be available for auth'ed users
* [ODPI-99] - Propagate jenkins user and add slave to the new bare-HW host
* [ODPI-100] - Sandbox images should be created and published by CI
* [ODPI-107] - spec-tests resources should be excluded from RAT check

version 1.0.0 draft of runtime specification

This is version 1.0.0 of the Runtime specification, in draft state. We are still taking feedback in preparation for the full 1.0.0 release.