Aggregate Release Notes

General Upgrade Steps

It is recommended that you first use ODK Briefcase to make a copy of all data on your ODK Aggregate before you upgrade. The general steps are:

disable all submissions to ODK Aggregate via the Forms Management tab.
use ODK Briefcase to pull a copy of all data to your computer.
search upward in the release notes below, starting from your current ODK Aggregate version to locate the most recent ODK Aggregate version that does not require any manual upgrade steps. e.g., if you were using v1.3, searching upward would reveal that v1.4 required manual upgrade actions. The release prior to that which did not require any manual upgrade actions is v1.3.4. Download that version of ODK Aggreate and upgrade to it (in this case, v1.3.4).
log onto your server to confirm that it is still functioning
download the next ODK Aggregate version, perform the manual upgrade steps and upgrade to that version (and perform any post-install upgrade steps, if any).
log onto your server to confirm that it is still functioning
repeat 3-6 until current.
enable submissions to ODK Aggregate via the Forms Management tab.

Note that the submission Filter broke beginning with ODK Aggregate 1.4.6, so if you use it, you should upgrade through to at least 1.4.8. Submission filters were fixed in ODK Aggregate 1.4.8.

e.g.,

If you are running 1.3, and don't use the JSON publisher or Tables features, then:

Disable submissions

Pull all data using ODK Briefcase
 (this is your catastrophic back-up)

Clear your browser cache 
 (the only applicable upgrade step for 1.4)

Upgrade to 1.4.2 (you can skip 
installing 1.3.4 and 1.4 since the
only upgrade step was to clear your
browser's cache, which doesn't impact
your computer or the server).

Verify you can login

You would then do the upgrade steps 
for 1.4.3 (installing java 7).

And then update to 1.4.3

Verify you can log in

And then update to 1.4.5 or 1.4.7 
if the submission-filter feature 
is not important.

Verify you can log in

Enable submissions.

Note that you need to know the exact "instance name" that was used in prior installs in order for your username and password to continue to work. If you add a space or change capitalization or spelling, the passwords will be invalid (you just need to re-run the installer with the correct string to correct the problem).

Installing older ODK Aggregate (pre-1.4.8) Releases

On April 12, 2016, Google disabled the login mechanism used by the older update scripts that were packaged with these installers.

To use these older installers:

first, download and run an ODK Aggregate 1.4.8 (or newer) installer; on the final screen, uncheck the Launch installation script (3-10 minutes). checkbox and click Finish.
go into the directory where the installer placed the files, go into the ODK Aggregate directory, and rename the ODKAggregate directory inside it to NewRemoval.
next, download and run the pre-1.4.8 ODK Aggregate installer; on the final screen, uncheck the Launch installation script (3-10 minutes). checkbox and click Finish.
go into the directory where the pre-1.4.8 installer placed its files, go into the ODK Aggregate directory, and copy the ODKAggregate directory from there to the installer directory created by the newer installer (effectively replacing the configuration produced by the newer installer with the configuration produced by the older one).
finally, in the directory created by the newer installer, if on: * Windows: double-click the ODKAggregateAppEngineUpdater.jar and use that to update your App Engine instance. * Mac OSX: double-click the uploadAggregateToAppEngine.app file in that directory. * Linux: open a bash shell, navigate to this directory, and run the uploadAggregateToAppEngine.sh file.

NOTE: If you are reverting from a 1.4.8 or later release to a 1.4.7 or earlier release, you must manually delete the background module of your App Engine using the Google Cloud Platform administration web pages.

ADDITIONAL NOTE: ODK Aggregate 1.4.7 and earlier use a deprecated backends technology. Google may terminate support for that at any time with little warning. You should consider upgrading to 1.4.8.

Determining your ODK Aggregate Version

If are able to log onto the server as a Site Administrator, the ODK Aggregate version is displayed at the top of the Site Admin / Preferences sub-tab. If no version information is displayed on that page, then you are running an extremely old version of ODK Aggregate; in that case, scroll down to the very bottom of this wiki to see additional instructions for identifying your ODK Aggregate version.

If you are unable to log onto your server, you will need to search for the version in the application logs. To do that,

open a browser to https://appengine.google.com
choose the application id for your ODK Aggregate server
click on the "Logs" link in the left sidebar
pop open "Options"
type afterPropertiesSet in the Filter text field
hit "Search"
now, find the occurrences of afterPropertiesSet on your webpage.

The first one will generally report the version. E.g.,

2015-09-22 08:19:12.855 org.opendatakit.common.security.Realm afterPropertiesSet: Version: v1.4.5 Production

Aggregate v1.4.15

(July 2017)

Update javarosa to the new opendatakit-javarosa-2.3.0.jar
Fix: rev 210 sync protocol. User permissions were incorrectly being computed and filtered. This prevented resetting the server with new content from the device (but syncing with existing content worked fine). The rev 210 sync protocol is incompatible with anything prior to rev 210.

Upgrading

Do not upgrade if you are using ODK 2.0 rev 208 or earlier.
See v1.4.13 upgrade notes.

Aggregate v1.4.14

(June 2017)

Update javarosa to the new opendatakit-javarosa-2.2.0.jar
Fix for JSON publisher correctly handling UTF-8
Added SHA-1 library to browser so that Basic Auth can now be configured and will now work (this capability is not exposed in the wizard-based installer/configurer; existing passwords need to be changed before they can be used in a Basic Auth configuration)
Revised sync protocol and database schema for ODK 2.0. This is incompatible with anything prior to rev 210 (not yet available).
Microsoft SQLServer configurations can now run on Linux and MacOSX boxes.
Updated to the latest AppEngine SDK.
Updated build environment to a mixed Java 7 and Java 8 environment.

Upgrading

Do not upgrade if you are using ODK 2.0 rev 208 or earlier.
See v1.4.13 upgrade notes.

Aggregate v1.4.13

(January 2017)

Add support for Microsoft SQL Server database and Azure SQL Server
Updated to expect Tomcat 8 and JDBC 4 libraries. This extensively impacted the datastore layer.
Rewrote the Google Sheets publisher to use a newer API.
Changed the PAUSED publisher state to expose a restart button. Fixed bug that prevented ABANDONED button from working. When a publisher enters the Paused state, this button triggers ODK Aggregate to resume from that state. This happens automatically every 8 minutes or so, but having a resume button simplifies integration testing. i.e., when a publisher is in the Paused state, you do NOT need to click this button -- but if you are testing a publisher to your own server, it can be useful to have.
Fixes for ODK 2.0 datatype form.
NaN +Infinity and -Infinity are now handled within Google AppEngine. These cannot be supported on systems using MySQL and SQL Server (a fundamental limitation of those databases). On PostgreSQL systems, if you alter your tables to use REAL instead of DECIMAL(38,0) columns, it will be able to store and process these special values.
Updated a multitude of jars (libraries).
If this is a first-time install using PostgreSQL, please see the upgrade steps below for the additional commands you need to execute to complete the PostgreSQL configuration. These commands are missing from the create_db_and_user.sql but are required for PostgreSQL operations.

Upgrading

If publishing to Google Sheets, you must go to the API Manager tab and enable the Google Sheets API.
Tomcat installs now require Tomcat 8 or higher. The war file produced by the installer will work as-is on those systems -- no need for any manual modifications to get things to work (though MySQL still requires downloading and copying the MySQL Connector/J, and SQL Server also has a manual configuration step).
If using an older MySQL database, please upgrade to MySQL 5.7 or higher; note that some MySQL releases expire all database passwords after 360 days. Please verify the behavior of your version of MySQL and either change the password expiration policy or create a calendar reminder to change the password before it expires. For ODK Aggregate, you will need to re-run the installer to specify any new password. For more information, see the MySQL documentation. e.g., MySQL password expiration policy.
The ODK 2.0 sync protocol in this release is compatible with the ODK 2.0 Rev 200, 202, 204, 206 and 208 tools. Note that you cannot just upgrade from a server earlier than 1.4.10; you have to delete everything and start over.
See v1.4.12 upgrade steps if upgrading from an earlier ODK Aggregate version.
If using an older PostgreSQL database, please upgrade to at least 9.4. For all PostgreSQL servers, you must execute an additional command to grant permissions to the schema on the server. This is a change from the earlier 9.1 install and the additional command is missing from the create_db_and_user.sql script. The commands to be executed in the pgAdmin query window are as follows, with your_database, your_schema and database_username replaced with the values you specified when you ran the installer:

\c "your_database";
alter schema "your_schema" owner to "database_username";

Aggregate v1.4.12

(September 2016)

update to installer with new language and links to documentation.
revise upload tool for Google AppEngine deployments to more-consistently display the token-entry dialog.
add .csv-based downloading and bulk updating of configured users and their capabilities.
change MySQL ODK 1.x and ODK 2.0 data table constructions to use MEDIUMBLOB columns for media attachments. Fix several issues around accessing attachments. See Updating instructions (below) and MySQL configuration instructions (on opendatakit.org) for configuration that is REQUIRED for MySQL.
tweak Google AppEngine configuration to reduce frontend usage hours and thereby reduce costs.
performance improvements to Google AppEngine task locks (primarily for ODK 2.0 support).
add "/users/list" URL that returns the list of all configured users and the roles they have been granted. Access to this URL requires authentication and is restricted to ODK Aggregate usernames and Google accounts (anonymousUser access is forbidden). If the authenticated user does not have Tables Super-user, Administer Tables, or Site Administrator permissions, a singleton list is returned that contains only information about their own identity and its capabilities.
changes to support row-level filtering in ODK 2.0.
implement a paginated view of ODK 2.0 data rows; 100 records per page.
upgrade to Google AppEngine 1.9.42 SDK

Known Issues

the ODK 2.0 "datatypes" example form and table (in the app-designer repository) fail to upload to ODK Aggregate. This will be addressed in a future release.

Upgrading

For MySQL installations, please make sure you have this server configuration (if this is a new database install, you can reduce max_allowed_packet to 16842752):

character_set_server=utf8
collation_server=utf8_unicode_ci
max_allowed_packet=1073741824

For ODK 1.x uses, no special upgrade steps are required.
If you are using ODK 2.0 features, you must visit the Site Admin / Permissions page and click Save Changes to complete the upgrade to this release (you do not need to modify anything on that page, just click the button).
The ODK 2.0 sync protocol in this release is compatible with the ODK 2.0 Rev 200, 202, 204, 206 and 208 tools. Note that you cannot just upgrade from a server earlier than 1.4.10; you have to delete everything and start over.
See v1.4.11 upgrade steps if upgrading from an earlier ODK Aggregate version.

Aggregate v1.4.11

(July 2016)

add a mutex around ODK 1.x form submission creation and updating. This should eliminate submission data corruption.
if only ODK 2.0 privileges are assigned, hide the Submissions tab.
fix: advisory memcache interaction to delete the entry when rolling back the datastore update.
add "Tables Super-user" as a configurable user capability. Users with this capability will be able to modify the table-level and row-level privileges on ODK 2.0 rev 206 data tables. These privileges are advisory and do not provide strong access or revision control.
add "/roles/granted" URL that returns the roles granted to an authenticated username or google account. This will be used by ODK 2.0 rev 206 tools to determine and save the capabilities of the configured user on a device. Those capabilities are then used to apply advisory access controls on the tables and rows.

Upgrading

For MySQL installations, please make sure you have this server configuration:

character_set_server=utf8
collation_server=utf8_unicode_ci
max_allowed_packet=1073741824

For ODK 1.x uses, no special upgrade steps are required.
If you are using ODK 2.0 features, you must visit the Site Admin / Permissions page and click Save Changes to complete the upgrade to this release (you do not need to modify anything on that page, just click the button).
The ODK 2.0 sync protocol in this release is compatible with the ODK 2.0 Rev 200, 202 and 204 tools. Note that you cannot just upgrade from a server earlier than 1.4.10; you have to delete everything and start over.
See v1.4.10 upgrade steps if upgrading from an earlier ODK Aggregate version.

Aggregate v1.4.10

(July 2016)

on AppEngine, add advisory memcache interactions inside database mutex implementation (TaskLockImpl) to detect and thereby reduce race conditions that could lead to database corruption. These changes do not alter the fundamental mutex mechanism. They may reduce the likelihood of a mutex failure.
fix ODK 2.0 sync: table-specific pre-loaded instance attachments under config/assets/csv/tableid/instances/... were not being identified as table-specific.
fix ODK 2.0 sync: add mutex around app-level file, table-level file and row-level attachment REST interactions. Eliminates the possibility of joint-updating that could corrupt the app-, table- or row-level manifests.
fix ODK 2.0 sync: change row-level file attachment bulk-update to not refetch the manifest after adding each file attachment. This had caused 1% failure rate on row-level attachment syncing when there were large numbers of file attachments (30-60) for each row.
fix ODK 2.0 sync: when authorization fails (permission denied), return UNAUTHORIZED response rather than DENIED. This reports an authorization failure on the client vs. a protocol error.

Upgrading

For MySQL installations, please make sure you have this server configuration:

character_set_server=utf8
collation_server=utf8_unicode_ci
max_allowed_packet=1073741824

For ODK 1.x uses, no special upgrade steps are required.
The ODK 2.0 sync protocol is compatible with the ODK 2.0 Rev 200, 202 and 204 tools.
For ODK 2.0 uses, if you have pre-loaded datasets (via tables.init) that have row-level attachments (e.g., geotagger example dataset), you should: 1. remove these incorrectly-categorized files from the server (anything under assets/csv/table_id/instances/... on the Manage App Level Files sub-tab) 1. deploy your device configuration to a clean device 1. Reset App Server to correctly upload these files so that they appear under the Manage Table Level Files sub-tab.

Aggregate v1.4.9

(May 2016)

update to javarosa 2016-04-21 * add XPath 3.1 math functions (e.g., exp, log, sin, cos, etc.)
fix AppEngine adaptation layer for Apache HttpClient so that Oauth2 authentications work
change version string in ODKAggregateAppEngineUpdater.jar to be a date stamp (2016-04-21)
fix ODK 2.0 interfaces to support ODK Aggregate username / password for authentication.
fix ODK 2.0 Sync so that a delete-table request can be repeated until successful or Not Found (404) when it times out.

Upgrading

For MySQL installations, please make sure you have this server configuration:

character_set_server=utf8
collation_server=utf8_unicode_ci
max_allowed_packet=1073741824

For all other installations, there are no special upgrade steps required.
The ODK 2.0 sync protocol is compatible with the ODK 2.0 Rev 200, 202 and 204 tools.

Aggregate v1.4.8

(April 2016)

Submission Filters are once again working.
Update javarosa library with fixes for:
- fix date, time, dateTime handling of time zones (in collaboration with SurveyCTO)
- fix itemset choice lists -- various issues.
- fix jr:choice-name() context resolution so that it can work with relative paths and repeat groups (SurveyCTO contribution).
- add CONTROL_FILE_CAPTURE for future arbitrary-file-attachment handling
- use consistent UTF-8 treatment when writing and reading files
- better detection and handling of binary-file-format changes across versions
- build with Java 7
Add support for geotrace and geoshape to Export-to-KML (for ODK 1.x).
Update Apache HTTP Client libraries to 4.5.2; this should support SNI protocol interactions of external publishers (ODK 1.x - untested).
Change Google App Engine update mechanism to use new wizard update tool.
Change Google App Engine code to use EAR / module format.
Widespread jar update.
Correct bug in csv-export library shared with ODK 2.0 tools for exporting datasets in those tools (does not impact Export as CSV in the 1.x toolchain).
Add new bulk attachment-POST APIs for ODK 2.0 sync protocol.
Rework ui tests to use Chrome for the Selenium web testing.
Rework all of the eclipse projects and add documentation for setting up a tomcat eclipse debug environment.
Rework the Google App Engine maven projects to use the EAR / module format.
Update maven plugins and tools to newer versions.

Upgrading

For MySQL installations, please make sure you have this server configuration:

character_set_server=utf8
collation_server=utf8_unicode_ci
max_allowed_packet=1073741824

For all other installations, there are no special upgrade steps required.
The ODK 2.0 sync protocol is compatible with the ODK 2.0 Rev 200 and higher tools.

Aggregate v1.4.7

(May 2015) 🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

NOTE: Submission Filters are broken in this release.
when submissions rows are corrupted, consistently report the information necessary for end-users to delete or repair the corrupted submission. 4 error messages needed to be enhanced.
correct ODK 2.0 delete-table functionality so that it does not generate errors when it is not running on AppEngine. The code had been performing an unguarded explicit cast to an AppEngine-only implementation class.

Upgrading

For MySQL installations, please make sure you have this server configuration:

character_set_server=utf8
collation_server=utf8_unicode_ci
max_allowed_packet=1073741824

For all other installations, there are no special upgrade steps required.

Aggregate v1.4.6

(April 2015) 🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

NOTE: Submission Filters are broken in this release.
fix for Google Sheets publisher-creation problem caused by April 20th deprecation of the Google Docs API. We were making a single call to that obsolete API to create a blank Google Sheets document.
update to javarosa-2015-04-17 jar. That jar adds OpenStreetMap support to javarosa.
removal of the "Sign in with Google" mechanisms that used OpenID.
widespread update to jars, custom repository configuration, and the elimination of several custom-built jars because of the elimination of OpenID and the maturity of Maven as a version repository.

Upgrading

No special upgrade steps required.

Aggregate v1.4.5

(January 2015) 🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

update to javarosa-2015-01-10 jar.
update to jQuery 1.11.1
update the ODK 2.0 Data model and Sync protocol (incompatible with device releases: rev 122 and earlier).
Fix: support performing mark-as-complete on encrypted submissions (requires ODK Briefcase v1.4.5 or higher).
Fix: add server preference to ignore partially inserted/deleted submissions. Logs them but ignores them so that you can access all other rows in your dataset. Disabled by default. By default, all actions fail upon encountering any malformed submission. You should not ignore these failures but should correct them as soon as is practical.
Attempted fix: for "Log In" issue -- insert sleep to give Google a chance to propagate clearing of session cookie. Tweak the webpage resize/layout calculations to be more efficient.
fix: make incomplete deletions and insertions more recoverable.
performance: change MySQL table creation to not declare primary keys and just use ordinary indices for the primary key.
security: Support Enketo-express (allow non-https communications with Enketo server). When the communications are not secure, this change discloses the Enketo access token to eavesdroppers.
security: Filter out forbidden characters in redirect string to prevent XSS attacks.
security: Add clickjacking prevention header as detailed here: https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet

Upgrading

If you were testing out ODK Tables, you should delete all ODK Tables files and data tables before upgrading, as the database schema has changed. Note that the server no longer works with ODK Sync 2.0 rev 122 (or earlier releases).

Aggregate v1.4.4

(August 2014) 🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

installer now asks for an ODK Aggregate username for the super-user (not a Google e-mail account). Default password is aggregate.
banner displayed if super-user's password has not been changed from aggregate.
fix: column name generator bug (upload of form definition failed)
fix: add more detailed error messages when a submission is corrupted
IE6 and IE7 are no longer supported. Upgrade your browser.
allow user to specify the ODK 2.0 App Name.
allow anonymous access to ODK 2.0 Sync APIs.
enforce 'Administer Tables' access for ODK 2.0 Sync APIs that alter server configuration.
for ODK 2.0 Sync APIs, remove JBoss Resteasy; use Apache Wink instead.
extensive version updates to supporting software libraries.
update CONFIGURE.txt instructions for maven builds. Define ANT scripts for external dependencies.

Upgrading

Upgrades require several manual interventions:

You must have Java 7 installed - the GAE 1.9.7 SDK used within the installer now requires that version of Java. If you are upgrading from ODK Aggregate 1.4.3, you have already done this.
If you were testing out ODK Tables, you should delete all ODK Tables data tables before upgrading, as the database schema has changed.
You must flush the session cookies on the server. Session cookies are used to identify the logged-in users of the system. The security software versions were updated in this release, causing the older cookies to become invalid. See below for the upgrade steps.

Assuming you have done (1) and (2) above, the upgrade steps after those are as follows:

Open a browser and go to your Google AppEngine dashboard via: dashboard
Click through to your application id. Then click on "Memcache Viewer" under the "Data" heading in the left sidebar. Keep this window open.
Now, run the installer and deploy ODK Aggregate 1.4.4 to this application id.
After it has deployed, click on Flush Cache in the dashboard window's Memcache Viewer screen.
Follow the instructions here for changing the password of the super-user username.

Other than the need to flush this cache, and the need to delete any ODK Tables data before upgrading, this should be a seamless upgrade from ODK Aggregate 1.4.x

Aggregate v1.4.3

(May 2014) 🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

updated javarosa jar
add stub Tomcat/MySQL Eclipse project with readme
update selenium test environment and various 3rd party jars
update the GAE SDK inside the installer to 1.9.0
new ODK Tables sync protocol and UI. Breaks ODK Tables Alpha 2 sync.
new sync protocol exchanges SAVEPOINT_TYPE, FILTER_TYPE, FILTER_VALUE

Upgrading

Upgrades require a manual intervention:

You must have Java 7 installed - the GAE 1.9.0 SDK used within the installer now requires that version of Java.

If you were testing out ODK Tables, you should delete all ODK Tables data tables before upgrading, as the database schema has changed.

Otherwise, this should be a seamless upgrade from ODK Aggregate 1.4.

Aggregate v1.4.2

(March 2014) 🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

Fixes to ODK Tables sync protocol for the combined release. Due to interactions with Google AppEngine, the ODK Tables Synchronization protocol does not currently work on Google AppEngine servers. You must install a local server or a VM image in order to use that mechanism

Upgrading

If you were testing out ODK Tables, you should delete all tables before upgrading, as the database schema has changed.

Otherwise, this should be a seamless upgrade from ODK Aggregate 1.4.

Aggregate v1.4.1

(February 2014) 🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

Summary of changes:

Enketo Webforms integration. You can now use Enketo's browser-based Webforms to fill-in and publish submissions directly into ODK Aggregate. This feature was developed and donated by SDRC India. To enable Enketo integration, go to the Site Admin / Preferences tab and click on Enketo API Configuration.
fix the Z-ALPHA JSON publisher and the JSON File export to emit an array of zero or more objects, one object per submission, with proper treatment of embedded quotes, etc. Confirmed that the output passes JSLint.
fix the CVS File export functionality to double-up all occurrences of double-quotes in a field before surrounding that field with double quotes (per RFC 4180).
clean up date and time handling in REDCap publisher and enforce GMT time zone interpretation when rendering date and time strings.
various GWT interfaces have been changed to use concrete ArrayList types (reduces code size).
new permissions have been added in support of ODK Tables -- Synchronize Tables and Administer Tables and for most ODK Tables interactions, the user is required to have Synchronize Tables permissions.
extensive changes to ODK Tables sync protocol and database structures. There will be further changes in the next update.
fix sizing calculations and CSS so that the ODK logo does not get clipped or overwritten.

Upgrading

If you were testing out ODK Tables, you should delete all tables before upgrading, as the database schema has changed.

Otherwise, this should be a seamless upgrade from ODK Aggregate 1.4.

Aggregate v1.4

(October 2013) 🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

Summary: Major changes to Aggregate's publishers to update to the latest jars and protocols, add new publishers, and improve quota usage.

changed behavior: simple JSON publisher now POSTs an application/json entity body; added option for how to treat binary content. Incompatible change; see Upgrading section if you were using the Z-ALPHA JSON Server
changed representation: ODK Tabled storage schema has been revised. Incompatible change; see Upgrading section if you used your server to upload or download data to ODK Tables. Syncing with ODK Tables is broken in this release. Only v1.3.4 on Google AppEngine works with the ODK Tables alpha (we are in the middle of changing table schemas).
fix: file exports were not properly writing UTF-8 character sets.
fix: postgreSQL failures on some forms (column names must be less than 63 characters)
Upgrade Aggregate’s Google publishers to use the updated Oauth2 libraries (e.g., Google Fusion Tables, Google Spreadsheet). Publishing from Google Apps domains does not work (it never has). This required a significant rewrite of the publishers so please notify the ODK core team of bugs!
NEW: Add support for publishing data to Google Maps Engine, more information can be found at GoogleMapsEngineInstructions
rework Spreadsheet and Fusion Table publishers to use Google libraries (consistent with the new Google Maps Engine publisher)
additional active-paused state in publisher to extend the delay in publishing attempts (and reduce quota usage) if the destination publisher is reporting an error.
improved how failures during form definition uploads are rolled back to increase the likelihood that the database is restored to a clean state.

Upgrading

You may need to clear your browser cache to complete the upgrade. If the browser screen flickers after upgrading, first clear your browser cache and reload the page.

If you were using the Z-ALPHA JSON Server, you must delete all instances of that publisher before upgrading. After upgrading, the updated publisher will send a single application/json entity body to the external server instead of a multi-part form containing that entity.

If you were using ODK Tables with ODK Aggregate, you need to delete all the ODK Tables data on ODK Aggregate before upgrading.

Aggregate v1.3.4

(August 2013) 🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

fix for the v1.3.3 fix for Google AppEngine -- The original fix caused the creation of new publishers to Google Spreadsheets to fail, export to files to fail, form deletions to fail, and purge-sent-submisions actions to fail. This fix should rectify those issues.

Aggregate v1.3.3

🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

fix for Google AppEngine -- entity keys (unique identifiers assigned by Google AppEngine and used internally by ODK Aggregate) may now contains slashes. Submissions that have been assigned an entity key containing a slash were breaking the 'SubmissionKey' parsing used when publishing, retrieving images, accessing repeat groups, or retrieving submissions using ODK Briefcase.

Aggregate v1.3.2

🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

expose the ODK Tables data and management tabs.
when installing for first time, ODK Aggregate will not require you to log in. Access restrictions are not altered when upgrading. This only affects new deployments. issue 710 (on Google Code) - upon an initial install, configure Anonymous User with Data Collector and Form Manager (and Data Viewer) permissions.
watchdog sweep interval shortened to 30 seconds in fast-publishing mode (from 60 seconds). Ensure watchdog is scheduled to be fired when there are records remaining to be published.
fix publisher failure to Google Spreadsheets and Fusion Tables by prepending 'n' to element names beginning with digits.
fix publishing failure with Fusion Tables when a form with repeat groups has submissions without any repeats.
issue 824 (on Google Code) - fix for malformed json publishing with null media attachments
issue 776 (on Google Code) - attempt to create Spreadsheets and Fusion Tables on behalf of the owner (vs. reassign ownership after the fact). Potentially fixes Google Apps Domain publishing issues (if service account is granted the permissions).
fix publisher-creation failure that can cause cycling UI refresh.
update javarosa library, adding format-date-time().
issue 806 (on Google Code) - briefcase not downloading instances with repeat groups containing images.
update to selemium 2.33.0 to resolve Firefox ESR 17.0.7 failures.

Aggregate v1.3.1

🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

change watchdog to run more frequently if there is an active publisher. Provide a 'disable' button on the Site Admin / Preferences page to restore older behavior (to conserve GAE quota).
issue 794 (on Google Code) - form upload failed for some forms on MySQL with stack exhaustion.
issue 761 (on Google Code) - Fix to simple JSON publisher. had caused instability when used.
issue 697 (on Google Code) - popups don't show centered in screen when displayed on top of scrolling regions.
issue 786 (on Google Code) - forms with repeat groups cannot be versioned.
issue 768 (on Google Code) - rows-per-page value keeps getting reset on refresh

Aggregate v1.3

(February 2013) 🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

A major motivation for this release was the impending OAuth 1.0 deactivation for Google Fusion Tables and, soon to follow, all Google services in general; Google's transition has been delayed because of a usage scenario that is not met with OAuth 2.0. The 1.3 release of ODK Aggregate makes a wholesale transition from OAuth 1.0 to OAuth 2.0, breaking all publishers. A lesser motivation is the end of support for Google Maps v1.0 API, scheduled for March 2013; but that shouldn't significantly impact users.

This release also contains 2 changes to the JSON file export and 3 new Alpha-quality publishers.

Installer now supports migrated AppEngine instances (for the Master-Slave -to- High-Replication Datastore migration).
Google Fusion Tables publisher now provides the links to the tables of all the repeat groups, the top-level record, and a left-outer-join view of the first repeat group and top-level record. This gives a 'flat' view of the data.
Google Spreadsheets and Google Fusion Tables publishers are now using OAuth 2.0 for authentication. This breaks all existing publishers (you need to republish). Additional user configuration is needed to register the necessary OAuth 2.0 credentials via the Site Admin / Preferences page. When setting up a publisher, you are no longer redirected to Google to allow access; the access credentials defined on the Site Admin / Preferences page bypass that step in the process; instead, you will receive an e-mail from Google notifying you of the newly published data having been shared with you.
Google Maps v3 API is now used for the visualization features.
issue 728 (on Google Code) 0 siteKey and userServiceKey have been removed from the installer and siteKey is now saved and accessed from the datastore (this enables easily-replicated AWS VM images to be constructed).
issue 718 (on Google Code), issue 730 (on Google Code) - some large forms cause table subdivision algorithm to fail, preventing the form from loading into ODK Aggregate. This alters the names of tables that are created and their contents.
Added a 'Published Through' and an 'Owner' column to the 'Published Data' table to communicate the progress of the publisher and who is receiving the data (really, the initial owner).
issue 724 (on Google Code) - move the watchdog to the frontend instance to minimize the use of the background thread. This change may cause issues on servers with a very large number of publishers or pending export jobs.
issue 653 (on Google Code) - spelling typo fix
issue 712 (on Google Code) - fix Fusion Tables link; changed Google Spreadsheets to be links, and to have these links open in new windows.
issue 558 (on Google Code) - Google Spreadsheet publisher failed badly if name was blank.
issue 587 (on Google Code) - extra comma in JSON file export (in repeat groups)
JSON file export now exports multiple-choice values as a JSON array of string values, rather than a space-separated string.
issue 648 (on Google Code) - arbitrary intermingling of http and https requests are problematic
issue 714 (on Google Code) - cannot save filter unless display metadata is checked.
issue 732 (on Google Code) - version string should be an integer that fits in an int.
issue 742 (on Google Code) - unable to change the title of an existing form
forms are now listed alphabetically
MySQL media attachments are stored as BLOB types, allowing the default MySQL configuration to work (the LONG_BLOB setting had required changes to the out-of-the-box MySQL database configuration)
annual update of all jars and jar dependencies (extensive); removed all OAuth 1.0 jars.
NEW! Alpha release REDCap (XML) publisher
NEW! Alpha release Simple JSON publisher
NEW! Alpha release Ohmage JSON publisher

See Publisher Details for information about the new publishers.

Upgrading

The upgrade will break all existing publishers. After the upgrade, ODK Aggregate needs to be configured with OAuth 2.0 credentials on the Site Admin / Permissions page. Once configured, you will then be able to create new publishers for your data (it is not possible to resume or restore publishing to the original publishers).

To avoid having to create new publishers that re-publish already-published data, you can:

Before upgrading: 1. go to the Form Management page 1. uncheck Accept Submissions 1. verify that all submissions appearing on the submissions tab have been successfully published to Fusion Tables and Google Spreadsheets.
at this point, because ODK Aggregate is not accepting any new submissions, your surveyors are unable to send filled-in forms and we can be assured that no data is in transit during the upgrade process.
Deploy ODK Aggregate 1.3.0
Go to the Publishers page, and create replacement publishers using 'Stream New Submissions ONLY`
Now go to Fusion Tables or Spreadsheets and copy the data from the 1.2 tables into the newly-created publisher tables.
Finally, go to ODK Aggregate 1.3.0's Form Management page and check Accept Submissions
at this point, new submissions will stream into the new publishers and you have manually copied the old data into the new publisher, so these new publishers now have all of your data.

The database tables for the new publishers and older publishers do not overlap, so if you roll back to the ODK Aggregate 1.2 release, you will not see the new publishers, but the earlier 1.2 publishers will 'reappear.' If you want to, after the upgrade, in MySQL or PostgreSQL, you can drop the unused old tables:

_server_preferences (has been replaced by _server_preference_properties)
_form_service_cursor (has been replaced by _form_service_cursor_2)
_fusion_table (has been replaced by _fusion_table_2)
_fusion_table_repeat (has been replaced by _fusion_table_repeat_2)
_google_spreadsheet (has been replaced by _google_spreadsheet_2)
_google_spreadsheet_repeat (has been replaced by _google_spreadsheet_repeat_2)

Aggregate v1.2

(July 2012) 🔻 Google disabled functionality used by this installer to upload ODK Aggregate to Google App Engine. See here for the work-around.

Updated javarosa library with cascading select support (as with KoBo Collect).
Add a Delete button to the Exports list to enable deleting the generated files.
Exported files using filters now export the metadata if displayed by the filter.
Improved Map visualization display and pop-ups (showing images)
Improved filter, export and publishing pop-ups.
Update to use LONGBLOB and LONGSTRING on MySQL (new tables only) See MySQL Usage Note Below
cache thumbnail images for 1 hr for improved performance and lower AppEngine datastore usage.
Form definition files and media attachments can now be altered and those changes uploaded to the ODK Aggregate server. The server still maintains only one version of the form, and all alterations must not affect the number of questions in the form or change the data type of any field (e.g., from int to decimal or string, etc.).
Whenever a form or any of its media files are modified, the version attribute in the top-level element (where the form id is defined) must be changed. Version attributes are recommended to be of the form "yyyymmddnn", e.g., 2012060400 -- the last two digits are the form iteration within the given day. They must be integer values and the new value must compare lexically greater than the prior value (this means, for example, since "9" compares lexically greater than "10", you cannot update a version from 9 to 10 -- but you could upgrade from "09" to "10").
There is a 15-minute grace period for uploading revisions after which the version must be incremented (e.g., incremented to 2012060401).
Note that you will need the soon-to-be-released updates to ODK Briefcase and ODK Form Uploader in order to use those tools with this version of ODK Aggregate.
Fix odd start-up failures on Google AppEngine
issue 596 (on Google Code) - timestamp comment in wrong location

MySQL Usage Note

The use of LONGBLOB and LONGSTRING requires a configuration change in the MySQL server. The server requires the transmisison packet size to be configured large enough to hold the largest LONGBLOB or LONGSTRING you will ever send to the server. See http://dev.mysql.com/doc/refman/5.5/en/connector-net-programming-blob.html section 22.2.5.10.1. Preparing the MySQL Server for configuring the max_allowed_packet size.

Alternatively, after creating your tables, you can use the MySQL ALTER TABLE command to change the LONGBLOB field to a BLOB field. This was the pre-1.2 setting, and will be the 1.2.1 setting for image fields, returning the system to use 65kB image chunks and avoiding the need to change the server configuration. If you do this, you will need to stop and restart your ODK Aggregate server for the change to be detected and take effect.

Upgrading

Existing 1.0 installations can upgrade to the 1.2 release, but, once upgraded, if you use the new form-versioning feature,these installations cannot downgrade from 1.2 to the earlier 1.0 releases.

NOTE: MySQL and PostgreSQL require special upgrade steps.

Upgrading Google AppEngine

See v1.1 Production Upgrading instructions.

Upgrading MySQL or PostgreSQL

The _filter_group table has a new column. If you are running ODK Aggregate 1.0 or 1.1, you will need to issue an "alter table" command on this table to add this column.

MySQL

The alter table command would be:

ALTER TABLE `_filter_group` ADD COLUMN `INCLUDE_METADATA` char(1) NULL;

PostgreSQL

The alter table command would be:

ALTER TABLE "_filter_group" ADD COLUMN "INCLUDE_METADATA" boolean NULL;

Depending upon your database management tool, you may need to qualify the table name with the schema.

PREVIEW: Aggregate v1.1

(June 2012)

PREVIEW RELEASE Production deployments should remain on the v1.0.9 release unless the features of this release are critical to your deployments.
Form definition files and media attachments can now be altered and those changes uploaded to the ODK Aggregate server. The server still maintains only one version of the form, and all alterations must not affect the number of questions in the form or change the data type of any field (e.g., from int to decimal or string, etc.).
Whenever a form or any of its media files are modified, the version attribute in the top-level element (where the form id is defined) must be changed. Version attributes are recommended to be of the form "yyyymmddnn", e.g., 2012060400 -- the last two digits are the form iteration within the given day. They must be integer values and the new value must compare lexically greater than the prior value (this means, for example, since "9" compares lexically greater than "10", you cannot update a version from 9 to 10 -- but you could upgrade from "09" to "10").
There is a 15-minute grace period for uploading revisions after which the version must be incremented (e.g., incremented to 2012060401).
Note that you will need the soon-to-be-released updates to ODK Briefcase and ODK Form Uploader in order to use those tools with this version of ODK Aggregate. Compatibility issues may arise as we understand the full ramifications of this change.
Note that ODK Collect 1.1.7 has buggy support for versioned forms. If you make use of versioning, you should use ODK Collect 1.2 RC1.
An updated ODK Aggregate 1.1 will be released at the time ODK Collect 1.2 becomes the official production release.

Upgrading

This is a preview release. The form-versioning capability may be tweaked between now and the 1.1.1 release.

Existing 1.0 installations can upgrade to the 1.1 release, but, once upgraded, if you use the new form-versioning feature,these installations cannot downgrade from 1.1 to the earlier 1.0 releases.

Aggregate v1.0.9 Production

fix issue 598 (on Google Code) - AppEngine over-quota page showing erroneously.

Upgrading

See v1.0.4 Production Upgrading instructions.

Aggregate v1.0.8 Production

fix erroneous release of TaskLock that could cause 60-second lock-out of "Add Forms", causing slow uploads of multimedia forms using FormUploader or Briefcase
fix performance bugs in the TaskLock for GAE.
fix correctness bugs in the TaskLock for MySQL and PostgreSQL that can cause two TaskLocks to be obtained for the same resource if they are both requested at exactly the same second.

Upgrading

See v1.0.4 Production Upgrading instructions.

Aggregate v1.0.7 Production

fix issue 505 (on Google Code) - double "Add Forms" leaves UI in inoperable state
fix issue 506 (on Google Code) - double "Add Forms" leaves UI in inoperable state
JSON file export supported
update installer to use Google AppEngine 1.6.4 SDK
remove uiVersion comparison tests (deprecated functionality).

Upgrading

See v1.0.4 Production Upgrading instructions.

Aggregate v1.0.6 Production

fix issue 542 (on Google Code) - Add a Refresh Credentials button on publishers. This button appears when the publishing to an external service (Google Spreadsheet or Fusion Tables) fails with a 401 (Unauthorized) error. Clicking the button will walk through the authorization screens for publishing to that service, yielding a new credential for that publisher.
fix issue 512 (on Google Code) - Add a Purge Submission Data button on the Form Management / Submission Admin tab. This button enables you to delete submission data up to a specified date. Incomplete submissions are not deleted.
fix issue 422 (on Google Code) - digest auth failing in 1.0.5 Release Candidate.
Includes all functionality from the 1.0.5 release candidate.

Upgrading

See v1.0.4 Production Upgrading instructions.

Aggregate v1.0.5 Release Candidate

Reduce memory requirements for KML and CSV generation to enable generation of larger datasets.
Restore bookmark navigation after login challenges - v1.0.4 did not navigate to a requested bookmark page if the user was challenged to log onto the site.
Add support for Oauth 1.0 publishing from Build into Appspot-hosted Aggregate instances. Oauth 1.0 tokens do not work on Tomcat servers.
Add support for Oauth 2.0 access to Aggregate. Aggregate will accept any Google token granting access to https://www.googleapis.com/auth/userinfo.email with access rights equivalent to the indicated Google user.

Upgrading

This is a Release Candidate and should be considered less reliable than a Production release; please test it thoroughly before deploying.

The primary changes in this release are in the areas of authentication, permissions and page navigation. All the typical interactions have been tested, but failures may produce unexpected errors.

No special upgrade steps are required when upgrading from v1.0.4; see v1.0.4 Production Upgrading instructions when upgrading from versions prior to v1.0.4.

Aggregate v1.0.4 Production Release

(January 2012)

issue 483 (on Google Code) - unable to log into MySQL or PostgreSQL deployments with openId.
issue 484 (on Google Code) - streaming does not function.
issue 486 (on Google Code) - odk:length not respected on Google AppEngine
Fix layout issue with HelpBook and misc. popups.
Update all security jars and misc. others to latest releases.
Update to Google AppEngine 1.6.1 SDK

Upgrading

NOTE: Google AppEngine requires special upgrade steps if you utilize odk:length.

Upgrading Google AppEngine

See v1.0 Production Upgrading instructions.

If none of your forms use odk:length to alter the maximum string length, then the upgrade is complete.

Otherwise, for ONLY those forms that use odk:length:

Download the forms that use odk:length using the ODK Briefcase application.
Delete the forms on Google AppEngine (this may take several minutes or hours if you have many submissions).
Upload the forms from ODK Briefcase back onto your Google AppEngine instance.

Upgrading MySQL or PostgreSQL

See v1.0 Production Upgrading instructions.

odk:length has always been respected on MySQL and PostgreSQL; there are no additional steps to be performed on those systems.

Aggregate v1.0.3 Production Release

NOTE: MySQL and PostgreSQL deployments of 1.0.3 suffer from issue 483 (on Google Code). As a work-around, if you can live without OpenID logins for google accounts, you can install 1.0.2, configure an ODK Aggregate username with password and Site Admin privileges, then upgrade to 1.0.3 and use the ODK Aggregate username to administer your site. Otherwise, continue to use 1.0.2; the fix for this issue will be in 1.0.4.

updates to make ODK Aggregate OpenRosa 1.0 compliant provided the version is an integer value.
issue 416 (on Google Code) - avoid full scan of submissions on the Submission Admin sub-tab.
issue 416 (on Google Code) - improvements to submission processing and publishing to minimize quota usage.
issue 416 (on Google Code) - change attachment handling to use lazy fetches, reducing quota. Eliminate immediate refetch upon change.
issue 468 (on Google Code) - form delete consuming significant quota.
issue 478 (on Google Code) - excessive quota during submission processing.
issue 478 (on Google Code) - improvements to submission processing to regulate the number of background tasks launched. Fixes to limit rate of Watchdog firings.
issue 478 (on Google Code) - move upload-submission background task to run on the foreground instance.
issue 474 (on Google Code) - case-sensitive hostname compare fails.
update libraries to the latest Apache commons and others
plumb missing quota exceptions up through services API
improve logging during background task activities
add logging and improve tracking of watchdog executions

Upgrading

See v1.0 Production Upgrading instructions.

Downgrading

See v1.0.2 Production Downgrading instructions if downgrading to a version prior to v1.0.2.

Aggregate v1.0.2 Production Release

fix issue 458 (on Google Code) - incorrectly presenting &frasl; for slashes in form ids.
fix issue 457 (on Google Code) - forms containing slashes or characters that should be escaped could not be deleted.
fix issue 426 (on Google Code) - publishing of dates and date-time values to Google Spreadsheet and Fusion Tables -- now recognized as dates.
fix bug in publishing that was using wrong date during retries. Could cause some data to not be published to Google Spreadsheets and/or Fusion Tables.
fix bug in image thumbnails that improperly set the size of the thumbnail
fix bug in kml export that didn't properly escape strings
fix bug in Google Spreadsheet publishing that delayed publishing start
log application version during start-up.
adjust refresh rates of the UI pages to lessen database accesses. issue 416 (on Google Code)
add over-quota error pages.
plumb up to the UI all over-quota exceptions -- as distinct from generic datastore failures.
optimize all queries for GAE operations. issue 416 (on Google Code)
move background process (/gae/watchdog) to run in backend instance.
move background activities (watchdog, publishing, csv/kml export, purge submissions and delete form) to backend instance.
watchdog is now run every 40 minutes, but only when activity is occuring on the webserver. If there is no work pending, the watchdog is not run. issue 416 (on Google Code)
enable multi-threaded webserver operations (should reduce instance-hour consumption on heavily-used sites).
improve low- and out-of-memory performance within GAE QueryImpl class. Lower the pre-fetch size and reduce the fetch limit to avoid timeouts on large dataset reads. issue 416 (on Google Code)
change publishing to use longer timeout values to Fusion Tables and to dynamically adjust for slower (or faster) external service connections.
reduce the batch size during publishing so as to minimize unnecessary datastore reads. issue 416 (on Google Code)
improve error logging during publishing and form deletion.
move Form creation and cache-management into FormFactory. Reduce cache lifetime to 3 seconds. issue 416 (on Google Code)
better encapsulate and hide FormDefinition object within Form object.
track and emit datastore query, query-read, get, put and delete usage counts (does not yet flush to log during webserver shutdown). issue 416 (on Google Code)

Upgrading

See v1.0 Production Upgrading instructions.

Downgrading

On Google AppEngine, should you need to roll back to an earlier release (e.g., v1.0.1 Production), you will need to manually delete the "background" backend. This can be done through the AppEngine dashboard, via the Backends link.

On MySQL and PostgreSQL, no special action is required.

Aggregate v1.0.1 Production Release

add 30-second cache for Form definition issue 416 (on Google Code).
change GAE queue configuration to limit retries on failure issue 416 (on Google Code).
add logging on datastore accesses issue 416 (on Google Code). Disabled for performance.
double watchdog interval to 6 minutes issue 416 (on Google Code).
double webpage background-refresh interval to 10 seconds issue 416 (on Google Code).
change the default number of records to display to 100 (from 500) issue 416 (on Google Code).
fix issue 410 (on Google Code) publishing does not flush all data to external services.
fix spurious error message on Form Management tab when Aggregate 1.0 has no form loaded (noted on issue 412 (on Google Code)).
fix integration test breakage.
fix datetime string parse error that was causing rounding errors in the millisecond digit. Impacts all data display, extraction, publishing and Briefcase activities.

Upgrading

See v1.0 Production Upgrading instructions.

Aggregate v1.0 Production Release

add version information on Site Admin / Preferences tab
fix date/time parsing problems that were affecting Briefcase data transfers.

Upgrading

If you are running RC4, RC3, RC2, or RC1 (on Google's AppEngine or with MySQL or PostgreSQL), there are no special upgrade steps. It should just work.

If you are using Beta 4, follow the upgrade instructions described under RC1.

For all other Alpha or Beta releases, you must either use a new appspot instance or delete ALL data in your appspot instance; if you are using MySQL or PostgreSQL with an Alpha or Beta release, you should start with an empty database.

Aggregate v1.0 Release Candidate 4 (RC4)

update javarosa library
allow null URI in resume cursor
fix issue 387 (on Google Code) (issue 380 (on Google Code)) - Briefcase download generates invalid Submission XML
NOTE: date/time data may not be properly parsed during Briefcase transfers.

Upgrading

If you are running RC3, RC2, or RC1 (on Google's AppEngine or with MySQL or PostgreSQL), there are no special upgrade steps. It should just work.

If you are using Beta 4, follow the upgrade instructions described under RC1.

For all other Alpha or Beta releases, you must either use a new appspot instance or delete ALL data in your appspot instance; if you are using MySQL or PostgreSQL with an Alpha or Beta release, you should start with an empty database.

Aggregate v1.0 Release Candidate 3 (RC3)

fix issue 345 (on Google Code) - unable to change user passwords on Tomcat deployments
fix issue 317 (on Google Code) - bookmarked pages weren't working because the hashes and query parameters were being stripped during authentication.
fix issue 375 (on Google Code) - csv downloads (and kml downloads) only emited the first 500 records, not the entire dataset.
filters could not be defined and/or were not working on Tomcat (MySQL, PostgreSQL) deployments.
KML file exports now honor the active filter and emit only what that allows (plus the image, title and location fields).
conform to OpenRosa Form Discovery API by handling the verbose and formID parameters.
include a specially-built javax imageIO library that is redistributable so that Tomcat instances show thumbnail images out-of-the-box.
integration tests are now working in the Maven build, using seliniumhq 2.8.0. Coverage reports are not yet available and test suite only verifies the app starts, the superUser can log in, and a form can be uploaded.

Upgrading

If you are running RC2 or RC1 (on Google's AppEngine or with MySQL or PostgreSQL), there are no special upgrade steps. It should just work.

If you are using Beta 4, follow the upgrade instructions described under RC1.

For all other Alpha or Beta releases, you must either use a new appspot instance or delete ALL data in your appspot instance; if you are using MySQL or PostgreSQL with an Alpha or Beta release, you should start with an empty database.

Aggregate v1.0 Release Candidate 2 (RC2)

improved help content, including links to youtube videos.
show only "complete" submissions on the Submissions tab.
row filters now properly implement hide logic
now paginates results on Submissions tab.
fix for Google Spreadsheets publishing issue.
new "Submission Admin" tab under "Form Management" supports marking as complete an incomplete submission (incomplete submissions are those lacking one or more of their audio, image or video captures).
only submissions that are marked as complete are published.
media file count on Forms List tab is now clickable and displays the filename, file type and file size of each uploaded media file associated with a form.
fix bug causing csv and kml downloads to not automatically open Excel and Google Earth, respectively.
fix bug causing Change Password to report failures (now waits up to 15 seconds for a server response before failing this operation).
fix bug causing Permissions page to only display 15 usernames.
update to gdata 1.46.0 libraries

Upgrading

If you are running RC1 (on Google's AppEngine or with MySQL or PostgreSQL), there are no special upgrade steps. It should just work.

If you are using Beta 4, follow the upgrade instructions described under RC1.

For all other Alpha or Beta releases, you must either use a new appspot instance or delete ALL data in your appspot instance; if you are using MySQL or PostgreSQL with an Alpha or Beta release, you should start with an empty database.

Aggregate v1.0 Release Candidate 1 (RC1)

further improvements to filters
csv file export now applies the display filter active at the moment the user presses the Export button.
restructure external services to make it easier to add new publishers
restartable cursors for paging through large sets of submissions (implemented at the datastore layer, but not yet up into the UI).
allow aggregate username to be designated as a superUser (not exposed in Installer; partial support in UI).
updated to GWT 2.4 and AppEngine SDK 1.5.4
remove Briefcase applet (functionality will be replaced by the Briefcase application).
add large dataset (310,000 record) tests for restartable cursor.

Beginning with Release Candidate 1, there will be a data migration path across all further upgrades; we don't expect any further incompatible schema changes. However, in the event that there is an incompatible schema change, you will be able to migrate your data to a newer ODK Aggregate 1.0 instance via a Briefcase application that can download submissions from ODK Aggregate 1.0 RC1 and onward and upload those submissions into future ODK Aggregate releases.

An initial release of the Briefcase application will be available soon.

Upgrading

To upgrade from Beta rev.4, follow the instructions below.

For all other Alpha or Beta releases, you must either use a new appspot instance or delete ALL data in your appspot instance before deploying the RC1. If you are using MySQL or PostgreSQL, you should start with an empty database.

On GAE

You don't need to do anything. It should just work.

On MySQL or PostgreSQL

The persistent results tables that hold the exported csv and kml files have changed (adding support for emitting csv files filtered by the active filter).

Stop Tomcat
In your database's administration tool, connect to the database and: * drop table _persistent_results; * drop table _persistent_result_file_bin; * drop table _persistent_result_file_ref; * drop table _persistent_result_file_blb;
copy the new WAR to the /webapps directory.
Start Tomcat

Aggregate v1.0 Beta rev.4

split management tab into 2 tabs (Management and Site Admin).
hide tabs that the user doesn't have permission to access.
add help tooltips and documentation pop-ups.
single-click hide/show of metadata in UI.
delete individual submissions through the UI.
make explicit that string fields are truncated to 255 characters.
allow upload of forms that do not have data types defined for all fields (with suitable warnings).
add odktables features.
added a new metadata field, _MARKED_AS_COMPLETE_DATE that tracks the date the submission was last marked as complete.
fix issue 311 (on Google Code) that causes infinite redraw loop if a form definition is corrupted.
fixed several bugs related to incomplete uploaded submissions -- submissions with only some of their media attachments present.
gain a global lock during instance start-up to ensure that only one instance might be changing the configuration during that time.
improve GAE query and task locking logic; introduce datastore settle-time logic, remove read-after-write attempts and inserted retries in key places to improve real-world performance.
extend timeouts for FusionTables to minimize publishing failures.
refactor MiscTasks, PersistentResults and FormInfo to no longer themselves be form and submission structures.
rework treatment of metadata so that it is more like other data.
clear session cookies more completely when logging out.

Upgrading

Changes to the database schema necessitate either using a clean appspot instance or deleting ALL data in your appspot instance before deploying the beta 4. If you are using MySQL or PostgreSQL, you should start with an empty database.

Aggregate v1.0 Beta rev.3 utf-8

This fixes 4 issues:

utf-8 characters are not displaying or downloading properly from a Google AppEngine server. (Issue 286 (on Google Code))
filtering on metadata fields does not work. (Issue 299 (on Google Code))
filtering does not work with date fields or select1 fields. (Issue 299 (on Google Code))
insert a 3 second delay in order for the datastore to settle before streaming newly-submitted data records to external services. TBD: This may not be enough time when GAE datastore is having performance issues.

This update also exposes the "meta-is-complete" metadata on a submission. If a submission with a media attachment (e.g., image, audio recording or video capture) is uploaded but the attachment is not, the submission will be marked as not complete (false). Only completed submissions are published to external services.

Upgrading

If you are running Beta rev.3, you should just be able to deploy this on top of that installation. If UTF-8 characters do not display properly in your form definition, you will need to delete the form (and submitted data) then upload the form anew.

Otherwise, if you are not running Beta rev.3, then follow the steps required for upgrading to Beta rev.3 prior to moving to Beta rev.3 utf-8.

Aggregate v1.0 Beta rev.3

Upgrading

To upgrade from earlier Alpha or Beta releases, you must...

On GAE

If running builds prior to Beta 2 on Google AppEngine cloud services, any forms with Decimal data will need to be deleted and reloaded. Decimal data had been represented as strings and is now represented properly as double-precision numbers in Google AppEngine. Note: while Briefcase can be used to download data, there is currently no way to upload your downloaded data into the beta 3.
Disable writes to the datastore (via Application Settings / Disable Datastore Writes )
Deploy to GAE.
Delete all the records in these kinds (using Datastore Viewer): * opendatakit._granted_authority_hierarchy * opendatakit._registered_users * opendatakit._user_granted_authority
Disable your application (via Application Settings / Disable Application )
Enable writes,
Enable your application,

On MySQL or PostgreSQL

Stop Tomcat
In your database's administration tool, connect to the database and: * drop table _granted_authority_hierarchy; * drop table _registered_users; * drop table _user_granted_authority;
copy the new WAR to the /webapps directory.
Start Tomcat

Upgrade Impacts

The primary impact is the loss of all registered users and their privileges (what the manual steps above do). Beta 3 changes the user configuration. Registered users are now either gmail accounts or ODK Aggregate usernames. ODK Aggregate now supports only anonymous access and/or registered users.

Known Issues

issue 265 (on Google Code) -- IE 8 doesn't refresh the submissions page on adding or deleting forms.
issue 261 (on Google Code) -- logins can time out while sitting on permissions page, causing unsaved changes to be lost.
issue 231 (on Google Code) -- large submission sets may cause UI to time out.

Changelist

recognize OpenRosa meta/instanceID tag and use it if present. Generalize to use the first meta/instanceID as the unique ID for a submission (don't require a namespace).
change-password now uses https when available (JSOP).
digest auth (ODK Aggregate logins) now have nonce timeout set to 30 minutes.
better logging of submissions and reasons for failures (for GAE diagnosis, since ODK Collect doesn't necessarily report this)
disable or hide UI functionality if user does not have required permissions for that functionality.
update to AppEngine SDK 1.5.1
update installer screens; https automatic on GAE deployment.
installer now deletes its temp directory
update security model (extensive changes)
hide password when deploying to Appspot (issue 222 (on Google Code))
fix launching of scripts and display of README on windows (issue 223 (on Google Code))
show the number of media files associated with a form (issue 224 (on Google Code))
fix crash on MacOS when some fonts are present (issue 236 (on Google Code))
fix lockouts when changing super-user (issue 242 (on Google Code))
hide forms that are queued for deletion from the UI (issue 255 (on Google Code))
remove the old UI (servlet) pages.
remove old applets
fix operations behind proxies (issue 212 (on Google Code))
fix lingering UTF-8 character display issues (issue 218 (on Google Code), 215)
fix new form and upload submissions links (issue 243 (on Google Code); 257) and IE page display and upload submissions bug.
fix PostgreSQL create-database scripting errors. (issue 256 (on Google Code))
changes to improve client-side javascript operations, especially the refresh timer operations.
changes to poll for configuration changes so that instances running in multi-server farms all eventually synchronize their security config and forms list.
changes to get working (depressingly...) minimal unit tests for gae, mysql, postgresql under maven.

Aggregate v1.0 Beta rev.2

fixes for Tomcat operations (missing images, styles)
updates to tabs, stylesheets and layout of new UI
configuration wizard improvements (Issue 222 (on Google Code), 223)
Postgresql install works (issue 227 (on Google Code))
updated to AppEngine SDK 1.5.0.1
add Tomcat image preview bean
simplified permissions tab
change Google AppEngine decimal value storage to use double.

Limitations:

will only display 1000 records
will only export 1000 records to csv and kml files
publishing broke as of Oct 2011.

To work around the publishing failures and the 1000-record limitation, use the Briefcase applet to download the data to a local csv file. Access the Briefcase applet through the URL:

https://your-application-id-here.appspot.com/Briefcase

Aggregate v1.0 Beta

Useful URLs during the install process:

Google AppEngine

Apache Tomcat 6

PostgreSQL 9.0

MySQL 5.1 Community Edition

Installer Special Instructions

You must have the Java runtime environment (JRE) installed. To download:

Browse to: http://www.oracle.com/technetwork/java/javase/downloads/index.html
Choose the Download JRE button.
Follow the instructions for installing the JRE.

Installer Known Issues

Can only select install folder; can't create it.

Aggregate 1.0 Beta Known Issues

Security and Multiple-Running-Instance Concerns -- see AggregateSecurity

Aggregate 1.0 Beta Feature Requests

Need indication of whether a form has associated media files

Determining Aggregate 1.0 Version

n the 1.0 Production release, the Version appears on the Site Admin Tab, under the Preferences sub-tab.

Publishing to Google is broken prior to RC2 (due to a change in Google infrastructure over the summer).

Here's my cheat sheet:

Beta 1 and earlier: permissions tab looks nothing like the tab in the Production release.
Beta 2: permissions tab looks like the Production release; Forms tab does not show the number of media files associated with a form.
Beta 3: display the number of media files associated with a form; publishing to Google is broken.
Beta 4: there are "Form Management" and "Site Admin tabs; publishing to Google is broken.
RC1: csv exports apply the active Filter to the exported columns; publishing to Google is broken.
RC2: submissions tab is paginated (shows 500 records per page by default); publishing to Google works.
RC3: kml exports apply the active Filter to the exported data. exports all data -- no longer truncates to the first 500 records.
RC4: submissions for forms with non-repeating groups are now properly reconstructed into Briefcase. date fields are broken.
Production: Version info is shown on the Site Admin / Preferences sub-tab; date fields work with Briefcase.