Tool Suite Javascript framework and formDef.json (Survey) format

Javascript Structure

This document is accurate as of rev 210, July 6, 2017.

Injected Interfaces

The Java framework on the Android device injects two Java interfaces (odkCommonIf and odkDataIf) into both ODK Tables and ODK Survey WebKits. Additionally, it injects one additional Java interface into each: odkTablesIf into ODK Tables WebKits and odkSurveyStateManagement into ODK Survey WebKits.

Within the Javascript, it is expected that all interactions with these interfaces will be done through wrapper objects. Specifically, for odkCommonIf and odkDataIf, Javascript programmers would invoke methods on the odkCommon and odkData objects defined in

system/js/odkCommon.js
system/js/odkData.js

The ODK Tables-specific interface is interacted with via the odkTables object defined in:

system/tables/js/odkTables.js

This wrapper object mostly invokes odkCommon to perform its actions, but does call the odkTablesIf injected interface's one method to load the list view portion of the split-screen detail-with-list-view layout.

The ODK Survey interface is invoked within the Javascript that implements the survey presentation and navigation logic and should not be directly called by form designers.

odkCommon.js

This creates a window.odkCommon object that wraps calls to the injected odkCommonIf Java interface. When loaded inside the App Designer, it also creates a mock implementation of the injected interface.

This class provides support for:

1. obtaining information about the runtime environment (e.g., Android OS version, etc.)
2. obtaining information about the currently-selected locale.
3. obtain the active user.
4. obtain system properties (e.g., deviceId).
5. emitting log messages to an application log.
6. translations of text, media files and urls.
7. conversion functions for retrieving and storing timestamps and intervals.
8. storing and retrieving session variables (transient values that persist for the lifetime of this WebKit).
9. converting relative paths of configuration files and of row-level attachments into URLs suitable for use in HTML documents (e.g., image src attributes).
10. constructing form references used to launch ODK Survey.
11. invoking arbitrary intents (external programs) on Android devices.
12. obtaining the results from an intent that was previously invoked.
13. exiting the current WebKit and specifying a return intent status value and extras bundle.

The explicit session variable interfaces (odkCommon.getSessionVariable(elementPath) and odkCommon.setSessionVariable(elementPath, value)) provide a mechanism to preserve the state of a webpage within the Java Activity stack so that no matter how nested the call stack to external applications becomes, it can be unwound and the state of the webpage recovered. Similarly, the invoking of arbitrary intents and the retrieving of their result intent status and extras bundle (excluding byte arrays) provides direct access to Android's native application inter-operation capabilities from within the WebKit. This interface is used within ODK Survey for media captures; the internal methods that accomplish this are in system/survey/js/odkSurvey.js. Within ODK Tables, this capability is used to navigate between HTML pages for general content, list views, and detail views (largely via the higher-level methods of the odkTables wrapper object). As a webpage designer, there is nothing preventing you from performing media captures from ODK Tables web pages, or from defining custom prompts within ODK Survey that launch into ODK Tables list views, etc. by leveraging one or the other of the odkSurvey or odkTables objects.

odkData.js

This creates a window.odkData object that wraps calls to the injected odkDataIf Java interface. When loaded inside the App Designer, a mock implementation of the injected interface is loaded that uses W3C SQL to emulate the injected interface's capabilities.

This class provides support for asynchronous interactions with a SQL database (internally, this is implemented via a SQLite database).

The interaction to get the active user's roles would be:

// declare a success function
var successFn = function( resultObj ) {
  // do success handling
  var roles = resultObj.getRoles();
  // this will be a list of the roles and groups the user
  // belongs to.
};
// declare the failure function
var failureFn = function( errorMsg) {
  // errorMsg is a text string. Typically the getMessage()
  // of the Java Exception that occurred during processing. 
  // do failure handling
};
//
// make the asynchronous request
odkData.getRoles(successFn, failureFn);

If the request failed, the errorMsg is the message returned from within the Java layer. As noted, this is typically the getMessage() of an exception.

Otherwise, the resultObj returned contains information about the outcome. This object is a wrapper object with accessor methods defined here

NOTE:

1. the color information is only present within ODK Tables. It is not computed and returned within ODK Survey.
2. the display names will need to be localized before use. See the APIs provided by odkCommon.

odkTables.js

As noted, this is here:

system/tables/js/odkTables.js

It provides methods to open ODK Tables generic web pages and list and detail views. These are generally just wrappers for calls to odkCommon to invoke the intents for those views.

odkSurvey.js

As noted, this is here:

system/survey/js/odkSurvey.js

It provides methods to capture media files and. like odkTables these are generally just wrappers for calls to odkCommon to invoke the intents for those actions.

other system/survey/js files

These files are generally not used by web page developers. They implement the survey form execution logic and their functions will be broadly covered later in this document.

Configuration File Structure

A complimentary description is provided in the user-level documentation. The tool suite stores its configuration and data files on the SDCard in a directory structure rooted at opendatakit. The directories underneath this are application names and provide isolation of data and configuration from one user-defined application to the next. By default, if not specified, the application name is assumed to be default. Underneath this are 4 top-level directories:

/opendatakit/{appName}
    /config
        /assets -- common shared configuration
            app.properties -- application properties
            index.html -- HTML home page for *ODK Tables* (if configured)
            /css -- common css files
            /fonts -- common font files
            /img -- common image files
            /js -- common javascript files written by you
            /libs -- common 3rd party javascript libraries
            ... additional directories and files
            ... 
            tables.init -- identifies what to process during initialization
            /csv/{tableId}[.{qualifier}].csv
            /csv/{tableId}/instances/{cleanrowId}/{row-level-attachment-files}
            ...
            /commonDefinitions.js -- shared translations (available in all webkits)
            /framework/frameworkDefinitions.js -- *ODK Survey* template translations 
            /framework/forms/framework/framework.xlsx -- XLSX framework form
            /framework/forms/framework/formDef.json   -- created from XLSX

        /tables
            /{tableId}/definition.csv  -- defines the data model of the table
            /{tableId}/properties.csv  -- properties of the tableId
            /{tableId}/forms/{formId}/{formId}.xlsx   -- XLSX form
            /{tableId}/forms/{formId}/formDef.json    -- created from XLSX
            ... optional additional form definition files
            /{tableId}/forms/{formId}/customTheme.css
            /{tableId}/forms/{formId}/customStyles.css
            /{tableId}/forms/{formId}/customPromptTypes.js
            /{tableId}/forms/{formId}/customScreeenTypes.js
            /{tableId}/forms/{formId}/{other-files}
            ... end optional additional form definition files
            /{tableId}/html/{optional-html-files}.html
            /{tableId}/js/{optional-js-files}.js
            ... any other directories and files you might want
            ... 
    /data       - database and file attachments for this application
    /system     - internally managed javascript files and configuration
        /js/odkData.js
        /js/odkCommon.js
        /tables/js/odkTables.js -- wrapper for odkTables functionality
        /survey/js/odkSurvey.js -- wrapper for odkSurvey functionality
        ... the remaining files are not directly accessed
        /lib/...  -- 3rd party libraries used by *ODK Survey*
        index.html -- *ODK Survey* top-level HTML
        /js/mock/... -- mock interfaces used in App Designer
        /survey/js/... -- *ODK Survey* javascript
        /survey/templates/... -- *ODK Survey* handlebars templates
    /output     - holds logging files, exported data
    /permanent  - available for device-only content (e.g., map tiles)

Internationalization

Internationalization of web page content is achieved through the API exposed in the odkCommon object and the configuration contained in the XLSX files for the framework form and for the forms with formIds that match their tableIds. I.e.,

/opendatakit/{appName}/config
   /assets/framework/forms/framework/framework.xlsx
   /tables/{tableId}/forms/{tableId}/{tableId}.xlsx

Within the framework XLSX file, the framework_translations sheet defines the translations for all of the ODK Survey form labels and prompts. On this page, the string_token column contains the identifier for a particular label or prompt translation. The text.default column provides the default translation for that label or prompt, and all subsequent text.{langCode} columns provide translations for each of those specific language codes (e.g., {langCode} might be es for Spanish). If the label or prompt supports image or media enhancements, there will also be image.default and image.{langCode} or audio... or video... columns providing differing content for those.

Also within the framework XLSX file, there can be an optional common_translations sheet following the same format as the framework_translations sheet. This can be used to provide an application-wide set of translations.

Similarly, within the {tableId}.xlsx file, there can be an optional table_specific_translations sheet that also follows the same format as the framework_translations sheet. This is used to provide a table-specific set of translations.

Defining the available locales

The list of {appName}-wide locales and the default locale are specified on the framework form's settings sheet. Individual ODK Survey forms may define additional translations, but those will be form-specific and are not available to ODK Tables web pages.

Locales are defined in two steps.

First, the survey row under the setting_name column on the settings sheet should have an explicit translation for each locale. Do this by creating columns labeled display.title.text.{langCode} for each locale {langCode}. It is recommended that you use the Android 2-letter language codes. These are the 2-letter ISO 639-1 language codes, with the exception that iw is used for Hebrew, ji is used for Yiddish, and in is used for Indonesian. Using these 2-letter codes will enable use of the Android system locale for selection of the display language if the {apppName} is so configured.

Second, for each of these {langCode} values, create a row on the settings sheet with that {langCode} value under the setting_name column. Then create columns labeled display.locale.text.{langCode} across the top of the settings sheet. Provide translations for this language choice or, alternatively, define those translations on the common_translations sheet and reference the corresponding string_token under a single display.locale column.

The default locale will be the top-most {langCode} locale that you specify on the settings sheet.

Referencing translations in ODK Survey XLSX files

To reference these translations within a question prompt in a survey, instead of specifying a value under a display.prompt.text column, you would create a display.prompt column and place the string_token from the translations sheet into that column, leaving any display.prompt.... columns empty. The same applies for hint and title text.

And, finally, in all surveys, you can always provide on-the-spot translations for a prompt label by creating another column display.prompt.text.{langCode} and specifying the translation for that language code directly on the survey sheet.

Referencing translations in ODK Tables web pages

In order to access translations, your web page must load the commonDefinitions.js file and the tableSpecificDefinitions.js files. Within ODK Tables web pages, you can then obtain the appropriate translation via:

var locale = odkCommon.getPreferredLocale();
// obtain the text translation for the 'my_string_token' token.
var translatedString = odkCommon.localizeText(locale, 'my_string_token');

Additional methods are available within odkCommon to test whether a translation exists, and to obtain a localized image, audio or video URL. Refer to that file for the available methods.

Accessing the list of locales and default locale

And finally, to access the list of locales, you can directly access that via:

window.odkCommonDefinitions._locales.value

This is an array of objects (no particular order). Each object has a display.locale entry that can be translated to the current display language, and a name which is the {langCode} for that locale.

And the default locale is available at:

window.odkCommonDefinitions._default_locale.value

XLSXConverter Production of Translations

After defining your translations on the framework and tableId XLSX files, the XLSXConverter must be run on these files to generate the translation files.

When the XLSXConverter processes the framework.xlsx file and emits two files (in addition to the formDef.json):

/opendatakit/{appName}/config
   /assets/commonDefinitions.js
   /assets/framework/frameworkDefinitions.js

Of these, the frameworkDefinitions.js just contains a representation for the content of the framework_translations sheet.

The commonDefinitions.js contains the content of the common_translations sheet and the list of locales and the default locale from the settings sheet (as described in the previous section)

When the XLSXConverter processes the {tableId}.xlsx file, it emits three files (in addition to the formDef.json):

/opendatakit/{appName}/config
   /tables/{tableId}/definition.csv -- data definition
   /tables/{tableId}/properties.csv -- table properties
   /tables/{tableId}/tableSpecificDefinitions.js 

The last of these, tableSpecificDefinitions.js, holds a representation for the content of the table_specific_translations sheet.

ODK Tables web pages

ODK Tables does not impose any structure to the HTML files a user may write. While the odkDataIf and odkCommonIf interfaces will always be injected into the WebKit, it is up to the user whether or not they make use of them (which generally means loading the odkData and odkCommon wrapper objects). A typical HTML file might be as follows:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
    <head>
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <link href="../../../assets/css/your.css" type="text/css" rel="stylesheet" />
        <script type="text/javascript" src="../../../assets/commonDefinitions.js"></script>
        <script type="text/javascript" src="../tableSpecificDefinitions.js"></script>
        <script type="text/javascript" src="../../../../system/js/odkCommon.js"></script>
        <script type="text/javascript" src="../../../../system/js/odkData.js"></script>
        <script type="text/javascript" src="../../../../system/tables/js/odkTables.js"></script>
        <script type="text/javascript" src="../../../assets/libs/jquery-3.2.1.js"></script>
        <script type="text/javascript" src="../js/example.js"></script>
    </head>
    <body>
        <br>
        <a href="#" onclick="odkTables.openTableToListView(null,
          'visit',
          'plot_id = ?',
          [exampleResultSet.getRowId(0)],
          'config/tables/example/html/example_list.html');">Examples</a>
        <script>
            $(display);  // calls the display() function in example.js when document ready
        </script>
    </body>
</html>

The DOCTYPE header defines the file compliance level (in this case, HTML 4.01 Transitional).

The <head> section sets the viewport and loads your css file. It will then typically load the 5 standard javascript files needed to support translations and the injected interfaces (commonDefinitions.js, tableSpecificDefinitions.js, odkCommon.js, odkData.js, and odkTables.js). This example then loads the 3rd-party javascript libraries that the user needs and which the user has provided in the /config/assets/libs directory and, finally, loads the javascript file they crafted that is specific to this web page (example.js).

Once all the scripts and resources on the page have loaded, the script tag at the bottom of the <body> section will invoke the display() function which was presumably specified in the user's example.js file.

If the web page can launch other web pages or external applications, and if it cares about the status of those requests or needs to process the extras in the result intents returned from those requests (e.g., to interpret a barcode), then the user's display() function, after it has initialized the page, should register a listener for action outcomes and call that listener once, directly, to process any outcome received prior to that registration (it will commonly be the case that these will have been received prior to the registration of this listener).

See the comments at the top of the odkCommon.js file for details.

ODK Survey form processing

The XLSXConverter converts the XLSX file defining an ODK Survey form into a JSON representation of that form. This representation is then layered on top of the generic ODK Survey javascript framework to produce the javascript code that is executed when filling out the form.

The primary building blocks of this generic ODK Survey javascript framework are:

• bootstrap - for prompt UI and behavior
• Handlebars - for HTML content rendering
• Backbone - for event handling within prompts
• requirejs - for module dependencies and loading
• jquery - generic utility functions
• underscore - generic utility functions
• moment - date and time handling support

Some additional libraries are use for specific widgets and capabilities (e.g., d3 for graphing, combodate for calendar widgets).

The ODK Survey javascript framework then adds form navigation, data validation, data storage and data retrieval functions. Central to this framework is the calling context which provides a continuation abstraction for chaining and resuming processing during asynchronous interactions.

The following sections will outline:

• ODK Survey Calling Contexts (ctxt)
• ODK Survey javascript modules
• ODK Survey control flow overview
• ODK Survey primitive controller actions
• ODK Survey formDef.json structure

ODK Survey Calling Contexts (ctxt)

The success and failure callbacks used within the odkData API are also used throughout the ODK Survey javascript. These are so common, that they are passed into functions as a single "calling context" argument, generally named ctxt. Whereas many libraries have success and failure callbacks:

obj.action(successCallbackFn, failureCallbackFn);

the ODK Survey javascript would just pass in the ctxt object:

obj.action(ctxt);

This ctxt object consists, at a minimum, of a success function and a failure function. The failure function generally takes one argument which is an object containing a message field that holds an error message. The success function may pass in an argument or not.

These calling contexts are created, tracked and managed by the controller class via:

window.controller.newContext( event )  -- when needed during event processing
window.controller.newCallbackContext() -- on callbacks from Java shim
window.controller.newStartContext() -- special case
window.controller.newFatalContext() -- special case

The ctxt object extends the baseContext defined within controller, which has:

{
	contextChain: [],
	append: function( method, detail ) {...},
	success: function() {...},
	failure: function(msg) {...},
}

A well-written success() or failure(msg) function will perform its actions then call the success or failure function of the parent instance from which it is extended. So you will often see code like this in ODK Survey javascript:

var that = this;
this.render($.extend({}, ctxt, { success: function() {
        that.postRender(ctxt);
    }, failure: function(msg) {
        ctxt.append(“mymethod”, “unable to render”);
        ctxt.failure(msg);
    } });

Where postRender(ctxt) will be responsible for calling the success or failure methods of the ctxt object that was extended and passed into the render() method. The failure(msg) code, in contrast, just logs a message to the context log (via append(), discussed below), and calls the parent instance’s failure function.

By always calling the parent instance’s success or failure function, you can do interesting things, like implement mutexes (an advanced software construct) -- because you are always assured that if you extend a ctxt, that one of your failure(msg) and success() functions will always be called.

The failure(msg) function takes an argument, which is an object that may contain an optional ‘message’ parameter, which could be a description of what the failure was. This is used during validation.

The use of the ctxt object enables you to store values within the ctxt, and ensure that these are available later in your code, or, via extending it, to change the success function so that it takes an argument, etc., as needed by your code (the database layer quite frequently needs to pass values into the ctxt success method).

The append() function on the context enables you to append a log record to the context. The baseContext’s success() and failure(msg) methods both cause the accumulated log messages to be written via the odkCommon.log(). On Chrome, the log message is suppressed. On Android, it is written to the /opendatakit/{appName}/output/logging directory and emitted in the system log if an error or warning.

The ‘seq:’ and ‘seqAtEnd:’ values emitted in these logs are useful for understanding what events are processed concurrently within the javascript. ‘seq’ is the sequence number of this context, and ‘seqAtEnd’ is the sequence number of the newest context in-process at the time this context completes.

Note that when interacting with other asynchronous frameworks, it is easy to convert from ctxt-based style to the success/failure function style:

fwk.action( function() { ctxt.success(); }, function() { ctxt.failure(); } );

Finally, these calling contexts are very similar to javascript promises. However, within the ODK Survey javascript, the typical construction is to insert processing steps before taking the success or failure action of the incoming calling context. In contrast, with promises, the typical construction is to append processing steps upon completion of the promise.

In the rare cases when it is necessary to append actions after a calling context chain completes (like the Promise model), two APIs are provided:

ctxt.setChainedContext(aCtxt);
ctxt.setTerminalContext(aCtxt);

Chained contexts are executed in-order, depth-first, from first registered to last registered, after which all terminal contexts are executed in the order in which they were collected from within all of the executed chained contexts. In practice, the ODK Survey javascript framework only makes use of terminal contexts, and those usages only register a single terminal context.

ODK Survey javascript modules

All user forms processed within ODK Survey load the same HTML file. Form-specific content and behaviors are specified via the window.location.hash portion of the URL. The common HTML file is here:

/opendatakit/{appName}/system/index.html

and its contents (as of July 2017) are:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>OpenDataKit Common Javascript Framework</title>
    <link rel="stylesheet" type="text/css" id="custom-styles" />
    <link rel="stylesheet" type="text/css" id="theme" href="libs/bootstrap-3.3.7-    dist/css/bootstrap.min.css" />
    <link rel="stylesheet" type="text/css" href="../config/assets/css/odk-survey.css" />
    <link rel="stylesheet" type="text/css" id="theme" href="libs/spinner/waitMe.css" />
	<script type="text/javascript" src="../config/assets/framework/frameworkDefinitions.js"></script>
	<script type="text/javascript" src="../config/assets/commonDefinitions.js"></script>
    <script type="text/javascript" src="js/odkCommon.js"></script>
    <script type="text/javascript" src="js/odkData.js"></script>
    <script type="text/javascript" src="tables/js/odkTables.js"></script>
    <script type="text/javascript" src="survey/js/odkSurvey.js"></script>
    <script type="text/javascript" src="survey/js/odkSurveyStateManagement.js"></script>
    <noscript>This page requires javascript and a Chrome or WebKit browser</noscript>
</head>
<body>
    <div id="block-ui"></div>
    <div class="odk-page">
        <div class="odk-screen">
            <div class="odk-toolbar"></div>
            <div class="odk-scroll">
                <div class="odk-container">Please wait...</div>
            </div>
            <div class="odk-footer"></div>
        </div>
    </div>
    <script type="text/javascript" data-main="survey/js/main" src="libs/require.2.3.3.js"></script>
</body>
</html>

This loads a /config/assets/css/odk-survey.css file that users can customize, loads the common javascript wrapper objects and translation files, and finally triggers requirejs to load the framework and (eventually) process the window.location.hash to load and interpret the form definition.

The requirejs module management framework, under the direction of the /system/survey/js/main.js configuration and initialization file, loads the javascript files used by the ODK Survey form framework.

Listed alphabetically, these are:

• builder - responsible for reading the formDef.json and initializing the controller with the list of prompts in the survey.
• controller - handles the logic for moving from one prompt to the next; this includes pre- and post- actions and performing the validation logic.
• database - Handles the interactions with the odkData interface to the database. This also constructs and maintains the in-memory model description holding the form definition and the instance’s data and of the structure of the table in which it is stored.
• databaseUtils - contains utility functions for transforming between the database storage strings and the javascript reconstructions in the model.
• formulaFunctions - common functions accessible from the user's Javascript eval environment (for use within their formulas).
• handlebarsHelpers - Handlebars helper functions for use within handlebars templates. These are invoked via {{helperFunction arg1}} or {{helperFunction arg1 arg2}} within the handlebars templates.
• main - the requirejs configuration and initialization file loaded via index.html that guides the javascript loading process. It waits for various components to load, cleans up the WebKit URL, and invokes parsequery.changeUrlHash(ctxt).
• odkSurvey - simple wrapper for invoking the various media capture actions exposed by ODK Survey
• odkSurveyStateManagement - this is used only within App Designer to simulate the injected Java interface of the same name.
• opendatakit - a random collection of methods that don't quite belong anywhere. Some of these cache and wrap requests to the odkCommon layer.
• parsequery - responsible for parsing the hash fragment and triggering the building of the form, the triggering the initialization of the data table, changing of the viewed page, etc.
• **prompts **- the core set of prompts defined by the ODK Survey javascript framework. The first of these, base, defines the basic operation of a prompt.
• promptTypes - due to the way requirejs works, this defines an empty object into which the prompts (above) are inserted.
• screenManager - handles the rendering of a screen, including any please-wait or other in-progress notifications, and the events that initiate actions on that screen (e.g., change language, swipe left/right, back/forward button clicks). Many of those actions invoke methods on the controller to complete. Note that rendering of the prompts within a screen (equivalent to an ODK Collect field-list) are handled within the definition of the screen.
• screens - the core set of screen renderers defined by the ODK Survey javascript framework. This includes the templating screen for customized layouts and the standard screen renderer.
• screenTypes - due to the way requirejs works, this defines an empty object into which the screens (above) are inserted.

ODK Survey control flow overview

index.html Initialization Sequence

The index.html file explicitly loads these script files:

• frameworkDefinitions.js - translations for standard ODK Survey buttons and prompts
• commonDefinitions.js - application-wide translations defined by the user
• odkCommon.js - wrapper object for odkCommonIf injected Java interface
• odkData.js - wrapper object for odkDataIf injected Java interface
• odkTables.js - wrapper object for odkTablesIf injected Java interface and convenience methods for ODK Tables navigation actions.
• odkSurvey.js - wrapper object providing convenience methods for media capture interactions.
• odkSurveyStateManagement.js - mock object used only within App Designer to provide functionality equivalent to the injected Java interface by the same name.
• require.js - the requirejs module management library
• main.js - loaded indirectly by requirejs to begin the module-load process

The relatively rapid loading of index.html very quickly presents ‘Please wait...’ to the user. This is not internationalized. Once the ODK Survey framework is initialized, this will change to an internationalized prompt (using the waiting_text translations), and then be replaced by the requested screen in the form (or first screen of the form) when the form definition is fully processed.

Main

The main.js file declares the interdependencies among the various javascript frameworks. It relies on requirejs for package dependency management and loading. The code first loads jquery and an extended regEx library (for Unicode strings). Once those are loaded, it then loads additional 3rd party libraries and the main ODK Survey javascript framework files via:

require([ 'spinner', 'databaseUtils', 'opendatakit', 'database', 'parsequery',
                        'builder', 'controller', 'd3', 'jqueryCsv', 'combodate'],
  function(...) {...})

Once the ODK frameworks has loaded, the body of the function is executed. The body then initializes the parsequery object (needed to avoid circular references):

parsequery.initialize(controller,builder);

And then either triggers a reload to clean up the window.location value or initiates the parsing of the formDef.json specified in the URL location.hash via:

parsequery.changeUrlHash(ctxt);

Parsequery

parsequery has two main entry points. The first:

parsequery.changeUrlHash(ctxt) {
    parsequery._parseParameters(wrappedCtxt);
    // when complete:
    that.controller.registerQueuedActionAvailableListener(ctxt, opendatakit.getRefId());

parses the formDef and calls the controller to initiate the processing of data callbacks from the Java layer.

The second entry point is _prepAndSwitchUI, which is called deep within the processing performed inside changeUrlHash(ctxt) and also by the controller when opening a specific instanceId within a form. That entry point assumes that the tableId and formId have not changed from what they currently are.

parsequery._parseParameters(ctxt) has the following flow (accomplished with many asynchronous processing steps -- arguments are omitted):

parsequery._parseParameters() {
    if ( !sameForm ) {
        controller.reset( function() {
            // webpage now displays ‘Please wait...’ with translations
            parseQuery._parseFormDefFile();
        });
    } else {
        parseQuery._parseQueryParameterContinuation();
    }
}

// called to load the (new) formDef.json
parseQuery._parseFormDefFile() {
    requirejs( ‘formDef.json’, function() {
        parseQuery._parseQueryParameterContinuation();
    })
}

// called to interpret hash parameters after formDef.json loaded
// If the tableId is changed, load information about the tableId
// from the database layer so we know what fields are in it.
// Otherwise, interpret the formDef.json and construct the 
// javascript objects that are used to render that form.
// And, once the object tree is initialized, call
// _prepAndSwitchUI() to render the specified screen in that form.
parseQuery._parseQueryParameterContinuation() {
    if ( !sameTable ) {
        controller.reset( function() {
            // webpage now displays ‘Please wait...’ with translations
            // Load information about the tableId from the database
            // layer so we know what fields are in it.
            database.initializeTables(function() {
                // parse and construct form objects
                builder.buildSurvey( function() {
                    // render the specified screen in this form
                    parseQuery._prepAndSwitchUI();
                });
            });
        });
    } else if ( !sameForm ) {
        controller.reset( function() {
            // webpage now displays ‘Please wait...’ with translations
            // parse and construct form objects
            builder.buildSurvey( function() {
                // render the specified screen in this form
                parseQuery._prepAndSwitchUI();
            });
        });
    } else if ( !sameInstance ) {
        controller.reset( function() {
            // webpage now displays ‘Please wait...’ with translations
            // render the specified screen in this form
            parseQuery._prepAndSwitchUI();
        });
    } else {
        // render the specified screen in this form
        parseQuery._prepAndSwitchUI();
    }
}

// retrieve and cache information for the instanceId (row) 
// being manipulated (if any) and render the specified screen
// in the current form
parseQuery._prepAndSwitchUI() {
    database.initializeInstance( function() {
        controller.startAtScreenPath(ctxt, screenPath);
    });
}

From this flow, you can see that the rough sequence of flow is:

1. controller.reset() is called to display ‘Please wait...’
2. database.initializeTables() to retrieve metadata about the tableId.
3. builder.buildSurvey() to process the raw formDef.json file.
4. database.initializeInstance() creates the initial (largely empty) row of an instanceId (if it is new) and reads the data for the instanceId from the database (if it is pre-existing), sets the current instance id and populates the mdl with the values for that instance id.
5. controller.startAtScreenPath() is called to direct theODK Survey javascript framework to display the requested screen.
6. controller.registerQueuedActionAvailableListener() is called to initiate the processing of any Java data callbacks (e.g., responses from intents).

Builder

Builder's only entry point is buildSurvey. This attempts to load several well-known files and then processes the formDef.json.

It begins by attempting to load (in order):

/opendatakit/{appName}
     /config/tables/{tableId}/tableSpecificDefinitions.js
     /config/tables/{tableId}/forms/{formId}/customScreenTypes.js
     /config/tables/{tableId}/forms/{formId}/customPromptTypes.js

The file tableSpecificDefinitions.js contains the translations described earlier.

The customScreenTypes.js file contains user-defined screen types. These should follow the constructions of the basic screens defined in /system/survey/js/screens.js and should be stored as property fields inside the screenTypes object.

The customPromptTypes.js file contains user-defined prompt types. These should follow the constructions of the basic prompts defined in /system/survey/js/prompts .js and should be stored as property fields inside the promptTypes object.

The column_types field in the specification object within the formDef.json is a map consisting of column names and their expected column types. This is used to convert ordinary text describing a calculation into javascript functions that perform the calculation (via eval). For simplicity, these column names are interpreted independent of the sheet within the XLSX file from which the formDef.json is constructed. The allowed values for column types is only partially extensible as it must be interpreted and processed within the builder. The valid column types are:

• function
• formula
• formula(arg1[, arg2[,...]])
• requirejs_path

Columns with the function type are expected to contain column values ({columnValue}) that are a text string that can be evaluated as a function definition -- e.g., {columnValue} would be something like: function() { return 3; }.

The formula type and the formula(...) type are expected to have {columnValue} be an expression that is the return value of a function. These are wrapped by the builder to construct either

function() { return ({columnValue}); }

or

function(arg1[, arg2[,...]) { return ({columnValue}); }

Function and formula column types have their content evaluated in the context of the methods exposed by formulaFunctions to produce javascript functions. Because they are evaluated within the formulaFunctions context, they only have limited access to the internals of the ODK Survey framework. This intentionally limits their power and the potential for damage that they might otherwise wreak.

The requirejs_path type causes builder to prefix the path to the form's directory. This supports referencing custom prompt templates and, potentially, images and other media, that are stored in the form directory.

The default column_types map can be extended in the XLSX file by defining a column_types sheet with headings that are column names and a single row beneath that defines the column type for that column name.

The default column_types map consists of:

{
    _screen_block: 'function',
    condition: 'formula',
    constraint: 'formula',
    required: 'formula',
    calculation: 'formula', // 'assign' prompt and on calculates sheet.
    newRowInitialElementKeyToValueMap: 'formula',
    openRowInitialElementKeyToValueMap: 'formula',
    selectionArgs: 'formula',
    url: 'formula', // external_link prompt
    uri: 'formula', // queries
    callback: 'formula(context)', // queries
    choice_filter: 'formula(choice_item)', // expects "choice_item" context arg.
    templatePath: 'requirejs_path'
}

Builder uses the column_types field in the specification object within the formDef.json to convert fields (column names) into their appropriate types. This conversion consists of a a full traversal of content from the calculates, settings, choices, queries, and all the survey sheets in the original XLSX file.

Next, for each of the survey sheets, builder creates Backbone instances of the prompt types referenced on those sheets, one instance for each declared prompt. These instances fold the field definitions the user specified in the XLSX file on top of the default values provided by the prompt definitions (and custom prompt definitions), allowing the user to customize the prompt through explicit changes in the XLSX file. These prompt instances are used when rendering the survey.

Lastly, the builder attempts to load:

/opendatakit/{appName}
     /config/tables/{tableId}/forms/{formId}/customStyles.css

It then attempts to load:

/opendatakit/{appName}
     /config/tables/{tableId}/forms/{formId}/customTheme.css

Or, if that doesn't exist, it examines the formDef.json to see if there was a theme defined on the settings sheet of the XLSX file and attempts to load:

/opendatakit/{appName}
     /config/assets/css/{theme}.css

And, lastly, it examines the formDef.json to see if there was a font-size defined on the settings sheet of the XLSX file and attempts to set it in the body:

$('body').css("font-size", fontSize.value);

Database

The ODK Survey database layer is a fairly thin wrapper around the odkData object. It maintains a cache of all of the field values in the referenced instanceId (row) within the current form. This cache is synchronously referenced and modified within the presentation layer and asynchronously updated via calls to the odkData object. In general, these asynchronous writes occur during lose-focus event processing.

Additionally, it maintains a copy of the properties of that table (e.g., display name of the table and display names of the fields) and a description of the field types in the database table (the table definition). These are returned via the odkData object. This information is used within ODK Survey to enable formulas to refer to field values either via their elementPath or via the database column in which they are stored (elementKey). A prime example of this is a geopoint. If the name of the geopoint field is mylocation then the individual latitude, longitude, etc. values are maintained within the cache as individual keys within a mylocation object -- you can refer to them naturally as mylocation.latitude, mylocation.longitude, etc. This is the elementPath representation of these fields. However, within the database layer, these are stored as individual columns with column names of mylocation_latitude, mylocation_longitude etc. That is the elementKey representation. A similar transformation occurs for file attachments and any user-defined complex data type (multi-valued prompts). Simple select-multiple prompts, which manipulate arrays of values, have an elementPath representation within the cache as a Javascript array of selected values. Within the database layer, their elementKey representation is a JSON serialization of this array (in contrast, select-multiple prompts that reference linked tables would not store their selections in the dominant data table but rely upon filter conditions and storing a (foreign) key in the subordinate table, or in an association table, to establish their linkage).

The support this synchronous cache and this data abstraction, the main entrypoints for this layer can be divided into 4 sections:

1. retrieving information about a database table
2. creating and deleting a database row
3. getting and modifying fields in a database row
4. utility functions for parsing selection and order-by clauses

• Retrieving information about a database table

Two methods:

initializeTables(ctxt, formDef, tableId, formPath)
readTableDefinition(ctxt, formDef, tableId, formPath)

The first is called during the initial loading of the form; the second is used by linked table prompts.

• Creating and deleting a database row

Five methods:

initializeInstance(ctxt, model, formId, instanceId, sameInstance, keyValueMap)
get_linked_instances(ctxt, dbTableName, selection, selectionArgs, displayElementName, orderBy)
save_all_changes(ctxt, model, formId, instanceId, asComplete)
ignore_all_changes(ctxt, model, formId, instanceId)
delete_checkpoints_and_row(ctxt, model, instanceId)

The first method, initializeInstance is used to initialize the synchronous cache with data values. It takes a boolean, sameInstance that is true if this is a reload of values for the current instanceId (row). It also takes a map of data changes keyValueMap to apply to this instance.

If sameInstance is true, this array is ignored.

If sameInstance is false and instanceId is null (we are not yet editing a row) then any initial values for the form's session variables that are specified in the keyValueMap are applied, and any initial values for any of the row's fields are ignored.

If sameInstance is false and instanceId is not null, the row's values are fetched from the database. If the row does not exist, it is initialized with the default values specified in the form for each of the row's fields, and then those changes are overlayed with the changes specified in the keyValueMap. And, finally, any initial values for the form's session variables that are specified within the keyValueMap are applied.

The second method, get_linked_instances is used by linked table prompts to retrieve rows from other data tables (e.g., for linked table prompts).

The remaining methods (save_all_changes, ignore_all_changes and delete_checkpoints_and_row) manage the retention and deletion of the row in the database table.

• getting and modifying fields in a database row

Five methods:

setValueDeferredChange( name, value )
getDataValue(name)
getInstanceMetaDataValue(name)
applyDeferredChanges(ctxt)
setInstanceMetaData(ctxt, name, value)

The first 3 of these methods are the standard setters and getters of values. In general, the metadata fields of a row are read-only within ODK Survey javascript. For this reason, there is no synchronous setter method for these fields.

The last 2 methods, applyDeferredChanges and setInstanceMetaData, are used internally within the ODK Survey javascript framework to flush the changes in the synchronous cache through to the database via calls to odkData. Nearly all manipulation of a row's instance metadata is done within the Java layer. The exception is the changing of the current row's locale, which is effected via the call to setInstanceMetaData.

• utility functions for parsing selection and order-by clauses

Two methods:

convertSelectionString(linkedModel, selection)
convertOrderByString(linkedModel, order_by)

These functions examine where clauses and order-by clauses to replace any elementPath expressions with elementKey values. Because this is not within the database layer, these conversions are not entirely fool-proof.

Controller

The initial load of a form ends with a call to controller.startAtScreenPath() followed by a call to controller.registerQueuedActionAvailableListener().

The controller object is responsible for navigating the form, ensuring that required fields are populated, that constraints are applied, that all validation logic is executed, and that appropriate actions are taken when the user launches an external application (e.g., for media capture), launches a sub-form, saves the form, exits without saving, or elects to delete a row from the database.

To implement back button functionality, the controller maintains a history of how the user has navigated through the form. This navigation history is necessary because there is no fixed execution path through an ODK Survey form (user-directed navigation is one of the big changes between the javarosa-based tools and ODK Survey). The odkSurveyStateManagement injected Java interface provides the underlying storage mechanism for this functionality and is directly called by controller during its processing.

The types of actions that the controller can perform, and how these are defined in the formDef.json will be described later in this document. At this time, it is sufficient to know that the controller is executing a program that performs actions, such as the rendering of a screen containing one or more prompts, as well as performing conditional and unconditional branches within that program.

The controller's progress through this program is tracked by the history stack maintained within odkSurveyStateManagement and the top of that history stack identifies the operation which the controller is currently executing. The controller's (vastly simplified) form processing flow is as follows:

controller.startAtScreenPath(ctxt, screenPath) {
    var op = operation corresponding to screenPath.
    controller._doActionAt(op);
}
//
// starting at the operation referenced by 'op',
// execute operations until a screen is rendered
controller._doActionAt(op) {
    controller._doActionAtLoop(op);
    // when the above completes, we are 
    // given a screenOp (screen rendering 
    // operation) to transition to, or 
    // have already produced a pop-up to 
    // communicate an error to the user.
    if ( screenOp !== null ) {
        controller.setScreenWithMessagePopup(ctxt, screenOp, ...);
    }
}
//
// main execution loop
controller._doActionAtLoop(op) {
    while () {
        switch ( op._token_type ) {
        case “goto_label”:
            // jump (possibly conditionally)
            // to another operation
            break;
        …
        // other control flow options
        // some of these can return out 
        // of this while without returning
        // a screen rendering operation.
        // any that do will have already
        // produced an alert or error pop-up
        …
        case “assign”:
            // do assignment
            break;
        case “begin_screen”:
            // render a screen
            return op; // the ‘screenOp’ in _doActionAt();
        }
    }
}
//
// render a screen
controller.setScreenWithMessagePopup(ctxt, screenOp, options, msg) {
    // set up a 500ms delay timer to render the ‘msg’ pop-up
    // so that the UI can settle on the new page before we 
    // display the message. Otherwise, it might be lost
    // during the rendering of the screen.
    setTimeout(function() {
        screenManager.showScreenPopup(m);
    }, 500);
    screenManager.setScreen(ctxt, screenOp, options);
}

i.e., the processing flow eventually calls screenManager to display a screen (via setScreen(ctxt, screenOp, options)) and perhaps also shows a pop-up with some sort of alert or error message (via showScreenPopup(m)).

When the next button is pressed or the screen is swiped forwards, the framework calls controller.gotoNextScreen() which verifies that all required fields are filled-in and all constraints are applied. It then triggers much the same processing sequence -- calling doActionAt() with the operation after the currently-rendered screen.

When the back button is pressed or the screen is swiped backward, the framework calls controller.gotoPreviousScreen() which pops the operation history stack for the current survey sheet until a screen-rendering operation is found, and that screen is then rendered. And, if the history for the current survey sheet is exhausted, then the contents screen for that sheet is displayed.

Finally, returning to the discussion of the control flow on the initial load of a form, after the current screen is rendered, the call to controller.registerQueuedActionAvailableListener() causes an action listener to be registered with odkCommon and then calls that listener to process any results that became available before the listener was registered. If there are any results from a previous odkCommon.doAction(...intentArgs...) request (e.g., a media-file capture request), then the controller's action listener will interpret the results to identify what prompt in the current screen should receive and process these results and then invoke that prompt to complete the processing. Otherwise, if there are no results, no additional actions are taken. This completes the control flow on the initial load of the form.

screenManager

The screenManager provides event handling for swiping and the navigation bars at the top and bottom of a screen. It delegates to the screen object to construct the DOM representation for that content and also delegates to the screen object to register and unregister event handlers for any other DOM elements via calls to recursiveUndelegateEvents() and recursiveDelegateEvents(). Those event handlers are expected to be defined in the Backbone-based screen objects and prompt objects.

The high-level actions of the screen manager are:

screenManager.setScreen(ctxt, screen) {
    // show 'loading...' spinner
    screenManager.showSpinnerOverlay();
    // stop processing all events on the current screen
    screenManager.disableSwipeNavigation();
    screenManager.activeScreen.recursiveUndelegateEvents();
    // construct the DOM objects in the page (heavily nested)
    screen.buildRenderContext(... {
        screen.render(... {
            screenManager.activeScreen = screen;
            // replace the screen
            screenManager.$el.find(".odk-page").replaceWith(screen.$el);
        });
    });
    //
    // and via a ctxt.terminalContext()  registration
    // so that the DOM replacement and redraw can take effect
    screenManager.activeScreen.afterRender();
    screenManager.activeScreen.recursiveDelegateEvents();
    screenManager.hideSpinnerOverlay();
}

screen

The screen object determines the set of prompts that should be displayed and lays them out. The custom screen example shows how this can be done within an arbitrary HTML template by using ids on DOM elements to identify where the inner HTML for a prompt should be injected.

Immediately prior to screen rendering, any unsaved changes in data values are asynchronously flushed to the database.

The screen object also enforces required fields and constraints and can reject any attempts by the controller object to move off of this screen or pop-up a confirmation for the user to accept.

See the screens.js file.

prompt

Prompts register event handlers for their DOM elements and are responsible for restoring and saving values displayed in those DOM elements into the synchronous data cache and for validating those values and enforcing any constraints (if so directed).

See the prompts.js file.

ODK Survey controller actions (details)

As mentioned earlier, the main processing loop within the controller executes a program derived from the form's XLSX file and encoded in the formDef.json. The 10 primitive operations in this program are:

• assign -- a synchronous assignment statement storing a calculation in a field.
• begin_screen -- push this operation on the navigation history stack and render its screen (this object is both an operation and an instance of a screen)
• goto_label -- jump to an operation identified by the given label. This may be a conditional jump. i.e., the 'condition' property, if present, will be a boolean predicate. If it evaluates to false, then skip to the next question; if it evaluates to true or is not present, then execute the 'goto'
• do_section -- push this operation onto the section history stack and mark it as 'advanceOnReturn' then start processing operation 0 in the specified section.
• exit_section -- pop the section history stack (removing the entire navigation history for this section) and if the last operation from the calling section is marked 'advanceOnReturn' then advance to the next operation otherwise execute the operation (unused). This undoes do_section.
• back_in_history -- pop the navigation history stack (unwind to the previous rendered screen) or, if this would exit the current section, display the contents screen for that section, or, if there is no previous screen, navigate to the screen specified when the form was initially loaded.
• advance -- either execute back_in_history if the current operation is marked 'popHistoryOnExit' or advance to the next operation.
• validate -- if this operation (validate) is not yet on the section history stack, push the operation and mark it with 'validateInProgress' (this enables the controller to return to the validate operation and resume the validation check after the user corrects the first invalid field value). Then traverse all fields with the indicated sweep_name and verify their compliance. Upon finding a field that fails validation, retrieve the begin_screen operation that contains that prompt, set that operation as the new current operation and mark it as 'popHistoryOnExit'. If all fields validate successfully, pop the section history stack (removing the 'validateInProgress' entry) and execute advance.
• resume -- pop the history stack (unwind to the previous rendered screen) and render that screen. If popping the history stack would have exited the section, exit the section and advance to the next screen after the 'do_section' command.
• save_and_terminate -- save all changes and close the WebKit.

ODK Survey formDef.json structure

The XSLSXConverter in the AppDesigner reads the XLSX form definition file and produces a set of output files, including formDef.json, as described earlier in this document.

In general, users of ODK Survey will not directly interact with this file. Instead, they would write their forms in the XLSX file. Several features of ODK Survey support that simplified usage:

1. custom screens and prompts can be defined in separate javascript files (see the earlier discussion of the Builder processing sequence). These custom prompts and screens can then be referenced by name in your XLSX form definition. This allows new widgets to be defined without extending the XLSX syntax or the XLSXConverter.
2. if additional member variables or functions are needed in your prompt logic, you can define these simply by adding a column with that member variable name to the survey sheet. If there is a value in that field, the member variable will be created and present in the prompt instance when the form is being rendered. Use the column_types sheet to define the interpretation of these members (e.g., to interpret them as functions or formulas). Member variables that are objects can be created by using '.' to separate the member variable name from the field name, as is done with the display.* columns, which define a member variable that is an object with various fields (e.g., display.prompt, display.text, etc.). Array-valued member variables can be created using [0], [1], etc. to specify the values in the array.
3. custom css styles and themes can be defined. Additionally, the full power of javascript is available when needed.

A primary goal of the formDef.json format was to preserve enough of the original source document (XLSX) to enable that document to be roughly reconstructed. The cell locations and values of all the cells in the XLSX file are retained, but formatting and coloring of the cells is not preserved. One can theoretically write an inversion program that would take this information and reconstruct an XLSX file without formatting or cell coloring that would be functionally equivalent to the original source XLSX file. This solves a common issue in the javascript-based tools where the conversion process fails to preserve enough information about the source document to enable it to be recreated.

Additionally, while the XLSXConverter performs numerous cross-checks, some errors cannot be reported until the form is executed. When these occur, error messages must identify the sheet, row and column in which the error occurred so that it is easy for form designers to correct the issues.

With this understanding, the structure of the formDef.json file consists, at the top level, of an object with 2 fields:

{
    xlsx: {...},
    specification: {...}
}

xlsx component

Note that alternative form description environments, such as drag-and-drop form builders, are expected to produce content that might be stored under a different field name. As those other tools develop, it is expected that some error message handling will need to be revised to properly report errors against those other source descriptions.

The purpose of this component is to enable an inversion tool to generate an XLSX file that is functionally equivalent to the source XLSX that generated this formDef.json.

NOTE: Writing custom prompts that directly reference the content of the xlsx component is fragile and should be discouraged. Future versions of the ODK Survey javascript framework may delete this component from the formDef.json structure that is retained during form execution. Remember that prompts will automatically possess member values and functions that you have defined on the survey sheets, so there is little need to retrieve information by directly access the formDef.json structure.

The xlsx object has field names that correspond to the names of the sheets in the originating XLSX file. Each sheet in the XLSX file is assumed to have a header row followed by data rows beneath it. The values for these sheet-name fields are arrays of objects, one or each data-row on that sheet. i.e., the header row is omitted. Each of these row objects will contain a _row_num field with the corresponding row number in the original XLSX file.

If a cell in the originating XLSX file's data-row was not empty, the corresponding data-row object will have a field with the column name from the header-row and this value as the field-value. For complex header-row column names, like display.prompt.text, the resulting data-row object will have a display field with an object value with a prompt field with an object value with a text field with the cell content. In cases where a value for the root cell: display.prompt.text and a cell field: display.prompt.text.en are both specified, the value in the root cell (display.prompt.text) will be pushed down into a default field.

Here is a portion of the xlsx structure showing the content of the first data row of the survey sheet from the example form:

"xlsx": {
    "survey": [
      {
        "type": "integer",
        "name": "default_rating",
        "display": {
          "prompt": "first_prompt",
          "hint": {
            "text": "If the form does not yet have a rating, this will be proposed for the rating value. This value is not retained in the survey result set and exists only for the duration of this survey session."
          }
        },
        "model": {
          "isSessionVariable": true
        },
        "_row_num": 2
      },
  ... 

Note that recreating the XLSX file from this structure is mechanical but the reconstruction cannot preserve the order of the header columns, since that information has already been discarded.

specification component

The specification component of the formDef.json object is the only part of that is active used by the ODK Survey javascript framework. This component contains the following fields:

• column_types -- used by builder. Can be extended by adding a column_types sheet in the XLSX file.
• settings -- content from the settings sheet in the XLSX file. This is an object with field names corresponding to the setting_name on that sheet with values corresponding to the data-row matching that setting name. Retrieve a given setting_name via a call to opendatakit.getSettingObject(opendatakit.getCurrentFormDef(), setting_name) There are accessor methods defined in the opendatakit.js javascript file for retrieving common settings values.
• choices -- content from the choices sheet in the XLSX file. This is an object with field names corresponding to the choice_list_name on that sheet. The values for these fields are arrays of objects, one object per row matching that choice_list_name in the order in which they appear in the choices sheet. This information is returned as part of all data row fetches and queries and is accessible on the odkData result object via calls to resultObj.getColumnChoicesList(elementPath) and, for individual data values, you can access the object corresponding to that data value most efficiently via resultObj.getColumnChoiceDataValueObject(elementPath, choiceDataValue). Within ODK Survey, the prompts use a wrapper function: opendatakit.getChoicesDefinition(choice_list_name) to access the choices list. The choices field should eventually be removed as the above calls on the result object are definitive and those choice lists come from the choices sheet of the form whose formId matches the tableId. The choices sheet within each form XLSX file will be retained until the AppDesigner and XLSXConverter can become smarter.
• queries -- content from the queries sheet in the XLSX file. This is an object with field names corresponding to each query_name on that sheet. The values for these fields are objects corresponding to the data-row matching that setting name. Retrieve a given query_name via a call to opendatakit.getQueriesDefinition(query_name)
• calculates -- content from the calculates sheet in the XLSX file. This is an object with field names corresponding to each calculation_name on that sheet. The values for these fields are objects corresponding to the data-row matching that setting name.
• section_names -- an array of the survey sections from the XLSX file. This includes the synthesized initial sheet if one is not explicitly specified.
• sections -- an object with field names corresponding to each of the section_names. Each such field defines the form content for that section (that sheet in the XLSX file).

After the builder has processed the form definition, the following fields are added:

• currentPromptTypes -- a list of all standard and custom Backbone prompt classes.
• currenScreenTypes -- a list of all standard and custom Backbone screen classes.

Additionally, the builder also scans and alters all of these fields from their original formDef.json content by applying the column_types field mappings to their content. See the Builder section, earlier, for how it replaces or modifies some string value content.

The following fields are present, but are not the authoritative source for this information and may be removed in future releases. They are present only to support the emulation of the Java environment when running in App Designer and are candidates for removal as that environment evolves:

• dataTableModel -- the authoritative version of this content is returned in response to a database query on a table. This is used within the App Designer to emulate the Java environment.
• model -- this content is an intermediate synthesis of the model sheet and all datatype attributions in the survey and survey sections. It is used to generate the definition.csv file. And, on the Java side, that file is used to create the database table and construct the dataTableModel returned by the odkData object.
• properties -- this content is used to generate the properties.csv file and is returned through a database query as metadata by the odkData object. It is only used directly when rendering the framework form, which is only done within the App Designer. The App Designer also uses it during database initialization.
• table_specific_definitions -- this content is written to tableSpecificDefinitions.js
• framework_definitions -- this content is written to /config/assets/framework/frameworkDefinitions.js
• common_definitions -- this content is written to /config/assets/commonDefinitions.js
• choices -- as noted above, this should eventually disappear and the choices sheet should eventually only be present in the form whose formId matches the tableId. That can't happen until XLSXConverter and the AppDesigner get smarter (i.e., this will likely persist in the formDef.json for longer than any of the above fields).

sections sub-component

Each section object in the sections sub-component contains a heavily processed and cross-checked version of that section of the survey. These objects have the following fields:

• section_name -- the section name -- i.e., the name of the sheet in the original XLSX file.
• nested_sections -- a map of all the section names that are targets of do_section actions within this section.
• reachable_sections -- a map that is the closure of all section names that can be recursively reached by all nested sections and by this section. This is used to ensure there are no cycles among the sections.
• prompts -- a list of all prompts within this section.
• validation_tag_map -- a map of all validation tag names and the array of prompts that reference that tag name (and that have value constraints). Prompts can specify a list of validation tag names that will enforce the prompt's constraints by specifying a space-separated list of values for a validation_tags column in their XLSX sheet. Intermediate validation of some prompt values can be achieved via the validate {tagName} action at any point in a survey. If nothing is specified for the validation_tags column, the prompt is automatically added to the finalize validation tag, which is processed when the Save as Complete action is initiated within the form.
• operations -- an array of operations that the controller iterates through to process this section of the form. Unless otherwise specified, processing starts at index zero in this array.
• branch_label_map -- a map of all branch (go-to) label names and the index within the operation array to which they correspond. Used to map the 'goto label' operation to a destination within the operations array.

After the builder has processed the form definition, the following fields are added:

• parsed_prompts -- a list of Backbone instances corresponding to the extension of the referenced Backbone prompt type with the field values found in the prompts list.

And builder also scans the operations list applying the column_types rules.

During form navigation, the parsed_prompts list of Backbone instances will be used to render DOM content and handle events. The prompts array may be removed by the builder in some future release. Each of these prompts has an XLSXConverter-generated field, _branch_label_enclosing_screen that identifies the branch label for the operation that will render the screen containing this prompt. This is used during validation to map back from the prompt whose constraints are violated to the begin_screen operation that will render the screen containing that prompt. Prompts also have _row_num and __rowNum_ fields that reference the XLSX row in the section that defines the prompt and the line number within the section (one less than the XLSX row number due to the presence of the header row), respectively. These are used for reporting exceptions during form loading and processing (i.e., malformed formulas, etc.).

operations sub-sub-element

Each element in the operations array describes an action the controller should execute when processing the form. The 10 primitive operation types were described in an earlier section. Below are brief examples of these various primitive operations.

Within all operations objects:

1. an operationIdx field contains the index into the operations array under which this operation object is stored.
2. the _token_type field contains the operation type.
3. the _row_num field contains the (first) row in the section that corresponds to this action. i.e., if a screen contained multiple prompts, this would be the row containing the begin screen action.

Here are specifics for each operation type:

• assign

assign actions can appear within begin screen ... end screen regions or outside of them. If they appear outside of them, they are interpreted as a separate operation by the controller. Here is an example of such an assign action:

  {
    "type": "assign",
    "name": "default_rating",
    "calculation": 8,
    "_row_num": 2,
    "__rowNum__": 1,
    "_token_type": "assign",
    "operationIdx": 0
  },

The key fields in this are:

1. name -- the field (session variable or a field in the data row) to assign.
2. calculation -- the expression to evaluate and assign in the field. This is converted by builder into a javascript function (i.e., transforming it into: 'function() { return (8); }' which is then evaluated).

• begin_screen

This is an example of a begin_screen operation object:

      {
        "clause": "begin screen",
        "_row_num": 20,
        "__rowNum__": 19,
        "_token_type": "begin_screen",
        "_end_screen_clause": {
          "clause": "end screen",
          "_row_num": 23,
          "__rowNum__": 22,
          "_token_type": "end_screen"
        },
        "_screen_block": "function() {var activePromptIndicies = [];\nassign('coffee_today', (( data('coffee_today') == null ) ? data('avg_coffee') : data('coffee_today')));\nactivePromptIndicies.push(11);\n\nreturn activePromptIndicies;\n}\n",
        "operationIdx": 22
      },

In addition to the standard fields, this contains:

• clause -- the action clause that this corresponds to. If this were generated by a lone prompt, the clause field would be missing.
• _end_screen_clause -- the clause that marks the end screen statement.
• _screen_block -- this field will be processed by builder to generate a javascript function. It encapsulates any assign operations and any if-then-else logic within the begin screen ... end screen region that determined which prompts should be shown on that screen. The function returns an array of the prompt indicies that should be rendered at this time.

Note that the above example shows how an if-then-else clause and assign action are transformed into a _screen_block

Here is another example, this one for a prompt that is not wrapped by a begin screen ... end screen action: { "_row_num": 2, "_token_type": "begin_screen", "_screen_block": "function() {var activePromptIndicies = [];\nactivePromptIndicies.push(0);\n\nreturn activePromptIndicies;\n}\n", "operationIdx": 0 },

• goto_label

If-then-else clauses outside of begin screen ... end screen regions are converted into branch labels and conditional and unconditional goto_label commands. Additionally, users may explicitly jump to a label using a goto clause in the XLSX file (and do that conditionally if they specify a condition predicate). Here is an example of a conditional goto_label operation object generated from an if clause.

      {
        "clause": "if",
        "condition": "selected(data('examples'), 'intents')",
        "_row_num": 4,
        "__rowNum__": 3,
        "_token_type": "goto_label",
        "_branch_label": "_then4",
        "operationIdx": 2
      },

The key fields for this are:

• condition -- present if the goto is conditional. If present, this is converted by builder into a javascript function.
• _branch_label -- where the goto should jump to.

Here is another example of an unconditional goto generated as a result of an end if clause:

      {
        "clause": "end if",
        "_token_type": "goto_label",
        "_branch_label": "_else9",
        "_row_num": 9,
        "operationIdx": 3
      },

• do_section

Here is an example of a do_section operation:

  {
    "clause": "do section household",
    "_row_num": 2,
    "__rowNum__": 1,
    "_token_type": "do_section",
    "_do_section_name": "household",
    "operationIdx": 0
  },

The key field here is:

_do_section_name which identifies the section name that should be jumped into.

• exit_section

Here is an example of an exit_section operation:

  {
    "_token_type": "exit_section",
    "clause": "exit section",
    "_row_num": 7,
    "operationIdx": 3
  },

• back_in_history

This is primarily used as a psuedo-instruction (an instruction injected into the operation stream) when the user hits the Back button or swipes backward. This is also emitted as a real operation when a back clause is specified in the XLSX file. Used in that manner, it can create a "dead-end" screen that the user cannot swipe through (they can only go backward) and can be useful when presenting a user with a user_branch prompt (where the user must choose the next action and there is no default action).

  {
    "clause": "back",
    "_row_num": 4,
    "__rowNum__": 3,
    "_token_type": "back_in_history",
    "operationIdx": 3
  },

• advance

This is only used as a psuedo-instruction (an instruction injected into the operation stream) when the user hits the Next button or swipes forward.

• validate

This is an example of a validate operation:

  {
    "clause": "validate user_info",
    "_row_num": 12,
    "__rowNum__": 11,
    "_token_type": "validate",
    "_sweep_name": "user_info",
    "operationIdx": 7
  },

Partial validation of a form is one of the advanced features of ODK Survey. In this instance, only the fields tagged with the user_info validation tag will be verified. The key field for this operation is:

_sweep_name -- the name of a validation tag. Any fields that have this name in their space-separated list of validation tags under the validation_tags column in the XLSX file will have their constraints validated.

If a field has a constraint but no values under the validation_tags column, finalize will automatically be assumed to be in that list. 'validate finalize' is called when a form is saved-as-complete.

• resume

None of our examples explicitly use this clause in the XLSX file. However, it is used in the construction of the default Contents screen handler for a section which is emitted if the form designer did not specify their own '_contents' branch label and define their own screen for this purpose. Choosing to view the contents screen causes a jump to the '_contents' branch. The default implementation of that branch is a begin_screen operation to display the Contents screen followed by a resume. The default Contents screen has its hideInBackHistory field set to true. This causes that screen to not be saved in the back history. When a user swipes forward, the resume operation will scan backward to the screen before the Contents screen (since it is skipped) and will render that screen (returning the user to the screen they were last at).

  {
    "_token_type": "resume",
    "clause": "resume",
    "_row_num": 9,
    "operationIdx": 12
  }

• save_and_terminate

This is not explicitly used in our examples, but it is used within the automatically-generated 'initial' section if the user has not defined their own. This operation corresponds to a 'save and terminate' clause. That clause takes a 'condition' expression that indicates whether the content should be saved-as-complete or saved-as-incomplete (this clause does not itself determine the validation status and hence completeness of the data). Because of this, any save-as-complete action should be preceded by a 'validate finalize' clause to ensure that the form is validated. After saving the form contents, the ODK Survey window is then closed.

 {
    "_token_type": "save_and_terminate",
    "clause": "save and terminate",
    "calculation": true,
    "_row_num": 9,
    "screen": {
      "hideInBackHistory": true
    },
    "operationIdx": 11
  },