Preparing to upgrade to CHT 4.0

Introduction

Medic uses Semantic Versioning (aka “SemVer”) which means that the CHT upgrade from the major 3.x version to the 4.x version denotes there are breaking changes. The key to a successful upgrade will be to understand and plan for these breaking changes. Aside from the Docker hosting infrastructure (out of scope for this prep document), the two breaking changes are around CHT Android and Enketo.

While CHT 4.0 has not been released yet, the effort to be prepared can be quite time consuming, especially for large deployments that may need to do handset upgrades in a worst case. The sooner deployments start preparing for the upgrade, the easier it will be when it comes to the upgrade itself. Conveniently, all device and Android app changes to prepare for 4.x are backwards compatible with 3.x. Prepare now, so you will be ready to upgrade sooner than later!

CHT Android v1.0.0+

This change is straightforward in that CHT 4.x no longer supports versions before 1.0.0, so deployments need to update their Play Store app. As of this writing, CHT Android is at 1.0.4. Please see the Android docs on how to update your app and release it. Note that Google’s Play Store can often have delays which deployments have no control over. Again, the sooner you start, the better.

Versions in use

After you have published your app, you need to instruct your users to check the Play Store for upgrades. You can then check CHT Telemetry to see what CHT Android versions are in use. Assuming you have couch2pg set up to pull in your CouchDB data to a PostgreSQL database, this query will list Android versions counts for the current year, broken out by month, year and version:

SELECT
	concat(doc#>>'{metadata,year}','-',lpad(doc#>>'{metadata,month}',2,'0')) as telemetry_month, 	
	doc#>>'{device,deviceInfo,app,version}' as cht_android_version,
	count(distinct(doc#>>'{metadata,user}')) AS count_distinct_users,
	count(distinct(doc#>>'{metadata,deviceId}')) AS count_distinct_devices,
	count(*) AS count_telemetry
FROM
	couchdb_users_meta 
WHERE
	doc#>>'{type}' = 'telemetry'  
	AND doc#>>'{metadata,year}' = to_char(current_date, 'yyyy')
GROUP BY 
	telemetry_month, cht_android_version	
ORDER BY 
	telemetry_month DESC,
	cht_android_version DESC NULLS LAST

Note that each user can submit many telemetry docs (count_telemetry), so the query breaks out users (count_distinct_users) and devices (count_distinct_devices) for the given month. This means that telemetry counts will be higher than the number of active users. As well, early in the current month, many users may not have had a chance to synchronize their telemetry data yet. For example, this report was run on the 5th of October, so the counts for all three tables are low. Refer to prior months in this case.

Active user counts

You can check the monitoring API with curl to check active users for the last 30 days. Remember that the users can send a telemetry report once per day, so this active user count will be less than the counts from the query above.

We’ll use jq to filter out the unrelated metrics. Be sure to replace CHT-URL-HERE with your production CHT URL:

curl -s https://CHT-URL-HERE/api/v2/monitoring\?connected_user_interval\=30 | \
  jq '. | {connected_users}'

This call will show the JSON with active users for the last 30 days:

{
  "connected_users": {
    "count": 80
  }
}

Users in need of upgrade

In the above table we can see there’s 4 users in the most recent month that are on version v0.11.0-webview. Let’s find their username so we can follow up with them directly. This query is hard coded for the current year (2022), the current month (3) and the version v0.11.0-webview. Be sure to update the query to fit your needs according to which month, year and version you need to search for:

select 
	distinct doc#>>'{metadata,user}' as user, 
	doc#>>'{device,deviceInfo,app,version}' as cht_android_version
from 
	couchdb_users_meta 
where 
	doc#>>'{type}' = 'telemetry'  
	and doc#>>'{device,deviceInfo,app,version}' is not null 
	and doc#>>'{metadata,year}' = to_char(current_date, 'yyyy')
	and doc#>>'{metadata,month}' = to_char(current_date, 'FMMM')
	and doc#>>'{device,deviceInfo,app,version}' = 'v0.11.0-webview'

In this case only one user is found:

A twist on this query is to remove the version filter. Do not remove the year and month filter as you will get duplicate values for users who have upgraded over time:

select 
	distinct doc#>>'{metadata,user}' as user, 
	doc#>>'{device,deviceInfo,app,version}' as cht_android_version
from 
	couchdb_users_meta 
where 
	doc#>>'{type}' = 'telemetry'  
	and doc#>>'{device,deviceInfo,app,version}' is not null 
	and doc#>>'{metadata,year}' = to_char(current_date, 'yyyy')
	and doc#>>'{metadata,month}' = to_char(current_date, 'FMMM')

This will give a list of every user and the version they’re running as of the current month and year:

CHT Conf

CHT Conf has been upgraded to be aware of forms written to work in CHT 3.x that may not work in CHT 4.x. Upgrading, can help you identify forms in need of fixing when pushing to dev instances as outlined below.

To upgrade app, run npm update cht-conf

cht-conf-test-harness

The cht-conf-test-harness has been upgraded (in version 3.x of the cht-conf-test-harness) to support testing CHT 4.x forms. When preparing to upgrade to CHT 4.x, it is important to use the latest version of the cht-conf-test-harness for automated testing.

(Note that the 3.x version of cht-conf-test-harness only supports CHT 4.x. If you are still running CHT 3.x, you should continue using cht-conf-test-harness 2.x.)

The breaking Enketo changes included in CHT 4.x are reflected in cht-conf-test-harness 3.x. Running your automated tests with the latest test harness can help identify potential form issues.

Enketo

CHT 4.0 upgrades the version of Enketo used to render forms. This upgrade provides a ton of bug fixes and enhancements (particularly around ODK spec compliance) which will make the forms experience in the CHT even better! (For example, we now have proper support for repeats with a dynamic length, including the various XPath functions necessary to take full advantage of this functionality!) That being said, it does introduce a few changes which may affect the way your forms function (or even cause some forms to fail to load at all).

Manual testing

You can also manually test your forms on a non-prod CHT instance. It is possible to test your forms against the new Enekto changes without having to uplift your non-prod CHT instance to the new 4.0 architecture.

An easy way of doing this is to use the CHT Docker Helper to deploy a 3.x CHT instance. After you have your dev instance up and running, use Horticulturalist to upgrade to the 3.17.0-FR-enketo-upgrade feature release:

COUCH_URL=https://medic:password@*your-my.local.ip.co-address*:8443/medic horti --local --install=3.17.0-FR-enketo-upgrade-beta.1

After pushing your app config (see “CHT Conf” above), you can proceed to go through each of your forms in a browser and on a device to ensure there’s no errors.

Notable changes to form behavior

XPath expressions

• Proper syntax in XPath expressions is more strictly enforced (e.g. parameters passed to the concat function must be separated by commas)
  • The + operator can no longer be used to concatenate string values in an expression. Although previous versions of Enketo supported this functionality, it was never part of the ODK Specification. The concat function should be used instead.
• The behavior of expressions referencing invalid XPath paths (both absolute and relative) has changed. Previously, an invalid XPath path (one pointing to a non-existent node) was evaluated as being equivalent to an empty string. So, /invalid/xpath/path = '' would evaluate to true. Now that expression will evaluate to false since invalid XPath paths are no longer considered equivalent to empty strings.
  • Validation has been added to cht-conf that can detect many invalid XPath paths and will provide an error when trying to upload a form.
• The value returned for an unanswered number question, when referenced from an XPath expression, has changed from 0 to NaN. This can affect existing logic comparing number values to 0.

Layout

• The horizontal and horizontal-compact appearances are now deprecated (a warning will be displayed by cht-conf when uploading to the server). The columns, columns-pack, and columns-n appearances should be used instead. See the documentation for more details.
• Markdown syntax is now supported for all question labels (and not just note fields).

Updated XPath functions

• The format-date and format-date-time functions no longer accept month values that are <= 0 (e.g. 1984-00-23). This is notable because some patterns for calculating dates based on an offset of a certain amount of years/months relied on this functionality (e.g. birth date).
  • See the details below regarding the new add-date function for a cleaner way of calculating dates based on an offset.
• The behavior of the today function has changed to return the current date at midnight in the current timezone instead of at the current time. To get the current date and current time use the now function.
• decimal-date-time:
  • The behavior of this function has changed with regard to the default timezone used when calculating the decimal value of a date that does not include any timezone information. (Note that the values from basic date questions do NOT include time zone information.) Previously, the timezone used in the calculation for dates with no timezone information was UTC. Now, the user’s current timezone will be used.
    • Practically speaking, this means it is no longer safe to assume that the output from decimal-date-time, for a value from a basic date question, will be a whole number. Now it is likely that a decimal value will be returned (with the numbers after the decimal point representing the offset of the user’s timezone from UTC).
  • Previously this function would accept various string parameters (e.g. date strings with various formats) as input. Now, the only string values it will only accept are ones formatted according to ISO 8601 (e.g. 2022-10-03).
    • Strings containing date values should be parsed with the date function before calling decimal-date-time.

New XPath functions

• Custom CHT functions:
  • add-date - Adds the provided number of years/months/days/hours/minutes to a date value.
• ODK Functions:
  • Repeats and other node sets:
    • position - Returns the current iteration index within a repeat group.
    • count/count-non-empty - Returns the number of elements in a repeat group.
      • Now is re-calculated when the size of the repeat changes
  • once - Returns the value of the given expression if the question’s value is empty. Otherwise, returns the current value of the question.
  • checklist/weighted-checklist - Returns True when the required number of affirmative responses is given.

Known issues

• The answer to a non-relevant question is not immediately cleared when the question becomes non-relevant and is still provided to XPath expressions that reference the question. When the form is submitted, the non-relevant answers will be cleared and any dependent XPath expressions will be re-evaluated.
  • This behavior applies both to questions that were answered and later became non-relevant as well as to questions which have configured default values.
• In 3.x versions of the CHT you could simulate a bulleted list in a form label (e.g. a note) by using - or *. Now, in 4.x, the bullets are no longer properly displayed in the label due to this bug.

Community cht-upgrade-helper

An unofficial cht-upgrade-helper utility has been created by a community member that can help flag form elements that should be manually reviewed before upgrading to 4.0.0. It is not officially supported by Medic and is provided as-is.