User telemetry

Performance data of certain user actions

Introduced in v3.4.0

The app collects performance data on certain user actions which is then aggregated each day and replicated to the server. This can be used to evaluate the performance of the code and configuration and to evaluate where improvements can be made.

The aggregate doc for the previous day is created when the first telemetry item is recorded each day. This is stored in the medic-user-<username>-meta database on the device and replicated to the server when an internet connection is available. This user specific server db is then replicated into the medic-users-meta database which holds all aggregate telemetry docs for all users.

The aggregate docs’ IDs follow the pattern telemetry-<year>-<month>-<day>-<username>-<uuid>.

Performance data

Each aggregate data point has the following fields.

FieldDescription
sumA sum of all the recorded times in milliseconds.
minThe smallest time recorded in milliseconds.
maxThe largest time recorded in milliseconds.
countThe number of times recorded.
sumsqrThe sum of squares of the times recorded in milliseconds.

The telemetry data gathered changes with different versions of the framework. Currently, the data points collected are:

FieldDescription
boot_timeThe overall boot time including loading the code, purging, and accessing the database.
boot_time:1:to_first_code_executionThe time between the page loading and the JavaScript starting to run.
boot_time:2:to_bootstrapThe time between JavaScript starting and the bootstrapping (purging, initial replication, etc) to complete.
boot_time:3:to_angular_bootstrapThe time between bootstrapping completing and the webapp being ready to use.
boot_time:4:to_db_warmedThe time between the webapp being ready to use and the database being ready to use.
enketo:reports:<form>:<action>:<component>The time taken to fill in Enketo forms. The action can either be “add” or “edit”. The component is one of: “render” covers getting the form and rendering it on screen; “user_edit_time” is the time the user took to fill in and submit the form; or “save” is about converting the form into a report and saving it.
enketo:contacts:<form>:add:<component>As above but for Contact creation forms.
enketo:tasks:<form>:<action>:<component>As above but for forms on the Tasks tab.
search:contactsThe time taken to list all contacts.
search:contacts:<filter[:filter]>The time taken to search all contacts using the given filters.
search:reportsThe time taken to list all reports.
search:reports:<filter[:filter]>The time taken to search all reports using the given filters.
client-date-offsetThe difference between the client datetime and the server datetime. Only recorded if the difference is large enough that it may cause issues.
analytics:targets:loadThe time taken to load the targets page. Added in 3.9
tasks:loadThe time taken to load the tasks page. Added in 3.9
tasks:refreshThe time taken to refresh tasks on the tasks page. Added in 3.9
rules-engine:initializeThe time taken to initialize the rules-engine . Added in 3.9
rules-engine:update-emissionsThe time taken to update emissions in the rules-engine, when receiving a change. Added in 3.9
rules-engine:tasks:all-contactsThe time taken to fetch tasks for all contacts in rules-engine. Added in 3.9
rules-engine:tasks:some-contactsThe time taken to fetch tasks for some specific contacts in rules-engine. Added in 3.9
rules-engine:targetsTime taken for the rules-engine to fetch targets. Added in 3.9
rules-engine:targets:dirty-contactsNumber of “dirty” contacts[1] when fetching targets in the rules-engine. Added in 3.9
rules-engine:tasks:dirty-contactsNumber of “dirty” contacts[1] when fetching tasks in the rules-engine. Added in 3.9
rules-engine:ensureTaskFreshness:cancelThe time taken to cancel the automated task freshness thread in the rules-engine. This event is only recorded when the thread is cancelled before executing the refresh. Added in 3.9
rules-engine:ensureTargetFreshness:cancelThe time taken to cancel the automated target freshness thread in the rules-engine. This event is only recorded when the thread is cancelled before executing the refresh. Added in 3.9
replication:user-initiatedNumber of times the user clicked “Sync now”. Added in 3.12.
replication:<database>:<direction>:successTime taken to replicate, when replication was successful. Added in 3.12.
replication:<database>:<direction>:failureTime taken to replicate, when replication failed. Added in 3.12.
replication:<database>:<direction>:failure:reason:offline:clientNumber of times replication failed because of connection errors, and the app detects the client is offline. Added in 3.12.
replication:<database>:<direction>:failure:reason:offline:serverNumber of times replication failed because of connection errors, and the app detects the client is online. Added in 3.12.
replication:<database>:<direction>:failure:reason:errorNumber of times replication failed because of errors other than connection errors. Added in 3.12.
replication:<database>:<direction>:docsNumber of replicated docs. For medic replication, stores number of “read” docs, for meta replication, stores sum of read docs for every direction. Added in 3.12.
replication:medic:<direction>:ms-since-last-replicated-dateTime between a replication attempt and the last successful replication. Only recorded for medic database, every time replication is attempted. Added in 3.12.
replication:medic:<direction>:deniedNumber of times replication was denied[2]. Added in 3.12.

[1] “Dirty” indicates that the contact’s task documents are not up to date. They will be refreshed before being used.
[2] Replication can be denied when the user doesn’t have permissions to create a doc (hierarchy permissions) or when a doc fails a validate_doc_update check.

Unless otherwise specified, database and direction placeholders stand for any combination of:

databasedirection
medicfrom or to
metasync

Metadata

When the aggregate doc is created the Telemetry service also includes a snapshot of some metadata.

FieldDescription
yearThe year the data was collected. Added in 3.4.0.
monthThe month the data was collected. Initially the month was 0 indexed (eg: 0=Jan, 1=Feb, …), but from 3.8.0 this bug was fixed so it changed to 1 indexed (eg: 1=Jan, 2=Feb, …). Added in 3.4.0.
dayThe day of the month the data was collected. Added in 3.12.0.
userThe username of the logged in user. Added in 3.4.0.
deviceIdA unique key for this device. Added in 3.4.0.
versions.appThe version of the webapp. Added in 3.5.0.
versions.forms.<form>The version of each form. Added in 3.5.0.
userAgentThe userAgent string from the user’s browser. Added in 3.4.0.
hardwareConcurrencyThe number of cores reported from the browser. Added in 3.4.0.
screen.widthThe width of the screen in pixels. Added in 3.4.0.
screen.heightThe height of the screen in pixels. Added in 3.4.0.
deviceInfo.app.versionThe version name of the Android app. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.app.packageNameThe package name of the Android app. Only captured when running in the Android wrapper v0.9.0+. Added in 3.12.0.
deviceInfo.app.versionCodeThe Internal version code of the Android app. Only captured when running in the Android wrapper v0.9.0+. Added in 3.12.0.
deviceInfo.software.androidVersionThe version of Android OS. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.software.osApiLevelThe API of the Android OS. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.software.osVersionThe version of Android OS (detailed). Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.hardware.deviceThe Android device name. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.hardware.modelThe Android model name. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.hardware.manufacturerThe Android device manufacturer. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.hardware.hardwareThe Android device hardware. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.hardware.cpuInfoThe Android device CPU information. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.storage.freeThe available storage on the device. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.storage.totalThe total storage on the device. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.ram.freeThe available RAM on the device. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.ram.totalThe total RAM on the device. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.ram.thresholdThe level of RAM at which certain services will be killed by Android. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.network.downSpeedThe reported download speed of the network. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
deviceInfo.network.upSpeedThe reported upload speed of the network. Only captured when running in the Android wrapper v0.4.0+. Added in 3.8.0.
dbInfo.doc_countThe number of docs in the local database. Added in 3.4.0.
dbInfo.update_seqThe update sequence of the local database. Added in 3.4.0.
dbInfo.idb_attachment_formatThe format of database attachments. Added in 3.4.0.
dbInfo.db_nameThe name of the local database. Added in 3.4.0.
dbInfo.auto_compactionWhether or not auto compaction is set. Added in 3.4.0.
dbInfo.adapterThe database adapter being used. Added in 3.4.0.

Export data to JSON

To export all telemetry in JSON for further analysis or visualization, first meet these prerequisites:

  1. Ensure that both node and npm are installed and that the needed node libraries are installed: npm install inquirer pouchdb-core fs path minimist pouchdb-adapter-http
  2. Get a current copy of the export script: curl -s -o get_users_meta_docs.js https://raw.githubusercontent.com/medic/cht-core/master/scripts/get_users_meta_docs.js

To export the data open a terminal in the folder where you want to save the export, and run this command:

node get_users_meta_docs.js --mode batch --type telemetry https://USERNAME:PASSWORD@COUCHDB_SERVER:COUCHDB_PORT/medic-users-meta > telemetry.json

For example, if your username is admin, your password is pass, your CouchDB server is localhost and your CouchDB port is 5984, you would run:

node get_users_meta_docs.js --mode batch --type telemetry https://admin:pass@localhost:5984/medic-users-meta > telemetry.json

This will save a file named telemetry.json containing all the telemetry data in the current directory.

Code Samples

Logged in from the browser

{
  "_id": "telemetry-2020-5-medic-016304ab-7167-4c97-93bb-a6626ef6128d",
  "_rev": "1-90a94d76eb30ac47e2f498b80cf54cd1",
  "type": "telemetry",
  "metrics": {
    "boot_time": {
      "sum": 3879.43500012625,
      "min": 308.0550000304356,
      "max": 1110.6599999475293,
      "count": 7,
      "sumsqr": 2616858.529000253
    },
    "boot_time:1:to_first_code_execution": {
      "sum": 2096.309999935329,
      "min": 141.66500000283122,
      "max": 558.6599999805912,
      "count": 8,
      "sumsqr": 672987.1618553334
    },
    "boot_time:2:to_bootstrap": {
      "sum": 628.5150001058355,
      "min": 70.50499995239079,
      "max": 129.88000002223998,
      "count": 7,
      "sumsqr": 58704.48214660012
    },
    "boot_time:3:to_angular_bootstrap": {
      "sum": 1415.5550000723451,
      "min": 45.25000008288771,
      "max": 787.9549999488518,
      "count": 7,
      "sumsqr": 732805.5498113176
    },
    "search:contacts:types": {
      "sum": 3728.9899999741465,
      "min": 96.60000004805624,
      "max": 1134.1099999845028,
      "count": 8,
      "sumsqr": 2939824.639961114
    },
    "search:reports:subjectIds": {
      "sum": 3825.224999745842,
      "min": 27.304999995976686,
      "max": 982.2900000144728,
      "count": 25,
      "sumsqr": 1461182.0137715642
    }
  },
  "metadata": {
    "year": 2020,
    "month": 5,
    "day": 20,
    "user": "medic",
    "deviceId": "016304ab-7167-4c97-93bb-a6626ef6128d",
    "versions": {
      "app": "master",
      "forms": {
        "client_review": "2-51ad4bd1c86c18ea768196da8264147e",
        "contact:clinic:create": "16-8dbb4dde6f8f9067e9cbed5b44ce5fd0",
        "contact:clinic:edit": "2-c1c955042db963a95470bdf891bcef6f",
        "contact:district_hospital:create": "2-42b8b579b38ca6c2978c238e7d9c78fe",
        "contact:district_hospital:edit": "2-4360bb0eb6f6ff863849a07adac7e851",
        "contact:health_center:create": "2-c9b86820479f211c8e505eba150c5203",
        "contact:health_center:edit": "2-19b3da5f9f146be07ec1854a372e374b",
        "contact:person:create": "2-230d39c41682da720b83e4909b138c9b",
        "contact:person:edit": "6-75781492122157beb7c2451861970ece",
        "death_report": "2-97cfa3cc24ee743dfea03c019d644bec",
        "delivery": "2-adf37c0da7d5e58c8e4d27fc940f70e6",
        "enroll": "10-13322d437e301f3c436e3b989a3fd384",
        "no_contact": "4-0901afd126ec6c87fe952ca3c3a652ed",
        "pnc_danger_sign_follow_up_baby": "2-12e8989f6774828bb10aa4b675bc43ff",
        "pnc_danger_sign_follow_up_mother": "2-62f58b6e0ade10d70f15df73ddb6a025",
        "pregnancy": "2-4fa3db96cae638c4cb041125c9970496",
        "pregnancy_danger_sign": "2-49758bfd894fb5db6e719bff864f9da9",
        "pregnancy_danger_sign_follow_up": "2-a9a2b6800d6f93e151a3a63359620976",
        "pregnancy_facility_visit_reminder": "2-0e3deaf22966166d05bcf803615499c8",
        "pregnancy_home_visit": "2-283eb323e61a7fc756bda61f9d9be382",
        "referral_for_care": "2-627c6c21bd54e482133acc8188a71f8d",
        "undo_death_report": "2-3a4ca8d678f0338d7044cb1bc35d103d"
      }
    }
  },
  "device": {
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Safari/537.36",
    "hardwareConcurrency": 4,
    "screen": {
      "width": 1920,
      "height": 1080
    },
    "deviceInfo": {}
  },
  "dbInfo": {
    "db_name": "medic",
    "purge_seq": "0-g1AAAAEzeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjPlsQBJhgNA6v____ezEhnwqnsAUfefkLoFEHX7CalrgKibj1tdkgKQTLLHa2dSAkhNPX41DiA18XjVJDIkyUMUZAEAuYBi9g",
    "update_seq": "1456-g1AAAAE2eJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjMlMiTJ____PytxAg4FSQpAMskerGYhLjUOIDXxYDVTcKlJAKmpB6tZhENNHguQZGgAUkBl87MSl-JVtwCibn9WYjtedQcg6u7jdj9E3QOIuv9ZSQwMjDVZAFZfZ6g",
    "sizes": {
      "file": 67196510,
      "external": 65800620,
      "active": 65357308
    },
    "other": {
      "data_size": 65800620
    },
    "doc_del_count": 41,
    "doc_count": 497,
    "disk_size": 67196510,
    "disk_format_version": 7,
    "data_size": 65357308,
    "compact_running": false,
    "cluster": {
      "q": 8,
      "n": 1,
      "w": 1,
      "r": 1
    },
    "instance_start_time": "0",
    "host": "http://localhost:5988/medic/",
    "auto_compaction": false,
    "adapter": "http"
  }
}

CHT Core Framework > Overview > Data Flows

An overview of data flows in the CHT for analytics, impact monitoring, and data science