O3 & OCL Config Workflow: How to show concept errors?


Hi @suruchi @burke @dkayiwa & @michaelbontyes, (& CC @ibacher & @mksrom),

Currently trying to solve a concept-related issue/error message w/ @hadijah315 & @jwnasambu. (Related thread here.) I assumed that if I went into dev3’s Admin page → OCL Module → Status Page I would find the usual list of subscription-related errors. image But when I do, this is all I see:

You’re probably thinking “well duh, we’re using the /ocl zip file workflow for dev3, not the Subscription-link-workflow, so of course there’s no error messages showing here”. My questions are:

  • Is there another way of seeing concept/metadata-related errors in a single place, other than here?
  • If not: Should we show any ocl-related error logs here? That’s where I expected to find them at least.

Brief explainer video: Loom | Free Screen & Video Recording Software | Loom

Curious to hear your opinion.

1 Like

Discussing now w/ @mksrom: We think there are 2 needs:

  1. Easier human-readable files, eg. upload a CSV directly instead of a zip file. OCL does already provide CSV exports for concepts and mappings. @suruchi @burke do you think we could make the /ocl iniz setup support uploading CSVs as well as the zip files with the JSON?

  2. Improve logging upon importing concepts when using “Iniz + OCL Module”. Currently there’s no way to see or check any errors associated with your concepts unless you use the Subscription Workflow (which I’m doing in our demo environment but this is both tedious and double-loads concepts which feels precarious for a real-world implication!)

CC @ruhanga

Grace, do you mean zip file upload at import page?

No, I mean the files we upload here in the Iniz config file for our distro: openmrs-distro-referenceapplication/distro/configuration/ocl at main · openmrs/openmrs-distro-referenceapplication · GitHub

Currently these are all .zip files with JSON inside. Per Romain, It would be easier for many implementers if they could also/instead be CSVs.

1 Like

A few notes here:

  1. The OCL “exports” to CSV are not equivalent to the JSON exports. Among other things, they are limited to 1000 entries (so only 1000 concepts and only 1000 entries per file) and I don’t see an easy way of generating multiple files. They also just do not have all the data we’d expect (for example, names other than the default name).
  2. Data fidelity: CSV are sometimes easier for humans to read, but they aren’t capable of capturing as much data as the JSON documents are. This is because, fundamentally, in both OCL and OpenMRS concepts are best represented as graphs of objects rather than flat lines. This gets difficult when we need to start specifying attributes of sub-components (for example, in both CIEL and OCL, we have a full set of properties for concept names to include things like: UUID, locale, actual text, whether it’s preferred / fully specified, etc.). While there are hacks that we can perform, we start losing most of the properties that make CSVs seem desirable (more attributes mean more columns which stop getting to a point where things are “easy” for humans to parse).

Just to try to make it clear, here’s the representation of a concept from the OCL CSV file:

id public access created at updated at created_by_id updated_by_id is active extras uri version released retired is latest version name full name default locale supported locales website description external id concept class datatype comment parent_id versioned_object_id mnemonic counted index
2186712 View 2022-08-16T22:59:45.100619+00:00 2022-08-16T22:59:45.112942+00:00 1 1 True {} /orgs/CIEL/sources/CIEL/concepts/1098/2186712/ 2186712 False False False en 1098AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA Frequency N/A 42 2620 1098 True True

And here’s the same concept in the JSON format:

    "child_concept_urls": [],
    "concept_class": "Frequency",
    "created_on": "2022-08-16T22:59:45.100619Z",
    "datatype": "N/A",
    "descriptions": [
            "description": "Time descriptor.  Every month.",
            "description_type": null,
            "external_id": "1101FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF",
            "locale": "en",
            "locale_preferred": false,
            "type": "ConceptDescription",
            "uuid": "2641931"
    "display_locale": "en",
    "display_name": "Monthly",
    "extras": {},
    "id": "1098",
    "is_latest_version": false,
    "locale": null,
    "names": [
            "external_id": "1172BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB",
            "locale": "en",
            "locale_preferred": true,
            "name": "Monthly",
            "name_type": "FULLY_SPECIFIED",
            "type": "ConceptName",
            "uuid": "2641928"
            "external_id": "125258BBBBBBBBBBBBBBBBBBBBBBBBBBBBBB",
            "locale": "en",
            "locale_preferred": false,
            "name": "Once per month",
            "name_type": null,
            "type": "ConceptName",
            "uuid": "2641929"
            "external_id": "125280BBBBBBBBBBBBBBBBBBBBBBBBBBBBBB",
            "locale": "en",
            "locale_preferred": false,
            "name": "Once a month",
            "name_type": null,
            "type": "ConceptName",
            "uuid": "2641930"
    "owner": "CIEL",
    "owner_name": "CIEL",
    "owner_type": "Organization",
    "owner_url": "/orgs/CIEL/",
    "parent_concept_urls": [],
    "previous_version_url": "/orgs/CIEL/sources/CIEL/concepts/1098/1675480/",
    "retired": false,
    "source": "CIEL",
    "source_url": "/orgs/CIEL/sources/CIEL/",
    "type": "Concept",
    "update_comment": null,
    "updated_on": "2022-08-16T22:59:45.112942Z",
    "url": "/orgs/CIEL/sources/CIEL/concepts/1098/",
    "uuid": "2186712",
    "version": "2186712",
    "version_created_by": "ocladmin",
    "version_created_on": "2022-08-16T22:59:45.100619Z",
    "version_url": "/orgs/CIEL/sources/CIEL/concepts/1098/2186712/"
1 Like

I find the JSON easier to read than the CSV. Probably because am a dev. :smiley: