Preparing OCL for OpenMRS for OCL API version 2

OCL Squad,

It’s time for us to start testing API v2. The OCL dev team has been hard at work and API v2 is expected to be ready for production soon (i.e., within the next few weeks). We have a copy of the OCL client running at openmrs.qa2.openconceptlab.org for testing and this points to the current version of QA for API v2 at api.qa.aws.openconceptlab.org. I’ve created a test user for us (test/Test123!).

You should be able to log in and create a dictionary. I’ve been able to create some example concepts. You’ll notice concept counts aren’t displaying properly on the dictionary detail page; this involves one of the few changes from API v1 to v2 and I’ve documented changes needed in OCLOMRS-923.

When I try to import a concept from CIEL, it currently takes ~6 minutes. This is a combination of issues:

  • Some mappings requests are taking >15 seconds to respond (#553), which the ocl dev team will tackle for us
  • Adding CIEL 1226, makes ~20 calls to one of these slow mapping requests, asking for mappings for CIEL concepts Unknown (1067), Insertion of central venous catheter (162625), and Yes (1065). I’m not sure why this is happening (is it a client-side issue? or the client misbehaving because of a changed response from the API?)
  • A final call does a PUT to https://api.qa.aws.openconceptlab.org/users/test/collections/test1/references/?cascade=sourcemappings that appears to be rejected due to a CORS error. I’m guessing this might be an issue with API v2, but could use eyes from a dev on the client side to see if they agree.

Posting this to hopefully jumpstart some interest/testing & bug fixing for the imminent switch to OCL API v2. We can discuss more on Wednesday’s [OCL Squad call](PUT https://api.qa.aws.openconceptlab.org/users/burke/collections/test1/references/?cascade=sourcemappings). Hopefully, we’ll have some OCL devs visiting to share some of their frontend work as well.

/cc @grace

3 Likes

So this final PUT request looks to be pathologic and it seems to be related to the issue highlighted in the bullet point before this. In particular, despite the error about “CORS header ‘Access-Control-Allow-Origin’ missing”, the actual pre-flight request (i.e., the OPTIONS request) seems to succeed and the PUT request fails with a 504 (Gateway Timeout) error after ~1 minute.

The reason I think this is linked to the issue above is that the request payload for that request is this:

{"data":{"expressions":["/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1067/","/orgs/CIEL/sources/CIEL/concepts/162625/","/orgs/CIEL/sources/CIEL/concepts/1065/","/orgs/CIEL/sources/CIEL/concepts/1226/"]}}

I’m not sure why the client would be making the ~20 calls to the mappings request, especially for the same value. That looks like there is an issue on the client side. However, the API v2 is either operating with a very different version of CIEL or just outright returning an incorrect response for the request to:

/orgs/CIEL/sources/CIEL/mappings/?fromConcept=1226&limit=0

In prod, this query returns 5 results without any of them being 1067, 1065, or 162625. In the QA v2, the same same query returns 24 results, including 1067, 1065, 162625, every single one of them wrong (i.e., none of the results match from_concept_code == 1226 whereas I’d expect every result in that result set to have from_concept_code == 1226).

It seems like the “solution” is what Sny suggested here, i.e., adding &q= to the end of the query string results in the right answers and in a reasonable time frame. Locally, I can change the query here to include the &q= and the pathological behaviour stops (we I get one GET request to get the mappings and one working PUT request to add them in).

That said, this kind of issue makes me nervous that we’re ready to cut over to API v2 without extensive testing.

Agreed. The time for intensive testing is now and we can be very helpful to the OCL team since we have very practical use cases (and a client) from which to help test.

Holy cow! Looks like there are some significant issues here. Suggest we discuss tomorrow on our OCL for OpenMRS call :slight_smile:

Thanks for your investigation, @ibacher. It looks like most of this will be rectified by address #553 – i.e., fixing the unexpected behavior when the q parameter is omitted. I expect Sunny will quickly have it fixed.

While we certainly need to do testing against v2, I suspect any issues (just like this one) will look nastier than the actual problem. For example, fixing API v2 to behave properly when the query (q) parameter is omitted is probably a relatively straightforward fix for the OCL dev team and will probably fix all of the issues I enumerated above (delays, seemingly random concepts getting imported, etc.).

1 Like

Yeah, sorry for the long narrative. It looks like there’s a (small) bug in the backend and a (hopefully small) bug in the front-end and the two of them interacted very poorly.

1 Like

Good news! :grinning: Sunny has already fixed the bug (#553) and this solved all the problems noted above (importing a CIEL concept is instantaneous again and works as expected). Thanks Sunny!

FYI – While testing that, I noticed mappings are not showing on the concept detail view and reported this to the OCL dev team as #555.

As we test the OCL Client for OpenMRS against OCL API v2, we should be aware of the few known breaking changes (from #463)…

Breaking Changes in OCL API v2

  • Search for custom attributes changed to support Elastic Search (full details)
    • Use . instead of __ for nested queries (e.g., extras.my_custom_attribute )
    • Use =! instead of != for negation queries
  • references are no longer returned when retrieving a collection, unless explicitly requested (e.g., /orgs/MyOrg/collections/MyCollection/?includeReferences=true). They can still be retrieved at the references endpoint (e.g., /orgs/MyOrg/collections/MyCollection/references/ )
  • url and version_url changed for some some responses (e.g., mappings). See #65.
  • limit=0 is disabled for most/all requests

(for example, I see several calls from the client containing “limit=0” meaning “give me everything”, which has been deprecated… this particular change won’t affect us in most cases, but could cause problems when the client asks for something that does produce a long list, assumes its getting a comprehensive response, and is actually getting a partial response)

1 Like

IIRC we’re really just fetching those references so we can distinguish between concepts actually in the collection vs concepts referenced in another source. If it were possible to adjust API v2 to just return those as separate fields we could eliminate the need to load all the references in the case where we don’t need them.

I actually only see limit=0 being used twice: once for the list of user organisations, which shouldn’t be too hard to change (we already paginate this), and once for the list of mappings for a concept (which probably requires a bit more work).

Update – Sunny has already fixed the missing mappings.

@grace, OCL for OpenMRS client running on QA2 against API v2 is looking pretty good. We’re reaching a point where errors I’m seeing are changes needed in our client (e.g., new location of signup page).

I would encourage any folks in the OCL for OpenMRS Squad to sign up for an account via email from qa.aws.openconceptlab.org and do some testing of our client on QA2 at openmrs.qa2.openconceptlab.org.

FYI #1 – there’s a relatively recent (unofficial) version of CIEL installed on QA2, which helps for testing.

FYI #2 – I’ve added the “oclapi2” label to OCLOMRS client issues against API v2 I uncover.

image

1 Like

FYI for those who are following the OCL API2 changes: @suruchi has been leading our QA process and has done 99% of the QA needed for the client by going through our QA checklist for the OCL client.

The issues found are being tracked here: https://issues.openmrs.org/browse/OCLOMRS-940?jql=project%20%3D%2014701%20AND%20labels%20%3D%20oclapi2

2 Likes