Lots of LDAP errors with email verification process in past few days

Projects ID Dashboard
1

This week, we’ve received several reports of people having problems with the email verification process. When they try to click the link in their verification email, they get the following error from ID Dashboard:

 LDAP Error
Already exists

Error at Object.<anonymous> (/opt/id/node_modules/LDAP/LDAP.js:16:23) at Module._compile (module.js:449:26) at Object.Module._extensions..js (module.js:467:10) at Module.load (module.js:356:32) at Function.Module._load (module.js:312:12) at Module.require (module.js:362:17) at require (module.js:378:17) at Object.<anonymous> (/opt/id/app/ldap.js:16:18) at Module._compile (module.js:449:26) at Object.Module._extensions..js (module.js:467:10)

They can’t log in or reset their password after this, and the verification doesn’t work of course due to the above error.

A workaround that we’ve been using is:

  1. Delete the “Locked” account in Formage
  2. Recreate it via the normal ID Dashboard page, using the exact same names, email address, and a temporary password, and get an error message that the email address and username exists.
  3. Check that the account now re-appears in Formage, but no longer “Locked”
  4. Log in to Dashboard with the temporary password to verify
  5. Instruct the user to use the password reset feature to reset their password and continue as normal.

This is stopping several people from successfully getting accounts, so we should try to figure out what’s wrong ASAP.

Any ideas? @r0bby @plypy @elliott /cc @ryan

2

I can look into this but @plypy and @elliott know more and probably can solve it quicker than me – I’d have to look through the code…part of GSoC this year is fixing backend stuff – so this falls under GSoC – it will definitely be fixed but a short-term fix should be implemented ASAP…

3

What is interesting is that this seems to have happened much more frequently in the last week or so…

4

From the error message ‘Already exists’, it seems that OpenLDAP has already stored that entry before we ldap_add it. So I think, we’d better check our OpenLDAP.

FYI, here is the flowchart of the normal process of creating a OpenMRS ID, and the error happened in the very last stage associated with OpenLDAP. flowchart

1 Like
5

Hi, @michael. It seems the problems are getting more and more annoying. If you don’t mind, I want to get the SSH access to our production server, so I can dive deeper into this problem and maybe solve it. Or the only thing I can do now is to guess randomly based on the very limited error log…

And also, I am very sorry to those who are disturbed, I’ll try to figure out why… Some part of the Dashboard seems to get rusty :weary:

1 Like
6

During my development… I noticed that we’ve lost the track of our production’s version… I think we’d better create a production branch or something else that could track it.

What’s more, @michael please tell me the commit hash of the current Dashboard on production. I’m trying to implement the auto-injection feature for TALK, so I need to know it.

1 Like
7

The current master should NOT be deployed in its current state – even post GSoC – I have work I need to do to bring it to production quality. Currently there’s a few things we need to resolve.

8

So this is still happening several times each week. What’s the best way to track down what’s failing in the code?

9

Scrutinizing again to the backtrace, this error should be related with ldapadd procedure. It’s either the problem of OpenLDAP or somehow the code have sent more than one ldapadd request.

And based on your workaround and my guess to the current state of Dashboard in production, which should have no delete sync logic from Mongo to LDAP and could dynamic sync user from LDAP to Mongo. See, in your workaround,

  1. User is deleted from Mongo, but remained still in LDAP
  2. Sync is triggered from LDAP to Mongo
  3. Sync in 2. doesn’t create locked flag …

I’m examining the code as well. But really, the commit hash of current production server would be much helpful!!!

1 Like
10

I believe I can work out a temporary solution for this, please provide me the hash and I’ll change the code to ignore this error when it happens.

2 Likes
11

Anything I can do to help out with this? @plypy, I can do some work on production if you need it. I’ll PM you the prod hash.

1 Like
12

Thanks @elliott; I was tied up this weekend. @plypy in case you need access to production or have specific questions, probably best way to reliably get answers is through email to helpdesk@ or new case at https://help.openmrs.org/

Keep up the good work!

13

I can’t find the commit @elliott has provided…

So for the sake of efficiency I’ve already sent my request for SSH access.

14

I think we should just patch production and then fix the resulting line in our development…

15

From the error log I downloaded from production server and some tests I did, I can say,

  1. This error happens frequently and earlier than we thought… from April or even earlier, 5 or more times a week.

  2. This problem has created lots of orphan data that only exist in LDAP but not in mongo.

  3. Personally I’d like to say, it should be other LDAP stuff that’s to blame not Dashboard. As they’ll report “Already Exists” though Dashboard only performed only one insert operation… This sucks

Besides… the log is very poor to provide useful information, this needs to changing.

16

@elliott @r0bby @michael I’ve already fixed this by ignoring LDAP Error(68) Already exists. And the work is published in production branch of my repo.

We shall create a production branch on openmrs repo as well deploy this ASAP.

2 Likes
17

Interesting. Was it really just an ignorable error?

When I looked that the “bad” accounts in Formage, it showed as status Locked. Is that just because they did not complete email confirmation process?

18

Yep, see this flowchart I posted

and this anlyzing

19

Something related with LDAP operation is broken, that would report Already Exists error. But on Dashboard’s perspective, we can safely ignore this.

1 Like
20

I made a big mistake of rewriting git history – so the code in production was a bit behind…how diverged is your production branch? That’s my only concern.

Protip: rewriting git commit history which others have worked on is bad…if it’s only your work and others don’t have it…then it’s fine.