Surely. I will go through the information on the ID dashboard mail thread. Will also have an eye on methods used to detect spam accounts in this kind of domain. @r0bby nice work to present the all information needed to get on it with. Thank you very much.
@michael made a comment on the wiki page about one approach to how we ban spammers. I can’t wait to see your proposal
I saw it in the evening itself. IP based filtering is a good practice which is practically used. Specially for a system like openmrs, where users are not going to use computers in cyber cafes or other multi user or multi task PCs to access our resource. Because if they do it would not be nice to block a good user for the sake of vulnerable (bad) user. Will read a little more and come-up with the draft within next three days with an organized plan and suggestion. So that we can discuss and sharpen our ideas to make a great dashboard.
One quick question, I found my concern of using IP as the blcoking entity for spams was raised here by @plypy. How about introducing a set of rules, instead of using IPs for this. We can use a rule based spam detective system. This will help in more precise spam detection. Also the rules can be changed as the system (and the attackers ) is evolving. WDYT?
That is a concern, however we’d only ever ban an IP in extraordinary circumstances, whereas the blacklist is a list we check against. I’d (and I can probably speak for the rest of the infrastructure team) would only ever ban the ip if I saw repeated spam accounts from a given ip.
Hi @r0bby and devs, Why do we need to have same information in both LDAP and MongoDB? What are the exact situations that a user goes to LDAP but not to MongoDB?
I reffered to  and some other resources. Based on that and my little kind of experience we can easily handle the authentication part using OpenLDAP. We can keep the user attributes and information in MongoDB. Considering the objective of the project “to see all users in MongoDB and OpenLDAP and edit their user details have the updates happen in both systems.”, this would be a great option. For take this decision it is better to know the answers for above questions.
To make our development easier. LDAP exists because that is how Atlassian products fetch user data. @elliott answered your question succinctly and way better than I ever could. See his answer here:
Hi @r0bby and devs, Talking about the UI of the dashboard, wouldn’t it be great if we can graphically show the contribution of the user for each entity (such as wiki, jira and soon.). What I basically have in my mind is something like github. We don’t need many and complex graphs. Just simple but attractive ones should be used.
Additionally we can add a word cloud, the weight of the word will indicate the frequency it is used currently. This will help the users to find out what is mostly important.
Basically what I have in my mind is presenting some statistics about current activities and the user’s activities. By that the user can get all the information about different places it self from the dash board.
@r0bby, WDYT? Will this add unnecessary complexity. If we utilize UI interfaces well I don’t think so.
That’s a nice to have. Let’s first focus on the basic functionality (being able to see users in both data stores (OpenLDAP and Mongo) as one unified listing.
I closed the other thread, it’s old and continuing it here:
The workflow I think we should do (We’re going to have to cache a copy of LDAP, we don’t wanna hammer the server – there are a lot of user accounts – consider maybe redis or something similar):
- For each user in mongo
- Check if it is in LDAP, if it is, add some kind of an indication
- If we update a user, we don’t need to worry about updating both, the backend handles keeping both in sync. However, if we delete a user, we should handle deleting from both LDAP and Mongo.
Hi @r0bby, surely these are the things that I am working on right now. Regarding the UI s, I was thinking about the ways how we can icing the cake. I will give updates with the possible solutions for synchronizing data between two databases. Then we can decide which will is the efficient manner. Thank you very much for the information. Hope to comeback with a summary of possible solutions tonight.
Don’t go overboard…I don’t want you to get too excited and then realize you’re in over your head. That said get the basics down first…if after you finish that – we can discuss improvements. I do like your enthusiasm!
LDAP is an industry standard for user directories & authentication services. Many web apps are able to authenticate to LDAP, including all webapps used in the OpenMRS community, so we need to support it.
With regards to situations where a user may not be in both LDAP and Mongo - we had a bug recently where users saved fine in LDAP but due to us not handling a exception correctly…dashboard assume they did not…so we have a bunch of orphan accounts – not sure how many. Now, users are near guaranteed to be in both, though there may be the scattered case where it might not happen.
Surely we can use Redis, to cache the LDAP and use it for easy searching against Mongo.
Do we use solr or elasticsearch at the moment?
I am heading towards one solution, which uses one of these two (preferably elasticsearch), to search users efficiently and to get the A intersection B and A-B where A is OpenLDAP and B is Mongo. This will help us in visualizing the users who are correctly handled as expected. It will get even more better with Redis. Of course this is not the only way we can do this. But performance wise I hope this will be a good approach. Also this will help in the process of easy verifying of successful deleting and updating of user accounts.
And there are existing mongo-conncectors like transporter, mongo-connector etc, which we can use to develop a kind of little functional search component for users using AndulaJS and node.js. What I need to point-out is if we need one of these two (solr or elasticsearch) the benefits will not limited to this project only. WDYT?
@r0bby the exception you have mentioned is the one that give an exception when a new user is registered. And it will give an exception on the mail verification too. Does the following exception at verifying account related to this?
Timeout Error at Object. (/opt/id/node_modules/LDAP/LDAP.js:16:23) at Module._compile (module.js:449:26) at Object.Module._extensions…js (module.js:467:10) at Module.load (module.js:356:32) at Function.Module._load (module.js:312:12) at Module.require (module.js:362:17) at require (module.js:378:17) at Object. (/opt/id/app/ldap.js:16:18) at Module._compile (module.js:449:26) at Object.Module._extensions…js (module.js:467:10)
No, our current tooling is Atlassian Crowd’s search feature plus Formage’s – neither of which use Solr/Elasticsearch…actually I can’t speak 100% for what Crowd uses. It’s an option but keep in mind that the server we run Dashboard on currently has it competing for resources. So whatever we choose, has to be: 1) easy to maintain 2) not be resource heavy. I’m all for it – but it has to add value – a lot of it. ElasticSearch for our use-case…I’m not entirely sure it’d scale well. Solr…again not sure. We have a large community.
, HOWEVER – see my points above about the current resource situation. The server we are using for Dashboard shares with JIRA/Crowd, I’m not sure it can handle another app – esp. Solr/ElasticSearch – even if it adds value.
No, it was a different one. It was actually mentioned awhile back. I’m too lazy right now to look it up.
Keep in mind that an admin search happens rarely and is usually targeted to finding a single user. That search is usually done to find a specific user to make edits on that user record.
My concern with adding something like Solr/Elasticsearch is we simply don’t have the resources. I do see the value, just our hosting situation would need to change.
Actually elastic search is not a official tool to attend the objective of seeing all users in MongoDB and OpenLDAP in oneplace.
However there are connectors  which can transform data from LDAP or Redis and MongoDB to elastic search. Which we can use to see the users as it was in one place. We can use that (elasticsearch) indexing to update the two databases upon update of an user. So it could be leveraged to attend the objective while creating more space for future analytics, searching, crawling etc. Note that once we have elastic search there are multiple things we can do easily.
Surely this may consume more resources which we have to consider. We should not use this approach if it doesn’t make sense economically. It may not be feasible if we are not going to grab the best outcome of analytics, searching, crawling etc. ASAP from elasticsearch. WDYT?
It’s a possibility but I don’t think our infrastructure can handle it. I actually have to get dashboard runnable…I’m gonna dockerize it and have a vagrant box also