Error: Unable to deserialize object

I’m working with Cohort Queries in the Reporting Module. I notice that some of them are disappearing. For example, last night I had 26, and this morning there are 19 in the list. The items that are not deserialized are various types; SQL, CodedObs, and Inverse.

When I look in the log file, get errors like…

WARN - SerializedDefinitionServiceImpl.getAllDefinitions(145) |2015-10-29 12:51:38,177| Unable to deserialize Definition: Serialized org.openmrs.module.reporting.cohort.definition.CodedObsCohortDefinition named <Referred In>
WARN - SerializedDefinitionServiceImpl.getDefinitionByUuid(129) |2015-10-29 12:51:38,195| Unable to deserialize Definition : Serialized org.openmrs.module.reporting.cohort.definition.CodedObsCohortDefinition named <Referred In>
WARN - SerializedDefinitionServiceImpl.getAllDefinitions(145) |2015-10-29 12:51:38,208| Unable to deserialize Definition: Serialized org.openmrs.module.reporting.cohort.definition.CodedObsCohortDefinition named <Referred Out>
WARN - SerializedDefinitionServiceImpl.getDefinitionByUuid(129) |2015-10-29 12:51:38,229| Unable to deserialize Definition : Serialized org.openmrs.module.reporting.cohort.definition.CodedObsCohortDefinition named <Referred Out>
WARN - SerializedDefinitionServiceImpl.getAllDefinitions(145) |2015-10-29 12:51:38,281| Unable to deserialize Definition: Serialized org.openmrs.module.reporting.cohort.definition.InverseCohortDefinition named <Address Non-Metropolitan>
WARN - SerializedDefinitionServiceImpl.getAllDefinitions(145) |2015-10-29 12:51:38,291| Unable to deserialize Definition: Serialized org.openmrs.module.reporting.cohort.definition.SqlCohortDefinition named <Final Diagnosis - Asthma>
WARN - SerializedDefinitionServiceImpl.getAllDefinitions(145) |2015-10-29 12:51:38,301| Unable to deserialize Definition: Serialized org.openmrs.module.reporting.cohort.definition.SqlCohortDefinition named <Final Diagnosis - Prostatectomy>
WARN - SerializedDefinitionServiceImpl.getAllDefinitions(145) |2015-10-29 12:51:38,335| Unable to deserialize Definition: Serialized org.openmrs.module.reporting.cohort.definition.SqlCohortDefinition named <Address Metropolitan>
WARN - SerializedDefinitionServiceImpl.getAllDefinitions(145) |2015-10-29 12:51:38,345| Unable to deserialize Definition: Serialized org.openmrs.module.reporting.cohort.definition.SqlCohortDefinition named <Admission Night>
WARN - SerializedDefinitionServiceImpl.getAllDefinitions(145) |2015-10-29 12:51:38,359| Unable to deserialize Definition: Serialized org.openmrs.module.reporting.cohort.definition.SqlCohortDefinition named <Admission Day>
WARN - SerializedDefinitionServiceImpl.getAllDefinitions(145) |2015-10-29 12:51:38,371| Unable to deserialize Definition: Serialized org.openmrs.module.reporting.cohort.definition.SqlCohortDefinition named <Final Diagnosis - Circumcision >

The catalina.out log says… http://pastebin.com/bKUZqb7W

I’m running…

  • OpenMRS 2.3, but noticed the problem with 2.3 Snapshot previously.
  • Reporting 0.9.7
  • Serialization Xstream 0.2.9
  • Apache Tomcat/7.0.52 (Ubuntu)
  • JVM 1.7.0_85-b01 and tried JVM 1.6.0_36-b36 because of a similar ticket; REPORT-468. The error persists.

Anybody have any suggestions, like @mseaton or @raff? I’m scared to keep creating cohort queries because I don’t know what will disappear next. For example, some I have re-created, but they end up disappearing too.

I remember experiencing something like this. For my case, it was the reports which were disappearing on restart. On checking the database, they actually existed but the serializer was failing to load some of them. I had not done any upgrade of any module, not even the platform. I did not investigate it much, but just wanted to add a voice that we need to look into this seriously. :slight_smile:

Thanks @dkayiwa for your confirmation that we may really have a problem here. The items are still in the database as you say.

I have uploaded a database dump of the serialized_objects table in case that helps with troubleshooting. serialized_object.txt (70.8 KB)

@mogoodrich recently ran into this problem and found that it was caused by the recent Hibernate upgrade ( I believe ). @arbaughj, are you running the latest version? Mark, did you release the fix you put in for that?

Mike

Yes, the problem was that after upgrading to a newer version of javaassist, serialization-xstream was (potentially) corrupting reports when saving them.

I put the fix to serialization-xstream and released it (v2.9). @arbaughj it does look like you are using the latest version, so you should be okay going forward. However, you will have to manually clean up the existing reports. Thanks for including the dump of your serialization export table!

You’ll need to go in and manually remove all the “_$$_jvst35_41” strings from your database. ie, change:

class=“org.openmrs.User_$$_jvst35_41” class=“org.openmrs.User”

Looks like there are 15 matches the dump you send me. Note that the numbers after the 'jvst" vary, so unfortunately you can’t correct with a single search and replace.

fwiw, I would fix a couple first, test to make sure it worked, and then resave and make sure the problem isn’t reocurring.

Hope this helps!

Take care, Mark

1 Like

Thats it. Those were the exact changes i had to make to my table data. I just had not idea what was causing it and i ended up with the same mess, every time i saved. Let me also upgrade the module and will report back if i ever get the problem again.

Thanks @mogoodrich! You hit the nail on the head. I removed the _$$_jvst*** from the records and they now appear in the list.

FYI: I only upgraded yesterday after I noticed items were dissapearing, so it was most likely the bug you fixed causing the problem. I’ll let you know should this occur again.

Thanks so much for your assistance!

Glad I could help… and that there was an easier one compared to the pesky address hierarchy issue!

Dear @mogoodrich, I ran into this again while adding indicators to a Cohort Crosstab Dataset. I added a comment to the previous ticket…

We tried to trouble shoot this a little more, and found the problem still persists. It seems if you go really fast that’s when it starts to loose them; almost like it’s trying to serialize it, and gets interrupted with another request to serialize and save, so does that one instead. It’s typically the ones in the middle that disappear.

@dkayiwa, you didn’t happen to notice loosing indicators off of a Cohort Crosstab Dataset?

Any thoughts @mseaton or @mogoodrich?

@arbaughj, I don’t have a good answer unfortunately. Running off of a simple demo database with 1.11.4 and the latest snapshot versions of reporting and serialization.xstream, I haven’t managed to reproduce this. If you can find a way to reliably reproduce the error with test data that would be very helpful. Are there particular types of definitions that tend to have issues (e.g. Coded Obs Cohort, etc)?

We tried to reproduce the problem with the Standalone, but were unsuccessful. I don’t think we were able to create a complex enough report, and didn’t have the Internet delays.

We were able to add all of the indicators by going slowly, waiting for the page to fully load each time, which in some cases took nearly 5-minutes per indicator, and doing a backup of the serialized_items table after adding each indicator, so we could fail-back to before we lost the indicators we had added.

The work-around is just to slow down!

Thank you everyone for having this discussion on talk. This helped me.

Just as a reminder. Anyone still using Reference Application 2.2 (with Serialization.Xstream v2.7), might still face this issue.