Support Article
Passivation daemon is locking up
SA-2365
Summary
Server becomes unresponsive with hung threads every hour. It has been reported that the hang seems to occur when the SAP Connect-SOAP rule is invoked and times out waiting for a response from the external webservice / endpoint.
Error Messages
"PegaRULES PassivationDaemon" Id=21 in TIMED_WAITING
BlockedCount : 0, BlockedTime : -1, WaitedCount : 12521, WaitedTime : -1
at java.lang.Thread.sleep(Native Method)
at com.pega.pegarules.session.internal.mgmt.base.RequestorChildren.waitOnAllBatchActivities(RequestorChildren.java:650)
at com.pega.pegarules.session.internal.mgmt.PRRequestorImpl.waitOnAllBatchActivities(PRRequestorImpl.java:1182)
at com.pegarules.generated.activity.ra_action_load_salesorderwbsdetailsread_sync_288faa978f7c3d33dc865a90e3833716.step4_circum0(ra_action_load_salesorderwbsdetailsread_sync_288faa978f7c3d33dc865a90e3833716.java:786)
at com.pegarules.generated.activity.ra_action_load_salesorderwbsdetailsread_sync_288faa978f7c3d33dc865a90e3833716.perform(ra_action_load_salesorderwbsdetailsread_sync_288faa978f7c3d33dc865a90e3833716.java:137)
at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3265)
at com.pega.pegarules.exec.internal.declare.infengine.pages.DeclarativePageDirectoryImpl.runRule(DeclarativePageDirectoryImpl.java:1575)
at com.pega.pegarules.exec.internal.declare.infengine.pages.DeclarativePageDirectoryImpl.runLoadActivity(DeclarativePageDirectoryImpl.java:1114)
at com.pega.pegarules.exec.internal.declare.infengine.pages.DeclarativePageDirectoryImpl.loadAndReturnDeclarePage(DeclarativePageDirectoryImpl.java:776)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPageDir.loadAndReturnDeclarePage(ThreadPageDir.java:1520)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPageDir.getRODeclarePage(ThreadPageDir.java:1097)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPageDir.getOrLoadDeclarePage(ThreadPageDir.java:871)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPageDir.getDirectPage(ThreadPageDir.java:785)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPageDir.getDirectPage(ThreadPageDir.java:718)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPageDir.getPageForAutoPopulate(ThreadPageDir.java:507)
at com.pega.pegarules.session.internal.mgmt.PRThreadImpl.getPageForAutoPopulate(PRThreadImpl.java:578)
at com.pega.pegarules.data.internal.clipboard.PropertyDataPageWrapper.getOldClipboardObject(PropertyDataPageWrapper.java:1753)
at com.pega.pegarules.data.internal.clipboard.PropertyDataPageWrapper.populateDataFromPage(PropertyDataPageWrapper.java:1092)
at com.pega.pegarules.data.internal.clipboard.PropertyDataPageWrapper.getAssociatedObject(PropertyDataPageWrapper.java:933)
at com.pega.pegarules.data.internal.clipboard.PropertyDataPageWrapper.getAssociatedPage(PropertyDataPageWrapper.java:916)
at com.pega.pegarules.data.internal.clipboard.PropertyDataPageWrapper.prGetPropertyNames(PropertyDataPageWrapper.java:373)
at com.pega.pegarules.data.internal.clipboard.ClipboardPageImpl.getKeySetFromWrapper(ClipboardPageImpl.java:4316)
at com.pega.pegarules.data.internal.clipboard.ClipboardPageImpl.values(ClipboardPageImpl.java:4333)
at com.pega.pegarules.data.internal.clipboard.ClipboardPropertyBase.iterator(ClipboardPropertyBase.java:3289)
at com.pega.pegarules.data.internal.clipboard.StorageStreamCommonImpl.serializeAutoPopulateReferenceData(StorageStreamCommonImpl.java:1088)
at com.pega.pegarules.data.internal.clipboard.StorageStreamCommonImpl.getPageValueForStream(StorageStreamCommonImpl.java:991)
at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$PageSection.addProperty(DirectStreamEncoder.java:1083)
at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$PageSection.addPage(DirectStreamEncoder.java:778)
at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$PageListSection.addPageList(DirectStreamEncoder.java:1610)
at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$PageSection.addProperty(DirectStreamEncoder.java:1226)
at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder$PageSection.addPage(DirectStreamEncoder.java:778)
at com.pega.pegarules.data.internal.clipboard.DirectStreamEncoder.encode(DirectStreamEncoder.java:353)
at com.pega.pegarules.data.internal.clipboard.StorageStreamCommonImpl.encode(StorageStreamCommonImpl.java:793)
at com.pega.pegarules.data.internal.clipboard.ClipboardPageImpl.getStreamForm(ClipboardPageImpl.java:3282)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPassivation.getPageBytes(ThreadPassivation.java:653)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPassivation.passivatePageIntoThreadMap(ThreadPassivation.java:579)
at com.pega.pegarules.session.internal.mgmt.base.ThreadPassivation.passivateThread(ThreadPassivation.java:462)
at com.pega.pegarules.session.internal.mgmt.PRThreadImpl.passivateThread(PRThreadImpl.java:1074)
at com.pega.pegarules.session.internal.mgmt.base.RequestorPassivation.passivateRequestor(RequestorPassivation.java:814)
at com.pega.pegarules.session.internal.mgmt.PRRequestorImpl.passivateRequestor(PRRequestorImpl.java:1521)
at com.pega.pegarules.session.internal.mgmt.base.NodeRequestorMgt.getContextForPassivation(NodeRequestorMgt.java:1605)
at com.pega.pegarules.session.internal.mgmt.base.NodeRequestorMgt.passivateRequestor(NodeRequestorMgt.java:2463)
at com.pega.pegarules.session.internal.mgmt.base.RequestorPassivation.cleanupImpl(RequestorPassivation.java:614)
- locked com.pega.pegarules.session.internal.mgmt.PRRequestorBase@59e086f8
at com.pega.pegarules.session.internal.mgmt.PRRequestorBase.cleanup(PRRequestorBase.java:894)
- locked com.pega.pegarules.session.internal.mgmt.PRRequestorBase@59e086f8
at com.pega.pegarules.session.internal.mgmt.PRRequestorImpl.cleanup(PRRequestorImpl.java:363)
- locked com.pega.pegarules.session.internal.mgmt.PRRequestorImpl@33ac44cc
at com.pega.pegarules.session.internal.mgmt.base.NodeRequestorMgt.cleanupConditionally(NodeRequestorMgt.java:1120)
- locked com.pega.pegarules.session.internal.mgmt.PRNodeImpl@69865b10
Root Cause
The thread dumps indicate that salesorderwbsdetailsread_sync is creating child requestors which are taking too long to execute. This led to the DEADLOCK between the “PegaRULES PassivationDaemon (Id=21)” and all other “PegaRULES-Batch-n” threads BLOCKED on lock=com.pega.pegarules.session.internal.mgmt.PRNodeImpl@69865b10.
Resolution
The following actions prevent this issue from reoccurring:
- Short term – The thread dump indicate that salesorderwbsdetailsread_sync is creating child requestors which are taking too long to execute. As a local-change, reduce the “timeout” value (waiting time) from the salesorderwbsdetailsread_sync activity.
- Long term – The vulnerability of threads to BLOCK on PassivationDaemon has been fixed in Pega 7.1.6.
Published January 31, 2016 - Updated October 8, 2020
Have a question? Get answers now.
Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.