SA-24104

Summary

A systems administrator is reviewing their JVM Garbage Collection log data after performing an intense user load test. 

The GC log data reveals a gradually increasing usage of the JVM heap over a period of just 24 hours. This consumption gradually leads to a consumption of the machine: because GC operations increasingly consume the CPU, the memory consumption becomes so great that the machine does not have enough memory to satisfy new memory requests, and GC operations consume all available CPU.  The Process Commander environment becomes non-responsive. Note that there is no java.lang.outofmemory error in the logs.

Error Messages

No error message are displayed on screen.  No applicable error messages are found in the Pega logfile.

However, approximately 60% of the entire Pega ALERT log is composed of the following EXCP0001 ALERT:

[MSG][Exception][STACK][com.pega.pegarules.pub.services.RemoteApplicationException: SOAP service failed, error:The system cannot infer the transport information from the  URL.
 at com.pegarules.generated.activity.ra_action_invokeaxis2_3df4180fcce95ef34867d5fe4255d3cf.step13_circum0(ra_action_invokeaxis2_3df4180fcce95ef34867d5fe4255d3cf.java:2183)
 at com.pegarules.generated.activity.ra_action_invokeaxis2_3df4180fcce95ef34867d5fe4255d3cf.perform(ra_action_invokeaxis2_3df4180fcce95ef34867d5fe4255d3cf.java:273)
 at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3375)
 at com.pegarules.generated.activity.ra_action_invoke_5e9e0cecbbfdfa9a2b1306e703ec4af8.step8_circum0(ra_action_invoke_5e9e0cecbbfdfa9a2b1306e703ec4af8.java:1032)
 at com.pegarules.generated.activity.ra_action_invoke_5e9e0cecbbfdfa9a2b1306e703ec4af8.perform(ra_action_invoke_5e9e0cecbbfdfa9a2b1306e703ec4af8.java:189)
 at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3375)
 at com.pegarules.generated.activity.ra_action_invokesoapconnector_632911c40a94541994d4d7930f7c6360.step6_circum0(ra_action_invokesoapconnector_632911c40a94541994d4d7930f7c6360.java:701)
 at com.pegarules.generated.activity.ra_action_invokesoapconnector_632911c40a94541994d4d7930f7c6360.perform(ra_action_invokesoapconnector_632911c40a94541994d4d7930f7c6360.java:163)
 at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:3375)<CR> at com.pega.pegarules.exec.internal.declare.infengine.pages.DeclarativePageDirectoryImpl.runRule(DeclarativePageDirectoryImpl.java:918)
 at com.pega.pegarules.exec.internal.declare.infengine.pages.DeclarativePageDirectoryImpl.runLoadActivity(DeclarativePageDirectoryImpl.java:440)
 at com.pega.pegarules.session.internal.mgmt.base.handler.EditableDataPageHandler.loadDataPage(EditableDataPageHandler.java:127)
 at com.pega.pegarules.session.internal.mgmt.base.handler.EditableDataPageHandler.findDataPage(EditableDataPageHandler.java:258)
 at com.pega.pegarules.session.internal.mgmt.base.AbstractPageDirectory.findDataPageInCurrentDir(AbstractPageDirectory.java:893)
...

Steps to Reproduce

To reproduce this error, user crafted a load test that explicitly performed CONNECT-SOAP calls using invalid end point URL values, in other words, the CONNECT-SOAP operations were attempting to contact a remote host that did not exist.

Root Cause

A composite root cause was identified during the course of this investigation.  Heap dump analyses revealed:

- a defect in the Dynamic System Settings Security Switch URLACCESSMODE.  When this switch was set to ALLOW, the underlying code both failed to actuate as advertised, and orphaned memory objects.  This translated into a slow memory leak in the GC log data.

- a defect in the CONNECT-SOAP standard code.  This defect orphaned CATCH BLOCK objects whenever the CONNECT-SOAP operation returned an error condition.

Resolution

Apply the following hot fixes to resolve this issue:

- HFix-27843
This software fix resolved the issue of a memory leak that occurred when a CONNECT-SOAP operation threw an error condition

- HFix-27820
This software fix resolved the issue of a memory leak that occurred around the use of the Dynamic System Setting SECURITY/URLACCESSMODE when set to ALLOW