SA-30167

Summary

User observes that their service requestors are active even after 24 hours, so the service requests pool size is getting filled up and new requestors are timing out. This is happening in Oracle 12, after recent upgrade from prior Oracle version.

ConnectionException: Database-General Problem encountered when getting connection for database pegarules 17008 08003 Closed Connection

Error Messages

2016-10-24 05:41:38,017 [fault (self-tuning)'] [  STANDARD] [                    ] [     App:03.04] (l.access.RDBPageResultPackager) ERROR IPADDRESS|IPADDRESS username - There was a database problem when performing an RDBExecute
com.pega.pegarules.pub.database.ConnectionException: There was a database problem when performing an RDBExecute: code: 17002 SQLState: 08006 Message: IO Error: Bad packet type
From: (H1FDEC367E983CB47CFB67106828F2DB6:HOSTNAME) 
Caused by SQL Problems.
Problem #1, SQLState 08006, Error code 17002: java.sql.SQLRecoverableException: IO Error: Bad packet type
** ** **
** ** **
Caused by: oracle.net.ns.NetException: Bad packet type
       at oracle.net.ns.Packet.processHeader(Packet.java:368)
       at oracle.net.ns.Packet.receive(Packet.java:304)
       at oracle.net.ns.DataPacket.receive(DataPacket.java:104)
       at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
       at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
       at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)
       at oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1504)
       at oracle.jdbc.driver.T4CMAREngine.unmarshalNBytes(T4CMAREngine.java:1478)

2016-10-24 05:41:38,075 [fault (self-tuning)'] [  STANDARD] [                    ] [     App:03.04] (internal.mgmt.MessageEvaluator) ERROR IPADDRESS|IPADDRESS username - Database error trying to retrieve message "There was a database problem when performing an RDBExecute: code: 17002 SQLState: 08006 Message: IO Error: Bad packet type with the following parameters: 
com.pega.pegarules.pub.database.ConnectionException: Database-General      Problem encountered when getting connection for database pegarules 17008  08003  Closed Connection
From: (H1FDEC367E983CB47CFB67106828F2DB6:HOSTNAME) 
Caused by SQL Problems.
Problem #1, SQLState 08003, Error code 17008: java.sql.SQLRecoverableException: Closed Connection
** ** ** 
Caused by: java.sql.SQLRecoverableException: Closed Connection
       at oracle.jdbc.driver.PhysicalConnection.setAutoCommit(PhysicalConnection.java:3741)
       at weblogic.jdbc.wrapper.PoolConnection.setAutoCommit(PoolConnection.java:148)
       at com.pega.pegarules.data.internal.access.DatabaseConnectionImpl.setAutoCommit(DatabaseConnectionImpl.java:794)
       at com.pega.pegarules.data.internal.access.ConnectionManagerImpl.getConnectionFromDataSourceIdentity(ConnectionManagerImpl.java:1027)   

 

Steps to Reproduce

1.) Access the HTTP service which will create a service requestor.
2.) Observe new requestors getting timed out with closed conenction.

Root Cause

A defect or configuration issue in the operating environment.

The NetException is caused due to a data packet mismatch between data that is being sent from the database and the data being read by the Oracle JDBC thin driver (Application server). "SQLRecoverableException: Closed Connection" indicates that the JDBC Connection in the pool was successfully established at some point, but when the application thread tried to use it, the connection was no longer open.

The root cause is either the firewall between the application server and the database is timing-out the connections or the database server itself is closing the client connections based on a set timeout. 

Resolution

Make the following change to the operating environment:

Tune the Oracle Weblogic JDBC Connection Pool configurations as below:

https://docs.oracle.com/cd/E13222_01/wls/docs81/ConsoleHelp/domain_jdbcconnectionpool_config_connections.html
​
Test Reserved Connections: true
Test Released Connections: true
Test Frequency: 5 seconds
Test Table Name: Enter a simple query (e.g. – “select count(*) from DUAL;”)