eBase server hangs after a while

[EMvdB]

We have the following problem: UFSserver hangs after a while, sometimes after 30 minutes, sometime after several hours.

eBase installation:
Windows 2003 server, IIS disabled
Standard eBase installation (with tomcat), version 3.40 (02052008)
Repository database: Oracle
JRE 1.5.0.13
Only tools.jar from JDK
ojdb14.jar from februati 2008

This is what I can see in the logfiles:

STDout.log
Mon Jun 09 13:29:30: Info : Save successful for Form TER_CONTACT_ALG user maassee
Mon Jun 09 13:30:09: Error : SQL error on statement select * from forms where form_id = ?
Mon Jun 09 13:30:09: Error : I/O-uitzondering: Connection reset
Mon Jun 09 13:30:09: Error : SQL error on statement select * from ufs_components where component_id = ?
Mon Jun 09 13:30:09: Error : StandardXAConnectionHandle:prepareStatement getTransaction exception: javax.transaction.SystemException: Cannot send XA(StandardXAConnection:
commit on prepare =<false>
is closed =<false>
this autoCommit =<true>
listeners size =<1>
next timeOut =<0>
timeOut period =<60000>
timeOut secs =<0>
transaction manager=<org>
StandardXADataSource:
connection count=<2>
number of dead connection=<0>
dead lock max wait=<300000>
dead lock retry wait=<10000>
driver=<oracle>
driver name=<oracle>
number of *free* connections=<1>
max con=<0>
min con=<50>
prepared stmt cache size=<0>
transaction manager=<org>
xid connection size=<1>
StandardConnectionPoolDataSource:
master prepared stmt cache size=<0>
prepared stmt cache size =<0>
StandardDataSource:
driver=<oracle>
url=<jdbc>
user=<ufs>
CoreDataSource :
debug =<false>
description =<null>
login time out =<60>
user =<ufs>
verbose =<false>
StandardXADataSource:
connection count=<2>
number of dead connection=<0>
dead lock max wait=<300000>
dead lock retry wait=<10000>
driver=<oracle>
driver name=<oracle>
number of *free* connections=<1>
max con=<0>
min con=<50>
prepared stmt cache size=<0>
transaction manager=<org>
xid connection size=<1>
StandardConnectionPoolDataSource:
master prepared stmt cache size=<0>
prepared stmt cache size =<0>
StandardDataSource:
driver=<oracle>
url=<jdbc>
user=<ufs>
CoreDataSource :
debug =<false>
description =<null>
login time out =<60>
user =<ufs>
verbose =<false>
StandardXAStatefulConnection:
commit on prepare =<false>
timed out =<false>
id =<3>
state =<0>
time out =<0>
xid =<bb14>
StandardXAConnectionHandle:
global transaction =<true>
is really used =<false>
this autoCommit =<true>
in use size =<0>
master prepared stmt cache size =<0>
transaction =<bb14>
connection =<oracle>
oracle.jdbc.driver.T4CConnection@159780d) start:javax.transaction.xa.XAException: java.sql.SQLException: Gesloten verbinding (error code = 0) --java.sql.SQLException: Gesloten verbinding
Mon Jun 09 13:30:09: Error : Transaction error - illegal state detected when issuing rollback with transaction status active


XAPool.log
189766 [xapool] ERROR - Logger.error(55) : StandardPoolDataSource:checkThisObject can't closed the connection: java.sql.SQLException: Gesloten verbinding
196813 [xapool] ERROR - Logger.error(55) : StandardPoolDataSource:checkThisObject can't closed the connection: java.sql.SQLException: Gesloten verbinding
196813 [xapool] ERROR - Logger.error(55) : StandardPoolDataSource:checkThisObject can't closed the connection: java.sql.SQLException: Gesloten verbinding
206266 [xapool] ERROR - Logger.error(55) : StandardPoolDataSource:checkThisObject can't closed the connection: java.sql.SQLException: Gesloten verbinding
209047 [xapool] ERROR - Logger.error(55) : StandardPoolDataSource:checkThisObject can't closed the connection: java.sql.SQLException: Gesloten verbinding


Does someone have a clue what is happening, or what I can do to prevent this.

[AJDulk]

From the log you sent, I would say that you are losing your connection to the database. I suspect that you are running your database on another [virtual] machine. If so, are there any firewalls between the machines? If there are, the firewall may be closing the connection due to what it sees as inactivity. Try running Ebase without the firewall enabled between the 2 machines and see if that helps.

Without knowing exactly what and how your system is running it is difficult to give any further insight into this.

BTW: Have you added this to a log in the customer portal?

[EMvdB]

Anthony,
You are right about loosing the connection, but the question is why? And why isn't the connection not automatically restored?

What would you like to know about our system?

I will add it is asap to the customer portal.

Greetings, Erwin
(Apeldoorn)

[AJDulk]

Erwin,

I don't get anything on a web search where the problem is encountered. I did find some source code though.

The code checks for a connection and if there is one tries to close it. For some reason while your connection is registering as open it is already closed and therefor throws an error when trying to close an already closed connection.

One option is (as mentioned before) a firewall rule. Depending when the firewall last saw something happening on the port it may close the port. A lot of firewalls are intelligent enough now-a-days that they will allow a connection to a normally blocked port if certain conditions are met - such as an already existing connection on another port. With a firewall, you would expect a fairly consistent pattern - example: no one uses the server for 30 minutes then error.

Another option is broken hardware, maybe one of the network cards is glitching randomly. When Ebase tries to connect to the database just as the glitch occurs then you get the error.

I suspect that xapool sees this as a fatal problem and so does not try to restore the connection.

I will let Ebase troubleshoot your system. If you wanted me to do it then I would need detailed information about the system. The following is a list off the top of my head:

• Hardware configuration.
  • How many servers.
    Network cards.
    Firewalls.
  Software configuration.
  • What operating systems are being used (Windows 2003 for Ebase, ? for the Oracle Database).
    Fix packs installed on the servers.
    Which versions of Ebase and Oracle (Ebase: 3.4.0 build 02052008 and ?).
    Fix packs installed for the Oracle database.
    Firewall settings (Hardware and Software) on the various servers.
  Logs.
  • Ebase (what is happening for example before and after the posted error).
    Firewall (what connections are being made and when).
    Oracle (what is getting through to the database and what is being sent back).
    OS (not likely to hold much more information than already gleaned from the other logs, though you never know).

[Sarah]

This query was also logged with Ebase via the Customer Portal and, therefore, Ebase are looking into it.

The current situation is that EMvdB is changing the CheckLevel in the Oracle datasource in ufs.xml from '1' to '2', to see if that solves the problem.

When we have a definite solution, we'll post it here.