oracle 12569,ListenerLogIntermittentlyReports:unknownconnectdataandeither*12537or*12569error

Listener

Log Intermittently Reports: < unknown connect data> and

either * 12537 or * 12569 errors (Doc ID 1664901.1)

In this Document

APPLIES TO:

Oracle

Database Cloud Schema Service - Version N/A and

laterOracle Database Exadata Cloud

Machine - Version N/A and later

Oracle Cloud Infrastructure -

Database Service - Version N/A and later

Oracle Database Exadata Express

Cloud Service - Version N/A and later

Oracle Database Backup Service -

Version N/A and later

Information in this document applies

to any platform.

GOAL

Intermittently, or repeatedly, the listener log contains

messages such as:

* * 12537

TNS-12537: TNS:connection closed

TNS-12560: TNS:protocol adapter error

TNS-00507: Connection closed

For example, on AIX:

28-AUG-2018 09:14:40 * * 12537

TNS-12537: TNS:connection closed

TNS-12560: TNS:protocol adapter error

TNS-00507: Connection closed

IBM/AIX RISC System/6000

Error: 55: Operation now in progress

or

14-SEP-2017 13:11:20 * * 12569

TNS-12569: TNS:packet checksum failure

We often see a pattern to these errors such as the occurrence in

even intervals.

Here's an example log except where a TCP monitor process is

'touching' the listener port every 5 seconds. Each of these

attempts causes this error to be thrown in the log.

Line

22: 27-JUL-2018 10:07:55 * *

12537

Line

43: 27-JUL-2018 10:08:00 * *

12537

Line

58: 27-JUL-2018 10:08:05 * *

12537

Line

73: 27-JUL-2018 10:08:10 * *

12537

Line

94: 27-JUL-2018 10:08:15 * *

12537

Line

139: 27-JUL-2018 10:08:20 * *

12537

Line

166: 27-JUL-2018 10:08:25 * *

12537

Line

181: 27-JUL-2018 10:08:30 * *

12537

Line

196: 27-JUL-2018 10:08:35 * *

12537

Line

223: 27-JUL-2018 10:08:40 * *

12537

Line

250: 27-JUL-2018 10:08:45 * *

12537

Line

295: 27-JUL-2018 10:08:50 * *

12537

Line

328: 27-JUL-2018 10:08:55 * *

12537

Line

343: 27-JUL-2018 10:09:00 * *

12537

Line

358: 27-JUL-2018 10:09:05 * *

12537

Each of these listener log events is accompanied by this

stack:

TNS-12560: TNS:protocol adapter

error

TNS-00507: Connection closed

Solaris Error: 150: Operation

now in progress

or

TNS-12560: TNS:protocol adapter error

TNS-00507: Connection closed

IBM/AIX RISC System/6000

Error: 55: Operation now in progress

SOLUTION

This is a 3rd party tool (script, daemon, program) probing

the listener. The tnslsnr does not understand the request because

the connection does not use TNS headers.

Longer explanation:

These errors are not caused by an Oracle product, but are

most likely caused by a 3rd party tool. This can

be easily reproduced to provide further understanding into the

error.

What is happening is something, such as a 3rd party probe, is

trying to connect to the listener, but the listener does not

understand this connection because it is NOT a sqlnet connection.

It knows this because sqlnet uses a unique header

to identify itself.

After 60 seconds, the listener will give up trying to

identify this connection and report a TNS-12525 error.

However, if the unknown connection attempt is terminated BEFORE the

60 seconds, then you get the error in the listener log:

... * * 12537

TNS-12537: TNS:connection closed

TNS-12560: TNS:protocol adapter error

TNS-00507: Connection closed

Reproducing the error is easy:

On the server, simply ssh to yourself on the listener port.

For example: ssh -p 1521 oracle@listener_hostname

If you kill this command after a few seconds with CTRL-C, check the

appropriate listener log to see the error.

Who or what is doing this, and what IP address is it originating

from?

A listener trace file will reveal the source IP of the machine

where the unknown connection came from. However,

we only identify the IP address - not the program name.

To find the IP in a trace file search for

"ns=12537". When a hit is found, search UP (above

the error) for "Accepted Entry". This is the

source IP address.

Following is an extracted and edited example:

1. Searching for "12537" -->

...

[28-AUG-2018 09:14:40:546] ntt2err: Read unexpected EOF ERROR on

14

[28-AUG-2018 09:14:40:546] ntt2err: exit

[28-AUG-2018 09:14:40:546] nsprecv: error exit

[28-AUG-2018 09:14:40:546] nserror: entry

[28-AUG-2018 09:14:40:546] nserror: nsres: id=5, op=68, ns=12537,

ns2=12560; nt[0]=507, nt[1]=0, nt[2]=0; ora[0]=0, ora[1]=0,

ora[2]=0

[28-AUG-2018 09:14:40:546] nscon: error exit

[28-AUG-2018 09:14:40:546] nsdo: nsctxrnk=0

[28-AUG-2018 09:14:40:546] nsdo: error exit

...

2. Checking BACK from this to find the origin -->

...

[28-AUG-2018 09:14:40:542] nsevfnt: cxd: 0x1058e5b0 stage 0: NT

events set:

CONNECTION REQUEST

[28-AUG-2018 09:14:40:542] nsevfnt: cxd: 0x1058e5b0 stage 0: NS

events set:

INCOMING CALL

...

[28-AUG-2018 09:14:40:543] nttcnr: connected on ipaddr

xx.y.15z.197

[28-AUG-2018 09:14:40:543] nttvlser: entry

[28-AUG-2018 09:14:40:543] snlinGetNameInfo: entry

[28-AUG-2018 09:14:40:543] snlinGetNameInfo: exit

[28-AUG-2018 09:14:40:543] nttvlser: valid node check on incoming

node xx.xx.40.32

[28-AUG-2018 09:14:40:543] nttvlser: Accepted Entry:

xx.xx.40.32

...

*NOTES:

- If the listener log shows the and TNS-12537 occurring at exactly

09:14:40, then this is a match.

This means, there is a program "pinging" (polling) the Listener

from xx.xx.40.32

- Check for more occurrences to make sure you cover the full number

of incoming IP addresses.

Unfortunately, an issue like this is best left up to be resolved by

the experts of the environment - normally the customer or an

on-site support group.

The goal is to find out what tool or daemon or script is running

that wants to probe the listener.

It may be possible to change the listener ports, and possibly find

a tool or script suddenly throwing errors.

To conclude: "something", non-oracle related, is attempting to

connect to the listener. Additional

~~~~~~~~

In some environments, it was found that the "12569" error was due

to one of the following causes:

1. An "Apache Tomcat" Application / program set to the Listener

port; or

2. A Weblogic (WLS) Server whose ONS settings were incorrectly set

to the SCAN port instead of the ONS port.

Both of these were found to be sending non-TNS packets for

connection requests.

This shows as follows in a level 16 Listener trace:

[14-SEP-2015 13:10:54:246] nsprecv: packet

hdr

[14-SEP-2015 13:10:54:256] nsprecv: 50 4F 53 54 20 2F 00 00

|POST./..|

[14-SEP-2015 13:10:54:267] nsprecv: error exit

[14-SEP-2015 13:10:54:278] nserror: entry

[14-SEP-2015 13:10:54:286] nserror: nsres: id=7, op=68, ns=12569,

ns2=0; nt[0]=0, nt[1]=0, nt[2]=0; ora[0]=0, ora[1]=0,

ora[2]=0

[14-SEP-2015 13:10:54:296] nscon: error exit

The word "POST" and trailing back slash is the non-Oracle

Client's attempt to "connect" and is why the Listener reports

it.

It needs to be reviewed by whoever is administering the

Application(s) processes as this is not caused by the Listener and

it is merely the Listener disallowing the connection attempt.

3. This can also be caused by an "F5 BIG-IP" load balancer used by

an Application Server environment.

This might be performing a TCP request (ping) to each server on the

Listener port and which the Listener is picking up as an illegal

connection attempt.

Disable any TCP pinging from devices and programs such as Load

Balancers (F5 being more common).