Category Archives: General

How to never change connection strings again

The other day a colleague mentioned an interesting side-effect of CMAN: your clients never have to change the connection string ever again.
I’ve used CMAN for various other reasons (firewall channeling, etc.) but never thought of it as a solution to ever changing connection strings when you frequently move databases around. And from experience I can tell that for some application support teams changing connection strings can be a real struggle.

For this use case give the CMAN servers a round-robin DNS alias and publish service names that don’t need changing. This way, wherever your CMANs and databases are located the connection string for the client remains the same, e.g.:

my_service =
  (DESCRIPTION =
    (FAILOVER = ON)
    (LOAD_BALANCE = OFF)
    (TRANSPORT_CONNECT_TIMEOUT = 3)
    (CONNECT_TIMEOUT = 6) (RETRY_COUNT = 1)
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = cmans)(PORT = 1521))
    )
    (CONNECT_DATA = (SERVICE_NAME = my_service))
  )

So far everybody is happy…

Now, imagine you run Data Guard across two datacenters, A and B, for multiple databases. Some primary databases run in datacenter A, some in B. Because a few applications are extremely sensitive to latency, whenever a primary/standby role change happens you move your application server VMs to the same datacenter where the primary database runs.
Let’s introduce two CMAN instances in both datacenters and all of a sudden half your user base is complaining about performance. The problem is that about 50% of your connections are now going to the other datacenter just for CMAN to connect back, thus creating a loop:

If your sole purpose for using CMAN is to avoid changing client connect strings there are probably other (read better) solutions, for instance OID TNS lookups.

Or, hear me out, remote listeners πŸ˜‰

Replace CMAN with remote listeners to be the single point of contact for connection requests. The remote listener’s redirect messages will point to the local listener which will then spawn the server process. This way the application always connects to the database without being looped through the other datacenter. (for more details about listener redirects check this out)

How-to
For this example I’m assuming following hostnames, IPs, and DNS entries
DB server 1 in DC A: db-1 / 192.168.55.10
DB server 2 in DC B: db-2 / 192.168.55.20
Remote listeners in DC A: rl-1 / 192.168.55.30, rl-2 / 192.168.55.31
Remote listeners in DC B: rl-3 / 192.168.55.32, rl-4 / 192.168.55.33
DNS Alias: oracle-portal with IPs 192.168.55.30, 192.168.55.31, 192.168.55.32, 192.168.55.33 (round-robin for connection load balancing)

Create VMs/containers of the technology of your choosing and install the Oracle client.

INVENTORY_LOCATION=/u01/app/oraInventory
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=${ORACLE_BASE}/product/client19

./runInstaller -silent -waitForCompletion -noconfig \
oracle.install.responseFileVersion=/oracle/install/rspfmt_clientinstall_response_schema_v19.0.0 \
UNIX_GROUP_NAME=oinstall \
INVENTORY_LOCATION=${INVENTORY_LOCATION} \
ORACLE_HOME=${ORACLE_HOME} \
ORACLE_BASE=${ORACLE_BASE} \
oracle.install.client.installType=Custom \
oracle.install.client.customComponents=oracle.network.listener:19.0.0.0.0

Put this in ${ORACLE_HOME}/network/admin/listener.ora.

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = rl-1)(PORT = 1521))
    )
  )
ADMIN_RESTRICTIONS_LISTENER = ON
VALID_NODE_CHECKING_REGISTRATION_LISTENER = ON
REGISTRATION_INVITED_NODES_LISTENER = (rl-1, db-1, db-2)

Start the listener.

lsnrctl start LISTENER

Repeat that for the other 3 remote listener VMs/containers (don’t forget to chenge the hostname in listener.ora).
Invited nodes (Valid Node Checking Registration) is a good security practice for listeners to restrict the hosts that can register services (remember CVE-2012-1675 and COST? What a drama this was πŸ™‚ ).

Let the databases register their services.

alter system set remote_listener = "rl-1:1521","rl-2:1521","rl-3:1521","rl-4:1521" scope=both sid='*';
alter system register;

That’s it. From now on, your clients no longer need to know on which server your database is running.

my_service =
  (DESCRIPTION =
    (FAILOVER = ON)
    (LOAD_BALANCE = OFF)
    (TRANSPORT_CONNECT_TIMEOUT = 3)
    (CONNECT_TIMEOUT = 6) (RETRY_COUNT = 1)
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = oracle-portal)(PORT = 1521))
    )
    (CONNECT_DATA = (SERVICE_NAME = my_service))
  )

All roads lead to Rome – Importing custom TLS certificate into Oracle wallet

It appears that Oracle’s preferred way to get a certificate into a wallet is by generating a key pair and then create certificate signing request using “orapki” (that’s what you mostly see in the docs and on MOS). Once the request is singed by the CA you import the certificate into the wallet.
But, there are situations where you don’t get to create the signing request from the Oracle wallet. For instance, at one of my clients, they create everything at the CA and just send a PKCS#12 file to the DBAs. Since PKCS is a standard and the Oracle wallet itself is a PKCS#12 file one could (naively) assume that these things are compatible. Unfortunately, this is not always the case. When trying to generate an SSO file with “orapki” from an openssl created PKCS#12 file I get the following error:

Exception in thread "main" java.lang.NullPointerException
        at oracle.security.pki.OracleKeyStoreSpi.a(Unknown Source)
        at oracle.security.pki.OracleSSOKeyStoreSpi.a(Unknown Source)
        at oracle.security.pki.OracleFileWalletImpl.b(Unknown Source)
        at oracle.security.pki.OracleWallet.saveLSSO(Unknown Source)
        at oracle.security.pki.textui.OracleWalletTextUI.create(Unknown Source)
        at oracle.security.pki.textui.OracleWalletTextUI.command(Unknown Source)
        at oracle.security.pki.textui.OraclePKITextUI.main(Unknown Source)

This is just one example, I’ve encountered other problems when trying to work with PKCS#12 files that were created by tools other than “orapki” (or mkstore). Therefore you should always create the PKCS#12 file using “orapki” and then work from there.
I’m going to show a few practical ways how to import a “pre-created” custom TLS certificate into an Oracle wallet.

Assuming we get a PKCS#12 file containing everything we need:

  • Root certificate
  • Intermediate certificated
  • Private key
  • Signed certificate
ORACLE_WALLET_LOC=/home/oracle/wallet
ORACLE_WALLET_PWD=*****
SOURCE_PKCS12_FILE/home/oracle/custom-ca.p12
SOURCE_PKCS12_PWD=*****

orapki wallet create -wallet "${ORACLE_WALLET_LOC}" -compat_v12 -pwd "${ORACLE_WALLET_PWD}"
orapki wallet import_pkcs12 \
    -wallet "${ORACLE_WALLET_LOC}" \
   -pkcs12file "${SOURCE_PKCS12_FILE}" \
   -pkcs12pwd "${SOURCE_PKCS12_PWD}" \
   -pwd "${ORACLE_WALLET_PWD}"
orapki wallet create -wallet "${ORACLE_WALLET_LOC}" -compat_v12 -auto_login_local -pwd "${ORACLE_WALLET_PWD}"
orapki wallet display -wallet "${ORACLE_WALLET_LOC}"

Maybe we get a Java key store (JKS) containing everything:

ORACLE_WALLET_LOC=/home/oracle/wallet
ORACLE_WALLET_PWD=*****
SOURCE_JKS_FILE=/home/oracle/custom-ca.jks
SOURCE_JKS_PWD=*****

orapki wallet create -wallet "${ORACLE_WALLET_LOC}" -compat_v12 -pwd "${ORACLE_WALLET_PWD}"
orapki wallet jks_to_pkcs12 \
    -wallet "${ORACLE_WALLET_LOC}" \
    -keystore "${SOURCE_JKS_FILE}" \
    -jkspwd "${SOURCE_JKS_PWD}" \
    -pwd "${ORACLE_WALLET_PWD}"
orapki wallet create -wallet "${ORACLE_WALLET_LOC}" -compat_v12 -auto_login_local -pwd "${ORACLE_WALLET_PWD}"
orapki wallet display -wallet "${ORACLE_WALLET_LOC}"

Or, we might get everything as individual base64 encoded files:

  • root.cer
  • inter.cer
  • private-pwd.key (password protected)
  • custom.cer
CERT_FILES_LOC=/home/oracle
ORACLE_WALLET_LOC=/home/oracle/wallet
ORACLE_WALLET_PWD=*****
PRIVATE_KEY_PWD=*****

orapki wallet create -wallet "${ORACLE_WALLET_LOC}" -compat_v12 -pwd "${ORACLE_WALLET_PWD}"
orapki wallet add \
    -wallet "${ORACLE_WALLET_LOC}" \
    -trusted_cert \
    -cert "${CERT_FILES_LOC}/root.cer" \
    -pwd "${ORACLE_WALLET_PWD}"
orapki wallet add \
    -wallet "${ORACLE_WALLET_LOC}" \
    -trusted_cert \
    -cert "${CERT_FILES_LOC}/inter.cer" \
    -pwd "${ORACLE_WALLET_PWD}"
orapki wallet import_private_key \
    -wallet "${ORACLE_WALLET_LOC}" \
    -pvtkeyfile "${CERT_FILES_LOC}/private-pwd.key" \
    -pvtkeypwd "${PRIVATE_KEY_PWD}" \
    -cert "${CERT_FILES_LOC}/custom.cer" \
    -pwd "${ORACLE_WALLET_PWD}"
orapki wallet create -wallet "${ORACLE_WALLET_LOC}" -compat_v12 -auto_login_local -pwd "${ORACLE_WALLET_PWD}"
orapki wallet display -wallet "${ORACLE_WALLET_LOC}"

“Helper functions”
Extracting individual components from a PKCS#12 file:

CERT_FILES_LOC=/home/oracle
SOURCE_PKCS12_FILE/home/oracle/custom-ca.p12
SOURCE_PKCS12_PWD=*****
PRIVATE_KEY_PWD=*****

# extract private key from PKCS#12 file
openssl pkcs12 -nocerts -in "${SOURCE_PKCS12_FILE}" -out "${CERT_FILES_LOC}/private-pwd.key" -password pass:"${SOURCE_PKCS12_PWD}" -passout pass:"${PRIVATE_KEY_PWD}"
# cleanup file: remove Bag/Key Attribute information
sed -i -n '/-BEGIN ENCRYPTED PRIVATE KEY-/,/-END ENCRYPTED PRIVATE KEY-/p' "${CERT_FILES_LOC}/private-pwd.key"

# remove passphrase from private key file (!!!security risk!!!)
openssl rsa -in "${CERT_FILES_LOC}/private-pwd.key" -out "${CERT_FILES_LOC}/private-nopwd.key" -passin pass:"${PRIVATE_KEY_PWD}"

# extract certificate from PKCS#12 file
openssl pkcs12 -clcerts -nokeys -in "${SOURCE_PKCS12_FILE}" -out "${CERT_FILES_LOC}/custom.cer" -password pass:"${SOURCE_PKCS12_PWD}"

# extract root and intermediate certs from PKCS#12 file
#   manually split ca-chain.cer into root.cer and inter.cer
openssl pkcs12 -in "${SOURCE_PKCS12_FILE}" -cacerts -nokeys -chain -out "${CERT_FILES_LOC}/ca-chain.cer" -password pass:"${SOURCE_PKCS12_PWD}"

Create PKCS#12 file with “openssl”:

CERT_FILES_LOC=/home/oracle
TARGET_PKCS12_FILE/home/oracle/opennssl.p12
TARGET_PKCS12_PWD=*****
PRIVATE_KEY_PWD=*****

cat "${CERT_FILES_LOC}/root.cer" "${CERT_FILES_LOC}/inter.cer" > "${CERT_FILES_LOC}/ca-chain.cer"
openssl pkcs12 -export -out "${TARGET_PKCS12_FILE}" -inkey "${CERT_FILES_LOC}/private-pwd.key" -in "${CERT_FILES_LOC}/custom.cer" -certfile "${CERT_FILES_LOC}/ca-chain.cer" -password pass:"${TARGET_PKCS12_PWD}" -passin pass:"${PRIVATE_KEY_PWD}"

Convert a PKCS#12 file to Java key store (JKS):

OPENSSL_PKCS12_FILE=/home/oracle/wallet/openssl.p12
OPENSSL_PKCS12_PWD=*****
JKS_FILE=/home/oracle/wallet/keystore.jks
JKS_PWD=*****

keytool -v -importkeystore -srckeystore "${OPENSSL_PKCS12_FILE}" -srcstoretype pkcs12 -srcstorepass "${OPENSSL_PKCS12_PWD}" -destkeystore "${JKS_FILE}" -deststoretype jks -deststorepass "${JKS_PWD}"

Footnote: tested “orapki” from Oracle 19.5 database home

Does the listener cache TLS certificates?

A while ago a fellow DBA asked me if the listener cached TLS certificates. My immediate answer was “Sure, not caching would hurt performance severely.”
But, I couldn’t be certain so I ran a trace on it.

As the listener.log shows I did connect three times using TLS enabled endpoint:

...
07-SEP-2018 11:05:30 * (CONNECT_DATA=(SERVICE_NAME=DEV1.localdomain)(CID=(PROGRAM=C:\app\oracle\product\client1830\bin\sqlplus.exe)(HOST=WIN2012CLI1)(USER=user1))) * (ADDRESS=(PROTOCOL=tcps)(HOST=192.168.56.29)(PORT=49211)) * establish * DEV1.localdomain * 0
07-SEP-2018 11:05:46 * (CONNECT_DATA=(SERVICE_NAME=DEV1.localdomain)(CID=(PROGRAM=C:\app\oracle\product\client1830\bin\sqlplus.exe)(HOST=WIN2012CLI1)(USER=user1))) * (ADDRESS=(PROTOCOL=tcps)(HOST=192.168.56.29)(PORT=49212)) * establish * DEV1.localdomain * 0
07-SEP-2018 11:05:48 * (CONNECT_DATA=(SERVICE_NAME=DEV1.localdomain)(CID=(PROGRAM=C:\app\oracle\product\client1830\bin\sqlplus.exe)(HOST=WIN2012CLI1)(USER=user1))) * (ADDRESS=(PROTOCOL=tcps)(HOST=192.168.56.29)(PORT=49213)) * establish * DEV1.localdomain * 0
...

During the entire time I had a listener trace and a β€œstrace” on the “tnslsnr” process.
Going through the strace output I found the open calls for both wallet files (ewallet.p12 and cwallet.sso).

Line 419: open("/u01/app/oracle/etc/wallet/auth/ewallet.p12", O_RDONLY) = 19
Line 506: open("/u01/app/oracle/etc/wallet/auth/cwallet.sso", O_RDONLY) = 20

Then the listener maps anonymous memory and reads data from cwallet.sso (file descriptor 20).

Line 514: mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0520168000
Line 517: read(20, "\272\333\241\211\10\to\264\306\247/w\217#\0n+[\t\371\v\266\244\230d\214e3\246ZV\22"..., 1149) = 1149
...
Line 531: read(20, "\241\370N8\0\0\0\6\0\0\0!\6\303\20]{\207\16_\246\247\3579'\234h\35I\301m="..., 4096) = 4096
...
Line 542: read(20, "\272\333\241\211\10\to\264\306\247/w\217#\0n+[\t\371\v\266\244\230d\214e3\246ZV\22"..., 4096) = 1149

Shortly thereafter, the listener closes the file handles and unmaps the anonymous memory.

Line 551: close(19)
Line 561: close(20)
Line 562: munmap(0x7f0520168000, 4096)            = 0

All this happens on the first incoming TLS connection request. After that it never touches any of the wallet files again.

The same can be observed in the listener.og: it opens/reads/closes the wallet file on the first incoming TLS connection request only.

Line 4473: CONNECTION REQUEST
Line 4627: snzdfo_open_file:Opening file /u01/app/oracle/etc/wallet/auth/ewallet.p12 with READ ONLY permissions
Line 4631: snzdfo_open_file:Opening file /u01/app/oracle/etc/wallet/auth/cwallet.sso with READ ONLY permissions
Line 4667: nztwOpenWallet:exit

I didn’t do any long running tests but I this proves that the listener does cache the certificate from the wallet (at least temporarily).

SCAN listener client connections

I can see how this depiction of client connections through SCAN listeners can be confusing. You might think that SCAN listeners connect to local listeners and pass through the connection from the client to the local listeners (like a proxy).

Source: support.oracle.com (MOS Doc ID 887522.1)

This would be a bad idea as all database traffic would now flow through the SCAN listeners. Instead, what actually happens is this: the SCAN listener sends a “redirect” message to the client and the client then establishes a new connection to the local listener given in the message.

I’m using a fail-over connect string to illustrate what happens if the primary database is on “ora122racB” cluster. This means the client will first go through all SCAN listeners on “ora122racA” cluster and then fail-over to the next address.

(DESCRIPTION=
  (ADDRESS_LIST=
    (ADDRESS=
      (PROTOCOL=TCP)(HOST=ora122racA-scan.localdomain)(PORT=1521)
    )
    (ADDRESS=
      (PROTOCOL=TCP)(HOST=ora122racB-scan.localdomain)(PORT=1521)
    )
  )
  (CONNECT_DATA=(SERVICE_NAME=racpdb1.localdomain))
)

Running a SQL*Net trace reveals the following sequence.
First, it expands the SCAN IPs:

nlad_expand_hst: Result: (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.25)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.27)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.26)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.35)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.36)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.37)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=racpdb1.localdomain)(CID=(PROGRAM=C:\app\oracle\product\ora12201\bin\sqlplus.exe)(HOST=T11967)(USER=spotonoracle))))
nlad_pr: entry
nlad_pr: description processing
nlad_pr: entry
snlinGetAddrInfo: entry
snlinGetAddrInfo: exit
nlad_pr: hostname 192.168.56.25
snlinFreeAddrInfo: entry
snlinFreeAddrInfo: exit
snlinGetAddrInfo: entry
snlinGetAddrInfo: exit
nlad_pr: hostname 192.168.56.27
snlinFreeAddrInfo: entry
snlinFreeAddrInfo: exit
snlinGetAddrInfo: entry
snlinGetAddrInfo: exit
nlad_pr: hostname 192.168.56.26
snlinFreeAddrInfo: entry
snlinFreeAddrInfo: exit
snlinGetAddrInfo: entry
snlinGetAddrInfo: exit
nlad_pr: hostname 192.168.56.35
snlinFreeAddrInfo: entry
snlinFreeAddrInfo: exit
snlinGetAddrInfo: entry
snlinGetAddrInfo: exit
nlad_pr: hostname 192.168.56.36
snlinFreeAddrInfo: entry
snlinFreeAddrInfo: exit
snlinGetAddrInfo: entry
snlinGetAddrInfo: exit
nlad_pr: hostname 192.168.56.37
snlinFreeAddrInfo: entry
snlinFreeAddrInfo: exit

The first connection is established to the first SCAN IP in above list.

nsc2addr: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.25)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=racpdb1.localdomain)(CID=(PROGRAM=C:\app\oracle\product\ora12201\bin\sqlplus.exe)(HOST=WIN10VM01)(USER=spotonoracle))))
...
nsprecv: 00 67 01 00 04 00 02 00  |.g......|
nsprecv: 22 00 20 5B 28 44 15 52  |"..[(DES|
nsprecv: 53 52 49 50 74 49 4F 4E  |CRIPTION|
nsprecv: 1D 28 54 4E 50 3D 29 98  |=(TMP=)(|
nsprecv: 26 52 4E 4D 55 4D 3D 32  |VSNNUM=2|
nsprecv: 40 32 33 37 35 36 38 50  |02375680|
nsprecv: 39 18 45 52 52 3D 31 22  |)(ERR=12|
nsprecv: 25 31 34 26 38 55 52 62  |514)(ERR|
nsprecv: 4F 54 5E 53 54 21 43 4B  |OR_STACK|
nsprecv: 3D 28 25 52 52 4F 12 3D  |=(ERROR=|
nsprecv: 28 23 4F 44 45 3D 71 32  |(CODE=12|
nsprecv: 35 11 34 49 24 45 4E 46  |514)(EMF|
nsprecv: 49 3F 34 29 29 29 30     |I=4)))) |

Since the SCAN listener does not know the service “racpdb1.localdomain” it does return an error. Remember, the primary database is running on the other cluster.
Then it tries the second SCAN IP from the list.

nsc2addr: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.27)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=racpdb1.localdomain)(CID=(PROGRAM=C:\app\oracle\product\ora12201\bin\sqlplus.exe)(HOST=WIN10VM01)(USER=spotonoracle))))
...
nsprecv: 00 67 01 00 04 00 02 00  |.g......|
nsprecv: 22 00 20 5B 28 44 15 52  |"..[(DES|
nsprecv: 53 52 49 50 74 49 4F 4E  |CRIPTION|
nsprecv: 1D 28 54 4E 50 3D 29 98  |=(TMP=)(|
nsprecv: 26 52 4E 4D 55 4D 3D 32  |VSNNUM=2|
nsprecv: 40 32 33 37 35 36 38 50  |02375680|
nsprecv: 39 18 45 52 52 3D 31 22  |)(ERR=12|
nsprecv: 25 31 34 26 38 55 52 62  |514)(ERR|
nsprecv: 4F 54 5E 53 54 21 43 4B  |OR_STACK|
nsprecv: 3D 28 25 52 52 4F 12 3D  |=(ERROR=|
nsprecv: 28 23 4F 44 45 3D 71 32  |(CODE=12|
nsprecv: 35 11 34 49 24 45 4E 46  |514)(EMF|
nsprecv: 49 3F 34 29 29 29 30     |I=4)))) |

Same error return message from this SCAN listener.
And then the third SCAN IP.

nsc2addr: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.26)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=racpdb1.localdomain)(CID=(PROGRAM=C:\app\oracle\product\ora12201\bin\sqlplus.exe)(HOST=WIN10VM01)(USER=spotonoracle))))
...
nsprecv: 00 67 01 00 04 00 02 00  |.g......|
nsprecv: 22 00 20 5B 28 44 15 52  |"..[(DES|
nsprecv: 53 52 49 50 74 49 4F 4E  |CRIPTION|
nsprecv: 1D 28 54 4E 50 3D 29 98  |=(TMP=)(|
nsprecv: 26 52 4E 4D 55 4D 3D 32  |VSNNUM=2|
nsprecv: 40 32 33 37 35 36 38 50  |02375680|
nsprecv: 39 18 45 52 52 3D 31 22  |)(ERR=12|
nsprecv: 25 31 34 26 38 55 52 62  |514)(ERR|
nsprecv: 4F 54 5E 53 54 21 43 4B  |OR_STACK|
nsprecv: 3D 28 25 52 52 4F 12 3D  |=(ERROR=|
nsprecv: 28 23 4F 44 45 3D 71 32  |(CODE=12|
nsprecv: 35 11 34 49 24 45 4E 46  |514)(EMF|
nsprecv: 49 3F 34 29 29 29 30     |I=4)))) |

Again, obvously, we get the same return message.

Now, the client fails over to the second address in the connection descriptor. It connects to the first SCAN IP from the list for cluster “ora122racB”. This time we expect a positive response from the SCAN listener.

nsc2addr: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.35)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=racpdb1.localdomain)(CID=(PROGRAM=C:\app\oracle\product\ora12201\bin\sqlplus.exe)(HOST=WIN10VM01)(USER=spotonoracle))))
...
nsprecv: 02 32 00 11 06 00 00 00  |.2......|
nsprecv: 01 40 54 41 44 44 52 45  |.@(ADDRE|
nsprecv: 43 53 3D 98 50 52 4F 54  |SS=(PROT|
nsprecv: 45 43 4F 4C 3D 54 43 50  |OCOL=TCP|
nsprecv: 22 28 71 4F 53 54 3D 24  |)(HOST=1|
nsprecv: 33 2E 32 95 32 2E 47 24  |92.168.5|
nsprecv: 3F 1E 15 39 29 14 87 2F  |6.33)(PO|
nsprecv: 92 54 3D 58 35 32 12 30  |RT=1521)|
nsprecv: 39 00 28 37 45 53 43 52  |).(DESCR|
nsprecv: 44 50 94 49 4F 4E 3D 28  |IPTION=(|
nsprecv: 46 44 44 27 45 53 53 3D  |ADDRESS=|
nsprecv: 28 50 85 4F 54 4F 43 4F  |(PROTOCO|
nsprecv: 9C 3D 54 23 50 29 28 48  |L=TCP)(H|
nsprecv: 48 53 65 3D 51 33 2F 12  |OST=192.| 
nsprecv: 34 58 88 74 87 45 1F 21  |168.56.3|
nsprecv: 98 29 68 50 4F 52 54 3D  |5)(PORT=|
nsprecv: 51 35 32 58 35 32 12 30  |1521))(C|
nsprecv: 5F 4E 4E 87 43 54 5F 44  |ONNECT_D|
nsprecv: 51 54 45 3D 28 53 45 52  |ATA=(SER|
nsprecv: 26 49 43 36 5F 4E 41 4D  |VICE_NAM|
nsprecv: 35 3D 61 77 47 83 19 84  |E=racpdb|
nsprecv: 35 30 4D 34 7C 3E 4C 56  |1.locald|
nsprecv: 63 6E 67 36 61 72 64 2E  |omain)(C|
nsprecv: 62 6F 8E 29 28 43 49 44  |ID=(PROG|
nsprecv: 3E 28 50 17 4F 47 52 41  |RAM=C:\a|
nsprecv: 4C 3D 22 37 5C 3F 72 51  |pp\oracl|
nsprecv: 68 6E 52 31 38 6C 6E 52  |e\produc|
nsprecv: 62 31 41 33 5E 92 73 2E  |t\ora122|
nsprecv: 3C 73 71 3F 79 7C 78 23  |01\bin\s|
nsprecv: 1E 65 29 65 29 28 48 4F  |qlplus.e|
nsprecv: 91 54 41 2F 28 53 45 52  |xe)(HOST|
nsprecv: 51 84 3F 27 34 87 37 51  |=WIN10VM|
nsprecv: 38 29 95 55 53 45 52 3D  |01)(USER|
nsprecv: 73 72 63 35 29 29 28 53  |=spotono|
nsprecv: 2F 4E 4E 18 43 54 5F 44  |racle))(|
nsprecv: 35 52 97 45 52 3F 64 65  |SERVER=d|
nsprecv: 87 69 63 65 74 65 64 29  |edicated|
nsprecv: 29 49 6F 53 54 41 4E 43  |)(INSTAN|
nsprecv: 46 5F 4E 67 4D 45 3D 45  |CE_NAME=|
nsprecv: 61 44 19 7E 34 38 14 57  |raccdb11|
nsprecv: 39 36                    |)))     |

Indeed, the listener did respond with the following redirect address (VIP from local listener):

(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.33)(PORT=1521))

Now, the client connects the local listener given in the redirect message.

nsc2addr: (ADDRESS=(PROTOCOL=TCP)(HOST=129.168.56.33)(PORT=1521))

From here on the connection protocol takes its regular turn (same with or without SCAN).

I hope this makes it a bit more clear how clients connection to databases using SCAN listeners.

SQL Developer – enabling TLS for JDBC thin driver

I don’t think this is documented anywhere (and no hits on google) but it shows once again how awsome SQL Developer is. πŸ™‚
Instead of using “jdbc:oracle:oci” for TLS enabled connections we’re going to set SQL Developer up to do just that with “jdbc:oracle:thin”. In that regard, SQL Developer is no more special than any Java application.
To keep it short I’m sticking to the basics, maybe I’ll add some variations into the comments later.
First, we need to add “OraclePKIProvider” to the JRE security provider’s list. The provider’s list is defined in “$JAVA_HOME/jre/lib/security/java.security”.
I’m going to append to it by creating a new file that I only source from SQL Developer. In my case, the next free slot is 10. Depending on your OS and JRE version the list may slightly vary.
(also make sure that the java.security file is appendable: “security.overridePropertiesFile=true”)

$ vi /opt/sqldeveloper/sqldeveloper/bin/sqldev.java.security
security.provider.10=oracle.security.pki.OraclePKIProvider

Next, we add a few options to the sqldeveloper.conf file:

$ vi /opt/sqldeveloper/sqldeveloper/bin/sqldeveloper.conf
# add this to end of the config file

# specify the path to your SSO wallet file
AddVMOption -Doracle.net.wallet_location="(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/home/btr/tns/wallet)))"
# this is required if you want to use TLS authentication next to in-flight traffic encryption
AddVMOption -Doracle.net.authentication_services="(TCPS)"
# optionally, if you want the client to verify the server certificate
AddVMOption -Doracle.net.ssl_server_dn_match=true
# append/override JRE security config to add Oracle provider
AddVMOption -Djava.security.properties=/opt/sqldeveloper/sqldeveloper/bin/sqldev.java.security
# even SQLDev 18.1 ships with a very old version that is not compatible - just download the latest one from OTN
AddJavaLibFile /home/btr/orasec/jlib/oraclepki.jar

That’s it, we’re done.

Now you can create connections using TNS, LDAP, or Advanced. Basic/EZCONNECT does not support to specify the protocol, so it can’t be used for TLS.

Here we only use TLS for in-flight traffic encryption:

And here, we make use of TLS user authentication:

Little trick with lsnrctl

Are you running RAC/GI with role separation? You’re tired of constantly switching between “oracle” and “grid” users or you don’t have access to “grid” at all?
This little trick saves me lots of time whenever I need to check the status or services details of a listener running as “grid”. I hope it serves you as well as it does serve me πŸ™‚

This gives me the listener names running on the connected node:

$ ps -ef|grep tnslsnr
grid      3468     1  0 21:50 ?        00:00:00 /u01/app/gi12201/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
grid      3643     1  0 21:50 ?        00:00:00 /u01/app/gi12201/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid      3646     1  0 21:50 ?        00:00:00 /u01/app/gi12201/bin/tnslsnr MGMTLSNR -no_crs_notify -inherit
grid      3652     1  0 21:50 ?        00:00:00 /u01/app/gi12201/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit
grid      3665     1  0 21:50 ?        00:00:00 /u01/app/gi12201/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit

By default, every listener in a GI/RAC setup has an IPC end-point with the KEY being the same as the listener name. For instance:

Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=192.168.56.55)(PORT=1523)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.55)(PORT=1521)))

So you can simply run lsnrctl with the connect descriptor on the command line (as oracle):

$ lsnrctl status "(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))"

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 17-MAY-2018 22:12:49

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN1
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                17-MAY-2018 21:52:41
Uptime                    0 days 0 hr. 20 min. 8 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/gi12201/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/ol7122rac2/listener_scan1/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=192.168.56.55)(PORT=1523)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.55)(PORT=1521)))
Services Summary...
Service "65388e79ee096b69e0533238a8c08c32.localdomain" has 2 instance(s).
  Instance "RAC11", status READY, has 2 handler(s) for this service...
  Instance "RAC12", status READY, has 2 handler(s) for this service...
Service "RAC1.localdomain" has 2 instance(s).
  Instance "RAC11", status READY, has 2 handler(s) for this service...
  Instance "RAC12", status READY, has 2 handler(s) for this service...
Service "pdbrac1.localdomain" has 2 instance(s).
  Instance "RAC11", status READY, has 2 handler(s) for this service...
  Instance "RAC12", status READY, has 2 handler(s) for this service...
The command completed successfully

Yes, this only works for listeners that run on the node you’re connected to. You can’t do it across nodes, but I still find it extremely useful.

For your convenience, you can put the connect descriptors in the TNSNAMES.ORA sourced by “oracle”:

$ cat $TNS_ADMIN/tnsnames.ora 
listener_scan1=(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))

$ lsnrctl services listener_scan1

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 17-MAY-2018 22:30:10

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
Services Summary...
Service "65388e79ee096b69e0533238a8c08c32.localdomain" has 2 instance(s).
  Instance "RAC11", status READY, has 2 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.56.53)(PORT=1523))
...

It also works within the lsnrctl command line:

$ lsnrctl

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 17-MAY-2018 22:39:58

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Welcome to LSNRCTL, type "help" for information.

LSNRCTL> set current_listener (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
Current Listener is (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
LSNRCTL> services
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
Services Summary...
Service "65388e79ee096b69e0533238a8c08c32.localdomain" has 2 instance(s).
  Instance "RAC11", status READY, has 2 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.56.53)(PORT=1523))
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.53)(PORT=1521))
...

Any comments? I’m listening…

Play fetch – get exactly one row at a time

Every once in a while I want SQL*Plus to fetch one row at a time, so I set ARRAYSIZE to 1.
But:

SQL> set autotrace traceonly statistics
SQL> set arraysize 1
SQL> select * from user1.cf1 where rownum <= 100;

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
        102  consistent gets
          0  physical reads
          0  redo size
     258194  bytes sent via SQL*Net to client
       1091  bytes received via SQL*Net from client
         51  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
        100  rows processed

We only did 51 network round trips to fetch 100 rows. This is because SQL*Plus is using OCI and the default for prefetch (OCI_ATTR_PREFETCH_ROWS) is set to 1. So we end up with 2 rows per fetch call (note that with prefetch 1, the first round trip already returns 1 row).

Now, I really, really want to fetch just one row at a time. Starting with Oracle 12.1 you can create an OCI configuration file to specify various settings, one of them being prefetch. See documentation for more details.

vi ${TNS_ADMIN}/oraaccess.xml

<?xml version="1.0"?>
 <oraaccess xmlns="http://xmlns.oracle.com/oci/oraaccess" xmlns:oci="http://xmlns.oracle.com/oci/oraaccess" schemaLocation="http://xmlns.oracle.com/oci/oraaccess http://xmlns.oracle.com/oci/oraaccess.xsd">
  <default_parameters>
    <prefetch>
      <rows>0</rows>
    </prefetch>
  </default_parameters>
</oraaccess>

With this configuration in place we should see 100+ network round trips.

SQL> set autotrace traceonly statistics
SQL> set arraysize 1
SQL> select * from user1.cf1 where rownum <= 100;

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
        102  consistent gets
          0  physical reads
          0  redo size
     267988  bytes sent via SQL*Net to client
       1652  bytes received via SQL*Net from client
        102  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
        100  rows processed

Go play fetch πŸ˜‰

Are you fishing or catching? – Server-side SQL*Net tracing for specific clients

SQL*Net tracing on the database server is all or nothing (sqlnet.ora):
TRACE_LEVEL_SERVER=support

On a busy system this quickly gets out of hand. More often than not I find myself in need of enabling SQL*Net trace on the server for a particular database client.
The first step is to setup a separate TNS_ADMIN location that we want the sever process for our client connections to pick up:

$ mkdir /u01/app/oracle/network/admin-trc
$ copy /u01/app/oracle/network/admin/sqlnet.ora /u01/app/oracle/network/admin-trc/.

Then enable SQL*Net tracing in the new TNS_ADMIN location:

vi /u01/app/oracle/network/admin-trc/sqlnet.ora

...
TRACE_LEVEL_SERVER=support

If you can modify the client’s connection string things are easy. All we need is a separate service on the existing listener that the client then connects to:
vi /u01/app/oracle/network/admin/listener.ora

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = pdbdev1_trc.localdomain)
      (ORACLE_HOME = /u01/app/oracle/product/ora12201)
      (SID_NAME = DEV1)
      (ENVS="TNS_ADMIN=/u01/app/oracle/network/admin-trc")
    )
  )

Line 4: Defines our new service name
Line 7: This makes server processes for connections through that service read our sqlnet.ora in the “admin-trc” directory and, therefore enable SQL*Net tracing.

Now, change the client’s connection string to use “pdbdev_trc” service and we’re done.
But, what if for some reason you cannot easily change the client’s connect string. That’s when I like to resort to port-forwarding using firewalld.

First, we spin up an additional listener on another port. Let’s assume our original listener runs on port 1521.
vi /u01/app/oracle/network/admin/listener.ora

LISTENER_TRC =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = ol7ora122dev1.localdomain)(PORT = 1522))
    )
  )

SID_LIST_LISTENER_TRC =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = pdbdev1.localdomain)
      (ORACLE_HOME = /u01/app/oracle/product/ora12201)
      (SID_NAME = DEV1)
      (ENVS="TNS_ADMIN=/u01/app/oracle/network/admin-trc")
    )
  )

Line 4: listening on new port
Line 11: the service name is the same as on the original listener (where the client connects to)
Line 14: Again, this sets the TNS_ADMIN location for the server process to pick up our sqlnet.ora with SQL*Net tracing enabled.

Secondly, add the port forwarding rule:

firewall-cmd --zone=oradb-server \
  --add-rich-rule='rule family="ipv4" source address="<client-IP>" forward-port port=1521 protocol=tcp to-port=1522'

Now, every connection request from the client we want to have a server-side SQL*Net trace from gets forwarded to our new listener and picks up our modified sqlnet.ora.
Obviously, this is assuming you have firewalld configured and running already. If not, await my next blog post on basic firewalld configuration for Oracle database servers.

Enjoy!

P.s. when you have all the traces you need you can remove the port forwading rule like this:

firewall-cmd --zone=oradb-server \
  --remove-rich-rule='rule family="ipv4" source address="<client-IP>" forward-port port=1521 protocol=tcp to-port=1522'

Tracking down PGA memory leak – a walkthrough

The other day I tracked down a nasty problem one application was having. It was first noticed in the alert.log:

ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT
...
PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT of 7000 MB
Immediate Kill Session#: 686, Serial#: 26646
Immediate Kill Session: sess: 0x19e7c2a78  OS pid: 35173

Checking the PGA memory from gv$process revealed a shocking picture:

SQL> select prc.inst_id, ses.sid, ses.serial#, prc.pid, prc.spid, round(prc.pga_used_mem/1024/1024) used_mb, round(prc.pga_alloc_mem/1024/1024) alloc_mb, round((prc.pga_alloc_mem - prc.pga_used_mem)/1024/1024, 0) unused_mb, round(prc.pga_freeable_mem/1024/1024) freeable_mb, round(prc.pga_max_mem/1024/1024) max_mb
from gv$process prc
  left outer join gv$session ses on (ses.inst_id = prc.inst_id and ses.paddr = prc.addr and ses.type = 'USER')
order by used_mb desc
fetch first 10 rows only;

   INST_ID        SID    SERIAL#        PID SPID                        USED_MB   ALLOC_MB  UNUSED_MB                             FREEABLE_MB     MAX_MB
---------- ---------- ---------- ---------- ------------------------ ---------- ---------- ---------- --------------------------------------- ----------
         1        655      50496         89 338934                          227        284         57                                       0        499
         1        675      45630        137 338938                          223        280         57                                       0        296
         1        416       7124        285 315339                          222        281         58                                       1        298
         1        290      45321        296 81710                           184        233         49                                       0        280
         2        296      52565         98 223329                          163        207         45                                       0        270
         2        710       8463        293 386687                          162        207         45                                       0        270
         1         67      15923         54 98311                           147        187         40                                       0        264
         2        295      25163        242 316189                          143        185         42                                       0        266
         1         32      55120        270 358206                          137        178         41                                       0        260
         2        309      13006        230 167307                          135        175         39                                       0        258

 10 rows selected 

A lot of sessions used between 100MB and 230MB. Monitoring these sessions over some time showed that the memory was never released, something’s definitely wrong. PGA dumps of a couple of connections all showed the same pattern.

oradebug setmypid
oradebug dump pga_detail_get <pid>
SQL> select category, name, heap_name, round(bytes/1024/1024, 0) size_mb, allocation_count
from gv$process_memory_detail
where pid = 89
order by pid, bytes desc
fetch first 10 rows only;

CATEGORY        NAME                       HEAP_NAME          SIZE_MB                        ALLOCATION_COUNT
--------------- -------------------------- --------------- ---------- ---------------------------------------
Other           permanent memory           kolarsCreateCt         214                                   10541
Other           free memory                top uga heap            57                                    1571
Other           free memory                kolarsCreateCt           3                                    4907
Other           free memory                session heap             2                                    1016
Other           kollalos2                  koh-kghu sessi           1                                      86
Other           permanent memory           pga heap                 1                                      43
Other           kxsFrame4kPage             session heap             0                                     123
Other           qmushtCreate               qmtmInit                 0                                       6
PL/SQL          static frame of inst       koh-kghu sessi           0                                      36
Other           inode->map_kdlimem         buckets_kdliug           0                                       9

The closest I come to kolarsCreateCt in the oradebug doc is this:

oradebug doc component
...
Components in library GENERIC:
--------------------------
...
  LOB                          LOB (koll, kola)
...

This is the clue that there is apparently a LOB resource management problem. Luckily, the PL/SQL code base was not too big and only a couple of places where LOB handling was done. But, all LOBs seemed to be handled correctly.
One thing, though, I first didn’t pay much attention to was a few “XMLTYPE.getClobVal()” in SELECT statements. I’ve used that myself in the past a couple of times and knew this returned a persistent LOB handle and should not leak.
I took one of these SQL statements and ran them in my favorite tool, SQL Developer, and BOOOOOM! My session just consumed a couple of hundreds of megabytes of PGA memory and wouldn’t release it until I closed the connection.

So I crafted a little test case to check what’s going on.

SQL> create table xmltest(id number, data xmltype, primary key (id))
SQL>   xmltype column data store as binary xml;
SQL> insert into xmltest (id, data) values (1, '<someelement></someelement>');
SQL> commit;
SQL> select t.id, dbms_lob.istemporary(t.data.getclobval()) is_temp from xmltest t;

        ID    IS_TEMP
---------- ----------
         1          1

I don’t think this is what I knew from past experience so I did a quick MOS search and found following bug report:
Bug 23061709 : HIGH KOLARSCREATECT AND ORA-04036 WHEN XMLTYPE COLUMN IS STORED AS BINARY

OK, lets change the XMLType storage to CLOB and see what happens:

SQL> create table xmltest(id number, data xmltype, primary key (id))
SQL>   xmltype column data store as securefile clob;
SQL> insert into xmltest (id, data) values (1, '<someelement></someelement>');
SQL> commit;
SQL> select t.id, dbms_lob.istemporary(t.data.getclobval()) is_temp from xmltest t;

        ID    IS_TEMP
---------- ----------
         1          0

To be fair, the documentation for 11.2, 12.1, and 12.2 all state the following:
“This member function returns a CLOB containing the serialized XML representation. If the CLOB returned is temporary, it must be freed after use.”

Because I always run my tests in SQL*Plus additionally to SQL Developer I was puzzled to see that there’s no memory leak. Why does it not happen in SQL*Plus? I thought that maybe SQL*Plus is clever enough to always check for temporary LOBs and free them.

Firing up gdb with this script proved me right.

break OCILobFreeTemporary
  commands
    where
    continue
  end

This SQL fetches five rows with CLOB data.

SQL> select t.id, t.data.getclobval() from xmltest t where rownum <= 5;

Output from gdb:

Breakpoint 1, 0x00007f678cc3d250 in OCILobFreeTemporary () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libclntsh.so.12.1
#0  0x00007f678cc3d250 in OCILobFreeTemporary () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libclntsh.so.12.1
#1  0x00007f678eafb1ef in afioci () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libsqlplus.so
#2  0x00007f678eaed912 in afifpilobfreetemp () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libsqlplus.so
#3  0x00007f678eaed799 in afifpifretmp () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libsqlplus.so
#4  0x00007f678eae8f20 in afidet () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libsqlplus.so
#5  0x00007f678eb272f3 in safiqry () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libsqlplus.so
#6  0x00007f678eb096e3 in afiqry () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libsqlplus.so
#7  0x00007f678eb21bf2 in afixeqqry () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libsqlplus.so
#8  0x00007f678eb21521 in afixeqsql () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libsqlplus.so
#9  0x00007f678eb20ed5 in afixeqr () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libsqlplus.so
#10 0x00007f678eabfd74 in aficfd () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libsqlplus.so
#11 0x00007f678eabec49 in aficdr () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libsqlplus.so
#12 0x00007f678eaeacf2 in afidrv () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libsqlplus.so
#13 0x00000000004007a1 in main ()

Breakpoint 1, 0x00007f678cc3d250 in OCILobFreeTemporary () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libclntsh.so.12.1
...
#13 0x00000000004007a1 in main ()

Breakpoint 1, 0x00007f678cc3d250 in OCILobFreeTemporary () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libclntsh.so.12.1
...
#13 0x00000000004007a1 in main ()

Breakpoint 1, 0x00007f678cc3d250 in OCILobFreeTemporary () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libclntsh.so.12.1
...
#13 0x00000000004007a1 in main ()

Breakpoint 1, 0x00007f678cc3d250 in OCILobFreeTemporary () from /u01/app/oracle/product/12.1.0.2.161018/RAC/lib/libclntsh.so.12.1
...
#13 0x00000000004007a1 in main ()

Five rows fetched and gdb shows five calls to OCILobFreeTemporary. The call to OCILobFreeTemporary originates from libsqlplus.so which tells me it’s not the OCI layer itself handling it, it’s the logic in SQL*Plus.

Next, I wanted to see myself what happens in a Java/JDBC application. Here’s the relevant snippet from my test case:

      String sql = "select t.id, t.data.getclobval() from xmltest t where rownum <= 5";
      stmt = conn.prepareStatement(sql);
      
      for (int i=0; i<3; i++) {
      stmt = conn.prepareStatement(sql);
        rset = stmt.executeQuery();

        while (rset.next ()) {
          System.out.println (rset.getInt(1));
          xmlclob = rset.getClob(2);
          xmlclob.free();
        }

        rset.close();

        c.format("\nPress ENTER 1 to proceed.\n");
        c.readLine();
      }

Like SQL*Plus being an application handling the LOB resources, you have to do it programmatically in Java as well. Line 11 frees the temporary LOB.
Without the free() call on the Clob you will see the memory leak when there XMLType storage clause is XML BINARY. It happens, even when do don’t reference the LOB column in Java:

        while (rset.next ()) {
          System.out.println (rset.getInt(1));
        }

Lesson learnt: never trust what you think you know! Sometimes things change in subtle ways.

Rant: Does Oracle care anymore?

It seems Oracle has lost all interest in its on-premises software. I’ve noticed this with other features and behaviour changes, this is just a small example.
Remember the article about DBCA silent installation? I mentioned the following:

  • Controlfiles are not multiplexed to different paths as specified. First given path is used for all controlfile members (SR is "Work In Progress")

Last Thursday, after 3+ month, my Service Request has finally been updated with an answer. They’ve changed DBCA to ignore the <ControlfileAttributes> section and the CONTROL_FILES “initParam” value unless you specify “-honorControlFileInitParam” argument. This argument does not show in the dbca help “dbca -createDatabase -help” nor is it anywhere in the documentation – how I’m I supposed to know?. And this is what bugs me the most: Oracle 12.2 on-premises is out now for quite a while but it seems that the documentation has not been updated in many places. Guess that’s how it’s gonna be from now on πŸ™