In this Document
APPLIES TO: Oracle SOA Suite - Version 11.1.1.1.0 and later
Oracle HTTP Server - Version 12.1.3.0.0 and later
Oracle WebLogic Server - Version 8.1 and later
Linux x86
Linux x86-64
SYMPTOMS
It is observed on some Linux boxes that WebLogic server startup takes several minutes and hangs for a while. Similar behavior happens during the domain creation, when the security information gets populated.
If you take a thread dump on the troubled process you will observe that WebLogic is waiting for random data generation since the OS is running out of entropy.
- locked <0x00000000e061f4b8> (a java.lang.Object)
at
sun.security.provider.NativePRNG$RandomIO.access$300(NativePRNG.java:108)
at sun.security.provider.NativePRNG.engineGenerateSeed(NativePRNG.java:102)
at java.security.SecureRandom.generateSeed(SecureRandom.java:495)
at com.bea.security.utils.random.AbstractRandomData.ensureInittedAndSeeded(AbstractRandomData.java:91)
- locked <0x00000000f8c7d7b8> (a com.bea.security.utils.random.SecureRandomData)
at
com.bea.security.utils.random.AbstractRandomData.getRandomBytes(AbstractRandomData.java:105)
- locked <0x00000000f8c7d7b8> (a com.bea.security.utils.random.SecureRandomData)
at
com.bea.security.utils.random.AbstractRandomData.getRandomBytes(AbstractRandomData.java:100)
at com.bea.console.utils.CSRFUtils.getSecret(CSRFUtils.java:56)
at jsp_servlet._jsp._changemgmt.__changemanager._jspService(__changemanager.java:156)
CAUSE
According to the official Kernel documentation, Linux has two devices to provide random data at any time:/dev/random and /dev/urandom. Both ways should be secure enough to use them in generating PGP keys, ssh challenges, and other applications where secure random numbers are required. Starting on kernel 2.6, default entropy is 4096 bits and problem arises when the entropy available on the system is minimum (around 100 bits or less).
The main difference between those two devices is that /dev/random runs out of random bits and makes you wait for more to be accumulated. Note that on some systems, it can block for a long time waiting for new user-generated entropy to be entered into the system.
In terms of the outcome,/dev/random is categorized as a high quality entropy device if we compare it with /dev/urandom. The latter uses the entropy pool as long as it is available, but falls back on pseudo random numeric algorithms when depleted.
Why a system could be running out of entropy?
You have to consider that an Operating System performs cryptographic operations frequently (on ssh challenges, https connections, etc.) so the /dev/random pool gets consumed quite quickly. OS also expects to feed that pool with I/O operations coming from disk, network, mouse or keyboard but that situation does not happen as quickly. This is a common pattern on virtualized environments or headless boxes.
Is important to mention that Java uses /dev/random by defaultas entropy generator device.
How to verify if you are encountering this issue?
Check the default system entropy.
$ cat /proc/sys/kernel/random/poolsize
4096
Check the available entropy.
$ cat /proc/sys/kernel/random/entropy_avail
125 On previous example, entropy is too low.
Monitor the current entropy of the system by using the following command:
$ for i in $(seq 500); do cat /proc/sys/kernel/random/entropy_avail ; sleep 5; done
Start a WebLogic server instance. You should see that entropy decreases or stalls.
The following links explain in detail how Entropy affects Java on Linux Environments: bug 6202721 and6521844.
SOLUTION
Choose one of the following approaches.
1. Feed the /dev/random device with additional I/O operations or using a random number generator tool (likerngd).
2. Instruct WebLogic server to use a non-blocking entropy device.
2.1 Long term solution
a) WebLogic Server Scope
i. Edit the Weblogic startup script ($DOMAIN_HOME/bin/startWebLogic.sh)
ii. Add the following to the JAVA_OPTIONS variable: -Djava.security.egd=file:/dev/./urandom
iii. Save the file.
iv. Set the domain environment. ($DOMAIN_HOME/bin/setDomainEnv.sh)
v. Start WebLogic instances.
b) JDK Scope
i. Edit the Java Security Properties file ($JAVA_HOME/jre/lib/security/java.security)
ii. The securerandom.source property specifies the source of seed data for secure random. If that property points to /dev/random, set it as one of the options listed below.
securerandom.source=file:/dev/./urandom
securerandom.source=file:/dev/urandom
iii. Save changes and start the WebLogic Server instances.
2.2 Temporary solution (usually applied for testing purposes)
i. Override the JAVA_OPTIONS environment variable before starting WebLogic Server via shell scripts.
$ export JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.security.egd=file:/dev/./urandom"
ii. Start WebLogic instances.