[Perf issue diagnostic] dotnet core takes long time to verify x509 certificate on Ubuntu 18.04

After installing dotnet core 3.0 preview 5 and openssl 1.1.1 on my Ubuntu 18.04, I found the x509 certificate validation takes long time (> 3 min for 1000 validations). The x509 certificate validtion benchmark is from https://github.com/dotnet/corefx/issues/35086. What happens?

Diagnostic process:

1. Check CPU usage: less than 10%, it is almost idle. So, there may be other reasons.

2. Check network. I used "iftop" and found there is access to "apps.digsigtrust.com" (or "192.35.177.xx") when the benchmark running. After the benchmark finishes, that access disappears. It looks like during x509 validation, it goes to network to download something.

3. Further diagnostic: when the benchmark is running, use "top" to get its PID. Then use "strace -p $PID" to check all system calls. I saw below. Obviously, the benchmark does not find expected CA.

stat("/home/xx/.dotnet/corefx/cryptography/x509stores/disallowed", 0x7fff27e3ea30) = -1 ENOENT (No such file or directory)
lstat("/home/xx/.dotnet/corefx/cryptography/x509stores/disallowed", 0x7fff27e3ea40) = -1 ENOENT (No such file or directory)
lstat("/usr/local/ssl/cert.pem", 0x7fff27e3e9f0) = -1 ENOENT (No such file or directory)
lstat("/usr/local/ssl/certs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/xx/.dotnet/corefx/cryptography/x509stores/root", 0x7fff27e3ea10) = -1 ENOENT (No such file or directory)
lstat("/home/xx/.dotnet/corefx/cryptography/x509stores/ca", 0x7fff27e3ea10) = -1 ENOENT (No such file or directory)
lstat("/home/xx/.dotnet/corefx/cryptography/x509stores/my", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=554, ...}) = 0
recvmsg(45, 0x7fff27e3dce0, 0) = -1 EAGAIN (Resource temporarily unavailable)
sendmsg(45, {msg_name(0)=NULL, msg_iov(1)=[{"GET / HTTP/1.1\r\nHost: cert.int-x"..., 53}], msg_controllen=0, msg_flags=0}, 0) = 53

After checking those folders, I saw it check "/usr/local/ssl/certs". Typically, that is the path where all installed CA certificates locate. But it is empty now. Instead, I found "/usr/lib/ssl/certs" is pointed to /etc/ssl/certs/ where I can find many CA certficates.

It looks like the root cause is openssl did not find the correct CA path. Finally, I thought that were caused by updating openssl.

Before updating openssl

$ openssl version -d
OPENSSLDIR: "/usr/lib/ssl"

After updating openssl

$ openssl version -d
OPENSSLDIR: "/usr/local/ssl"

Solution:

create softlink for /usr/local/ssl/certs to point to /etc/ssl/certs. Then the benchmark indicates 1000 iteration only takes 0.6 second.

Lession learn:

After updating openssl, make sure its certificate searching path is correct!