EVP_PKEY_meth_add0 #645 issue reopen

[rudenkornk]

Intro

Hello :)
It looks like I faced the same issue which was mentioned in #645
(I cannot reopen it, so I am creating a new one. Hopefully this is OK).

My honest confession -- I do not know what is going on, except for these two things, which I know for sure:

1. I tried to setup my corporate laptop on NixOS to use tpm2-backed signed office Wi-Fi. (My company uses tpm2 chips to authenticate employees on corp network).
2. It did not work out of the box due to newer version of libp11 in nixpkgs. And the exact revision it stopped working is 60329e0 (this was verified manually by myself using git bisect).

Having this said, all the report next was produced by Fable 5. It looks plausible to me, at least there is a working reproduction: https://github.com/rudenkornk/libp11-645-repro

---

Current Behavior

After PKCS11_get_private_key() is called, RSA operations on unrelated software keys in the same process break.
Specifically, X509_sign() with a provider-backed software RSA key that was created before the PKCS#11 key was loaded fails with:

error:03000093:digital envelope routines:default_check:command not supported:crypto/evp/ctrl_params_translate.c:337

Additionally, software RSA keys generated after PKCS11_get_private_key() silently become legacy keys
(EVP_PKEY_get0_provider() returns NULL, where it returned the default provider before).
Those keys happen to still work, which masks the bug in tests that generate their software key after loading the PKCS#11 key —
this is most likely why the issue could not be reproduced in #645.

The regression was introduced in commit 60329e0 (first released in 0.4.18):
pkcs11_pkey_method_rsa_new() registers a custom RSA EVP_PKEY_METHOD process-globally via EVP_PKEY_meth_add0()
whenever an RSA private key is loaded.
On OpenSSL 3.x an application-registered EVP_PKEY_METHOD takes priority over the provider machinery for all RSA keys in the process,
and for pre-existing provider-backed keys the legacy/provider ctrl translation fails as above.

libp11 0.4.17 and earlier are not affected (the method was scoped to the engine via ENGINE_set_pkey_meths()).

Real-world impact: wpa_supplicant EAP-TLS with a TPM2 client key (tpm2-pkcs11 via the pkcs11 engine) stops connecting
after upgrading libp11 from 0.4.13 to 0.4.18, because the TLS stack performs software RSA operations
(server certificate chain processing) in the same process after the engine key is loaded.

Expected Behavior

PKCS11_get_private_key() should not affect RSA operations on software keys that have no PKCS#11 association.
Software keys should keep working and should keep being provider-backed, exactly as with libp11 <= 0.4.17.

Steps To Reproduce & Observed Output

Self-contained Docker reproducer (three files attached: Dockerfile, repro.c, run.sh).
It builds libp11 0.4.17 (last good) and 0.4.18 (first bad) from the official release tarballs on Debian 13,
initializes a SoftHSM token with one RSA key, and runs the identical program against both versions
(selected via LD_LIBRARY_PATH).
The program generates a software RSA key, runs X509_sign()+verify, RSA-PSS EVP_DigestSign()+verify and
RSA-OAEP encrypt+decrypt on software keys before and after PKCS11_get_private_key().

• Full command line used:

docker build -t libp11-645 .
docker run --rm libp11-645

• Full stdout/stderr output:

=============== libp11 0.4.17 ===============
OpenSSL 3.5.6 7 Apr 2026
default provider loaded: yes
[before] software key provider: default
[before] X509_sign+verify (PKCS#1 v1.5)           OK
[before] DigestSign+verify (RSA-PSS)              OK
[before] encrypt+decrypt (RSA-OAEP)               OK
[before, pre-existing key] software key provider: default
[before, pre-existing key] X509_sign+verify (PKCS#1 v1.5)           OK
[before, pre-existing key] DigestSign+verify (RSA-PSS)              OK
[before, pre-existing key] encrypt+decrypt (RSA-OAEP)               OK
PKCS11_get_private_key(): OK (key type RSA)
[after] software key provider: default
[after] X509_sign+verify (PKCS#1 v1.5)           OK
[after] DigestSign+verify (RSA-PSS)              OK
[after] encrypt+decrypt (RSA-OAEP)               OK
[after, pre-existing key] software key provider: default
[after, pre-existing key] X509_sign+verify (PKCS#1 v1.5)           OK
[after, pre-existing key] DigestSign+verify (RSA-PSS)              OK
[after, pre-existing key] encrypt+decrypt (RSA-OAEP)               OK

not reproduced: all software-key RSA operations still work

=============== libp11 0.4.18 ===============
OpenSSL 3.5.6 7 Apr 2026
default provider loaded: yes
[before] software key provider: default
[before] X509_sign+verify (PKCS#1 v1.5)           OK
[before] DigestSign+verify (RSA-PSS)              OK
[before] encrypt+decrypt (RSA-OAEP)               OK
[before, pre-existing key] software key provider: default
[before, pre-existing key] X509_sign+verify (PKCS#1 v1.5)           OK
[before, pre-existing key] DigestSign+verify (RSA-PSS)              OK
[before, pre-existing key] encrypt+decrypt (RSA-OAEP)               OK
PKCS11_get_private_key(): OK (key type RSA)
[after] software key provider: (legacy/no provider)
[after] X509_sign+verify (PKCS#1 v1.5)           OK
[after] DigestSign+verify (RSA-PSS)              OK
[after] encrypt+decrypt (RSA-OAEP)               OK
[after, pre-existing key] software key provider: default
[after, pre-existing key] X509_sign+verify (PKCS#1 v1.5)           FAILED
80173815FD710000:error:03000093:digital envelope routines:default_check:command not supported:../crypto/evp/ctrl_params_translate.c:337:
[after, pre-existing key] DigestSign+verify (RSA-PSS)              OK
[after, pre-existing key] encrypt+decrypt (RSA-OAEP)               OK

REPRODUCED: software-key RSA operations broke after PKCS11_get_private_key()

=============== summary ===============
libp11 0.4.17: all software-key RSA operations OK
libp11 0.4.18: REPRODUCED - X509_sign() broken after PKCS11_get_private_key()

PKCS#11 Spy log

The failing operation (X509_sign() with a pure software key) performs no PKCS#11 calls;
the token is only touched during setup and by PKCS11_get_private_key(), both of which succeed.
I can provide a PKCS#11 Spy trace of the full run on request.

Environment

• Operating system and version: Debian 13 (trixie) container; also reproduced on NixOS 26.05 with OpenSSL 3.6.3
• Architecture (x86_64, arm64, etc.): x86_64
• PKCS#11 module used: SoftHSM v2 (libsofthsm2.so), version 2.6.1 (Debian) / 2.7.0 (NixOS)
• Token / HSM type: SoftHSM software token; the regression was originally hit with tpm2-pkcs11 backed by a hardware TPM 2.0

Versions

• libp11 built from:
  • upstream master (commit 6b45984 — reproduced)
  • upstream release (tag): libp11-0.4.18 (first bad release; libp11-0.4.17 is the last good one), libp11-0.4.20 contains the same unguarded code path
  • distribution package (name and version): nixpkgs libp11 0.4.18 (also affected)
• PKCS#11 module and version: SoftHSM 2.6.1 (Debian trixie) and 2.7.0 (NixOS)
• openssl version -a (in the reproduction container):

OpenSSL 3.5.6 7 Apr 2026 (Library: OpenSSL 3.5.6 7 Apr 2026)
built on: Sat Jun  6 19:55:35 2026 UTC
platform: debian-amd64
options:  bn(64,64)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -Wa,--noexecstack -g -O2 -Werror=implicit-function-declaration -ffile-prefix-map=/build/reproducible-path/openssl-3.5.6=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DZLIB -DZSTD -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-3"
MODULESDIR: "/usr/lib/x86_64-linux-gnu/ossl-modules"
Seeding source: os-specific
CPUINFO: OPENSSL_ia32cap=0x7ffaf3bfffebffff:0x18c05fdef3bfa7eb:0x00000000fc100510:0x0000000000040000:0x0000000000000000

Configuration / Settings

None.
Stock distribution OpenSSL with no custom OPENSSL_CONF, no engine or provider configuration.
The reproducer uses the plain libp11 C API (PKCS11_CTX_load / PKCS11_get_private_key), no engine involved.
Only environment variables set: SOFTHSM2_CONF, PKCS11_MODULE, PKCS11_PIN.

Reproducibility

• Reproducible with a different PKCS#11 module: yes — the Docker reproducer uses SoftHSM;
  the original real-world failure was with tpm2-pkcs11 (wpa_supplicant EAP-TLS via the pkcs11 engine).
  The module is irrelevant to the mechanism: the global EVP_PKEY_meth_add0() registration happens on any RSA private key load.

Files / Artifacts

Attached: Dockerfile, repro.c, run.sh (self-contained, ~90 second build, no host dependencies besides Docker).

Anything else

• This is a working reproducer for EVP_PKEY_meth_add0() in p11_rsa.c globally overrides RSA signing, breaks X509_sign() with software keys (OpenSSL 3.x) #645, which was closed as not reproducible.
  The two ingredients the original attempts were missing:
  1. The software key must be created before PKCS11_get_private_key() and used after.
  2. Keys generated after the call silently downgrade to legacy keys and keep working, hiding the problem.
• Bisect over releases: 0.4.17 good, 0.4.18 bad; first bad commit 60329e0.
• The PKCS11_FLAG_NO_METHODS guard added after 0.4.18 is only set in the provider code path
  (provider_helpers.c), so the C API and engine paths still trigger the global registration on master.
• Workaround: pin libp11 to <= 0.4.17.

[rudenkornk]

@olszomal @mtrojnar may I kindly ask you to take a look on this once again? :)