Add PQC signature support

[olszomal]

Pull Request Type

• Bug fix
• New feature
• Code style / formatting / renaming
• Refactoring (no functional or API changes)
• Build / CI related changes
• Documentation
• Other (please describe):

Related Issue

Fixes #606

Current Behavior

The PKCS#11 provider supports traditional key types and EdDSA, but it does not handle post-quantum signature algorithms.

New Behavior

ML-DSA, SLH-DSA, and FALCON keys can be detected, generated, signed with, and verified where supported by the token and OpenSSL version.

Scope of Changes

• Added PKCS#11 provider support for ML-DSA, SLH-DSA, and FALCON signing and verification.
• Added PQC key pair generation.
• Added PQC examples and provider tests.
• Fixed PKCS#11 session handling and cleanup paths.
• Fixed attribute template cleanup to avoid leaks.
• Improved compatibility with newer OpenSSL APIs.
• Fixed MSVC compiler warnings.
• Updated documentation, build files, and tests.

Testing

• Existing tests
• New tests added
• Manual testing -> performed with Thales TCT Luna T-5000 Network HSM

Additional Notes

• OpenSSL 3.5 adds native support for the PQC algorithms ML-DSA and SLH-DSA.
• FALCON is not supported by OpenSSL, so public-key operations for FALCON are performed through the PKCS#11 token instead of relying on the OpenSSL provider implementation.
• FALCON and SLH-DSA support could not be tested because no token supporting these mechanisms was available during development.

License Declaration

• I hereby agree to license my contribution under the project's license.

[mtrojnar]

Thanks for PR. I found few issues worth fixing before merge:

1. ML-DSA signing fails with SoftHSM

   • src/p11_pkey.c:869
   • CKM_ML_DSA signing uses pParameter = NULL. SoftHSM ML-DSA expects CK_SIGN_ADDITIONAL_CONTEXT even for empty context.
   • New test fails:
     • make -C tests check TESTS='provider-mldsa87-keygen.softhsm'
     • failure: EVP_Digest_sign_verify_test() failed with err code: -6
   • pkcs11-spy shows C_Sign returns CKR_GENERAL_ERROR.
   • pkcs11-tool signs same key successfully because it passes CKH_HEDGE_PREFERRED, pContext = NULL, ulContextLen = 0.
   • Suggested fix: define/pass CK_SIGN_ADDITIONAL_CONTEXT for ML-DSA signing.

2. SLH-DSA mechanism constants are off by one

   • src/pkcs11.h:537
   • PR defines:
     • CKM_SLH_DSA_KEY_PAIR_GEN = 0x2E
     • CKM_SLH_DSA = 0x2F
   • PKCS#11 3.2 / p11-kit define:
     • CKM_SLH_DSA_KEY_PAIR_GEN = 0x2D
     • CKM_SLH_DSA = 0x2E
   • Compliant tokens will reject or run wrong mechanism. Please adjust constants.

3. Public API symbol declared but not exported

   • src/libp11.h:568
   • PKCS11_evp_pkey_verify() is declared public, but missing from src/libp11.exports.
   • External users would compile but fail linking. Either export it and review ABI/versioning, or keep it internal.

4. New ML-DSA DER fixtures missing from dist data

   • tests/Makefile.am:81
   • Added files mldsa87-cert.der, mldsa87-privkey.der, mldsa87-pubkey.der are not in dist_check_DATA.
   • make distcheck / release tarballs will miss them. Please add them.

Also git diff --check reports trailing whitespace:

• src/p11_pkey.c:221

[olszomal]

Thanks a lot for the review.

Regarding point 1, I checked the ML-DSA signing failure with SoftHSM2. This appears to be caused by the current SoftHSM2 master issue fixed by SoftHSMv2 PR softhsm/SoftHSMv2#881

That change explicitly uses "provider=default" when creating internal ML-DSA key contexts. With that fix applied, ML-DSA signing works correctly.

The same ML-DSA signing path also works with a Thales TCT Luna T-5000 Network HSM.

I fixed the remaining issues:

• corrected the SLH-DSA mechanism constant values,
• added PKCS11_evp_pkey_verify to libp11.exports,
• added the new ML-DSA DER fixtures to dist_check_DATA,
• fixed the trailing whitespace reported by git diff --check.