Fix test failures with OpenSSL 4.0. by junaruga · Pull Request #1019 · ruby/openssl

junaruga · 2026-03-25T20:34:32Z

This PR is based on the #1005, fixing the following test failures with OpenSSL master case. I don't intend to merge this PR. It's just show the result of the CI. I added the 5th commit on the #1005.

Use AES-256-CBC for OpenSSL to avoid errors on the error stack with OpenSSL 4.0. LibreSSL and AWS-LC don't support AES-256-CBC well, so use PBE-SHA1-3DES for them.

I haven't checked which commit of the openssl/openssl master branch caused the error.

This commit fixes the following test failures.
https://github.com/ruby/openssl/actions/runs/23093927984/job/67083113067?pr=1005

2) Failure: test_create_no_pass(OpenSSL::TestPKCS12)
/home/runner/work/openssl/openssl/test/openssl/utils.rb:140:in `teardown'
<[]> expected but was
<["error:0308010C:digital envelope routines::unsupported"]>

diff:
? ["error:0308010C:digital envelope routines::unsupported"]

However, I see other kind of test failure. I cannot reproduce this failure on my local, even when using the same commit of the OpenSSL openssl/openssl@5f084f8.

https://github.com/junaruga/ruby-openssl/actions/runs/23562223843/job/68604792678#step:11:590

1) Error: test_private_encoding_encrypted(OpenSSL::TestPKeyRSA): OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_pkey_rsa.rb:465:in `private_to_der'
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_pkey_rsa.rb:465:in `test_private_encoding_encrypted'
     462: 
     463:   def test_private_encoding_encrypted
     464:     rsa = Fixtures.pkey("rsa2048")
  => 465:     encoded = rsa.private_to_der("aes-128-cbc", "abcdef")
     466:     asn1 = OpenSSL::ASN1.decode(encoded) # PKCS #8 EncryptedPrivateKeyInfo
     467:     assert_kind_of OpenSSL::ASN1::Sequence, asn1
     468:     assert_equal 2, asn1.value.size
Error: OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_pkey_rsa.rb:465:in `private_to_der'
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_pkey_rsa.rb:465:in `test_private_encoding_encrypted'

OpenSSL's master branch is changing functions to return const pointers where the returned objects are not meant to be modified by the caller. Update ossl_*_new() to take const pointers accordingly. Unfortunately, *_dup() in older versions of OpenSSL and in LibreSSL/AWS-LC take non-const pointers, so const casts are required.

ossl_ec_new() was removed in commit 94aeab2 (pkey: simplify ossl_pkey_new(), 2017-03-16), but it forgot to remove the declaration while doing so.

ASN1_STRING has been made opaque in OpenSSL's master branch. Use the new accessor functions instead of accessing fields directly. Other uses of ASN1_STRING fields were already updated in <ruby#978>. This patch converts the remaining ones, which require the new functions added in OpenSSL 4.0 and were not available at that time.

OpenSSL master added support for RFC 7919 groups in TLS 1.2. They are preferred over SSLContext#tmp_dh= or #tmp_dh_callback= values if the client advertises them in the supported_groups extension.

junaruga · 2026-03-25T20:41:10Z

Here is the error.

https://github.com/ruby/openssl/actions/runs/23562768731/job/68606686661?pr=1019#step:11:590

1) Error: test_private_encoding_encrypted(OpenSSL::TestPKeyRSA): OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error
/home/runner/work/openssl/openssl/test/openssl/test_pkey_rsa.rb:465:in `private_to_der'
/home/runner/work/openssl/openssl/test/openssl/test_pkey_rsa.rb:465:in `test_private_encoding_encrypted'
     462: 
     463:   def test_private_encoding_encrypted
     464:     rsa = Fixtures.pkey("rsa2048")
  => 465:     encoded = rsa.private_to_der("aes-128-cbc", "abcdef")
     466:     asn1 = OpenSSL::ASN1.decode(encoded) # PKCS #8 EncryptedPrivateKeyInfo
     467:     assert_kind_of OpenSSL::ASN1::Sequence, asn1
     468:     assert_equal 2, asn1.value.size
Error: OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error
/home/runner/work/openssl/openssl/test/openssl/test_pkey_rsa.rb:465:in `private_to_der'
/home/runner/work/openssl/openssl/test/openssl/test_pkey_rsa.rb:465:in `test_private_encoding_encrypted'

junaruga · 2026-03-25T22:27:44Z

I got the following CI log when I ran without the 5th commit (my commit) now. Interestingly there is no error about Error: test_private_encoding_encrypted(OpenSSL::TestPKeyRSA): OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error in the log. That means the 5th commit modifying test/openssl/test_pkcs12.rb caused the test_private_encoding_encrypted error in the test/openssl/test_pkey_rsa.rb. It is weird. Do I need to clean something up in the teardown method in the test/openssl/test_pkcs12.rb?

https://github.com/junaruga/ruby-openssl/actions/runs/23566877802/job/68620449742

junaruga · 2026-03-26T13:55:48Z

There are differences between my local and GitHub Actions openssl-master environment. The differences may make the GitHub Actions specific test failure.

My local

OS: Fedora Linux 43

Ruby version

$ which ruby
~/.local/ruby-4.1.0dev-debug-dcfbbdc38c-openssl-4.0.0-dev-1cb0d36b39/bin/ruby

$ ruby -v
ruby 4.1.0dev (2026-01-08T09:48:38Z master dcfbbdc38c) +PRISM [x86_64-linux]

GitHub Actions openssl-master

OS: Ubuntu 24.04.4 LTS

Ruby version:

ruby 3.0.7p220 (2024-04-23 revision 724a071175) [x86_64-linux]

junaruga · 2026-03-26T13:59:42Z

Oh I just noticed the test_private_encoding_encrypted error in test/openssl/test_pkey_rsa.rb rake test_fips step. So, that means that rake test passed by my 5th commit, then CI moved to the next step rake test_fips.

https://github.com/ruby/openssl/actions/runs/23562768731/job/68606686661?pr=1019#step:11:600

junaruga · 2026-03-26T17:58:27Z

For the est_private_encoding_encrypted error in test/openssl/test_pkey_rsa.rb .

$ cat test.rb
require 'openssl'

OpenSSL.debug = true
rsa = OpenSSL::PKey.read(File.read('test/openssl/fixtures/pkey/rsa2048.pem'))
rsa.private_to_der('aes-128-cbc', 'abcdef')

$ OPENSSL_DIR=$HOME/.local/openssl-4.1.0-dev-fips-debug-5f084f8a99
$ OPENSSL_CONF=$OPENSSL_DIR/ssl/openssl_fips.cnf \
  ruby -I ./lib test.rb
...
test.rb:5: warning: error on stack: error:1C8000FE:Provider routines:kdf_pbkdf2_set_ctx_params:password strength too weak
test.rb:5: warning: error on stack: error:11800067:PKCS12 routines:PKCS12_item_i2d_encrypt_ex:encrypt error
test.rb:5: warning: error on stack: error:11800067:PKCS12 routines:PKCS8_set0_pbe_ex:encrypt error
test.rb:5:in 'OpenSSL::PKey::PKey#private_to_der': i2d_PKCS8PrivateKey_bio: encrypt error (OpenSSL::PKey::PKeyError)
OpenSSL error queue reported 3 errors:
error:1C8000FE:Provider routines:kdf_pbkdf2_set_ctx_params:password strength too weak
error:11800067:PKCS12 routines:PKCS12_item_i2d_encrypt_ex:encrypt error
error:11800067:PKCS12 routines:PKCS8_set0_pbe_ex:encrypt error
	from test.rb:5:in '<main>'

It seems the test failed because the length of the password (abcdef): 6 is less than KDF_PBKDF2_MIN_PASSWORD_LEN (= 8).

$ gdb --args ruby -I ./lib test.rb
(gdb) set env OPENSSL_CONF /home/jaruga/.local/openssl-4.1.0-dev-fips-debug-5f084f8a99/ssl/openssl_fips.cnf
(gdb) b kdf_pbkdf2_set_ctx_params
...
(gdb) f
#0  kdf_pbkdf2_set_ctx_params (vctx=0x8e0240, params=0x7fffffffae10)
    at providers/implementations/kdfs/pbkdf2.c:369
369	            ERR_raise(ERR_LIB_PROV, PROV_R_PASSWORD_STRENGTH_TOO_WEAK);
(gdb) l
364	    }
365	
366	    if (p.pw != NULL) {
367	        if (ctx->lower_bound_checks != 0
368	            && p.pw->data_size < KDF_PBKDF2_MIN_PASSWORD_LEN) {
369	            ERR_raise(ERR_LIB_PROV, PROV_R_PASSWORD_STRENGTH_TOO_WEAK);
370	            return 0;
371	        }
372	        if (!pbkdf2_set_membuf(&ctx->pass, &ctx->pass_len, p.pw))
373	            return 0;
(gdb) f
#0  kdf_pbkdf2_set_ctx_params (vctx=0x8e0240, params=0x7fffffffae10) at providers/implementations/kdfs/pbkdf2.c:369
369	            ERR_raise(ERR_LIB_PROV, PROV_R_PASSWORD_STRENGTH_TOO_WEAK);
(gdb) p KDF_PBKDF2_MIN_PASSWORD_LEN
$3 = 8
(gdb) p p.pw->data_size
$4 = 6

The stack trace is below.

(gdb) bt
#0  kdf_pbkdf2_set_ctx_params (vctx=0x8e0240, params=0x7fffffffae10) at providers/implementations/kdfs/pbkdf2.c:369
#1  0x00007fffcd22edb5 in kdf_pbkdf2_derive (vctx=0x8e0240, key=0x7fffffffaf90 "\320\300\215", keylen=16, params=0x7fffffffae10) at providers/implementations/kdfs/pbkdf2.c:312
#2  0x00007fffcd768066 in EVP_KDF_derive (ctx=0x8e0220, key=0x7fffffffaf90 "\320\300\215", keylen=16, params=0x7fffffffae10) at crypto/evp/kdf_lib.c:145
#3  0x00007fffcd7718f2 in ossl_pkcs5_pbkdf2_hmac_ex (pass=0x7fffffffb3d0 "abcdef", passlen=6, salt=0x8dc6b0 "4\020\005\232o\025\351\330\324_0\223\232Ir>", saltlen=16, iter=2048, digest=0x8ae4a0, keylen=16,
    out=0x7fffffffaf90 "\320\300\215", libctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0) at crypto/evp/p5_crpt2.c:61
#4  0x00007fffcd772235 in PKCS5_v2_PBKDF2_keyivgen_ex (ctx=0x85ff90, pass=0x7fffffffb3d0 "abcdef", passlen=6, param=0x8dc090, c=0x0, md=0x0, en_de=1, libctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0)
    at crypto/evp/p5_crpt2.c:245
#5  0x00007fffcd771e06 in PKCS5_v2_PBE_keyivgen_ex (ctx=0x85ff90, pass=0x7fffffffb3d0 "abcdef", passlen=6, param=0x8dc070, c=0x0, md=0x0, en_de=1, libctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0)
    at crypto/evp/p5_crpt2.c:163
#6  0x00007fffcd7632ac in EVP_PBE_CipherInit_ex (pbe_obj=0x7fffcdca8268 <nid_objs+6440>, pass=0x7fffffffb3d0 "abcdef", passlen=6, param=0x8dc070, ctx=0x85ff90, en_de=1, libctx=0x7fffcdd16ee0 <default_context_int>,
    propq=0x0) at crypto/evp/evp_pbe.c:144
#7  0x00007fffcd89b29d in PKCS12_pbe_crypt_ex (algor=0x8da460, pass=0x7fffffffb3d0 "abcdef", passlen=6, in=0x8dc1e0 "0\202\004\276\002\001", inlen=1218, data=0x8dabf8, datalen=0x8dabf0, en_de=1,
    libctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0) at crypto/pkcs12/p12_decr.c:39
#8  0x00007fffcd89ba22 in PKCS12_item_i2d_encrypt_ex (algor=0x8da460, it=0x7fffcdc9a780 <local_it>, pass=0x7fffffffb3d0 "abcdef", passlen=6, obj=0x8dafe0, zbuf=1, ctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0)
    at crypto/pkcs12/p12_decr.c:209
#9  0x00007fffcd89f997 in PKCS8_set0_pbe_ex (pass=0x7fffffffb3d0 "abcdef", passlen=6, p8inf=0x8dafe0, pbe=0x8da460, ctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0) at crypto/pkcs12/p12_p8e.c:76
#10 0x00007fffcd89f8da in PKCS8_encrypt_ex (pbe_nid=-1, cipher=0x8d0120, pass=0x7fffffffb3d0 "abcdef", passlen=6, salt=0x0, saltlen=0, iter=0, p8inf=0x8dafe0, libctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0)
    at crypto/pkcs12/p12_p8e.c:51
#11 0x00007fffcd9a7fc5 in p8info_to_encp8 (p8info=0x8dafe0, ctx=0x8cef20) at providers/implementations/encode_decode/encode_key2any.c:133
#12 0x00007fffcd9a8067 in key_to_encp8 (key=0x8adc60, key_nid=6, params=0x0, params_type=5, k2d=0x7fffcd9a98ce <rsa_prv_k2d>, ctx=0x8cef20) at providers/implementations/encode_decode/encode_key2any.c:150
#13 0x00007fffcd9a8224 in key_to_epki_der_priv_bio (out=0x8da610, key=0x8adc60, key_nid=6, pemname=0x7fffcdbdb571 "RSA PRIVATE KEY", p2s=0x7fffcd9a96c0 <prepare_rsa_params>, k2d=0x7fffcd9a98ce <rsa_prv_k2d>, ctx=0x8cef20)
    at providers/implementations/encode_decode/encode_key2any.c:217
#14 0x00007fffcd9a8397 in key_to_pki_der_priv_bio (out=0x8da610, key=0x8adc60, key_nid=6, pemname=0x7fffcdbdb571 "RSA PRIVATE KEY", p2s=0x7fffcd9a96c0 <prepare_rsa_params>, k2d=0x7fffcd9a98ce <rsa_prv_k2d>, ctx=0x8cef20)
    at providers/implementations/encode_decode/encode_key2any.c:266
#15 0x00007fffcd9a9f47 in key2any_encode (ctx=0x8cef20, cout=0x8da280, key=0x8adc60, type=6, pemname=0x7fffcdbdb571 "RSA PRIVATE KEY", checker=0x7fffcd9a9920 <rsa_check_key_type>,
    writer=0x7fffcd9a832b <key_to_pki_der_priv_bio>, pwcb=0x7fffcd7ad43d <ossl_pw_passphrase_callback_enc>, pwcbarg=0x8b31f8, key2paramstring=0x7fffcd9a96c0 <prepare_rsa_params>, key2der=0x7fffcd9a98ce <rsa_prv_k2d>)
    at providers/implementations/encode_decode/encode_key2any.c:1255
#16 0x00007fffcd9ab9ca in rsa_to_PrivateKeyInfo_der_encode (ctx=0x8cef20, cout=0x8da280, key=0x8adc60, key_abstract=0x0, selection=135, cb=0x7fffcd7ad43d <ossl_pw_passphrase_callback_enc>, cbarg=0x8b31f8)
    at providers/implementations/encode_decode/encode_key2any.c:1546
#17 0x00007fffcd7348e9 in encoder_process (data=0x7fffffffbcf0) at crypto/encode_decode/encoder_lib.c:711
#18 0x00007fffcd733203 in OSSL_ENCODER_to_bio (ctx=0x8b31c0, out=0x6f28a0) at crypto/encode_decode/encoder_lib.c:91
#19 0x00007fffcd893a36 in do_pk8pkey (bp=0x6f28a0, x=0x8b9900, isder=1, nid=-1, enc=0x7fffcdb87060 <aes_128_cbc>, kstr=0x0, klen=0, cb=0x7fffce019e5b <ossl_pem_passwd_cb>, u=0x7fffce1177b0, propq=0x0)
    at crypto/pem/pem_pk8.c:124
#20 0x00007fffcd893815 in i2d_PKCS8PrivateKey_bio (bp=0x6f28a0, x=0x8b9900, enc=0x7fffcdb87060 <aes_128_cbc>, kstr=0x0, klen=0, cb=0x7fffce019e5b <ossl_pem_passwd_cb>, u=0x7fffce1177b0) at crypto/pem/pem_pk8.c:59
#21 0x00007fffce036437 in do_pkcs8_export (argc=2, argv=0x7fffe92ff050, self=140736650639400, to_der=1) at ../../../../ext/openssl/ossl_pkey.c:871
#22 0x00007fffce036515 in ossl_pkey_private_to_der (argc=2, argv=0x7fffe92ff050, self=140736650639400) at ../../../../ext/openssl/ossl_pkey.c:900
...

junaruga · 2026-03-26T19:47:20Z

For the test failures in test/openssl/test_pkcs12.rb, I am debugging.

This is a reproducer.

$ ruby -I ./lib test/openssl/test_pkcs12.rb -n test_create_no_pass
Loaded suite test/openssl/test_pkcs12
Started
F
=====================================================================================================
Failure: test_create_no_pass(OpenSSL::TestPKCS12)
/home/jaruga/var/git/ruby/openssl/test/openssl/utils.rb:140:in 'OpenSSL::TestCase#teardown'
<[]> expected but was
<["error:0308010C:digital envelope routines::unsupported"]>

diff:
? ["error:0308010C:digital envelope routines::unsupported"]
=====================================================================================================
Finished in 0.083926823 seconds.
-----------------------------------------------------------------------------------------------------
1 tests, 5 assertions, 1 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
-----------------------------------------------------------------------------------------------------
11.92 tests/s, 59.58 assertions/s

The call (char*)ERR_error_string(ERR_peek_error(), 0) is calling the functions to check the value of the OpenSSL.errors.

$ gdb --args ruby -I ./lib test/openssl/test_pkcs12.rb -n test_create_no_pass
...
(gdb) call (char*)ERR_error_string(ERR_peek_error(), 0)
$28 = 0x7fffcdd16d40 <buf> "error:00000000:lib(0)::reason(0)"
(gdb) f
#0  inner_evp_generic_fetch (methdata=0x7fffffff5210, prov=0x0, operation_id=2,
    name=0x7fffcdb9f35a "PBE-SHA1-3DES", properties=0x0,
    new_method=0x7fffcd75d19a <evp_cipher_from_algorithm>,
    up_ref_method=0x7fffcd75d990 <evp_cipher_up_ref>, free_method=0x7fffcd75d9aa <evp_cipher_free>)
    at crypto/evp/evp_fetch.c:376
376	        ERR_raise_data(ERR_LIB_EVP, code,
(gdb) bt
#0  inner_evp_generic_fetch (methdata=0x7fffffff5210, prov=0x0, operation_id=2, name=0x7fffcdb9f35a "PBE-SHA1-3DES", properties=0x0, new_method=0x7fffcd75d19a <evp_cipher_from_algorithm>,
    up_ref_method=0x7fffcd75d990 <evp_cipher_up_ref>, free_method=0x7fffcd75d9aa <evp_cipher_free>) at crypto/evp/evp_fetch.c:376
#1  0x00007fffcd75e98c in evp_generic_fetch (libctx=0x0, operation_id=2, name=0x7fffcdb9f35a "PBE-SHA1-3DES", properties=0x0, new_method=0x7fffcd75d19a <evp_cipher_from_algorithm>,
    up_ref_method=0x7fffcd75d990 <evp_cipher_up_ref>, free_method=0x7fffcd75d9aa <evp_cipher_free>) at crypto/evp/evp_fetch.c:404
#2  0x00007fffcd75da12 in EVP_CIPHER_fetch (ctx=0x0, algorithm=0x7fffcdb9f35a "PBE-SHA1-3DES", properties=0x0) at crypto/evp/evp_enc.c:1521
#3  0x00007fffcd8991e4 in PKCS12_pack_p7encdata_ex (pbe_nid=146, pass=0x0, passlen=-1, salt=0x0, saltlen=0, iter=2048, bags=0xd9b930, ctx=0x0, propq=0x0) at crypto/pkcs12/p12_add.c:112
#4  0x00007fffcd89aed3 in PKCS12_add_safe_ex (psafes=0x7fffffff5440, bags=0xd9b930, nid_safe=146, iter=2048, pass=0x0, ctx=0x0, propq=0x0) at crypto/pkcs12/p12_crt.c:323
#5  0x00007fffcd89a6fc in PKCS12_create_ex2 (pass=0x0, name=0x7fffce09f408 "hello", pkey=0xda0680, cert=0xd98510, ca=0x0, nid_key=146, nid_cert=146, iter=2048, mac_iter=2048, keytype=0, ctx=0x0, propq=0x0, cb=0x0,
    cbarg=0x0) at crypto/pkcs12/p12_crt.c:111
#6  0x00007fffcd89aa87 in PKCS12_create_ex (pass=0x0, name=0x7fffce09f408 "hello", pkey=0xda0680, cert=0xd98510, ca=0x0, nid_key=146, nid_cert=146, iter=0, mac_iter=0, keytype=0, ctx=0x0, propq=0x0)
    at crypto/pkcs12/p12_crt.c:176
#7  0x00007fffcd89aae1 in PKCS12_create (pass=0x0, name=0x7fffce09f408 "hello", pkey=0xda0680, cert=0xd98510, ca=0x0, nid_key=146, nid_cert=146, iter=0, mac_iter=0, keytype=0) at crypto/pkcs12/p12_crt.c:185
#8  0x00007fffce0305ba in ossl_pkcs12_s_create (argc=7, argv=0x7fffe92ff830, self=140736647901600) at ../../../../ext/openssl/ossl_pkcs12.c:150
...
(gdb) n
371	    if ((name_id != 0 || name != NULL) && method == NULL) {
(gdb) call (char*)ERR_error_string(ERR_peek_error(), 0)
$29 = 0x7fffcdd16d40 <buf> "error:0308010C:digital envelope routines::unsupported"

junaruga · 2026-03-26T19:52:26Z

However, I couldn't find which commit caused the error in the PKCS12_create called in the OpenSSL::PKCS12.create.

junaruga · 2026-03-27T00:54:20Z

However, I couldn't find which commit caused the error in the PKCS12_create called in the OpenSSL::PKCS12.create.

I am trying to run git bisect to find which openssl/openssl commit caused this issue.

junaruga · 2026-03-27T01:57:07Z

I am trying to run git bisect to find which openssl/openssl commit caused this issue.

I found the first bad commit openssl/openssl@2ea6e78 caused the failure by git bisect.

Here are the command log.

Apply the following patch based on #1005 to fix a compile error with OpenSSL 4.0 master branch. There are openssl/openssl commits where the ASN1_BIT_STRING_get_length exists but ASN1_BIT_STRING_set1 doesn't exist yet.

diff --git a/ext/openssl/extconf.rb b/ext/openssl/extconf.rb
index 06ed4f6..163537e 100644
--- a/ext/openssl/extconf.rb
+++ b/ext/openssl/extconf.rb
@@ -171,6 +171,7 @@ have_func("SSL_get0_peer_signature_name(NULL, NULL)", ssl_h)

 # added in 4.0.0
 have_func("ASN1_BIT_STRING_set1(NULL, NULL, 0, 0)", "openssl/asn1.h")
+have_func("ASN1_BIT_STRING_get_length(NULL, 0, 0)", "openssl/asn1.h")

 Logging::message "=== Checking done. ===\n"

diff --git a/ext/openssl/openssl_missing.h b/ext/openssl/openssl_missing.h
index ed3b5b7..206ee37 100644
--- a/ext/openssl/openssl_missing.h
+++ b/ext/openssl/openssl_missing.h
@@ -41,7 +41,9 @@ ASN1_BIT_STRING_set1(ASN1_BIT_STRING *bitstr, const uint8_t *data,
     bitstr->flags |= ASN1_STRING_FLAG_BITS_LEFT | unused_bits;
     return 1;
 }
+#endif

+#ifndef HAVE_ASN1_BIT_STRING_GET_LENGTH
 static inline int
 ASN1_BIT_STRING_get_length(const ASN1_BIT_STRING *bitstr, size_t *length,
                            int *unused_bits)

Prepare the following script.

bisect.sh

#!/bin/bash

set -ex

OPENSSL_DIR="/home/jaruga/git/openssl2/dist"
RUBY_OPENSSL_DIR="/home/jaruga/git/ruby/openssl"

git clean -fdx

./Configure \
  --prefix="${OPENSSL_DIR}" \
  --libdir=lib \
  --debug \
  shared \
  enable-fips \
  enable-trace \
  '-Wl,-rpath,$(LIBRPATH)' \
  no-docs \
  -O0 -g3 -ggdb3 -gdwarf-5
make -j8
make install

sed -e "s|OPENSSL_DIR|$OPENSSL_DIR|" "${RUBY_OPENSSL_DIR}/tool/openssl_fips.cnf.tmpl" \
  > "${OPENSSL_DIR}/ssl/openssl_fips.cnf"
OPENSSL_CONF="$OPENSSL_DIR/ssl/openssl_fips.cnf" \
  "${OPENSSL_DIR}/bin/openssl" list -providers

pushd "${RUBY_OPENSSL_DIR}"

bundle exec rake clean
rm -rf tmp/ lib/openssl.so
MAKEFLAGS="V=1" \
  RUBY_OPENSSL_EXTCFLAGS="-O0 -g3 -ggdb3 -gdwarf-5" \
  bundle exec rake compile -- \
  --with-openssl-dir="${OPENSSL_DIR}"

# OPENSSL_CONF="$OPENSSL_DIR/ssl/openssl_fips.cnf" \
bundle exec rake debug
# OPENSSL_CONF="$OPENSSL_DIR/ssl/openssl_fips.cnf" \
bundle exec ruby -I ./lib test/openssl/test_pkcs12.rb -n test_create_no_pass

popd

$ pwd
/home/jaruga/var/git/openssl2

$ git bisect start
$ git bisect bad 74d47c8e66e02c1b6d97323ad4507292b02e9e5b
$ git bisect good d36cdcdce9be1f493d2f648fb63c14e1e29a35d2
$ git bisect run /path/to/bisect.sh

$ git bisect log
git bisect start
# status: waiting for both good and bad commits
# bad: [74d47c8e66e02c1b6d97323ad4507292b02e9e5b] Provide ASN1_BIT_STRING_set1()
git bisect bad 74d47c8e66e02c1b6d97323ad4507292b02e9e5b
# status: waiting for good commit(s), bad commit known
# good: [d36cdcdce9be1f493d2f648fb63c14e1e29a35d2] Fix const spec in ssl
git bisect good d36cdcdce9be1f493d2f648fb63c14e1e29a35d2
# good: [25a51c5cacbe7b1d1aadb3037a96bdfd09faaf45] add cmp_thunk function to ossl_bsearch
git bisect good 25a51c5cacbe7b1d1aadb3037a96bdfd09faaf45
# good: [0a99c3e9df51414a01188a2d18b20f744a93efa4] interop tests: update
git bisect good 0a99c3e9df51414a01188a2d18b20f744a93efa4
# good: [7b8f9b1c1c1944e8c748ce7160ffa5a3968d73e3] Update NOTES-VALGRIND.md to document valgrind compaints on reachable memory blocks
git bisect good 7b8f9b1c1c1944e8c748ce7160ffa5a3968d73e3
# good: [be42447469dcccc68b7e115c7947e3c25124f3f2] Fix the converters between the old and new BIO_read functions to handle end-of-file state properly.
git bisect good be42447469dcccc68b7e115c7947e3c25124f3f2
# bad: [2ea6e785f526f88f913cc6f49372aae9dc54bc63] EVP_get_digestbynid/EVP_get_cipherbynid turns into...
git bisect bad 2ea6e785f526f88f913cc6f49372aae9dc54bc63
# good: [039baf390e5e6a3ea8cbb564302a4d5cf2cec502] doc: fix RSA_set_method return value documentation
git bisect good 039baf390e5e6a3ea8cbb564302a4d5cf2cec502
# first bad commit: [2ea6e785f526f88f913cc6f49372aae9dc54bc63] EVP_get_digestbynid/EVP_get_cipherbynid turns into...

rhenium · 2026-03-27T07:07:37Z

Thanks for digging!

I am trying to run git bisect to find which openssl/openssl commit caused this issue.

I found the first bad commit openssl/openssl@2ea6e78 caused the failure by git bisect.

It appears to be leaking the error queue entry on a success path, which seems to me a simple oversight. Would you open an issue/send a patch to openssl/openssl?

I don't think we have to introduce a workaround for the PKCS12 issue for now.

junaruga · 2026-03-27T08:25:16Z

It appears to be leaking the error queue entry on a success path, which seems to me a simple oversight. Would you open an issue/send a patch to openssl/openssl?

I don't think we have to introduce a workaround for the PKCS12 issue for now.

All right. That makes sense. I will open an issue or send a patch to openssl/openssl, and will let you know here.

junaruga · 2026-03-27T21:31:23Z

All right. That makes sense. I will open an issue or send a patch to openssl/openssl, and will let you know here.

@rhenium I sent the patch openssl/openssl#30607 to openssl/openssl. I confirmed that the test_pkcs12.rb test passed with this patch.

junaruga · 2026-03-27T22:06:37Z

I rebased this PR changing my 5th commit. The 5th commit is a temporary workaround to fix this error. Hopefully my patch to openssl/openssl will be merged to the master branch and backported to the openssl-4.0 branch until OpenSSL 4.0.0 official release.

… and later This commit is a temporary workaround for an error message printed in a valid case OpenSSL::PKCS12.create with PBE-SHA1-3DES. This commit fixes the following test failures in test/openssl/test_pkcs12.rb in OpenSSL 4.0.0-alpha1 and later versions. https://github.com/ruby/openssl/actions/runs/23093927984/job/67083113067?pr=1005 ``` 2) Failure: test_create_no_pass(OpenSSL::TestPKCS12) /home/runner/work/openssl/openssl/test/openssl/utils.rb:140:in `teardown' <[]> expected but was <["error:0308010C:digital envelope routines::unsupported"]> diff: ? ["error:0308010C:digital envelope routines::unsupported"] ``` The following OpenSSL commit triggered the test failures. openssl/openssl@2ea6e78

…FIPS OpenSSL 4.0.0 added a check for Password-Based Key Derivation Function 2 (PBKDF2) to require the minimal password length 8 in FIPS by the following commit. openssl/openssl@71ed0fc This commit fixes the following test failure in OpenSSL 4.0 FIPS by changing testing password length from 6 to 8.. ``` 1) Error: test_private_encoding_encrypted(OpenSSL::TestPKeyRSA): OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error /home/runner/work/openssl/openssl/test/openssl/test_pkey_rsa.rb:465:in `private_to_der' /home/runner/work/openssl/openssl/test/openssl/test_pkey_rsa.rb:465:in `test_private_encoding_encrypted' 462: 463: def test_private_encoding_encrypted 464: rsa = Fixtures.pkey("rsa2048") => 465: encoded = rsa.private_to_der("aes-128-cbc", "abcdef") 466: asn1 = OpenSSL::ASN1.decode(encoded) # PKCS ruby#8 EncryptedPrivateKeyInfo 467: assert_kind_of OpenSSL::ASN1::Sequence, asn1 468: assert_equal 2, asn1.value.size Error: OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error ```

junaruga · 2026-03-29T15:18:35Z

I think we can merge the 5th commit of the ossl_pkcs12.c, a temporary workaround to fix the test failures to make CI pass, and we can revert the commit later after we confirm that the fix openssl/openssl#30607 is included in OpenSSL 4.0.0 official release. FOr now we are not sure if the fix will be backported to the openssl-4.0 branch before OpenSSL 4.0.0 official release.

rhenium · 2026-03-29T16:12:00Z

I'd prefer not to pollute our git blame history for an issue that only exists in their development branches. If the fix doesn't make it into the final release, I think a better course of action might be to just skip PKCS12 tests. However, I'm fairly optimistic it can be merged since the final release appears to be 2 weeks away.

I think other changes are good to merge now.

rhenium added 4 commits March 16, 2026 22:11

pkey: remove unnecessary prototype from ossl_pkey.h

faad7a0

ossl_ec_new() was removed in commit 94aeab2 (pkey: simplify ossl_pkey_new(), 2017-03-16), but it forgot to remove the declaration while doing so.

ssl: fix test_tmp_dh and test_tmp_dh_callback with OpenSSL 4.0

3e01c80

OpenSSL master added support for RFC 7919 groups in TLS 1.2. They are preferred over SSLContext#tmp_dh= or #tmp_dh_callback= values if the client advertises them in the supported_groups extension.

junaruga mentioned this pull request Mar 25, 2026

Improve OpenSSL 4.0 compatibility, part 1 #1005

Closed

junaruga changed the title ~~test_pkcs12.rb: Fix test failures with OpenSSL 4.0.~~ Fix test failures with OpenSSL 4.0. Mar 26, 2026

junaruga force-pushed the wip/openssl-4.0-compatibility branch from 528c991 to f909fd5 Compare March 26, 2026 18:38

junaruga marked this pull request as ready for review March 26, 2026 19:03

junaruga force-pushed the wip/openssl-4.0-compatibility branch from f909fd5 to 6e409d2 Compare March 27, 2026 02:02

junaruga force-pushed the wip/openssl-4.0-compatibility branch from 6e409d2 to 52a0321 Compare March 27, 2026 13:30

junaruga force-pushed the wip/openssl-4.0-compatibility branch from 52a0321 to 82911b1 Compare March 27, 2026 22:00

junaruga added 2 commits March 29, 2026 16:10

junaruga force-pushed the wip/openssl-4.0-compatibility branch from 82911b1 to ff13d3b Compare March 29, 2026 15:12

Conversation

junaruga commented Mar 25, 2026

Uh oh!

junaruga commented Mar 25, 2026

Uh oh!

junaruga commented Mar 25, 2026

Uh oh!

junaruga commented Mar 26, 2026

My local

GitHub Actions openssl-master

Uh oh!

junaruga commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

junaruga commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

junaruga commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

junaruga commented Mar 26, 2026

Uh oh!

junaruga commented Mar 27, 2026

Uh oh!

junaruga commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rhenium commented Mar 27, 2026

Uh oh!

junaruga commented Mar 27, 2026

Uh oh!

junaruga commented Mar 27, 2026

Uh oh!

junaruga commented Mar 27, 2026

Uh oh!

junaruga commented Mar 29, 2026

Uh oh!

rhenium commented Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

junaruga commented Mar 26, 2026 •

edited

Loading

junaruga commented Mar 26, 2026 •

edited

Loading

junaruga commented Mar 26, 2026 •

edited

Loading

junaruga commented Mar 27, 2026 •

edited

Loading

rhenium commented Mar 29, 2026 •

edited

Loading