Repository navigation
urllib3 v2.4.0 on Python 3.13 doesn't work with EKS #2394
Description
Activity
We've run into this issue as well. AWS support told us this is only the case with EKS clusters created with version 1.16 and below. Creating a new cluster would fix the issue, but updating does not.
As a workaround:
client_configuration = None if sys.version_info >= (3, 13): # https://docs.python.org/3/whatsnew/3.13.html#ssl client_configuration = Configuration() client_configuration.verify_ssl = False api_client = ApiClient(configuration=client_configuration) kubernetes.client.VersionApi(api_client)
Of course, it's insecure to completely disable SSL verification, but it does avoid the issue.
- added a commit that references this issue
on May 19, 2025 We ran into this as well, with an on-prem RKE 1.28 cluster.
The issue seems to be related to a change in behavior in the urllib3 library starting with 2.4.0: https://github.com/urllib3/urllib3/releases/tag/2.4.0
Added
verify_flagsoption tocreate_urllib3_contextwith a default ofVERIFY_X509_PARTIAL_CHAINandVERIFY_X509_STRICTfor Python 3.13+. (urllib3/urllib3#3571)This seems to have changed the default behavior of TLS connections made with urllib3, specifically on Python 3.13+ Workarounds include downgrading to Python 3.12, or downgrading urllib3 to 2.3.0.
I suspect the fix for this in the
kuberneteslibary is to update calls tocreate_urllib3_contextwithverify_flagsset to the previous default value.Reacted by YanickDelWe're also running into this issue with an EKS cluster that was created back in 2019. We need to keep the updates flowing, and it seems like the easiest solution for now is to pin urllib3 to v2.3.0, but that's not a viable long-term solution. We would also appreciate a way to configure urllib3 to use the old behaviour, without disabling TLS verification entirely.
could you please file a PR for the fix? Thanks
What do you see an acceptable fix as being?
I guess you either can specify urllib3 should be a version <2.4.0, or perhaps expose an easy way to disable this new sticker check in urllib3?
we can specify a version < 2.4.0 for urllib3, please add a brief comment about why that is added.
urllib3 only did this because Python 3.13 does it too. See What’s New In Python 3.13?:
ssl.create_default_context() sets ssl.VERIFY_X509_PARTIAL_CHAIN and ssl.VERIFY_X509_STRICT as default flags.
The actual fix is to ensure the certificate generated by Kubernetes is compliant and is not rejected when those two flags are set. Which is the case for Kubernetes 1.17 and later, apparently.
Reacted by Gaute SolemdalRecent versions of Python (3.13+) enforce stricter SSL certificate validation and require modern X.509 extensions for the certificate chain.
EKS clusters that were originally created on Kubernetes v1.16 or earlier have a root CA certificate without the Subject Key Identifier (SKID) and Authority Key Identifier (AKI) extensions. Most tools like curl, openssl, or kubectl work fine with such certificates, but Python 3.13+ (and recent urllib3/requests) will fail with a CERTIFICATE_VERIFY_FAILED/Missing Authority Key Identifier error, because they now require these fields to be present for secure chain validation.
If your cluster was created on Kubernetes v1.17 or newer, its CA certificate includes the necessary SKID and AKI, so Python 3.13+ clients are able to connect without any problems.
To summarize:
If your EKS cluster was created on Kubernetes v1.16 or lower, SSL connections from recent Python versions will fail due to missing X.509 extensions in the CA.
For clusters created on v1.17 or higher, everything works as expected.# Old EKS cluster (Kubernetes v1.16 or lower) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment, Certificate Sign X509v3 Basic Constraints: critical CA:TRUE // (Subject Key Identifier, Authority Key Identifier not present) # New EKS cluster (Kubernetes v1.17 or higher) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment, Certificate Sign X509v3 Basic Constraints: critical CA:TRUE X509v3 Subject Key Identifier: XX:XX:XX:... // (Authority Key Identifier often not present for root CA, but SKID is present)Reacted by Gaute Solemdal, siriu5b, Ben Gray, vish-ad and Calvin TaylorReacted by Aarni Koskela- added a commit that references this issue
on Jul 16, 2025 An additional problem is that there are two known vulnerabilities that affect any
urllib3versions lower than2.5.0, with apparently no released backports for2.3.x:Had the same issue.
For me, the "fix" was to use pyenv from https://github.com/pyenv/pyenv to download python 3.12 and try again.Everything worked after that.
EDIT: yes I was running this against an EKS kubernetes cluster provisioned in ~2019.
If you don't want to downgrade Python or
urllib, or disable TLS validation, you can use this workaround when creating the Kubernetes API client:api_client = ApiClient() # Ugly workaround to make the OpenSSL TLS validation laxer # to be compatible to old root CA certificates (created using k8s <= 1.16). # These are not compliant to openSSL's VERIFY_X509_STRICT that is used per default with Python 3.13. ctx = ssl.create_default_context() ctx.verify_flags = ctx.verify_flags & ~ssl.VERIFY_X509_STRICT api_client.rest_client.pool_manager = urllib3.PoolManager( num_pools=4, # default value from the kubernetes library that cannot be extracted programmatically. ssl_context=ctx, **api_client.rest_client.pool_manager.connection_pool_kw, )
Reacted by Greg Najda, Lucas Melchior, phnmn, Thomas PERROT, Cole Ratner, Oleksii Karpenko and Feliks Yaretskiy- added a commit that references this issue
on Nov 26, 2025 - added 2 commits that reference this issue
on Jan 14, 2026 For those looking for a workaround that doesn't disable SSL verification entirely or pin urllib3: you can use a
sitecustomize.pyto patchssl.SSLContext.verify_flagsat interpreter startup, clearing only theVERIFY_X509_STRICTflag that enforces AKI presence.# sitecustomize.py — place in your site-packages/ directory # Python auto-imports this at startup (https://docs.python.org/3/library/site.html#module-sitecustomize) import ssl as _ssl _real_set = _ssl.SSLContext.verify_flags.fset _real_get = _ssl.SSLContext.verify_flags.fget _mask = ~_ssl.VERIFY_X509_STRICT _ssl.SSLContext.verify_flags = property( lambda self: _real_get(self) & _mask, lambda self, val: _real_set(self, val & _mask), )
This keeps cert chain validation, hostname checking, and everything else intact — it only disables the strict X.509 extension check that OpenSSL 3.6 added to
VERIFY_X509_STRICT. Works with Python 3.13 + OpenSSL 3.6.2 + urllib3 2.4.0+ on EKS clusters created before K8s 1.16.Related: aws/containers-roadmap#2638 (request for EKS to reissue CA certs with AKI)
What happened (please include outputs or screenshots):
The following exception is raised whenever calling the Kubernetes API of an EKS cluster:
Full stacktrace
What you expected to happen:
The exception shouldn't be raised and the call to the Kubernetes API should be made successfully.
How to reproduce it (as minimally and precisely as possible):
Use the latest version of this project with urllib3 v2.4.0 on Python 3.13.
Anything else we need to know?:
This seems to be caused by the following change in urllib3 v2.4.0: issue, PR, which only takes effect on Python 3.13.
I've only experienced the issue with EKS, which must use self-signed certificates that aren't fully compatible with RFC 5280, notably because they don't provide an Authority Key Identifier.
I don't know if the same issue is the case of other Kubernetes providers.
Environment:
kubectl version): v1.32.3-eks-bcf3d70python --version): 3.13.3pip list | grep kubernetes): 32.0.1