Windows: cert already in hash table

[cshowers]

My env
Windows 7
chefdk 0.7
PowerShell 4
chef-sever-core 12.1.2

---

I've been testing the new chefdk policyfile by following the policyfile guided tour https://www.chef.io/blog/2015/08/18/policyfiles-a-guided-tour/#more-16518 Every time I run chef provision it failes with SSL_read errors

I created the special provision cookbook as per example and ran following
chef provision staging --sync -n aar-staging-01 -o port=8000
Uploading policy to policy group staging
[2015-08-30T22:00:25-04:00] ERROR: SSL Validation failure connecting to host: server-centos-65 - SSL_read: cert already in hash table
[2015-08-30T22:00:27-04:00] ERROR: SSL Validation failure connecting to host: server-centos-65 - SSL_write: cert already in hash table
Error: Failed to upload policy to policy group staging
Reason: (OpenSSL::SSL::SSLError) SSL_write: cert already in hash table

I setup chef server with ssl. I ran knife ssl fetch and knife ssl check is good

PS C:\Users\cshowers\git\chef-cookbooks\aar> knife ssl check
Connecting to host server-centos-65:443
Successfully verified certificates from `server-centos-65'

even if I run 'chef push staging' I get the same ssl errors. Am I missing something? It's almost like chef provision command doesn't use the same ssl as knife. I am pretty new to chef and I am trying to get my head around policfiles.

---

My knife.rb
current_dir = File.dirname(FILE)
log_level :info
log_location STDOUT
node_name "cshowers"
client_key "#{current_dir}/cshowers.pem"
chef_server_url "https://server-centos-65/organizations/test"
cookbook_path ["#{current_dir}/../cookbooks"]

[danielsdeleo]

Does it work if you give the -c CONFIG_FILE option? Trying to figure out if it's skipping the config file load if you don't pass the option explicitly.

[cshowers]

no it gives the same error. I tried a few tricks with config file. like relocate it to a different dir and ref it using -c. I did not add client.pem on purpose to see if it complains. And yes it complained about not having proper ssl. So I added pem and it resumes with cert already in hash.

[cshowers]

Output of using -c

PS C:\Users\cshowers\git\chef-cookbooks\aar> chef provision staging --sync -n aar-staging-01 -o port=8000 -c c:\chef\client.rb
Uploading policy to policy group staging
[2015-09-02T14:01:15-04:00] ERROR: SSL Validation failure connecting to host: server-centos-65 - SSL_read: cert already in hash table
[2015-09-02T14:01:16-04:00] ERROR: SSL Validation failure connecting to host: server-centos-65 - SSL_write: cert already in hash table
[2015-09-02T14:01:16-04:00] ERROR: SSL Validation failure connecting to host: server-centos-65 - SSL_write: cert already in hash table
Error: Failed to upload policy to policy group staging
Reason: (OpenSSL::SSL::SSLError) SSL_write: cert already in hash table

PS C:\Users\cshowers\git\chef-cookbooks\aar> knife ssl check -c c:\chef\client.rb
Connecting to host server-centos-65:443
Successfully verified certificates from `server-centos-65'

[danielsdeleo]

Thanks for that, I wanted to make sure this wasn't the same thing as a different bug where the config file simply wasn't getting loaded.

This seems to be an issue with Ruby itself, where when you have active threads and are using the SSL_CERT_FILE environment variable to configure OpenSSL (which we have to do on Windows), you can get the (OpenSSL::SSL::SSLError) SSL_write: cert already in hash table error. We are looking into ways we could fix this in the build.

The answer on this SO post might help, but ChefDK patches ruby to set the SSL_CERT_FILE variable (otherwise SSL cert verification wouldn't work in any circumstance), which might break the workaround. http://stackoverflow.com/questions/23263121/opensslx509storeerror-cert-already-in-hash-table

[cshowers]

ok I see. So since this problem only shows up on Windows I could test this out on linux until there is a chefDK release that fixes this.

[danielsdeleo]

Yes, we are looking at whether Ruby 2.2 would fix it, or if we could build Ruby on Windows ourselves rather than relying on RubyInstaller/DevKit builds (this would allow us to control the build-time configuration of OpenSSL so we wouldn't need to set SSL_CERT_FILE via a patch).

[SympaHannuPiki]

Any change on this yet? When trying to upload even a bit more complex cookbook into server, I will get the same error. Simple cookbooks are fine, but "windows" cookbook from supermarket isn't something it will digest very well. We are using valid certificate from Thawte and everything else work just fine, including "knife ssl check" is returning successful results.

Currently using ChefDK on Windows 8.1

Chef Development Kit Version: 0.10.0
chef-client version: 12.5.1
berks version: 4.0.1
kitchen version: 1.4.2

[danielsdeleo]

If you make a policyfile and upload it with chef push does that fail in the same way? Can you knife cookbook upload the same cookbook?

[danielsdeleo]

Also, what chef server are you using, and have you modified its SSL configuration?

[gitmlynch]

I'm having the same problem if I do a cookbook upload with --include-dependencies and the windows cookbook is one of the dependencies. This wasn't a problem until I upgraded from Chefdk 0.4.0 and knife-windows 0.8.5.

I was able to upload the windows cookbook by itself, so that got me past my immediate problem, but for automated deployments, that's not a useful work around.

Currently using ChefDK on Windows Server 2008 R2

Chef Development Kit Version: 0.9.0
chef-client version: 12.5.1
knife-windows: 1.1.1

[danielsdeleo]

Are you specifically seeing SSL_read: cert already in hash table ? If not, it may be a different issue.

[gitmlynch]

I am seeing SSL_read: cert already in hash table

[gitmlynch]

FWIW, I have found that using the --concurrency option, set to 1, in my 'knife cookbook upload' commands to be 100% successful at avoiding the SSL error problem described above.

[SympaHannuPiki]

@danielsdeleo, tested "knife cookbook upload" and problem seems to be the same. Chef server is 12.3.1. Regarding SSL configuration, I let Chef to generate self-signed certificate and then replaced (crt/key) files with new (valid) certificate... so no actual modifications are made for it. Most of the modifications for any configs are for Analytics.

When uploading depended cookbooks one by one, I finally managed to upload original "windows" cookbook.

PS D:\xxx\sandbox\cookbooks> knife cookbook upload windows
Uploading windows        [1.38.4]
ERROR: SSL Validation failure connecting to host: chef.rs-sympa.com - SSL_read: cert already in hash table
ERROR: SSL Validation failure connecting to host: chef.rs-sympa.com - SSL_read: cert already in hash table
ERROR: Could not establish a secure connection to the server.
Use `knife ssl check` to troubleshoot your SSL configuration.
If your Chef Server uses a self-signed certificate, you can use
`knife ssl fetch` to make knife trust the server's certificates.

Original Exception: OpenSSL::SSL::SSLError: SSL_read: cert already in hash table

[bradwilliamson]

I am also getting this error consistently . I try to the knife ssl check and fetch. Everything looks good. I get this randomly when using chef-client -o 'cookbook::recipe' -c C:\chef-repo.chef\knife.rb for provisioning virtual machines with the chef-provisioning-vsphere gem.

[bradwilliamson]

This issue is still present in the latest rub gems provided with chefdk 0.11.2

[bradwilliamson]

This issue is still persistent in chefdk 0.12.0. It seems to always occur if there is a syntax error in your provisioning recipe & you try to run it again after. If I reboot that sometimes clears it up. Doing a knife ssl fetch does not fix the issue. This occurs with machine_batch paralegal provisioning of servers.

[bradwilliamson]

I have started to use a Linux Chef Workstation as a work around when I provision more than one server at a time. The bug doesn't appear in the ruby gems provided with chefdk on linux.

[jdivine]

I am also seeing this when my Windows nodes converge - usually when I first provision them and they do the initial converge. Not on my workstation where I trigger the provisioning (although I have seen that too!) but on the nodes themselves when they converge with chef-client, as they synchronize the cookbooks from the server. It seems to happen about half of the time - frequently I just manually kick off the Chef Provisioning run again and the nodes will successfully converge.

Same issue with both chef-client 12.5.1 and 12.12.5

[2016-07-19T19:53:29-05:00] INFO: Storing updated cookbooks/java/recipes/ibm_tar.rb in the cache. [2016-07-19T19:53:29-05:00] ERROR: SSL Validation failure connecting to host: chef.spokvdev.com - SSL_read: cert already in hash table [2016-07-19T19:53:29-05:00] INFO: Storing updated cookbooks/java/recipes/oracle.rb in the cache.

                   [2016-07-19T19:53:31-05:00] ERROR: SSL Validation failure connecting to host: chef.spokvdev.com - SSL_read: cert already in hash table

                   ================================================================================
                   [2016-07-19T19:53:31-05:00] INFO: Storing updated cookbooks/apt/templates/default/01proxy.erb in the cache.
                   Error Syncing Cookbooks:
                   [2016-07-19T19:53:31-05:00] INFO: Storing updated cookbooks/apt/templates/default/10recommends.erb in the cache.
                   ================================================================================

                   Unexpected Error:
                   -----------------
                   OpenSSL::SSL::SSLError: SSL_read: cert already in hash table

                   [2016-07-19T19:53:31-05:00] INFO: Storing updated cookbooks/apt/templates/default/20auto-upgrades.erb in the cache.
                   [2016-07-19T19:53:31-05:00] ERROR: Running exception handlers
                   [2016-07-19T19:53:31-05:00] ERROR: Exception handlers complete```

[glennsarti]

Could this be a Chef manifestation of the similar issue in Puppet?

puppetlabs/puppet@35572a9

• The solution is to update the OpenSSL::X509::Store monkey path to
  pass a predicate to Rubys uniq function, to ensure that certs are
  compared by their DER byte representation. Without this change, the
  default uniq implementation uses the .hash function on the
  X509::Certificate object, which varies by object instance, not actual
  cert contents. Although the PEM representation of a cert could be
  used for comparison, DER generates smaller objects for comparison.

...

• Duplicate detection is necessary to perform in Puppet, because Rubys
  OpenSSL wrapper provides no ability to enumerate the store, or query
  if a cert has been added already. The code simply fails if dupes are
  added:

https://github.com/ruby/ruby/blob/ruby_2_1/ext/openssl/ossl_x509store.c#L303-L316

That code delegates to the OpenSSL function X509_STORE_add_crl, which
ultimately uses X509_OBJECT_retrieve_match to determine if a given
cert matches any in the store already:

https://github.com/openssl/openssl/blob/OpenSSL_1_0_2g/crypto/x509/x509_lu.c#L572-L597

[bradwilliamson]

This error still occurs as of latest Chef DK release. I get this error mainly when using machine_batch on a Windows Chef Workstation provisioning multiple Virtual Machines.

[irvingpop]

If I'm reading this Ruby issue correctly, it should be fixed in Ruby 2.4 which is used by ChefDK 2.x - can those impacted confirm?

cc: @thommay @trevorghess