Re: [squid-users] FATAL: The ssl_crtd helpers are crashing too rapidly, need help!

On Sep 11, 2012, at 5:40 AM, Alex Crow <alex_at_nanogherkin.com> wrote:

> Hi all, Amos.
>
> I've been running 3.2.1 for 2-3 of weeks in production. All was well for a couple of weeks, but over the last few days, approximately every 2 days we get people saying they have lost web access. This coincided with the above error message repeating and squid workers constantly restarting.
>
> This morning I had a look back in the logs and it seemingly started just after this point:
>
> 2012/09/11 10:13:32 kid1| WARNING: ssl_crtd #1 exited
> 2012/09/11 10:13:32 kid1| Too few ssl_crtd processes are running (need 1/32)
> 2012/09/11 10:13:32 kid1| Starting new helpers
> 2012/09/11 10:13:32 kid1| helperOpenServers: Starting 1/32 'ssl_crtd' processes
> 2012/09/11 10:13:32 kid1| client_side.cc(3477) sslCrtdHandleReply: "ssl_crtd" helper return <NULL> reply
> 2012/09/11 10:13:33 kid1| clientNegotiateSSL: Error negotiating SSL connection on FD 315: error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate (1/0)
> (ssl_crtd): Cannot create ssl certificate or private key.
> 2012/09/11 10:19:08 kid1| WARNING: ssl_crtd #1 exited
> 2012/09/11 10:19:08 kid1| Too few ssl_crtd processes are running (need 1/32)
> 2012/09/11 10:19:08 kid1| Starting new helpers
> 2012/09/11 10:19:08 kid1| helperOpenServers: Starting 1/32 'ssl_crtd' processes
> (ssl_crtd): Cannot create ssl certificate or private key.
> 2012/09/11 10:19:08 kid1| client_side.cc(3477) sslCrtdHandleReply: "ssl_crtd" helper return <NULL> reply
> 2012/09/11 10:19:08 kid1| WARNING: ssl_crtd #1 exited
> 2012/09/11 10:19:08 kid1| Too few ssl_crtd processes are running (need 1/32)
> 2012/09/11 10:19:08 kid1| Starting new helpers
> 2012/09/11 10:19:08 kid1| helperOpenServers: Starting 1/32 'ssl_crtd' processes
> 2012/09/11 10:19:09 kid1| client_side.cc(3477) sslCrtdHandleReply: "ssl_crtd" helper return <NULL> reply
> 2012/09/11 10:19:09 kid1| clientNegotiateSSL: Error negotiating SSL connection on FD 692: error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate (1/0)
> 2012/09/11 10:19:09 kid1| clientNegotiateSSL: Error negotiating SSL connection on FD 694: error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate (1/0)
> 2012/09/11 10:19:21 kid1| fwdNegotiateSSL: Error negotiating SSL connection on FD 936: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0)
> (ssl_crtd): Cannot create ssl certificate or private key.
> 2012/09/11 10:19:38 kid1| WARNING: ssl_crtd #1 exited
> 2012/09/11 10:19:38 kid1| Too few ssl_crtd processes are running (need 1/32)
> 2012/09/11 10:19:38 kid1| Starting new helpers
> ...
>
> From then on this seems to repeat until I do this:
>
> /etc/init.d/squid3 stop
> rm -rf /var/lib/ssl_db
> /usr/lib/squid3/ssl_crtd -c -s /var/lib/ssl_db
> chown -R proxy:proxy /var/lib/ssl_db
> /etc/init.d/squid3 start
>
> Sometimes I have to do this twice before squid settles down again and the ssl_crtd processes stop being reported as crashing.
>
> Any ideas?

I have occasionally seen a couple of different problems with the SSL certificate database. One is where invalid certificates are generated somehow, such as when the signing certificate is no longer valid, and another is where the size file is empty. I think the problem with the size file has been fixed in 3.3-head, but I'm not sure about 3.2.1.

This is an old patch that I used to help diagnose problems in the SSL certificate database. I have no idea if it will still apply to the sources, but maybe you can manually apply it and see if it helps track down the problem…

--- src/ssl/certificate_db.cc.ORIG 2012-01-31 14:07:49.000000000 -0600
+++ src/ssl/certificate_db.cc 2012-02-02 10:36:50.000000000 -0600
@@ -10,6 +10,9 @@
 #if HAVE_FSTREAM
 #include <fstream>
 #endif
+#if HAVE_IOSTREAM
+#include <iostream>
+#endif
 #if HAVE_STDEXCEPT
 #include <stdexcept>
 #endif
@@ -207,8 +210,16 @@
 {
     const Locker locker(dbLock, Here);
     load();
- if (!db || !cert || !pkey || min_db_size > max_db_size)
+ if (!db || !cert || !pkey)
+ {
+ std::cerr << "addCertAndPrivateKey: Error: db or cert or pkey not valid" << std::endl;
+ return false;
+ }
+ else if (min_db_size > max_db_size)
+ {
+ std::cerr << "addCertAndPrivateKey: Error: min_db_size " << min_db_size << " > max_db_size " << max_db_size << std::endl;
         return false;
+ }
     Row row;
     ASN1_INTEGER * ai = X509_get_serialNumber(cert.get());
     std::string serial_string;
@@ -220,7 +231,10 @@
     row.setValue(cnlSerial, serial_string.c_str());
     char ** rrow = TXT_DB_get_by_index(db.get(), cnlSerial, row.getRow());
     if (rrow != NULL)
+ {
+ std::cerr << "addCertAndPrivateKey: Error: rrow != NULL" << std::endl;
         return false;
+ }
 
     {
         TidyPointer<char, tidyFree> subject(X509_NAME_oneline(X509_get_subject_name(cert.get()), NULL, 0));
@@ -234,7 +248,10 @@
     }
 
     while (max_db_size < size()) {
- deleteOldestCertificate();
+ if (!deleteOldestCertificate()) {
+ std::cerr << "addCertAndPrivateKey: Warning: deleteOldestCertificate() failed" << std::endl;
+ break;
+ }
     }
 
     row.setValue(cnlType, "V");
@@ -249,12 +266,18 @@
     }
 
     if (!TXT_DB_insert(db.get(), row.getRow()))
+ {
+ std::cerr << "addCertAndPrivateKey: Error: TXT_DB_insert() failed" << std::endl;
         return false;
+ }
 
     row.reset();
     std::string filename(cert_full + "/" + serial_string + ".pem");
     if (!writeCertAndPrivateKeyToFile(cert, pkey, filename.c_str()))
+ {
+ std::cerr << "addCertAndPrivateKey: Error: writeCertAndPrivateKeyToFile() failed" << std::endl;
         return false;
+ }
     addSize(filename);
 
     save();
@@ -410,7 +433,7 @@
 
 size_t Ssl::CertificateDb::readSize() const
 {
- size_t db_size;
+ size_t db_size = 0;
     std::ifstream size_file(size_full.c_str());
     if (!size_file && enabled_disk_store)
         throw std::runtime_error("cannot read \"" + size_full + "\" file");
Received on Tue Sep 11 2012 - 13:15:36 MDT