Why does SSL handshake give 'Could not generate DH keypair' exception?

ssl handshake wireshark
ssl handshake steps
ssl handshake wiki
ssl handshake protocol
ssl handshake diagram
ssl handshake client certificate
ssl handshake failed
ssl handshake cisco

When I make an SSL connection with some IRC servers (but not others - presumably due to the server's preferred encryption method) I get the following exception:

Caused by: java.lang.RuntimeException: Could not generate DH keypair
    at com.sun.net.ssl.internal.ssl.DHCrypt.<init>(DHCrypt.java:106)
    at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:556)
    at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:183)
    at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
    at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:893)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1165)
    ... 3 more

Final cause:

Caused by: java.security.InvalidAlgorithmParameterException: Prime size must be multiple of 64, and can only range from 512 to 1024 (inclusive)
    at com.sun.crypto.provider.DHKeyPairGenerator.initialize(DashoA13*..)
    at java.security.KeyPairGenerator$Delegate.initialize(KeyPairGenerator.java:627)
    at com.sun.net.ssl.internal.ssl.DHCrypt.<init>(DHCrypt.java:100)
    ... 10 more

An example of a server that demonstrates this problem is aperture.esper.net:6697 (this is an IRC server). An example of a server that does not demonstrate the problem is kornbluth.freenode.net:6697. [Not surprisingly, all servers on each network share the same respective behaviour.]

My code (which as noted does work when connecting to some SSL servers) is:

    SSLContext sslContext = SSLContext.getInstance("SSL");
    sslContext.init(null, trustAllCerts, new SecureRandom());
    s = (SSLSocket)sslContext.getSocketFactory().createSocket();
    s.connect(new InetSocketAddress(host, port), timeout);
    s.setSoTimeout(0);
    ((SSLSocket)s).startHandshake();

It's that last startHandshake that throws the exception. And yes there is some magic going on with the 'trustAllCerts'; that code forces the SSL system not to validate certs. (So... not a cert problem.)

Obviously one possibility is that esper's server is misconfigured, but I searched and didn't find any other references to people having problems with esper's SSL ports, and 'openssl' connects to it (see below). So I'm wondering if this is a limitation of Java default SSL support, or something. Any suggestions?

Here's what happens when I connect to aperture.esper.net 6697 using 'openssl' from commandline:

~ $ openssl s_client -connect aperture.esper.net:6697
CONNECTED(00000003)
depth=0 /C=GB/ST=England/L=London/O=EsperNet/OU=aperture.esper.net/CN=*.esper.net/emailAddress=support@esper.net
verify error:num=18:self signed certificate
verify return:1
depth=0 /C=GB/ST=England/L=London/O=EsperNet/OU=aperture.esper.net/CN=*.esper.net/emailAddress=support@esper.net
verify return:1
---
Certificate chain
 0 s:/C=GB/ST=England/L=London/O=EsperNet/OU=aperture.esper.net/CN=*.esper.net/emailAddress=support@esper.net
   i:/C=GB/ST=England/L=London/O=EsperNet/OU=aperture.esper.net/CN=*.esper.net/emailAddress=support@esper.net
---
Server certificate
-----BEGIN CERTIFICATE-----
[There was a certificate here, but I deleted it to save space]
-----END CERTIFICATE-----
subject=/C=GB/ST=England/L=London/O=EsperNet/OU=aperture.esper.net/CN=*.esper.net/emailAddress=support@esper.net
issuer=/C=GB/ST=England/L=London/O=EsperNet/OU=aperture.esper.net/CN=*.esper.net/emailAddress=support@esper.net
---
No client certificate CA names sent
---
SSL handshake has read 2178 bytes and written 468 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 51F1D40A1B044700365D3BD1C61ABC745FB0C347A334E1410946DCB5EFE37AFD
    Session-ID-ctx: 
    Master-Key: DF8194F6A60B073E049C87284856B5561476315145B55E35811028C4D97F77696F676DB019BB6E271E9965F289A99083
    Key-Arg   : None
    Start Time: 1311801833
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)
---

As noted, after all that, it does connect successfully which is more than you can say for my Java app.

Should it be relevant, I'm using OS X 10.6.8, Java version 1.6.0_26.

The problem is the prime size. The maximum-acceptable size that Java accepts is 1024 bits. This is a known issue (see JDK-6521495).

The bug report that I linked to mentions a workaround using BouncyCastle's JCE implementation. Hopefully that should work for you.

UPDATE

This was reported as bug JDK-7044060 and fixed recently.

Note, however, that the limit was only raised to 2048 bit. For sizes > 2048 bit, there is JDK-8072452 - Remove the maximum prime size of DH Keys; the fix appears to be for 9.

How Does an SSL Certificate Work? What is SSL Handshake?, Error The server doesn't support the cipher suite used by the client. SNI-enabled servers can't communicate with the client. The name on the certificate doesn't match with the hostname in the URL. Incomplete or invalid certificate chain. When I make an SSL connection with some IRC servers (but not others - presumably due to the server's preferred encryption method) I get the following exception: An example of a server that demonstrates this problem is aperture.esper.net:6697 (this is an IRC server). An example of a server that does not demonstrate the problem is kornbluth

How to Fix the SSL/TLS Handshake Failed Error?, ) might have been wrongly installed on the website server. Note that the SSL Handshake is invisible to the user and happens instantaneously. Essentially, three keys are used to set up the SSL connection: the public, private, and session keys. Anything encrypted with the public key can only be decrypted with the private key, and vice versa.

Here is my solution (java 1.6), also would be interested why I had to do this:

I noticed from the javax.security.debug=ssl, that sometimes the used cipher suite is TLS_DHE_... and sometime it is TLS_ECDHE_.... The later would happen if I added BouncyCastle. If TLS_ECDHE_ was selected, MOST OF the time it worked, but not ALWAYS, so adding even BouncyCastle provider was unreliable (failed with same error, every other time or so). I guess somewhere in the Sun SSL implementation sometimes it choose DHE, sometimes it choose ECDHE.

So the solution posted here relies on removing TLS_DHE_ ciphers completely. NOTE: BouncyCastle is NOT required for the solution.

So create the server certification file by:

echo |openssl s_client -connect example.org:443 2>&1 |sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'

Save this as it will be referenced later, than here is the solution for an SSL http get, excluding the TLS_DHE_ cipher suites.

package org.example.security;

import java.io.BufferedInputStream;
import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.InetAddress;
import java.net.Socket;
import java.net.URL;
import java.net.UnknownHostException;
import java.security.KeyStore;
import java.security.cert.Certificate;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.util.ArrayList;
import java.util.List;

import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLParameters;
import javax.net.ssl.SSLSocket;
import javax.net.ssl.SSLSocketFactory;
import javax.net.ssl.TrustManagerFactory;

import org.apache.log4j.Logger;

public class SSLExcludeCipherConnectionHelper {

    private Logger logger = Logger.getLogger(SSLExcludeCipherConnectionHelper.class);

    private String[] exludedCipherSuites = {"_DHE_","_DH_"};

    private String trustCert = null;

    private TrustManagerFactory tmf;

    public void setExludedCipherSuites(String[] exludedCipherSuites) {
        this.exludedCipherSuites = exludedCipherSuites;
    }

    public SSLExcludeCipherConnectionHelper(String trustCert) {
        super();
        this.trustCert = trustCert;
        //Security.addProvider(new BouncyCastleProvider());
        try {
            this.initTrustManager();
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }

    private void initTrustManager() throws Exception {
        CertificateFactory cf = CertificateFactory.getInstance("X.509");
        InputStream caInput = new BufferedInputStream(new FileInputStream(trustCert));
        Certificate ca = null;
        try {
            ca = cf.generateCertificate(caInput);
            logger.debug("ca=" + ((X509Certificate) ca).getSubjectDN());
        } finally {
            caInput.close();
        }

        // Create a KeyStore containing our trusted CAs
        KeyStore keyStore = KeyStore.getInstance("jks");
        keyStore.load(null, null);
        keyStore.setCertificateEntry("ca", ca);

        // Create a TrustManager that trusts the CAs in our KeyStore
        String tmfAlgorithm = TrustManagerFactory.getDefaultAlgorithm();
        tmf = TrustManagerFactory.getInstance(tmfAlgorithm);
        tmf.init(keyStore);
    }

    public String get(URL url) throws Exception {
        // Create an SSLContext that uses our TrustManager
        SSLContext context = SSLContext.getInstance("TLS");
        context.init(null, tmf.getTrustManagers(), null);
        SSLParameters params = context.getSupportedSSLParameters();
        List<String> enabledCiphers = new ArrayList<String>();
        for (String cipher : params.getCipherSuites()) {
            boolean exclude = false;
            if (exludedCipherSuites != null) {
                for (int i=0; i<exludedCipherSuites.length && !exclude; i++) {
                    exclude = cipher.indexOf(exludedCipherSuites[i]) >= 0;
                }
            }
            if (!exclude) {
                enabledCiphers.add(cipher);
            }
        }
        String[] cArray = new String[enabledCiphers.size()];
        enabledCiphers.toArray(cArray);

        // Tell the URLConnection to use a SocketFactory from our SSLContext
        HttpsURLConnection urlConnection =
            (HttpsURLConnection)url.openConnection();
        SSLSocketFactory sf = context.getSocketFactory();
        sf = new DOSSLSocketFactory(sf, cArray);
        urlConnection.setSSLSocketFactory(sf);
        BufferedReader in = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
        String inputLine;
        StringBuffer buffer = new StringBuffer();
        while ((inputLine = in.readLine()) != null) 
            buffer.append(inputLine);
        in.close();

        return buffer.toString();
    }

    private class DOSSLSocketFactory extends javax.net.ssl.SSLSocketFactory {

        private SSLSocketFactory sf = null;
        private String[] enabledCiphers = null;

        private DOSSLSocketFactory(SSLSocketFactory sf, String[] enabledCiphers) {
            super();
            this.sf = sf;
            this.enabledCiphers = enabledCiphers;
        }

        private Socket getSocketWithEnabledCiphers(Socket socket) {
            if (enabledCiphers != null && socket != null && socket instanceof SSLSocket)
                ((SSLSocket)socket).setEnabledCipherSuites(enabledCiphers);

            return socket;
        }

        @Override
        public Socket createSocket(Socket s, String host, int port,
                boolean autoClose) throws IOException {
            return getSocketWithEnabledCiphers(sf.createSocket(s, host, port, autoClose));
        }

        @Override
        public String[] getDefaultCipherSuites() {
            return sf.getDefaultCipherSuites();
        }

        @Override
        public String[] getSupportedCipherSuites() {
            if (enabledCiphers == null)
                return sf.getSupportedCipherSuites();
            else
                return enabledCiphers;
        }

        @Override
        public Socket createSocket(String host, int port) throws IOException,
                UnknownHostException {
            return getSocketWithEnabledCiphers(sf.createSocket(host, port));
        }

        @Override
        public Socket createSocket(InetAddress address, int port)
                throws IOException {
            return getSocketWithEnabledCiphers(sf.createSocket(address, port));
        }

        @Override
        public Socket createSocket(String host, int port, InetAddress localAddress,
                int localPort) throws IOException, UnknownHostException {
            return getSocketWithEnabledCiphers(sf.createSocket(host, port, localAddress, localPort));
        }

        @Override
        public Socket createSocket(InetAddress address, int port,
                InetAddress localaddress, int localport) throws IOException {
            return getSocketWithEnabledCiphers(sf.createSocket(address, port, localaddress, localport));
        }

    }
}

Finally here is how it is used (certFilePath if the path of the certificate saved from openssl):

try {
            URL url = new URL("https://www.example.org?q=somedata");            
            SSLExcludeCipherConnectionHelper sslExclHelper = new SSLExcludeCipherConnectionHelper(certFilePath);
            logger.debug(
                    sslExclHelper.get(url)
            );
        } catch (Exception ex) {
            ex.printStackTrace();
        }

SSL Connection Error On Android: How to Fix? [6 Working Steps], SSL/TLS are protocols used for encrypting information between two points. The following is a standard SSL handshake when RSA key exchange algorithm is used: 1. DigiCert SSL/TLS Certificates with Extended Validation (EV) provide  The SSL handshake builds a secure connection and customers or users of a website will have a safe environment for potential web transactions.

The answer above is correct, but in terms of the workaround, I had problems with the BouncyCastle implementation when I set it as preferred provider:

java.lang.ArrayIndexOutOfBoundsException: 64
    at com.sun.crypto.provider.TlsPrfGenerator.expand(DashoA13*..)

This is also discussed in one forum thread I found, which doesn't mention a solution. http://www.javakb.com/Uwe/Forum.aspx/java-programmer/47512/TLS-problems

I found an alternative solution which works for my case, although I'm not at all happy with it. The solution is to set it so that the Diffie-Hellman algorithm is not available at all. Then, supposing the server supports an alternative algorithm, it will be selecting during normal negotiation. Obviously the downside of this is that if somebody somehow manages to find a server that only supports Diffie-Hellman at 1024 bits or less then this actually means it will not work where it used to work before.

Here is code which works given an SSLSocket (before you connect it):

List<String> limited = new LinkedList<String>();
for(String suite : ((SSLSocket)s).getEnabledCipherSuites())
{
    if(!suite.contains("_DHE_"))
    {
        limited.add(suite);
    }
}
((SSLSocket)s).setEnabledCipherSuites(limited.toArray(
    new String[limited.size()]));

Nasty.

How does SSL work? What is an SSL handshake?, This section does not attempt to provide full details of the messages exchanged during In overview, the steps involved in the SSL handshake are as follows:. An example of a server that does not demonstrate the problem is kornbluth.freenode.net:6697. [Not surprisingly, all servers on each network share the same respective behaviour.] My code (which as noted does work when connecting to some SSL servers) is:

What Happens in a TLS Handshake? | SSL Handshake, The SSL handshake is initiated when your browser issues a secure connection request to a Web server. The server sends a public key to your computer, and  SSL starts to work after the TCP connection is established, initiating what is called an SSL handshake. 2 The server sends its certificate to the user along with a number of specifications (including which version of SSL/TLS and which encryption methods to use, etc.).

The TLS Handshake: Taking a closer look, A handshake is a process that enables the TLS/SSL client and server to establish a set of secret keys with which they can communicate. During  create SSL_CTX; create a new socket connection (e.g. I'm using libuv) create two memory BIOs: one is filled with data I receive from the server (readBio) the other one is used to in the application code to read from. (writeBio) create a SSL* and set state to SSL_connect_state; start the handshake process with SSL_do_handshake [loop] receive / send data

The SSL/TLS Handshake: an Overview, Learn what a TLS handshake is, the steps of a handshake, and potential when interacting in an SSL handshake with help from this post from Venafi. the server gives the client a Session Ticket that will help it remember it. 4. Make Contact. To ensure the right level of contact between your hand and your partner’s, keep your palm flat—not cupped—when you reach out to shake. Then, make contact diagonally. Try to wrap your fingers around your partner’s hand, scaling them one by one, as if you were giving a hug with your hand.

An overview of the SSL or TLS handshake, On an SSL encrypted website, the data transmission starts off with SSL/TLS handshake process. Understand what SSL/TLS handshake is and how it works. SSL Handshake Explained The reason it’s called a handshake is that it’s when two parties – client and server come across each other for the first time. The handshake involves a number of steps that start from validating the identity of the other party and concludes with the generation of a common key – secret key if you may call it.

Comments
  • The reason seems to be this: Prime size must be multiple of 64, and can only range from 512 to 1024 (inclusive). No idea what size was sent by the server here, and what the specification says about this.
  • @PaŭloEbermann: Actually you can see the size the server used in the openssl output in the question: "Cipher is DHE-RSA-AES256-SHA, Server public key is 2048 bit". And 2048 > 1024 :-).
  • @sleske: not exactly. Server public key (size) was, and is, the key in the cert. s_client in 2011 didn't show ephemeral key at all; 1.0.2 in 2015 and up does as Server Temp Key several lines higher. Although a good server usually should make DHE size same as RSA-auth size.
  • +1. Everyone who has a Sun Developer Network Account, please vote for this bug.
  • Thanks. Seems a pretty serious problem given the existence of servers which request a larger size! :( I tried BouncyCastle; if you set it up as preferred provider it crashes with a different exception (sigh), and I can't see an obvious way to use that just for DH. However, I found an alternative solution, which I'll add as a new answer. (It's not pretty.)
  • Cool. This has been fixed in newer versions of Java. But my question is about using older version.. When I use older version, sometimes it works and sometimes it gives above exception..Why so random behaviour? If its a bug in java, then I guess it should never work?
  • The 2048 fix was backported to IcedTea 2.5.3 . Newer versions of IcedTea increased it to 4096.
  • The problem is the DH prime size. The maximum-acceptable size that Java accepts is 2048 bits. While we wait for Oracle to expand the limit, you can compile with Excelsior Jet which has an expanded limit.
  • works for me! though not quite sure what I am doing.
  • security.provider.2=org.bouncycastle.jce.provider.BouncyCastleProvider did work better, putting it on 1. resulted in errors in default software.
  • This works for me too, but I have added a provider dynamically. Reffer my answer here for details.
  • Thanks a ton, this worked for me and was required in order to successfully build the Tomcat 7.0.78 source.
  • @mjj1409, in Java 1.5 we don't have $JAVA_HOME/lib/security path
  • Just tested your solution. It's working as intended. Thanks. Actually, just adding jdk.tls.disabledAlgorithms=DHE, ECDHE in JDK_HOME/jre/lib/security/java.security also works and avoid all this code.