Subscribe to our blog

A new vulnerability (CVE-2020-11501) has been discovered in the Datagram Transport Layer Security (DTLS) implementation in GnuTLS, where clients always send a fixed value (all-zero bytes) instead of random bytes in the first handshake message (ClientHello). The GnuTLS releases from 3.6.3 to 3.6.12 are affected by this vulnerability.

This vulnerability impacts Red Hat Enterprise Linux 8 and has been rated as having a Moderate impact by Red Hat Product Security. A fix for this issue has been delivered as part of RHSA-2020:1998, shipped on April 30, 2020.

This post investigates how this vulnerability affects real-world usage of the GnuTLS library. In summary, if DTLS is used with ephemeral Diffie-Hellman and RSA based key exchanges, the impact is considered to be limited; on the other hand, if it is used with plain Pre-Shared Key (PSK) exchange, applications are susceptible to repeated command execution and plaintext recovery.

We will also look at a similar issue found in another TLS library (Mozilla’s Network Security Services), and explore ways to prevent such flaws in the future.

1: TLS handshake and the use of randomness

In TLS and DTLS, a client and server first negotiate a set of security parameters necessary to start packet protection. This process is called handshake and can be illustrated as below:


Figure 1: DTLS 1.2 handshake

Figure 1: DTLS 1.2 handshake

The security parameters include keys used for encryption and for integrity checks with Message Authentication Code (MAC). In DTLS 1.2, a total of 4 keys – separate keys for encryption and MAC for each direction – are installed after a successful handshake.

To calculate those keys, both peers first share a premaster secret, which depends on the underlying key exchange algorithm. If ephemeral Diffie-Hellman (DHE) key exchange or its Elliptic Curve variant (ECDHE) is used, then a shared secret value obtained from the key agreement process is used. If RSA key exchange is used, then a random value generated by the client is used.

Afterwards, a master secret is derived from the premaster secret in conjunction with the random nonces exchanged between the client and server:

master_secret = PRF(pre_master_secret, "master secret",
                    ClientHello.random + ServerHello.random)

Where PRF is a pseudorandom function, that is based on HMAC in TLS 1.2. ClientHello.random and ServerHello.random are the random values included in the ClientHello and ServerHello handshake messages.

Finally, encryption and MAC keys are derived from the master secret, they also use the random nonces:

key_block = PRF(master_secret, "key expansion",
                ServerHello.random + ClientHello.random);

The purpose of those random nonces is to bind the master secret and the keys to a particular handshake. Even if the same premaster secret is reused in multiple handshakes, attackers cannot simply replay the handshake as long as each peer properly contributes randomness.

The GnuTLS flaw invalidates the client’s contribution of randomness: the value that is supposed to be random is instead set to be all-zeros. Before going into an analysis of the impact, let’s take a look at a similar issue found in another TLS library a couple of years ago.

2: The elder twin: all-zero ServerHello.random issue in NSS

In 2018, Mozilla’s Network Security Services (NSS) encountered a similar issue (CVE-2018-12384) in the code handling old style (SSLv2 compatible) ClientHello messages. This issue, unlike the GnuTLS issue, was on the server side and affected ServerHello.random.

Since an attacker can cause the server to process any ClientHello message, the issue was originally perceived as high severity. However, it turned out that in this case the handshake simply fails while verifying the final (Finished) message that also checks all handshake messages exchanged.

struct {
    opaque verify_data[verify_data_length];
} Finished;

   PRF(master_secret, finished_label, Hash(handshake_messages))

The latest analysis of this issue raises the following points:

  1. The flaw mentioned previously could allow the attacker to replicate handshake messages including the ClientHello. If a stream cipher (or block cipher with counter mode) is used, it may lead to bad failure modes. We will discuss this scenario later.

  2. With TLS False Start, the attacker can order the server to process application data before receiving the server's Finished message. The handshake will then nevertheless fail because of Finished verification as discussed above.

  3. If the client allows a weak hash such as MD5, and the server reuses the Diffie-Hellman parameters, it could be affected by a variant of the SLOTH attack.

As for the GnuTLS issue, (3) is not relevant because the default configuration disables weak hashes.

3: The practical impact of all-zero ClientHello.random

Going back to the all-zero ClientHello.random issue, it looks less severe than the ServerHello.random issue at first glance, because an attacker, posing as a legitimate server, cannot initiate a handshake by itself.

A major concern is whether it is possible to decrypt intercepted past conversations between legitimate parties. This is impractical because the client is still responsible for generating additional randomness as part of premaster secret calculation: when (EC)DHE is used, the client’s public key is derived from a random value generated on-the-fly, and when RSA is used, the premaster secret is a random value generated by the client.

With the default configuration, GnuTLS generates at least 32-byte long random values for (EC)DHE and 46-byte long values for RSA. Moreover, server parameters for (EC)DHE are digitally signed with the server’s key as a proof of origin to prevent impersonation, though the signature would be meaningless if neither ClientHello.random nor ServerHello.random contributed randomness.

To this point, we have only looked at the key exchange algorithms used with certificate authentication, though GnuTLS supports another set of key exchange algorithms: the PSK algorithms, which use symmetric keys, typically passwords, shared in advance. PSKs are particularly useful in virtual private network (VPN) applications, where explicit access control can be implemented based on user accounts on the server.

PSKs are categorized in three variants: plain PSK, (EC)DHE_PSK, and RSA_PSK. In terms of the premaster secret, the latter two algorithms share the same characteristics of the (EC)DHE and RSA key exchanges: they prefix the result of (EC)DH key agreement or random value to the PSK value, as other_secret:

struct {
    opaque other_secret<0..2^16-1>;
    opaque psk<0..2^16-1>;

On the other hand, other_secret is set to all-zero for plain PSK. If ClientHello.random is all-zero, the client will always send identical ClientHello messages and this allows a man-in-the-middle attacker to replay handshakes without having the server’s key.


Figure 2: Replicated handshake

Figure 2: Replicated handshake

Such replays not only allow the attacker to instruct the client to process the same application-level operation twice (S2 in Figure 2), but also enable key-reuse attack in counter mode ciphers such as AES-GCM because the same key and initialization vector (IV) are used. This allows plaintext recovery when the client sends application data after the handshake. In Figure 2, while C1 and C2 are both encrypted, the attacker can easily calculate the XOR of the original plaintext: C1 ⊕ C2 = (P1 ⊕ K1) ⊕ (P2 ⊕ K1) = P1 ⊕ P2, where K1 is the first key block, and P1 and P2 are the plaintexts corresponding to C1 and C2.

One of the popular VPN applications using the GnuTLS’s DTLS implementation is OpenConnect VPN, which makes use of plain PSK, although the above attack does not apply because it does not use static keys - they are derived from an initial TLS session. OpenConnect VPN 8.08 is already working around the issue by disabling DTLS on the affected versions of GnuTLS.

4: How TLS implementations could prevent similar issues in the future

Catching and preventing these flaws in the library is not trivial because DTLS and TLS peers occasionally need to resend unchanged handshake messages, including the ClientHello message. In GnuTLS the faulty code change was meant to exclude the case where the client is sending a ClientHello in response to a HelloRetryRequest:

  /* Generate random data 
- if (!IS_DTLS(session)
-     || session->internals.dtls.hsk_hello_verify_requests ==
-     0) {
+ if (!(session->internals.hsk_flags & HSK_HRR_RECEIVED) &&
+     !(IS_DTLS(session) && session->internals.dtls.hsk_hello_verify_requests == 0)) {
          ret = _gnutls_gen_client_random(session);

The NSS flaw was also caused by an attempt at code reorganization that increased code reuse with other protocol versions. The unit test coverage couldn’t catch relevant cases because the security parameters are set to all-zero at session initialization time.

One way to detect such flaws is to enable the Valgrind memory checker in a continuous integration (CI) environment. In the session initialization, the memory regions that require initialization at run-time can be explicitly marked as undefined with a client request:


The function generating a ClientHello.random value is supposed to unmark it with VALGRIND_MAKE_MEM_DEFINED. That way, when the library tries to send an uninitialized ClientHello.random, Valgrind reports it as an “uninitialised value” error. This measure is already applied in the GnuTLS master branch.

5: Conclusion

The analysis we conducted in this post leads us to the conclusion that this bug allows practical attacks only with non-common configurations where bare Pre-Shared Keys are used. Moreover it can be mitigated by transitioning to key exchanges that make use of (EC)DHE or RSA. Therefore we rated this vulnerability as being of moderate severity and we will provide a fix in a future scheduled update.

As seen in this and the NSS issue, there are common patterns of faulty code changes that result in weakening the security assumptions of transport protocols. While TLS and DTLS are resilient in typical use-cases, certain configurations can be vulnerable for a limited time. That underlines the importance of handling vulnerabilities transparently in upstream communities, and the need of an organizational practice that ensures that software stays up to date. It also highlights the importance of cryptographic agility and the ability to change the algorithm used via configuration options.

The author of this post would like to thank Pieter Hameete for reporting the issue upstream, the Eclipse Californium community for investigating it, and Stefan Bühler for providing the fix. We also appreciate the prior analysis of the related NSS issue coordinated by Eric Rescorla and Martin Thomson.

About the author

Daiki Ueno is a programmer in the crypto team at Red Hat, where he works on TLS libraries including NSS and GnuTLS.

Read full bio

Browse by channel

automation icon


The latest on IT automation that spans tech, teams, and environments

AI icon

Artificial intelligence

Explore the platforms and partners building a faster path for AI

open hybrid cloud icon

Open hybrid cloud

Explore how we build a more flexible future with hybrid cloud

security icon


Explore how we reduce risks across environments and technologies

edge icon

Edge computing

Updates on the solutions that simplify infrastructure at the edge

Infrastructure icon


Stay up to date on the world’s leading enterprise Linux platform

application development icon


The latest on our solutions to the toughest application challenges

Original series icon

Original shows

Entertaining stories from the makers and leaders in enterprise tech