Secure Socket Layer (SSL) also known as Transport Layer Security (TLS) is a cryptographic protocol which defines how 2 entities (client and server) communicate with each other securely. TLS is the successor of SSL. You can read more about it here: http://en.wikipedia.org/wiki/Transport_Layer_Security

These are the following protocols which are most commonly used:

  • SSL 2.0
  • SSL 3.0
  • TLS 1.0 (SSL 3.1)
  • TLS 1.1 (SSL 3.2)
  • TLS 1.2 (SSL 3.3)

SSL 2.0 had many security flaws which led to the development of its successor SSL 3.0. It is present only for backward compatibility. I have rarely seen anyone using this version and I would highly recommend against it.

As we know TLS/SSL is an application layer protocol. Below is a diagram depicting the TCP/IP model:

I am not going to discuss the SSL/TLS protocol in this post as it is beyond the scope of this topic. However I would be discussing SSL handshake in brief and relate it to IIS.

The above diagram makes it clear that TLS/SSL runs on top of TCP/IP like any other application layer protocol. Before we delve into SSL handshake we need to know something about TCP handshake too.

TCP/IP Handshake

Microsoft has published a support article explaining the 3-way TCP/IP handshake. Here is the link: http://support.microsoft.com/kb/172983

Below diagram should give you a gist of the TCP/IP handshake:

If we were to capture a network trace (or a TCP Dump) and look at the details available and analyze the details available; the IP Layer provides the TCP layer with IP Address of the client and server. The TCP layer contains the details about the source port and the destination port, TCP Flags and other details like checksum, Windows Size etc.

When the user launches a browser and punches in the web address, let’s say https://www.kaushalz.com, the client and the server would perform the TCP/IP handshake as seen below

So basically this is what is passed on from the TCP/IP layer to the application layer:

  • IP Address of the source and destination
  • Source Port and Destination Port

The host header is neither present in the IP or the TCP layer. This actually leads to a problem which was addressed via the introduction of SERVER NAME INDICATION (a TLS Extension).

Problem due to above Limitation

Before I describe the problem we need to understand a little about the server side bindings. When routing a HTTP request to a website the server determines which process the request to be routed based on the IP, PORT & the HOSTNAME. These 3 are always available to the server during a normal HTTP communication. So basically the combination of IP+PORT+HOSTNAME is used as a unique identity to route the site to a specific process. The server admin can have the same IP+PORT for all the HTTP websites and alter only the HOSTNAME and maintain the uniqueness throughout. Which also makes the server scalable.

However in case of SSL the server has access to IP & Port only. Since the HOSTNAME is not available, the server has to route the request to the process depending on IP+PORT. This limitation leaves the server handicapped, as it has to changes the design for websites running on HTTPS. Due to this, the uniqueness for the websites running on HTTPS is determined through combination of IP+PORT. In real world, having a separate IP for a website is not ideal due to hardware & monetary limitations. Also changing the port number for all SSL bindings may not be ideal as changing the port number to anything other than the default SSL port would require the client to specifically put out the port number in the request. As a result, the server is not scalable for HTTPS sites.

This was a protocol limitation and severely affected the scalability of the sites

This problem was addressed by introducing a TLS Extension called Server name Indication. The client sends the server the hostname it is requesting for as a part of the CLIENT HELLO in the form of TLS EXTENSIONS. You can read more about it here: RFC 3546 (Section 3.1)

TLS/SSL Handshake

Consider a scenario, where a client browser is accessing the site https://www.kaushalz.com

  • Client will try to resolve the hostname to an IPAddress via DNS.
  • Once the client has the Destination IP, it will send a TCP SYN to the server.
  • The Server responds with ACK to this SYN.
  • The client responds with an ACK to the ACK it received from the server. Now a TCP connection has been established between the client and the server. The client will now forward the requests to the Destination IP on port 443 (Default TLS/SSL port)
  • The control is now transferred to the SSL Protocol in the application layer. It has the IP & the Port information handy from previous steps. However, it still has no clue whatsoever about the hostname.
  • The client creates a TLS Packet called as CLIENT HELLO. This contains the following details:
    • SSL Protocol version
    • Session ID
    • List of Cipher Suites supported by the client.
    • List of CLIENT HELLO Extensions

The Client typically selects the most secure protocol version and sends it to the server. Below is a snippet from the RFC 3546:

Blake-Wilson, et. al.            Standards Track                       [Page 4]
RFC 3546                         TLS Extensions                       June 2003 </p>
2.1. Extended Client Hello </p>
Clients MAY request extended functionality from servers by sending the extended client hello message format in place of the client hello message format. The extended client hello message format is: </p>

struct {
         ProtocolVersion client_version;
         Random random;
         SessionID session_id;
         CipherSuite cipher_suites<2..2^16-1>;
         CompressionMethod compression_methods<1..2^8-1>;
         Extension client_hello_extension_list<0..2^16-1>;
} ClientHello;

Here the new “client_hello_extension_list” field contains a list of extensions. The actual “Extension” format is defined in Section 2.3. </p> In the event that a client requests additional functionality using the extended client hello, and this functionality is not supplied by the server, the client MAY abort the handshake. </p> Note that [TLS], Section 7.4.1.2, allows additional information to be added to the client hello message.  Thus the use of the extended client hello defined above should not “break” existing TLS 1.0 servers. </p>  A server that supports the extensions mechanism MUST accept only client hello messages in either the original or extended ClientHello format, and (as for all other messages) MUST check that the amount of data in the message precisely matches one of these formats; if not then it MUST send a fatal “decode_error” alert.  This overrides the “Forward compatibility note” in [TLS]. </p>
  • The client sends a CLIENT HELLO to the server on the IP & Port it obtained during TCP handshake.
  • For this scenario I will consider IIS 7.5 as the SERVER entity. Upon receiving the CLIENT HELLO, the server has access to the following information:
    • IP Address (10.168.3.213)
    • Port Number (443)
    • Protocol Version (TLS 1.0)
    • List of Cipher Suites
    • Session ID
    • List of CLIENT HELLO Extensions etc.

The Server will first check if it supports the above protocol version and if any of the cipher suites in the provided list. If not, the handshake fails there itself.

The Server will now try to determine if there is an end point listening on the IP and PORT. If it finds an endpoint and if it is IIS, then the TCPIP.SYS driver moves the packet to the HTTP.SYS layer.

  • HTTP.SYS moves the request into the generic SSL Queue.
  • Until IIS 7.5 the SSL bindings were IP based i.e., IP+ Port and were associated with a certificate hash.
  • The HTTP.SYS tries to determine the certificate has corresponding to this IP+Port combination. It does so by enumerating the following registry key: - HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters\SslBindingInfo
  • From the above, the certificate hash corresponding to the IP+PORT combination is determined. Now the HTTP.SYS calls the CRYPTO API’s by passing on the cert hash to retrieve certificate blob, which calls the - certificate store, finds the certificate and sends it back to the HTTP.SYS.
  • The Server responds to the client with SERVER HELLO. RFC 3546 defines the format of the SERVER HELLO:
Blake-Wilson, et. al.            Standards Track                       [Page 5]
RFC 3546                          TLS Extensions                      June 2003
2.2. Extended Server Hello The extended server hello message format MAY be sent in place of the server hello message when the client has requested extended functionality via the extended client hello message specified in Section 2.1.  The extended server hello message format is:
     struct {
         ProtocolVersion server_version;
         Random random;
         SessionID session_id;
         CipherSuite cipher_suite;
         CompressionMethod compression_method;
         Extension server_hello_extension_list<0..2^16-1>;
     } ServerHello;
Here the new “server_hello_extension_list” field contains a list of extensions.  The actual “Extension” format is defined in Section 2.3.
Note that the extended server hello message is only sent in response to an extended client hello message.  This prevents the possibility that the extended server hello message could “break” existing TLS 1.0 clients.
  • The Server typically responds back with the following details:
    • SSL/TLS Protocol version.
    • One of the cipher suites from the list of cipher suites provided by client. (whichever is the most secure)
    • Certificate of the server (Without the private key of course)
    • List of SERVER HELLO Extensions.
    • (OPTIONAL) If the web app associated with this binding requires a Client Certificate for authentication then it would request the client to send the certificate. Here the IIS Sever would send the client the distinguished names of the list of TRUSTED ROOT CA it supports.

Below is a snippet of the network trace:

Frame 320: 257 bytes on wire (2056 bits), 257 bytes captured (2056 bits) on interface 0
Ethernet II, Src: Cisco_e5:44:00 (10:bd:18:e5:44:00), Dst: WistronI_86:74:54 (3c:97:0e:86:74:54)
Internet Protocol Version 4, Src: 10.168.3.213 (10.168.3.213), Dst: 10.171.71.21 (10.171.71.21)
Transmission Control Protocol, Src Port: http (80), Dst Port: 42079 (42079), Seq: 1576, Ack: 359, Len: 203
[2 Reassembled TCP Segments (1663 bytes): #319(1460), #320(203)]
Secure Sockets Layer
    TLSv1 Record Layer: Handshake Protocol: Multiple Handshake Messages
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 1658
        Handshake Protocol: Server Hello
            Handshake Type: Server Hello (2)
            Length: 81
            Version: TLS 1.0 (0x0301)
            Random
            Session ID Length: 32
            Session ID: 8d0a0000efffe1ad6a82edc6d6a8967bd759cd0f3bdf70e9…
            Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)
            Compression Method: null (0)
            Extensions Length: 9
            Extension: renegotiation_info
            Extension: server_name
        Handshake Protocol: Certificate
            Handshake Type: Certificate (11)
            Length: 1565
            Certificates Length: 1562
             Certificates (1562 bytes)
                Certificate Length: 1559
                Certificate (id-at-commonName=www.kaushalz.com,id-at-organizationalUnitName=Azure,id-at-organizationName=Microsoft,id-at-localityName=Bangalore,id-at-stateOrProvinceName=India,id-at-countryName=IN)
                    signedCertificate
                    algorithmIdentifier (shaWithRSAEncryption)
                    Padding: 0
                    encrypted: bcd1c6d0a5e548eea94749e950d9ed8d7b73a79ac63306f0…
        Handshake Protocol: Server Hello Done
            Handshake Type: Server Hello Done (14)
            Length: 0
  • The Client uses the SERVER HELLO to perform SERVER AUTHENTICATION. This is described in detail here: http://support.microsoft.com/kb/257587.If the server cannot be authenticated, the user is warned and informed that an encrypted and authenticated connection cannot be established. If the server is successfully authenticated, the client proceeds to the next step.
NOTE: If you captured a network trace for a SSL Handshake you could see the details until SERVER HELLO, after that the encryption begins and nothing would be available and would make sense as the packets are encrypted.
  • The Client uses the data provided from the server to generate a pre-master secret for the session, **encrypts it with the server’s public key (obtained from the server’s certificate)**, and then sends the encrypted pre-master secret to the server. If the server had requested for CLIENT CERTIFICATE, then client also signs another piece of data that is unique to this handshake and known by both the client and server. In this case, the client sends both the signed data and the client’s own certificate to the server along with the encrypted pre-master secret.
  • If the server had requested for client authentication, the server attempts to authenticate the client. If the client cannot be authenticated, the session ends. If the client is successfully authenticated, the server uses its private key to decrypt the pre-master secret, and then performs a series of steps (which the client also performs, starting from the same pre-master secret) to generate the master secret.
  • Both the client and the server use the master secret to generate the session keys, which are symmetric keys used to encrypt and decrypt information exchanged during the SSL session and to verify its integrity (that is, to detect any changes in the data between the time it was sent and the time it is received over the SSL connection).
  • The CLIENT & the SERVER send each other a message informing that future messages from them will be encrypted with the session key. It then sends a separate (encrypted) message indicating that its portion of the handshake is finished.
  • The SSL Handshake is done. The Client and the Server send each other messages which are encrypted/decrypted using the session keys generated in the previous step.
  • It is now that the Client sends the actual HTTP Request packet to the Server in the encrypted form.
  • The Server decrypts the request via the symmetric key and generates a response, encrypts it and sends it back to the client.
  • This continues normally for the entire session of secure communication. However, at any time either the client or the server may renegotiate the connection. In this case the process repeats again.

Below is a diagrammatic representation of the SSL Handshake:

Identifying problems during SSL Handshake

Eventually, once the handshake completes and the data exchange has been done, either both or one of the entities will eventually close down the connection gracefully. If there was a problem during the SSL Handshake then you there would be an exception raised within the SSL Layer (SSL ALERT PROTOCOL). These exceptions may or may not be fatal i.e. not all exceptions would cause the handshake to fail.

As we know there we can see details only until SERVER HELLO. Anything beyond this point is not visible. However, in case of a SSL ALERT we would see some notification which can be viewed in the network traces.

Servers also tend to propagate this information through some sort of server logging. On Windows we have SCHANNEL logging which throws a corresponding SCHANNEL event in the SYSTEM event logs. Watch out for these events. Below is a snippet of one such event:

Log Name:      System
 Source:        Schannel
Date:          05-08-2013 20:16:02
Event ID:      36888
Task Category: None
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      My-Computer
Description:</p> The following fatal alert was generated: 40. The internal error state is 1205.

However, not all the alerts are fatal. Reproduce the issue and review the error message being logged.

You could also use WireShark/ Network Monitor/ Netsh to collect a network trace.

Hopefully this would give you some idea on how SSL handshake works. Let me know if you have any queries/suggestions.

MORE INFORMATION