Axway Gateway: Managing Security

SSL and TLS protocols

About SSL and TLS protocols

Using SSL or TLS with transfer protocols

TLS instance

Incoming connection

Outgoing connection

Related topics

Managing TLS

Using TLS

Managing TLS Security Profiles

Managing TLS Security Profiles (command line)

About SSL and TLS protocols

SSL (Secure Sockets Layer) was created by Netscape. Version 3.0 of SSL was used by the Internet Engineering Task Force (IETF) to define the TLS protocol (Transport Layer Security). These very similar protocols operate between network protocol layers (for example, TCP/IP... ) and application protocol layers (for example, HTTP). They both provide the following services:

SSL and TLS are based on four protocols:

  • Handshake Protocol: sets up the client and server session
  • Change Cipher Spec Protocol: establishes new agreement on the cipher suite used for the session
  • Alert Protocol: transmits warning and error messages between client and server
  • Record Protocol: comprises the above protocols and the application protocols

Protocol stack

When you use TLS or SSL, the protocol stack can be represented as follows:


Change Cipher Spec Protocol

Alert Protocol

Application Protocols
(FTP, ...)

Record Protocol

Network Protocols (TCP/IP ...)

The Record Protocol layer is responsible for:

  • Splitting data (maximum data block size is 16 KB)
  • Compressing data (optional)
  • Adding an integrity checksum (MAC)
  • Encrypting data

To establish an SSL or TLS connection, the connection procedure:

  • Sets up the record protocol layer: the layer is initialized with no integrity check and no encryption method (record layer state is then referred to as null with null null)
  • Executes the "handshake" protocol: client and server exchange their certificates and agree on an encryption method
  • Executes the "change cipher" protocol: following this exchange, the record layer is reconfigured using the security parameters exchanged during the handshake

To test the connection, a finish message is sent to test the established security parameters. The message contains a checksum of all the data sent since the beginning of the "handshake".

Payload splitting

Gateway supports payload splitting for the following TLS handshake messages:

  • CERTIFICATE, containing as payload Certificate Chains
  • CERTIFICATE REQUEST, containing as payload Authority DN lists

Gateway can thus handle messages of this type with payloads up to 64KB. The payload is split into several TLS handshake messages sent (or received) in a row, each carrying a fragment of the total payload not more than 16KB in size.

Note   Gateway does not support TLS Handshake messages longer than 16KB, that are not RFC-compliant. Be aware of the fact that current versions of multiple third party software such as openSSL, fileZilla, Mozilla and others, implement support for long (>16KB) Certificate Chains and Authority DN lists, by using handshake messages longer than 16KB in order to accommodate the entire size of the payload.

Using SSL or TLS with transfer protocols

Use of SSL or TLS is transparent for application protocols (for example, HTTPS, FTP, OFTP, and PeSIT). In most cases the SSL or TLS negotiation is triggered as soon as network connection is established.

Note: To secure a protocol with TLS or SSL you must use a different SAP (Service Access Point) than used for a connection in that protocol that is not secure.


FTP implements the use of TLS by introducing commands that start a TLS session on both the client and the server.

You can configure Gateway to use TLS in implicit and explicit mode for an FTP connection using the parameter ft_sec_option. In the implicit mode the session starts directly in a secured mode. In the explicit mode the session is an FTP session until the AUTH TLS-P command, which activates the SSL layer, is sent to the server.

An FTP session involves two connections:

  • Control connection
  • Data connection

Both connections must be secured.

You can establish the data connection in either of the two following modes:

  • Passive mode (the client initiates it)
  • Active mode (the server initiates it)

In passive mode, the data connection:

  • Uses TLS session caching
  • Reuses the security parameters negotiated for the control connection (there is a single handshake for both control and data connections)


SMTP implements the use of TLS by introducing commands that start a TLS session on both the client and the server.

SMTP servers are usually secured using TLS explicitly and most will not accept an implicit TLS connection. However an implicit TLS connection is possible with some SMTP servers. As well, SMTP servers for a proxy mailer may impose TLS in implicit mode.

You can configure Gateway to use TLS in implicit and explicit mode for an SMTP or POP3 connection using the parameter -ft_sec_option.

POP3 can be secured using implicit TLS if the server requires it. Since implicit TLS connections are established during the protocol connection, immediate TLS negotiation is imposed.

TLS/SSL and secure protocols

Some protocols, such as ETEBAC 5, are already secured. If you use SSL or TLS with those protocols, then the machine load is increased, but security is not increased. Recent protocols (or recent protocol extensions) implement the use of TLS by introducing commands that start a TLS session on both the client and the server (for example, SMTP, FTP). When you use SSL or TLS with FTP, all sessions (control and data) are encrypted, and, if possible, security parameters are reused to avoid a completely new handshake.

TLS instance

As described in SSL and TLS protocols, a TLS session involves two steps:

  • A handshake session, where the communicating partners exchange security parameters
  • The data transfer itself

You can configure cryptographic and authentication parameters via the TLS Profile object. Gateway selects the Profile to use for a TLS session depending on the way the connection between the partners is established (incoming or outgoing connection).

TLS handshake protocol initialization

You can impose the following handshake parameters via the TLS Security Profile configuration:

  • Negotiation/Renegotiation
  • Cache
  • Cipher suites
  • Protocol version

Incoming connection

An incoming connection in TLS has a minimum of two levels of verification. The first is identification of a corresponding Network Profile object and the second is the identification and check of the Remote Site object and its parameters.

Retrieving the Network Profile object (netprof)

For incoming connections, the network parameters are used to retrieve a Network Profile object. The Network Profile object indicates:

  • Whether to use TLS for this connection (Network security option set to TLS)
  • The name of the Security Profile to use

Checking the Remote Site

Following the TLS Handshake and the protocol connection, Gateway locates the Remote Site object involved in the current transfer.

Gateway then checks that the parameters negotiated during the TLS Handshake and the incoming Security Profile fields of the located Site (tls_sprof_in) match. If the tls_sprof_in parameter value is empty, no additional control is processed. If this control fails, the transfer is aborted.

Matching the Security Profile relies on:

  • Security Profile type (always server for incoming connections)
  • Exit scheduling value
  • Client and server authentication
  • Cipher suite used
  • Protocol used (TLSV1 or SSLV3)
  • User certificate sent
  • Remote certification chain received

The following diagram illustrates how Gateway implements security for incoming TLS connections:

Outgoing connection

The Remote Site configuration determines whether or not to secure an outgoing connection.

If you specified a Security Profile in the outgoing Security Profile field (tls_sprof_out) of the Site, then this Security Profile is used to secure the connection.

The following diagram schematically illustrates how Gateway implements security for outgoing TLS connections:

TLS Extensions

When connecting as a client, Gateway uses the host name of the remote partner as the value for the Server Name Indication extension, as specified by the RFC6606 from the IETF.

Gateway uses this value as is, as long as it doesn't resemble an IP address (IPV4 or IPV6), and as long as it is exclusively written with ASCII characters. Also, the RFC requires that the value sent with this extension must be the fully qualified domain name of the server.


  • In server mode, this extension is ignored.
  • This extension is sent only when Gateway attempts to negotiate version 1.2 of TLS.
  • The server_name extension is not added if TLS is done in Secure Relay (XSR termination for outgoing connections)

Related topics

Managing TLS


Related Links