About multi-node architecture

This topic describes the Transfer CFT multi-node feature, which provides you with horizontal scalability and high availability for fail overs.

Multi-node benefits include:

  • Horizontal scalability: increased transfer flow capacity
  • High availability: active/active including patch management on an active cluster, and automatic node restart after fail over
  • Traffic management: balancing load between nodes through an external load balancer or DNS

Prerequisites

Transfer CFT in multi-node architecture requires:

  • One key per node must have the cluster option (see key), and if there is more than one host you require at least one valid key per host
  • A shared file system for use of a multi-node architecture on several hosts (active/active). Additionally, the system must be configured prior to the multi-node installation and the shared disk ready when starting the Copilot server.
    • Windows only: You must map the shared disk to a drive letter. Windows UNC is not supported.
    • Windows only: The Copilot Service Mode cannot be started as the LocalSystem account.
    • Windows only: If you are running Copilot in Service Mode, you must set up a dependency with the shared disk's service for multi-node.

Platform and shared file system support

Transfer CFT supports the following OS for multi-node architecture: Windows-x86, Linux-x86, AIX-power, HPUX-parisc, HPUX-ia64, Sun-SPARC, Sun-x86, OpenVMS and z/OS.

IBM i supports a mono-host, multi-node architecture. For more information, refer to the IBM i Transfer CFT 3.2.2 Installation and Operations Guides.

Supported shared file systems for multi-node, multi-host architecture

The following non-exhaustive table lists shared file systems that have been tested with Transfer CFT.

Operating system Supported Unsupported
AIX GPFS (recommended), NFSv4 NFSv3, CXFS, VeritasSF
HP-UX NFSv4 NFSv3, CXFS, VeritasSF
Linux-x86 GPFS (recommended), NFSv4 NFSv3, CXFS, ACFS, OCFSv1, OCFSv2, QFS, VeritasSF
OpenVMS RMS  
Solaris NFSv4 NFSv3, CXFS, QFS, VeritasSF
Windows-x86 CIFS CXFS, NFS
z/OS Sharing DASD across Sysplex  

Architecture

The Transfer CFT multi-node architecture is based on hosts, nodes, and a shared file system.

Regardless of the number of servers hosting the nodes from outside the cluster, the nodes are viewed as a single machine on an internal network.

What is a host?

A host refers to a physical or virtual server located behind a load balancer, or other DNS mechanism. A server can host zero or multiple nodes. Transfer CFT itself does not manage the round robin or other load balancing mechanisms.

What is a node?

A node is a Transfer CFT runtime running on a host. Multiple nodes are called a Transfer CFT cluster, which can run on one or multiple hosts.

What is a shared file system?

A shared file system is a file system resource that is shared by several hosts.

General architectural overview

  • Transfer CFT provides a node manager per host that monitors every node and checks that its nodes are active. If a node goes down, the node manager detects the inactivity and takes over that node's activity.
  • In addition to the node manager, Transfer CFT provides a connection dispatcher that listens on the Transfer CFT server port, and monitors incoming traffic.
  • For multiple nodes to be able to access the same files, using the same set configuration, the system requires the use of a shared file system. The shared disk provides communication, configuration, partners, data flows, internal datafiles and nodes.
  • Incoming flow passes by a load balancer. Note that the exiting flow does not pass through an external load balancer.

Normal behavior

Host B goes down Host A takes over Node_1

About active/active

In a Transfer CFT active/active architecture, multi-node with multi-host functionality, services include:

  • Patch management for an active cluster to provide continuous availability
  • Scalability to overcome Transfer CFT transfer and session restrictions (for example, max. number of transfers per instance)
  • Automatic node restart by Transfer CFT after a fail over allowing a second host to take over if the first node fails

These services provide more than a balance of the total load among nodes, in active/active if one host goes down the other host recuperates the entire load.

About active/passive

In a Transfer CFT active/passive architecture services include:

  • Patch management for an active cluster to provide continuous availability
  • Ability to bring each node (instance of Transfer CFT) online if the primary host fails with complete redundancy

Architectural component details

Copilot

Copilot operates various services for multi-node management: the node manager, the connection dispatcher, and the synchronous communication media dispatcher. Only one Copilot runs per host.

Node manager

The node manager monitors all nodes that are part of the Transfer CFT multi-node environment (independent of the host). The monitoring mechanism is based on locks provided by the file system lock manager.

When a node is not running correctly, the node manager tries to start it locally.

Connection dispatcher

The connection dispatcher checks for inbound connections, and dispatches requests to each node.

On start up, the Transfer CFT nodes connect to the connection dispatcher service and register their TCP interface/port pairs. The service listens on the access points and passes incoming connections to the various Transfer CFT nodes.

The connection dispatcher service allow you to run multiple nodes, which listen at the same access points (interface/port pairs) on the same host.

You can set the dispatcher to use either a round-robin load balancing, or define a one-to-one relationship between a partner and a node. A one-to-one relationship ensures that for any given partner the transfers are kept in the correct chronological order.

Synchronous communication media dispatcher

This service listens for TCP connections that originate from synchronous media communication clients. It is responsible for balancing clients' transfer requests among the nodes, and redirecting the following commands, SWAITCAT, SEND TYPE=REPLY, or SEND TYPE=NACK, to the node that owns the corresponding transfer request.

Runtime files

All runtime data are stored on a shared file system.

The following internal datafiles are shared between nodes:

  • Parameter internal datafiles (CFTPARM)
  • Partners internal datafiles (CFTPART)
  • PKI base (CFTPKU)
  • Main communication media file (CFTCOM)
  • Unified configuration (UCONF)

The following internal datafiles are node specific, and the filename is flagged by the node identifier:

  • Catalog (cftcata00, cftcata01,...) located in <cft_runtime_dir>/data
  • Communication media file (cftcom00, cftcom01,...) located in <cft_runtime_dir>/data
  • Log files (cftlog00, cftlog01,...) located in <cft_runtime_dir>/log
  • Output file (cft00.out, cft01.out,...) located in <cft_runtime_dir>/run
  • Account file (cftaccnt00, cftaccnt01,...) located in <cft_runtime_dir>/accnt

Node recovery overview

There are two possibilities when a node manager detects a node failure:

  • If it is a local node fail over, the node manager automatically attempts to restart the node on the local server.
  • If it is a remote node fail over, the node manager waits for the remote node manager to restart the node. If the remote node manager does not restart the node  before the timeout, the local node manager will restart the node on the local server.

After the node is restarted, whether local or remote, it will complete all transfer requests that were active when the failure occurred.

  • After a fail over, manual intervention is required to rebalance the cluster.
  • When a node manager fails but nodes are still active, after a timeout nodes are stopped and restarted on another server. A manual intervention is required to restore the node manager.

Transfer recovery overview

When a node receives an incoming request - a transfer receive, restart, acknowledgement or negative acknowledgement - if the corresponding transfer record cannot be found in the node's own catalog, the node requests the transfer record from other nodes.

Possible scenarios include:

  • If another node has the catalog record, the node retrieves it and performs the transfer.
  • If no nodes have the record, an error is returned.
  • If any one of the nodes does not respond, the requesting node continues to retry all nodes until the session's timeout. Once the timeout is reached, the node ends the connection. After this, the remote partner retries the request according to its retry parameters.

In the case of node failure during the transfer recovery process, the catalog record is locked in both catalogs until both nodes are available for recovery.

Modify the installation for multi-node

Restrictions

The following multi-node restrictions apply:

  • Bandwidth control is calculated by node.
  • Accounting statistics are generated by node.
  • Nodes are not automatically re-balanced when a host is recovered after a failure.
  • Duplicate file detection is not supported.
  • When using COMS, the SEND/RECV commands does not support STATE=HOLD with WSTATE.
  • You cannot have more than one defined CFTCOM TYPE=FILE object.

When using Central Governance with Transfer CFT multi node, you cannot manage Transfer CFT updates. You are required to manually update the nodes.

Related topics

Related Links