Transfer CFT folder monitoring

This topic describes how to set up folder monitoring for Transfer CFT directories. When activated in Transfer CFT, folder monitoring periodically checks the status of files contained in a defined set of directories to see if there are transfer candidates. A set of configurable options allow you to define which files are monitored and the condition that can trigger an automatic SEND command for these files.

The following sections describe the concepts, parameters, and examples you can use to set up Transfer CFT folder monitoring:

About folder monitoring

See the Folder monitoring parameters table for a list of the parameters and values used to configure folder monitoring.

Checking the state of a monitored file

For each monitored file, Transfer CFT tracks the file size and the date of its last modification. These two pieces of information constitute the state of the file. If the state does not change within a certain delay in seconds, the file is considered to be available to submit for transfer. A delay of zero seconds indicates that the files are immediately ready for submission.

Folder monitoring directories

For each directory to be monitored by Transfer CFT, the scan_dir, there must be another directory for Transfer CFT to use to record the state information for each file. This second directory is called the work_dir. These two directories must be separate entities (the work_dir must differ from the scan_dir, and the work_dir cannot be a sub-directory of the scan_dir), and allow files to move from one directory to the other. Note that Transfer CFT does not create the scan_dir directory. However, the work_dir is automatically created in case of non existence.

Tracking files

To track files from the scan_dir that have been submitted, Transfer CFT can either:

  • Move (by renaming) these files to work_dir prior to submitting them, or
  • Leave these files in place and create another file with the same name in the work_dir prior to submitting the transfer

SEND parameters

In addition to the file path name, two other parameters are supplied to the SEND command, the IDF and PART names. You can set these as fixed parameter values, or extract them from the first or second sub-directory names. Transfer CFT then automatically creates the corresponding sub-directories, in the work_dir directory tree, as needed.

Prerequisites

This section lists the prerequisites and "dos and don'ts" for folder monitoring. Before starting to configure folder monitoring, read the following:

  • The new folder monitoring feature is managed through Transfer CFT uconf.
  • All of the folder monitoring uconf parameters are dynamic, meaning that after a modification you must run the command: CFTUTIL RECONFIG TYPE=UCONF
  • The folder monitoring option must be configured by a Transfer CFT user who has Transfer CFT Administrator rights, as it is managed through uconf.
  • The system user management feature (CFTSU) is not supported for Folder Monitoring configurations.
  • There are 2 options (FILE and MOVE methods); the MOVE method is recommended as explained in the sections below.
  • Do not define a scan_directory that is identical to a work_directory.
  • Do not define a work_directory as a subdirectory of a scan_directory.
  • When using the File method, you should never delete the .met files that are automatically created in the working directory.

Limitations

  • Directory scanning does not allow transferring a file whose name contains a wildcard character, for example the asterisk character (*) on UNIX-like platforms. Files containing wildcard characters in their file names are not be processed; they remain in the scanning directory, and an error message displays in the log. For a list of wildcard characters per platform, see Platform-specific characters and functions.
  • In IBM i and z/OS environments you can only use folder monitoring on UNIX file systems.
  • The WILDMAT parameter is available exclusively on Unix/Windows systems.

Configure folder monitoring

For each monitored directory you must provide a unique name to identify the set of configuration parameters corresponding to this directory. This enables you to individually configure each directory to be monitored.

Step overview

  1. Activate the folder monitoring option.
    • Set uconf parameter folder_monitoring.enable to Yes.
  2. Declare your logical directories to monitor.
    • Add to your uconf parameter folder_monitoring.folders 1 logical name by root directory you want to monitor.
  3. For each logical directory defined, configure the specific options you want to use for each:
    • File management method
    • Used sub-directories
    • Set the IDF
    • Set the partner name
    • Define the delay to take into account the file
    • Define other uconf Folder Monitoring parameters (described in the following sections)

Folder monitoring parameters

Use the following UCONF parameters to configure folder monitoring for each directory as needed. See the section UCONF if you are not familiar with unified configuration settings.

Parameter Type Default Description
folder_monitoring.enable Boolean No
  • No: No folder monitoring occurs.
  • Yes: Enable Transfer CFT folder monitoring.

See the examples below.

folder_monitoring.folders node None

Add the logical folders to monitor (list of logical identifiers).

You should provide a unique name to identify the set of configuration parameters corresponding to this directory. If you have more than one Folder to monitor, use a space between each logical value.

See the Comment*** below this table for additional information.

folder_monitoring.folders .<logical_name>.enable Boolean Yes Enables a scan of the folder.
folder_monitoring.folders .<logical_name>.scan_dir string None

Absolute path name of the top level directory to scan.

This directory must exist before restarting CFT.
folder_monitoring.folders .<logical_name>.work_dir string None

Absolute path name of the top level directory available for file state information.

  • If you are using the MOVE method, files that are ready to be submitted are available in the work_dir.
  • If you are using the FILE method, the .met files are stored in the work_dir.
Caution     Never delete any .met files.
folder_monitoring.folders .<logical_name>.enable_subdir Boolean Yes

Values:

  • Yes: The entire scan_dir sub-directory tree is monitored.
  • No: No scan is performed.
folder_monitoring.folders .<logical_name>.method enum MOVE

Values:

  • MOVE: Files are moved (by renaming), to the work_dir prior to being submitted.
  • FILE: Files are left in the scan_dir, and a state file with the same name is created in work_dir prior to submitting the file.

See also Defining the METHOD parameter.

folder_monitoring.folders .<logical_name>.file_idle_delay integer 5 If the state of a file has not changed within this delay in seconds, the file becomes a candidate for submission.
folder_monitoring.folders .<logical_name>.idf string ""

The IDF name to use in the SEND command. Use one of the following:

  • A fixed name.
  • "(0)": The name of the first directory sub-level is used.
  • "(1)": The name of the second directory sub-level is used.
folder_monitoring.folders .<logical_name>.part string ""

The PART name to use in the SEND command. Use one of the following:

  • A fixed name.
  • "(0)": The name of the first directory sub-level is used.
  • "(1)": The name of the second directory sub-level is used.
folder_monitoring.folders .<logical_name>.interval int 60 The interval between two scans of the directory files in seconds.
folder_monitoring.folders .<logical_name>.file_count int -1 Maximum number of file submissions for each scan.
folder_monitoring.folders .<logical_name>.file_size_min int -1 Files shorter than this value, in bytes, are not candidates for submission.
folder_monitoring.folders .<logical_name>.file_size_max int -1 Files larger than this value, in bytes, are not candidates for submission.
folder_monitoring.folders .<logical_name>.file_include_filter string "" If this parameter is defined, only files whose names match this pattern are monitored.
folder_monitoring.folders.<logical_name>.file_exclude_filter string "" If this parameter is defined, files whose names match this pattern are not monitored.
folder_monitoring.folders .<logical_name>.resubmit_changed_file Boolean Yes

This parameter has no effect when the configured method is MOVE.

When the method parameter value is set to FILE:

  • Yes: When the state of a previously submitted file is seen as having changed, the file is submitted again.
  • No: Files are not resubmitted, regardless of changes.
Note The file is resubmitted after any change regardless of if the modification is a small change, or purging and replacing the file with another file having the same name.
folder_monitoring.folders .<logical_name>.filter_type enum WILDMAT

Defines the pattern matching algorithm to use for file name filtering. Values:

  • STRJCMP: The Transfer CFT pattern matching algorithm.
  • WILDMAT: A well known public domain algorithm, and is the default. Unix/Windows only

See Create inclusion and exclusion filters for details.

folder_monitoring.folders .<logical_name>.renaming_method Enum TIMESTAMP

This parameter applies only to the MOVE method. When set to TIMESTAMP, a timestamp of the pattern YYYYMMDDHHMMSS is added at the end of the name of the renamed file but before the last '.'.

For example, using timestamp_separators=".":
  • myfile is renamed myfile.20131025
  • myfile.txt is renamed myfile.20131025.txt
folder_monitoring.folders .<logical_name>.renaming_separators string  

This parameter only applies to the MOVE method. It must contain at most 2 characters from among the following:

.[]()i_-

The first character defines the separator before the timestamp. The second one, when present, defines the separator after the timestamp.

For example, using timestamp_separators "[]": - myfile is renamed myfile.[20131025] - myfile.txt is renamed myfile.[20131025].txt

folder_monitoring.folders .<logical_name>.control string   Metadata used to control user changes.

Comment***: You can use either Copilot or CFTUTIL to create the list of folders <logical_names>. When using CFTUTIL, be careful to correctly enter the command. For example, where FM1, FM2 and FM3 are 3 logical folders to be managed by Transfer CFT, enter:

CFTUTIL uconfset id= folder_monitoring.folders, value=”’FM1 FM2 FM3’”

How Transfer CFT handles monitored files

This section describes how the various file monitoring parameters work.

Parameter settings and actions

  • The delay between scans of a given directory is defined by its interval parameter value.
  • By default the enable_subdir parameter is set to YES, and the directory and all its sub-directories are scanned.
  • For each file detected, the name is checked against the configured parameters values in the include and exclude file filters. Files that match the combined criteria are monitored, all others are ignored.

For a file to become a candidate to be submitted, the following conditions must be met:

  • File size: If these values are configured, the following rules apply.
    • file_size_min: The current size must not be less than this value.
    • file_size_max: The current size must not be greater than this value.
  • The last modification time and duration must not have changed within a number of seconds as defined in the file_idle_delay parameter value.

Defining the METHOD parameter

Submitting the SEND command occurs differently depending on if the METHOD parameter value is set to MOVE or FILE.

MOVE option

If the METHOD parameter is set to MOVE:

  1. The file is moved (by renaming) to the work_dir in the same relative directory position as its original position in the scan_dir. By default an optional timestamp is added to the name of the moved file (see configuration parameters above).
  2. A command using the following pattern is submitted internally:

CFTUTIL SEND=<part>, IDF=<idf>, FNAME=<pathname>

where:

  • <part> is the partner name as indicated in the configuration.
  • <idf> is the idf name as indicated in the configuration.
  • <pathname> is the absolute pathname of the file in the work_dir directory.

Example

The following example demonstrates how the PART name is used for the first directory sub-level, and the IDF name for the second level sub-directory.

Original file:     /dir_c/scan/newyork/idf1/my_file.txt

Renamed file: /dir_c/work/newyork/idf1/my_file.20131025.txt

Command

CFTUTIL SEND part=newyork, idf=idf1, fname=/dir_c/work/newyork/idf1/my_file.txt

FILE option

When the METHOD parameter is set to FILE, the original file is not renamed. However, to prevent the file from being sent again when Transfer CFT is restarted, and to keep track of persistent information about the file, a new file is created in the work_dir directory. This new file contains a certain amount of metadata about the file that Transfer CFT needs.

Submitting the SEND command occurs as follows:

  1. A new file with the same name as the original file, but suffixed with .met, is created in the directory work_dir in the same relative position as the original file. Metadata about the file needed by Transfer CFT are written to this .met file.

    NOTE: These .met files must not be deleted, as they are internal Transfer CFT files. To delete the .met files, you must delete the associated files in the SCAN directory (Transfer CFT must be running).

  2. The same CFTUTIL command as used in the MOVE method is submitted internally, but with an fname corresponding to the original file in the scan_dir.

Example

The following example uses the partner name in the first directory sub-level and the IDF name in the second.

Original file:    /dir_c/scan/newyork/idf1/my_file.txt

Metadata file: /dir_c/work/newyork/idf1/my_file.txt.met

Command

CFTUTIL SEND part=newyork, idf=idf1, fname=/dir_c/scan/newyork/idf1/my_file.txt

Although both methods provide the same level of functionality, it is recommended that you use the MOVE method (renaming file)  when the renaming of the file prior to its submission is acceptable.

The FILE  method has some drawbacks that the MOVE method does not, especially related to performance. FILE method limitations include:

  • For each file to send, a second file (the .met file) is created.
  • Since all of the files remain in the directory, the directory to be scanned may contain a large number of files.
  • All of the met files must be checked by Transfer CFT at start up.
  • Transfer CFT cannot differentiate between when a file has been purged and replaced by a new one, or if this same file has just been modified.

Create inclusion and exclusion filters

You can use the file_include_filter and file_exclude_filter parameters to define file name patterns to include or exclude files from folder monitoring.

The filter_type parameter value then indicates how the comparison of file names against the patterns occurs. There are two possible filter_type parameter values, STRJCMP and WILDMAT.

STRJCMP filter_type

A STRJCMP pattern-matching filter can contain the asterisk (*) and/or the question mark (?) characters. The STRJCMP filter characters are interpreted as follows:

Character Description Example
* Indicates any sequence of zero or more characters. The filter "*.dat" selects any file name that has the extension ".dat".
? Indicates any single character. The filter "T*.???" selects any file name starting with a 'T' and having an extension of exactly three characters.

WILDMAT filter_type

Unix/Windows only

The WILDMAT pattern-matching filter offers more operations than the STRJCMP filter_type. The WILDMAT filter characters are interpreted as follows, where x and y are used to indicate any character:

Character Description Example
* Indicates any sequence of zero or more characters. The filter "*.dat" selects any file name that has the extension ".dat".
? Indicates any single character. The filter "T*.???" selects any file name starting with a 'T' and having an extension of exactly three characters.
\x Indicates that when x is a special character, x is interpreted as a normal character. This is generally used to invalidate the meaning of the * and ? characters.
[x...y] Indicates a single character set defined by "x...y".

The filter [0-9]

indicates any decimal digit.

-

Indicates a range of characters. However, the minus character (or hyphen) has no special meaning if it is either the first or the last character in the set.

The filter [0-9a-zA-Z] indicates any alphanumeric character (in English).
] Has no special meaning if it is the first character in the set.  
[^x...y] Indicates any character other than those  specified in the set "x...y".

The  filter [^0-9]

indicates any character which is not a decimal digit.

The filter [^]-] indicates any character other than a closed bracket or minus sign.

*.[tT][xX][tT] Indicates any string terminated by .TXT regardless of the case.  

Modify and apply configuration changes

The act of starting Transfer CFT causes Transfer CFT to check for and reload configuration changes. Alternatively, you can dynamically execute the CFTUCONF RECONFIG type=uconf command to check and reload the configuration.

Upon reloading, if there are any modified configuration parameters or detected errors in the new configuration, Transfer CFT records these in the log. Additionally, Transfer CFT verifies that the updated configuration is compatible with the contents of the current directories.

In particular, if you change the METHOD parameter from FILE to MOVE without modifying the scan_dir and work_dir parameters, and if the work_dir directory is not empty, Transfer CFT displays an error message in the log and will not monitor the corresponding directory.

To deactivate compatibility checks of a folder’s new configuration, unset the value of the folder_monitoring.folders .<logical_name>.control parameter using the uconfunset command.

Example

If the folder's logical name is A, execute the following command prior to the reconfiguration (or start) command:

CFTUTIL UCONFUNSET id = folder_monitoring.folders. A.control

When you then reconfigure (or start) Transfer CFT, the A folder is not checked.

CFTUCONF RECONFIG type=uconf

Directory configuration examples

This section presents an example that consists of configuring 3 directories for monitoring, each having a different set of configuration parameter values. In this example, the three different directories are called A, B, and C.

Note that the configuration parameter folder_monitoring must contain a list with these directory names, separated by blanks. Additionally, you must enable the folder monitoring functionality.

Note In all of the examples in this topic, you must enter CFTUTIL in upper case.

For this example, you would execute the following command:

CFTUTIL uconfset id=folder_monitoring.enable , value='Yes'

CFTUTIL uconfset id=folder_monitoring.folders , value= 'A B C'

*Note that the "' '"characters are used to protect the spaces between each folder monitoring nodes declarations.

Note All of the examples in this section were written for a UNIX platform. Modify to suit your environment accordingly.

Directory A requirements

The first directory presents the simplest possible configuration, leaving most parameters set to their default values.

  • All of the files in the directory sub-tree are candidates for the SEND submission.
  • The files are sent to a given partner, newyork, using an IDF name of IDFA.

The following commands create the configuration defined for directory A.

#

# Create all of the needed directories (UNIX platform example)

#

mkdir /home/CFT/fm/dir_a

mkdir /home/CFT/fm/dir_a/scan

mkdir /home/CFT/fm/dir_a/work

#

# Define the needed Transfer CFT configuration parameters leaving all others set to their default value.

#

CFTUTIL uconfset id=folder_monitoring.folders.A.scan_dir , value='/home/CFT/fm/dir_a/scan'

CFTUTIL uconfset id=folder_monitoring.folders.A.work_dir , value='/home/CFT/fm/dir_a/work'

CFTUTIL uconfset id=folder_monitoring.folders.A.part , value='NEWYORK'

CFTUTIL uconfset id=folder_monitoring.folders.A.idf , value='IDFA'

Directory B requirements

For the second directory, directory B, we want to:

  • Be able to send files to the following partners, newyork, berlin, london, rome, brussels, and paris.
  • Use the id given as the IDF, in this example TXT.
  • Send only files suffixed by .txt.

The following commands create the required directory B configuration.

#

# Create all needed directories (example for UNIX platforms)

#

mkdir /home/CFT/fm/dir_b

mkdir /home/CFT/fm/dir_b/scan

mkdir /home/CFT/fm/dir_b/work

mkdir /home/CFT/fm/dir_b/scan/newyork

mkdir /home/CFT/fm/dir_b/scan/berlin

mkdir /home/CFT/fm/dir_b/scan/london

mkdir /home/CFT/fm/dir_b/scan/rome

mkdir /home/CFT/fm/dir_b/scan/brussels

mkdir /home/CFT/fm/dir_b/scan/paris

#

# Define all of the needed Transfer CFT configuration parameters, while leaving the others set to their default value.

#

CFTUTIL uconfset id=folder_monitoring.folders.B.scan_dir , value='/home/CFT/fm/dir_b/scan'

CFTUTIL uconfset id=folder_monitoring.folders.B.work_dir , value='/home/CFT/fm/dir_b/work'

CFTUTIL uconfset id=folder_monitoring.folders.B.part , value='(0)'

CFTUTIL uconfset id=folder_monitoring.folders.B.idf , value='TXT'

CFTUTIL uconfset id=folder_monitoring.folders.B.file_include_filter , value='*.txt'

The files to be sent must be moved to the directory that corresponds to the destination partner name, for example /home/CFT/fm/dir_b/newyork for the partner named newyork.

Directory C requirements

For the third directory, directory C, we want to:

  • Be able to send files to multiple partners, newyork and paris.
  • Use idf1, idf2, or idf3 as the newyork partner IDF.
  • Use idfa, idfb, idfc, or idfd as the paris partner IDF.
  • Not send files suffixed by .tmp.
  • Automatically move the files to be sent to the scan_dir, so the file_idle_delay parameter value is set to zero.
  • Submit files within a delay of approximately 10 seconds (interval).
  • Limit the number of send submissions per interval to 4 (file_count).

The following commands create the described directory C configuration.

#

# Create all needed directories (example for UNIX platforms)

#

mkdir /home/CFT/fm/dir_c

mkdir /home/CFT/fm/dir_c/scan

mkdir /home/CFT/fm/dir_c/work

mkdir /home/CFT/fm/dir_c/scan/newyork/idf1

mkdir /home/CFT/fm/dir_c/scan/newyork/idf2

mkdir /home/CFT/fm/dir_c/scan/newyork/idf3

mkdir /home/CFT/fm/dir_c/scan/paris/idfa

mkdir /home/CFT/fm/dir_c/scan/paris/idfb

mkdir /home/CFT/fm/dir_c/scan/paris/idfc

mkdir /home/CFT/fm/dir_c/scan/paris/idfd

#

# Define all necessary Transfer CFT configuration parameters leaving others set to their default value.

#

CFTUTIL uconfset id=folder_monitoring.folders.C.file_idle_delay , value='0'

CFTUTIL uconfset id=folder_monitoring.folders.C.idf , value='(1)'

CFTUTIL uconfset id=folder_monitoring.folders.C.part , value='(0)'

CFTUTIL uconfset id=folder_monitoring.folders.C.scan_dir , value='/home/CFT/fm/dir_c/scan'

CFTUTIL uconfset id=folder_monitoring.folders.C.work_dir , value='/home/CFT/fm/dir_c/work'

CFTUTIL uconfset id=folder_monitoring.folders.C.interval , value='10'

CFTUTIL uconfset id=folder_monitoring.folders.C.file_count , value='4'

CFTUTIL uconfset id=folder_monitoring.folders.C.file_exclude_filter , value='*.tmp'

The files to be sent must be moved to the directory that corresponds to the destination partner and idf names, for example /home/CFT/fm/dir_c/newyork/idf1 for the partner newyork and idf idf1.

Folder monitoring using the Transfer CFT UI

From the Transfer CFT UI, select the Unified Configuration icon. From the Unified configuration dialog box, select folder_monitoring. To display all possible parameters, first create at least one folder (folder_monitoring.folders) for Transfer CFT to monitor, and click Apply.

For more information on setting unified configuration parameters, refer to Using UCONF in CFTUTIL or About the unified configuration topics.

 

Related Links