Folder monitoring CFTFOLDER

This section provides a description of how to use Transfer CFT objects to manage folder monitoring.

Note There are two ways to implement Transfer CFT folder monitoring, either using UCONF or Transfer CFT objects. We recommend the CFTFOLDER method of configuring folder monitoring. Users that presently are using UCONF to manage folder monitoring can migrate to a CFTFOLDER configuration as described in Migrate to CFTFOLDER folder monitoring.

Step overview

  1. Activate the folder monitoring option.
    • Set uconf parameter folder_monitoring.enable to Yes.
  2. Declare your logical directories to monitor.
    • Add CFTFOLDER objects.
  3. For each logical directory defined, configure the specific options you want to use for each:
    • File management method
    • Used sub-directories
    • Set the IDF
    • Set the partner name
    • Define the delay to take into account the file
    • Define other Folder Monitoring parameters
  4. Optionally, to specify file-system event monitoring you additionally must set this in CFTFOLDER object.

CFTFOLDER object

The following table describes the parameters you can use to define the CFTFOLDER object. Additionally the table lists the UCONF equivalent for users who have opted for a polling type of folder monitoring.

Folder monitoring parameters

Use the following CFTFOLDER parameters to configure folder monitoring for each directory as needed.

Parameter descriptions

Parameter Type

Default  

Description

same as in UCONF

<folder_monitoring.enable>

Boolean No
  • No: No folder monitoring occurs.
  • Yes: Enable Transfer CFT folder monitoring.

ID

Mandatory

node None

Add the logical folders to monitor (list of logical identifiers).

You should provide a unique name to identify the set of configuration parameters corresponding to this directory. If you have more than one Folder to monitor, use a space between each logical value.

STATE Boolean Active

Enables a scan of the folder.

Note NO = NOACTIVE.

SCANDIR

Mandatory
string None

Absolute path name of the top level directory to scan.

This directory must exist before restarting Transfer CFT.

*See NOTE.

WORKDIR

Mandatory
string None

Absolute path name of the top level directory available for file state information.

  • If you are using the MOVE method, files that are ready to be submitted are available in the work_dir.
  • If you are using the FILE method, the .met files are stored in the work_dir.
Caution     Never delete any .met files.

*See NOTE.

ENABLESUBDIR Boolean Yes

Values:

  • Yes: The entire scan_dir sub-directory tree is monitored.
  • No: No scan is performed.
METHOD enum MOVE

Values:

  • MOVE: Files are moved to the work_dir prior to being submitted.
  • FILE: Files are left in the scan_dir, and a state file with the same name is created in work_dir prior to submitting the file.
Note Before changing the method from FILE to MOVE, you should remove all files (metadata .met files) located in the associated working directory.
Note Changing the method from MOVE to FILE, deletes all files located in the associated working directory. Therefore, we recommend removing all files from the scan and working directory before changing the METHOD type.
FILEIDLEDELAY integer 5 If the state of a file has not changed within this delay in seconds, the file becomes a candidate for submission.

IDF

Mandatory

string ""

The IDF name to use in the SEND command. Use one of the following:

  • A fixed name.
  • "(0)": The name of the first directory sub-level is used.
  • "(1)": The name of the second directory sub-level is used.

PART

Mandatory
string ""

The PART name to use in the SEND command. Use one of the following:

  • A fixed name.
  • "(0)": The name of the first directory sub-level is used.
  • "(1)": The name of the second directory sub-level is used.
INTERVAL int 60 The interval between two scans of the directory files in seconds.
FILECOUNT int 0

Maximum number of file submissions for each scan. Using the default value indicates that there is no maximum.

FILESIZEMIN int 0

Files shorter than this value, in bytes, are not candidates for submission. Using the default value indicates that there is no lower limit on the file size.

FILESIZEMAX int 0

Files larger than this value, in bytes, are not candidates for submission. Using the default value indicates that there is no upper limit on the file size.

INCLUDEFILTER string "" If this parameter is defined, only files whose names match this pattern are monitored.
EXCLUDEFILTER string "" If this parameter is defined, files whose names match this pattern are not monitored.
RESUBMITCHANGED Boolean Yes

This parameter has no effect when the configured method is MOVE.

When the method parameter value is set to FILE:

  • Yes: When the state of a previously submitted file is seen as having changed, the file is submitted again.
  • No: Files are not resubmitted, regardless of changes.
Note The file is resubmitted after any change regardless of if the modification is a small change, or purging and replacing the file with another file having the same name.
FILTERTYPE enum WILDMAT

Defines the pattern matching algorithm to use for file name filtering.

Values:

  • STRJCMP: The Transfer CFT pattern matching algorithm.
  • WILDMAT: A well known public domain algorithm, and is the default. Unix/Windows only
  • EREGEX:  Extended regular expression syntax.

See Create inclusion and exclusion filters for details.

RENAMEMETHOD Enum TIMESTAMP

This parameter applies only to the MOVE method. When set to TIMESTAMP, a timestamp of the pattern YYYYMMDDHHMMSS is added at the end of the name of the renamed file but before the last '.'.

For example, using timestamp_separators=".":
  • myfile is renamed myfile.20131025
  • myfile.txt is renamed myfile.20131025.txt
NoteUnset the default value and use " " to MOVE without adding a timestamp.
RENAMESEPARATOR string

OpenVMS:

" _ "

All other OS:

"."

This parameter only applies to the MOVE method.

You can use no more than two characters from among the following:

.[]()_-

The first character defines the separator before the timestamp. The second one, when present, defines the separator after the timestamp.

For example, using timestamp_separators "[]": - myfile is renamed myfile.[20131025] - myfile.txt is renamed myfile.[20131025].txt

N/A in this version string   Metadata used to control user changes.

USEFSEVENTS

 

More information

Boolean No Set to YES to enable the file system events monitoring service to detect newly available files.
Note *You cannot use the following characters in the SCANDIR or WORKDIR definition. Additionally you cannot use a comma (,) in the CFTFOLDER SCANDIR or WORKDIR definition.
  • UNIX /
  • For Windows \ / : * ? " < > |

Parameter settings and actions

  • The delay between scans of a given directory is defined by its interval parameter value.
  • By default the ENABLESUBDIR [enable_subdir] parameter is set to YES, and the directory and all its sub-directories are scanned.
  • For each file detected, the name is checked against the configured parameters values in the include and exclude file filters. Files that match the combined criteria are monitored, all others are ignored.

For a file to become a candidate to be submitted, the following conditions must be met:

  • File size: If these values are configured, the following rules apply.
    • FILESIZEMIN [file_size_min]: The current size must not be less than this value.
    • FILESIZEMAX [file_size_max]: The current size must not be greater than this value.
  • The last modification time and duration must not have changed within a number of seconds as defined in the FILEIDLEDELAY [file_idle_delay] parameter value.

Create or modify a CFTFOLDER object and apply configuration changes

The act of starting Transfer CFT causes Transfer CFT to check for and reload configuration changes. Alternatively, you can dynamically execute the CFTUTIL RECONFIG type=FOLDER command to check and reload the configuration.

Upon reloading, if there are any modified configuration parameters or detected errors in the new configuration, Transfer CFT records these in the log. Additionally, Transfer CFT verifies that the updated configuration is compatible with the contents of the current directories.

In particular, if you change the METHOD parameter from FILE to MOVE without modifying the scan_dir and work_dir parameters, and if the work_dir directory is not empty, Transfer CFT displays an error message in the log and will not monitor the corresponding directory.

If you create or modify a folder while Transfer CFT is running, you must execute the ACT command to reload the configuration.

Example

The following command reloads the FM40 configuration.

CFTUTIL ACT ID=FM40, type=FOLDER

Enable the file-system event monitoring

This feature enables you to use file-system events monitoring to detect newly available files for an immediate Transfer CFT action.

Available on Linux/Windows only

See Supported OS for file-system event monitoring.

To enable file-system event monitoring modify as follows:

CFTUTIL CFTFOLDER ID=<myfolderobject>, USEFSEVENTS=YES, ...

Attention

This feature can be resource intensive for Transfer CFT and the system in general in the following situations:

  • You have a large number of directories and sub-directories monitored using file-system events.
  • The activity in terms of file additions, removals, changes of files in those directories is high.

We recommended that you only use file-system event monitoring when immediate attention by Transfer CFT is a functional requirement.

Activate and deactivate folder monitoring

To turn the file-system event monitoring off for a given folder object, use the command:

CFTUTIL INACT TYPE=FOLDER,ID=<myfolderobject>

To turn on the file-system event monitoring for a given folder object, use the command:

CFTUTIL ACT TYPE=FOLDER,ID=<myfolderobject>

To view all of the CFTFOLDER objects, use the command:

CFTUTIL LISTPARM TYPE=FOLDER

To extract the folder objects, use the command:

CFTUTIL CFTEXT TYPE=FOLDER

Remove or change a folder

Change a folder

If Transfer CFT is running and you create or change a folder or multiple folders, you can execute RECONFIG TYPE=FOLDER so that all changes are taken into account.

Delete a folder

Prior to deleting a folder object, check that it is inactive. You can execute INACT on this folder if unsure.

CFTUTIL INACT TYPE=FOLDER, ID=<myfolder>

CFTUTIL CFTFOLDER ID=<myfolder>, MODE=DELETE

Note If you delete an active folder object while Transfer CFT is running without first deactivating the folder (INACT), you must execute a RECONFIG TYPE=FOLDER. If you do not, the folder remains active.

Directory configuration examples

This section presents an example that consists of configuring 3 directories for monitoring, each having a different set of configuration parameter values. In this example, the three different directories are called A, B, and C.

Note All of the examples in this section were written for a UNIX platform. Modify to suit your environment accordingly.

Directory A requirements

The first directory presents the simplest possible configuration, leaving most parameters set to their default values.

  • All of the files in the directory sub-tree are candidates for the SEND submission.
  • The files are sent to a given partner, newyork, using an IDF name of IDFA.

The following commands create the configuration defined for directory A.

#

# Create all of the needed directories (UNIX platform example)

#

mkdir /home/CFT/fm/dir_a

mkdir /home/CFT/fm/dir_a/scan

mkdir /home/CFT/fm/dir_a/work

#

CFTUTIL CFTFOLDER ID=A, SCANDIR='/home/CFT/fm/dir_a/scan', WORKDIR='/home/CFT/fm/dir_a/work', PART='NEWYORK', IDF='IDFA'

Directory B requirements

For the second directory, directory B, we want to:

  • Be able to send files to the following partners, newyork, berlin, london, rome, brussels, and paris.
  • Use the id given as the IDF, in this example TXT.
  • Send only files suffixed by .txt.

The following commands create the required directory B configuration.

#

# Create all needed directories (example for UNIX platforms)

#

mkdir /home/CFT/fm/dir_b

mkdir /home/CFT/fm/dir_b/scan

mkdir /home/CFT/fm/dir_b/work

mkdir /home/CFT/fm/dir_b/scan/newyork

mkdir /home/CFT/fm/dir_b/scan/berlin

mkdir /home/CFT/fm/dir_b/scan/london

mkdir /home/CFT/fm/dir_b/scan/rome

mkdir /home/CFT/fm/dir_b/scan/brussels

mkdir /home/CFT/fm/dir_b/scan/paris

#

CFTUTIL CFTFOLDER ID=B, SCANDIR='/home/CFT/fm/dir_b/scan', WORKDIR='/home/CFT/fm/dir_b/work', PART='(0)', IDF='TXT', INCLUDEFILTER='*.txt'

The files to be sent must be moved to the directory that corresponds to the destination partner name, for example /home/CFT/fm/dir_b/newyork for the partner named newyork.

Directory C requirements

For directory C we want to:

  • Send files to multiple partners, newyork and paris.
  • Use idf1, idf2, or idf3 as the newyork partner IDF.
  • Use idfa, idfb, idfc, or idfd as the paris partner IDF.
  • Not send files suffixed by .tmp.
  • Automatically move the files to be sent to the SCANDIR, so the FILEIDLEDELAY parameter value is set to zero.
  • Submit files within a delay of approximately 10 seconds (INTERVAL).
  • Limit the number of send submissions per interval to 4 (FILECOUNT).

The following commands create the described directory C configuration.

#

# Create all needed directories (example for UNIX platforms)

#

mkdir /home/CFT/fm/dir_c

mkdir /home/CFT/fm/dir_c/scan

mkdir /home/CFT/fm/dir_c/work

mkdir /home/CFT/fm/dir_c/scan/newyork/idf1

mkdir /home/CFT/fm/dir_c/scan/newyork/idf2

mkdir /home/CFT/fm/dir_c/scan/newyork/idf3

mkdir /home/CFT/fm/dir_c/scan/paris/idfa

mkdir /home/CFT/fm/dir_c/scan/paris/idfb

mkdir /home/CFT/fm/dir_c/scan/paris/idfc

mkdir /home/CFT/fm/dir_c/scan/paris/idfd

#

CFTUTIL CFTFOLDER ID=C, FILEIDLEDELAY='0', IDF='(1)', PART='(0)', SCANDIR='/home/CFT/fm/dir_c/scan',

WORKDIR='/home/CFT/fm/dir_c/work', INTERVAL='10', FILECOUNT='4', FILEEXCLUDEFILTER='*.tmp'

The files to be sent must be moved to the directory that corresponds to the destination partner and idf names, for example /home/CFT/fm/dir_c/newyork/idf1 for the partner newyork and idf idf1.

Related topics

Related Links