Segment transformation, invalid characters policy

Segment Transformation policy controls if DataHub can change folder and file names to comply to platform's restrictions. 

Enabling this behavior allows DataHub to change the names of folders and files that contain characters that are not supported by the destination before transferring the file. This will be done instead of flagging the file for manual remediation and preventing it from being transferred. When this occurs the unsupported character will be transformed into a underscore.

For example, if you attempted to transfer the file "Congrats!.txt" from box to NFS it would be transformed to "Congrats_.txt" and appear that way on the destination. 

In the Create Job User Interface (UI), "Allow Unsupported File Names to be Changed" is the same as "segment_transform": "true" configuration in the job JSON

Job JSON Example

Segment Transformation Policy options include:

  • true (default)
  • false

Example

{
    "name":"Simple Transfer Job with Segment Transform Policy",
    "kind": "transfer",
    "transfer": {
        "segment_transform": false,
        "transfer_type": "copy",
        "source": {
            "connection": { "id": "{{nfs_connection}}" },
            "target": {
                "path": "/SourcePath"
            }
        },
        "destination": {
            "connection": { "id": "{{cloud_connection}}" },
            "target": {
                "path": "/DestinationPath"
            }
        }
    },
    "schedule": {
        "mode": "manual"
    }
}

How to List Connection Features & Limitations

Ensure you have an established connection before making the following call

Connection Features

GET {{url}}v1/connections/{{cloud_connection}}?include=features

To list general supported features for a given platform, make the following call. No connection is required

Platform Features

GET {{url}}v1/connections/platforms/{{platform_id}}?fields=all

Related Links