Item overwrite policy

Overwrite policy controls DataHub's behavior when it detects the same file name on the target platform.

Overwrite (default)

This is the recommended and default setting

When overwrite policy is configured, it will overwrite the target location any time DataHub detects an event indicating that a change occurred. If no event is detected; item overwrite policy is not applicable. Item overwrite behaves the same, both on the first and subsequent runs. 

Conflict Policy and Delete Policy are also triggered by event detection. However, overwrite policy will take precedence. 

Skip

Not recommended / Special case configuration only

When overwrite policy skip is configured, DataHub will skip any files that exist on both sides of the given platforms. All other policies may still apply. 

This is not a recommended configuration because it can lead to unintended discrepancies between the source and destination. For example, when an event is detected to transfer the file using a conflict policy, overwrite skip setting will take precedence. This will veto the transfer decision for existing files and log a warning message. 

Fail

Not recommended / Special case configuration only

When overwrite policy fail is configured, DataHub will fail any files that exist on both sides of the given platforms. All other policies may still apply. 

This is not a recommended configuration because it can lead to unintended discrepancies between the source and destination. For example, when an event is detected to transfer the file using a conflict policy, overwrite fail setting will take precedence. This will veto the transfer decision for existing files and log a failure message. 

Job JSON Example

Item Overwrite policy options include:

  • overwrite (default) - overwrite the file
  • fail - do not overwrite the file, report as error
  • skip - do not overwrite the file, do not report as error

Example

{
    "name":"Simple Transfer Job with Overwrite Policy",
    "kind": "transfer",
    "transfer": {
        "item_overwrite": "overwrite",
        "transfer_type": "copy",
        "source": {
            "connection": { "id": "{{nfs_connection}}" },
            "target": {
                "path": "/SourcePath"
            }
        },
        "destination": {
            "connection": { "id": "{{cloud_connection}}" },
            "target": {
                "path": "/DestinationPath"
            }
        }
    },
    "schedule": {
        "mode": "manual"
    }
}

Related Links