Folder mapping

Convention jobs such as folder mapping are ideal for migrations where you wish to control the transfer at a granular level without the effort of creating individual jobs. DataHub will automatically create a unique job for each folder in your hierarchy, inheriting configurations from the parent/master job. 

Folder mapping jobs are a type of convention or master job that create child jobs a single level deep in the folder hierarchy at the path you specify. The folder mapping job may be controlled and manipulated just like a transfer job, but when executed, it will not transfer data. Instead, each execution creates, modifies, or deletes its child jobs, which are then responsible for the transfer of data.  For example, if your source contains three sub-folders, a folder mapping job would create a child job for each of those three folders in the specified path. As new folders are created in the source, additional child jobs are created for them automatically.  Data will be transferred when the child jobs are run.

Performance Considerations

The parent for a folder mapping job has minimal impact on performance. However, the child jobs it creates follow the same performance consideration as any job type available in DataHub. 

If DataHub is installed on a single instance, Parallel Writes will be a limiting factor, regardless of how many child jobs are created. In a multi-node scenario, more jobs running concurrently may be configured without impacting performance. 

User-Interface - Creating a Folder Mapping Job

Folder Mapping supports the following:

Impersonation

All job features defined while creating the parent job will be applied to the child jobs it creates

Defining your Source and Destination Paths

If you are an administrator using Impersonation, enable Run as user..., and choose the user you wish to access.

Source / Destination Path: If you wish to transfer all content, leave the source path blank. A child job will be created for every top level folder. If a folder is selected for the source path, a child job will be created for every sub folder within the parent folder. 

Child Job Source / Destination Path: This directory within each folder will be used as the source.

Target the root of each folder: The child job will be created for the first level folder relative to the source path.

Target a specific directory within each folder: If there is a folder that is exists in every directory, you can define it with this option.

Create the Folder Mapping Job

After configuring policies, behaviors, and advanced features, you will be prompted to schedule the job. This schedule will be applied to the child jobs

  • The parent job will run immediately to create the child jobs.
  • After the child jobs are created, the default schedule is set to run every 6 hours so it can review the source for any new content.
  • The parent and child job schedules can be changed at any time.

DataHub API - Creating a Folder Mapping Job

In general, creating a folder mapping job is not much different than creating a transfer job. 

The parameter to note here is "kind".  It must be set to "folder_mapping".  

To set the schedule for the child jobs, set the schedule parameter within the transfer block.  The schedule parameter outside the transfer block is the schedule for the parent job only.

In the example below,

  • The parent job is set to run automatically after creation and run every 15 mins thereafter.
  • The child jobs are set to a manual schedule.

POST {{url}}v1/jobs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91

{
    "name":"Convention Job | Folder Mapping",
    "kind": "folder_mapping",
    "transfer": {
      "transfer_type": "copy",
      "audit_level": "trace",
      "batch_mode": "always",
      "conflict_resolution": "latest",
      "delete_propagation": "ignore_destination",
      "failure_policy": "continue",
      "large_item": "skip",
      "lock_propagation": "ignore",
      "max_items_per_container": 10000,
      "performance": {
                "parallel_writes": {
                    "requested": 2
                }
      },         
      "permissions": {
            "policy": "add",
            "links": true,
            "failures": "none"
      },
      "preserve_owners": true,
      "timestamps": true,
      "empty_containers": "create",
      "duplicate_names": "rename",
      "item_overwrite": "overwrite",
      "restricted_content": "convert",
      "segment_transform": true,
      "versioning": {
            "preserve": "native",
            "select": "all"
      },
      "account_map": {
         "id": "{{account_map_id}}",
         "type": "account_map"
    },
    "filter": {
                "source": [
                    {
                        "action": "exclude",
                        "rules": [
                            {
                                "type": "filter_hidden"
                            }
                        ],
                        "type": "filter_rule"
                    }
                ],
                "destination": [
                    {
                        "action": "exclude",
                        "rules": [
                            {
                                "type": "filter_hidden"
                            }
                        ],
                        "type": "filter_rule"
                    }
                ]
        },
        "source": {
            "connection": { "id": "{{cloud_connection_source}}" },
            "target": {
                "path": "/SourcePath"
            }
        },
        "destination": {
            "connection": { "id": "{{cloud_connection_destination}}" },
            "target": {
                "path": "/DestinationPath"
            }
        },
        "simulation_mode": false,
        "schedule": {
            "mode": "manual"
        }
    },
    "schedule": {
        "mode": "auto"
    },
    "stop_policy": {
        "on_success": 5,
        "on_failure": 5,
        "on_execute": 25
    },
    "category": {
      "name": "Convention Jobs | Folder Mapping"
    }
}

DataHub API - Creating a Folder Mapping Job by Including Specific Folders

POST {{url}}v1/jobs

{
  "name":"Folder Mapping Convention",
  "kind""folder_mapping",
  "transfer": {
      "transfer_type""copy",
      "source": {
        "connection": { "id""{{nfs_connection}}" }
      },
      "destination": {
        "connection": { "id""{{cloud_connection}}" },
        "target": {
          "path""/FolderMappingDestination"
        }
      },
      "schedule": {
        "mode""manual"
      }
  },
  "schedule": {
    "mode""manual"
  },
  "convention": {
    "inclusions": [
      "folder1",
      "folder2",
      "folder4"
    ]
  }
}

DataHub API | Creating a Folder Mapping Job by Excluding Specific Folders

POST {{url}}v1/jobs

{
  "name":"Folder Mapping Convention",
  "kind""folder_mapping",
  "transfer": {
      "transfer_type""copy",
      "source": {
        "connection": { "id""{{nfs_connection}}" }
      },
      "destination": {
        "connection": { "id""{{cloud_connection}}" },
        "target": {
          "path""/FolderMappingDestination"
        }
      },
      "schedule": {
        "mode""manual"
      }
  },
  "schedule": {
    "mode""manual"
  },
 
  "convention": {
    "exclusions": [
      "folder1",
      "folder2",
      "folder4"
    ]
  }
}

Related Links