Bulk migration API batch mode

Batch mode is an optimized way to upload high volume of data using special API provided by few of the platforms. Currently DropBox, DropBox For Business, Office 365 and OneDrive For Business have support for this feature. When running in batch mode, instead of committing one file at a time, files are committed in batches optimized for each platform.

Requirement

To leverage the Bulk (Batch) Migration API, transfer accounts must be an Site Collection Administrator

Non-Administrator transfer accounts will default to use CSOM, even when batch mode is requested with the job.

Configure Your Job

Batch mode policy can be changed or disabled by setting transfer.batch_mode to:

  • none - disable the batch mode for all job's executions
  • initial -  use batch mode for the first job execution only
  • always - (default behavior) use batch mode for all job executions

Example

{
    "name":"Simple Transfer Job with no batch mode",
    "kind": "transfer",
    "transfer": {
        "batch_mode": "none",
        "transfer_type": "copy",
        "source": {
            "connection": { "id": "{{nfs_connection}}" }
        },
        "destination": {
            "connection": { "id": "{{cloud_connection}}" },
            "target": {
                "path": "/MyDestination1"
            }
        }
    },
    "schedule": {
        "mode": "manual"
    }
}

Enable Batch Manifests

Batch Manifests are configured in your appSettings.json file.

  • You need the platform ID. See List Platform ID's for the call to return the ID. 
  • You must restart your DataHub Service to enable the batch manifest feature.
  • Batch manifests are located in your log directory.

This is the element that needs to be included to enable batch manifests. Replace "platform_id" with the corresponding platform ID. 

}
 "platform_id": {
   "batch_save_manifests": true
}

The example code below shows enabling batch manifest for Office 365 (platform_id = office365-oauth2) and OneDrive for Business (platform ID = onedrive-business-oauth2).

Enable Batch Manifests

{
  "data": {
    "provider": "sqlserver",
    "connection": "SERVER=localhost;DATABASE=SkySyncV4;Integrated Security=SSPI;",
    "directory": "C:\\ProgramData\\Syncplicity DataHub\\v4"
  },
  "server": {
    "proxy_url": "http://192.168.1.76:9090/"
  },
  "deployment": {
        "packageDirectory": "C:\\ProgramData"
  },
    "office365-oauth2": {
    "batch_save_manifests": true
  },
    "onedrive-business-oauth2": {
    "batch_save_manifests": true
  },
  "license": {
    "activation_key": "ab4e994a-342f-0000-0000-b5da2577579b"
    }
  }
}

Configure Batch Size

Batch Size is configured in your appSettings.json file.

  • You need the platform ID. See List Platform ID's for the call to return the ID. 
  • You must restart your DataHub Service for the batch size to be applied.
  • The Default Batch Size is 100.


Configure Batch Size

{
  "data": {
    "provider": "sqlserver",
    "connection": "SERVER=localhost;DATABASE=SkySyncV4;Integrated Security=SSPI;",
    "directory": "C:\\ProgramData\\Syncplicity DataHub\\v4"
  },
  "server": {
    "proxy_url": "http://192.168.1.76:9090/"
  },
  "deployment": {
        "packageDirectory": "C:\\ProgramData"
  },
    "office365": {
    "batch_save_manifests": true,
    "batch_item_limit": "5"
  },
    "onedrive-business": {
    "batch_save_manifests": true,
    "batch_item_limit": "5"
  },
  "license": {
    "activation_key": "ab4e994a-342f-0000-0000-b5da2577579b"
    }
  }
}

Settings through the UI

Add the following to the JSON block in Create Job Advanced Features.

{
"transfer": {
        "batch_mode": "none"
    }
}

Related Links