Bandwidth throttling

Bandwidth throttling allows the user to limit how much upload or download bandwidth DataHub's jobs use. Throttling can be setup to take affect at certain times of day.

Bandwidth Throttling Options

Bandwidth throttling can be specified on per job basis. Additionally, default throttling setting can be set for all jobs which do not have bandwidth specified as part of the job options.

Option

Description

Job's static upload and download throttling

Sets max upload and download bandwidth for the job:

  • Example:
{
...
	"transfer": {
		...
		"performance": {
			"upload": {
				"bytes_per_second": 102400
			},
			"download": {
				"bytes_per_second": 102400
		}
		...
	}
...
}
Job with throttling window during business hours

Limits upload during business hours. Note that hours must be specified in UTC and bytes_per_second = 0 is unlimited.

  • Example:
{
...
	"transfer": {
	...
		"performance": {
			"upload": {
				"window": [
					{
						"bytes_per_second": 102400, 
						"time": {
							"hr": 9,
							"min": 3,
							"sec": 0
						}
					},
					{
						"bytes_per_second": 0, 
						"time": {
							"hr": 17,
							"min": 0,
							"sec": 0
						}
					}
				]
			}
		}
	...
	}
...
}
Default job bandwidth settings

Apply bandwidth throttling to all jobs which do not have it specified in transfer options.

Example setting in appSettings.json: 

"performance" : {
             "throttle" : {
                 "upload" :  102400 ,
                 "download"   102400
             }
}

Default job bandwidth settings with bandwidth window

Apply bandwidth throttling during business hours to all jobs which do not have it specified in transfer options.

Example setting in appSettings.json: 

"performance": {
	"throttle": {
		"upload": {
			"window": [
				{
					"bytes_per_second": 102400, 
					"time": {
						hr: 9,
						min: 30,
						sec: 00
					}
				},
				{
					"bytes_per_second": 0, 
					"time": {
						"hr": 17,
						"min": 00,
						"sec": 00
					}
				}
			]
		}
	}
}

Example JSON

The following copy job will upload at rate no higher than 100kBps.

POST v1/jobs

{
     "name" : "Simple Transfer Job with upload throttling" ,
     "kind" :  "transfer" ,
     "transfer" : {
         "transfer_type" :  "copy" ,
         "source" : {
             "connection" : {  "id" :  "{{nfs_connection_id}}"   },
             "target" : {
                 "path" :  "/w/source"
             }
         },
         "destination" : {
             "connection" : {  "id" :  "{{cloud_connection_id}}"   },
             "target" : {
                 "path" :  "/w/destination"
             }
         },
         "performance" : {
             "upload" : {
                 "bytes_per_second" :  102400
             }
         }
     },
     "schedule" : {
         "mode" :  "auto"
     }
}

Related Links