Machine parameters
- machine:
- type:
dict
argument path:machine
- batch_type:
- type:
str
argument path:machine/batch_type
The batch job system type. Option: Slurm, SlurmJobArray, Lebesgue, DistributedShell, Shell, DpCloudServer, LSF, Torque, PBS
- local_root:
- type:
str
argument path:machine/local_root
The dir where the tasks and relating files locate. Typically the project dir.
- remote_root:
- type:
str
, optionalargument path:machine/remote_root
The dir where the tasks are executed on the remote machine. Only needed when context is not lazy-local.
- clean_asynchronously:
- type:
bool
, optional, default:False
argument path:machine/clean_asynchronously
Clean the remote directory asynchronously after the job finishes.
Depending on the value of context_type, different sub args are accepted.
- context_type:
- type:
str
(flag key)argument path:machine/context_type
possible choices: DpCloudServerContext, HDFSContext, LebesgueContext, SSHContext, LazyLocalContext, LocalContextThe connection used to remote machine. Option: HDFSContext, SSHContext, DpCloudServerContext, LazyLocalContext, LocalContext, LebesgueContext
When context_type is set to
DpCloudServerContext
(or its aliasesdpcloudservercontext
,DpCloudServer
,dpcloudserver
):- remote_profile:
- type:
dict
argument path:machine[DpCloudServerContext]/remote_profile
The information used to maintain the connection with remote machine.
- email:
- type:
str
argument path:machine[DpCloudServerContext]/remote_profile/email
Email
- password:
- type:
str
argument path:machine[DpCloudServerContext]/remote_profile/password
Password
- program_id:
- type:
int
argument path:machine[DpCloudServerContext]/remote_profile/program_id
Program ID
- input_data:
- type:
dict
argument path:machine[DpCloudServerContext]/remote_profile/input_data
Configuration of job
- job_name:
- type:
str
, optionalargument path:machine[DpCloudServerContext]/remote_profile/input_data/job_name
Job name
- image_name:
- type:
str
, optionalargument path:machine[DpCloudServerContext]/remote_profile/input_data/image_name
Name of the image which run the job, optional when platform is not ali/oss.
- disk_size:
- type:
str
, optionalargument path:machine[DpCloudServerContext]/remote_profile/input_data/disk_size
disk size (GB), optional when platform is not ali/oss.
- scass_type:
- type:
str
argument path:machine[DpCloudServerContext]/remote_profile/input_data/scass_type
machine configuration.
- platform:
- type:
str
argument path:machine[DpCloudServerContext]/remote_profile/input_data/platform
Job run in which platform.
- log_file:
- type:
str
, optionalargument path:machine[DpCloudServerContext]/remote_profile/input_data/log_file
location of log file.
- checkpoint_files:
- type:
list
|str
, optionalargument path:machine[DpCloudServerContext]/remote_profile/input_data/checkpoint_files
location of checkpoint files when it is list type. record file changes when it is string value ‘sync_files’
- checkpoint_time:
- type:
int
, optional, default:15
argument path:machine[DpCloudServerContext]/remote_profile/input_data/checkpoint_time
interval of checkpoint data been stored minimum 15.
- backward_files:
- type:
list
, optionalargument path:machine[DpCloudServerContext]/remote_profile/input_data/backward_files
which files to be uploaded to remote resources. Upload all the files when it is None or empty.
When context_type is set to
HDFSContext
(or its aliaseshdfscontext
,HDFS
,hdfs
):- remote_profile:
- type:
dict
, optionalargument path:machine[HDFSContext]/remote_profile
The information used to maintain the connection with remote machine. This field is empty for this context.
When context_type is set to
LebesgueContext
(or its aliaseslebesguecontext
,Lebesgue
,lebesgue
):- remote_profile:
- type:
dict
argument path:machine[LebesgueContext]/remote_profile
The information used to maintain the connection with remote machine.
- email:
- type:
str
argument path:machine[LebesgueContext]/remote_profile/email
Email
- password:
- type:
str
argument path:machine[LebesgueContext]/remote_profile/password
Password
- program_id:
- type:
int
argument path:machine[LebesgueContext]/remote_profile/program_id
Program ID
- input_data:
- type:
dict
argument path:machine[LebesgueContext]/remote_profile/input_data
Configuration of job
- job_name:
- type:
str
, optionalargument path:machine[LebesgueContext]/remote_profile/input_data/job_name
Job name
- image_name:
- type:
str
, optionalargument path:machine[LebesgueContext]/remote_profile/input_data/image_name
Name of the image which run the job, optional when platform is not ali/oss.
- disk_size:
- type:
str
, optionalargument path:machine[LebesgueContext]/remote_profile/input_data/disk_size
disk size (GB), optional when platform is not ali/oss.
- scass_type:
- type:
str
argument path:machine[LebesgueContext]/remote_profile/input_data/scass_type
machine configuration.
- platform:
- type:
str
argument path:machine[LebesgueContext]/remote_profile/input_data/platform
Job run in which platform.
- log_file:
- type:
str
, optionalargument path:machine[LebesgueContext]/remote_profile/input_data/log_file
location of log file.
- checkpoint_files:
- type:
list
|str
, optionalargument path:machine[LebesgueContext]/remote_profile/input_data/checkpoint_files
location of checkpoint files when it is list type. record file changes when it is string value ‘sync_files’
- checkpoint_time:
- type:
int
, optional, default:15
argument path:machine[LebesgueContext]/remote_profile/input_data/checkpoint_time
interval of checkpoint data been stored minimum 15.
- backward_files:
- type:
list
, optionalargument path:machine[LebesgueContext]/remote_profile/input_data/backward_files
which files to be uploaded to remote resources. Upload all the files when it is None or empty.
When context_type is set to
SSHContext
(or its aliasessshcontext
,SSH
,ssh
):- remote_profile:
- type:
dict
argument path:machine[SSHContext]/remote_profile
The information used to maintain the connection with remote machine.
- hostname:
- type:
str
argument path:machine[SSHContext]/remote_profile/hostname
hostname or ip of ssh connection.
- username:
- type:
str
argument path:machine[SSHContext]/remote_profile/username
username of target linux system
- password:
- type:
str
, optionalargument path:machine[SSHContext]/remote_profile/password
password of linux system
- port:
- type:
int
, optional, default:22
argument path:machine[SSHContext]/remote_profile/port
ssh connection port.
- key_filename:
- type:
NoneType
|str
, optional, default:None
argument path:machine[SSHContext]/remote_profile/key_filename
key filename used by ssh connection. If left None, find key in ~/.ssh or use password for login
- passphrase:
- type:
NoneType
|str
, optional, default:None
argument path:machine[SSHContext]/remote_profile/passphrase
passphrase of key used by ssh connection
- timeout:
- type:
int
, optional, default:10
argument path:machine[SSHContext]/remote_profile/timeout
timeout of ssh connection
- totp_secret:
- type:
NoneType
|str
, optional, default:None
argument path:machine[SSHContext]/remote_profile/totp_secret
Time-based one time password secret. It should be a base32-encoded string extracted from the 2D code.
When context_type is set to
LazyLocalContext
(or its aliaseslazylocalcontext
,LazyLocal
,lazylocal
):- remote_profile:
- type:
dict
, optionalargument path:machine[LazyLocalContext]/remote_profile
The information used to maintain the connection with remote machine. This field is empty for this context.
When context_type is set to
LocalContext
(or its aliaseslocalcontext
,Local
,local
):- remote_profile:
- type:
dict
, optionalargument path:machine[LocalContext]/remote_profile
The information used to maintain the connection with remote machine. This field is empty for this context.