renate.benchmark.experimentation module#
- renate.benchmark.experimentation.create_cumulative_metrics()[source]#
Gets the cumulative metrics for a given task along with a name of the metric to include in any potential results table.
- Return type:
List
[Tuple
[str
,Callable
]]
- renate.benchmark.experimentation.cumulative_metrics_summary(results, cumulative_metrics, num_tasks, num_instances)[source]#
Creates a pandas DataFrame summary with respect to the observed tasks, specified by
num_tasks
.- Parameters:
results¶ (
Dict
[str
,List
[List
[float
]]]) – The results dictionary holding all the results with respect to all recorded metrics.cumulative_metrics¶ (
List
[Tuple
[str
,Callable
]]) – The list of (name, metric) tuples.num_tasks¶ (
int
) – The total number of tasks.num_instances¶ (
List
[int
]) – Count of test data points for each task.
- Return type:
DataFrame
- renate.benchmark.experimentation.individual_metrics_summary(results, current_task, num_tasks)[source]#
Creates a pandas DataFrame summary for all individual metrics with respect to all observed tasks.
- renate.benchmark.experimentation.execute_experiment_job(backend, config_file, config_space, experiment_outputs_url, mode, metric, num_updates, working_directory='renate_working_dir', requirements_file=None, dependencies=None, role=None, instance_type='ml.c5.xlarge', instance_count=1, instance_max_time=259200, max_time=None, max_num_trials_started=None, max_num_trials_completed=None, max_num_trials_finished=None, n_workers=1, seed=0, accelerator='auto', devices=1, deterministic_trainer=True, gradient_clip_val=None, gradient_clip_algorithm=None, job_name='renate', strategy='ddp', precision='32', save_state=True)[source]#
Executes the experiment job.
- Parameters:
backend¶ (
Literal
['local'
,'sagemaker'
]) – Backend of the experiment job.config_file¶ (
str
) – Path to the Renate config file.config_space¶ (
Dict
[str
,Any
]) – Details for defining your own search space is provided in the Syne Tune Documentation.experiment_outputs_url¶ (
str
) – Path to the experiment outputs.mode¶ (
Literal
['min'
,'max'
]) – Whether to minimize or maximize the metric.metric¶ (
str
) – Metric of the experiment job.num_updates¶ (
int
) – Number of updates of the experiment job.working_directory¶ (
Optional
[str
]) – Path to the working directory.requirements_file¶ (
Optional
[str
]) – Path to the requirements file.dependencies¶ (
Optional
[List
[str
]]) – (SageMaker backend only) List of strings containing absolute or relative paths to files and directories that will be uploaded as part of the SageMaker training job.role¶ (
Optional
[str
]) – Role of the experiment job.instance_type¶ (
str
) – Instance type of the experiment job.instance_count¶ (
int
) – Instance count of the experiment job.instance_max_time¶ (
float
) – Instance max time of the experiment job.max_time¶ (
Optional
[float
]) – Max time of the experiment job.max_num_trials_started¶ (
Optional
[int
]) – Max number of trials started of the experiment job.max_num_trials_completed¶ (
Optional
[int
]) – Max number of trials completed of the experiment job.max_num_trials_finished¶ (
Optional
[int
]) – Max number of trials finished of the experiment job.n_workers¶ (
int
) – Number of workers of the experiment job.seed¶ (
int
) – Seed of the experiment job.accelerator¶ (
Literal
['auto'
,'cpu'
,'gpu'
,'tpu'
]) – Type of accelerator to use.devices¶ (
int
) – Number of devices to use.deterministic_trainer¶ (
bool
) – When true the Trainer adopts a deterministic behaviour also on GPU. In this function this parameter is set to True by default.gradient_clip_val¶ (
Optional
[float
]) – The value at which to clip gradients. Passing None disables it. More detailsgradient_clip_algorithm¶ (
Optional
[str
]) – The gradient clipping algorithm to use. Can be norm or value. More detailsjob_name¶ (
str
) – Name of the experiment job.strategy¶ (
str
) – Name of the distributed training strategy to use. More detailsprecision¶ (
str
) – Type of bit precision to use. More detailssave_state¶ (
bool
) – Flag to retain models and buffer states of each update step. Disable to save storage.
- Return type:
None