renate.benchmark.experimentation module

renate.benchmark.experimentation.experiment_config_file()

renate.benchmark.experimentation.create_cumulative_metrics()

Gets the cumulative metrics for a given task along with a name of the metric to include in any potential results table.

Return type:

List[Tuple[str, Callable]]

renate.benchmark.experimentation.cumulative_metrics_summary(results, cumulative_metrics, num_tasks, num_instances)

Creates a pandas DataFrame summary with respect to the observed tasks, specified by num_tasks.

Parameters:

• results (Dict[str, List[List[float]]]) – The results dictionary holding all the results with respect to all recorded metrics.

• cumulative_metrics (List[Tuple[str, Callable]]) – The list of (name, metric) tuples.

• num_tasks (int) – The total number of tasks.

• num_instances (List[int]) – Count of test data points for each task.

Return type:

DataFrame

renate.benchmark.experimentation.individual_metrics_summary(results, current_task, num_tasks)

Creates a pandas DataFrame summary for all individual metrics with respect to all observed tasks.

Parameters:

• results (Dict[str, List[List[float]]]) – The results dictionary holding all the results with respect to all recorded metrics.

• current_task (int) – The current task ID.

• num_tasks (int) – The total number of tasks.

Return type:

DataFrame

renate.benchmark.experimentation.execute_experiment_job(backend, config_file, config_space, experiment_outputs_url, mode, metric, num_updates, working_directory='renate_working_dir', requirements_file=None, dependencies=None, role=None, instance_type='ml.c5.xlarge', instance_count=1, instance_max_time=259200, max_time=None, max_num_trials_started=None, max_num_trials_completed=None, max_num_trials_finished=None, n_workers=1, seed=0, accelerator='auto', devices=1, deterministic_trainer=True, gradient_clip_val=None, gradient_clip_algorithm=None, job_name='renate', strategy='ddp', precision='32', save_state=True)

Executes the experiment job.

Parameters:

• backend (Literal['local', 'sagemaker']) – Backend of the experiment job.

• config_file (str) – Path to the Renate config file.

• config_space (Dict[str, Any]) – Details for defining your own search space is provided in the Syne Tune Documentation.

• experiment_outputs_url (str) – Path to the experiment outputs.

• mode (Literal['min', 'max']) – Whether to minimize or maximize the metric.

• metric (str) – Metric of the experiment job.

• num_updates (int) – Number of updates of the experiment job.

• working_directory (Optional[str]) – Path to the working directory.

• requirements_file (Optional[str]) – Path to the requirements file.

• dependencies (Optional[List[str]]) – (SageMaker backend only) List of strings containing absolute or relative paths to files and directories that will be uploaded as part of the SageMaker training job.

• role (Optional[str]) – Role of the experiment job.

• instance_type (str) – Instance type of the experiment job.

• instance_count (int) – Instance count of the experiment job.

• instance_max_time (float) – Instance max time of the experiment job.

• max_time (Optional[float]) – Max time of the experiment job.

• max_num_trials_started (Optional[int]) – Max number of trials started of the experiment job.

• max_num_trials_completed (Optional[int]) – Max number of trials completed of the experiment job.

• max_num_trials_finished (Optional[int]) – Max number of trials finished of the experiment job.

• n_workers (int) – Number of workers of the experiment job.

• seed (int) – Seed of the experiment job.

• accelerator (Literal['auto', 'cpu', 'gpu', 'tpu']) – Type of accelerator to use.

• devices (int) – Number of devices to use.

• deterministic_trainer (bool) – When true the Trainer adopts a deterministic behaviour also on GPU. In this function this parameter is set to True by default.

• gradient_clip_val (Optional[float]) – The value at which to clip gradients. Passing None disables it. More details

• gradient_clip_algorithm (Optional[str]) – The gradient clipping algorithm to use. Can be norm or value. More details

• job_name (str) – Name of the experiment job.

• strategy (str) – Name of the distributed training strategy to use. More details

• precision (str) – Type of bit precision to use. More details

• save_state (bool) – Flag to retain models and buffer states of each update step. Disable to save storage.

Return type:

None