renate.benchmark.experimentation module#

renate.benchmark.experimentation.experiment_config_file()[source]#
renate.benchmark.experimentation.create_cumulative_metrics()[source]#

Gets the cumulative metrics for a given task along with a name of the metric to include in any potential results table.

Return type:

List[Tuple[str, Callable]]

renate.benchmark.experimentation.cumulative_metrics_summary(results, cumulative_metrics, num_tasks, num_instances)[source]#

Creates a pandas DataFrame summary with respect to the observed tasks, specified by num_tasks.

Parameters:

  • results (Dict[str, List[List[float]]]) – The results dictionary holding all the results with respect to all recorded metrics.

  • cumulative_metrics (List[Tuple[str, Callable]]) – The list of (name, metric) tuples.

  • num_tasks (int) – The total number of tasks.

  • num_instances (List[int]) – Count of test data points for each task.

Return type:

DataFrame

renate.benchmark.experimentation.individual_metrics_summary(results, current_task, num_tasks)[source]#

Creates a pandas DataFrame summary for all individual metrics with respect to all observed tasks.

Parameters:

  • results (Dict[str, List[List[float]]]) – The results dictionary holding all the results with respect to all recorded metrics.

  • current_task (int) – The current task ID.

  • num_tasks (int) – The total number of tasks.

Return type:

DataFrame

renate.benchmark.experimentation.execute_experiment_job(backend, config_file, config_space, experiment_outputs_url, mode, metric, num_updates, working_directory='renate_working_dir', requirements_file=None, dependencies=None, role=None, instance_type='ml.c5.xlarge', instance_count=1, instance_max_time=259200, max_time=None, max_num_trials_started=None, max_num_trials_completed=None, max_num_trials_finished=None, n_workers=1, seed=0, accelerator='auto', devices=1, deterministic_trainer=True, gradient_clip_val=None, gradient_clip_algorithm=None, job_name='renate', strategy='ddp', precision='32', save_state=True)[source]#

Executes the experiment job.

Parameters:

  • backend (Literal['local', 'sagemaker']) – Backend of the experiment job.

  • config_file (str) – Path to the Renate config file.

  • config_space (Dict[str, Any]) – Details for defining your own search space is provided in the Syne Tune Documentation.

  • experiment_outputs_url (str) – Path to the experiment outputs.

  • mode (Literal['min', 'max']) – Whether to minimize or maximize the metric.

  • metric (str) – Metric of the experiment job.

  • num_updates (int) – Number of updates of the experiment job.

  • working_directory (Optional[str]) – Path to the working directory.

  • requirements_file (Optional[str]) – Path to the requirements file.

  • dependencies (Optional[List[str]]) – (SageMaker backend only) List of strings containing absolute or relative paths to files and directories that will be uploaded as part of the SageMaker training job.

  • role (Optional[str]) – Role of the experiment job.

  • instance_type (str) – Instance type of the experiment job.

  • instance_count (int) – Instance count of the experiment job.

  • instance_max_time (float) – Instance max time of the experiment job.

  • max_time (Optional[float]) – Max time of the experiment job.

  • max_num_trials_started (Optional[int]) – Max number of trials started of the experiment job.

  • max_num_trials_completed (Optional[int]) – Max number of trials completed of the experiment job.

  • max_num_trials_finished (Optional[int]) – Max number of trials finished of the experiment job.

  • n_workers (int) – Number of workers of the experiment job.

  • seed (int) – Seed of the experiment job.

  • accelerator (Literal['auto', 'cpu', 'gpu', 'tpu']) – Type of accelerator to use.

  • devices (int) – Number of devices to use.

  • deterministic_trainer (bool) – When true the Trainer adopts a deterministic behaviour also on GPU. In this function this parameter is set to True by default.

  • gradient_clip_val (Optional[float]) – The value at which to clip gradients. Passing None disables it. More details

  • gradient_clip_algorithm (Optional[str]) – The gradient clipping algorithm to use. Can be norm or value. More details

  • job_name (str) – Name of the experiment job.

  • strategy (str) – Name of the distributed training strategy to use. More details

  • precision (str) – Type of bit precision to use. More details

  • save_state (bool) – Flag to retain models and buffer states of each update step. Disable to save storage.

Return type:

None