Create Custom Benchmarks

There are several reasons to create a custom benchmark. For example, you want to use your own dataset, your own model, your own scenario or your own data splits. Creating a benchmark config file is very similar to creating the Renate config file. It is not required to build it from scratch but can be build on top of experiment_config in case you want to reuse parts of the Renate benchmarks.

Your Own Function Arguments

Benchmarking still relies on the functions model_fn and data_module_fn. You can add further arguments as long as you use typing and data of type bool, float, int, str, list, and tuple. A more detailed explanation is provided in the Renate config chapter.

Your Own Model

If you want to use your own model, simply provide the model_fn function as explained in the Renate config chapter.

Your Own Data

As the first step, you need to create a RenateDataModule as described in the Renate config chapter. In order to use it, you additionally need to split the train and test data into partitions using a Scenario. For this purpose, you can use one of the Renate scenarios which we described in the last chapter. For example, you can combine the ClassIncrementalScenario with your data module as follows.

def data_module_fn(
    data_path: str, chunk_id: int, seed: int, class_groupings: Tuple[Tuple[int]]
):
    data_module = CustomDataModule(data_path=data_path, seed=seed)
    return ClassIncrementalScenario(
        data_module=data_module,
        class_groupings=class_groupings,
        chunk_id=chunk_id,
    )

Please note, that data_module_fn takes chunk_id as a special argument which indicates the partition id. The chunk_id will be in range \(0\ldots\text{num_updates}-1\) and indicates the partition that should be loaded as training and validation data.

Your Own Scenario

If you want to use a custom scenario, create your own class extending Scenario. You will need to implement prepare_data() such that it assigns the right subset of your data to self._train_data and self._val_data based on self._chunk_id. Assign a list of datasets to self._test_data where each entry of the list corresponds to one self._chunk_id. At each update step, your model will be evaluated on the test datasets up to self._chunk_id. You can modify this behavior to assign different values to self._test_data. Please check the implementation of IIDScenario for a simple example.