renate.memory.storage module

renate.memory.storage.mmap_tensor(filename, size, dtype)

Creates or accesses a memory-mapped tensor.

Return type:

Tensor

class renate.memory.storage.Storage(directory)

Bases: Dataset

An abstract class for permanent storage of datasets.

dump_dataset(ds)

Return type:

None

load_dataset(directory)

class renate.memory.storage.MemoryMappedTensorStorage(directory)

Bases: Storage

A class implementing permanent storage of nested tensor datasets.

This implements storage for length data points consisting of nested tensors of fixed types and shapes. Storage implements __len__ and __getitem__ and therefore can be used as a torch Dataset. To populate the storage, it also implements dump_dataset. It does _not_ keep track which slots have or have not been populated.

Storage is given a path to a directory, where it creates (or accesses, if they already exist) memory-mapped tensor files.

Parameters:

• directory (str) – Path to a directory.

• data_point – Prototypical datapoint from which to infer shapes/dtypes.

• length – Number of items to be stored.

dump_dataset(ds)

class renate.memory.storage.FileTensorStorage(directory)

Bases: Storage

A class implementing permanent storage of nested tensor datasets to disk as pickle files.

This implements storage for length data points consisting of nested tensors of fixed types and shapes. Storage implements __len__ and __getitem__ and therefore can be used as a torch Dataset. To populate the storage, it also implements dump_dataset. It does _not_ keep track which slots have or have not been populated.

Storage is given a path to a directory, where it creates (or accesses, if they already exist) pickle files one for each point in the dataset.

Parameters:

directory (str) – Path to a directory.

dump_dataset(ds)

Return type:

None

load_dataset(directory)