renate.memory.storage module#
- renate.memory.storage.mmap_tensor(filename, size, dtype)[source]#
Creates or accesses a memory-mapped tensor.
- Return type:
Tensor
- class renate.memory.storage.Storage(directory)[source]#
Bases:
Dataset
An abstract class for permanent storage of datasets.
- class renate.memory.storage.MemoryMappedTensorStorage(directory)[source]#
Bases:
Storage
A class implementing permanent storage of nested tensor datasets.
This implements storage for
length
data points consisting of nested tensors of fixed types and shapes.Storage
implements__len__
and__getitem__
and therefore can be used as a torchDataset
. To populate the storage, it also implementsdump_dataset
. It does _not_ keep track which slots have or have not been populated.Storage
is given a path to a directory, where it creates (or accesses, if they already exist) memory-mapped tensor files.- Parameters:
- class renate.memory.storage.FileTensorStorage(directory)[source]#
Bases:
Storage
A class implementing permanent storage of nested tensor datasets to disk as pickle files.
This implements storage for
length
data points consisting of nested tensors of fixed types and shapes.Storage
implements__len__
and__getitem__
and therefore can be used as a torchDataset
. To populate the storage, it also implementsdump_dataset
. It does _not_ keep track which slots have or have not been populated.Storage
is given a path to a directory, where it creates (or accesses, if they already exist) pickle files one for each point in the dataset.- Parameters:
directory¶ (
str
) – Path to a directory.