data_helper

to_device

Overview:

Transfer data to certain device

Arguments:

  • item (Any): the item to be transferred

  • device (str): the device wanted

  • ignore_keys (list): the keys to be ignored in transfer, defalut set to empty

Returns:

  • item (Any): the transferred item

to_dtype

Overview:

Change data to certain dtype

Arguments:

  • item (Any): the item to be dtype changed

  • dtype (type): the type wanted

Returns:

  • item (object): the dtype changed item

to_tensor

Overview:

Change numpy.ndarray, sequence of scalars to torch.Tensor, and keep other data types unchanged.

Arguments:

  • item (Any): the item to be changed

  • dtype (type): the type of wanted tensor

Returns:

  • item (torch.Tensor): the change tensor

to_ndarray

Overview:

Change torch.Tensor, sequence of scalars to ndarray, and keep other data types unchanged.

Arguments:

  • item (object): the item to be changed

  • dtype (type): the type of wanted ndarray

Returns:

  • item (object): the changed ndarray

to_list

Overview:

Transform torch.Tensor, numpy.ndarray to list, keep other data types unchanged

Arguments:

  • item (Any): the item to be transformed

Returns:

  • item (list): the list after transformation

Note

Now supports item type: torch.Tensor,:obj:numpy.ndarray, dict, list, tuple and None

tensor_to_list

Overview:

Transform torch.Tensor to list, keep other data types unchanged

Arguments:

  • item (Any): the item to be transformed

Returns:

  • item (list): the list after transformation

Note

Now supports item type: torch.Tensor, dict, list, tuple and None

same_shape

Overview:

Judge whether all data elements in a list have the same shape.

Arguments:

  • data (list): the list of data

Returns:

  • same (bool): whether the list of data all have the same shape

build_log_buffer

Overview:

Builg log buffer, a subclass of dict, which can transform the input data into log format.

Returns:

  • log_buffer (LogDict): Log buffer dict

CudaFetcher

class ding.torch_utils.data_helper.CudaFetcher(data_source: Iterable, device: str, queue_size: int = 4, sleep: float = 0.1)[source]
Overview:

Fetch data from source, and transfer it to specified device.

Interfaces:

run, close

close() → None[source]
Overview:

Stop producer thread by setting end_flag to True.

run() → None[source]
Overview:

Start producer thread: Keep fetching data from source, change the device, and put into queue for request.

get_tensor_data

Overview:

Get pure tensor data from the given data(without disturbing grad computation graph)