utils.file_helper

file_helper

Please Reference ding/ding/utils/file_helper.py for usage.

read_from_ceph

Overview:

Read file from ceph

Arguments:

• path (str): File path in ceph, start with "s3://"

Returns:

• (data): Deserialized data

read_from_redis

Overview:

Read file from redis

Arguments:

• path (str): Dile path in redis, could be a string key

Returns:

• (data): Deserialized data

read_from_rediscluster

Overview:

Read file from rediscluster

Arguments:

• path (str): Dile path in rediscluster, could be a string key

Returns:

• (data): Deserialized data

read_from_mc

Overview:

Read file from memcache, file must be saved by torch.save()

Arguments:

• path (str): File path in local system

Returns:

• (data): Deserialized data

read_from_file

Overview:

Read file from local file system

Arguments:

• path (str): File path in local file system

Returns:

• (data): Deserialized data

read_from_path

Overview:

Read file from ceph

Arguments:

• path (str): File path in ceph, start with "s3://", or use local file system

Returns:

• (data): Deserialized data

save_file_ceph

Overview:

Save pickle dumped data file to ceph

Arguments:

• path (str): File path in ceph, start with "s3://", use file system when not

• data (Any): Could be dict, list or tensor etc.

save_file_redis

Overview:

Save pickle dumped data file to redis

Arguments:

• path (str): File path (could be a string key) in redis

• data (Any): Could be dict, list or tensor etc.

save_file_rediscluster

Overview:

Save pickle dumped data file to rediscluster

Arguments:

• path (str): File path (could be a string key) in redis

• data (Any): Could be dict, list or tensor etc.

read_file

Overview:

Read file from path

Arguments:

• path (str): The path of file to read

• fs_type (str or None): The file system type, support {'normal', 'ceph'}

• use_lock (bool): Whether use_lock is in local normal file system

save_file

Overview:

Save data to file of path

Arguments:

• path (str): The path of file to save to

• data (object): The data to save

• fs_type (str or None): The file system type, support {'normal', 'ceph'}

• use_lock (bool): Whether use_lock is in local normal file system

remove_file

Overview:

Remove file

Arguments:

• path (str): The path of file you want to remove

• fs_type (str or None): The file system type, support {'normal', 'ceph'}