pyslk.transfer package

Submodules

pyslk.transfer.archive module

pyslk.transfer.archive.archive(resource, dst_gns, recursive=False, retry=False, **kwargs) → tuple[list, str]

Upload data to tape archive using slk archive

This archiving function wraps some functionalities around archive_raw(). It returns a list of archived files or directories and can use tenacity for retrying failed archive attempts.

Parameters:

• resource (str or list) – Path, pattern or list of files that should be archived.

• dst_gns (str) – Destination directory for archiving.

• recursive (bool) – Archive recursively.

• retry (bool) – Retry archiving if an ArchiveError is encountered. If retry is True and archiving fails due to ArchiveError, archiving will wait for 10 seconds and retry. Maximal 3 attempts are made. After that, ArchiveError will be raised after all. Requires tenacity to be installed.

Returns:

a tuple containing archived files and stdout from archive_raw.

Return type:

(list, str)

See also

• archive_raw()

pyslk.transfer.recall module

pyslk.transfer.recall.recall_dev(resource: str | int | Path | list[str] | GroupCollection | None = None, search_id: str | int | None = None, search_query: str | None = None, recursive: bool = False, n_parallel_recalls_soft_max: int = 5, proceed_on_error: bool = False, verbose: bool = False, **kwargs) → GroupCollection

kwargs: * n_parallel_recalls_hard_max * split_recalls * wait_time_check_running_jobs (in seconds) * max_tape_number_per_search * wait_until_finished

Parameters:

• resource (str, list, Path) – list of files or a namespaces with files that should be recalled.

• search_id (int, str) – id of a search

• search_query (str) – a search query

• recursive (bool) – do recursive search in the namespaces

• n_parallel_recalls_soft_max (int) – number of parallel recalls to perform

• proceed_on_error (bool) – do not throw an error if a recall job cannot be submitted

• verbose (dict) – verbose mode

• kwargs – keyword arguments

Returns:

a collection of all

Return type:

GroupCollection

pyslk.transfer.retrieve module

pyslk.transfer.retrieve.retrieve(resource: str | list | int, dest_dir: str | None = None, recursive: bool = False, group: bool | None = None, delayed: bool = False, **kwargs) → str | list

Retrieve data from tape archive using slk retrieve

If group is True or resource is a list of files, the retrieve calls will be grouped by tape id to optimize retrieve calls. This functions aims at implementing the recommendations of DKRZ for speeding up retrievals.

Parameters:

• resource (str, list or int) – Path, pattern, SearchID, file or list of files that should be retrieved.

• dest_dir (str) – Destination directory for retrieval. Retrieves to current directory by default.

• recursive (bool) – Retrieve recursively.

• group (bool) – Group files by tape id and make one retrieve call per group. If group is None, retrieve calls will only be grouped if resource is a list of files. To totally avoid grouping, set group to False.

• delayed (bool) – Delay retrieve calls using dask delayed.

Returns:

StdOut from slk calls. If delayed is True, a list of dask.delayed.Delayed objects per tape_id is returned for later (maybe parallel) execution.

Return type:

str or list

See also

• retrieve_raw()

Module contents

pyslk.transfer.archive(resource, dst_gns, recursive=False, retry=False, **kwargs) → tuple[list, str]

Upload data to tape archive using slk archive

This archiving function wraps some functionalities around archive_raw(). It returns a list of archived files or directories and can use tenacity for retrying failed archive attempts.

Parameters:

• resource (str or list) – Path, pattern or list of files that should be archived.

• dst_gns (str) – Destination directory for archiving.

• recursive (bool) – Archive recursively.

• retry (bool) – Retry archiving if an ArchiveError is encountered. If retry is True and archiving fails due to ArchiveError, archiving will wait for 10 seconds and retry. Maximal 3 attempts are made. After that, ArchiveError will be raised after all. Requires tenacity to be installed.

Returns:

a tuple containing archived files and stdout from archive_raw.

Return type:

(list, str)

See also

• archive_raw()

pyslk.transfer.recall_dev(resource: str | int | Path | list[str] | GroupCollection | None = None, search_id: str | int | None = None, search_query: str | None = None, recursive: bool = False, n_parallel_recalls_soft_max: int = 5, proceed_on_error: bool = False, verbose: bool = False, **kwargs) → GroupCollection

kwargs: * n_parallel_recalls_hard_max * split_recalls * wait_time_check_running_jobs (in seconds) * max_tape_number_per_search * wait_until_finished

Parameters:

• resource (str, list, Path) – list of files or a namespaces with files that should be recalled.

• search_id (int, str) – id of a search

• search_query (str) – a search query

• recursive (bool) – do recursive search in the namespaces

• n_parallel_recalls_soft_max (int) – number of parallel recalls to perform

• proceed_on_error (bool) – do not throw an error if a recall job cannot be submitted

• verbose (dict) – verbose mode

• kwargs – keyword arguments

Returns:

a collection of all

Return type:

GroupCollection

pyslk.transfer.retrieve(resource: str | list | int, dest_dir: str | None = None, recursive: bool = False, group: bool | None = None, delayed: bool = False, **kwargs) → str | list

Retrieve data from tape archive using slk retrieve

If group is True or resource is a list of files, the retrieve calls will be grouped by tape id to optimize retrieve calls. This functions aims at implementing the recommendations of DKRZ for speeding up retrievals.

Parameters:

• resource (str, list or int) – Path, pattern, SearchID, file or list of files that should be retrieved.

• dest_dir (str) – Destination directory for retrieval. Retrieves to current directory by default.

• recursive (bool) – Retrieve recursively.

• group (bool) – Group files by tape id and make one retrieve call per group. If group is None, retrieve calls will only be grouped if resource is a list of files. To totally avoid grouping, set group to False.

• delayed (bool) – Delay retrieve calls using dask delayed.

Returns:

StdOut from slk calls. If delayed is True, a list of dask.delayed.Delayed objects per tape_id is returned for later (maybe parallel) execution.

Return type:

str or list

See also

• retrieve_raw()