pyslk.raw package

pyslk.raw package#

Submodules#

pyslk.raw.jobs module#

pyslk.raw.jobs.job_exists_raw(job_id: int, return_type: int = 0) → str | int | CompletedProcess#

Check if tape read job exists

Parameters:

job_id (int) – id of a recall job in StrongLink
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.jobs.job_queue_raw(interpret: str | None = None, return_type: int = 0) → str | int | CompletedProcess#

Prints status of the queue of tape read jobs

Parameters:

interpret (str or None) – interpret the queue length; possible values: DETAILS, NUMBER, TEXT, RAW, JSON
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

return job report

Parameters:

job_id (str or int) – a job id
outfile (str or Path) – optional output file if job report should be written into a file
overwrite (bool) – overwrite destination output file if it already exists
return_incomplete_report (bool) – try to write out job report even if job is not finished yet
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.jobs.job_status_raw(job_id: int, return_type: int = 0) → str | int | CompletedProcess#

Check the status of a tape read job with the given ID

Possible status values are AVAILABLE, BLOCKED and ERRORSTATE

Parameters:

job_id (int) – id of a recall job in StrongLink
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.jobs_verify module#

pyslk.raw.jobs_verify.result_verify_job_raw(job_id: int | str, header: bool = False, json: bool = False, json_no_source: bool = False, number_errors: bool = False, number_sources: bool = False, sources: bool = False, return_type: int = 0) → str | int | CompletedProcess#

prints result verification job information

Parameters:

job_id (int or str) – job id
header (bool) – print header
json (bool) – print json
json_no_source (bool) – print json without source
number_errors (bool) – print number of errors
number_sources (bool) – print number of sources
sources (bool) – print sources
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

submit a verify job

Parameters:

resources (str or Path or list[str] or list[Path] or None) – provide a list of resources over which the verify job should be run
resource_ids (str or int or list[str] or list[int] or None) – provide a list of ids of resources over which the verify job should be run
search_id (str or int or None) – provide a search id pointing to files over which the verify job should be run
search_query (str or None) – provide a search query to select files over which the verify job should be run
json_input_file (str or Path or None) – read a JSON file which contains payload of the verify job (admin users only)
recursive (bool) – iterate namespaces recursively
resume_on_page (int) – resume the command and start with search result page resume_on_page; internally, 1000 search results are on one ‘page’ and fetched by one request; you do not necessarily have read permissions for all of these files
results_per_page (int) – number of search results requested per page
end_on_page (int) – end with search result page end_on_page; internally, 1000 search results are on one ‘page’ and fetched by one request; you do not necessarily have read permissions for all of these files
save_mode (bool) – save mode suggested to be used in times of many timeouts; please do not regularly use this parameter; start one verify job per page of search results instead of one verify job for 50 pages of search results
verbose (bool) – verbose output
json (bool) – print output as JSON
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.listing module#

pyslk.raw.listing.list_clone_file_raw(resource_path: str | Path, return_type: int = 0) → str | int | CompletedProcess#

List file information

Parameters:

resource_path (str or Path) – namespace or resource
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

List results from search id and important time stamps

Parameters:

search_id (int or str) – search id of search which results should be printed
only_files (bool) – print only files (like default for slk list)
only_namespaces (bool) – print only namespaces
start (int) – collect search results starting with the result ‘start’; if set to 1 then collect from the beginning
count (int) – Collect as many search results as defined by this parameter. If set to 0 then collect until end.
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.listing.list_raw(path_or_id: str | int | Path, show_hidden: bool = False, numeric_ids: bool = False, recursive: bool = False, text: bool = False, print_bytes: bool = False, return_type: int = 0) → str | int | CompletedProcess#

List results from search id or GNS path

Parameters:

path_or_id (str or int or Path) – search id or gns path
show_hidden (bool) – show ‘.’ files, default: False (don’t show these files)
numeric_ids (bool) – show numeric values for user and group, default: False (show user and group names)
recursive (bool) – use the -R flag to list recursively, default: False
text (bool) – print result to file ‘slk_${USER}_list.txt’, default: False (print to command line / print non-empty return string)
print_bytes (bool) – use the -b to show sizes in bytes, default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.login module#

pyslk.raw.login.hostname_raw(return_type: int = 0) → str | int | CompletedProcess#

Shows current hostname you are connected to

Returns:: stdout of the slk_helpers call
Return type:: Union[str, int, subprocess.CompletedProcess]

pyslk.raw.login.session_raw(return_type: int = 0) → str | int | CompletedProcess#

Shows expiration date of your token

Parameters:: return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)
Returns:: stdout of the slk_helpers call
Return type:: Union[str, int, subprocess.CompletedProcess]

pyslk.raw.metadata module#

pyslk.raw.metadata.hsm2json_raw(resources: str | list | Path | None = None, search_id: int = -1, recursive: bool = False, outfile: str | Path | None = None, restart_file: str | Path | None = None, schema: str | list | None = None, write_json_lines: bool = False, verbose: bool = False, quiet: bool = False, write_mode: str | None = None, print_summary: bool = False, print_json_summary: bool = False, write_compact_json: bool = False, instant_metadata_record_output: bool = False, print_hidden: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Extract metadata from HSM file(s) and return them in JSON structure

Parameters:

resources (str or list or Path-like or None) – list of resources to be searched for [default: None]
search_id (int) – id of a search
recursive (bool) – export metadata from all files in gns_path recursively
outfile (str or Path-like or None) – Write the output into a file instead to the stdout [default: None]
restart_file (str or Path-like or None) – set a restart file in which the processed metadata entries are listed [default: None]
schema (str, list or None) – import only metadata fields of listed schemata; if str: comma-separated list without spaces
write_json_lines (bool = False) – write JSON-lines instead of JSON
verbose (bool = False) – verbose mode
quiet (bool = False) – quiet mode
write_mode (str = None) – applies when ‘output’ is set; possible values: OVERWRITE, ERROR
print_summary (bool = False) – print summary on how many metadata records have been processed [default: False]
print_json_summary (bool = False) – print summary on how many metadata records have been processed as JSON [default: False]
write_compact_json (bool = False) – write JSON metadata in a compact form with less line breaks [default: False]
instant_metadata_record_output (bool) – False (default): read metadata of all files and write/print out afterwards; True: write/print each metadata record after it has been read (requires ‘write_json_lines’)
print_hidden (bool) – print read-only not-searchable metadata fields (sidecar file) [default: False]
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.metadata.json2hsm_raw(json_file: str | Path | None = None, restart_file: str | Path | None = None, schema: str | list | None = None, expect_json_lines: bool = False, verbose: bool = False, quiet: bool = False, ignore_non_existing_metadata_fields: bool = False, write_mode: str | None = None, instant_metadata_record_update: bool = False, use_res_id: bool = False, skip_bad_metadata_sets: bool = False, print_summary: bool = False, print_json_summary: bool = False, json_string: str | None = None, return_type: int = 0) → str | int | CompletedProcess#

Read metadata from JSON file and write them to archived files into HSM. Use absolute paths from metadata records to identify target files.

Parameters:

json_file (str or Path-like or None) – JSON input file containing metadata; incompatible with json_string [default: None]
restart_file (str or Path-like or None,) – set a restart file in which the processed metadata entries are listed [default: None]
schema (str or list) – import only metadata fields of listed schemata; if str: comma-separated list without spaces
expect_json_lines (bool) – read JSON-lines from file instead of JSON
verbose (bool) – verbose mode
quiet (bool) – quiet mode
ignore_non_existing_metadata_fields (bool) – throw no error if metadata field does not exist in StrongLink
write_mode (str) – select write mode for metadata: OVERWRITE, KEEP, ERROR, CLEAN
use_res_id (bool) – use resource_id instead of path to identify file
skip_bad_metadata_sets (bool) – skip damaged / incomplete metadata sets [default: throw error]
instant_metadata_record_update (bool) – False (default): read metadata records of all files and import into StrongLink afterwards; True: import each metadata record after it has been read
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)
print_summary (bool) – print a summary
print_json_summary (bool) – print a summary as JSON
json_string (str) – provide a json string instead of a json file; incompatible with json_file

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.metadata.metadata_raw(resource_path: str | Path, alternative_output_format: bool = False, print_hidden: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Get metadata

Parameters:

resource_path (str or Path) – resource (full path)
alternative_output_format (bool) – print the name of the metadata schema in front of each field name [default: False]
print_hidden (bool) – print read-only not-searchable metadata fields (sidecar file) [default: False]
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Apply metadata to the namespace and child resources

Parameters:

path_or_id (str or int or Path) – search id or gns path of resources to retrieve
metadata (dict) – dict that holds as keys “[metadata schema].[field]” and as values the metadata values
recursive (bool) – use the -R flag to tag recursively, Default: False
display (bool) – print metadata as JSON (ignore “metadata” if this is True), Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.resource module#

pyslk.raw.resource.checksum_raw(resource_path: str | Path, checksum_type: str | None = None, return_type: int = 0) → str | int | CompletedProcess#

Get a checksum of a resource

Parameters:

resource_path (str or Path) – resource (full path)
checksum_type (str) – checksum_type (possible values: None, “sha512”, “adler32”; None => print all)
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Change the access mode of a resource or namespace

Parameters:

gns_path (str or list or Path) – namespace or file (full GNS path); can be file list
mode (str or int) – new mode/permissions of a file (as known from bash’s chmod)
recursive (bool) – use the -R flag to delete recursively, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.resource.delete_raw(gns_path: Path | str | list, recursive: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Soft delete a namespace (optionally all child objects of a non-empty: namespace) or a specific file

Parameters:

gns_path (str or list or Path) – namespace or file (full GNS path); can be file list
recursive (bool) – use the -R flag to delete recursively, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.resource.exists_raw(gns_path: str | Path, return_type: int = 0) → str | int | CompletedProcess#

Check if resource exists

Parameters:

gns_path (str) – namespace or resource
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.resource.group_raw(gns_path: str | Path, group: str | int, recursive: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Change the group of a resource or namespace

Parameters:

gns_path (str or Path) – namespace or file (full GNS path)
group (str or int) – new group of a file (group name or gid)
recursive (bool) – use the -R flag to delete recursively, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Get info whether file is flagged as partial file

Parameters:

resource_path (str or Path or list or None) – resource (full path)
resource_id (str or int) – a resource id
search_id (int or str) – id of a search
recursive (bool) – go through all files in resource_path recursively
verbose (bool) – single verbose mode, Default: False
double_verbose (bool) – double verbose mode, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.resource.mkdir_raw(gns_path: str | Path, recursive: bool = False, parents: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Create a directory

Parameters:

gns_path (str or Path) – gns path to create
recursive (bool) – use the -R create folders recursively; throw error if folder already exists
parents (bool) – create parent folders recursively; throw no error if folder already exists (like ‘mkdir -p’)
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.resource.move_raw(src_path: str | Path, dst_gns: str | Path, interactive: bool, return_type: int = 0) → str | int | CompletedProcess#

Move namespaces/files from one parent folder to another; renaming is not possible

Parameters:

src_path (str or Path) – namespace or file (full GNS path)
dst_gns (str or Path) – new parent namespace
interactive (bool) – prompt user before overwrite; none-interactive: don’t overwrite
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.resource.owner_raw(gns_path: str | Path, owner: str | int, recursive: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Change the owner of a resource or namespace

Parameters:

gns_path (str or Path) – namespace or file (full GNS path)
owner (str or int) – new owner of a file (username or uid)
recursive (bool) – use the -R flag to delete recursively, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.resource.rename_raw(old_name: str | Path, new_name: str | Path, return_type: int = 0) → str | int | CompletedProcess#

Rename a folder or file; moving is not possible

Parameters:

old_name (str or Path) – folder or file name (full GNS path)
new_name (str or Path) – new name (only name; no full GNS path)
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.resource.resource_path_raw(resource_id: str | int, return_type: int = 0) → str | int | CompletedProcess#

Get path for a resource id

Parameters:

resource_id (str or int) – a resource_id
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

Get permissions of a resource provided via resource id or a resource path

Parameters:

resource_path (str or path-like) – resource path (either resource_id or resource_path have to be provided; not both)
resource_id (str or int) – a resource id
as_octal_number (bool) – Do not return the permissions as combination of x/w/r/- but as three digit octal number
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.resource.resource_tape_raw(resource: str, json: bool = False, return_type: int = 0) → str | int | CompletedProcess#

prints resource tape information

Parameters:

resource (str) – resource (full path)
json (bool) – print json
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Get type of resource provided via resource id or a resource path

Parameters:

resource_path (str or path-like or None) – resource path (either resource_id or resource_path have to be provided; not both)
resource_id (str or int) – a resource id
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

Returns file size in byte

Parameters:

resource_path (str or Path or None) – namespace or resource
resource_id (str or int) – a resource id
recursive (bool) – use the -R to calculate size recursively
pad_spaces_left (int) – pad spaces left
verbose (bool) – single verbose mode, Default: False
double_verbose (bool) – double verbose mode, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.search module#

Generates a search query that searches for the listed resources

A search query will be generated which connects all elements of ‘resources’ with and ‘or’. A ‘resource’ (an element of ‘resources’) might be one of these:

a filename with full path (e.g. /arch/bm0146/k204221/INDEX.txt)
a filename without full path (e.g. INDEX.txt)
a regex describing a filename (e.g. /arch/bm0146/k204221/.*.txt)
a namespace (e.g. /arch/bm0146/k204221 or /arch/bm0146/k204221/)

Details are given in the slk_helpers documentation at https://docs.dkrz.de

Parameters:

resources (str or list or Path) – list of resources to be searched for
recursive (bool) – do recursive search in the namespaces
no_newline (bool) – do recursive search in the namespaces
cached_only (bool) – do recursive search in the namespaces
not_cached (bool) – do recursive search in the namespaces
tape_barcodes (list[str]) – do recursive search in the namespaces
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.search.gen_search_query_raw(key_value_pairs: str | list[str], recursive: bool = False, search_query: str | None = None, return_type: int = 0) → str | int | CompletedProcess#

Generates a search query that searches based on the conditions provided as key-values pairs

These key-value pairs are actually key-operator-value-triples. E.g.

path=/arch/bm0146/k204221/iow
resources.size < 1024
resources.created > 2023-01-01

Details are given in the slk_helpers documentation at https://docs.dkrz.de

Parameters:

key_value_pairs (str or list) – list of key-value pairs connected via an operator
recursive (bool) – do recursive search in the namespaces
search_query (str) – an existing search query to be extended
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.search.search_immediately_raw(search_string: str, return_type: int = 0) → str | int | CompletedProcess | Popen#

submits a search query string to StrongLink and immediately returns the search id; search not finished

Parameters:

search_string (str) – JSON search query string
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output), 3 (running subproc.)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess, subprocess.Popen]

pyslk.raw.search.search_incomplete_raw(search_id: str | int, return_type: int = 0) → str | int | CompletedProcess#

Check if search is incomplete

Parameters:

search_id (str or int) – a search_id
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

Creates search and returns search id

Either group, user and/or name can be set or a search_string can be provided.

search_string has to be a valid JSON search string as described in the SLK-CLI manual. Simple double quotes have to be used in the JSON expression (no escaped double quotes, no escaped special characters).

Example for search_string:: search_string=’{“resources.mtime”: {“$gt”: “2021-09-02”}}’

Parameters:

search_string (str) – JSON search query string
group (str or int) – search for files belonging to the provided group name or GID
user (str or int) – search for files belonging to the provided username or UID
name (str) – search files having the provided name
partial (bool) – search for files with ‘partial file’ flag
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.search.search_status_raw(search_id: str | int, json: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Check if search is successful

Parameters:

search_id (str or int) – a search_id
json (bool) – if the output should be in json format
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.search.search_successful_raw(search_id: str | int, return_type: int = 0) → str | int | CompletedProcess#

Check if search is successful

Parameters:

search_id (str or int) – a search_id
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.search.searchid_exists_raw(search_id: str | int, return_type: int = 0) → str | int | CompletedProcess#

Check if search id exists

Parameters:

search_id (str or int) – a search_id
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.search.tnsr_raw(search_id: str | int, quiet: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Print the total number of search results regardless of read permissions of the current user

Parameters:

search_id (str or int) – a search_id
quiet (bool) – should the command be quiet or print warnings?
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.search.total_number_search_results_raw(search_id: str | int, quiet: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Print the total number of search results regardless of read permissions of the current user

Parameters:

search_id (str or int) – a search_id
quiet (bool) – should the command be quiet or print warnings?
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.storage module#

pyslk.raw.storage.group_files_by_tape_raw(resources: str | list | Path | None = None, search_id: int | str | None = None, search_query: str | None = None, recursive: bool = False, print_tape_id: bool = False, print_tape_barcode: bool = False, print_tape_status: bool = False, print_resource_id: bool = False, count_files: bool = False, gen_search_query: bool = False, run_search_query: bool = False, set_max_tape_number_per_search: int = -1, json: bool = False, full: bool = False, details: bool = False, count_tapes: bool = False, return_type: int = 0) → str | int | CompletedProcess | Popen#

groups a list of files based on which tape they are stored on

Receives a list of files or a search id as input. Looks up which files are stored in the HSM-Cache and which are not stored in the HSM-Cache but only on tape. Files on tape are grouped by tape: each line of the output contains all files which are on one tape. The user can directly create a search query for retrieving all files from one tape (–gen-search-query) or directly run this search (–run-search-query). In the latter case, the search id is printed per tape. If the user wants to know the tape id and the tape status, she/he might use –print-tape-id and –print-tape-status, respectively.

Parameters:

resources (str or list or Path-like or None) – list of resources to be searched for
search_id (int or str or None) – id of a search
search_query (str) – a search query
recursive (bool) – do recursive search in the namespaces
print_tape_id (bool) – print the tape id on the far left followed by a ‘:’, Default: False
print_tape_barcode (bool) – print the tape barcode on the far left followed by a ‘:’, Default: False
print_tape_status (bool) – print the status (‘avail’ or ‘blocked’) of the tape of each file group; if print_tape_id is set, this is printed: ‘TAPE_ID, TAPE_STATUS: FILES’; if this is not set, this is printed: TAPE_STATUS: FILES, Default: False
print_resource_id (bool) – print the resource id for each file instead of its path. Is ignored when ‘–gen-search-query’, ‘–run-search-query’, ‘–full’ or ‘–count-files’ are set. Default: False.
count_files (bool) – count files instead of printing file list
gen_search_query (bool) – generate and print (a) search query strings instead of the lists of files per tape, Default: False
run_search_query (bool) – generate and run (a) search query strings instead of the lists of files per tape and print the search i, Default: False
set_max_tape_number_per_search (int) – number of tapes per search; if ‘-1’ => the parameter is not set
json (bool) – return the output as JSON
full (bool) – print useful information and run search query per tape; implies print_tape_barcode, print_tape_status and run_search_query
details (bool) – print useful information; implies print_tape_barcode and print_tape_status
count_tapes (bool) – only count the number of involved tapes and print the number incl. some text
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output), 3 (running subproc.)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess, subprocess.Popen]

See also

group_files_by_tape()

Get info whether file is stored on a tape or not (independent on whether it is stored in the cache)

Parameters:

resource_path (str or Path or None) – resource (full path)
resource_id (str or int or None) – a resource id
search_id (int or str or None) – id of a search
recursive (bool) – export metadata from all files in resource_path recursively
verbose (bool) – single verbose mode, Default: False
double_verbose (bool) – double verbose mode, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Get info whether file is in HSM cache or not

Parameters:

resource_path (str or Path or None) – resource (full path)
resource_id (str or int or None) – a resource id
search_id (int or str or None) – id of a search
recursive (bool) – export metadata from all files in gns_path recursively
verbose (bool) – single verbose mode, Default: False
double_verbose (bool) – double verbose mode, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

prints resource content record (rcr) information

Parameters:

resource_path (str or Path or None) – resource (full path)
resource_id (str or int or None) – a resource id
json (bool) – print json
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.storage.tape_barcode_raw(tape_id: int | str, return_type: int = 0) → str | int | CompletedProcess#

return tape barcode for provided tape id

Parameters:

tape_id (int or str) – id of a tape in the tape library
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Check if tape exists

Parameters:

tape_id (int or str or None) – id of a tape in the tape library
tape_barcode (str or None) – barcode of a tape in the tape library
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.storage.tape_id_raw(tape_barcode: str, return_type: int = 0) → str | int | CompletedProcess#

return tape id for provided tape barcode

Parameters:

tape_barcode (str) – barcode of a tape in the tape library
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Return the name of the library in which the tape is stored

Parameters:

tape_id (int or str or None) – id of a tape in the tape library
tape_barcode (str or None) – barcode of a tape in the tape library
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Check the status of a tape

Parameters:

tape_id (int or str or None) – id of a tape in the tape library
tape_barcode (str or None) – barcode of a tape in the tape library
details (bool) – print a more detailed description of the retrieval status
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.transfer module#

pyslk.raw.transfer.archive_raw(src_path: str | list | Path, dst_gns: str | Path, partsize: int | None = None, streams: int | None = None, stats: bool = False, recursive: bool = False, preserve_permissions: bool = False, exclude_hidden: bool = False, verbose: bool = False, double_verbose: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Upload files in a directory and optionally tag resources: using directory path and GNS path

Parameters:

src_path (str or list or Path) – search id or gns path of resources to retrieve
dst_gns (str or Path) – destination directory for retrieval
partsize (int) – size of each file to download per stream, Default: 500
streams (int) – number of file part streams to use per node, Default: 4
stats (bool) – Display part & I/O stats for each node; Default: False
recursive (bool) – use the -R flag to archive recursively, Default: False
preserve_permissions (bool) – preserve original file permission, Default: False
exclude_hidden (bool) – exclude . (hidden) files, Default: False
verbose (bool) – single verbose mode, Default: False
double_verbose (bool) – double verbose mode, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Recall files from tape to cache via search id or GNS path

Parameters:

path_or_id (str or int or Path) – search id or gns path of resources to recall
recursive (bool) – use the -R flag to recall recursively, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output), 3 (running subproc.)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess, subprocess.Popen, FakeProc]

pyslk.raw.transfer.retrieve_raw(path_or_id: str | int | Path, dest_dir: str | Path, partsize: int = -1, streams: int = -1, stats: bool = False, writeblocksize: int = -1, recursive: bool = False, duplicate: bool = False, preserve_path: bool = True, skip_if_exists: bool = False, return_type: int = 0) → str | int | CompletedProcess | Popen | None#

Retrieve files via search id or GNS path.

Overwrite files if they already exists. Prevent this by ‘duplicate’ or ‘skip_if_exists’.

Parameters:

path_or_id (str or int or Path) – search id or gns path of resources to retrieve
dest_dir (str or Path) – destination directory for retrieval
partsize (int) – size of each file to download per stream, Default: 500 (default is used if not >0)
streams (int) – number of file part streams to use per node, Default: 4 (default is used if not >0)
stats (bool) – Display part and I/O stats for each node
writeblocksize (int) – I/O block size for file writes (KB), Default: 4096 (default is used if not >0)
recursive (bool) – use the -R flag to retrieve recursively, Default: False
duplicate (bool) – create a duplicate file if file exists
preserve_path (bool) – preserve namespace in destination [default: True]
skip_if_exists (bool) – Skip if file exists
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output), 3 (running subproc.)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess, subprocess.Popen] or None

See also

retrieve()

pyslk.raw.versioning module#

pyslk.raw.versioning.version_slk_helpers_raw(return_type: int = 0) → str | int | CompletedProcess#

List the version of slk_helpers

Parameters:: return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)
Returns:: stdout of the slk_helpers call
Return type:: Union[str, int, subprocess.CompletedProcess]

pyslk.raw.versioning.version_slk_raw(return_type: int = 0) → str | int | CompletedProcess#

List the version of slk

Parameters:: return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)
Returns:: stdout of the slk call
Return type:: Union[str, int, subprocess.CompletedProcess]

Module contents#

pyslk.raw.archive_raw(src_path: str | list | Path, dst_gns: str | Path, partsize: int | None = None, streams: int | None = None, stats: bool = False, recursive: bool = False, preserve_permissions: bool = False, exclude_hidden: bool = False, verbose: bool = False, double_verbose: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Upload files in a directory and optionally tag resources: using directory path and GNS path

Parameters:

src_path (str or list or Path) – search id or gns path of resources to retrieve
dst_gns (str or Path) – destination directory for retrieval
partsize (int) – size of each file to download per stream, Default: 500
streams (int) – number of file part streams to use per node, Default: 4
stats (bool) – Display part & I/O stats for each node; Default: False
recursive (bool) – use the -R flag to archive recursively, Default: False
preserve_permissions (bool) – preserve original file permission, Default: False
exclude_hidden (bool) – exclude . (hidden) files, Default: False
verbose (bool) – single verbose mode, Default: False
double_verbose (bool) – double verbose mode, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.checksum_raw(resource_path: str | Path, checksum_type: str | None = None, return_type: int = 0) → str | int | CompletedProcess#

Get a checksum of a resource

Parameters:

resource_path (str or Path) – resource (full path)
checksum_type (str) – checksum_type (possible values: None, “sha512”, “adler32”; None => print all)
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Change the access mode of a resource or namespace

Parameters:

gns_path (str or list or Path) – namespace or file (full GNS path); can be file list
mode (str or int) – new mode/permissions of a file (as known from bash’s chmod)
recursive (bool) – use the -R flag to delete recursively, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.delete_raw(gns_path: Path | str | list, recursive: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Soft delete a namespace (optionally all child objects of a non-empty: namespace) or a specific file

Parameters:

gns_path (str or list or Path) – namespace or file (full GNS path); can be file list
recursive (bool) – use the -R flag to delete recursively, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.exists_raw(gns_path: str | Path, return_type: int = 0) → str | int | CompletedProcess#

Check if resource exists

Parameters:

gns_path (str) – namespace or resource
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Generates a search query that searches for the listed resources

A search query will be generated which connects all elements of ‘resources’ with and ‘or’. A ‘resource’ (an element of ‘resources’) might be one of these:

a filename with full path (e.g. /arch/bm0146/k204221/INDEX.txt)
a filename without full path (e.g. INDEX.txt)
a regex describing a filename (e.g. /arch/bm0146/k204221/.*.txt)
a namespace (e.g. /arch/bm0146/k204221 or /arch/bm0146/k204221/)

Details are given in the slk_helpers documentation at https://docs.dkrz.de

Parameters:

resources (str or list or Path) – list of resources to be searched for
recursive (bool) – do recursive search in the namespaces
no_newline (bool) – do recursive search in the namespaces
cached_only (bool) – do recursive search in the namespaces
not_cached (bool) – do recursive search in the namespaces
tape_barcodes (list[str]) – do recursive search in the namespaces
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.gen_search_query_raw(key_value_pairs: str | list[str], recursive: bool = False, search_query: str | None = None, return_type: int = 0) → str | int | CompletedProcess#

Generates a search query that searches based on the conditions provided as key-values pairs

These key-value pairs are actually key-operator-value-triples. E.g.

path=/arch/bm0146/k204221/iow
resources.size < 1024
resources.created > 2023-01-01

Details are given in the slk_helpers documentation at https://docs.dkrz.de

Parameters:

key_value_pairs (str or list) – list of key-value pairs connected via an operator
recursive (bool) – do recursive search in the namespaces
search_query (str) – an existing search query to be extended
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.group_files_by_tape_raw(resources: str | list | Path | None = None, search_id: int | str | None = None, search_query: str | None = None, recursive: bool = False, print_tape_id: bool = False, print_tape_barcode: bool = False, print_tape_status: bool = False, print_resource_id: bool = False, count_files: bool = False, gen_search_query: bool = False, run_search_query: bool = False, set_max_tape_number_per_search: int = -1, json: bool = False, full: bool = False, details: bool = False, count_tapes: bool = False, return_type: int = 0) → str | int | CompletedProcess | Popen#

groups a list of files based on which tape they are stored on

Parameters:

resources (str or list or Path-like or None) – list of resources to be searched for
search_id (int or str or None) – id of a search
search_query (str) – a search query
recursive (bool) – do recursive search in the namespaces
print_tape_id (bool) – print the tape id on the far left followed by a ‘:’, Default: False
print_tape_barcode (bool) – print the tape barcode on the far left followed by a ‘:’, Default: False
print_tape_status (bool) – print the status (‘avail’ or ‘blocked’) of the tape of each file group; if print_tape_id is set, this is printed: ‘TAPE_ID, TAPE_STATUS: FILES’; if this is not set, this is printed: TAPE_STATUS: FILES, Default: False
print_resource_id (bool) – print the resource id for each file instead of its path. Is ignored when ‘–gen-search-query’, ‘–run-search-query’, ‘–full’ or ‘–count-files’ are set. Default: False.
count_files (bool) – count files instead of printing file list
gen_search_query (bool) – generate and print (a) search query strings instead of the lists of files per tape, Default: False
run_search_query (bool) – generate and run (a) search query strings instead of the lists of files per tape and print the search i, Default: False
set_max_tape_number_per_search (int) – number of tapes per search; if ‘-1’ => the parameter is not set
json (bool) – return the output as JSON
full (bool) – print useful information and run search query per tape; implies print_tape_barcode, print_tape_status and run_search_query
details (bool) – print useful information; implies print_tape_barcode and print_tape_status
count_tapes (bool) – only count the number of involved tapes and print the number incl. some text
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output), 3 (running subproc.)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess, subprocess.Popen]

See also

group_files_by_tape()

pyslk.raw.group_raw(gns_path: str | Path, group: str | int, recursive: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Change the group of a resource or namespace

Parameters:

gns_path (str or Path) – namespace or file (full GNS path)
group (str or int) – new group of a file (group name or gid)
recursive (bool) – use the -R flag to delete recursively, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Get info whether file is flagged as partial file

Parameters:

resource_path (str or Path or list or None) – resource (full path)
resource_id (str or int) – a resource id
search_id (int or str) – id of a search
recursive (bool) – go through all files in resource_path recursively
verbose (bool) – single verbose mode, Default: False
double_verbose (bool) – double verbose mode, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.hostname_raw(return_type: int = 0) → str | int | CompletedProcess#

Shows current hostname you are connected to

Returns:: stdout of the slk_helpers call
Return type:: Union[str, int, subprocess.CompletedProcess]

pyslk.raw.hsm2json_raw(resources: str | list | Path | None = None, search_id: int = -1, recursive: bool = False, outfile: str | Path | None = None, restart_file: str | Path | None = None, schema: str | list | None = None, write_json_lines: bool = False, verbose: bool = False, quiet: bool = False, write_mode: str | None = None, print_summary: bool = False, print_json_summary: bool = False, write_compact_json: bool = False, instant_metadata_record_output: bool = False, print_hidden: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Extract metadata from HSM file(s) and return them in JSON structure

Parameters:

resources (str or list or Path-like or None) – list of resources to be searched for [default: None]
search_id (int) – id of a search
recursive (bool) – export metadata from all files in gns_path recursively
outfile (str or Path-like or None) – Write the output into a file instead to the stdout [default: None]
restart_file (str or Path-like or None) – set a restart file in which the processed metadata entries are listed [default: None]
schema (str, list or None) – import only metadata fields of listed schemata; if str: comma-separated list without spaces
write_json_lines (bool = False) – write JSON-lines instead of JSON
verbose (bool = False) – verbose mode
quiet (bool = False) – quiet mode
write_mode (str = None) – applies when ‘output’ is set; possible values: OVERWRITE, ERROR
print_summary (bool = False) – print summary on how many metadata records have been processed [default: False]
print_json_summary (bool = False) – print summary on how many metadata records have been processed as JSON [default: False]
write_compact_json (bool = False) – write JSON metadata in a compact form with less line breaks [default: False]
instant_metadata_record_output (bool) – False (default): read metadata of all files and write/print out afterwards; True: write/print each metadata record after it has been read (requires ‘write_json_lines’)
print_hidden (bool) – print read-only not-searchable metadata fields (sidecar file) [default: False]
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Get info whether file is stored on a tape or not (independent on whether it is stored in the cache)

Parameters:

resource_path (str or Path or None) – resource (full path)
resource_id (str or int or None) – a resource id
search_id (int or str or None) – id of a search
recursive (bool) – export metadata from all files in resource_path recursively
verbose (bool) – single verbose mode, Default: False
double_verbose (bool) – double verbose mode, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Get info whether file is in HSM cache or not

Parameters:

resource_path (str or Path or None) – resource (full path)
resource_id (str or int or None) – a resource id
search_id (int or str or None) – id of a search
recursive (bool) – export metadata from all files in gns_path recursively
verbose (bool) – single verbose mode, Default: False
double_verbose (bool) – double verbose mode, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.job_exists_raw(job_id: int, return_type: int = 0) → str | int | CompletedProcess#

Check if tape read job exists

Parameters:

job_id (int) – id of a recall job in StrongLink
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.job_queue_raw(interpret: str | None = None, return_type: int = 0) → str | int | CompletedProcess#

Prints status of the queue of tape read jobs

Parameters:

interpret (str or None) – interpret the queue length; possible values: DETAILS, NUMBER, TEXT, RAW, JSON
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

return job report

Parameters:

job_id (str or int) – a job id
outfile (str or Path) – optional output file if job report should be written into a file
overwrite (bool) – overwrite destination output file if it already exists
return_incomplete_report (bool) – try to write out job report even if job is not finished yet
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.job_status_raw(job_id: int, return_type: int = 0) → str | int | CompletedProcess#

Check the status of a tape read job with the given ID

Possible status values are AVAILABLE, BLOCKED and ERRORSTATE

Parameters:

job_id (int) – id of a recall job in StrongLink
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.json2hsm_raw(json_file: str | Path | None = None, restart_file: str | Path | None = None, schema: str | list | None = None, expect_json_lines: bool = False, verbose: bool = False, quiet: bool = False, ignore_non_existing_metadata_fields: bool = False, write_mode: str | None = None, instant_metadata_record_update: bool = False, use_res_id: bool = False, skip_bad_metadata_sets: bool = False, print_summary: bool = False, print_json_summary: bool = False, json_string: str | None = None, return_type: int = 0) → str | int | CompletedProcess#

Read metadata from JSON file and write them to archived files into HSM. Use absolute paths from metadata records to identify target files.

Parameters:

json_file (str or Path-like or None) – JSON input file containing metadata; incompatible with json_string [default: None]
restart_file (str or Path-like or None,) – set a restart file in which the processed metadata entries are listed [default: None]
schema (str or list) – import only metadata fields of listed schemata; if str: comma-separated list without spaces
expect_json_lines (bool) – read JSON-lines from file instead of JSON
verbose (bool) – verbose mode
quiet (bool) – quiet mode
ignore_non_existing_metadata_fields (bool) – throw no error if metadata field does not exist in StrongLink
write_mode (str) – select write mode for metadata: OVERWRITE, KEEP, ERROR, CLEAN
use_res_id (bool) – use resource_id instead of path to identify file
skip_bad_metadata_sets (bool) – skip damaged / incomplete metadata sets [default: throw error]
instant_metadata_record_update (bool) – False (default): read metadata records of all files and import into StrongLink afterwards; True: import each metadata record after it has been read
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)
print_summary (bool) – print a summary
print_json_summary (bool) – print a summary as JSON
json_string (str) – provide a json string instead of a json file; incompatible with json_file

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.list_clone_file_raw(resource_path: str | Path, return_type: int = 0) → str | int | CompletedProcess#

List file information

Parameters:

resource_path (str or Path) – namespace or resource
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

List results from search id and important time stamps

Parameters:

search_id (int or str) – search id of search which results should be printed
only_files (bool) – print only files (like default for slk list)
only_namespaces (bool) – print only namespaces
start (int) – collect search results starting with the result ‘start’; if set to 1 then collect from the beginning
count (int) – Collect as many search results as defined by this parameter. If set to 0 then collect until end.
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.list_raw(path_or_id: str | int | Path, show_hidden: bool = False, numeric_ids: bool = False, recursive: bool = False, text: bool = False, print_bytes: bool = False, return_type: int = 0) → str | int | CompletedProcess#

List results from search id or GNS path

Parameters:

path_or_id (str or int or Path) – search id or gns path
show_hidden (bool) – show ‘.’ files, default: False (don’t show these files)
numeric_ids (bool) – show numeric values for user and group, default: False (show user and group names)
recursive (bool) – use the -R flag to list recursively, default: False
text (bool) – print result to file ‘slk_${USER}_list.txt’, default: False (print to command line / print non-empty return string)
print_bytes (bool) – use the -b to show sizes in bytes, default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.metadata_raw(resource_path: str | Path, alternative_output_format: bool = False, print_hidden: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Get metadata

Parameters:

resource_path (str or Path) – resource (full path)
alternative_output_format (bool) – print the name of the metadata schema in front of each field name [default: False]
print_hidden (bool) – print read-only not-searchable metadata fields (sidecar file) [default: False]
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.mkdir_raw(gns_path: str | Path, recursive: bool = False, parents: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Create a directory

Parameters:

gns_path (str or Path) – gns path to create
recursive (bool) – use the -R create folders recursively; throw error if folder already exists
parents (bool) – create parent folders recursively; throw no error if folder already exists (like ‘mkdir -p’)
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.move_raw(src_path: str | Path, dst_gns: str | Path, interactive: bool, return_type: int = 0) → str | int | CompletedProcess#

Move namespaces/files from one parent folder to another; renaming is not possible

Parameters:

src_path (str or Path) – namespace or file (full GNS path)
dst_gns (str or Path) – new parent namespace
interactive (bool) – prompt user before overwrite; none-interactive: don’t overwrite
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.owner_raw(gns_path: str | Path, owner: str | int, recursive: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Change the owner of a resource or namespace

Parameters:

gns_path (str or Path) – namespace or file (full GNS path)
owner (str or int) – new owner of a file (username or uid)
recursive (bool) – use the -R flag to delete recursively, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

prints resource content record (rcr) information

Parameters:

resource_path (str or Path or None) – resource (full path)
resource_id (str or int or None) – a resource id
json (bool) – print json
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Recall files from tape to cache via search id or GNS path

Parameters:

path_or_id (str or int or Path) – search id or gns path of resources to recall
recursive (bool) – use the -R flag to recall recursively, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output), 3 (running subproc.)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess, subprocess.Popen, FakeProc]

pyslk.raw.rename_raw(old_name: str | Path, new_name: str | Path, return_type: int = 0) → str | int | CompletedProcess#

Rename a folder or file; moving is not possible

Parameters:

old_name (str or Path) – folder or file name (full GNS path)
new_name (str or Path) – new name (only name; no full GNS path)
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.resource_path_raw(resource_id: str | int, return_type: int = 0) → str | int | CompletedProcess#

Get path for a resource id

Parameters:

resource_id (str or int) – a resource_id
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

Get permissions of a resource provided via resource id or a resource path

Parameters:

resource_path (str or path-like) – resource path (either resource_id or resource_path have to be provided; not both)
resource_id (str or int) – a resource id
as_octal_number (bool) – Do not return the permissions as combination of x/w/r/- but as three digit octal number
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.resource_tape_raw(resource: str, json: bool = False, return_type: int = 0) → str | int | CompletedProcess#

prints resource tape information

Parameters:

resource (str) – resource (full path)
json (bool) – print json
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Get type of resource provided via resource id or a resource path

Parameters:

resource_path (str or path-like or None) – resource path (either resource_id or resource_path have to be provided; not both)
resource_id (str or int) – a resource id
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.result_verify_job_raw(job_id: int | str, header: bool = False, json: bool = False, json_no_source: bool = False, number_errors: bool = False, number_sources: bool = False, sources: bool = False, return_type: int = 0) → str | int | CompletedProcess#

prints result verification job information

Parameters:

job_id (int or str) – job id
header (bool) – print header
json (bool) – print json
json_no_source (bool) – print json without source
number_errors (bool) – print number of errors
number_sources (bool) – print number of sources
sources (bool) – print sources
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.retrieve_raw(path_or_id: str | int | Path, dest_dir: str | Path, partsize: int = -1, streams: int = -1, stats: bool = False, writeblocksize: int = -1, recursive: bool = False, duplicate: bool = False, preserve_path: bool = True, skip_if_exists: bool = False, return_type: int = 0) → str | int | CompletedProcess | Popen | None#

Retrieve files via search id or GNS path.

Overwrite files if they already exists. Prevent this by ‘duplicate’ or ‘skip_if_exists’.

Parameters:

path_or_id (str or int or Path) – search id or gns path of resources to retrieve
dest_dir (str or Path) – destination directory for retrieval
partsize (int) – size of each file to download per stream, Default: 500 (default is used if not >0)
streams (int) – number of file part streams to use per node, Default: 4 (default is used if not >0)
stats (bool) – Display part and I/O stats for each node
writeblocksize (int) – I/O block size for file writes (KB), Default: 4096 (default is used if not >0)
recursive (bool) – use the -R flag to retrieve recursively, Default: False
duplicate (bool) – create a duplicate file if file exists
preserve_path (bool) – preserve namespace in destination [default: True]
skip_if_exists (bool) – Skip if file exists
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output), 3 (running subproc.)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess, subprocess.Popen] or None

See also

retrieve()

pyslk.raw.search_immediately_raw(search_string: str, return_type: int = 0) → str | int | CompletedProcess | Popen#

submits a search query string to StrongLink and immediately returns the search id; search not finished

Parameters:

search_string (str) – JSON search query string
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output), 3 (running subproc.)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess, subprocess.Popen]

pyslk.raw.search_incomplete_raw(search_id: str | int, return_type: int = 0) → str | int | CompletedProcess#

Check if search is incomplete

Parameters:

search_id (str or int) – a search_id
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

Creates search and returns search id

Either group, user and/or name can be set or a search_string can be provided.

Example for search_string:: search_string=’{“resources.mtime”: {“$gt”: “2021-09-02”}}’

Parameters:

search_string (str) – JSON search query string
group (str or int) – search for files belonging to the provided group name or GID
user (str or int) – search for files belonging to the provided username or UID
name (str) – search files having the provided name
partial (bool) – search for files with ‘partial file’ flag
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.search_status_raw(search_id: str | int, json: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Check if search is successful

Parameters:

search_id (str or int) – a search_id
json (bool) – if the output should be in json format
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.search_successful_raw(search_id: str | int, return_type: int = 0) → str | int | CompletedProcess#

Check if search is successful

Parameters:

search_id (str or int) – a search_id
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.searchid_exists_raw(search_id: str | int, return_type: int = 0) → str | int | CompletedProcess#

Check if search id exists

Parameters:

search_id (str or int) – a search_id
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.session_raw(return_type: int = 0) → str | int | CompletedProcess#

Shows expiration date of your token

Parameters:: return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)
Returns:: stdout of the slk_helpers call
Return type:: Union[str, int, subprocess.CompletedProcess]

Returns file size in byte

Parameters:

resource_path (str or Path or None) – namespace or resource
resource_id (str or int) – a resource id
recursive (bool) – use the -R to calculate size recursively
pad_spaces_left (int) – pad spaces left
verbose (bool) – single verbose mode, Default: False
double_verbose (bool) – double verbose mode, Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

submit a verify job

Parameters:

resources (str or Path or list[str] or list[Path] or None) – provide a list of resources over which the verify job should be run
resource_ids (str or int or list[str] or list[int] or None) – provide a list of ids of resources over which the verify job should be run
search_id (str or int or None) – provide a search id pointing to files over which the verify job should be run
search_query (str or None) – provide a search query to select files over which the verify job should be run
json_input_file (str or Path or None) – read a JSON file which contains payload of the verify job (admin users only)
recursive (bool) – iterate namespaces recursively
resume_on_page (int) – resume the command and start with search result page resume_on_page; internally, 1000 search results are on one ‘page’ and fetched by one request; you do not necessarily have read permissions for all of these files
results_per_page (int) – number of search results requested per page
end_on_page (int) – end with search result page end_on_page; internally, 1000 search results are on one ‘page’ and fetched by one request; you do not necessarily have read permissions for all of these files
save_mode (bool) – save mode suggested to be used in times of many timeouts; please do not regularly use this parameter; start one verify job per page of search results instead of one verify job for 50 pages of search results
verbose (bool) – verbose output
json (bool) – print output as JSON
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk call

Return type:

Union[str, int, subprocess.CompletedProcess]

Apply metadata to the namespace and child resources

Parameters:

path_or_id (str or int or Path) – search id or gns path of resources to retrieve
metadata (dict) – dict that holds as keys “[metadata schema].[field]” and as values the metadata values
recursive (bool) – use the -R flag to tag recursively, Default: False
display (bool) – print metadata as JSON (ignore “metadata” if this is True), Default: False
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.tape_barcode_raw(tape_id: int | str, return_type: int = 0) → str | int | CompletedProcess#

return tape barcode for provided tape id

Parameters:

tape_id (int or str) – id of a tape in the tape library
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Check if tape exists

Parameters:

tape_id (int or str or None) – id of a tape in the tape library
tape_barcode (str or None) – barcode of a tape in the tape library
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.tape_id_raw(tape_barcode: str, return_type: int = 0) → str | int | CompletedProcess#

return tape id for provided tape barcode

Parameters:

tape_barcode (str) – barcode of a tape in the tape library
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Return the name of the library in which the tape is stored

Parameters:

tape_id (int or str or None) – id of a tape in the tape library
tape_barcode (str or None) – barcode of a tape in the tape library
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

Check the status of a tape

Parameters:

tape_id (int or str or None) – id of a tape in the tape library
tape_barcode (str or None) – barcode of a tape in the tape library
details (bool) – print a more detailed description of the retrieval status
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.tnsr_raw(search_id: str | int, quiet: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Print the total number of search results regardless of read permissions of the current user

Parameters:

search_id (str or int) – a search_id
quiet (bool) – should the command be quiet or print warnings?
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.total_number_search_results_raw(search_id: str | int, quiet: bool = False, return_type: int = 0) → str | int | CompletedProcess#

Print the total number of search results regardless of read permissions of the current user

Parameters:

search_id (str or int) – a search_id
quiet (bool) – should the command be quiet or print warnings?
return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)

Returns:

stdout of the slk_helpers call

Return type:

Union[str, int, subprocess.CompletedProcess]

pyslk.raw.version_slk_helpers_raw(return_type: int = 0) → str | int | CompletedProcess#

List the version of slk_helpers

Parameters:: return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)
Returns:: stdout of the slk_helpers call
Return type:: Union[str, int, subprocess.CompletedProcess]

pyslk.raw.version_slk_raw(return_type: int = 0) → str | int | CompletedProcess#

List the version of slk

Parameters:: return_type (int) – select between 0 (== str output), 1 (== exit code), 2 (subprocess output)
Returns:: stdout of the slk call
Return type:: Union[str, int, subprocess.CompletedProcess]

pyslk.raw package

Contents

pyslk.raw package#

Submodules#

pyslk.raw.jobs module#

pyslk.raw.jobs_verify module#

pyslk.raw.listing module#

pyslk.raw.login module#

pyslk.raw.metadata module#

pyslk.raw.resource module#

pyslk.raw.search module#

pyslk.raw.storage module#

pyslk.raw.transfer module#

pyslk.raw.versioning module#

Module contents#