deriva.core.utils package

Submodules

deriva.core.utils.core_utils module

class deriva.core.utils.core_utils.AttrDict

Bases: dict

Dictionary with optional attribute-based lookup.

For keys that are valid attributes, self.key is equivalent to self[key].

update([E, ]**F) → None. Update D from dict/iterable E and F.

If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

exception deriva.core.utils.core_utils.ConcurrentUpdate

Bases: ValueError

exception deriva.core.utils.core_utils.NotModified

Bases: ValueError

class deriva.core.utils.core_utils.TimeoutHTTPAdapter(*args, **kwargs)

Bases: requests.adapters.HTTPAdapter

send(request, **kwargs)

Sends PreparedRequest object. Returns Response object.

Parameters:

• request – The PreparedRequest being sent.
• stream – (optional) Whether to stream the request content.
• timeout (float or tuple or urllib3 Timeout object) – (optional) How long to wait for the server to send data before giving up, as a float, or a (connect timeout, read timeout) tuple.
• verify – (optional) Either a boolean, in which case it controls whether we verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use
• cert – (optional) Any user-provided SSL certificate to be trusted.
• proxies – (optional) The proxies dictionary to apply to the request.

Return type:

requests.Response

deriva.core.utils.core_utils.add_logging_level(level_name, level_num, method_name=None)

deriva.core.utils.core_utils.bootstrap(logging_level=20)

deriva.core.utils.core_utils.calculate_optimal_transfer_shape(size, chunk_limit=10000, requested_chunk_size=26214400, byte_align=65536)

deriva.core.utils.core_utils.copy_config(src, dst)

deriva.core.utils.core_utils.force_unicode_py2(s)

Reliably return a Unicode string given a possible unicode or byte string

deriva.core.utils.core_utils.format_credential(token=None, oauth2_token=None, username=None, password=None)

deriva.core.utils.core_utils.format_exception(e)

deriva.core.utils.core_utils.get_new_requests_session(url=None, session_config={'allow_retry_on_all_methods': False, 'cookie_jar': '/root/.deriva/cookies.txt', 'max_chunk_limit': 10000, 'max_request_size': 104857600, 'retry_backoff_factor': 1.0, 'retry_connect': 2, 'retry_read': 4, 'retry_status_forcelist': [500, 502, 503, 504], 'timeout': (6, 63)})

deriva.core.utils.core_utils.get_oauth_scopes_for_host(host, config_file='/root/.deriva/config.json', force_refresh=False, warn_on_discovery_failure=False)

deriva.core.utils.core_utils.get_transfer_summary(total_bytes, elapsed_time)

deriva.core.utils.core_utils.init_logging(level=20, log_format=None, file_path=None, file_mode='w', capture_warnings=True, logger_config={'boto3': 30, 'botocore': 30, 'globus_sdk': 30})

deriva.core.utils.core_utils.json_item_handler(input_file, callback)

deriva.core.utils.core_utils.load_cookies_from_file(cookie_file=None)

deriva.core.utils.core_utils.lock_file(file_path, mode, exclusive=True, timeout=60)

deriva.core.utils.core_utils.make_dirs(path, mode=511)

deriva.core.utils.core_utils.read_config(config_file='/root/.deriva/config.json', create_default=False, default={'download_processor_whitelist': [], 'oauth2_scopes': {}, 'server': {'catalog_id': 1, 'host': 'localhost', 'protocol': 'https'}, 'session': {'allow_retry_on_all_methods': False, 'cookie_jar': '/root/.deriva/cookies.txt', 'max_chunk_limit': 10000, 'max_request_size': 104857600, 'retry_backoff_factor': 1.0, 'retry_connect': 2, 'retry_read': 4, 'retry_status_forcelist': [500, 502, 503, 504], 'timeout': (6, 63)}})

deriva.core.utils.core_utils.read_credential(credential_file='/root/.deriva/credential.json', create_default=False, default={})

deriva.core.utils.core_utils.resource_path(relative_path, default='/deriva-docs/docs-src')

deriva.core.utils.core_utils.stob(string)

deriva.core.utils.core_utils.topo_sorted(depmap)

Return list of items topologically sorted.

depmap: { item: [required_item, …], … }

Raises ValueError if a required_item cannot be satisfied in any order.

The per-item required_item iterables must allow revisiting on multiple iterations.

deriva.core.utils.core_utils.urlquote(s, safe='')

Quote all reserved characters according to RFC3986 unless told otherwise.

The urllib.urlquote has a weird default which excludes ‘/’ from quoting even though it is a reserved character. We would never want this when encoding elements in Deriva REST API URLs, so this wrapper changes the default to have no declared safe characters.

deriva.core.utils.core_utils.urlquote_dcctx(s, safe='~{}", :')

Quote for use with Deriva-Client-Context or other HTTP headers.

Defaults to allow additional safe characters for less aggressive encoding of JSON content for use in an HTTP header value.

deriva.core.utils.core_utils.write_config(config_file='/root/.deriva/config.json', config={'download_processor_whitelist': [], 'oauth2_scopes': {}, 'server': {'catalog_id': 1, 'host': 'localhost', 'protocol': 'https'}, 'session': {'allow_retry_on_all_methods': False, 'cookie_jar': '/root/.deriva/cookies.txt', 'max_chunk_limit': 10000, 'max_request_size': 104857600, 'retry_backoff_factor': 1.0, 'retry_connect': 2, 'retry_read': 4, 'retry_status_forcelist': [500, 502, 503, 504], 'timeout': (6, 63)}})

deriva.core.utils.core_utils.write_credential(credential_file='/root/.deriva/credential.json', credential={})

deriva.core.utils.hash_utils module

deriva.core.utils.hash_utils.compute_file_hashes(file_path, hashes=frozenset({'md5'}))

Digests data read from file denoted by file_path.

deriva.core.utils.hash_utils.compute_hashes(obj, hashes=frozenset({'md5'}))

Digests input data read from file-like object fd or passed directly as bytes-like object. Compute hashes for multiple algorithms. Default is MD5. Returns a tuple of a hex-encoded digest string and a base64-encoded value suitable for an HTTP header.

deriva.core.utils.hash_utils.decodeBase64toHex(base64str)

deriva.core.utils.mime_utils module

deriva.core.utils.mime_utils.add_types(types)

deriva.core.utils.mime_utils.guess_content_type(file_path)

deriva.core.utils.mime_utils.parse_content_disposition(value)

deriva.core.utils.version_utils module

deriva.core.utils.version_utils.get_installed_version(ver)

Generates an external version string from an internal one. This is currently only being used to determine whether we are operating in a “frozen” environment or not, but other decorators could be added.

deriva.core.utils.version_utils.is_compatible(source_version, compat_versions)

Compare a source version string to a set of target version string specifications. Supports semantic versioning comparison via setuptools version comparison logic (http://setuptools.readthedocs.io/en/latest/setuptools.html#id7).

Parameters:

• source_version – a source version string
• compat_versions – an array of tuples with each tuple consisting of a set of strings of the form <operator><version>. The source_version is evaluated in a conjunction against each <operator><version> string in the tuple, using the packaging module version comparison function. The result of every tuple evaluation is then evaluated in a disjunction against other tuples in the array. If any one of the tuples evaluates to True, then the returned disjunction is logically true, and the source version is assumed to be compatible.

Returns:

boolean indicating compatibility

Example 1:

is_compatible("0.1.0", [[">=0.1.0", "<1.0.0"]])

Returns:True

Example 2:

is_compatible("1.1.0", [[">=0.1.0", "<1.0.0"]])

Returns:False

Example 3:

is_compatible("0.5.1-dev", [[">=0.3.5rc4", "<0.5.9-beta", "!=0.5.0"]])

Returns:True

Example 4:

is_compatible("0.7.7", [[">=0.7.0", "<0.7.6"], ["!=0.7.1"]])

Returns:True (disjunct of “!=0.7.1” cancelled out range “>=0.7.0”, “<0.7.6” – see below)

Example 5:

is_compatible("0.7.7", [[">=0.7.0", "<0.7.6", "!=0.7.1"]])

Returns:False

Example 5:

is_compatible("version-3.3", [["==version-3.0"], ["==version-3.1"], ["==version-3.3"]])

Returns:True

deriva.core.utils.webauthn_utils module

deriva.core.utils.webauthn_utils.get_wallet_entries(wallet, credential_type='oauth2', **kwargs)

Parameters:

• wallet – wallet - the wallet to examine (from a Client object)
• credential_type – the type of credential requested from the wallet
• kwargs – keyword arguments

Returns:

a list of oauth2 credentials obtained from auth.globus.org with the requested scope

Currently, only “oauth2” is supported as a credential type, and the following keyword args are supported (all of these are optional):

• credential_source - where the credentials came from (e.g., “https://auth.globus.org”).
• resource_server - the resource server associated with a credential.
• scopes - a list of desired scopes.

Example #1:

wallet = client.wallet.extra_values.get("wallet")
entries = get_wallet_entries(wallet, "oauth2", resource_server="identifiers.globus.org")

Returns:a list of oauth2 credentials associated with the resource server identifiers.globus.org.

Example #2:

entries = get_wallet_entries(wallet, "oauth2", credential_source="https://auth.globus.org")

Returns:a list of oauth2 credentials obtained from auth.globus.org.

Example #3:

entries = get_wallet_entries(
              wallet,
              "oauth2",
              credential_source="https://auth.globus.org",
              scopes=["https://auth.globus.org/scopes/identifiers.globus.org/create_update")

Returns:a list of oauth2 credentials obtained from auth.globus.org with the requested scope.

Note

Eventually, we may support wallets with multiple credential types (at the same time) for talking to a variety of remote servers, at which point the implementation will probably become a lot more generalized (with registered wallet providers, etc.).

Module contents

deriva.core.utils.eprint(*args, **kwargs)