deriva.core.utils package
Submodules
deriva.core.utils.core_utils module
- class deriva.core.utils.core_utils.AttrDict[source]
Bases:
dictDictionary with optional attribute-based lookup.
For keys that are valid attributes, self.key is equivalent to self[key].
- exception deriva.core.utils.core_utils.ConcurrentUpdate[source]
Bases:
ValueError
- exception deriva.core.utils.core_utils.NotModified[source]
Bases:
ValueError
- class deriva.core.utils.core_utils.TimeoutHTTPAdapter(*args, **kwargs)[source]
Bases:
HTTPAdapter- send(request, **kwargs)[source]
Sends PreparedRequest object. Returns Response object.
- Parameters:
request – The
PreparedRequestbeing sent.stream – (optional) Whether to stream the request content.
timeout (float or tuple or urllib3 Timeout object) – (optional) How long to wait for the server to send data before giving up, as a float, or a (connect timeout, read timeout) tuple.
verify – (optional) Either a boolean, in which case it controls whether we verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use
cert – (optional) Any user-provided SSL certificate to be trusted.
proxies – (optional) The proxies dictionary to apply to the request.
- Return type:
requests.Response
- deriva.core.utils.core_utils.calculate_optimal_transfer_shape(size, chunk_limit=10000, requested_chunk_size=26214400, byte_align=65536)[source]
- deriva.core.utils.core_utils.crockford_b32decode(s: str) int[source]
Decode Crockford base-32 string representation to non-negative integer.
- Parameters:
s – String to decode.
The input string is decoded as a sequence of Crockford Base-32 symbols, each encoding 5 bits, such that the first symbol represents the most-signficant bits.
The result is the non-negative integer corresponding to the decoded bit stream.
The Crockford decode process is case-insensitive and recognizes several synonyms for likely typographical errors. Namely, ‘O’->’0’, ‘I’->’1’, and ‘L’->’1’.
Optional hyphens may be present in the input string to break it into symbol groups. These bear no information and are simply ignored.
The optional checksum suffix from Crockford’s proposal is not supported.
- deriva.core.utils.core_utils.crockford_b32encode(v: int, grplen: int = 4) str[source]
Encode a non-negative integer using the Crockford Base-32 representation.
- Parameters:
v – Non-negative integer value to encode.
grplen – Non-negative number of output symbols in each group.
The input integer value is interpreted as an arbitrary-length bit stream of length v.bit_length(). The input integer is zero-extended to make the length a multiple of 5, i.e. effectively prefix-padded with zero bits.
The result is a string uses the Crockford Base-32 representation without any checksum suffix.
Output symbols are separated by a hyphen in groups of grplen symbols. Specify grplen=0 to suppress hyphenation.
This function is the inverse of crockford_b32decode().
Those wishing to encode negative integers must use their own convention to somehow multiplex sign information into the bit stream represented by the non-negative integer.
- deriva.core.utils.core_utils.datetime_to_epoch_microseconds(dt: datetime) int[source]
Convert a datatime to integer microseconds-since-epoch.
- Parameters:
dt – A timezone-aware datetime.datetime instance.
- deriva.core.utils.core_utils.epoch_microseconds_to_datetime(us: int) datetime[source]
Convert integer microseconds-since-epoch to timezone-aware datetime.
- Parameters:
us – Integer microseconds-since-epoch.
- deriva.core.utils.core_utils.format_credential(token=None, oauth2_token=None, username=None, password=None)[source]
- deriva.core.utils.core_utils.get_new_requests_session(url=None, session_config={'allow_retry_on_all_methods': False, 'bypass_cert_verify_host_list': ['localhost'], 'cookie_jar': '/root/.deriva/cookies.txt', 'max_chunk_limit': 10000, 'max_request_size': 104857600, 'retry_backoff_factor': 1.0, 'retry_connect': 2, 'retry_read': 4, 'retry_status_forcelist': [500, 502, 503, 504], 'timeout': (6, 63)})[source]
- deriva.core.utils.core_utils.get_oauth_scopes_for_host(host, config_file='/root/.deriva/config.json', force_refresh=False, warn_on_discovery_failure=False)[source]
- deriva.core.utils.core_utils.init_logging(level=20, log_format=None, file_path=None, file_mode='w', capture_warnings=True, logger_config={'globus_sdk': 30})[source]
- deriva.core.utils.core_utils.int_to_uintX(i: int, nbits: int) int[source]
Cast integer to an unsigned integer of desired width.
- Parameters:
i – Signed integer to encode.
nbits – Number output bits.
For negative inputs, the requested nbits must be equal or greater than i.bit_length(). For non-negative inputs, the requested nbits must be greater than i.bit_length(). The output bits are to be interpreted as 2’s complement, so the most-significant bit is set to represent negative inputs and kept clear to represent non-negative inputs.
This function is the inverse of uintX_to_int() when both are called using the same nbits operand.
- deriva.core.utils.core_utils.read_config(config_file='/root/.deriva/config.json', create_default=False, default={'download_processor_whitelist': [], 'oauth2_scopes': {}, 'server': {'catalog_id': 1, 'host': 'localhost', 'protocol': 'https'}, 'session': {'allow_retry_on_all_methods': False, 'bypass_cert_verify_host_list': ['localhost'], 'cookie_jar': '/root/.deriva/cookies.txt', 'max_chunk_limit': 10000, 'max_request_size': 104857600, 'retry_backoff_factor': 1.0, 'retry_connect': 2, 'retry_read': 4, 'retry_status_forcelist': [500, 502, 503, 504], 'timeout': (6, 63)}})[source]
- deriva.core.utils.core_utils.read_credential(credential_file='/root/.deriva/credential.json', create_default=False, default={})[source]
- deriva.core.utils.core_utils.stob(val)[source]
Convert a string representation of truth to True or False. Lifted and slightly modified from distutils.
True values are ‘y’, ‘yes’, ‘t’, ‘true’, ‘on’, and ‘1’; false values are ‘n’, ‘no’, ‘f’, ‘false’, ‘off’, and ‘0’. Raises ValueError if ‘val’ is anything else.
- deriva.core.utils.core_utils.topo_ranked(depmap: dict[Any, Union[set, collections.abc.Iterable]]) list[set][source]
Return list-of-sets representing values in ranked tiers as a topological partial order.
- Parameters:
depmap – Dictionary mapping of values to required values.
The entire set of values to rank must be represented as keys in depmap, and therefore must be hashable. For each depmap key, the corresponding value should be a set of required values, or an iterable of required values suitable to pass to set(). An empty set or iterable represents a lack of requirements to satisfy for a given key value.
The result list provides a partial order satisfying the requirements from the dependency map. Each entry in the list is a set representing a tier of values with equal rank. Values in a given tier do not require any value from a tier at a higher index.
Raises ValueError if a requirement cannot be satisfied in any order.
- deriva.core.utils.core_utils.topo_sorted(depmap: dict[Any, Union[set, collections.abc.Iterable]]) list[source]
Return list of items topologically sorted.
- Parameters:
depmap – Dictionary mapping of values to required values.
This is a simple wrapper to flatten the partially ordered output of topo_ranked(depmap) into an arbitrary total order.
The entire set of values to sort must be represented as keys in depmap, and therefore must be hashable. For each depmap key, the corresponding value should be a set of required values, or an iterable of required values suitable to pass to set(). An empty set or iterable represents a lack of requirements to satisfy for a given key value.
The result list provides a total order satisfying the requirements from the dependency map. Values at lower indices do not require values at higher indices.
Raises ValueError if a requirement cannot be satisfied in any order.
- deriva.core.utils.core_utils.uintX_to_int(b: int, nbits: int) int[source]
Cast unsigned integer of known width into signed integer.
- Parameters:
b – The non-negative integer holding bits to convert.
nbits – The number of input bits.
The specified input nbits must be equal or greater than i.bit_length(). The input bits are interpreted as 2’s complement, so values with the most-significant bit set are recast as negative numbers while inputs with the highest bit unset remain unchanged.
This function is the inverse of int_to_uintX() when both are called using the same nbits operand.
- deriva.core.utils.core_utils.urlquote(s, safe='')[source]
Quote all reserved characters according to RFC3986 unless told otherwise.
The urllib.urlquote has a weird default which excludes ‘/’ from quoting even though it is a reserved character. We would never want this when encoding elements in Deriva REST API URLs, so this wrapper changes the default to have no declared safe characters.
- deriva.core.utils.core_utils.urlquote_dcctx(s, safe='~{}",:')[source]
Quote for use with Deriva-Client-Context or other HTTP headers.
Defaults to allow additional safe characters for less aggressive encoding of JSON content for use in an HTTP header value.
- deriva.core.utils.core_utils.write_config(config_file='/root/.deriva/config.json', config={'download_processor_whitelist': [], 'oauth2_scopes': {}, 'server': {'catalog_id': 1, 'host': 'localhost', 'protocol': 'https'}, 'session': {'allow_retry_on_all_methods': False, 'bypass_cert_verify_host_list': ['localhost'], 'cookie_jar': '/root/.deriva/cookies.txt', 'max_chunk_limit': 10000, 'max_request_size': 104857600, 'retry_backoff_factor': 1.0, 'retry_connect': 2, 'retry_read': 4, 'retry_status_forcelist': [500, 502, 503, 504], 'timeout': (6, 63)}})[source]
deriva.core.utils.hash_utils module
- deriva.core.utils.hash_utils.compute_file_hashes(file_path, hashes=frozenset({'md5'}))[source]
Digests data read from file denoted by file_path.
- deriva.core.utils.hash_utils.compute_hashes(obj, hashes=frozenset({'md5'}))[source]
Digests input data read from file-like object fd or passed directly as bytes-like object. Compute hashes for multiple algorithms. Default is MD5. Returns a tuple of a hex-encoded digest string and a base64-encoded value suitable for an HTTP header.
deriva.core.utils.mime_utils module
deriva.core.utils.version_utils module
- deriva.core.utils.version_utils.get_installed_version(ver)[source]
Generates an external version string from an internal one. This is currently only being used to determine whether we are operating in a “frozen” environment or not, but other decorators could be added.
- deriva.core.utils.version_utils.is_compatible(source_version, compat_versions)[source]
Compare a source version string to a set of target version string specifications. Supports semantic versioning comparison via setuptools version comparison logic (http://setuptools.readthedocs.io/en/latest/setuptools.html#id7).
- Parameters:
source_version – a source version string
compat_versions – an array of tuples with each tuple consisting of a set of strings of the form <operator><version>. The source_version is evaluated in a conjunction against each <operator><version> string in the tuple, using the packaging module version comparison function. The result of every tuple evaluation is then evaluated in a disjunction against other tuples in the array. If any one of the tuples evaluates to True, then the returned disjunction is logically true, and the source version is assumed to be compatible.
- Returns:
boolean indicating compatibility
Example 1:
is_compatible("0.1.0", [[">=0.1.0", "<1.0.0"]])
- Returns:
True
Example 2:
is_compatible("1.1.0", [[">=0.1.0", "<1.0.0"]])
- Returns:
False
Example 3:
is_compatible("0.5.1-dev", [[">=0.3.5rc4", "<0.5.9-beta", "!=0.5.0"]])
- Returns:
True
Example 4:
is_compatible("0.7.7", [[">=0.7.0", "<0.7.6"], ["!=0.7.1"]])
- Returns:
True (disjunct of “!=0.7.1” cancelled out range “>=0.7.0”, “<0.7.6” – see below)
Example 5:
is_compatible("0.7.7", [[">=0.7.0", "<0.7.6", "!=0.7.1"]])
- Returns:
False
Example 5:
is_compatible("version-3.3", [["==version-3.0"], ["==version-3.1"], ["==version-3.3"]])
- Returns:
True
deriva.core.utils.webauthn_utils module
- deriva.core.utils.webauthn_utils.get_wallet_entries(wallet, credential_type='oauth2', **kwargs)[source]
- Parameters:
wallet – wallet - the wallet to examine (from a Client object)
credential_type – the type of credential requested from the wallet
kwargs – keyword arguments
- Returns:
a list of oauth2 credentials obtained from auth.globus.org with the requested scope
Currently, only “oauth2” is supported as a credential type, and the following keyword args are supported (all of these are optional):
credential_source - where the credentials came from (e.g., “https://auth.globus.org”).
resource_server - the resource server associated with a credential.
scopes - a list of desired scopes.
Example #1:
wallet = client.wallet.extra_values.get("wallet") entries = get_wallet_entries(wallet, "oauth2", resource_server="identifiers.globus.org")
- Returns:
a list of oauth2 credentials associated with the resource server identifiers.globus.org.
Example #2:
entries = get_wallet_entries(wallet, "oauth2", credential_source="https://auth.globus.org")
- Returns:
a list of oauth2 credentials obtained from auth.globus.org.
Example #3:
entries = get_wallet_entries( wallet, "oauth2", credential_source="https://auth.globus.org", scopes=["https://auth.globus.org/scopes/identifiers.globus.org/create_update")
- Returns:
a list of oauth2 credentials obtained from auth.globus.org with the requested scope.
Note
Eventually, we may support wallets with multiple credential types (at the same time) for talking to a variety of remote servers, at which point the implementation will probably become a lot more generalized (with registered wallet providers, etc.).