deriva.core package
Subpackages
- deriva.core.utils package
- Submodules
- deriva.core.utils.core_utils module
AttrDict
ConcurrentUpdate
NotModified
TimeoutHTTPAdapter
add_logging_level()
bootstrap()
calculate_optimal_transfer_shape()
copy_config()
format_credential()
format_exception()
get_new_requests_session()
get_oauth_scopes_for_host()
get_transfer_summary()
init_logging()
json_item_handler()
load_cookies_from_file()
lock_file()
make_dirs()
read_config()
read_credential()
resource_path()
stob()
topo_sorted()
urlquote()
urlquote_dcctx()
write_config()
write_credential()
- deriva.core.utils.hash_utils module
- deriva.core.utils.mime_utils module
- deriva.core.utils.version_utils module
- deriva.core.utils.webauthn_utils module
- Module contents
Submodules
deriva.core.annotation module
Definitions and implementation for validating ERMrest schema annotations.
- deriva.core.annotation.validate(model_obj, tag_name=None, validate_model_names=True)[source]
Validate the annotation(s) of the model object.
- Parameters:
model_obj – model object container of annotations
tag_name – tag name of the annotation to validate, if none, will validate all known annotations
validate_model_names – validate model names used in annotations
- Returns:
a list of validation errors, if any
deriva.core.base_cli module
deriva.core.datapath module
Definitions and implementations for data-path expressions to query and manipulate (insert, update, delete).
- class deriva.core.datapath.Array(arg)[source]
Bases:
AggregateFunction
Aggregate function for an array containing all values (including NULL).
- class deriva.core.datapath.ArrayD(arg)[source]
Bases:
AggregateFunction
Aggregate function for an array containing distinct values (including NULL).
- class deriva.core.datapath.Avg(arg)[source]
Bases:
AggregateFunction
Aggregate function for average of non-NULL values.
- class deriva.core.datapath.Bin(arg, nbins, minval=None, maxval=None)[source]
Bases:
AggregateFunction
Binning function.
- class deriva.core.datapath.Cnt(arg)[source]
Bases:
AggregateFunction
Aggregate function for count of non-NULL values.
- class deriva.core.datapath.CntD(arg)[source]
Bases:
AggregateFunction
Aggregate function for count of distinct non-NULL values.
- exception deriva.core.datapath.DataPathException(message, reason=None)[source]
Bases:
Exception
Exception in a datapath expression.
- class deriva.core.datapath.Max(arg)[source]
Bases:
AggregateFunction
Aggregate function for maximum non-NULL value.
deriva.core.deriva_binding module
- class deriva.core.deriva_binding.DerivaBinding(scheme, server, credentials=None, caching=True, session_config=None)[source]
Bases:
object
This is a base-class for implementation purposes. Not useful for clients.
- delete(path, headers={}, guard_response=None)[source]
Perform DELETE request, returning response object.
- Arguments:
path: the path within this bound server headers: headers to set in request guard_response: expected current resource state
as previously seen response object.
Uses guard_response to build appropriate ‘if-match’ header to assure change is only applied to expected state.
Raises ConcurrentUpdate for 412 status.
- get(path, headers={}, raise_not_modified=False, stream=False)[source]
Perform GET request, returning response object.
- Arguments:
path: the path within this bound server headers: headers to set in request raise_not_modified: raise HTTPError for 304 response
status when true.
- stream: whether to defer content retrieval to
streaming access mode on response object.
May consult built-in cache and apply ‘if-none-match’ request header unless input headers already include ‘if-none-match’ or ‘if-match’. On cache hit, returns cached response unless raise_not_modified=true.
Caching of new results is disabled when stream=True.
- head(path, headers={}, raise_not_modified=False)[source]
Perform HEAD request, returning response object.
- Arguments:
path: the path within this bound server headers: headers to set in request raise_not_modified: raise HTTPError for 304 response
status when true.
May consult built-in cache and apply ‘if-none-match’ request header unless input headers already include ‘if-none-match’ or ‘if-match’. On cache hit, returns cached response unless raise_not_modified=true. Cached response may include content retrieved by GET on the same resource.
- post(path, data=None, json=None, headers={})[source]
Perform POST request, returning response object.
- Arguments:
path: the path within this bound server data: a buffer or file-like content value json: data to serialize as JSON content headers: headers to set in request
Raises ConcurrentUpdate for 412 status.
- put(path, data=None, json=None, headers={}, guard_response=None)[source]
Perform PUT request, returning response object.
- Arguments:
path: the path within this bound server data: a buffer or file-like content value json: data to serialize as JSON content headers: headers to set in request guard_response: expected current resource state
as previously seen response object.
Uses guard_response to build appropriate ‘if-match’ header to assure change is only applied to expected state.
Raises ConcurrentUpdate for 412 status.
- class deriva.core.deriva_binding.DerivaClientContext(*args, **kwargs)[source]
Bases:
dict
Represent Deriva-Client-Context header content.
Well-known keys (originally defined for Chaise):
cid: client application ID i.e. program name
wid: window ID i.e. request stream ID
pid: page ID i.e. sub-stream ID
action: UX action embodied by request(s)
table: table upon which application is focused (if any)
uinit: True if request initiated by user action
Default values to use process-wide:
cid
: os.path.basename(sys.argv[0]) if availablewid
: a random UUID
The process-wide defaults MAY be customized by mutating
DerivaClientContext.defaults
prior to constructing instances.
- exception deriva.core.deriva_binding.DerivaPathError[source]
Bases:
ValueError
deriva.core.deriva_server module
deriva.core.ermrest_catalog module
- class deriva.core.ermrest_catalog.DerivaServer(scheme, server, credentials=None, caching=True, session_config=None)[source]
Bases:
DerivaBinding
Persistent handle for a Deriva server.
- connect_ermrest(catalog_id, snaptime=None)[source]
Connect to an ERMrest catalog and return the catalog binding.
- Parameters:
catalog_id – The id (or alias) of the existing catalog
snaptime – The id for a desired catalog snapshot (default None)
The catalog_id is normally a bare id (str), and the optional snaptime is a bare snapshot id (str). If the snaptime is None, the catalog_id may be a concatenated <id>@<snaptime> string, and it will be split to determine the snaptime.
If no snaptime is passed separately or compounded with catalog_id, an ErmrestCatalog binding will be returned. Conversely, if a snaptime is determined, an ErmrestSnapshot (immutable) binding will be returned.
- connect_ermrest_alias(id)[source]
Connect to an ERMrest alias and return the alias binding.
- Parameters:
id – The id of the existing alias
- create_ermrest_alias(id=None, owner=None, alias_target=None, name=None, description=None)[source]
Create an ERMrest catalog alias.
- Parameters:
id – The (str) id desired by the client (default None)
owner – The initial (list of str) ACL desired by the client (default None)
alias_target – The initial target catalog id binding desired by the client (default None)
name – Initial (str) catalog name if not None
description – Initial (str) catalog description if not None
The new alias id will be returned in the response, and used in future alias access. The use of the id parameter may yield errors if the supplied value is not available for use by the client. The value None will result in a server-assigned alias id.
The initial “owner” ACL on the new alias will be the client-supplied owner. The use of owner parameter may yield errors if the supplied ACL does not match the client, i.e. the client cannot lock themselves out of the alias. The value None will result in a server-assigned ACL with the requesting client’s identity.
The alias is bound to the client-supplied alias_target, if supplied. The use of alias_target may yield errors if the supplied value is not a valid target catalog id. The value None will reserve the alias in an unbound state.
Certain failure modes (or message loss) may leave the id reserved in the system. In this case, the effective owner_acl influences which client(s) are allowed to retry creation with the same id.
The name and description parameters are passed through to the alias creation service to initialize those respective metadata fields of the new aliase’s registry entry. See ERMrest documentation for more detail.
- create_ermrest_catalog(id=None, owner=None, name=None, description=None, is_persistent=None, clone_source=None)[source]
Create an ERMrest catalog.
- Parameters:
id – The (str) id desired by the client (default None)
owner – The initial (list of str) ACL desired by the client (default None)
name – Initial (str) catalog name if not None
description – Initial (str) catalog description if not None
is_persistent – Initial (bool) catalog persistence flag if not None
clone_source – Initial catalog clone_source if not None
The new catalog id will be returned in the response, and used in future catalog access. The use of the id parameter may yield errors if the supplied value is not available for use by the client. The value None will result in a server-assigned catalog id.
The initial “owner” ACL on the new catalog will be the client-supplied owner if provided. The use of owner parameter may yield errors if the supplied ACL does not match the client, i.e. the client cannot lock themselves out of the catalog. The value None will result in a server-assigned ACL with the requesting client’s identity.
Certain failure modes (or message loss) may leave the id reserved in the system. In this case, the effective owner ACL influences which client(s) are allowed to retry creation with the same id.
The name, description, is_persistent, and clone_source parameters are passed through to the catalog creation service to initialize those respective metadata fields of the new catalog’s registry entry. See ERMrest documentation for more detail. Authorization failures may occur when attempting to set the is_persistent flag. By default, these fields are not initialized in the catalog creation request, and they instead receive server-assigned defaults.
- class deriva.core.ermrest_catalog.ErmrestAlias(scheme, server, alias_id, credentials=None, caching=True, session_config=None)[source]
Bases:
DerivaBinding
Persistent handle for an ERMrest alias.
Provides basic REST client for HTTP methods on arbitrary paths. Caller has to understand ERMrest APIs and compose appropriate paths, headers, and/or content.
Additional utility methods provided for accessing alias metadata.
- property alias_id
- classmethod connect(deriva_server, alias_id)[source]
Connect to an ERMrest alias and return the alias binding.
- Parameters:
deriva_server – The DerivaServer binding which hosts ermrest
alias_id – The id of the existing alias
The alias_id is a bare id (str).
- classmethod create(deriva_server, id=None, owner=None, alias_target=None, name=None, description=None)[source]
Create an ERMrest catalog alias.
- Parameters:
deriva_server – The DerivaServer binding which hosts ermrest
id – The (str) id desired by the client (default None)
owner – The initial (list of str) ACL desired by the client (default None)
alias_target – The initial target catalog id desired by the client (default None)
name – Initial (str) catalog name if not None
description – Initial (str) catalog description if not None
The new alias id will be returned in the response, and used in future alias access. The use of the id parameter may yield errors if the supplied value is not available for use by the client. The value None will result in a server-assigned alias id.
The initial “owner” ACL on the new alias will be the client-supplied owner parameter. The use of owner may yield errors if the supplied ACL does not match the client, i.e. the client cannot lock themselves out of the alias. The value None will result in a server-assigned ACL with the requesting client’s identity.
The alias is bound to the client-supplied alias_target, if supplied. The use of alias_target may yield errors if the supplied value is not a valid target catalog id. The value None will reserve the alias in an unbound state.
Certain failure modes (or message loss) may leave the id reserved in the system. In this case, the effective owner ACL influences which client(s) are allowed to retry creation with the same id.
The name and description parameters are passed through to the alias creation service to initialize those respective metadata fields of the new aliase’s registry entry. See ERMrest documentation for more detail.
- delete_ermrest_alias(really=False)[source]
Perform DELETE request, destroying alias on server.
- Parameters:
really – delete when True, abort when False (default)
- retrieve()[source]
Retrieve current alias binding state as a dict.
The returned dictionary is suitable for local revision and being passed back into self.update:
state = self.retrieve() state.update({ “owner”: …, “alias_target”: …) self.update(**state)
- update(owner=<deriva.core.ermrest_model.NoChange object>, alias_target=<deriva.core.ermrest_model.NoChange object>, id=None)[source]
Update alias binding state in server, returning the response message dict.
- Parameters:
owner – Revised owner ACL for binding or nochange (default None)
alias_target – Revised target for binding or nochange (default None)
id – Current self.alias_id or None (default None)
The optional id parameter must be None or self.alias_id and does not affect state changes to the server. It is only specified in order to allow an idiom like:
state = self.retrieve() state.update(…) self.update(**state)
where the original “id” field of self.retrieve() is harmlessly passed through as a keyword.
- class deriva.core.ermrest_catalog.ErmrestCatalog(scheme, server, catalog_id, credentials=None, caching=True, session_config=None)[source]
Bases:
DerivaBinding
Persistent handle for an ERMrest catalog.
Provides basic REST client for HTTP methods on arbitrary paths. Caller has to understand ERMrest APIs and compose appropriate paths, headers, and/or content.
Additional utility methods provided for accessing catalog metadata.
- property alias_target
- property catalog_id
- clone_catalog(dst_catalog=None, copy_data=True, copy_annotations=True, copy_policy=True, truncate_after=True, exclude_schemas=None, dst_properties=None)[source]
Clone this catalog’s content into dest_catalog, creating a new catalog if needed.
- Parameters:
dst_catalog – Destination catalog or None to request creation of new destination (default).
copy_data – Copy table contents when True (default).
copy_annotations – Copy annotations when True (default).
copy_policy – Copy access-control policies when True (default).
truncate_after – Truncate destination history after cloning when True (default).
exclude_schemas – A list of schema names to exclude from the cloning process.
dst_properties – A dictionary of custom catalog-creation properties.
When dst_catalog is provided, attempt an idempotent clone, assuming content MAY be partially cloned already using the same parameters. This routine uses a table-level annotation “tag:isrd.isi.edu,2018:clone-state” to save progress markers which help it restart efficiently if interrupted.
When dst_catalog is not provided, a new catalog is provisioned. The optional dst_properties can customize metadata properties during this step:
name: str
description: str (markdown-formatted)
is_persistent: boolean
Cloning preserves source row RID values for application tables so that any RID-based foreign keys are still valid. It is not generally advisable to try to merge more than one source into the same clone, nor to clone on top of rows generated locally in the destination, since this could cause duplicate RID conflicts.
Cloning does not preserve all RID values for special ERMrest tables in the public schema (e.g. ERMrest_Client, ERMrest_Group) but normal applications should only consider the ID key of these tables.
Truncation after cloning avoids retaining incremental snapshots which contain partial clones.
- classmethod connect(deriva_server, catalog_id, snaptime=None)[source]
Connect to an ERMrest catalog and return the catalog binding.
- Parameters:
deriva_server – The DerivaServer binding which hosts ermrest
catalog_id – The id (or alias) of the existing catalog
snaptime – The id for a desired catalog snapshot (default None)
The catalog_id is normally a bare id (str), and the optional snaptime is a bare snapshot id (str). If the snaptime is None, the catalog_id may be a concatenated <id>@<snaptime> string, and it will be split to determine the snaptime.
If no snaptime is passed separately or compounded with catalog_id, an ErmrestCatalog binding will be returned. Conversely, if a snaptime is determined, an ErmrestSnapshot (immutable) binding will be returned.
- classmethod create(deriva_server, id=None, owner=None, name=None, description=None, is_persistent=None, clone_source=None)[source]
Create an ERMrest catalog and return the ERMrest catalog binding.
- Parameters:
deriva_server – The DerivaServer binding which hosts ermrest.
id – The (str) id desired by the client (default None)
owner – The initial (list of str) ACL desired by the client (default None)
name – Initial (str) catalog name if not None
description – Initial (str) catalog description if not None
is_persistent – Initial (bool) catalog persistence flag if not None
clone_source – Initial catalog clone_source if not None
The new catalog id will be returned in the response, and used in future catalog access. The use of the id parameter may yield errors if the supplied value is not available for use by the client. The value None will result in a server-assigned catalog id.
The initial “owner” ACL on the new catalog will be the client-supplied owner ACL. The use of owner parameter may yield errors if the supplied ACL does not match the client, i.e. the client cannot lock themselves out of the catalog. The value None will result in a server-assigned ACL with the requesting client’s identity.
Certain failure modes (or message loss) may leave the id reserved in the system. In this case, the effective owner ACL influences which client(s) are allowed to retry creation with the same id.
The name, description, is_persistent, and clone_source parameters are passed through to the catalog creation service to initialize those respective metadata fields of the new catalog’s registry entry. See ERMrest documentation for more detail. Authorization failures may occur when attempting to set the is_persistent flag. By default, these fields are not initialized in the catalog creation request, and they instead receive server-assigned defaults.
- delete(path, headers={}, guard_response=None)[source]
Perform DELETE request, returning response object.
- Arguments:
path: the path within this bound catalog headers: headers to set in request guard_response: expected current resource state
as previously seen response object.
Uses guard_response to build appropriate ‘if-match’ header to assure change is only applied to expected state.
Raises ConcurrentUpdate for 412 status.
- delete_ermrest_catalog(really=False)[source]
Perform DELETE request, destroying catalog on server.
- Arguments:
really: delete when True, abort when False (default)
- property deriva_server
Return DerivaServer binding for the same server this catalog belongs to.
- exists()[source]
Simple boolean test for catalog existence.
- Returns:
True if exists, False if not (404), otherwise raises exception
- getAsFile(path, destfilename, headers={}, callback=None, delete_if_empty=False, paged=False, page_size=100000, page_sort_columns=frozenset({'RID'}))[source]
Deprecated, call get_as_file instead.
- get_as_file(path, destfilename, headers={}, callback=None, delete_if_empty=False, paged=False, page_size=100000, page_sort_columns=frozenset({'RID'}))[source]
Retrieve catalog data streamed to destination file. Caller is responsible to clean up file even on error, when the file may or may not exist. If “delete_if_empty” is True, the file will be inspected for “empty” content. In the case of json/json-stream content, the presence of a single empty JSON object will be tested for. In the case of CSV content, the file will be parsed with CSV reader to determine that only a single header line and no row data is present.
- resolve_rid(rid: str, model: Model = None, builder: _CatalogWrapper = None) ResolveRidResult [source]
Resolve a RID value to return a ResolveRidResult (a named tuple).
- Parameters:
rid – The RID (str) to resolve
model – A result from self.getCatalogModel() to reuse
builder – A result from self.getPathBuilder() to reuse
Raises KeyError if RID is not found in the catalog.
The elements of the ResolveRidResult namedtuple provide more information about the entity identified by the supplied RID in this catalog:
datapath: datapath instance for querying the resolved entity
table: ermrest_model.Table instance containing the entity
rid: normalized version of the input RID value
Example to simply retrieve entity content:
path, _, _ = catalog.resolve_rid(‘1-0000’) data = path.entities().fetch()[0]
- table_schemas = {}
- class deriva.core.ermrest_catalog.ErmrestSnapshot(scheme, server, catalog_id, snaptime, credentials=None, caching=True, session_config=None)[source]
Bases:
ErmrestCatalog
Persistent handle for an ERMrest catalog snapshot.
Inherits from ErmrestCatalog and provides the same interfaces, except that the interfaces are now bound to a fixed snapshot of the catalog.
- property snaptime
The snaptime for this catalog snapshot instance.
deriva.core.ermrest_config module
deriva.core.ermrest_model module
- class deriva.core.ermrest_model.Column(table, column_doc)[source]
Bases:
object
Named column.
- alter(name=<deriva.core.ermrest_model.NoChange object>, type=<deriva.core.ermrest_model.NoChange object>, nullok=<deriva.core.ermrest_model.NoChange object>, default=<deriva.core.ermrest_model.NoChange object>, comment=<deriva.core.ermrest_model.NoChange object>, acls=<deriva.core.ermrest_model.NoChange object>, acl_bindings=<deriva.core.ermrest_model.NoChange object>, annotations=<deriva.core.ermrest_model.NoChange object>, update_mappings=UpdateMappings.no_update)[source]
Alter existing schema definition.
- Parameters:
name – Replacement column name (default nochange)
type – Replacement Type instance (default nochange)
nullok – Replacement nullok value (default nochange)
default – Replacement default value (default nochange)
comment – Replacement comment (default nochange)
acls – Replacement ACL configuration (default nochange)
acl_bindings – Replacement ACL bindings (default nochange)
annotations – Replacement annotations (default nochange)
update_mappings – Update annotations to reflect changes (default UpdateMappings.no_updates)
Returns self (to allow for optional chained access).
- apply(existing=None)[source]
Apply configuration to corresponding column in catalog unless existing already matches.
- Parameters:
existing – An instance comparable to self, or None to apply configuration unconditionally.
The state of self.comment, self.annotations, self.acls, and self.acl_bindings will be applied to the server unless they match their corresponding state in existing.
- property asset
Convenience property for managing content of object annotation tag:isrd.isi.edu,2017:asset
- property catalog
- clear(clear_comment=False, clear_annotations=True, clear_acls=True, clear_acl_bindings=True)[source]
Clear all configuration in column
NOTE: as a backwards-compatible heuristic, comments are retained by default so that a typical configuration-management client does not strip useful documentation from existing models.
- property column_display
Convenience property for managing content of object annotation tag:isrd.isi.edu,2016:column-display
- classmethod define(cname, ctype, nullok=True, default=None, comment=None, acls={}, acl_bindings={}, annotations={})[source]
Build a column definition.
- property display
Convenience property for managing content of object annotation tag:misd.isi.edu,2015:display
- drop(cascade=False, update_mappings=UpdateMappings.no_update)[source]
Remove this column from the remote database.
- Parameters:
cascade – drop dependent objects (default False)
update_mappings – Update annotations to reflect changes (default UpdateMappings.no_updates)
- property generated
Convenience property for managing presence of annotation tag:isrd.isi.edu,2016:generated
- property immutable
Convenience property for managing presence of annotation tag:isrd.isi.edu,2016:immutable
- prejson(prune=True)[source]
Produce a representation of configuration as generic Python data structures
- property required
Convenience property for managing presence of annotation tag:isrd.isi.edu,2018:required
- property uri_path
URI to this model resource.
- class deriva.core.ermrest_model.FindAssociationResult(table, self_fkey, other_fkeys)[source]
Bases:
object
Wrapper for results of Table.find_associations()
- class deriva.core.ermrest_model.ForeignKey(table, fkey_doc)[source]
Bases:
object
Named foreign key.
- alter(constraint_name=<deriva.core.ermrest_model.NoChange object>, on_update=<deriva.core.ermrest_model.NoChange object>, on_delete=<deriva.core.ermrest_model.NoChange object>, comment=<deriva.core.ermrest_model.NoChange object>, acls=<deriva.core.ermrest_model.NoChange object>, acl_bindings=<deriva.core.ermrest_model.NoChange object>, annotations=<deriva.core.ermrest_model.NoChange object>, update_mappings=UpdateMappings.no_update)[source]
Alter existing schema definition.
- Parameters:
constraint_name – Replacement constraint name string
on_update – Replacement on-update action string
on_delete – Replacement on-delete action string
comment – Replacement comment (default nochange)
acls – Replacement ACL configuration (default nochange)
acl_bindings – Replacement ACL bindings (default nochange)
annotations – Replacement annotations (default nochange)
update_mappings – Update annotations to reflect changes (default UpdateMappings.no_updates)
Returns self (to allow for optional chained access).
- apply(existing=None)[source]
Apply configuration to corresponding foreign key in catalog unless existing already matches.
- Parameters:
existing – An instance comparable to self, or None to apply configuration unconditionally.
The state of self.comment, self.annotations, self.acls, and self.acl_bindings will be applied to the server unless they match their corresponding state in existing.
- property catalog
- clear(clear_comment=False, clear_annotations=True, clear_acls=True, clear_acl_bindings=True)[source]
Clear all configuration in foreign key
NOTE: as a backwards-compatible heuristic, comments are retained by default so that a typical configuration-management client does not strip useful documentation from existing models.
- property column_map
Mapping of foreign_key_columns elements to referenced_columns elements.
- property columns
Sugared access to self.column_definitions
- classmethod define(fk_colnames, pk_sname, pk_tname, pk_colnames, on_update='NO ACTION', on_delete='NO ACTION', constraint_names=[], comment=None, acls={}, acl_bindings={}, annotations={}, constraint_name=None)[source]
Define a foreign key.
- Parameters:
fk_colnames – List of column names participating in the foreign key
pk_sname – Schema name string of the referenced primary key
pk_tname – Table name string of the referenced primary key
pk_colnames – List of column names participating in the referenced primary key
on_update – Constraint behavior when referenced primary keys are updated
on_update – Constraint behavior when referenced primary keys are deleted
constraint_names – Legacy input [ [ schema_name, constraint_name ] ] (for API backwards-compatibility)
comment – Comment string
acls – Dictionary of { acl_name: acl, … }
acl_bindings – Dictionary of { binding_name: acl_binding, … }
annotations – Dictionary of { annotation_uri: annotation_value, … }
constraint_name – Constraint name string
The contraint behavior values for on_update and on_delete must be one of the following literal strings:
‘NO ACTION’, ‘RESTRICT’, ‘CASCADE’, ‘SET NULL’, ‘SET DEFAULT’
The constraint_name kwarg takes a bare constraint name string and acts the same as setting the legacy constraint_names kwarg to: [ [ “placeholder”, constraint_name ] ]. This odd syntax is for backwards-compatibility with earlier API versions, and mirrors the structure of constraint names in ERMrest model description outputs. In those outputs, the “placeholder” field contains the schema name of the table containing the constraint.
- digest_referenced_columns(model)[source]
Finish construction deferred until model is known with all tables.
- drop(update_mappings=UpdateMappings.no_update)[source]
Remove this foreign key from the remote database.
- Parameters:
update_mappings – Update annotations to reflect changes (default UpdateMappings.no_updates)
- property foreign_key
Convenience property for managing content of object annotation tag:isrd.isi.edu,2016:foreign-key
- property name
Constraint name (schemaobj, name_str) used in API dictionaries.
- name_in_model(model)[source]
Constraint name (schemaobj, name_str) used in API dictionaries fetching schema from model.
While self.name works as a key within the same model tree, self.name_in_model(dstmodel) works in dstmodel tree by finding the equivalent schemaobj in that model via schema name lookup.
- property names
Constraint names field as seen in JSON document.
- prejson(prune=True)[source]
Produce a representation of configuration as generic Python data structures
- property uri_path
URI to this model resource.
- class deriva.core.ermrest_model.Key(table, key_doc)[source]
Bases:
object
Named key.
- alter(constraint_name=<deriva.core.ermrest_model.NoChange object>, comment=<deriva.core.ermrest_model.NoChange object>, annotations=<deriva.core.ermrest_model.NoChange object>, update_mappings=UpdateMappings.no_update)[source]
Alter existing schema definition.
- Parameters:
constraint_name – Unqualified constraint name string
comment – Replacement comment (default nochange)
annotations – Replacement annotations (default nochange)
update_mappings – Update annotations to reflect changes (default UpdateMappings.no_updates)
Returns self (to allow for optional chained access).
- apply(existing=None)[source]
Apply configuration to corresponding key in catalog unless existing already matches.
- Parameters:
existing – An instance comparable to self, or None to apply configuration unconditionally.
The state of self.comment and self.annotations will be applied to the server unless they match their corresponding state in existing.
- property catalog
- clear(clear_comment=False, clear_annotations=True)[source]
Clear all configuration in key
NOTE: as a backwards-compatible heuristic, comments are retained by default so that a typical configuration-management client does not strip useful documentation from existing models.
- property columns
Sugared access to self.unique_columns
- classmethod define(colnames, constraint_names=[], comment=None, annotations={}, constraint_name=None)[source]
Build a key definition.
- Parameters:
colnames – List of names of columns participating in the key
constraint_names – Legacy input [ [ schema_name, constraint_name ] ] (for API backwards-compatibility)
comment – Comment string
annotations – Dictionary of { annotation_uri: annotation_value, … }
constraint_name – Constraint name string
The constraint_name kwarg takes a bare constraint name string and acts the same as setting the legacy constraint_names kwarg to: [ [ “placeholder”, constraint_name ] ]. This odd syntax is for backwards-compatibility with earlier API versions, and mirrors the structure of constraint names in ERMrest model description outputs. In those outputs, the “placeholder” field contains the schema name of the table containing the constraint.
- drop(cascade=False, update_mappings=UpdateMappings.no_update)[source]
Remove this key from the remote database.
- Parameters:
cascade – drop dependent objects (default False)
update_mappings – Update annotations to reflect changes (default UpdateMappings.no_updates)
- property name
Constraint name (schemaobj, name_str) used in API dictionaries.
- name_in_model(model)[source]
Constraint name (schemaobj, name_str) used in API dictionaries fetching schema from model.
While self.name works as a key within the same model tree, self.name_in_model(dstmodel) works in dstmodel tree by finding the equivalent schemaobj in that model via schema name lookup.
- property names
Constraint names field as seen in JSON document.
- prejson(prune=True)[source]
Produce a representation of configuration as generic Python data structures
- property uri_path
URI to this model resource.
- class deriva.core.ermrest_model.Model(catalog, model_doc)[source]
Bases:
object
Top-level catalog model.
- apply(existing=None)[source]
Apply catalog configuration to catalog unless existing already matches.
- Parameters:
existing – An instance comparable to self.
The configuration in self will be applied recursively to the corresponding model nodes in schema.
If existing is not provided (default), the current whole configuration will be retrieved from the catalog and used automatically to determine whether the configuration goals under this Model tree are already met or need to be remotely applied.
- property bulk_upload
Convenience property for managing content of object annotation tag:isrd.isi.edu,2017:bulk-upload
- property catalog
- property chaise_config
Convenience property for managing content of object annotation tag:isrd.isi.edu,2019:chaise-config
- clear(clear_comment=False, clear_annotations=True, clear_acls=True, clear_acl_bindings=True)[source]
Clear all configuration in catalog and children.
NOTE: as a backwards-compatible heuristic, comments are retained by default so that a typical configuration-management client does not strip useful documentation from existing models.
- property column_defaults
Convenience property for managing content of object annotation tag:isrd.isi.edu,2023:column-defaults
- configure_baseline_catalog(apply=True, **kwargs)[source]
A baseline catalog configuration.
Update catalog to a baseline configuration: 1. Set default display mode to turn underscores to spaces in model element names. 2. Configure ERMrest_Client and ERMrest_Group to have readable names. 3. Create a schema WWW with Page and File tables in that schema configured to display web-page like
content.
Configure a basic navbar with links to all tables.
Afterwards, an ACL configuration should be applied to the catalog. See the deriva.config.examples package data for configuration templates.
- Parameters:
apply – if true, apply configuration changes before returning.
kwargs – a set of name-value pairs used to override default settings.
- configure_baseline_ermrest_client(apply=True)[source]
Baseline configuration of ERMrest_Client table.
Set up ERMrest_Client table so that it has readable names and uses the display name of the user as the row name.
- Parameters:
apply – if true, apply configuration changes before returning.
- configure_baseline_ermrest_group(apply=True)[source]
Baseline configuration of ERMrest_Group table.
Set up ERMrest_Group table so that it has readable names and uses the display name of the group as the row name.
- Parameters:
apply – if true, apply configuration changes before returning.
- create_schema(schema_def)[source]
Add a new schema to this model in the remote database based on schema_def.
Returns a new Schema instance based on the server-supplied representation of the newly created schema.
The returned Schema is also added to self.schemas.
- digest_fkeys()[source]
Finish second-pass digestion of foreign key definitions using full model w/ all schemas and tables.
- property display
Convenience property for managing content of object annotation tag:misd.isi.edu,2015:display
- property export_2019
Convenience property for managing content of object annotation tag:isrd.isi.edu,2019:export
- property export_fragment_definitions
Convenience property for managing content of object annotation tag:isrd.isi.edu,2021:export-fragment-definitions
- fkey(constraint_name_pair)[source]
Return configuration for foreign key with given name pair.
Accepts (schema_name, constraint_name) pairs as found in many faceting annotations and (schema_obj, constraint_name) pairs as found in fkey.name fields.
- classmethod fromfile(catalog, schema_file)[source]
Deserialize a JSON schema file as a Model management object.
- property generated
Convenience property for managing presence of annotation tag:isrd.isi.edu,2016:generated
- property immutable
Convenience property for managing presence of annotation tag:isrd.isi.edu,2016:immutable
- property non_deletable
Convenience property for managing presence of annotation tag:isrd.isi.edu,2016:non-deletable
- prejson(prune=True)[source]
Produce a representation of configuration as generic Python data structures
- property uri_path
URI to this model resource.
- class deriva.core.ermrest_model.NoChange[source]
Bases:
object
Special class used to distinguish no-change default arguments to methods.
Values for no-change are distinct from all valid values for
these arguments.
- class deriva.core.ermrest_model.Quantifier(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]
-
Logic quantifiers
- all = 'all'
- any = 'any'
- class deriva.core.ermrest_model.Schema(model, sname, schema_doc)[source]
Bases:
object
Named schema.
- alter(schema_name=<deriva.core.ermrest_model.NoChange object>, comment=<deriva.core.ermrest_model.NoChange object>, acls=<deriva.core.ermrest_model.NoChange object>, annotations=<deriva.core.ermrest_model.NoChange object>, update_mappings=UpdateMappings.no_update)[source]
Alter existing schema definition.
- Parameters:
schema_name – Replacement schema name (default nochange)
comment – Replacement comment (default nochange)
acls – Replacement ACL configuration (default nochange)
annotations – Replacement annotations (default nochange)
update_mappings – Update annotations to reflect changes (default UpdateMappings.no_updates)
Returns self (to allow for optional chained access).
- apply(existing=None)[source]
Apply configuration to corresponding schema in catalog unless existing already matches.
- Parameters:
existing – An instance comparable to self, or None to apply configuration unconditionally.
The state of self.comment, self.annotations, and self.acls will be applied to the server unless they match their corresponding state in existing.
- property catalog
- clear(clear_comment=False, clear_annotations=True, clear_acls=True, clear_acl_bindings=True)[source]
Clear all configuration in schema and children.
NOTE: as a backwards-compatible heuristic, comments are retained by default so that a typical configuration-management client does not strip useful documentation from existing models.
- property column_defaults
Convenience property for managing content of object annotation tag:isrd.isi.edu,2023:column-defaults
- create_table(table_def)[source]
Add a new table to this schema in the remote database based on table_def.
Returns a new Table instance based on the server-supplied representation of the newly created table.
The returned Table is also added to self.tables.
- classmethod define(sname, comment=None, acls={}, annotations={})[source]
Build a schema definition.
- classmethod define_www(sname, comment=None, acls={}, annotations={})[source]
Build a schema definition for wiki-like web content.
Defines a schema with a “Page” wiki-like page table definition and a “File” asset table definition for attachments to the wiki pages.
- Parameters:
sname – schema name
comment – a comment string for the table
acls – a dictionary of ACLs for specific access modes
annotations – a dictionary of annotations
- property display
Convenience property for managing content of object annotation tag:misd.isi.edu,2015:display
- drop(cascade=False, update_mappings=UpdateMappings.no_update)[source]
Remove this schema from the remote database.
- Parameters:
cascade – drop dependent objects.
update_mappings – Update annotations to reflect changes (default UpdateMappings.no_updates)
- property export_2019
Convenience property for managing content of object annotation tag:isrd.isi.edu,2019:export
- property export_fragment_definitions
Convenience property for managing content of object annotation tag:isrd.isi.edu,2021:export-fragment-definitions
- property generated
Convenience property for managing presence of annotation tag:isrd.isi.edu,2016:generated
- property immutable
Convenience property for managing presence of annotation tag:isrd.isi.edu,2016:immutable
- property non_deletable
Convenience property for managing presence of annotation tag:isrd.isi.edu,2016:non-deletable
- prejson(prune=True)[source]
Produce native Python representation of schema, suitable for JSON serialization.
- property uri_path
URI to this model resource.
- class deriva.core.ermrest_model.Table(schema, tname, table_doc)[source]
Bases:
object
Named table.
- alter(schema_name=<deriva.core.ermrest_model.NoChange object>, table_name=<deriva.core.ermrest_model.NoChange object>, comment=<deriva.core.ermrest_model.NoChange object>, acls=<deriva.core.ermrest_model.NoChange object>, acl_bindings=<deriva.core.ermrest_model.NoChange object>, annotations=<deriva.core.ermrest_model.NoChange object>, update_mappings=UpdateMappings.no_update)[source]
Alter existing schema definition.
- Parameters:
schema_name – Destination schema name (default nochange)
table_name – Replacement table name (default nochange)
comment – Replacement comment (default nochange)
acls – Replacement ACL configuration (default nochange)
acl_bindings – Replacement ACL bindings (default nochange)
annotations – Replacement annotations (default nochange)
update_mappings – Update annotations to reflect changes (default UpdateMappings.no_updates)
A change of schema name is a transfer of the existing table to an existing destination schema (not a rename of the current containing schema).
Returns self (to allow for optional chained access).
- property alternatives
Convenience property for managing content of object annotation tag:isrd.isi.edu,2016:table-alternatives
- apply(existing=None)[source]
Apply configuration to corresponding table in catalog unless existing already matches.
- Parameters:
existing – An instance comparable to self, or None to apply configuration unconditionally.
The state of self.comment, self.annotations, self.acls, and self.acl_bindings will be applied to the server unless they match their corresponding state in existing.
- property catalog
- property citation
Convenience property for managing content of object annotation tag:isrd.isi.edu,2018:citation
- clear(clear_comment=False, clear_annotations=True, clear_acls=True, clear_acl_bindings=True)[source]
Clear all configuration in table and children.
NOTE: as a backwards-compatible heuristic, comments are retained by default so that a typical configuration-management client does not strip useful documentation from existing models.
- property column_defaults
Convenience property for managing content of object annotation tag:isrd.isi.edu,2023:column-defaults
- property columns
Sugared access to self.column_definitions
- create_column(column_def: dict) Column [source]
Add a new column to this table in the remote database based on column_def.
Returns a new Column instance based on the server-supplied representation of the new column, and adds it to self.column_definitions too.
- create_fkey(fkey_def: dict) ForeignKey [source]
Add a new foreign key to this table in the remote database based on fkey_def.
Returns a new ForeignKey instance based on the server-supplied representation of the new foreign key, and adds it to self.foreign_keys too.
- create_key(key_def: dict) Key [source]
Add a new key to this table in the remote database based on key_def.
Returns a new Key instance based on the server-supplied representation of the new key, and adds it to self.keys too.
- create_reference(target: deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table | tuple[str, bool, deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table] | tuple[str, deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table], key_column_search_order: collections.abc.Iterable[str] | None = None) tuple[list[deriva.core.ermrest_model.Column], deriva.core.ermrest_model.ForeignKey] [source]
Add column(s) and a foreign key to this table in the remote database for a reference target.
See Table.define() documentation for more about reference targets.
Returns a list of new Column instances and a ForeignKey instance based on the server-supplied representation of the new model elements, and adds them to the self.columns and self.foreign_keys too.
- default_key_column_search_order = ['Name', 'name', 'ID', 'id']
- classmethod define(tname: str, column_defs: Iterable[dict | Key | Table | tuple[str, bool, Key | Table] | tuple[str, Key | Table]] = [], key_defs: Iterable[dict] = [], fkey_defs: Iterable[dict] = [], comment: str | None = None, acls: dict = {}, acl_bindings: dict = {}, annotations: dict = {}, provide_system: bool = True, provide_system_fkeys: book = True, key_column_search_order: Iterable[str] | None = None)[source]
Build a table definition.
- Parameters:
tname – the name of the newly defined table
column_defs – a list of custom Column.define() results and/or reference targets (see below)
key_defs – a list of Key.define() results for extra or overridden key constraint definitions
fkey_defs – a list of ForeignKey.define() results for foreign key definitions
comment – a comment string for the table
acls – a dictionary of ACLs for specific access modes
acl_bindings – a dictionary of dynamic ACL bindings
annotations – a dictionary of annotations
provide_system – whether to inject standard system column definitions when missing from column_defs
provide_system_fkeys – whether to also inject foreign key definitions for RCB/RMB
key_column_search_order – override heuristic for choosing a Key from a Table input
- Each reference target may be one of:
Key
Table
tuple (str, Key|Table)
tuple (str, bool, Key|Table)
A target Key specifies the columns of the remote table to reference. A target Table instance specifies the remote table and relies on heuristics to choose the target Key. A target tuple specifies a string “base name” and optionally a boolean “nullok” for the implied referencing columns. When omitted, a default base name is constructed from target table information, and a default nullok=False is specified for referencing columns. The key_column_search_order parameter influences the heuristic for selecting a target Key for an input target Table.
- classmethod define_asset(sname: str, tname: str, hatrac_template=None, column_defs: Iterable[dict | deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table | tuple[str, bool, deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table] | tuple[str, deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table]] = [], key_defs=[], fkey_defs=[], comment: str | None = None, acls: dict = {}, acl_bindings: dict = {}, annotations: dict = {}, provide_system: bool = True, provide_system_fkeys: bool = True, key_column_search_order: collections.abc.Iterable[str] | None = None)[source]
Build an asset table definition.
- Parameters:
sname – the name of the schema for the asset table
tname – the name of the newly defined table
hatrac_template –
template for the hatrac URL. Will undergo substitution to template can include elmenents such at {{{MD5}}} or {{{Filename}}}. The default template puts files in
/hatrac/schema_name/table_name/md5.filename
where the filename and md5 value is computed on upload and the schema_name and table_name are the values of the provided arguments. If value is set to False, no hatrac_template is used.
column_defs – a list of Column.define() results and/or reference targets (see below)
key_defs – a list of Key.define() results for extra or overridden key constraint definitions
fkey_defs – a list of ForeignKey.define() results for foreign key definitions
comment – a comment string for the table
acls – a dictionary of ACLs for specific access modes
acl_bindings – a dictionary of dynamic ACL bindings
annotations – a dictionary of annotations
provide_system – whether to inject standard system column definitions when missing from column_defs
provide_system_fkeys – whether to also inject foreign key definitions for RCB/RMB
key_column_search_order – override heuristic for choosing a Key from a Table input
These core asset table columns are generated automatically if absent from the input column_defs.
Filename: ermrest_curie, unique not null, default curie template “%s:{RID}” % curie_prefix
- URL: Location of the asset, unique not null. Default template is:
/hatrac/cat_id/sname/tname/{{{MD5}}}.{{{Filename}}} where tname is the name of the asset table.
Length: Length of the asset.
MD5: text
Description: markdown, not null
However, caller-supplied definitions override the default. See Table.define() documentation for an explanation reference targets in the column_defs list and the related key_column_search_order parameter.
In addition to creating the columns, this function also creates an asset annotation on the URL column to facilitate use of the table by Chaise.
- classmethod define_association(associates: Iterable[deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table | tuple[str, deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table]], metadata: Iterable[deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table | dict | tuple[str, bool, deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table]] = [], table_name: str | None = None, comment: str | None = None, provide_system: bool = True, provide_system_fkeys: bool = True, key_column_search_order: collections.abc.Iterable[str] | None = None) dict [source]
Build an association table definition.
- Parameters:
associates – reference targets being associated (see below)
metadata – additional metadata fields and/or reference targets for impure associations
table_name – name for the association table or None for default naming
comment – comment for the association table or None for default comment
provide_system – add ERMrest system columns when True
provide_system_fkeys – whether to also inject foreign key definitions for RCB/RMB
key_column_search_order – override heuristic for choosing a Key from a Table input
This is a utility function to help build an association table definition. It simplifies the task, but removes some control. For full customization, consider using Table.define() directly instead.
A normal (“pure”) N-ary association is a table with N foreign keys referencing N primary keys in referenced tables, with a composite primary key covering the N foreign keys. These pure association tables manage a set of distinct combinations of the associated foreign key values.
An “impure” association table adds additional metadata alongside the N foreign keys.
The “associates” parameter takes an iterable of reference targets. The association will be comprised of foreign keys referencing these associates. This includes columns to store the associated foreign key values, foreign key constraints to the associated tables, and a composite key constraint covering all the associated foreign key values.
The “metadata” parameter takes an iterable Column.define() results and/or reference targets. The association table will be augmented with extra metadata columns and foreign keys as directed by these inputs.
See the Table.define() method documentation for more on reference targets. Association columns must be defined with nullok=False, so the associates parameter is restricted to a more limited form of reference target input without caller-controlled nullok boolean values.
- classmethod define_page(tname, column_defs: Iterable[dict | deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table | tuple[str, bool, deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table] | tuple[str, deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table]] = [], key_defs=[], fkey_defs=[], comment: str | None = None, acls: dict = {}, acl_bindings: dict = {}, annotations: dict = {}, provide_system: bool = True, provide_system_fkeys: bool = True, key_column_search_order: collections.abc.Iterable[str] | None = None)[source]
Build a wiki-like “page” table definition.
- Parameters:
tname – the name of the newly defined table
column_defs – a list of Column.define() results for extra or overridden column definitions
key_defs – a list of Key.define() results and/or reference targets (see below)
fkey_defs – a list of ForeignKey.define() results for foreign key definitions
comment – a comment string for the table
acls – a dictionary of ACLs for specific access modes
acl_bindings – a dictionary of dynamic ACL bindings
annotations – a dictionary of annotations
provide_system – whether to inject standard system column definitions when missing from column_defs
provide_system_fkeys – whether to also inject foreign key definitions for RCB/RMB
key_column_search_order – override heuristic for choosing a Key from a Table input
These core page columns are generated automatically if absent from the input column_defs.
Title: text, unique not null
Content: markdown
However, caller-supplied definitions override the default. See Table.define() documentation for an explanation reference targets in the column_defs list and the related key_column_search_order parameter.
- classmethod define_vocabulary(tname: str, curie_template: str, uri_template: str = '/id/{RID}', column_defs: Iterable[dict | deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table | tuple[str, bool, deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table] | tuple[str, deriva.core.ermrest_model.Key | deriva.core.ermrest_model.Table]] = [], key_defs=[], fkey_defs=[], comment: str | None = None, acls: dict = {}, acl_bindings: dict = {}, annotations: dict = {}, provide_system: bool = True, provide_system_fkeys: bool = True, provide_name_key: bool = True, key_column_search_order: collections.abc.Iterable[str] | None = None)[source]
Build a vocabulary table definition.
- Parameters:
tname – the name of the newly defined table
curie_template – the RID-based template for the CURIE of locally-defined terms, e.g. ‘MYPROJECT:{RID}’
uri_template – the RID-based template for the URI of locally-defined terms, e.g. ‘https://server.example.org/id/{RID}’
column_defs – a list of Column.define() results and/or reference targets (see below)
key_defs – a list of Key.define() results for extra or overridden key constraint definitions
fkey_defs – a list of ForeignKey.define() results for foreign key definitions
comment – a comment string for the table
acls – a dictionary of ACLs for specific access modes
acl_bindings – a dictionary of dynamic ACL bindings
annotations – a dictionary of annotations
provide_system – whether to inject standard system column definitions when missing from column_defs
provide_system_fkeys – whether to also inject foreign key definitions for RCB/RMB
provide_name_key – whether to inject a key definition for the Name column
key_column_search_order – override heuristic for choosing a Key from a Table input
These core vocabulary columns are generated automatically if absent from the input column_defs.
ID: ermrest_curie, unique not null, default curie template “%s:{RID}” % curie_prefix
URI: ermrest_uri, unique not null, default URI template “/id/{RID}”
Name: text, unique not null
Description: markdown, not null
Synonyms: text[]
However, caller-supplied definitions override the default. See Table.define() documentation for an explanation reference targets in the column_defs list and the related key_column_search_order parameter.
- property display
Convenience property for managing content of object annotation tag:misd.isi.edu,2015:display
- drop(cascade=False, update_mappings=UpdateMappings.no_update)[source]
Remove this table from the remote database.
- Parameters:
cascade – drop dependent objects.
update_mappings – update annotations to reflect changes (default False)
- property export_2019
Convenience property for managing content of object annotation tag:isrd.isi.edu,2019:export
- property export_fragment_definitions
Convenience property for managing content of object annotation tag:isrd.isi.edu,2021:export-fragment-definitions
- find_associations(min_arity=2, max_arity=2, unqualified=True, pure=True, no_overlap=True) Iterable[FindAssociationResult] [source]
Yield (iterable) Association objects linking to this table and meeting all criteria.
min_arity: minimum number of associated fkeys (default 2) max_arity: maximum number of associated fkeys (default 2) or None unqualified: reject qualified associations when True (default True) pure: reject impure assocations when True (default True) no_overlap: reject overlapping associations when True (default True)
See documentation for sibling method Table.is_association(…) for more explanation of these association detection criteria.
- fkey_by_column_map(from_to_map, raise_nomatch=True)[source]
Return fkey from self.foreign_keys with matching {referencing: referenced} column mapping.
from_to_map: dict-like mapping with items() method yielding (from_col, to_col) pairs raise_nomatch: for True, raise KeyError on non-match, else return None
- fkeys_by_columns(from_columns, partial=False, raise_nomatch=True)[source]
Iterable of fkeys from self.foreign_keys with matching columns.
from_columns: iterable of referencing column instances or column names partial: include fkeys which cover a superset of from_columns raise_nomatch: for True, raise KeyError on empty iterable
- property generated
Convenience property for managing presence of annotation tag:isrd.isi.edu,2016:generated
- property google_dataset
Convenience property for managing content of object annotation tag:isrd.isi.edu,2021:google-dataset
- property immutable
Convenience property for managing presence of annotation tag:isrd.isi.edu,2016:immutable
- property indexing_preferences
Convenience property for managing content of object annotation tag:isrd.isi.edu,2018:indexing-preferences
- is_association(min_arity=2, max_arity=2, unqualified=True, pure=True, no_overlap=True, return_fkeys=False)[source]
Return (truthy) integer arity if self is a matching association, else False.
min_arity: minimum number of associated fkeys (default 2) max_arity: maximum number of associated fkeys (default 2) or None unqualified: reject qualified associations when True (default True) pure: reject impure assocations when True (default True) no_overlap: reject overlapping associations when True (default True) return_fkeys: return the set of N associated ForeignKeys if True
The default behavior with no arguments is to test for pure, unqualified, non-overlapping, binary assocations.
An association is comprised of several foreign keys which are covered by a non-nullable composite row key. This allows specific combinations of foreign keys to appear at most once.
The arity of an association is the number of foreign keys being associated. A typical binary association has arity=2.
An unqualified association contains only the foreign key material in its row key. Conversely, a qualified association mixes in other material which means that a specific combination of foreign keys may repeat with different qualifiers.
A pure association contains only row key material. Conversely, an impure association includes additional metadata columns not covered by the row key. Unlike qualifiers, impure metadata merely decorates an association without augmenting its identifying characteristics.
A non-overlapping association does not share any columns between multiple foreign keys. This means that all combinations of foreign keys are possible. Conversely, an overlapping association shares some columns between multiple foreign keys, potentially limiting the combinations which can be represented in an association row.
These tests ignore the five ERMrest system columns and any corresponding constraints.
- key_by_columns(unique_columns, raise_nomatch=True)[source]
Return key from self.keys with matching unique columns.
unique_columns: iterable of column instances or column names raise_nomatch: for True, raise KeyError on non-match, else return None
- property non_deletable
Convenience property for managing presence of annotation tag:isrd.isi.edu,2016:non-deletable
- property source_definitions
Convenience property for managing content of object annotation tag:isrd.isi.edu,2019:source-definitions
- sqlite3_ddl(keys: bool = True) str [source]
Return SQLite3 table definition DDL statement for this table.
- Parameters:
keys – If true, include unique constraints for each table key
Caveat: this utility does not produce: - column default expressions - foreign key constraint DDL
Both of these features are fragile in data export scenarios where we want to represent arbitrary ERMrest catalog dumps.
- classmethod system_column_defs(custom=[])[source]
Build standard system column definitions, merging optional custom definitions.
- classmethod system_fkey_defs(tname, custom=[])[source]
Build standard system fkey definitions, merging optional custom definitions.
- classmethod system_key_defs(custom=[])[source]
Build standard system key definitions, merging optional custom definitions.
- property table_display
Convenience property for managing content of object annotation tag:isrd.isi.edu,2016:table-display
- property uri_path
URI to this model element.
- property visible_columns
Convenience property for managing content of object annotation tag:isrd.isi.edu,2016:visible-columns
- property visible_foreign_keys
Convenience property for managing content of object annotation tag:isrd.isi.edu,2016:visible-foreign-keys
- property viz_3d_display
Convenience property for managing content of object annotation tag:isrd.isi.edu,2021:viz-3d-display
- class deriva.core.ermrest_model.UpdateMappings(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]
-
Update Mappings flag enum
- deferred = 'deferred'
- immediate = 'immediate'
- no_update = ''
- deriva.core.ermrest_model.equivalent(doc1, doc2, method=None)[source]
Determine whether two dict/array/literal documents are structurally equivalent.
- deriva.core.ermrest_model.find_tables_with_foreign_keys(target_tables: Iterable[Table], quantifier: Quantifier = Quantifier.all) set[deriva.core.ermrest_model.Table] [source]
Return set of tables with foreign key references to target tables.
- Parameters:
target_tables – an iterable of ermrest_model.Table instances
quantifier – one of the Quantifiers ‘any’ or ‘all’ (default ‘all’)
Each returned Table instance will be a table that references the targets according to the selected quantifier. A reference is a direct foreign key in the returned table that refers to a primary key of the target table.
quantifier==all: a returned table references ALL targets
quantifier==any: a returned table references AT LEAST ONE target
For proper function, all target_tables instances MUST come from the same root Model instance hierarchy.
- deriva.core.ermrest_model.make_id(*components)[source]
Build an identifier that will be OK for ERMrest and Postgres.
Naively, append as ‘_’.join(components).
Fallback to heuristics mixing truncation with short hashes.
- deriva.core.ermrest_model.make_type(type_doc)[source]
Create instance of Type, DomainType, or ArrayType as appropriate for type_doc.
- deriva.core.ermrest_model.object_annotation(tag_uri)[source]
Decorator to establish property getter/setter/deleter for object annotations.
Usage example:
@presence_annotation(tag.display) def display(self): pass
The stub method will be discarded.
deriva.core.hatrac_cli module
- class deriva.core.hatrac_cli.DerivaHatracCLI(description, epilog)[source]
Bases:
BaseCLI
Deriva Hatrac Command-line Interface.
- exception deriva.core.hatrac_cli.DerivaHatracCLIException(message)[source]
Bases:
Exception
Base exception class for DerivaHatracCli.
- exception deriva.core.hatrac_cli.ResourceException(message, cause)[source]
Bases:
DerivaHatracCLIException
Remote resource exception.
- exception deriva.core.hatrac_cli.UsageException(message)[source]
Bases:
DerivaHatracCLIException
Usage exception.
deriva.core.hatrac_store module
- exception deriva.core.hatrac_store.HatracHashMismatch[source]
Bases:
ValueError
- class deriva.core.hatrac_store.HatracStore(scheme, server, credentials=None, session_config=None)[source]
Bases:
DerivaBinding
- content_equals(path, filename=None, md5=None, sha256=None)[source]
Check if a remote object’s content is equal to the content of the at least one of the specified input file, input md5, or input sha256 by comparing MD5 hashes. :return: True IFF the object exists and the MD5 or SHA256 hash matches the MD5 or SHA256 hash of the input file
or the passed MD5 or SHA256 parameters.
- create_upload_job(path, file_path, md5, sha256, create_parents=True, chunk_size=26214400, content_type=None, content_disposition=None)[source]
- get_obj(path, headers={}, destfilename=None, callback=None, chunk_size=26214400)[source]
Retrieve resource optionally streamed to destination file.
If destfilename is provided, download content to file with that name. Caller is responsible to clean up file even on error, when the file may or may not be exist.
If hatrac provides a Content-MD5 response header, the resulting download file will be hash-verified on success or raise HatracHashMismatch on errors. This is not verified when destfilename is None, as the client must instead consume and validate content directly from the response object.
- put_loc(path, file_path, headers={}, md5=None, sha256=None, content_type=None, content_disposition=None, chunked=False, chunk_size=26214400, create_parents=True, allow_versioning=True, callback=None, cancel_job_on_error=True)[source]
- Parameters:
path –
file_path –
headers –
md5 –
sha256 –
content_type –
content_disposition –
chunked –
chunk_size –
create_parents –
allow_versioning –
callback –
cancel_job_on_error –
- Returns:
- put_obj(path, data, headers={}, md5=None, sha256=None, parents=True, content_type=None, content_disposition=None, allow_versioning=True)[source]
Idempotent upload of object, returning object location URI.
- Arguments:
path: name of object data: filename or seekable file-like object headers: additional headers md5: a base64 encoded md5 digest may be provided in order to skip the automatic hash computation sha256: a base64 encoded sha256 digest may be provided in order to skip the automatic hash computation parents: automatically create parent namespace(s) if missing content_type: the content-type of the object (optional) content_disposition: the preferred content-disposition of the object (optional) allow_versioning: reject with NotModified if content already exists (optional)
Automatically computes and sends Content-MD5 if no digests provided.
If an object-version already exists under the same name with the same Content-MD5, that location is returned instead of creating a new one.
deriva.core.polling_ermrest_catalog module
- class deriva.core.polling_ermrest_catalog.PollingErmrestCatalog(scheme, server, catalog_id, credentials={}, caching=True, session_config=None, amqp_server=None)[source]
Bases:
ErmrestCatalog
Persistent handle for an ERMrest catalog.
Provides a higher-level state_change_once() idiom to efficiently find candidate rows, transform them, and apply updates.
Provides a higher-level blocking_poll() idiom to efficiently poll a catalog, using AMQP to optimize polling where possible. (AMQP is currently limited to clients on localhost of catalog in practice.)
These features can be composed to implement condition-action agents with domain-specific logic, e.g.
catalog = ErmrestCatalog(…) idle_etag = None
- def look_for_work():
global idle_etag idle_etag, batch = catalog.state_change_once(
# claim up to 5 items per batch ‘/entity/Foo/state=actionable?limit=5’, ‘/attributegroup/Foo/id;state’, lambda row: {‘id’: row[‘id’], ‘state’: ‘claimed’}, idle_etag
) for candidate, update in batch:
# assume we have free reign on claimed candidates # using state=claimed as a semaphore revision = candidate.copy() revision[‘state’] = update[‘state’] … # do agent work revision[‘state’] = ‘complete’ catalog.put(‘/entity/Foo’, [revision])
catalog.blocking_poll(look_for_work)
- blocking_poll(look_for_work, polling_seconds=600, coalesce_seconds=0.1)[source]
Use ERMrest change-notice monitoring to optimize polled work processing.
Client-provided look_for_work function finds actual work in ERMrest and processes it. We only optimize the scheduling of this work.
Run look_for_work() whenever there might be more work in ERMrest.
If look_for_work() returns True, assume there is more work.
If look_for_work() returns non-True, wait for ERMrest change-notice or polling_seconds timeout before looking again (whichever comes first).
On any change-monitoring communication error, assume there might be more work and restart the monitoring process.
Other exceptions abort the blocking_poll() call.
- state_change_once(query_datapath, update_datapath, row_transform_func, idle_etag=None)[source]
Perform generic conditional state update via GET-PUT sequence.
- Arguments:
query_datapath: a query for candidate rows update_datapath: an update to consume update rows row_transform_func: maps candidate to update rows idle_etag: no-op if table is still in this state
- Returns: (idle_etag, [(candidate, update)…])
idle_etag: value to thread to future calls [(candidate, update)…]: each row that was updated
Exceptions from the transform or update process will abort without returning results.
GET query_datapath to get candidate row(s)
apply row_transform_func(row) to get updated content
PUT update_datapath to deliver transformed content – discards rows transformed to None
Uses opportunistic concurrency control with ETag, If-Match, etc. for safety.
Module contents
- deriva.core.get_credential(host, credential_file='/root/.deriva/credential.json', globus_credential_file='/root/.deriva/globus-credential.json', config_file='/root/.deriva/config.json', requested_scope=None, force_scope_lookup=False, match_scope_tag='deriva-all', update_bdbag_keychain=True)[source]
This function is used to get authorization credentials (in dict form) for use with various deriva-py API calls which take it as a parameter. A user must have already authenticated to the target host using either deriva-auth or deriva-globus-auth-utils login prior to calling this function, or the credential set for the host will not be found.
- Parameters:
host – The hostname to retrieve the credential set for.
credential_file – Optional path to non-default location of the webauthn cookie credential file.
globus_credential_file – Optional path to non-default location of the GlobusAuth bearer token store.
config_file – Optional path to the non-default location of the deriva-py config file.
requested_scope – Optional, specific scope request string for the given host. If not specified, the webauthn service on the host will be queried to determine the host-to-scope mapping that should be used.
force_scope_lookup – Optional parameter to force the webauthn scope query and update the cached value in the deriva-py config file. A scope lookup will always be performed the first time a host-to-scope mapping is needed and is not already present in the configuration file for a given host.
match_scope_tag – In the case that a host-to-scope mapping request returns multiple scopes, this is the key value (“tag”) to match against in the result dict. By convention, the default is set to “deriva-all”, which is the expected response from webauthn.
update_bdbag_keychain – Updates the bdbag keychain file with the bearer token mapped to host. This is done to ensure that the bdbag keychain is updated when a refreshable bearer-token gets refreshed during the login check. Defaults to True.
- Returns:
A dict containing credential authorization values mapped by authorization type
- deriva.core.get_credential(host, credential_file=DEFAULT_CREDENTIAL_FILE, globus_credential_file=DEFAULT_GLOBUS_CREDENTIAL_FILE, config_file=DEFAULT_CONFIG_FILE, requested_scope=None, force_scope_lookup=False, match_scope_tag='deriva-all')[source]
This function is used to get authorization credentials (in dict form) for use with various deriva-py API calls which take it as a parameter. A user must have already authenticated to the target host using either deriva-auth or deriva-globus-auth-utils login prior to calling this function, or the credential set for the host will not be found.
- Parameters:
host – The hostname to retrieve the credential set for.
credential_file – Optional path to non-default location of the webauthn cookie credential file.
globus_credential_file – Optional path to non-default location of the GlobusAuth bearer token store.
config_file – Optional path to the non-default location of the deriva-py config file.
requested_scope – Optional, specific scope request string for the given host. If not specified, the webauthn service on the host will be queried to determine the host-to-scope mapping that should be used.
force_scope_lookup – Optional parameter to force the webauthn scope query and update the cached value in the deriva-py config file. A scope lookup will always be performed the first time a host-to-scope mapping is needed and is not already present in the configuration file for a given host.
match_scope_tag – In the case that a host-to-scope mapping request returns multiple scopes, this is the key value (“tag”) to match against in the result dict. By convention, the default is set to “deriva-all”, which is the expected response from webauthn.
update_bdbag_keychain – Updates the bdbag keychain file with the bearer token mapped to host. This is done to ensure that the bdbag keychain is updated when a refreshable bearer-token gets refreshed during the login check. Defaults to True.
- Returns:
A dict containing credential authorization values mapped by authorization type
- deriva.core.read_credential(credential_file=DEFAULT_CREDENTIAL_FILE, create_default=False, default=DEFAULT_CREDENTIAL)[source]
- deriva.core.write_credential(credential_file=DEFAULT_CREDENTIAL_FILE, credential=DEFAULT_CREDENTIAL)[source]
- deriva.core.read_config(config_file=DEFAULT_CONFIG_FILE, create_default=False, default=DEFAULT_CONFIG)[source]
- core.DEFAULT_CONFIG_PATH = System dependent default path to the configuration directory.
- core.DEFAULT_CONFIG_FILE = System dependent default path to the config file.
- core.DEFAULT_CREDENTIAL_FILE = System dependent default path to the credential file.
- core.DEFAULT_GLOBUS_CREDENTIAL_FILE = System dependent default path to the Globus Auth credential file.