deriva.utils.catalog.components package

Submodules

deriva.utils.catalog.components.configure_catalog module

class deriva.utils.catalog.components.configure_catalog.DerivaCatalogConfigure(host, scheme='https', catalog_id=1, validate=True)

Bases: deriva.utils.catalog.components.deriva_model.DerivaCatalog

DerovaCatalogConfigure extends DerivaCatalog by providing methods for creating a default configuration for a catalog.

The only method of interest in this class is configure_baseline_catalog.

configure_baseline_catalog(catalog_name=None, admin=None, curator=None, writer=None, reader=None, set_policy=True, public=False)

Put catalog into standard configuration which includes:

1. Setting default display mode to be to turn underscores to spaces.
2. Set access control assuming admin, curator, writer, and reader groups.
3. Configure ermrest_client to have readable names.

4. Create a schema called WWW and create a Page table in that schema configured to display web-page like content.

Parameters:

• catalog_name – Name to use when looking up catalog groups. Defaults to host name if not provided.
• admin – Name of the admin group. Defaults to catalog-admin, where catalog is the catalog_name
• curator – Name of the curator group. Defaults to catalog-curator
• writer – Name of the writer group. Defaults to catalog-writer
• reader – Name of the reader group. Defaults to catalog-reader
• set_policy – Set policy for catalog to support reader/writer/curator/admin groups.
• public – Set to true if anonymous read access should be allowed.

class deriva.utils.catalog.components.configure_catalog.DerivaTableConfigure(catalog, table)

Bases: deriva.utils.catalog.components.deriva_model.DerivaTable

This class is used to create a default configuration for a Deriva Table.

configure_self_serve_policy(groups)

Set up a table so it has a self service policy. Add an owner column if one is not present, and set the acl binding so that it follows the self service policy.

Parameters:groups – dictionary of core catalog groups Returns:

configure_table_defaults(set_policy=True, public=False, reset_visible_columns=True)

This function adds the following basic configuration details to an existing table:

1. Creates a self service modification policy in which creators can update update any row they create. Optionally, an Owner column can be provided, which allows the creater of a row to delegate row ownership to a specific group.
2. Adds display annotations and foreign key declarations so that system columns RCB, RMB display in a user friendly way.
3. Adds a visible_foreign_keys and visible_columns annotation.

Parameters:

• set_policy – If true, then configure the table to have a self service policy
• public – Make table acessible without logging in.
• reset_visible_columns – Overwrite any existing visible_columns annotation.

create_default_visible_columns(really=False)

create_default_visible_foreign_keys(really=False)

deriva.utils.catalog.components.deriva_model module

class deriva.utils.catalog.components.deriva_model.DerivaModel(catalog)

Bases: deriva.utils.catalog.components.deriva_model.DerivaLogging

Representation of a deriva model. Is primarily used as a resource manager to group catalog operations so as to minimize network round trips. For example:

``` with DerivaModel(catalog)

table = schema.create_table(‘MyTable’,[]) table.display = ‘My Nice Table’

```

catalog_model()

contexts = {<DerivaContext.compact_brief: 'compact/brief'>, <DerivaContext.compact: 'compact'>, <DerivaContext.entry_edit: 'entry/edit'>, <DerivaContext.row_name: 'row_name'>, <DerivaContext.entry_create: 'entry/create'>, <DerivaContext.row_name_detailed: 'row_name/detailed'>, <DerivaContext.compact_select: 'compact/select'>, <DerivaContext.star: '*'>, <DerivaContext.row_name_compact: 'row_name/compact'>, <DerivaContext.filter: 'filter'>, <DerivaContext.row_name_title: 'row_name/title'>, <DerivaContext.entry: 'entry'>, <DerivaContext.detailed: 'detailed'>}

model_element(obj)

class deriva.utils.catalog.components.deriva_model.DerivaCatalog(host, scheme='https', catalog_id=1, ermrest_catalog=None)

Bases: deriva.utils.catalog.components.deriva_model.DerivaCore

A Dervia catalog. Operations on the catalog will alter both the ERMrest service as well as the annotations used by Chaise.

bulk_upload

Get/Set the navigation bar menu.

Returns:

catalog_id

Get catalog id

Returns:catalog identifier

create_schema(schema_name, comment=None, acls={}, annotations={})

Create a new schema in this catalog.

Parameters:

• schema_name – The name of the schema
• comment – A comment for the schema
• acls – ACLs for the schema
• annotations – Schema annotations.

Returns:

A DerivaSchema object

describe()

getPathBuilder()

get_groups()

host

Get catalog host.

Returns:Hostname of the current catalog

name

navbar_menu

Get/Set the navigation bar menu.

Returns:

refresh()

Refresh the any cached model values from the server. :return:

rename_visible_columns(column_map, validate=False)

schema(schema_name)

schemas

Return an interable for the schemas contained in the the catalog. The return value can be indexed by schema name, or iterated over.

Returns:

server_uri

URI for the catalog server

Returns:server uri

validate()

Validate all of the objects in the catalog.

Returns:

validate_display()

class deriva.utils.catalog.components.deriva_model.DerivaSchema(catalog, schema)

Bases: deriva.utils.catalog.components.deriva_model.DerivaCore

comment

create_asset(table_name, column_defs=[], key_defs=[], fkey_defs=[], comment=None, acls={}, acl_bindings={}, annotations={}, file_pattern='.*', extensions=[])

Create an asset table. This function creates a new table that has the standard asset columns in addition to columns provided by the caller.

Parameters:

• table_name –
• column_defs –
• key_defs –
• fkey_defs –
• comment –
• acls –
• acl_bindings –
• annotations –

Returns:

A DerivaTable object

create_table(table_name, column_defs, key_defs=[], fkey_defs=[], comment=None, acls={}, acl_bindings={}, annotations={}, default_config=True)

Create a new table from the provided arguments.

Parameters:

• table_name – The name of the new table to be created.
• column_defs –
• key_defs –
• fkey_defs –
• comment –
• acls –
• acl_bindings –
• annotations –
• default_config –

Returns:

create_vocabulary(vocab_name, curie_template, uri_template='/id/{RID}', column_defs=[], key_defs=[], fkey_defs=[], comment=None, acls={}, acl_bindings={}, annotations={})

Create a vocabulary table that can be used to reference externally defined terms. This funcion provides the option to add additional columns to the table, as well as set access control and additional table annotations.

Parameters:

• vocab_name – Name of the vocabulary table to be created.
• curie_template – Default shortform name for the term, in the form of ‘NAMESPACE:{RID}’,
• uri_template –
• column_defs – Additional columns to be added to the vocabulary table.
• key_defs –
• fkey_defs –
• comment – Comment string.
• acls –
• acl_bindings –
• annotations –

Returns:

A DerivaTable object

describe()

display

drop()

name

table(table_name)

Return a DerivaTable object for the named table.

Parameters:table_name – Returns:

tables

validate()

Validate the annotations associated with the tables in this schema. Look at all visable column, visible foreign key, display and other configurable fields associated with the catalog and check to ensure they use valid column and key definitions. Some limited syntax checking is done as well. Throws an exception if an invalid value is found.

Returns:True if all values are valid.

validate_display()

class deriva.utils.catalog.components.deriva_model.DerivaTable(catalog, table)

Bases: deriva.utils.catalog.components.deriva_model.DerivaCore

associate_tables(target_table, table_column='RID', target_column='RID', inline=True)

Create a pure binary association table that connects rows in the table to rows in the target table. Assume that RIDs are used for linking. however, this can be over riden.

Parameters:

• target_schema – Schema of the table that is to be associated with current table
• target_table – Name of the table that is to be associated with the current table
• table_column – Name of the column in the current table that is used for the foreign key, defaults to RID
• target_column – Name of the column in the target table that is to be used for the foreign key, defaults to RID

Returns:

Association table.

associate_vocabulary(term_table, table_column='RID')

Set an existing column in the table to refer to an existing vocabulary table.

Parameters:

• column_name – Name of the column whose value is to be from the vocabular
• term_table – The term table.

Returns:

None.

associated_tables()

Assuming the table is an pure binary association table, return the two table endpoints

Parameters:table – ermrest table object for a table that is a pure binary association table. Returns:list of 2-tuples that are the schema and table for the two tables in the M:N relationship

attributes(*attributes, **renamed_attributes)

chaise_uri

column(column_name)

columns

comment

copy_columns(column_map, dest_table=None)

Copy a set of columns, updating visible columns list and keys to mirror source columns. The columns to copy are specified by a column map. Column map can be a dictionary with entries SrcCol: DerviaColumnSpec or SrcCol:TargetCol.

Parameters:

• column_map – a column_map that describes the list of columns.
• dest_table – Table name of destination table
• column_map – A dictionary that specifies column name mapping

Returns:

copy_table(schema_name, table_name, column_map={}, clone=False, key_defs=[], fkey_defs=[], comment=None, acls={}, acl_bindings={}, annotations={})

Copy the current table to the specified target schema and table. All annotations and keys are modified to capture the new schema and table name. Columns can be renamed in the target table by providing a column mapping. Key and foreign key definitions can be augmented or overwritten by providing appropriate arguments. Lastly if the clone argument is set to true, the RIDs of the source table are reused, so that the equivalent of a move operation can be obtained.

Parameters:

• schema_name – Target schema name
• table_name – Target table name
• column_map – A dictionary that is used to rename columns in the target table.
• clone –
• key_defs –
• fkey_defs –
• comment –
• acls –
• acl_bindings –
• annotations –

Returns:

The new table

create_asset_table(key_column, extensions=[], file_pattern='.*', column_defs=[], key_defs=[], fkey_defs=[], comment=None, acls={}, acl_bindings={}, annotations={}, set_policy=True)

Create a basic asset table and configures the bulk upload annotation to load the table along with a table of associated metadata. This routine assumes that the metadata table has already been defined, and there is a key associated metadata. This routine assumes that the metadata table has already been defined, and there is a key column the metadata table that can be used to associate the asset with a row in the table. The default configuration assumes that the assets are in a directory named with the table name for the metadata and that they either are in a subdirectory named by the key value, or that they are in a file whose name starts with the key value.

Parameters:

• key_column – The column in the metadata table to be used to correlate assets with entries. Assets will be named using the key column.
• extensions – List file extensions to be matched. Default is to match any extension.
• file_pattern – Regex that identified the files to be considered for upload
• column_defs – a list of Column.define() results for extra or overridden column definitions
• key_defs – a list of Key.define() results for extra or overridden key constraint definitions
• fkey_defs – a list of ForeignKey.define() results for foreign key definitions
• comment – a comment string for the asset table
• acls – a dictionary of ACLs for specific access modes
• acl_bindings – a dictionary of dynamic ACL bindings
• annotations – a dictionary of annotations
• set_policy – If true, add ACLs for self serve policy to the asset table

Returns:

create_columns(columns, positions={}, visible=True)

Create a new column in the table.

Parameters:

• columns – A list of DerivaColumn.
• positions – Where the column should be added into the visible columns spec.
• visible – Include this column in the visible columns spec.

Returns:

create_foreign_key(columns, referenced_table, referenced_columns, name=None, comment=None, on_update='NO ACTION', on_delete='NO ACTION', acls=None, acl_bindings=None, annotations=None, position=None)

Parameters:

• columns – Column names in current table that are used for the foreign key
• referenced_table – Dervia table that is being referenced by this foreign key
• referenced_columns –
• name –
• comment –
• on_update –
• on_delete –
• acls – ACLs, defaults to {}
• acl_bindings – defaults to {}
• annotations – defaults to {}
• position – defaults to {}

Returns:

create_key(columns, name=None, comment=None, annotations={})

datapath

delete_columns(columns)

Drop a set of columns from a table, cleaning up visible columns and keys. You cannot delete columns if they are being used by a foreign key in another table, or if they are part of a composite key and you are only deleting a subset of the columns.

Parameters:columns – A list of column names or DerivaColumn instances for the current table

describe()

disassociate_tables(target_table)

display

drop()

Delete a table :return:

entities()

foreign_key

foreign_keys

is_pure_binary()

Check to see if the table has the propoerties of a pure binary association.

1. It only has two foreign keys,
2. There is a uniqueness constraint on the two keys.
3. NULL values are not allowed in the foreign keys.

Returns:Boolean

is_vocabulary_table()

Test to see if a table is a deriva vocabulary table. :return: True or False.

key

key_referenced(columns)

Given a set of columns that are a key, return the list of foreign keys that reference those columns. :param columns: :return:

keys

link_tables(target_table, column_name=None, target_column='RID', create_column=True)

Create a foreign key link from the specified column to the target table and column.

Parameters:

• column_name – Column or list of columns in current table which will hold the FK
• target_table – Target table to link to.
• target_column – Column to link the table on. Defaults to RID
• create_column – Create a new column for the foreign key. Name defaults to the target table name.

link_vocabulary(column_name, term_table)

Set an existing column in the table to refer to an existing vocabulary table. :param column_name: Name of the column whose value is to be from the vocabular :param term_table: The term table. :return: None.

move_table(schema_name, table_name, delete=True, column_map={}, key_defs=[], fkey_defs=[], comment=None, acls={}, acl_bindings={}, annotations={})

Move a table, renaming and inserting new columns.

Parameters:

• schema_name – Schema for new table
• table_name – Name of new table
• delete – Delete the origional table. Defaults to True
• column_map – A DerivaColumnMap that defines and column renaming or insertions.
• key_defs – New keys that should be defined in the target table
• fkey_defs –
• comment –
• acls –
• acl_bindings –
• annotations –

Returns:

New DerivaTable object

name

path

referenced_by

rename_column(from_column, to_column, default=None, nullok=None)

Rename a column by copying it and then deleting the origional column. THe type of the new column is the same as the old column. It is possible to alter the settings of nullok and default. :param from_column: Name of the column being copied. :param to_column: Name of the new column :param default: Set default value on new column, otherwise, copy existing :param nullok: Set NullOK to provided value on new column, otherwise copy existing :return:

rename_columns(column_map, dest_table=None, delete=True)

Rename a column by copying it and then deleting the origional column. :param dest_table: :param column_map: :param delete: :return:

schema

sources(merge_outbound=False, filter=None)

Create source lists from table columns.

Go through the columns and keys in the current table and create a list of DerivaSourceSpecs for each of them. If filter is provided, only the column or key names in the list are examined. If merge_outbound is true and a column is used in a simple foreign key, used return an outbound source rather then the column source.

Parameters:

• merge_outbound – If True and the column is in a simple foreign_key s
• filter – List of column or key names to include in the returned source lists.

Returns:

A triple of DerivaSourceSpec lists for columns, foreign_keys and incoming foreign_keys.

table_display

validate()

validate_display()

validate_table_display()

visible_columns

visible_foreign_keys

class deriva.utils.catalog.components.deriva_model.DerivaColumn(catalog, column)

Bases: deriva.utils.catalog.components.deriva_model.DerivaCore

Class that represents columns in Deriva catalog.

column_display

comment

default

classmethod define(name, type, nullok=True, default=None, fill=None, comment=None, acls={}, acl_bindings={}, annotations={})

display

drop()

Delete a single column. :return:

fill

get_acl_bindings()

Get dictionary from of acl_bindings :return:

get_acls()

Get dictionary form of ACL :return:

name

nullok

type

update_table(table)

validate()

validate_display()

class deriva.utils.catalog.components.deriva_model.DerivaKey(catalog, key)

Bases: deriva.utils.catalog.components.deriva_model.DerivaCore

columns

comment

static define(columns, name=[], comment=None, annotations={})

display

drop()

full_name

get_acls()

Get dictionary form of ACL :return:

key_display

name

table

update_table(table)

validate()

validate_display()

class deriva.utils.catalog.components.deriva_model.DerivaForeignKey(catalog, fkey)

Bases: deriva.utils.catalog.components.deriva_model.DerivaCore

column_map

columns

comment

static define(columns, dest_schema, dest_table, dest_columns, name=None, comment=None, on_update='NO ACTION', on_delete='NO ACTION', acls={}, acl_bindings={}, annotations={})

definition()

drop()

full_name

get_acl_bindings()

Get dictionary from of acl_bindings :return:

get_acls()

Get dictionary form of ACL :return:

name

on_delete

on_update

referenced_columns

referenced_table

standardize_name()

table

validate()

validate_display()