Uploading files via DERIVA-Upload and DERIVA-Auth

DERIVA provides client tools to upload data assets to a DERIVA deployment. DERIVA-Upload handles the movement of the data files. DERIVA-Auth provides an authentication token which is required when using the command-line interface.

There are two versions of the client tool:

Although the process for downloading and running the above tools are different, they both use the same directory structure designed for different data types.

From a desktop system

The desktop client is convenient for data files that are on your local system and aren’t too large or numerous.

1. Initial setup

The first time you launch deriva-upload (through the applications menu on Windows or MacOS, or with the deriva-upload command on Linux), the tool will ask you if you want to add a server configuration. Click “yes” to bring up the “Options” screen (you can also do this at any time by clicking the “Options” button at the top of the page).

Initial server configuration window

Click “Add” to bring up the “Server Configuration” form and enter the values provided by the DERIVA administrator.

The following are example values for the GUDMAP/RBK deployment.

Server configuration window

2. Uploading files

In the main Deriva-Upload window, click the “Login” button at the top to log in. This will pop up a login dialog window. Once you’ve logged in, you may see a window notifying you that an updated configuration is available and asking if you’d like to apply it; you should click “Yes” to update your configuration and dismiss the window.

Configuration update window

Next, in the main Deriva-Upload window, click the “Browse” button and select the deriva directory you created above. You’ll see all the files you created, listed as “Pending”. Before upload

Click the “Upload” button to start the upload process. The status of each file will change as it’s uploaded; for successful uploads, the status will change from “Pending” to “Complete”.

3. Logging out

Authentication tokens expire after 30 minutes of activity; you can log out explicitly by clicking on the “Logout” button at the top of the window.

Uploading over a slow or unreliable connection

By default, DERIVA-Upload is tuned for fast, reliable networks, such as a university or office connection. If you are uploading from home, over Wi-Fi, or over any connection with slow upload speeds, you may see uploads fail with “timeout” errors or too many retries, or large files that never seem to finish. Most home internet plans upload much more slowly than they download, so a connection that feels fast for streaming video can still be slow for uploading data.

Large files are sent to the server in pieces called “chunks”. By default each chunk is 25 MB, and each one has to finish within a fixed time limit. On a slow connection a chunk can run out of time before it finishes, and the uploader then has to send it again from the start.

Changing two settings usually fixes this. To adjust the settings:

  1. Click the “Options” button at the top of the main window.

  2. Select your server in the list and click “Edit” to open the “Server Configuration” form.

  3. Change the following settings:

    • Connect timeout (secs): despite its name, this setting also controls how long the uploader waits for each chunk to finish sending, not just how long it waits to connect. Increase it to 120.

    • Override chunk size (MB): check this box and set the chunk size to 10. Smaller chunks are each more likely to finish in time, and when one does fail, less data has to be sent again.

  4. Click “OK” to close the “Server Configuration” form, then “OK” again to close “Options”.

The new settings take effect the next time you click “Upload”. They are saved with your server configuration, so you only need to change them once.

If uploads still time out, try a higher connect timeout (for example 300) or a smaller chunk size (5 MB is the smallest allowed). If your connection is fast and reliable, you can return to the defaults by setting the connect timeout back to 6 and unchecking “Override chunk size”. You can hover your mouse over any of these settings for a short description.

You usually do not need to change the I/O timeout (secs) setting. It only controls how long the uploader waits for the server to reply after a chunk has been sent, not how long the sending itself may take.

From a remote server

If your data is on a remote server and/or the data is very large or there are many files involved, you’ll want to use the command-line interface (CLI).

Using the CLI on a remote server is a bit more complicated. First, you’ll need to get an authentication token by running the DERIVA-Auth tool on your desktop. Then you’ll run the command-line tool on the remote server.

1. Initial setup

On your desktop system, install the latest version of DERIVA-Auth here (for Mac or Windows desktops) or here (for Linux desktops).

On the remote server, install the latest version of deriva-py:

pip3 install --upgrade git+https://github.com/informatics-isi-edu/deriva-py.git

2. Getting an authentication token

Upload requests are authorized using OAuth2; to obtain an OAuth token, run the deriva-globus-auth-util utility on the host from which you’ll be uploading files:

deriva-globus-auth-utils login --host host

You can add the --refresh option if you want the token to last beyond the default timeout. This will generate a URL for a OAuth consent page and ask you to paste the resulting authorization code:

Please paste the following URL in a browser:
https://auth.globus.org/v2/oauth2/authorize?client_id=[long url redacted]
Please Paste your Auth Code Below:

Paste the URL into a browser, follow the directions there, and then paste the authorization code back into the deriva-globus-auth-utils session. This will create and store your OAuth token.

Please paste the following URL in a browser:
https://auth.globus.org/v2/oauth2/authorize?client_id=[long url redacted]
Please Paste your Auth Code Below: 
[code redacted]
Login Successful

To end the session, you can allow the token to expire or run:

deriva-globus-auth-utils logout

3. Uploading files

On the server, run the command:

deriva-upload-cli –catalog n host /path/to//deriva

where:

  • n is the catalog number (if specified by your DERIVA administrator; otherwise this option can be omitted),

  • host is the hostname provided by your DERIVA admin, and

  • /path/to//deriva is the path to the deriva directory you created above.

For example:

deriva-upload-cli --catalog 2 www.atlas-d2k.org $HOME/deriva

If uploads fail with timeout errors over a slow or unreliable connection, the same settings described in Uploading over a slow or unreliable connection are available as command-line options. Note that --chunk-size is given in bytes, not MB. For example, this uses a 120 second connect timeout and 10 MB chunks:

deriva-upload-cli --connect-timeout 120 --chunk-size 10485760 --catalog 2 www.atlas-d2k.org $HOME/deriva

4. Logging out

Authentication tokens expire after 30 minutes of activity; you can log out explicitly by clicking on the “Logout” button at the top of the DERIVA-Auth window.