KBI0002: Represent a Dataverse dataset as a DataLad dataset¶
- authors:
Michael Hanke <michael.hanke@gmail.com>
- discussion:
https://github.com/psychoinformatics-de/knowledge-base/pull/9
- keywords:
addurls, datalad-dataverse
The Dataverse extension package equips DataLad with the ability to deposit DataLad datasets on Dataverse and later retrieve such DataLad datasets from Dataverse again. However, if a non-DataLad dataset has been deposited on Dataverse already, there is no convenient command provided that can generate a DataLad dataset from such a deposit automatically. At the same time, it can be useful to nevertheless have such a DataLad dataset, for example, to be able to declare a (versioned) data dependency for using DataLad’s built-in provenance tracking features.
Requirements¶
The solution described here requires DataLad (tested with v0.18), and the Dataverse extension package (tested with v1.0). In principle, this can also be done with the pyDataverse package directly, but this has not been tested.
Solution¶
The starting point is a DataLad dataset. It either already exists, or is created
with DataLad’s create
command:
$ datalad create <path>
A new state of this dataset is populated to match the latest state of a Dataverse
dataset using the addurls-dataverse.py
script that is shown below. it must be
executed with a Python environment that has the aforementioned software dependencies
installed. The scripts assumes to run in the root directory of the DataLad dataset
that shall be populated. The script requires a Dataverse API token to be set via
the DATAVERSE_API_TOKEN
environment variable. This token can be obtained from
the Dataverse web UI (see the Dataverse extension package documentation for
details). Moreover, the script requires two positional arguments:
the base URL of the Dataverse instance that is hosting the dataset
the persistent identifier (PID) of the Dataverse dataset to be queried
The PID can be found in the “Metadata” tab of the dataset’s page on Dataverse.
An example script call looks like:
# needs to be using the `set` command on Windows
$ export DATAVERSE_API_TOKEN=99ad0817-....-....-....-279a39346c70
$ python <path-to-script> 'https://dataverse.example.com' 'doi:10.26165/...'
When executed, the script will output progress information on files being processed. No data will be downloaded. The local DataLad dataset is populated with information from the metadata query alone.
Once finished, the DataLad dataset will show (untracked) changes. Those need to be saved:
$ datalad save ...
It is recommended to include an appropriate commit message.
Afterwards the DataLad dataset is ready for (re-)use and supports standard DataLad
operations like get
, and can be added as a subdataset to other datasets. Please
note that the generated DataLad dataset needs to be hosted somewhere in order to be
accessible. However, there is no requirement to host a copy of the data files stored
on Dataverse.
Limitations¶
The script shown here assumes:
the targeted Dataverse instance is reporting SHA256 checksums. If that is not the case, and, for example MD5SUMs are reported. The script needs to be adjusted accordingly.
the queried Dataverse dataset must be public (file access permitted without particular permissions). The presented solution could be extended to access-protected dataset, but this has not been explored.
1import os
2import sys
3
4from datalad_dataverse.utils import get_native_api
5from datalad_next.datasets import Dataset
6
7# API token is specified via the environment
8dvtoken = os.environ['DATAVERSE_API_TOKEN']
9
10# the two positional arguments at the dataverse instance base URL
11# and the PID of the dataset
12dvurl = sys.argv[1]
13dvds_pid = sys.argv[2]
14
15api = get_native_api(dvurl, dvtoken)
16
17# get info and file listing for the latest version of that dataset
18dslisting = api.get_dataset(
19 dvds_pid,
20 version=':latest',
21).json()['data']['latestVersion']
22
23# this produces a list of dicts, where each dict represents a single
24# file in the dataset, and all relevant keys are accessible directly
25# at the top-level. The original structure is more complex (nested)
26dsfiles = [
27 dict(
28 f,
29 directory=f.get('directoryLabel', '.'),
30 sha265=f['dataFile']['checksum']['value'],
31 **f['dataFile'])
32 for f in dslisting['files']
33]
34
35dlds = Dataset('.')
36
37# feed file info to datalad's addurls
38# this call assumes that the dataverse instance is reporting
39# SHA256 checksums (other checksums could be used if available).
40# being able to define a git-annex key from the metadata alone
41# makes it possible to generate the dataset content without
42# having to download any file.
43dlds.addurls(
44 urlfile=dsfiles,
45 urlformat=f'{dvurl}/api/access/datafile/{{id}}',
46 filenameformat='{directory}/{label}',
47 key='et:SHA256-s{filesize}--{sha256}',
48 exclude_autometa='*',
49 fast=True,
50 save=False,
51)