Dataverse 4.19

This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project.

Release Highlights

Open ID Connect Support

Dataverse now provides basic support for any OpenID Connect (OIDC) compliant authentication provider.

Prior to supporting this standard, new authentication methods needed to be added by pull request. OIDC support provides a standardized way for authentication, sharing user information, and more. You are able to use any compliant provider just by loading a configuration file, without touching the codebase. While the usual prominent providers like Google and others feature OIDC support there are plenty of other options to easily attach your installation to a custom authentication provider, using enterprise grade software.

See the OpenID Connect Login Options documentation in the Installation Guide for more details.

This is to be extended with support for attribute mapping, group syncing and more in future versions of the code.

Python Installer

We are introducing a new installer script, written in Python. It is intended to eventually replace the old installer (written in Perl). For now it is being offered as an (experimental) alternative.

See README_python.txt in scripts/installer and/or in the installer bundle for more information.

Major Use Cases

Newly-supported use cases in this release include:

• Dataverse installation administrators will be able to experiment with a Python Installer (Issue #3937, PR #6484)
• Dataverse installation administrators will be able to set up an OIDC-compliant login options by editing a configuration file and with no need for a code change (Issue #6432, PR #6433)
• Following setup by a Dataverse administration, users will be able to log in using OIDC-compliant methods (Issue #6432, PR #6433)
• Users of the Search API will see additional fields in the JSON output (Issues #6300, #6396, PR #6441)
• Users loading the support form will now be presented with the math challenge as expected and will be able to successfully send an email to support (Issue #6307, PR #6462)
• Users of https://mybinder.org can now spin up Jupyter Notebooks and other computational environments from Dataverse DOIs (Issue #4714, PR #6453)

Notes for Dataverse Installation Administrators

Security vulnerability in Solr

A serious security issue has recently been identified in multiple versions of Solr search engine, including v.7.3 that Dataverse is currently using. Follow the instructions below to verify that your installation is safe from a potential attack. You can also consult the following link for a detailed description of the issue:

<A HREF="https://github.com/veracode-research/solr-injection#7-cve-2019-xxxx-rce-via-velocity-template-by-_s00py">RCE in Solr via Velocity Template</A>.

The vulnerability allows an intruder to execute arbitrary code on the system running Solr. Fortunately, it can only be exploited if Solr API access point is open to direct access from public networks (aka, "the outside world"), which is NOT needed in a Dataverse installation.

We have always recommended having Solr (port 8983) firewalled off from public access in our installation guides. But we recommend that you double-check your firewall settings and verify that the port is not accessible from outside networks. The simplest quick test is to try the following URL in your browser:

  `http://<your Solr server address>:8983`

and confirm that you get "access denied" or that it times out, etc.

In most cases, when Solr runs on the same server as the Dataverse web application, you will only want the port accessible from localhost. We also recommend that you add the following arguments to the Solr startup command: -j jetty.host=127.0.0.1. This will make Solr accept connections from localhost only; adding redundancy, in case of the firewall failure.

In a case where Solr needs to run on a different host, make sure that the firewall limits access to the port only to the Dataverse web host(s), by specific ip address(es).

We would also like to reiterate that it is simply never a good idea to run Solr as root! Running the process as a non-privileged user would substantially minimize any potential damage even in the event that the instance is compromised.

Citation and Geospatial Metadata Block Updates

We updated two metadata blocks in this release. Updating these metadata blocks is mentioned in the step-by-step upgrade instructions below.

Run ReExportall

We made changes to the JSON Export in this release (#6426). If you'd like these changes to reflected in your JSON exports, you should run ReExportall as part of the upgrade process. We've included this in the step-by-step instructions below.

BinderHub

https://mybinder.org now supports spinning up Jupyter Notebooks and other computational environments from Dataverse DOIs.

Widgets update for OpenScholar

We updated the code for widgets so that they will keep working in OpenScholar sites after the upcoming upgrade OpenScholar upgrade to Drupal 8. If users of your dataverse have embedded widgets on an Openscholar site that upgrades to Drupal 8, you will need to run this Dataverse version (or later) for the widgets to keep working.

Payara tech preview

Dataverse 4 has always run on Glassfish 4.1 but changes in this release (PR #6523) should open the door to upgrading to Payara 5 eventually. Production installations of Dataverse should remain on Glassfish 4.1 but feedback from any experiments running Dataverse on Payara 5 is welcome via the usual channels.

Notes for Tool Developers and Integrators

Search API

The boolean parameter query_entities has been removed from the Search API. The former "true" behavior of "whether entities are queried via direct database calls (for developer use)" is now always true.

Additional fields are now available via the Search API, mostly related to information about specific dataset versions.

Complete List of Changes

For the complete list of code changes in this release, see the <a href="https://github.com/IQSS/dataverse/milestone/86?closed=1">4.19 milestone</a> in Github.

For help with upgrading, installing, or general questions please post to the <a href="https://groups.google.com/forum/#!forum/dataverse-community">Dataverse Google Group</a> or email [email protected].

Installation

If this is a new installation, please see our <a href="http://guides.dataverse.org/en/4.19/installation/">Installation Guide</a>.

Upgrade

1. Undeploy the previous version.

• <glassfish install path>/glassfish4/bin/asadmin list-applications
• <glassfish install path>/glassfish4/bin/asadmin undeploy dataverse

1. Stop glassfish and remove the generated directory, start.

• service glassfish stop
• remove the generated directory: rm -rf <glassfish install path>glassfish4/glassfish/domains/domain1/generated
• service glassfish start

1. Deploy this version.

• <glassfish install path>/glassfish4/bin/asadmin deploy <path>dataverse-4.19.war

1. Restart glassfish.

2. Update Geospatial Metadata Block

• wget https://github.com/IQSS/dataverse/releases/download/v4.19/geospatial.tsv
• curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @geospatial.tsv -H "Content-type: text/tab-separated-values"

1. (Optional) Run ReExportall to update JSON Exports
   

http://guides.dataverse.org/en/4.19/admin/metadataexport.html?highlight=export#batch-exports-through-the-api