Client properties

This file contains the minimum service info to connect applications with dataClay. It is also loaded automatically during the initialization process and its default path is./cfgfiles/client.properties, which can be overriden by setting the environment variableDATACLAYCLIENTCONFIG.

Here is an example:

HOST=localhost TCPPORT=11034

As you can see, it only requires two properties to be defined:HOSTandTCPPORT; comprising the full address to be resolved in order to initialize a session with dataClay from your application.

9. Hands-on Examples

This section explains how to run theHelloPeopleapplications (introduced in section 2.1) with a minimal docker-based installation in your own machine (as explained in section 7).

The full process is based on Linux-based Bash scripts and both Java and Python versions of the application will be tested.

9.1 Requirements

To run the tests, your system needs to fulfill the following requirements:

1. Java 1.8: to run the Java version of HelloPeople and the dClayTool.

2. Python 3: to run Python version of HelloPeople.

3. Python pip: to install Python client library.

4. Docker support: to launch a Docker-based installation of dataClay including:

docker tools docker-compose 9.2 Download

Start downloading the zip file from our website (Downloads tab):

https://www.bsc.es/research-and-development/software-and-apps/software-list/dataclay Extract it into, for example, a directory nameddataClayand follow next steps.

9.3 Zip contents

After extracting the contents of the zip file you will find the following directories indataClayfolder:

examples/ apps including HelloPeople and its class model.

78

lib/ the Java client lib: dataClay.jar and dependencies.

tools/ dClayTool used to register class models.

containers/docker/ docker-compose examples to orchestrate dataClay services with Docker containers/singularity/ singularity setup example

Notice that Python client library is not included since it will be installed directly frompiprepositories.

9.4 Check and install

At this point you can run the main script that checks and installs dataClay:

> cd dataClay

> bash install.sh

This script checks that your system has proper versions of Java and Python, as well as docker support. It also retrieves our Docker images from Docker Hub if necessary and installs the Python client library from the Python Package Index (PyPI) usingpipcommand.

Analogously, you can also uninstall dataClay:

> cd dataClay

> bash uninstall.sh

9.5 Run HelloPeople

Now everything is ready to run your first dataClay application either in Java or Python. If you navigate throughexamples directory you will find thatHelloPeoplefolder contains both the Java and Python implementations.

To run the Java version with Object Store methods:

> cd dataClay/examples/HelloPeople/java

> bash demo_os.sh

To run the Java version with Object Oriented methods:

> cd dataClay/examples/HelloPeople/java

> bash demo_oo.sh

To run the Python version with Object Store methods:

> cd dataClay/examples/HelloPeople/python

> bash demo_os.sh

To run the Python version with Object Oriented methods:

> cd dataClay/examples/HelloPeople/python

> bash demo_oo.sh

9.6 Dissecting demo scripts

If you want to take a look at thedemo_*.shscripts used to execute HelloPeople both in Java and Python, you will basically find these following sections:

9.7 Other examples

1. Docker initialization. dataClay containers are initialized withdocker-composecommand.

2. Management operations. Basic user information is registered in the system as well as the application class model.

ThedClayToolis used to this end.

3. Stubs retrieval. ThedClayToolis now used to retrieve the stub classes corresponding to previously registered model and stores them instubsdirectory.

4. App compilation. HelloPeople is now compiled using the downloaded stubs and its classes are stored inbin directory. This is step is not necessary in Python.

5. App execution. HelloPeople is executed.

6. Shut down. dataClay containers are shut down withdocker-composecommand.

Notice that application examples also contain acfgfilesdirectory including thesession.propertiesand the client.propertiesfiles. As exposed in section 8.2 these files are required to establish the connection and initialize a session with dataClay.

9.7 Other examples

Besides HelloPeople, in the examples directory, you will find another two well-known applications with their correspond- ing class models: K-means and Wordcount. K-means is a data clustering algorithm commonly used for machine learning (http://en.wikipedia.org/wiki/K-means_clustering) and Wordcount is a simple application that counts all the occurrences for each unique word within a set of texts.

These applications not only benefit from dataClay persistence and its runtime environment, but also from its integration with the COMP Superscalar programming model [3] [4] for task parallelization. You can download COMP Superscalar from this link:

https://www.bsc.es/research-and-development/software-and-apps/software-list/comp-superscalar

In the case of Python applications, parallelism can be exploited as introduced in section 7.1.2 by running multiple Execution Environmentson a single node. A specific docker-compose file to orchestrate this kind of scenario is also provided.

If you want to learn more about these examples or need help with your own applications, please contact us:[email protected]

9.8 Singularity

dataClay can also be deployed using Singularity by converting our Docker images into Singularity ones. We provide an example to perform this setup and orchestrate Singularity containers in the directorycontainers/singularityof the downloadable zip (introduced in Section 9.2). Current supported Singularity version is≥2.4.2.

9.9 POM based projects

If you are deploying pom-based applications you can add the following dependency into your pom file to install the Java client library:

<dependency>

<groupId>dataclay</groupId>

<artifactId>dataclay</artifactId>

<version>1.0</version>

</dependency>

Current version of dataClay is not hosted through Maven central repository, so you also have to add the following repository to resolve previous dependency from our GitHub repository:

<repository>

<id>DataClay repository</id>

<url>https://raw.github.com/bsc-ssrg/dataclay-maven/repository</url>

</repository>

VI

Bibliography . . . 83 Index . . . 85

Bibliography and index

Bibliography

[1] Jonathan Martí, Anna Queralt, Daniel Gasull, Alex Barceló, Juan José Costa, and Toni Cortes.

“Dataclay: A distributed data store for effective inter-player data sharing”. In: Journal of Systems and Software131 (2017), pages 129–145. ISSN: 0164-1212.DOI:https://doi.

org/10.1016/j.jss.2017.05.080(cited on page 11).

[2] Jonathan Martí Fraiz. “dataClay : next generation object storage”. In:TDX (Tesis Doctorals en Xarxa)(Mar. 2017).URL:http://www.tdx.cat/handle/10803/405907(cited on page 11).

[3] Enric Tejedor, Yolanda Becerra, Guillem Alomar, Anna Queralt, Rosa M Badia, Jordi Torres, Toni Cortes, and Jesús Labarta. “Pycompss: Parallel computational workflows in python”.

In:The International Journal of High Performance Computing Applications31.1 (2017), pages 66–82.DOI:https://doi.org/10.1177/1094342015594678(cited on page 79).

[4] Rosa M. Badia, Javier Conejero, Carlos Diaz, Jorge Ejarque, Daniele Lezzi, Francesc Lordan, Cristian Ramon-Cortes, and Raul Sirvent. “COMP Superscalar, an interoperable programming framework”. In:SoftwareX3-4 (2015), pages 32–36.ISSN: 2352-7110.DOI:https://doi.

org/10.1016/j.softx.2015.10.004(cited on page 79).