Skip to content

Developer Guide#

Developing the metadata store involves at least the local setup of the services hub-repo and hub-search. This guide helps you step by step to set it up locally. This guide only involves the backend services. If you want to add the piveau frontend, have a look here.

Prerequisites#

The prerequisites are just tested recommendations. You can give it a try with different versions or other tools. In general, piveau will work on Linux, Windows, and MacOS. For clarity, this guide uses standard Linux commands.

  • Git
  • Docker >= 24
  • Docker Compose v2
  • JDK >= 17
  • Maven >= 3.6
  • Free ports on your machine

Getting the Code#

Create a new directory in your user space and navigate to it.

mkdir piveau && cd piveau
Clone hub-repo. 
git clone https://gitlab.com/piveau/hub/piveau-hub-repo
Clone hub-search. 
git clone https://gitlab.com/piveau/hub/piveau-hub-search

Important

On Windows: Make sure that you are using LF line separators!

This will give you the latest develop version of the code. Be aware it might include bugs or might not work at all in a given moment. If you want to make sure to get a stable version, checkout out the newest tag of each repo. The versioning of the two services does align, so don't worry if you check out two different versions.

.
└─ piveau
    ├─ piveau-hub-repo
    └─ piveau-hub-search

Starting the Databases#

The hub-repo repository comes with a convenient Docker Compose file, that starts all the required databases (piveau-hub-repo/docker-compose-dbs.yml).

Here are some insights about it:

services:
  elasticsearch: # (1)!
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.9 
    logging:
      options:
        max-size: "50m"
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data # (2)!
    environment: # (3)!
      - bootstrap.memory_lock=true
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms2G -Xmx4G
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - 9200:9200  # (4)!
      - 9300:9300

  virtuoso: # (5)!
    image: openlink/virtuoso-opensource-7
    logging:
      options:
        max-size: "50m"
    volumes: 
      - virtuoso_db:/opt/virtuoso-opensource/var/lib/virtuoso/db # (6)!
    ports:
      - 8890:8890 # (7)!
      - 1111:1111
    environment:
      - DBA_PASSWORD=dba # (8)!

  shadow-virtuoso: # (9)!
    image: openlink/virtuoso-opensource-7
    logging:
      options:
        max-size: "50m"
    volumes:
      - shadow_virtuoso_db:/opt/virtuoso-opensource/var/lib/virtuoso/db
    ports:
      - 9890:8890
      - 9111:1111
    environment:
      - DBA_PASSWORD=dba

volumes: # (10)!
  virtuoso_db:
  shadow_virtuoso_db:
  elasticsearch:
  1. Starts Elasticsearch, that is used by hub-search as a high-performance search backend.
  2. Mount the data directory to a volume for persistence.
  3. These settings are important, so Elasticsearch does not eat up your local resources.
  4. The Elasticsearch API will be available at this port. Convenient for debugging too.
  5. Start the Virtuoso Triplestore - the main database of piveau.
  6. Mount the database directory to a volume for persistence.
  7. This port gives access to the management backend of Virtuoso and the SPARQL endpoint.
  8. The password for accessing the API of Virtuoso.
  9. A second Virtuoso triplestore for storing non-public data. This is necessary since, the main Virtuoso opens the SPARQL endpoint and all data to the public in most use cases.
  10. These volumes are used to persist data across container restarts.

You can now start the database by executing:

$ cd piveau-hub-repo
$ docker compose -f docker-compose-dbs.yml up -d

You can verify that every is up and running by checking each database and/or execute $ docker ps.

If you are done with development you can just shutdown the databases with:

$ docker compose -f docker-compose-dbs.yml down

Sometimes you want to start from scratch with empty databases. You can achieve this, by removing the volumes when shutting down:

$ docker compose -f docker-compose-dbs.yml down -v

Set the Configuration#

Running hub-repo#

Setup Your IDE#

Post Installation Steps#

Load the Vocabularies#

You will need the vocabularies to use all functionalities. The most simple way to load them is to use the CLI of hub-repo. Please refer to the CLI reference for details.

Get Example Datasets#

Setup and Configure Keycloak#

  • Setup and configure Keycloak