The syntax of the scripting language presented here is outdated and needs a refresher. The general concepts still hold. If you want to do the current walkthrough, get a version of the avu dating back from early 2024.
Geoffrey is making a GUI for creating template, and Yanis is creating an adapter to do the ingestion work, that relies on the avu under the hood.
This page's content may evolve soon (today is 2025-01-24) to reflect these work initiatives.
¶ Templates
Templates are a way to describe how to create entities from a csv-like file. Templates are written in a templating language
The templating language allows us to create three types of things:
- Model
- Import
- Relationship
(There actually seem to be more type of things: model, configuration, relationship, import, export, enrichment, general. However there is nothing we know about the other types)
To use a model/import/relationship written in the templating language, one must first "compile" it to an entity. In other word, the templating language is useful for human to edit, but to be used one need to create a corresponding model/import/relationship entity.
This step will ensure that the template is correct (ie the code is valid templating language code), and will write it in a way that is easy to read for Orchestra.
¶ Model
Model describe what datapoint exist in a certain type of entity. It is the skeleton of the entity that is just missing the values, which will later be populated when we import it from an import
eg.
A Country model may be an entity of Class COUNTRY, and has a NAME_ATTRIBUTE and a POPULATION_ATTRIBUTE
¶ Import
Import are used to ingest a bunch of entities of a same model from a csv-like file. An import is specific to a certain model. It basically describes which column of the csv-like file will map to which datapoint in the target model.
eg.
The "name" column from the csv file shall be mapped to the "Name" of the model, and the "pop" column from the csv file shall be mapped to the "Population" of the model.
Note that it is possible to have multiple import for the same model. It is useful if multiple csv formatted differently describe the same thing.
¶ Relationships
This allows to create bilateral reference between entities ingested through a template.
¶ Walk through
This is a step-by-step tutorial to import your first entities.
We'll be importing very simple hotels and countries.
Here are the files that you will need:
walkthrough.zip
Assuming a linux system, and a running LEDR appliance.
After unziping the archive, the archive should contain these files:
$ ls walkthrough/
walkthrough/
├── countries.tsv
├── country_hotel_relationship.txt
├── country_import.txt
├── country_model.txt
├── hotel_import.txt
├── hotel_model.txt
├── hotels.tsv
The next step is to copy these files in the /AvesTerra/ folder of the avu container, which is one of the docker container shipped with the LEDR appliance. We need this to be able to use these templates with the avu.
Here is a little shell script to copy file from your host system to the /AvesTerra/ folder of the avu container
avu_put.sh:
docker cp $1 $(docker ps | grep avu | awk '{ print $1 }'):AvesTerra/
You can copy the files with
$ avu_put.sh ./walkthrough/
Successfully copied 12.3kB to 4d73c4a26176:AvesTerra/
Then we need to import the hotel_model.txt and hodel_import.txt in the same registry. The standard model registry is <30>, so we'll put it there.
Run the avu on localhost and the following commands
> template <0|0|30> "walkthrough/hotel_model.txt"
> template <0|0|30> "walkthrough/hotel_import.txt"
We now are ready to import the hotels data !
We do need a registry to create all the entities in, so let's create it with the avu:
> registry "hotel registry" ""
<0|0|139944>
This created a registry entity of name "hotel registry" and with key "".
In my case, its EUID is <0|0|139944>
Last thing we need is the EUID of the import template we created in <30>, since this is what we will use to read the csv file:
> <0|0|30>
Name: Model Registry
Key : Model
Properties:
Hotel Model Template [Hotel] <0|0|139942>
Hotel Import Template [Hotel Import] <0|0|139943>
In my case, its EUID is <0|0|139943>.
We can now import the "walkthrough/hotels.tsv" with this command
Read the command as:
Import to registry <139944> using model template <139943> with model registry <30> and data file "walkthrough/hotels.tsv"
We needto give the model registry <30> because the import template knows the key of the model template. In our case, the key is Hotel, that's why the key of the model template in the <30> registry is Hotel.
> import <139944> <0|0|139943> <0|0|30> "walkthrough/hotels.tsv"
Warnings: 0
Errors: 0
Records: 5
Verified: 5
Imported: 5
If everything went right, it should have imported the hotels in the target registry.
If we look inside the registry:
> <0|0|139944>
Name: hotel registry
Properties:
1900 [1] <0|0|139945>
30 BENCOOLEN [2] <0|0|139946>
7 WONDERS HOSTEL [3] <0|0|139947>
HOTEL DE LA PORTE SAINT MALO [4] <0|0|139948>
CIVILIAN HOTEL [5] <0|0|139949>
> <0|0|139947>
Name: 7 WONDERS HOSTEL
Key : 3
Facts:
LOCATION_ATTRIBUTE
Features:
Hotel Address [address] "257 JALAN BESAR"
ID_ATTRIBUTE
Features:
Hotel Identifier [id] "3"
Hotel Name [name] "7 WONDERS HOSTEL"
TODO: Complete the walkthrough with the importation of countries, and add relationshipt between the two
¶ Getting template code from template entities
Templates are saved as entities. One can download the 'source code' of a template with the archive avu command (whose name will soon change).
eg
archive <2|1|522499> "hotels_template.txt"
will download the hotel template from the orchestra server.
note: The file will be downloaded inside the avu docker container, which is a bit of a pain.
A useful bash utility to copy files from the avu docker container to your machine is
docker cp $(docker ps | grep avu | awk '{ print $1 }'):AvesTerra/$1 .
I called it avu_get.sh on my machine, I can use it like
$ cd /tmp/
$ avucp.sh hotels_template.txt
Successfully copied 4.1kB to /tmp/.
to copy the file "hotels_template.txt" I downloaded through the avu to my current working directory (in this example, to /tmp/)
Here is what the hotel_templates I downloaded looks like: