Overview

Meveo is a JavaEE8 web application and runs on the following technology stack:

  • JavaEE8

  • Wildfly15

  • Keycloak5

  • PostgreSQL9.6

  • ElasticSearch 2.3.4

Configuration

Module Versioning

To be able to introduced dependencies on Meveo module, it needed to be versioned.

Version Number

The version number would be a compound of 3 numbers separated by a digit, and following this pattern : MAJOR.MINOR.PATCH. A major version may imply a complete incompatibility between two versions of a module. A minor version implies that we can upgrade from an older version using a patch. A patch version implies only bug fixes that do not require any patch to be run.

Meveo Version Number

Sometimes, a module requires a specific version of meveo. For example, if it uses a connector, he will need a version superior to 6.1.0. However, if it needs a feature that was removed or refactored, like the scripting system before the refactor, the module will need a version inferior to 6.0.5. So we can set lower and upper bounds to the version.

Module A

Module B

Module C

Bounds

Resolved Version

Min: 6.0.5

>6.0.5

Bounds

Resolved Version

Max: 6.1.0

<6.1.0

Bounds

Resolved Version

Min: 6.2.0, Max: 6.2.0

6.2.0

Update Patch

An update patch is a script that has to be run in order to upgrade from a module version to another version. So it means that we need to add a mapping between old versions and a list of patches. For each patch, we must specify whether to execute it before or after the upgrade.

Here is an example :

Let’s suppose that we have made a module named "ExampleModule", with the first version named "1.0.0". It has no associated patch. Then, we run a Scrum Sprint and we end up having version "1.1.0" of our module. However, we changed the data type of a column from number to string so we need to apply a patch that saves the data, change the data type, and re-insert the data as string. So this patch, named "PatchA" is available for "1.0.0" → "1.1.0" and must be executed before migration. Later, we don’t touch the module but a new version of Meveo is released that implies some change in the data structure of the tables, so a new version of our module is released at the same time, with version "1.2.0" that is tied to the new Meveo version. A patch script named "PatchB" is associated that allows you to upgrade from version "1.1.0" → "1.2.0" and that must be executed before the upgrade. But we can also upgrade directly from version 1.0.0 to version 1.2.0 by executing the PatchA, then PatchB. Later, we modify the module but the modifications do not require any patch, it’s the 1.2.1 release. If we recap this in a table, that would give :

ExampleModule v1.0.0

ExampleModule v1.1.0

ExampleModule v1.2.0

ExampleModule v1.2.1

1.0.x→1.1.0

Before

After

PatchA

None

1.1.x→1.2.0

Before

After

PatchB

None

1.1.x→1.2.1

Before

After

PatchB

None

1.0.x→1.2.0

Before

After

PatchA

None

PatchB

None

1.0.x→1.2.1

Before

After

PatchA

None

PatchB

None

1.2.x→1.2.1

Release Process

Storage

For having a versioning process, we must have a release process. Once a release is done, we cannot modify it, it’s frozen. The deliverable file of a module is an XML file, so the release process should be easy. We can store them in a dedicated table (shared between meveo instances) along with the information defined previously. In the preliminary release, maybe we don’t need a shared database between meveo instances. In this case, we will need functionality to export / import releases. The export / import would consist of the following: the release information, the patches, and the XML file corresponding to the module. We should create a new listing to manage the releases of a module.

Releasing Action

When releasing a module version, we must specify the next iteration version and the patches that allow migration from a version to the released version. Optionally, we can also write some changelog. It implies we have to add a "current version" along with a "is in draft" field to the Meveo module model. A module that is not in draft cannot be modified, nor can its components. Before the release is done, we execute every test suite for the functions of the module and if there is a failure, we inform the user what has failed and the release is not realized. If the user still wants to make the release, he will either have to correct the function or remove the tests.

Migration Upgrade Process

Update from a release present in database

In a module detail, we must add a button "Upgrade" that allows to trigger the upgrade of a module. Only modules that are not in draft mode can be upgraded. The upgrade consists in :

1.) applying the "before" patches that match the current version of the module

2.) importing the XML module file

3.) applying the "after" patches that match the current version of the module

The user can only choose a release that matches the current meveo version and that has an available "from version" matching the current version of the module. In case the user wants to upgrade from an old version to the newest version, we must find an available "path" that leads from old version to the newest by applying intermediary upgrade. For instance, if we have version 1.0.0 of a module, and that we want to upgrade to version 1.4.0 directly but there is no direct way but we have version 1.1.0, 1.2.0 and 1.3.0 in database, we can apply successively the upgrade from 1.0.0 → 1.1.0 → 1.2.0 → 1.3.0 → 1.4.0. If we imagine there is a direct upgrade available from 1.2.0 to 1.4.0 it would give : 1.0.0 → 1.2.0 → 1.4.0. So we must find the shortest path, and if there is no path (ie: we don’t have version 1.1.0), the upgrade cannot be applied.

At the end of the update, each test suite of the function present in the module must be run and be ok, if not we will roll-back to the previous version of the module. Making a rollback implies that we must make backup of the concerned elements before starting the update.

Update from an XML file that is not present in database

If we want to upgrade the module in an environment that does not have the release in its database, we should be able to provide an XML module release file. In this case, we will first upload it to the database and then we will apply the same process as before. We should also look for releases in sibling meveo instances if not found in our own environment.

Module Changelogs

To keep traceability of our module release, we should be able to see the differences from one version to another.

Module Comparison

When in the detail of a module, we should have a "compare with version …" button that allow user to select any other version of the same module present in the database and to visualize, by type of entities, what have been change, for example : * Fields added / removed of a custom entity template

  • Code differences in a script instance

  • Different method (GET / POST) used for an endpoint

Changelog Algorithm

In order to produce a changelog, we first need to parse the XML of the other release, then convert it to module items that can be compared to items of current module. In order to do that, we have to find a way to mark comparable properties for each module item. For instance it is interesting for us to know if we changed the data store of a custom entity template, but we don’t want to compare its child templates. The output would look like the table below.

Custom Entity Templates

v1.1.2 (current)

MyTemplate:

- myNewField: String - add (style=color: green)

- myField: String (Long) - modified (style=String is red, Long is green)

- myLastField: Long - remove (style=color: red)

v1.0.0

MyTemplate:

- myField: String

- myLastField: String

Services

Scripts

In Meveo, you can create Java and Javascript scripts. The code of these scripts are stored in the Meveo git repository. You can either create them from the user interface under /pages/admin/storages/scriptInstances/scriptInstances.jsf or using the REST Api.

A script can be executed from the REST Api using the default endpoint, by a Notification, a Job or an Endpoint.

To ease the communication with other components of Meveo, the user can define named inputs and outputs.

Java scripts

All java scripts should be classes that implements the org.meveo.script.ScriptInterface interface. The code of the script is deduced from full name (package name + class name) of the java class.

The parameters can either be taken from the map or be defined through setters. The parameters taken from setters will automatically be added to the list of user defined inputs.

The output can either be added to the map (which is read after the script execution) or defined in a getter.The outputs taken from getters will automatically be added to the list of user defined outputs.

package org.meveo.test;

import org.meveo.service.script.Script;
import java.util.Map;
import org.meveo.admin.exception.BusinessException;

public class TestSetterAndGetters extends Script {

    private String field1;

    @Override
    public void execute(Map<String, Object> methodContext) throws BusinessException {
        String fieldFromMap = methodContext.get("fieldFromMap");
        methodContext.put("output2", "output2");
    }

    /**
    * @param fieldValue Value of the field.
    * Just a test.
    */
    public void setField1(String fieldValue){
        this.field1 = fieldValue;
    }

    /**
    * @return The field previously set
    */
    public String getOutput1(){
        return this.field1 + "increased";
    }
}

Javascript scripts

The javascript scripts should just be simple scripts that will be evaluated in a java context by the GraalJS library.

The parameters are injected as variables in the script context, so we can access them directly. We can also access them using the methodContext object.

The outputs of the script must be added in the methodContext map.

var inputA = methodContext.get('input');
methodContext.put('result', inputA * 2);
var declaredVar = input * 3;
methodContext.put('result2', declaredVar);

Git

Git server

You can access the git server like in any other git hosting provider. The remote origin will be accessible at https://{host}(:{port})/{context-root}/git/{repository-code}.

You must authenticate using your Meveo login / password

Git repository

A git repository entity reflect a git repository hosted by the Meveo instance. It is defined by a code, a description, a optional remote origin (with optional username and password), reading roles and writing roles. You can access the mangament page at /pages/admin/storages/repositories/gitRepositories.jsf.

Definition

Remote origin

When a git repository has a remote origin, the user can push / pull to / from the remote origin. Default credentials can be defined, but we can also specify them at execution of the action.

Reading roles

If reading roles are provided, only user with one of these roles will be able to pull and clone the concerned repository.

Writing roles

If writing roles are provided, only user with one of these roles will be able to push to the concerned repository.

Operations

Commit [API]

When committing a repository, you should provide a commit message and a pattern of the commited files (can be a regex). This operation is only available from API at the moment.

Push & Pull

When pushing or pulling a repository, you can specify credentials different from the default credentials. The Meveo instance will behave in the same way when he receives commits from a pull as when he receives a commit from a git client.

Import a repository

A zip file can be imported from the file system. If the repository already exist, it will be overriden. This operation is only available for repositories that have no remote origin.

Export a repository

The content of a repository can be exported as a zip file. The branch to export can be specified, default branch will be the current branch of the repository.

Managing branches [API]

We can checkout, create and delete branches of a git repository. This operation is only available from API at the moment

How to use Git in Meveo ?

Currently, the ontology elements and the scripts are stored in a dedicated git repository hosted by the running Meveo instance, called "Meveo" repository and accessible at https://{host}(:{port})/{context-root}/git/Meveo.

Scripts

If you clone the Meveo repository, then make some changes to a script, and finally push it, the concerned script will be re-compiled by Meveo and updated. If you create or delete scripts, the action will be reflected on the Meveo instance.

Ontology

The ontology elements are serialized under an extended JSON Schema specification. The same rules than for script applies, so if you create, modify or delete a json file, it will be reflected on the Meveo instance you pushed to.

Endpoints

When updating, creating, or deleting an endpoint, a javascript file will be created. This file contains a default function exported that make a fetch call to the corresponding endpoint. It takes into account the method (GET / POST), the path parameters and the body / query parameters. The return value of this function is a Response object that must be handled.

Endpoint Open API Documentation

Swagger Dynamically Generated Document

MEVEO has the capability to dynamically generate a Swagger standard schema of a given endpoint. This feature is available via API.

GET
/endpoint/openApi/{endpointCode}

Javascript Auto-Generated Interface

To automate the creation of GUI, MEVEO provides an endpoint that can be used to manage a custom entity template. It serves a dynamically generated endpoint javascript interface that can be used by the frontend application to send CRUD requests to the server.

Request Schema

The request schema is an Open API v3 Draft7 standard document that is created from the non-path parameters of an endpoint (field parametersMapping).

These parameters are passed to an endpoint and mapped to the linked script.

Currently two types of parameters are supported. Get and Body. Get is basically the query parameters, it’s data type corresponds to the Java native types. On the other hand a body parameter, is represented as a JSON object. It can be as complicated as needed. In Meveo, it can be a custom entity template on several layers, meaning custom entity template a can have a field custom entity template b.

This feature is available via API at:

GET
/endpoint/schema/{endpointCode}/request

Response Schema

The response schema is an Open API v3 Draft7 standard document. It represents the data type saved in endpoint’s returnedVariableName.

The returnedVariableName, is a name of a field inside a script where it is mapped from the endpoint. It can be a Java native data type and can be a custom entity template as well.

For example, we have a script ScriptTest that is linked to our endpoint. This ScriptTest has a custom entity template property named Account.

public class ScriptTest extends Script {

        private Account account;

        public Account getAccount() {
                return account;
        }

        public void setAccount(Account account) {
                this.account = account;
        }
}

To tell our endpoint that we want to return the value of the account after the execution, we need to set the value of endpoint.returnedVariableName=account.

This feature is available via API at:

GET
/endpoint/schema/{endpointCode}/response

Javascript Interface

An API that provides a working service or interface for managing CRUD operations of a custom entity template is available. This interface is automatically created and save in Meveo’s internal Git system, which is normally located at <PROVIDERS_DIR>\git\Meveo\endpoints.

GET
/endpoint/openApi/{endpointCode}

For reference, here is an example endpoint’s javascript interface

const buildRequestParameters = (parameters, schema) => {
    if (schema) {
        const errors = []
        const requestParameters = Object.keys(
            schema.properties,
        ).reduce((reqParameters, property) => {
            const value = parameters[property]
            const isRequired = schema.properties[property].required

            if (isRequired && !value) {
                errors.push(`${property} is required.`)
            } else if (!!value) {
                return {
                    ...reqParameters,
                    [property]: value
                }
            }
            return reqParameters
        }, {})
        if (errors.length > 0) {
            throw errors
        }
        return requestParameters
    }
    return null
}

const EVENT = {
    SUCCESS: "Updatepost-OpenApiGenerateCetTest-endpoint-SUCCESS",
    ERROR: "Updatepost-OpenApiGenerateCetTest-endpoint-ERROR"
};

export const registerEventListeners = (
    component,
    successCallback,
    errorCallback
) => {
    if (successCallback) {
        component.addEventListener(EVENT.SUCCESS, successCallback);
    }
    if (errorCallback) {
        component.addEventListener(EVENT.ERROR, errorCallback);
    }
};

export const getRequestSchema = async (parameters, config) => {
    return {
  "title": "post-OpenApiGenerateCetTest-endpointRequest",
  "id": "post-OpenApiGenerateCetTest-endpointRequest",
  "default": "Schema definition for post-OpenApiGenerateCetTest-endpoint",
  "$schema": "http://json-schema.org/draft-07/schema",
  "type": "object",
  "properties": {
    "qparam3": {
      "title": "Consumption",
      "description": "Consumption",
      "id": "Consumption",
      "storages": [
        "SQL"
      ],
      "type": "object",
      "properties": {
        "date": {
          "title": "Consumption.date",
          "description": "Date",
          "id": "CE_Consumption_date",
          "storages": [
            "SQL"
          ],
          "nullable": false,
          "readOnly": false,
          "versionable": false,
          "indexType": "INDEX_NOT_ANALYZE",
          "type": "string",
          "format": "date-time"
        },
        "amount": {
          "title": "Consumption.amount",
          "description": "Amount",
          "id": "CE_Consumption_amount",
          "storages": [
            "SQL"
          ],
          "nullable": false,
          "readOnly": false,
          "versionable": false,
          "indexType": "INDEX_NOT_ANALYZE",
          "type": "integer"
        },
        "account": {
          "title": "Consumption.account",
          "description": "Account",
          "id": "CE_Consumption_account",
          "storages": [
            "SQL"
          ],
          "nullable": false,
          "readOnly": false,
          "versionable": false,
          "indexType": "INDEX_NOT_ANALYZE",
          "type": "string",
          "minLength": 1,
          "maxLength": 255
        }
      },
      "required": [
        "account",
        "amount",
        "date"
      ]
    },
    "qparam2": {
      "title": "Account",
      "description": "Account",
      "id": "Account",
      "storages": [
        "SQL"
      ],
      "type": "object",
      "properties": {
        "accountCode": {
          "title": "Account.accountCode",
          "description": "Account code",
          "id": "CE_Account_accountCode",
          "storages": [
            "SQL"
          ],
          "nullable": false,
          "readOnly": false,
          "versionable": false,
          "indexType": "INDEX_NOT_ANALYZE",
          "type": "string",
          "minLength": 1,
          "maxLength": 255
        },
        "accountType": {
          "title": "Account.accountType",
          "description": "Account type",
          "id": "CE_Account_accountType",
          "storages": [
            "SQL"
          ],
          "nullable": false,
          "readOnly": false,
          "versionable": false,
          "indexType": "INDEX_NOT_ANALYZE",
          "type": "string",
          "minLength": 1,
          "maxLength": 255
        }
      },
      "required": [
        "accountCode",
        "accountType"
      ]
    },
    "qparam1": {
      "title": "qparam1",
      "type": "string",
      "minLength": 1
    }
  }
}
};

export const getResponseSchema = async (parameters, config) => {
    return {
  "title": "post-OpenApiGenerateCetTest-endpointResponse",
  "id": "post-OpenApiGenerateCetTest-endpointResponse",
  "default": "Schema definition for post-OpenApiGenerateCetTest-endpoint",
  "$schema": "http://json-schema.org/draft-07/schema",
  "type": "object",
  "properties": {
    "consumption": {
      "title": "Consumption",
      "description": "Consumption",
      "id": "Consumption",
      "storages": [
        "SQL"
      ],
      "type": "object",
      "properties": {
        "date": {
          "title": "Consumption.date",
          "description": "Date",
          "id": "CE_Consumption_date",
          "storages": [
            "SQL"
          ],
          "nullable": false,
          "readOnly": false,
          "versionable": false,
          "indexType": "INDEX_NOT_ANALYZE",
          "type": "string",
          "format": "date-time"
        },
        "amount": {
          "title": "Consumption.amount",
          "description": "Amount",
          "id": "CE_Consumption_amount",
          "storages": [
            "SQL"
          ],
          "nullable": false,
          "readOnly": false,
          "versionable": false,
          "indexType": "INDEX_NOT_ANALYZE",
          "type": "integer"
        },
        "account": {
          "title": "Consumption.account",
          "description": "Account",
          "id": "CE_Consumption_account",
          "storages": [
            "SQL"
          ],
          "nullable": false,
          "readOnly": false,
          "versionable": false,
          "indexType": "INDEX_NOT_ANALYZE",
          "type": "string",
          "minLength": 1,
          "maxLength": 255
        }
      },
      "required": [
        "account",
        "amount",
        "date"
      ]
    }
  }
}
}

export const executeApiCall = async (
    component,
    params,
    successCallback, // optional
    errorCallback // optional
) => {
    registerEventListeners(component, successCallback, errorCallback);
    const parameters = params || {};
    const {
        token,
        config
    } = parameters;

    // the name of the config variable is the name of the module
    const {
        Updatepost-OpenApiGenerateCetTest-endpoint: {
            OVERRIDE_URL,
            USE_MOCK
        }
    } = config || {};

    // the baseUrl can be overridden by indicating a OVERRIDE_URL in config,
    // by default it will use the same URL as the client application
    // or if this is auto-generated by meveo, it will have the server's host url
    const baseUrl = OVERRIDE_URL || window.location.origin; // || server.host.url

    // just an example how to use the useMock parameter to switch between mock and actual endpoints.
    const apiUrl = USE_MOCK ?
        `${baseUrl}/auth/realms/meveo/account?useMock=true` :
        `${baseUrl}/auth/realms/meveo/account`;

    //fetch request schema to filter out optional parameters that should not be passed into the request
    try {
        const requestSchema = await getRequestSchema(parameters);
        const requestParameters = buildRequestParameters(parameters, requestSchema);
        const parameterKeys = Object.keys(requestParameters || {});
        const hasParameters = requestParameters && parameterKeys.length > 0;

        const requestUrl = new URL(apiUrl);
        if (hasParameters) {
            parameterKeys.forEach(key => {
                requestUrl.searchParams.append(key, requestParameters[key]);
            });
        }

        const headers = new Headers();
        headers.append("Content-Type", "application/json");
        headers.append("Accept", "application/json");
        headers.append("Authorization", `Bearer ${token}`);

        const options = {
            method: "GET",
            headers
        };

        const response = await fetch(requestUrl, options);
        if (!response.ok) {
            throw [
                `Encountered error calling API: ${apiUrl}`,
                `Status code: ${response.status} [${response.statusText}]`
            ];
        }
        // if accept = "application/json" otherwise return response.text()
        const result = await response.json();
        component.dispatchEvent(
            new CustomEvent(EVENT.SUCCESS, {
                detail: {
                    result
                },
                bubbles: true
            })
        );
    } catch (error) {
        component.dispatchEvent(
            new CustomEvent(EVENT.ERROR, {
                detail: {
                    error
                },
                bubbles: true
            })
        );
    }
};