Overview
Meveo is a JavaEE8 web application and runs on the following technology stack:
-
JavaEE8
-
Wildfly15
-
Keycloak5
-
PostgreSQL9.6
-
ElasticSearch 2.3.4
Configuration
Module Versioning
To be able to introduced dependencies on Meveo module, it needed to be versioned.
Version Number
The version number would be a compound of 3 numbers separated by a digit, and following this pattern : MAJOR.MINOR.PATCH. A major version may imply a complete incompatibility between two versions of a module. A minor version implies that we can upgrade from an older version using a patch. A patch version implies only bug fixes that do not require any patch to be run.
Meveo Version Number
Sometimes, a module requires a specific version of meveo. For example, if it uses a connector, he will need a version superior to 6.1.0. However, if it needs a feature that was removed or refactored, like the scripting system before the refactor, the module will need a version inferior to 6.0.5. So we can set lower and upper bounds to the version.
Module A |
Module B |
Module C |
||||||||||||
|
|
|
Update Patch
An update patch is a script that has to be run in order to upgrade from a module version to another version. So it means that we need to add a mapping between old versions and a list of patches. For each patch, we must specify whether to execute it before or after the upgrade.
Here is an example :
Let’s suppose that we have made a module named "ExampleModule", with the first version named "1.0.0". It has no associated patch. Then, we run a Scrum Sprint and we end up having version "1.1.0" of our module. However, we changed the data type of a column from number to string so we need to apply a patch that saves the data, change the data type, and re-insert the data as string. So this patch, named "PatchA" is available for "1.0.0" → "1.1.0" and must be executed before migration. Later, we don’t touch the module but a new version of Meveo is released that implies some change in the data structure of the tables, so a new version of our module is released at the same time, with version "1.2.0" that is tied to the new Meveo version. A patch script named "PatchB" is associated that allows you to upgrade from version "1.1.0" → "1.2.0" and that must be executed before the upgrade. But we can also upgrade directly from version 1.0.0 to version 1.2.0 by executing the PatchA, then PatchB. Later, we modify the module but the modifications do not require any patch, it’s the 1.2.1 release. If we recap this in a table, that would give :
ExampleModule v1.0.0 |
ExampleModule v1.1.0 |
ExampleModule v1.2.0 |
ExampleModule v1.2.1 |
||||||||||||
1.0.x→1.1.0
|
1.1.x→1.2.0
|
1.1.x→1.2.1
|
|||||||||||||
1.0.x→1.2.0
|
1.0.x→1.2.1
|
||||||||||||||
1.2.x→1.2.1 |
Release Process
Storage
For having a versioning process, we must have a release process. Once a release is done, we cannot modify it, it’s frozen. The deliverable file of a module is an XML file, so the release process should be easy. We can store them in a dedicated table (shared between meveo instances) along with the information defined previously. In the preliminary release, maybe we don’t need a shared database between meveo instances. In this case, we will need functionality to export / import releases. The export / import would consist of the following: the release information, the patches, and the XML file corresponding to the module. We should create a new listing to manage the releases of a module.
Releasing Action
When releasing a module version, we must specify the next iteration version and the patches that allow migration from a version to the released version. Optionally, we can also write some changelog. It implies we have to add a "current version" along with a "is in draft" field to the Meveo module model. A module that is not in draft cannot be modified, nor can its components. Before the release is done, we execute every test suite for the functions of the module and if there is a failure, we inform the user what has failed and the release is not realized. If the user still wants to make the release, he will either have to correct the function or remove the tests.
Migration Upgrade Process
Update from a release present in database
In a module detail, we must add a button "Upgrade" that allows to trigger the upgrade of a module. Only modules that are not in draft mode can be upgraded. The upgrade consists in :
1.) applying the "before" patches that match the current version of the module
2.) importing the XML module file
3.) applying the "after" patches that match the current version of the module
The user can only choose a release that matches the current meveo version and that has an available "from version" matching the current version of the module. In case the user wants to upgrade from an old version to the newest version, we must find an available "path" that leads from old version to the newest by applying intermediary upgrade. For instance, if we have version 1.0.0 of a module, and that we want to upgrade to version 1.4.0 directly but there is no direct way but we have version 1.1.0, 1.2.0 and 1.3.0 in database, we can apply successively the upgrade from 1.0.0 → 1.1.0 → 1.2.0 → 1.3.0 → 1.4.0. If we imagine there is a direct upgrade available from 1.2.0 to 1.4.0 it would give : 1.0.0 → 1.2.0 → 1.4.0. So we must find the shortest path, and if there is no path (ie: we don’t have version 1.1.0), the upgrade cannot be applied.
At the end of the update, each test suite of the function present in the module must be run and be ok, if not we will roll-back to the previous version of the module. Making a rollback implies that we must make backup of the concerned elements before starting the update.
Update from an XML file that is not present in database
If we want to upgrade the module in an environment that does not have the release in its database, we should be able to provide an XML module release file. In this case, we will first upload it to the database and then we will apply the same process as before. We should also look for releases in sibling meveo instances if not found in our own environment.
Module Changelogs
To keep traceability of our module release, we should be able to see the differences from one version to another.
Module Comparison
When in the detail of a module, we should have a "compare with version …" button that allow user to select any other version of the same module present in the database and to visualize, by type of entities, what have been change, for example : * Fields added / removed of a custom entity template
-
Code differences in a script instance
-
Different method (GET / POST) used for an endpoint
Changelog Algorithm
In order to produce a changelog, we first need to parse the XML of the other release, then convert it to module items that can be compared to items of current module. In order to do that, we have to find a way to mark comparable properties for each module item. For instance it is interesting for us to know if we changed the data store of a custom entity template, but we don’t want to compare its child templates. The output would look like the table below.
Custom Entity Templates |
|
v1.1.2 (current) |
MyTemplate: - myNewField: String - add (style=color: green) - myField: String (Long) - modified (style=String is red, Long is green) - myLastField: Long - remove (style=color: red) |
v1.0.0 |
MyTemplate: - myField: String - myLastField: String |
Services
Scripts
In Meveo, you can create Java and Javascript scripts. The code of these scripts are stored in the Meveo git repository. You can either create them from the user interface under /pages/admin/storages/scriptInstances/scriptInstances.jsf
or using the REST Api.
A script can be executed from the REST Api using the default endpoint, by a Notification, a Job or an Endpoint.
To ease the communication with other components of Meveo, the user can define named inputs and outputs.
Java scripts
All java scripts should be classes that implements the org.meveo.script.ScriptInterface interface. The code of the script is deduced from full name (package name + class name) of the java class.
The parameters can either be taken from the map or be defined through setters. The parameters taken from setters will automatically be added to the list of user defined inputs.
The output can either be added to the map (which is read after the script execution) or defined in a getter.The outputs taken from getters will automatically be added to the list of user defined outputs.
package org.meveo.test;
import org.meveo.service.script.Script;
import java.util.Map;
import org.meveo.admin.exception.BusinessException;
public class TestSetterAndGetters extends Script {
private String field1;
@Override
public void execute(Map<String, Object> methodContext) throws BusinessException {
String fieldFromMap = methodContext.get("fieldFromMap");
methodContext.put("output2", "output2");
}
/**
* @param fieldValue Value of the field.
* Just a test.
*/
public void setField1(String fieldValue){
this.field1 = fieldValue;
}
/**
* @return The field previously set
*/
public String getOutput1(){
return this.field1 + "increased";
}
}
Javascript scripts
The javascript scripts should just be simple scripts that will be evaluated in a java context by the GraalJS library.
The parameters are injected as variables in the script context, so we can access them directly. We can also access them using the methodContext
object.
The outputs of the script must be added in the methodContext
map.
var inputA = methodContext.get('input');
methodContext.put('result', inputA * 2);
var declaredVar = input * 3;
methodContext.put('result2', declaredVar);
Git
Git server
You can access the git server like in any other git hosting provider. The remote origin will be accessible at https://{host}(:{port})/{context-root}/git/{repository-code}
.
You must authenticate using your Meveo login / password
Git repository
A git repository entity reflect a git repository hosted by the Meveo instance. It is defined by a code, a description, a optional remote origin (with optional username and password), reading roles and writing roles. You can access the mangament page at /pages/admin/storages/repositories/gitRepositories.jsf
.
Definition
Remote origin
When a git repository has a remote origin, the user can push / pull to / from the remote origin. Default credentials can be defined, but we can also specify them at execution of the action.
Operations
Commit [API]
When committing a repository, you should provide a commit message and a pattern of the commited files (can be a regex). This operation is only available from API at the moment.
Push & Pull
When pushing or pulling a repository, you can specify credentials different from the default credentials. The Meveo instance will behave in the same way when he receives commits from a pull as when he receives a commit from a git client.
Import a repository
A zip file can be imported from the file system. If the repository already exist, it will be overriden. This operation is only available for repositories that have no remote origin.
How to use Git in Meveo ?
Currently, the ontology elements and the scripts are stored in a dedicated git repository hosted by the running Meveo instance, called "Meveo" repository and accessible at https://{host}(:{port})/{context-root}/git/Meveo
.
Scripts
If you clone the Meveo repository, then make some changes to a script, and finally push it, the concerned script will be re-compiled by Meveo and updated. If you create or delete scripts, the action will be reflected on the Meveo instance.
Ontology
The ontology elements are serialized under an extended JSON Schema specification. The same rules than for script applies, so if you create, modify or delete a json file, it will be reflected on the Meveo instance you pushed to.
Endpoints
When updating, creating, or deleting an endpoint, a javascript file will be created. This file contains a default function exported that make a fetch call to the corresponding endpoint. It takes into account the method (GET / POST), the path parameters and the body / query parameters. The return value of this function is a Response object that must be handled.
Endpoint Open API Documentation
Swagger Dynamically Generated Document
MEVEO has the capability to dynamically generate a Swagger standard schema of a given endpoint. This feature is available via API.
GET
/endpoint/openApi/{endpointCode}
Javascript Auto-Generated Interface
To automate the creation of GUI, MEVEO provides an endpoint that can be used to manage a custom entity template. It serves a dynamically generated endpoint javascript interface that can be used by the frontend application to send CRUD requests to the server.
Request Schema
The request schema is an Open API v3 Draft7 standard document that is created from the non-path parameters of an endpoint (field parametersMapping).
These parameters are passed to an endpoint and mapped to the linked script.
Currently two types of parameters are supported. Get and Body. Get is basically the query parameters, it’s data type corresponds to the Java native types. On the other hand a body parameter, is represented as a JSON object. It can be as complicated as needed. In Meveo, it can be a custom entity template on several layers, meaning custom entity template a can have a field custom entity template b.
This feature is available via API at:
GET
/endpoint/schema/{endpointCode}/request
Response Schema
The response schema is an Open API v3 Draft7 standard document. It represents the data type saved in endpoint’s returnedVariableName.
The returnedVariableName, is a name of a field inside a script where it is mapped from the endpoint. It can be a Java native data type and can be a custom entity template as well.
For example, we have a script ScriptTest that is linked to our endpoint. This ScriptTest has a custom entity template property named Account.
public class ScriptTest extends Script {
private Account account;
public Account getAccount() {
return account;
}
public void setAccount(Account account) {
this.account = account;
}
}
To tell our endpoint that we want to return the value of the account after the execution, we need to set the value of endpoint.returnedVariableName=account.
This feature is available via API at:
GET
/endpoint/schema/{endpointCode}/response
Javascript Interface
An API that provides a working service or interface for managing CRUD operations of a custom entity template is available. This interface is automatically created and save in Meveo’s internal Git system, which is normally located at <PROVIDERS_DIR>\git\Meveo\endpoints.
GET
/endpoint/openApi/{endpointCode}
For reference, here is an example endpoint’s javascript interface
const buildRequestParameters = (parameters, schema) => {
if (schema) {
const errors = []
const requestParameters = Object.keys(
schema.properties,
).reduce((reqParameters, property) => {
const value = parameters[property]
const isRequired = schema.properties[property].required
if (isRequired && !value) {
errors.push(`${property} is required.`)
} else if (!!value) {
return {
...reqParameters,
[property]: value
}
}
return reqParameters
}, {})
if (errors.length > 0) {
throw errors
}
return requestParameters
}
return null
}
const EVENT = {
SUCCESS: "Updatepost-OpenApiGenerateCetTest-endpoint-SUCCESS",
ERROR: "Updatepost-OpenApiGenerateCetTest-endpoint-ERROR"
};
export const registerEventListeners = (
component,
successCallback,
errorCallback
) => {
if (successCallback) {
component.addEventListener(EVENT.SUCCESS, successCallback);
}
if (errorCallback) {
component.addEventListener(EVENT.ERROR, errorCallback);
}
};
export const getRequestSchema = async (parameters, config) => {
return {
"title": "post-OpenApiGenerateCetTest-endpointRequest",
"id": "post-OpenApiGenerateCetTest-endpointRequest",
"default": "Schema definition for post-OpenApiGenerateCetTest-endpoint",
"$schema": "http://json-schema.org/draft-07/schema",
"type": "object",
"properties": {
"qparam3": {
"title": "Consumption",
"description": "Consumption",
"id": "Consumption",
"storages": [
"SQL"
],
"type": "object",
"properties": {
"date": {
"title": "Consumption.date",
"description": "Date",
"id": "CE_Consumption_date",
"storages": [
"SQL"
],
"nullable": false,
"readOnly": false,
"versionable": false,
"indexType": "INDEX_NOT_ANALYZE",
"type": "string",
"format": "date-time"
},
"amount": {
"title": "Consumption.amount",
"description": "Amount",
"id": "CE_Consumption_amount",
"storages": [
"SQL"
],
"nullable": false,
"readOnly": false,
"versionable": false,
"indexType": "INDEX_NOT_ANALYZE",
"type": "integer"
},
"account": {
"title": "Consumption.account",
"description": "Account",
"id": "CE_Consumption_account",
"storages": [
"SQL"
],
"nullable": false,
"readOnly": false,
"versionable": false,
"indexType": "INDEX_NOT_ANALYZE",
"type": "string",
"minLength": 1,
"maxLength": 255
}
},
"required": [
"account",
"amount",
"date"
]
},
"qparam2": {
"title": "Account",
"description": "Account",
"id": "Account",
"storages": [
"SQL"
],
"type": "object",
"properties": {
"accountCode": {
"title": "Account.accountCode",
"description": "Account code",
"id": "CE_Account_accountCode",
"storages": [
"SQL"
],
"nullable": false,
"readOnly": false,
"versionable": false,
"indexType": "INDEX_NOT_ANALYZE",
"type": "string",
"minLength": 1,
"maxLength": 255
},
"accountType": {
"title": "Account.accountType",
"description": "Account type",
"id": "CE_Account_accountType",
"storages": [
"SQL"
],
"nullable": false,
"readOnly": false,
"versionable": false,
"indexType": "INDEX_NOT_ANALYZE",
"type": "string",
"minLength": 1,
"maxLength": 255
}
},
"required": [
"accountCode",
"accountType"
]
},
"qparam1": {
"title": "qparam1",
"type": "string",
"minLength": 1
}
}
}
};
export const getResponseSchema = async (parameters, config) => {
return {
"title": "post-OpenApiGenerateCetTest-endpointResponse",
"id": "post-OpenApiGenerateCetTest-endpointResponse",
"default": "Schema definition for post-OpenApiGenerateCetTest-endpoint",
"$schema": "http://json-schema.org/draft-07/schema",
"type": "object",
"properties": {
"consumption": {
"title": "Consumption",
"description": "Consumption",
"id": "Consumption",
"storages": [
"SQL"
],
"type": "object",
"properties": {
"date": {
"title": "Consumption.date",
"description": "Date",
"id": "CE_Consumption_date",
"storages": [
"SQL"
],
"nullable": false,
"readOnly": false,
"versionable": false,
"indexType": "INDEX_NOT_ANALYZE",
"type": "string",
"format": "date-time"
},
"amount": {
"title": "Consumption.amount",
"description": "Amount",
"id": "CE_Consumption_amount",
"storages": [
"SQL"
],
"nullable": false,
"readOnly": false,
"versionable": false,
"indexType": "INDEX_NOT_ANALYZE",
"type": "integer"
},
"account": {
"title": "Consumption.account",
"description": "Account",
"id": "CE_Consumption_account",
"storages": [
"SQL"
],
"nullable": false,
"readOnly": false,
"versionable": false,
"indexType": "INDEX_NOT_ANALYZE",
"type": "string",
"minLength": 1,
"maxLength": 255
}
},
"required": [
"account",
"amount",
"date"
]
}
}
}
}
export const executeApiCall = async (
component,
params,
successCallback, // optional
errorCallback // optional
) => {
registerEventListeners(component, successCallback, errorCallback);
const parameters = params || {};
const {
token,
config
} = parameters;
// the name of the config variable is the name of the module
const {
Updatepost-OpenApiGenerateCetTest-endpoint: {
OVERRIDE_URL,
USE_MOCK
}
} = config || {};
// the baseUrl can be overridden by indicating a OVERRIDE_URL in config,
// by default it will use the same URL as the client application
// or if this is auto-generated by meveo, it will have the server's host url
const baseUrl = OVERRIDE_URL || window.location.origin; // || server.host.url
// just an example how to use the useMock parameter to switch between mock and actual endpoints.
const apiUrl = USE_MOCK ?
`${baseUrl}/auth/realms/meveo/account?useMock=true` :
`${baseUrl}/auth/realms/meveo/account`;
//fetch request schema to filter out optional parameters that should not be passed into the request
try {
const requestSchema = await getRequestSchema(parameters);
const requestParameters = buildRequestParameters(parameters, requestSchema);
const parameterKeys = Object.keys(requestParameters || {});
const hasParameters = requestParameters && parameterKeys.length > 0;
const requestUrl = new URL(apiUrl);
if (hasParameters) {
parameterKeys.forEach(key => {
requestUrl.searchParams.append(key, requestParameters[key]);
});
}
const headers = new Headers();
headers.append("Content-Type", "application/json");
headers.append("Accept", "application/json");
headers.append("Authorization", `Bearer ${token}`);
const options = {
method: "GET",
headers
};
const response = await fetch(requestUrl, options);
if (!response.ok) {
throw [
`Encountered error calling API: ${apiUrl}`,
`Status code: ${response.status} [${response.statusText}]`
];
}
// if accept = "application/json" otherwise return response.text()
const result = await response.json();
component.dispatchEvent(
new CustomEvent(EVENT.SUCCESS, {
detail: {
result
},
bubbles: true
})
);
} catch (error) {
component.dispatchEvent(
new CustomEvent(EVENT.ERROR, {
detail: {
error
},
bubbles: true
})
);
}
};