ETL designer
etl design_sourcesetl design_transformsetl generateetl design_getetl generateetl space_saveetl space_getetl space_listetl space_viewetl space_deleteetl manual_checketl runetl mark_startetl join_armetl mark_doneetl join_readyetl join_readyetl error_marketl marksetl mark_reset
etl design_sources
Description
The families that can be dropped as source or destination: rows {kind, family, cm_type, label, category, source, destination, source_needs, destination_needs, formats}. A field item is {name, label, kind (text, query, json, bool, choice, columns), required, default, values, help}. Only the families whose functions exist on this server are listed
The families that can be dropped as source or destination: rows {kind, family, cm_type, label, category, source, destination, source_needs, destination_needs, formats}. A field item is {name, label, kind (text, query, json, bool, choice, columns), required, default, values, help}. Only the families whose functions exist on this server are listed
admin
etl design_sources
mentdb
[{"kind": "files", "family": "file", "cm_type": "file", "label": "Local files", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Relative to the folder of the file connection (empty: that folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Relative to the folder of the file connection (empty: that folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "ftp", "cm_type": "ftp", "label": "FTP", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "ftps", "cm_type": "ftps", "label": "FTPS", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "sftp", "cm_type": "sftp", "label": "SFTP", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "ssh", "cm_type": "ssh", "label": "SSH (scp)", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "cifs", "cm_type": "cifs", "label": "CIFS / SMB share", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "mentdb", "cm_type": "mentdb", "label": "Remote MentDB server (tunnel)", "category": "Files", "source": 1, "destination": 1, "source_needs": [{"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Remote folder (empty: the login folder)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "s3", "cm_type": "config", "label": "S3 (AWS, MinIO, OVH...)", "category": "Cloud storage", "source": 1, "destination": 1, "source_needs": [{"name": "bucket", "label": "Bucket", "kind": "text", "required": true}, {"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Prefix of the keys (empty: the root of the bucket)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "bucket", "label": "Bucket", "kind": "text", "required": true}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Prefix of the keys (empty: the root of the bucket)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "azure_blob", "cm_type": "config", "label": "Azure Blob", "category": "Cloud storage", "source": 1, "destination": 1, "source_needs": [{"name": "bucket", "label": "Bucket", "kind": "text", "required": true}, {"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Prefix of the keys (empty: the root of the bucket)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "bucket", "label": "Bucket", "kind": "text", "required": true}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Prefix of the keys (empty: the root of the bucket)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "files", "family": "gcs", "cm_type": "config", "label": "Google Cloud Storage", "category": "Cloud storage", "source": 1, "destination": 1, "source_needs": [{"name": "bucket", "label": "Bucket", "kind": "text", "required": true}, {"name": "filter", "label": "File filter", "kind": "text", "required": true, "default": "*.csv", "help": "File name pattern with * and ?"}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Prefix of the keys (empty: the root of the bucket)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the reader: csv read, json to_rows (path...), jsonl read, excelx read (sheet...); xml: json to_rows of xml to_json (path)"}, {"name": "delete_source", "label": "Delete the source files", "kind": "bool", "required": false, "default": false, "help": "Off by default: the files stay at the source"}], "destination_needs": [{"name": "bucket", "label": "Bucket", "kind": "text", "required": true}, {"name": "format", "label": "Format", "kind": "choice", "required": true, "default": "csv", "values": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"name": "dir", "label": "Folder", "kind": "text", "required": false, "help": "Prefix of the keys (empty: the root of the bucket)"}, {"name": "format_options", "label": "Format options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of the writer: csv write, json to_xml, excelx write; txt: csv from_json (header false, tab)"}, {"name": "file_name", "label": "File name", "kind": "text", "required": false, "help": "Markers [SOURCE] (source file without extension), [DATE] (yyyyMMddHHmmss), [FLOW_PID]; default [SOURCE]_[DATE].<format>"}], "formats": ["csv", "json", "jsonl", "xml", "txt", "excel", "excelx"]}, {"kind": "sql", "family": "db", "cm_type": "db", "label": "SQL database", "category": "Databases", "source": 1, "destination": 1, "source_needs": [{"name": "query", "label": "SELECT query", "kind": "query", "required": true}], "destination_needs": [{"name": "table", "label": "Table", "kind": "text", "required": true}, {"name": "mode", "label": "Mode", "kind": "choice", "required": false, "default": "insert", "values": ["insert", "upsert"]}, {"name": "keys", "label": "Key columns (upsert)", "kind": "columns", "required": false}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Parameters of sql bulk_insert / bulk_upsert (batch_size...)"}]}, {"kind": "rows", "family": "mongodb", "cm_type": "config", "label": "MongoDB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'mongodb find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'mongodb find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'mongodb insert'"}]}, {"kind": "rows", "family": "couchdb", "cm_type": "config", "label": "CouchDB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'couchdb find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'couchdb find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'couchdb insert'"}]}, {"kind": "rows", "family": "couchbase", "cm_type": "config", "label": "Couchbase", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'couchbase find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'couchbase find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'couchbase insert'"}]}, {"kind": "rows", "family": "surreal", "cm_type": "config", "label": "SurrealDB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'surreal find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'surreal find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'surreal insert'"}]}, {"kind": "rows", "family": "ravendb", "cm_type": "config", "label": "RavenDB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'ravendb find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'ravendb find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'ravendb insert'"}]}, {"kind": "rows", "family": "firestore", "cm_type": "config", "label": "Firestore", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'firestore find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'firestore find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'firestore insert'"}]}, {"kind": "rows", "family": "cosmos", "cm_type": "config", "label": "Cosmos DB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'cosmos find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'cosmos find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'cosmos insert'"}]}, {"kind": "rows", "family": "dynamodb", "cm_type": "config", "label": "DynamoDB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'dynamodb find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'dynamodb find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}]}, {"kind": "rows", "family": "cassandra", "cm_type": "config", "label": "Cassandra / ScyllaDB", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'cassandra find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'cassandra find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'cassandra insert'"}]}, {"kind": "rows", "family": "hbase", "cm_type": "config", "label": "HBase", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'hbase find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'hbase find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'hbase insert'"}]}, {"kind": "rows", "family": "elastic", "cm_type": "config", "label": "Elasticsearch / OpenSearch", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'elastic find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'elastic find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'elastic insert'"}]}, {"kind": "rows", "family": "solr", "cm_type": "config", "label": "Solr", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'solr find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'solr find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'solr insert'"}]}, {"kind": "rows", "family": "meili", "cm_type": "config", "label": "Meilisearch", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'meili find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'meili find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'meili insert'"}]}, {"kind": "rows", "family": "typesense", "cm_type": "config", "label": "Typesense", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'typesense find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'typesense find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'typesense insert'"}]}, {"kind": "rows", "family": "qdrant", "cm_type": "config", "label": "Qdrant", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'qdrant find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'qdrant find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'qdrant insert'"}]}, {"kind": "rows", "family": "chroma", "cm_type": "config", "label": "Chroma", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'chroma find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'chroma find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'chroma insert'"}]}, {"kind": "rows", "family": "weaviate", "cm_type": "config", "label": "Weaviate", "category": "NoSQL", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'weaviate find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'weaviate find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'weaviate insert'"}]}, {"kind": "rows", "family": "odata", "cm_type": "config", "label": "OData (SAP, Dynamics, Dataverse...)", "category": "Applications", "source": 1, "destination": 1, "source_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "filter", "label": "Filter", "kind": "json", "required": false, "default": "{}", "help": "The filter of 'odata find'"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"limit\": 10000}", "help": "Options of 'odata find' (limit: 10000 rows when not given)"}], "destination_needs": [{"name": "collection", "label": "Collection", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'odata insert'"}]}, {"kind": "messages", "family": "kafka", "cm_type": "config", "label": "Kafka", "category": "Messaging", "source": 1, "destination": 1, "source_needs": [{"name": "topic", "label": "Topic", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"max_messages\": 1000}", "help": "Options of 'kafka consume' (max_messages: 1000 when not given; wait_ms or timeout_ms)"}], "destination_needs": [{"name": "topic", "label": "Topic", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'kafka produce_batch'"}]}, {"kind": "messages", "family": "amqp", "cm_type": "config", "label": "AMQP (RabbitMQ...)", "category": "Messaging", "source": 1, "destination": 1, "source_needs": [{"name": "topic", "label": "Queue", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"max_messages\": 1000}", "help": "Options of 'amqp consume' (max_messages: 1000 when not given; wait_ms or timeout_ms)"}], "destination_needs": [{"name": "topic", "label": "Queue / routing key", "kind": "text", "required": true}, {"name": "exchange", "label": "Exchange", "kind": "text", "required": false, "default": "", "help": "Empty: the default exchange (the routing key is the queue)"}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'amqp publish_batch'"}]}, {"kind": "messages", "family": "mqtt", "cm_type": "config", "label": "MQTT", "category": "Messaging", "source": 1, "destination": 1, "source_needs": [{"name": "topic", "label": "Topic", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"max_messages\": 1000}", "help": "Options of 'mqtt receive' (max_messages: 1000 when not given; wait_ms or timeout_ms)"}], "destination_needs": [{"name": "topic", "label": "Topic", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'mqtt publish_batch'"}]}, {"kind": "messages", "family": "sqs", "cm_type": "config", "label": "Amazon SQS", "category": "Messaging", "source": 1, "destination": 1, "source_needs": [{"name": "topic", "label": "Queue", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"max_messages\": 1000}", "help": "Options of 'sqs consume' (max_messages: 1000 when not given; wait_ms or timeout_ms)"}], "destination_needs": [{"name": "topic", "label": "Queue / routing key", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'sqs publish_batch'"}]}, {"kind": "messages", "family": "nats", "cm_type": "config", "label": "NATS", "category": "Messaging", "source": 1, "destination": 1, "source_needs": [{"name": "topic", "label": "Subject", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"max_messages\": 1000}", "help": "Options of 'nats consume' (max_messages: 1000 when not given; wait_ms or timeout_ms)"}], "destination_needs": [{"name": "topic", "label": "Subject", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'nats publish_batch'"}]}, {"kind": "messages", "family": "pubsub", "cm_type": "config", "label": "Google Pub/Sub", "category": "Messaging", "source": 1, "destination": 1, "source_needs": [{"name": "topic", "label": "Subscription", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{\"max_messages\": 1000}", "help": "Options of 'pubsub consume' (max_messages: 1000 when not given; wait_ms or timeout_ms)"}], "destination_needs": [{"name": "topic", "label": "Topic", "kind": "text", "required": true}, {"name": "options", "label": "Options", "kind": "json", "required": false, "default": "{}", "help": "Options of 'pubsub publish_batch'"}]}]
etl design_transforms
Description
The transformations of the middle block: rows {type, label, function, params: [{name, label, kind}]}
The transformations of the middle block: rows {type, label, function, params: [{name, label, kind}]}
admin
etl design_transforms
mentdb
[{"type": "select", "label": "Keep some columns, in this order", "function": "table select", "params": [{"name": "columns", "label": "Columns", "kind": "columns"}]}, {"type": "rename", "label": "Rename columns", "function": "json to_rows (rename)", "params": [{"name": "map", "label": "Old name -> new name", "kind": "map"}]}, {"type": "where", "label": "Keep the rows that match", "function": "table where", "params": [{"name": "condition", "label": "Conditions: {\"field\": value} or {\"field\": {\"gt\": 10}} (eq, ne, gt, ge, lt, le, in, not_in, contains, starts_with, null)", "kind": "condition"}]}, {"type": "distinct", "label": "Distinct combinations of columns", "function": "table distinct", "params": [{"name": "columns", "label": "Columns", "kind": "columns"}]}, {"type": "sort", "label": "Sort the rows", "function": "table sort", "params": [{"name": "columns", "label": "Columns: [\"id\"] or [[\"amount\", \"desc\"]]", "kind": "order"}]}, {"type": "top", "label": "The first rows (per group)", "function": "table top", "params": [{"name": "n", "label": "Number of rows", "kind": "number"}, {"name": "columns", "label": "Order: [[\"amount\", \"desc\"]]", "kind": "order"}, {"name": "groups", "label": "Group columns (optional)", "kind": "columns"}]}, {"type": "mql", "label": "Free MQL on [rows] (must set [rows])", "function": "mql", "params": [{"name": "code", "label": "MQL code", "kind": "code"}]}, {"type": "script", "label": "Call a script with [rows] (returns the new rows)", "function": "execute", "params": [{"name": "name", "label": "Script name", "kind": "script"}]}]
etl generate <design>
Description
The MQL of the whole flow (group, 5 script merge, grants), not run. With a session the family of each block is read from its connection; without one (or to check it) the block gives "family". Clear errors for a bad name, a kind that does not match the connection, an unknown field, a value that cannot be written in MQL
The MQL of the whole flow (group, 5 script merge, grants), not run. With a session the family of each block is read from its connection; without one (or to check it) the block gives "family". Clear errors for a bad name, a kind that does not match the connection, an unknown field, a value that cannot be written in MQL
Parameters
design: The design JSON: {name, prefix, group (optional), source, transforms, destination} - String - required
design: The design JSON: {name, prefix, group (optional), source, transforms, destination} - String - required
admin
etl generate "{\"name\": \"orders_csv_to_json\", \"prefix\": \"demo.orders.csv_to_json\", \"source\": {\"cm\": \"demo_in\", \"kind\": \"files\", \"family\": \"file\", \"filter\": \"*.csv\", \"format\": \"csv\", \"delete_source\": true}, \"transforms\": [{\"type\": \"select\", \"columns\": [\"id\", \"client\", \"total\"]}, {\"type\": \"where\", \"condition\": {\"total\": {\"gt\": 100}}}, {\"type\": \"rename\", \"map\": {\"total\": \"amount\"}}, {\"type\": \"sort\", \"columns\": [[\"amount\", \"desc\"]]}], \"destination\": {\"cm\": \"demo_out\", \"kind\": \"files\", \"family\": \"file\", \"format\": \"json\", \"file_name\": \"orders_[SOURCE]_[FLOW_PID].json\"}}"
mentdb
# ETL flow 'orders_csv_to_json' generated by the ETL designer (etl generate): file (files) -> 4 transformation(s) -> file (files); if (not (group exist "demo_orders_csv_to_json")) { group add "demo_orders_csv_to_json"; }; script merge "demo.orders.csv_to_json.step_1_source_connect.exe" true 5 (param ) "Connect to the source (orders_csv_to_json)" { # ETL_DESIGN eyJuYW1lIjogIm9yZGVyc19jc3ZfdG9fanNvbiIsICJwcmVmaXgiOiAiZGVtby5vcmRlcnMuY3N2X3RvX2pzb24iLCAic291cmNlIjogeyJjbSI6ICJkZW1vX2luIiwgImtpbmQiOiAiZmlsZXMiLCAiZmFtaWx5IjogImZpbGUiLCAiZmlsdGVyIjogIiouY3N2IiwgImZvcm1hdCI6ICJjc3YiLCAiZGVsZXRlX3NvdXJjZSI6IHRydWV9LCAidHJhbnNmb3JtcyI6IFt7InR5cGUiOiAic2VsZWN0IiwgImNvbHVtbnMiOiBbImlkIiwgImNsaWVudCIsICJ0b3RhbCJdfSwgeyJ0eXBlIjogIndoZXJlIiwgImNvbmRpdGlvbiI6IHsidG90YWwiOiB7Imd0IjogMTAwfX19LCB7InR5cGUiOiAicmVuYW1lIiwgIm1hcCI6IHsidG90YWwiOiAiYW1vdW50In19LCB7InR5cGUiOiAic29ydCIsICJjb2x1bW5zIjogW1siYW1vdW50IiwgImRlc2MiXV19XSwgImRlc3RpbmF0aW9uIjogeyJjbSI6ICJkZW1vX291dCIsICJraW5kIjogImZpbGVzIiwgImZhbWlseSI6ICJmaWxlIiwgImZvcm1hdCI6ICJqc29uIiwgImZpbGVfbmFtZSI6ICJvcmRlcnNfW1NPVVJDRV1fW0ZMT1dfUElEXS5qc29uIn19; # ETL_HASH 32a0d0eee72d3556566dbb601be38a3a32b70a03d6bf1768165befc5b077cc64 ec25d3e9117b87d9 667358107360f60f 8595e42b6fc97fc0 9b180e46364afc82 903b217d96d2685a; # WORK FOLDER: every file of a run is in home/[FLOW_PID]/, [FLOW_PID] being the PID of this step 1; # home/[FLOW_PID]/<file>: a file extracted at step 2, extract.json for a sql, rows or messages source; # home/[FLOW_PID]/<file>.rows.json: the rows after step 3; # home/[FLOW_PID]/<file>.out/<name>: the file built at step 5 for a files destination; # Step 5 deletes the files of its file once loaded, then home/[FLOW_PID]/ when it is empty: all the files are loaded; # step 2 deletes it when there is nothing to extract. After an error the files stay for the replay of the step; # a replay of step 1 is a new run: a new PID, a new folder; # CONFIGURATION ; -> "[CONF_NAME_OF_THE_FLOW]" "orders_csv_to_json"; -> "[CONF_SOURCE_CM]" "demo_in"; # INITIALIZATION ; -> "[FLOW_PID]" [PID]; -> "[etl_connected]" 0; # HANDLE ; try { # Flow initialization; stack flow_init [FLOW_PID] [CONF_NAME_OF_THE_FLOW] "{}"; stack flow_step [FLOW_PID] 1 "source_connect..."; # Start mark: the last run of this flow (etl space_view); try {etl mark_start "demo.orders.csv_to_json" [FLOW_PID];} {log write (concat "ETL start mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; # Local connection: nothing to open; log write "Source: local files" OK null null; #Step 1 is valid; stack flow_step [FLOW_PID] 1 "source_connect_ok"; -> "[etl_connected]" 1; include "demo.orders.csv_to_json.step_2_extract.exe"; } { #Step 1 is not valid (an error of step 2 is already marked by step 2); if (== [etl_connected] 0) {stack flow_step [FLOW_PID] 1 "source_connect_KO";}; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.csv_to_json.step_1_source_connect.exe" "demo_orders_csv_to_json")) { group grant script "demo.orders.csv_to_json.step_1_source_connect.exe" "demo_orders_csv_to_json"; }; script merge "demo.orders.csv_to_json.step_2_extract.exe" true 1 (param ) "Extract from the source (orders_csv_to_json)" { # CONFIGURATION ; json load "etl_cm_source" {cm get [CONF_SOURCE_CM];}; -> "[CONF_DIR_SOURCE]" (json select "etl_cm_source" "/localDir"); json unload "etl_cm_source"; -> "[CONF_SOURCE_FILTER]" ".*[.]csv"; # HANDLE ; try { # Mark the flow as step 2; stack flow_step [FLOW_PID] 2 "extract..."; # Create the local directory PID; if (not (file exist (concat "home/" [FLOW_PID]))) {file mkdir (concat "home/" [FLOW_PID]);}; # Get files into PID directory; log write (concat "Get files from '" [CONF_DIR_SOURCE] "' (" [CONF_SOURCE_FILTER] ").") OK null null; json load "etl_names" (file dir_list_regex [CONF_DIR_SOURCE] [CONF_SOURCE_FILTER] true false); json load "valid_files" "[]"; json parse_array "etl_names" "/" "[etl_name]" { if (and (string matches [etl_name] "[^/\\\p{Cntrl}]+") (not (string matches [etl_name] "[.][.]?"))) {json iarray "valid_files" / [etl_name] STR;} {log write (concat "Skip the file name '" (string regexp_replace [etl_name] "\p{Cntrl}" "?") "' (not a plain file name).") OK null null;}; }; json unload "etl_names"; json parse_array "valid_files" "/" "[filename]" { file copy_file (concat [CONF_DIR_SOURCE] "/" [filename]) (concat "home/" [FLOW_PID] "/" [filename]); log write (concat "Get file 'home/" [FLOW_PID] "/" [filename] "'.") OK null null; }; # Delete the source files; json parse_array "valid_files" "/" "[filename]" { file delete (concat [CONF_DIR_SOURCE] "/" [filename]); log write (concat "Remove source file '" [CONF_DIR_SOURCE] "/" [filename] "'.") OK null null; }; } { #Step 2 is not valid; stack flow_step [FLOW_PID] 2 "extract_KO"; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; log write (concat (json count "valid_files" /) " file(s) to transform.") OK null null; if (> (json count "valid_files" /) 0) { #Step 2 is valid; stack flow_step [FLOW_PID] 2 (concat "extract_ok_" (json count "valid_files" /) "_files"); json parse_array "valid_files" "/" "[filename]" { stack (date now) "demo.orders.csv_to_json.step_3_transform.exe" "[FLOW_PID]" [FLOW_PID] "[filename]" [filename]; }; } { #Step 2 is valid; stack flow_step [FLOW_PID] 2 "extract_ok_zero_file"; # Nothing to transform; file delete (concat "home/" [FLOW_PID]); # END OF THE RUN (nothing to extract counts as a run ended OK): done mark for the joins; try {etl mark_done "demo.orders.csv_to_json";} {log write (concat "ETL done mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; }; } "Return nothing"; if (not (group is granted script "demo.orders.csv_to_json.step_2_extract.exe" "demo_orders_csv_to_json")) { group grant script "demo.orders.csv_to_json.step_2_extract.exe" "demo_orders_csv_to_json"; }; script merge "demo.orders.csv_to_json.step_3_transform.exe" true 1 (param (var "[FLOW_PID]" {true} "The flow PID" is_null:true is_empty:true "1") (var "[filename]" {true} "The filename" is_null:true is_empty:true "file.xxx") ) "Transform the rows (orders_csv_to_json)" { # CONFIGURATION ; -> "[CONF_SOURCE_FORMAT_OPTIONS]" "{}"; -> "[CONF_TRANSFORMED_FILENAME]" (concat [filename] ".rows.json"); # HANDLE ; try { # Mark the flow as step 3; stack flow_step [FLOW_PID] 3 "transform..."; -> "[etl_file]" (concat "home/" [FLOW_PID] "/" [filename]); log write (concat "Read the rows of '" [etl_file] "' (csv).") OK null null; -> "[rows]" (csv read [etl_file] [CONF_SOURCE_FORMAT_OPTIONS]); # Transformation 1: select; -> "[rows]" (table select [rows] "[\"id\", \"client\", \"total\"]"); # Transformation 2: where; -> "[rows]" (table where [rows] "{\"total\": {\"gt\": 100}}"); # Transformation 3: rename; -> "[rows]" (json to_rows [rows] "{\"rename\": {\"total\": \"amount\"}}"); # Transformation 4: sort; -> "[rows]" (table sort [rows] "[[\"amount\", \"desc\"]]"); json load "etl_rows" [rows]; log write (concat (json count "etl_rows" /) " row(s) after the transformations.") OK null null; json unload "etl_rows"; file create (concat "home/" [FLOW_PID] "/" [CONF_TRANSFORMED_FILENAME]) [rows] "UTF-8"; #Step 3 is valid; stack flow_step [FLOW_PID] 3 "transform_ok"; stack (date now) "demo.orders.csv_to_json.step_4_destination_connect.exe" "[FLOW_PID]" [FLOW_PID] "[filename]" [CONF_TRANSFORMED_FILENAME] "[source_filename]" [filename]; } { #Step 3 is not valid; stack flow_step [FLOW_PID] 3 "transform_KO"; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.csv_to_json.step_3_transform.exe" "demo_orders_csv_to_json")) { group grant script "demo.orders.csv_to_json.step_3_transform.exe" "demo_orders_csv_to_json"; }; script merge "demo.orders.csv_to_json.step_4_destination_connect.exe" true 5 (param (var "[FLOW_PID]" {true} "The flow PID" is_null:true is_empty:true "1") (var "[filename]" {true} "The filename" is_null:true is_empty:true "file.xxx") (var "[source_filename]" {true} "The file extracted at step 2" is_null:true is_empty:true "file.xxx") ) "Connect to the destination (orders_csv_to_json)" { # CONFIGURATION ; -> "[CONF_DESTINATION_CM]" "demo_out"; -> "[etl_step5]" 0; # HANDLE ; try { # Flow step 4; stack flow_step [FLOW_PID] 4 "destination_connect..."; # Local connection: nothing to open; log write "Destination: local files" OK null null; #Step 4 is valid; stack flow_step [FLOW_PID] 4 "destination_connect_ok"; include "demo.orders.csv_to_json.step_5_load.exe"; } { #Step 4 is not valid (an error of step 5 is already marked by step 5); if (not (== [etl_step5] 1)) {stack flow_step [FLOW_PID] 4 "destination_connect_KO";}; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.csv_to_json.step_4_destination_connect.exe" "demo_orders_csv_to_json")) { group grant script "demo.orders.csv_to_json.step_4_destination_connect.exe" "demo_orders_csv_to_json"; }; script merge "demo.orders.csv_to_json.step_5_load.exe" true 1 (param ) "Load to the destination (orders_csv_to_json)" { # CONFIGURATION ; -> "[CONF_DESTINATION_CM]" "demo_out"; json load "etl_cm_destination" {cm get [CONF_DESTINATION_CM];}; -> "[CONF_DIR_DESTINATION]" (json select "etl_cm_destination" "/localDir"); json unload "etl_cm_destination"; -> "[CONF_DESTINATION_FORMAT_OPTIONS]" "{}"; -> "[CONF_DESTINATION_FILENAME]" (concat "orders_" (string regexp_replace [source_filename] "\.[^.]*$" "") "_" [FLOW_PID] ".json"); -> "[etl_step5]" 1; -> "[etl_run_done]" 0; # HANDLE ; try { # Mark the flow as step 5; stack flow_step [FLOW_PID] 5 "load..."; -> "[rows]" (file load (concat "home/" [FLOW_PID] "/" [filename]) "UTF-8"); if (not (and (string matches [CONF_DESTINATION_FILENAME] "[^/\\\p{Cntrl}]+") (not (string matches [CONF_DESTINATION_FILENAME] "[.][.]?")))) {exception (1) (concat "The destination file name is not a plain file name: " (string regexp_replace [CONF_DESTINATION_FILENAME] "\p{Cntrl}" "?"));}; # The file is built in a folder of its own: home/[FLOW_PID]/<source file>.out/; -> "[etl_out_dir]" (concat "home/" [FLOW_PID] "/" [source_filename] ".out"); if (not (file exist [etl_out_dir])) {file mkdir [etl_out_dir];}; -> "[etl_out]" (concat [etl_out_dir] "/" [CONF_DESTINATION_FILENAME]); log write (concat "Build the file '" [etl_out] "' (json).") OK null null; file create [etl_out] [rows] "UTF-8"; log write (concat "Send '" [etl_out] "' to '" [CONF_DIR_DESTINATION] "/" [CONF_DESTINATION_FILENAME] "'...") OK null null; if (not (file exist [CONF_DIR_DESTINATION])) {file mkdir [CONF_DIR_DESTINATION];}; file copy_file [etl_out] (concat [CONF_DIR_DESTINATION] "/" [CONF_DESTINATION_FILENAME]); if (not (file exist (concat [CONF_DIR_DESTINATION] "/" [CONF_DESTINATION_FILENAME]))) { exception (1) (concat "Cannot copy the file '" [CONF_DESTINATION_FILENAME] "'."); }; file delete [etl_out]; try {file delete [etl_out_dir];} {} "[err]"; log write "Sent." OK null null; # Remove the work files of this file, then the folder of the run when it is empty (all its files are loaded); try {file delete (concat "home/" [FLOW_PID] "/" [filename]);} {} "[err]"; try {file delete (concat "home/" [FLOW_PID] "/" [source_filename]);} {} "[err]"; try { json load "etl_left" (file dir_list_regex (concat "home/" [FLOW_PID]) ".*" true true); if (== (json count "etl_left" /) 0) {file delete (concat "home/" [FLOW_PID]); -> "[etl_run_done]" 1;}; json unload "etl_left"; } {} "[err]"; #Step 5 is valid; stack flow_step [FLOW_PID] 5 "load_ok"; # END OF THE RUN (its last file is loaded: the folder of the run is deleted): done mark for the joins; if (== [etl_run_done] 1) { try {etl mark_done "demo.orders.csv_to_json";} {log write (concat "ETL done mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; }; } { try {file delete [etl_out];} {} "[err]"; try {file delete [etl_out_dir];} {} "[err]"; #Step 5 is not valid; stack flow_step [FLOW_PID] 5 "load_KO"; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.csv_to_json.step_5_load.exe" "demo_orders_csv_to_json")) { group grant script "demo.orders.csv_to_json.step_5_load.exe" "demo_orders_csv_to_json"; }; # No join: step 1 runs as soon as it is in the stack; script set delay "demo.orders.csv_to_json.step_1_source_connect.exe" 0 day {1};
a SELECT on the internal database -> free MQL -> another table
admin
etl generate "{\"name\": \"orders_sql_to_sql\", \"prefix\": \"demo.orders.sql_to_sql\", \"source\": {\"cm\": \"MENTDB\", \"kind\": \"sql\", \"family\": \"db\", \"query\": \"select id, client, total from etl_orders where status = 'NEW'\"}, \"transforms\": [{\"type\": \"rename\", \"map\": {\"total\": \"amount\"}}, {\"type\": \"mql\", \"code\": \"-> \\"[rows]\\" (table where [rows] \\"{\\\\"amount\\\\": {\\\\"ge\\\\": 100}}\\");\"}], \"destination\": {\"cm\": \"MENTDB\", \"kind\": \"sql\", \"family\": \"db\", \"table\": \"etl_orders_new\", \"mode\": \"insert\"}}"
mentdb
# ETL flow 'orders_sql_to_sql' generated by the ETL designer (etl generate): db (sql) -> 2 transformation(s) -> db (sql); if (not (group exist "demo_orders_sql_to_sql")) { group add "demo_orders_sql_to_sql"; }; script merge "demo.orders.sql_to_sql.step_1_source_connect.exe" true 5 (param ) "Connect to the source (orders_sql_to_sql)" { # ETL_DESIGN eyJuYW1lIjogIm9yZGVyc19zcWxfdG9fc3FsIiwgInByZWZpeCI6ICJkZW1vLm9yZGVycy5zcWxfdG9fc3FsIiwgInNvdXJjZSI6IHsiY20iOiAiTUVOVERCIiwgImtpbmQiOiAic3FsIiwgImZhbWlseSI6ICJkYiIsICJxdWVyeSI6ICJzZWxlY3QgaWQsIGNsaWVudCwgdG90YWwgZnJvbSBldGxfb3JkZXJzIHdoZXJlIHN0YXR1cyA9ICdORVcnIn0sICJ0cmFuc2Zvcm1zIjogW3sidHlwZSI6ICJyZW5hbWUiLCAibWFwIjogeyJ0b3RhbCI6ICJhbW91bnQifX0sIHsidHlwZSI6ICJtcWwiLCAiY29kZSI6ICItPiBcIltyb3dzXVwiICh0YWJsZSB3aGVyZSBbcm93c10gXCJ7XFxcImFtb3VudFxcXCI6IHtcXFwiZ2VcXFwiOiAxMDB9fVwiKTsifV0sICJkZXN0aW5hdGlvbiI6IHsiY20iOiAiTUVOVERCIiwgImtpbmQiOiAic3FsIiwgImZhbWlseSI6ICJkYiIsICJ0YWJsZSI6ICJldGxfb3JkZXJzX25ldyIsICJtb2RlIjogImluc2VydCJ9fQ==; # ETL_HASH f5afb9dcb788788ab6e7bd4fe4c6dd04e90398163c22bb1ab72312d60291cc67 c344ced2e4a9f42c 03232d42af1637ec b3a39ae6b1febd23 42b5d33342d896e1 9484834210f778d3; # WORK FOLDER: every file of a run is in home/[FLOW_PID]/, [FLOW_PID] being the PID of this step 1; # home/[FLOW_PID]/<file>: a file extracted at step 2, extract.json for a sql, rows or messages source; # home/[FLOW_PID]/<file>.rows.json: the rows after step 3; # home/[FLOW_PID]/<file>.out/<name>: the file built at step 5 for a files destination; # Step 5 deletes the files of its file once loaded, then home/[FLOW_PID]/ when it is empty: all the files are loaded; # step 2 deletes it when there is nothing to extract. After an error the files stay for the replay of the step; # a replay of step 1 is a new run: a new PID, a new folder; # CONFIGURATION ; -> "[CONF_NAME_OF_THE_FLOW]" "orders_sql_to_sql"; -> "[CONF_SOURCE_CM]" "MENTDB"; # INITIALIZATION ; -> "[FLOW_PID]" [PID]; -> "[etl_connected]" 0; # HANDLE ; try { # Flow initialization; stack flow_init [FLOW_PID] [CONF_NAME_OF_THE_FLOW] "{}"; stack flow_step [FLOW_PID] 1 "source_connect..."; # Start mark: the last run of this flow (etl space_view); try {etl mark_start "demo.orders.sql_to_sql" [FLOW_PID];} {log write (concat "ETL start mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; sql connect "session_remote" {cm get [CONF_SOURCE_CM];}; #Step 1 is valid; stack flow_step [FLOW_PID] 1 "source_connect_ok"; -> "[etl_connected]" 1; include "demo.orders.sql_to_sql.step_2_extract.exe"; } { try {sql disconnect "session_remote";} {} "[err]"; #Step 1 is not valid (an error of step 2 is already marked by step 2); if (== [etl_connected] 0) {stack flow_step [FLOW_PID] 1 "source_connect_KO";}; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.sql_to_sql.step_1_source_connect.exe" "demo_orders_sql_to_sql")) { group grant script "demo.orders.sql_to_sql.step_1_source_connect.exe" "demo_orders_sql_to_sql"; }; script merge "demo.orders.sql_to_sql.step_2_extract.exe" true 1 (param ) "Extract from the source (orders_sql_to_sql)" { # CONFIGURATION ; -> "[CONF_SOURCE_QUERY]" "select id, client, total from etl_orders where status = 'NEW'"; # HANDLE ; try { # Mark the flow as step 2; stack flow_step [FLOW_PID] 2 "extract..."; # Create the local directory PID; if (not (file exist (concat "home/" [FLOW_PID]))) {file mkdir (concat "home/" [FLOW_PID]);}; log write "Run the query of the source." OK null null; -> "[rows]" (json to_rows (sql to json "session_remote" "rows" [CONF_SOURCE_QUERY]) "{\"path\": \"$.data\"}"); json load "etl_rows" [rows]; log write (concat (json count "etl_rows" /) " row(s) extracted.") OK null null; if (> (json count "etl_rows" /) 0) { file create (concat "home/" [FLOW_PID] "/extract.json") [rows] "UTF-8"; json load "valid_files" "[\"extract.json\"]"; } { json load "valid_files" "[]"; }; json unload "etl_rows"; sql disconnect "session_remote"; } { try {sql disconnect "session_remote";} {} "[err]"; #Step 2 is not valid; stack flow_step [FLOW_PID] 2 "extract_KO"; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; log write (concat (json count "valid_files" /) " file(s) to transform.") OK null null; if (> (json count "valid_files" /) 0) { #Step 2 is valid; stack flow_step [FLOW_PID] 2 (concat "extract_ok_" (json count "valid_files" /) "_files"); json parse_array "valid_files" "/" "[filename]" { stack (date now) "demo.orders.sql_to_sql.step_3_transform.exe" "[FLOW_PID]" [FLOW_PID] "[filename]" [filename]; }; } { #Step 2 is valid; stack flow_step [FLOW_PID] 2 "extract_ok_zero_file"; # Nothing to transform; file delete (concat "home/" [FLOW_PID]); # END OF THE RUN (nothing to extract counts as a run ended OK): done mark for the joins; try {etl mark_done "demo.orders.sql_to_sql";} {log write (concat "ETL done mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; }; } "Return nothing"; if (not (group is granted script "demo.orders.sql_to_sql.step_2_extract.exe" "demo_orders_sql_to_sql")) { group grant script "demo.orders.sql_to_sql.step_2_extract.exe" "demo_orders_sql_to_sql"; }; script merge "demo.orders.sql_to_sql.step_3_transform.exe" true 1 (param (var "[FLOW_PID]" {true} "The flow PID" is_null:true is_empty:true "1") (var "[filename]" {true} "The filename" is_null:true is_empty:true "file.xxx") ) "Transform the rows (orders_sql_to_sql)" { # CONFIGURATION ; -> "[CONF_TRANSFORMED_FILENAME]" (concat [filename] ".rows.json"); # HANDLE ; try { # Mark the flow as step 3; stack flow_step [FLOW_PID] 3 "transform..."; -> "[etl_file]" (concat "home/" [FLOW_PID] "/" [filename]); log write (concat "Read the rows of '" [etl_file] "' (rows).") OK null null; -> "[rows]" (file load [etl_file] "UTF-8"); # Transformation 1: rename; -> "[rows]" (json to_rows [rows] "{\"rename\": {\"total\": \"amount\"}}"); # Transformation 2: mql; # Free MQL: it reads and sets [rows]; -> "[rows]" (table where [rows] "{\"amount\": {\"ge\": 100}}"); json load "etl_check" [rows]; json unload "etl_check"; json load "etl_rows" [rows]; log write (concat (json count "etl_rows" /) " row(s) after the transformations.") OK null null; json unload "etl_rows"; file create (concat "home/" [FLOW_PID] "/" [CONF_TRANSFORMED_FILENAME]) [rows] "UTF-8"; #Step 3 is valid; stack flow_step [FLOW_PID] 3 "transform_ok"; stack (date now) "demo.orders.sql_to_sql.step_4_destination_connect.exe" "[FLOW_PID]" [FLOW_PID] "[filename]" [CONF_TRANSFORMED_FILENAME] "[source_filename]" [filename]; } { #Step 3 is not valid; stack flow_step [FLOW_PID] 3 "transform_KO"; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.sql_to_sql.step_3_transform.exe" "demo_orders_sql_to_sql")) { group grant script "demo.orders.sql_to_sql.step_3_transform.exe" "demo_orders_sql_to_sql"; }; script merge "demo.orders.sql_to_sql.step_4_destination_connect.exe" true 5 (param (var "[FLOW_PID]" {true} "The flow PID" is_null:true is_empty:true "1") (var "[filename]" {true} "The filename" is_null:true is_empty:true "file.xxx") (var "[source_filename]" {true} "The file extracted at step 2" is_null:true is_empty:true "file.xxx") ) "Connect to the destination (orders_sql_to_sql)" { # CONFIGURATION ; -> "[CONF_DESTINATION_CM]" "MENTDB"; -> "[etl_step5]" 0; # HANDLE ; try { # Flow step 4; stack flow_step [FLOW_PID] 4 "destination_connect..."; sql connect "destination_sess" {cm get [CONF_DESTINATION_CM];}; sql auto_commit "destination_sess" false; #Step 4 is valid; stack flow_step [FLOW_PID] 4 "destination_connect_ok"; include "demo.orders.sql_to_sql.step_5_load.exe"; } { try {sql rollback "destination_sess";} {} "[err]"; try {sql disconnect "destination_sess";} {} "[err]"; #Step 4 is not valid (an error of step 5 is already marked by step 5); if (not (== [etl_step5] 1)) {stack flow_step [FLOW_PID] 4 "destination_connect_KO";}; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.sql_to_sql.step_4_destination_connect.exe" "demo_orders_sql_to_sql")) { group grant script "demo.orders.sql_to_sql.step_4_destination_connect.exe" "demo_orders_sql_to_sql"; }; script merge "demo.orders.sql_to_sql.step_5_load.exe" true 1 (param ) "Load to the destination (orders_sql_to_sql)" { # CONFIGURATION ; -> "[CONF_DESTINATION_CM]" "MENTDB"; -> "[CONF_DESTINATION_TABLE]" "etl_orders_new"; -> "[CONF_DESTINATION_OPTIONS]" "{}"; -> "[etl_step5]" 1; -> "[etl_run_done]" 0; # HANDLE ; try { # Mark the flow as step 5; stack flow_step [FLOW_PID] 5 "load..."; -> "[rows]" (file load (concat "home/" [FLOW_PID] "/" [filename]) "UTF-8"); log write (concat "Load the rows into the table '" [CONF_DESTINATION_TABLE] "'...") OK null null; json load "etl_rows" [rows]; if (> (json count "etl_rows" /) 0) { -> "[etl_result]" (sql bulk_insert "destination_sess" [CONF_DESTINATION_TABLE] [rows] [CONF_DESTINATION_OPTIONS]); log write (concat "Loaded: " [etl_result]) OK null null; } { log write "No row to load." OK null null; }; json unload "etl_rows"; sql commit "destination_sess"; sql disconnect "destination_sess"; # Remove the work files of this file, then the folder of the run when it is empty (all its files are loaded); try {file delete (concat "home/" [FLOW_PID] "/" [filename]);} {} "[err]"; try {file delete (concat "home/" [FLOW_PID] "/" [source_filename]);} {} "[err]"; try { json load "etl_left" (file dir_list_regex (concat "home/" [FLOW_PID]) ".*" true true); if (== (json count "etl_left" /) 0) {file delete (concat "home/" [FLOW_PID]); -> "[etl_run_done]" 1;}; json unload "etl_left"; } {} "[err]"; #Step 5 is valid; stack flow_step [FLOW_PID] 5 "load_ok"; # END OF THE RUN (its last file is loaded: the folder of the run is deleted): done mark for the joins; if (== [etl_run_done] 1) { try {etl mark_done "demo.orders.sql_to_sql";} {log write (concat "ETL done mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; }; } { try {sql rollback "destination_sess";} {} "[err]"; try {sql disconnect "destination_sess";} {} "[err]"; #Step 5 is not valid; stack flow_step [FLOW_PID] 5 "load_KO"; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.sql_to_sql.step_5_load.exe" "demo_orders_sql_to_sql")) { group grant script "demo.orders.sql_to_sql.step_5_load.exe" "demo_orders_sql_to_sql"; }; # No join: step 1 runs as soon as it is in the stack; script set delay "demo.orders.sql_to_sql.step_1_source_connect.exe" 0 day {1};
a kind that does not match the connection
admin
etl generate "{\"name\": \"bad\", \"prefix\": \"demo.bad\", \"source\": {\"cm\": \"demo_in\", \"kind\": \"sql\", \"family\": \"file\", \"filter\": \"*.csv\", \"format\": \"csv\"}, \"destination\": {\"cm\": \"demo_out\", \"family\": \"file\", \"format\": \"json\"}}"
mentdb
ERROR: Sorry, source.kind is 'sql' but the connection 'demo_in' (file) is of kind 'files'.
free MQL that would close the script is refused
admin
etl generate "{\"name\": \"bad\", \"prefix\": \"demo.bad\", \"source\": {\"cm\": \"demo_in\", \"family\": \"file\", \"filter\": \"*.csv\", \"format\": \"csv\"}, \"transforms\": [{\"type\": \"mql\", \"code\": \"} script delete \\"x\\"; {\"}], \"destination\": {\"cm\": \"demo_out\", \"family\": \"file\", \"format\": \"json\"}}"
mentdb
ERROR: Sorry, transforms[0].code: the braces { } are not balanced.
etl design_get <prefix>
Description
The design kept in step 1 of a generated flow, to reopen it in the designer
The design kept in step 1 of a generated flow, to reopen it in the designer
Parameters
prefix: The prefix of the scripts of the flow - String - required
prefix: The prefix of the scripts of the flow - String - required
admin
etl design_get "demo.orders.csv_to_json"
mentdb
{"name": "orders_csv_to_json", "prefix": "demo.orders.csv_to_json", "source": {"cm": "demo_in", "kind": "files", "family": "file", "filter": "*.csv", "format": "csv", "delete_source": true}, "transforms": [{"type": "select", "columns": ["id", "client", "total"]}, {"type": "where", "condition": {"total": {"gt": 100}}}, {"type": "rename", "map": {"total": "amount"}}, {"type": "sort", "columns": [["amount", "desc"]]}], "destination": {"cm": "demo_out", "kind": "files", "family": "file", "format": "json", "file_name": "orders_[SOURCE]_[FLOW_PID].json"}}
etl generate <design>
Description
Links (ETL spaces): next and wait_for in the design add to the MQL the start and done marks, the flows put in the stack at the end of a run (success) or on an error, the delay of step 1 and the arming of the join. A success link that makes a cycle is refused
Links (ETL spaces): next and wait_for in the design add to the MQL the start and done marks, the flows put in the stack at the end of a run (success) or on an error, the delay of step 1 and the arming of the join. A success link that makes a cycle is refused
Parameters
design: The design JSON with next [{prefix, on: success|error}] and/or wait_for [prefixes] - String - required
design: The design JSON with next [{prefix, on: success|error}] and/or wait_for [prefixes] - String - required
admin
etl generate "{\"name\": \"orders_join\", \"prefix\": \"demo.orders.join\", \"wait_for\": [\"demo.orders.csv_to_json\", \"demo.orders.sql_to_sql\"], \"next\": [{\"prefix\": \"demo.ops.alert\", \"on\": \"error\"}], \"source\": {\"cm\": \"MENTDB\", \"kind\": \"sql\", \"family\": \"db\", \"query\": \"select id, amount from etl_orders_new\"}, \"destination\": {\"cm\": \"demo_out\", \"kind\": \"files\", \"family\": \"file\", \"format\": \"csv\", \"file_name\": \"orders_[DATE].csv\"}}"
mentdb
# ETL flow 'orders_join' generated by the ETL designer (etl generate): db (sql) -> 0 transformation(s) -> file (files); if (not (group exist "demo_orders_join")) { group add "demo_orders_join"; }; script merge "demo.orders.join.step_1_source_connect.exe" true 5 (param ) "Connect to the source (orders_join)" { # ETL_DESIGN eyJuYW1lIjogIm9yZGVyc19qb2luIiwgInByZWZpeCI6ICJkZW1vLm9yZGVycy5qb2luIiwgIndhaXRfZm9yIjogWyJkZW1vLm9yZGVycy5jc3ZfdG9fanNvbiIsICJkZW1vLm9yZGVycy5zcWxfdG9fc3FsIl0sICJuZXh0IjogW3sicHJlZml4IjogImRlbW8ub3BzLmFsZXJ0IiwgIm9uIjogImVycm9yIn1dLCAic291cmNlIjogeyJjbSI6ICJNRU5UREIiLCAia2luZCI6ICJzcWwiLCAiZmFtaWx5IjogImRiIiwgInF1ZXJ5IjogInNlbGVjdCBpZCwgYW1vdW50IGZyb20gZXRsX29yZGVyc19uZXcifSwgImRlc3RpbmF0aW9uIjogeyJjbSI6ICJkZW1vX291dCIsICJraW5kIjogImZpbGVzIiwgImZhbWlseSI6ICJmaWxlIiwgImZvcm1hdCI6ICJjc3YiLCAiZmlsZV9uYW1lIjogIm9yZGVyc19bREFURV0uY3N2In19; # ETL_HASH 886a918bc49d868ad5bece067dab940bb9b3bfebfcc4b15000157525747cb51c 9f4a0854b610fc6f 2a1d1d0733afac73 9cb3bdcb2c1cd001 5a8d08b33a2d46fd 0eeb259bda4911cd; # WORK FOLDER: every file of a run is in home/[FLOW_PID]/, [FLOW_PID] being the PID of this step 1; # home/[FLOW_PID]/<file>: a file extracted at step 2, extract.json for a sql, rows or messages source; # home/[FLOW_PID]/<file>.rows.json: the rows after step 3; # home/[FLOW_PID]/<file>.out/<name>: the file built at step 5 for a files destination; # Step 5 deletes the files of its file once loaded, then home/[FLOW_PID]/ when it is empty: all the files are loaded; # step 2 deletes it when there is nothing to extract. After an error the files stay for the replay of the step; # a replay of step 1 is a new run: a new PID, a new folder; # CONFIGURATION ; -> "[CONF_NAME_OF_THE_FLOW]" "orders_join"; -> "[CONF_SOURCE_CM]" "MENTDB"; # INITIALIZATION ; -> "[FLOW_PID]" [PID]; -> "[etl_connected]" 0; -> "[etl_err_next]" 0; # HANDLE ; try { # Flow initialization; stack flow_init [FLOW_PID] [CONF_NAME_OF_THE_FLOW] "{}"; stack flow_step [FLOW_PID] 1 "source_connect..."; # Start mark: the last run of this flow (etl space_view); try {etl mark_start "demo.orders.join" [FLOW_PID];} {log write (concat "ETL start mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; sql connect "session_remote" {cm get [CONF_SOURCE_CM];}; #Step 1 is valid; stack flow_step [FLOW_PID] 1 "source_connect_ok"; -> "[etl_connected]" 1; include "demo.orders.join.step_2_extract.exe"; } { try {sql disconnect "session_remote";} {} "[err]"; #Step 1 is not valid (an error of step 2 is already marked by step 2); if (== [etl_connected] 0) {stack flow_step [FLOW_PID] 1 "source_connect_KO";}; # ERROR LINKS: once per run, at the last attempt (not twice when the flows already wait); if (not (== [etl_err_next] 1)) { -> "[etl_err_next]" 1; try { if (== (etl error_mark "demo.orders.join" [FLOW_PID] [PID]) 1) { try {etl run "demo.ops.alert";} {log write (concat "Error flow 'demo.ops.alert' not started: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; } {log write (concat "Error links not checked: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.join.step_1_source_connect.exe" "demo_orders_join")) { group grant script "demo.orders.join.step_1_source_connect.exe" "demo_orders_join"; }; script merge "demo.orders.join.step_2_extract.exe" true 1 (param ) "Extract from the source (orders_join)" { # CONFIGURATION ; -> "[CONF_SOURCE_QUERY]" "select id, amount from etl_orders_new"; # HANDLE ; try { # Mark the flow as step 2; stack flow_step [FLOW_PID] 2 "extract..."; # Create the local directory PID; if (not (file exist (concat "home/" [FLOW_PID]))) {file mkdir (concat "home/" [FLOW_PID]);}; log write "Run the query of the source." OK null null; -> "[rows]" (json to_rows (sql to json "session_remote" "rows" [CONF_SOURCE_QUERY]) "{\"path\": \"$.data\"}"); json load "etl_rows" [rows]; log write (concat (json count "etl_rows" /) " row(s) extracted.") OK null null; if (> (json count "etl_rows" /) 0) { file create (concat "home/" [FLOW_PID] "/extract.json") [rows] "UTF-8"; json load "valid_files" "[\"extract.json\"]"; } { json load "valid_files" "[]"; }; json unload "etl_rows"; sql disconnect "session_remote"; } { try {sql disconnect "session_remote";} {} "[err]"; #Step 2 is not valid; stack flow_step [FLOW_PID] 2 "extract_KO"; # ERROR LINKS: once per run, at the last attempt (not twice when the flows already wait); if (not (== [etl_err_next] 1)) { -> "[etl_err_next]" 1; try { if (== (etl error_mark "demo.orders.join" [FLOW_PID] [PID]) 1) { try {etl run "demo.ops.alert";} {log write (concat "Error flow 'demo.ops.alert' not started: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; } {log write (concat "Error links not checked: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; log write (concat (json count "valid_files" /) " file(s) to transform.") OK null null; if (> (json count "valid_files" /) 0) { #Step 2 is valid; stack flow_step [FLOW_PID] 2 (concat "extract_ok_" (json count "valid_files" /) "_files"); json parse_array "valid_files" "/" "[filename]" { stack (date now) "demo.orders.join.step_3_transform.exe" "[FLOW_PID]" [FLOW_PID] "[filename]" [filename]; }; } { #Step 2 is valid; stack flow_step [FLOW_PID] 2 "extract_ok_zero_file"; # Nothing to transform; file delete (concat "home/" [FLOW_PID]); # END OF THE RUN (nothing to extract counts as a run ended OK): done mark for the joins; try {etl mark_done "demo.orders.join";} {log write (concat "ETL done mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; }; } "Return nothing"; if (not (group is granted script "demo.orders.join.step_2_extract.exe" "demo_orders_join")) { group grant script "demo.orders.join.step_2_extract.exe" "demo_orders_join"; }; script merge "demo.orders.join.step_3_transform.exe" true 1 (param (var "[FLOW_PID]" {true} "The flow PID" is_null:true is_empty:true "1") (var "[filename]" {true} "The filename" is_null:true is_empty:true "file.xxx") ) "Transform the rows (orders_join)" { # CONFIGURATION ; -> "[CONF_TRANSFORMED_FILENAME]" (concat [filename] ".rows.json"); -> "[etl_err_next]" 0; # HANDLE ; try { # Mark the flow as step 3; stack flow_step [FLOW_PID] 3 "transform..."; -> "[etl_file]" (concat "home/" [FLOW_PID] "/" [filename]); log write (concat "Read the rows of '" [etl_file] "' (rows).") OK null null; -> "[rows]" (file load [etl_file] "UTF-8"); json load "etl_rows" [rows]; log write (concat (json count "etl_rows" /) " row(s) after the transformations.") OK null null; json unload "etl_rows"; file create (concat "home/" [FLOW_PID] "/" [CONF_TRANSFORMED_FILENAME]) [rows] "UTF-8"; #Step 3 is valid; stack flow_step [FLOW_PID] 3 "transform_ok"; stack (date now) "demo.orders.join.step_4_destination_connect.exe" "[FLOW_PID]" [FLOW_PID] "[filename]" [CONF_TRANSFORMED_FILENAME] "[source_filename]" [filename]; } { #Step 3 is not valid; stack flow_step [FLOW_PID] 3 "transform_KO"; # ERROR LINKS: once per run, at the last attempt (not twice when the flows already wait); if (not (== [etl_err_next] 1)) { -> "[etl_err_next]" 1; try { if (== (etl error_mark "demo.orders.join" [FLOW_PID] [PID]) 1) { try {etl run "demo.ops.alert";} {log write (concat "Error flow 'demo.ops.alert' not started: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; } {log write (concat "Error links not checked: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.join.step_3_transform.exe" "demo_orders_join")) { group grant script "demo.orders.join.step_3_transform.exe" "demo_orders_join"; }; script merge "demo.orders.join.step_4_destination_connect.exe" true 5 (param (var "[FLOW_PID]" {true} "The flow PID" is_null:true is_empty:true "1") (var "[filename]" {true} "The filename" is_null:true is_empty:true "file.xxx") (var "[source_filename]" {true} "The file extracted at step 2" is_null:true is_empty:true "file.xxx") ) "Connect to the destination (orders_join)" { # CONFIGURATION ; -> "[CONF_DESTINATION_CM]" "demo_out"; -> "[etl_step5]" 0; -> "[etl_err_next]" 0; # HANDLE ; try { # Flow step 4; stack flow_step [FLOW_PID] 4 "destination_connect..."; # Local connection: nothing to open; log write "Destination: local files" OK null null; #Step 4 is valid; stack flow_step [FLOW_PID] 4 "destination_connect_ok"; include "demo.orders.join.step_5_load.exe"; } { #Step 4 is not valid (an error of step 5 is already marked by step 5); if (not (== [etl_step5] 1)) {stack flow_step [FLOW_PID] 4 "destination_connect_KO";}; # ERROR LINKS: once per run, at the last attempt (not twice when the flows already wait); if (not (== [etl_err_next] 1)) { -> "[etl_err_next]" 1; try { if (== (etl error_mark "demo.orders.join" [FLOW_PID] [PID]) 1) { try {etl run "demo.ops.alert";} {log write (concat "Error flow 'demo.ops.alert' not started: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; } {log write (concat "Error links not checked: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.join.step_4_destination_connect.exe" "demo_orders_join")) { group grant script "demo.orders.join.step_4_destination_connect.exe" "demo_orders_join"; }; script merge "demo.orders.join.step_5_load.exe" true 1 (param ) "Load to the destination (orders_join)" { # CONFIGURATION ; -> "[CONF_DESTINATION_CM]" "demo_out"; json load "etl_cm_destination" {cm get [CONF_DESTINATION_CM];}; -> "[CONF_DIR_DESTINATION]" (json select "etl_cm_destination" "/localDir"); json unload "etl_cm_destination"; -> "[CONF_DESTINATION_FORMAT_OPTIONS]" "{}"; -> "[CONF_DESTINATION_FILENAME]" (concat "orders_" (date systimestamp_min) ".csv"); -> "[etl_step5]" 1; -> "[etl_run_done]" 0; # HANDLE ; try { # Mark the flow as step 5; stack flow_step [FLOW_PID] 5 "load..."; -> "[rows]" (file load (concat "home/" [FLOW_PID] "/" [filename]) "UTF-8"); if (not (and (string matches [CONF_DESTINATION_FILENAME] "[^/\\\p{Cntrl}]+") (not (string matches [CONF_DESTINATION_FILENAME] "[.][.]?")))) {exception (1) (concat "The destination file name is not a plain file name: " (string regexp_replace [CONF_DESTINATION_FILENAME] "\p{Cntrl}" "?"));}; # The file is built in a folder of its own: home/[FLOW_PID]/<source file>.out/; -> "[etl_out_dir]" (concat "home/" [FLOW_PID] "/" [source_filename] ".out"); if (not (file exist [etl_out_dir])) {file mkdir [etl_out_dir];}; -> "[etl_out]" (concat [etl_out_dir] "/" [CONF_DESTINATION_FILENAME]); log write (concat "Build the file '" [etl_out] "' (csv).") OK null null; csv write [etl_out] [rows] [CONF_DESTINATION_FORMAT_OPTIONS]; log write (concat "Send '" [etl_out] "' to '" [CONF_DIR_DESTINATION] "/" [CONF_DESTINATION_FILENAME] "'...") OK null null; if (not (file exist [CONF_DIR_DESTINATION])) {file mkdir [CONF_DIR_DESTINATION];}; file copy_file [etl_out] (concat [CONF_DIR_DESTINATION] "/" [CONF_DESTINATION_FILENAME]); if (not (file exist (concat [CONF_DIR_DESTINATION] "/" [CONF_DESTINATION_FILENAME]))) { exception (1) (concat "Cannot copy the file '" [CONF_DESTINATION_FILENAME] "'."); }; file delete [etl_out]; try {file delete [etl_out_dir];} {} "[err]"; log write "Sent." OK null null; # Remove the work files of this file, then the folder of the run when it is empty (all its files are loaded); try {file delete (concat "home/" [FLOW_PID] "/" [filename]);} {} "[err]"; try {file delete (concat "home/" [FLOW_PID] "/" [source_filename]);} {} "[err]"; try { json load "etl_left" (file dir_list_regex (concat "home/" [FLOW_PID]) ".*" true true); if (== (json count "etl_left" /) 0) {file delete (concat "home/" [FLOW_PID]); -> "[etl_run_done]" 1;}; json unload "etl_left"; } {} "[err]"; #Step 5 is valid; stack flow_step [FLOW_PID] 5 "load_ok"; # END OF THE RUN (its last file is loaded: the folder of the run is deleted): done mark for the joins; if (== [etl_run_done] 1) { try {etl mark_done "demo.orders.join";} {log write (concat "ETL done mark not written: " [etl_mark_err]) KO null null;} "[etl_mark_err]"; }; } { try {file delete [etl_out];} {} "[err]"; try {file delete [etl_out_dir];} {} "[err]"; #Step 5 is not valid; stack flow_step [FLOW_PID] 5 "load_KO"; # ERROR LINKS: once per run, at the last attempt (not twice when the flows already wait); if (not (== [etl_err_next] 1)) { -> "[etl_err_next]" 1; try { if (== (etl error_mark "demo.orders.join" [FLOW_PID] [PID]) 1) { try {etl run "demo.ops.alert";} {log write (concat "Error flow 'demo.ops.alert' not started: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; } {log write (concat "Error links not checked: " [etl_next_err]) KO null null;} "[etl_next_err]"; }; # Give the error to the stack and stop the process; exception (1) ([global_err]); } "[global_err]"; } "Return nothing"; if (not (group is granted script "demo.orders.join.step_5_load.exe" "demo_orders_join")) { group grant script "demo.orders.join.step_5_load.exe" "demo_orders_join"; }; # JOIN: step 1 waits for demo.orders.csv_to_json, demo.orders.sql_to_sql (each must end OK since the join was armed or last passed), checked again every minute; script set delay "demo.orders.join.step_1_source_connect.exe" 1 min {etl join_ready "demo.orders.join" "demo.orders.csv_to_json,demo.orders.sql_to_sql" [PID]}; # Arm the join now: the flows ended before this generation do not count; etl join_arm "demo.orders.join";
a flow cannot be linked to itself
admin
etl generate "{\"name\": \"orders_csv_to_json\", \"prefix\": \"demo.orders.csv_to_json\", \"source\": {\"cm\": \"demo_in\", \"kind\": \"files\", \"family\": \"file\", \"filter\": \"*.csv\", \"format\": \"csv\", \"delete_source\": true}, \"transforms\": [{\"type\": \"select\", \"columns\": [\"id\", \"client\", \"total\"]}, {\"type\": \"where\", \"condition\": {\"total\": {\"gt\": 100}}}, {\"type\": \"rename\", \"map\": {\"total\": \"amount\"}}, {\"type\": \"sort\", \"columns\": [[\"amount\", \"desc\"]]}], \"destination\": {\"cm\": \"demo_out\", \"kind\": \"files\", \"family\": \"file\", \"format\": \"json\", \"file_name\": \"orders_[SOURCE]_[FLOW_PID].json\"}, \"next\": [{\"prefix\": \"demo.orders.csv_to_json\"}]}"
mentdb
ERROR: Sorry, next[0].prefix: a flow cannot be linked to itself.
etl space_save <space>
Description
Create or replace a space: {name, description, nodes: [{prefix, x, y, draft (optional: the design of an object drawn but not generated yet), ui}], notes: [{text, x, y, ui}], view: {zoom, x, y}, ui}. updated and by are set by the server. Returns the new updated. To replace an existing space give the updated read with it: a space changed since (or a save without updated) is refused
Create or replace a space: {name, description, nodes: [{prefix, x, y, draft (optional: the design of an object drawn but not generated yet), ui}], notes: [{text, x, y, ui}], view: {zoom, x, y}, ui}. updated and by are set by the server. Returns the new updated. To replace an existing space give the updated read with it: a space changed since (or a save without updated) is refused
Parameters
space: The space JSON. Name: letters, digits, _ . - (100 characters at most) - String - required
space: The space JSON. Name: letters, digits, _ . - (100 characters at most) - String - required
admin
etl space_save "{\"name\": \"demo_erp\", \"description\": \"Orders of the ERP\", \"nodes\": [{\"prefix\": \"demo.orders.csv_to_json\", \"x\": 40, \"y\": 40}, {\"prefix\": \"demo.orders.sql_to_sql\", \"x\": 40, \"y\": 180}, {\"prefix\": \"demo.orders.join\", \"x\": 320, \"y\": 110, \"draft\": {\"name\": \"orders_join\", \"prefix\": \"demo.orders.join\", \"wait_for\": [\"demo.orders.csv_to_json\", \"demo.orders.sql_to_sql\"], \"next\": [{\"prefix\": \"demo.ops.alert\", \"on\": \"error\"}], \"source\": {\"cm\": \"MENTDB\", \"kind\": \"sql\", \"family\": \"db\", \"query\": \"select id, amount from etl_orders_new\"}, \"destination\": {\"cm\": \"demo_out\", \"kind\": \"files\", \"family\": \"file\", \"format\": \"csv\", \"file_name\": \"orders_[DATE].csv\"}}}], \"notes\": [{\"text\": \"Joins at night\", \"x\": 40, \"y\": 300}], \"view\": {\"zoom\": 1.0, \"x\": 0, \"y\": 0}}"
mentdb
2026-11-19 13:11:57.462
the same space saved again without its updated
admin
etl space_save "{\"name\": \"demo_erp\"}"
mentdb
ERROR: Sorry, the space 'demo_erp' already exists (saved by 'mentdb' at 2026-11-19 13:11:57.462): give its 'updated' (etl space_get) to replace it.
a node twice
admin
etl space_save "{\"name\": \"demo_bad\", \"nodes\": [{\"prefix\": \"a.b\"}, {\"prefix\": \"a.b\"}]}"
mentdb
ERROR: Sorry, nodes[1]: the flow 'a.b' is already in the space.
etl space_get <name>
Description
The space JSON (with updated and by); the nodes the user is not granted are {prefix, hidden: true}
The space JSON (with updated and by); the nodes the user is not granted are {prefix, hidden: true}
Parameters
name: The name of the space - String - required
name: The name of the space - String - required
admin
etl space_get "demo_erp"
mentdb
{"name": "demo_erp", "description": "Orders of the ERP", "nodes": [{"prefix": "demo.orders.csv_to_json", "x": 40, "y": 40}, {"prefix": "demo.orders.sql_to_sql", "x": 40, "y": 180}, {"prefix": "demo.orders.join", "x": 320, "y": 110, "draft": {"name": "orders_join", "prefix": "demo.orders.join", "wait_for": ["demo.orders.csv_to_json", "demo.orders.sql_to_sql"], "next": [{"prefix": "demo.ops.alert", "on": "error"}], "source": {"cm": "MENTDB", "kind": "sql", "family": "db", "query": "select id, amount from etl_orders_new"}, "destination": {"cm": "demo_out", "kind": "files", "family": "file", "format": "csv", "file_name": "orders_[DATE].csv"}}}], "notes": [{"text": "Joins at night", "x": 40, "y": 300}], "view": {"zoom": 1, "x": 0, "y": 0}, "updated": "2026-11-19 13:11:57.462", "by": "mentdb"}
etl space_list
Description
The spaces: rows {name, description, nodes (count), updated, by}. A user who is not sys sees the spaces he saved last and those whose generated nodes are all granted to him
The spaces: rows {name, description, nodes (count), updated, by}. A user who is not sys sees the spaces he saved last and those whose generated nodes are all granted to him
admin
etl space_list
mentdb
[{"name": "demo_erp", "description": "Orders of the ERP", "nodes": 3, "updated": "2026-11-19 13:11:57.462", "by": "mentdb"}]
etl space_view <name>
Description
Everything to draw a space in one call: {space, nodes: [{prefix, exists, draft, x, y, design: {name, description, source: {cm, kind, family, type}, destination: {cm, kind, family, type}, transforms (count)}, next, wait_for, manual_modified, state: {state (none, waiting, running, closed, error), pid, date, message}}], cycles: [[prefixes of a cycle of success links]]}. A node not generated gives its draft design; a node the user is not granted is {prefix, hidden: true}
Everything to draw a space in one call: {space, nodes: [{prefix, exists, draft, x, y, design: {name, description, source: {cm, kind, family, type}, destination: {cm, kind, family, type}, transforms (count)}, next, wait_for, manual_modified, state: {state (none, waiting, running, closed, error), pid, date, message}}], cycles: [[prefixes of a cycle of success links]]}. A node not generated gives its draft design; a node the user is not granted is {prefix, hidden: true}
Parameters
name: The name of the space - String - required
name: The name of the space - String - required
admin
etl space_view "demo_erp"
mentdb
{"space": {"name": "demo_erp", "description": "Orders of the ERP", "nodes": [{"prefix": "demo.orders.csv_to_json", "x": 40, "y": 40}, {"prefix": "demo.orders.sql_to_sql", "x": 40, "y": 180}, {"prefix": "demo.orders.join", "x": 320, "y": 110, "draft": {"name": "orders_join", "prefix": "demo.orders.join", "wait_for": ["demo.orders.csv_to_json", "demo.orders.sql_to_sql"], "next": [{"prefix": "demo.ops.alert", "on": "error"}], "source": {"cm": "MENTDB", "kind": "sql", "family": "db", "query": "select id, amount from etl_orders_new"}, "destination": {"cm": "demo_out", "kind": "files", "family": "file", "format": "csv", "file_name": "orders_[DATE].csv"}}}], "notes": [{"text": "Joins at night", "x": 40, "y": 300}], "view": {"zoom": 1, "x": 0, "y": 0}, "updated": "2026-11-19 13:11:57.462", "by": "mentdb"}, "nodes": [{"prefix": "demo.orders.csv_to_json", "exists": false, "draft": false, "x": 40, "y": 40, "design": null, "next": [], "wait_for": [], "manual_modified": false, "state": {"state": "none", "pid": null, "date": null, "message": null}}, {"prefix": "demo.orders.sql_to_sql", "exists": false, "draft": false, "x": 40, "y": 180, "design": null, "next": [], "wait_for": [], "manual_modified": false, "state": {"state": "none", "pid": null, "date": null, "message": null}}, {"prefix": "demo.orders.join", "exists": false, "draft": true, "x": 320, "y": 110, "design": {"name": "orders_join", "description": "", "source": {"cm": "MENTDB", "kind": "sql", "family": "db", "type": "sql"}, "destination": {"cm": "demo_out", "kind": "files", "family": "file", "type": "file"}, "transforms": 0}, "next": [{"prefix": "demo.ops.alert", "on": "error"}], "wait_for": ["demo.orders.csv_to_json", "demo.orders.sql_to_sql"], "manual_modified": false, "state": {"state": "none", "pid": null, "date": null, "message": null}}], "cycles": []}
etl space_delete <name>
Description
Delete a space (not its flows). Returns 1
Delete a space (not its flows). Returns 1
Parameters
name: The name of the space - String - required
name: The name of the space - String - required
admin
etl space_delete "demo_erp"
mentdb
1
etl manual_check <prefix>
Description
Were the scripts of a generated flow modified by hand since their generation? {prefix, exists, hash (the flow has an ETL_HASH line), modified, scripts: [{name, exists, modified}]}
Were the scripts of a generated flow modified by hand since their generation? {prefix, exists, hash (the flow has an ETL_HASH line), modified, scripts: [{name, exists, modified}]}
Parameters
prefix: The prefix of the scripts of the flow - String - required
prefix: The prefix of the scripts of the flow - String - required
admin
etl manual_check "demo.orders.csv_to_json"
mentdb
{"prefix": "demo.orders.csv_to_json", "exists": true, "hash": true, "modified": true, "scripts": [{"name": "demo.orders.csv_to_json.step_1_source_connect.exe", "exists": true, "modified": false}, {"name": "demo.orders.csv_to_json.step_2_extract.exe", "exists": true, "modified": false}, {"name": "demo.orders.csv_to_json.step_3_transform.exe", "exists": true, "modified": true}, {"name": "demo.orders.csv_to_json.step_4_destination_connect.exe", "exists": true, "modified": false}, {"name": "demo.orders.csv_to_json.step_5_load.exe", "exists": true, "modified": false}]}
etl run <prefix>
Description
Put step 1 of a generated flow in the stack, now: its pid, or a message when it already waits there (put in the stack, not started yet: not added again). A flow with wait_for starts only when its join is ready
Put step 1 of a generated flow in the stack, now: its pid, or a message when it already waits there (put in the stack, not started yet: not added again). A flow with wait_for starts only when its join is ready
Parameters
prefix: The prefix of the scripts of the flow - String - required
prefix: The prefix of the scripts of the flow - String - required
admin
etl run "demo.orders.csv_to_json"
mentdb
w
etl mark_start <prefix> <pid>
Description
Written by step 1 of a generated flow (just after stack flow_init): the start mark of the flow with the pid of the run (etl space_view finds the last run with it). Returns 1. The marks of a flow are written by the flow itself (its step 1 granted) or by sys
Written by step 1 of a generated flow (just after stack flow_init): the start mark of the flow with the pid of the run (etl space_view finds the last run with it). Returns 1. The marks of a flow are written by the flow itself (its step 1 granted) or by sys
Parameters
prefix: The prefix of the flow - String - required
pid: The pid of the run ([FLOW_PID]) - String - required
prefix: The prefix of the flow - String - required
pid: The pid of the run ([FLOW_PID]) - String - required
admin
etl mark_start "demo.orders.join" "w"
mentdb
1
etl join_arm <prefix>
Description
Written by the MQL of etl generate for a flow with wait_for: arms the join, the flows ended before do not count. Returns 1
Written by the MQL of etl generate for a flow with wait_for: arms the join, the flows ended before do not count. Returns 1
Parameters
prefix: The prefix of the flow that waits - String - required
prefix: The prefix of the flow that waits - String - required
admin
etl join_arm "demo.orders.join"
mentdb
1
etl mark_done <prefix>
Description
Written by step 5 of a generated flow at the end of a run that ended OK: the done mark of the flow. Returns 1
Written by step 5 of a generated flow at the end of a run that ended OK: the done mark of the flow. Returns 1
Parameters
prefix: The prefix of the flow - String - required
prefix: The prefix of the flow - String - required
admin
etl mark_done "demo.orders.csv_to_json"
mentdb
1
etl join_ready <prefix> <wait_for>
Description
Is the join ready? 1 when every flow of the list has a done mark more recent than the arming of the join and than its last pass, else 0 (nothing is recorded)
Is the join ready? 1 when every flow of the list has a done mark more recent than the arming of the join and than its last pass, else 0 (nothing is recorded)
Parameters
prefix: The prefix of the flow that waits - String - required
wait_for: The prefixes of the flows waited for, separated by commas - String - required
prefix: The prefix of the flow that waits - String - required
wait_for: The prefixes of the flows waited for, separated by commas - String - required
admin
etl join_ready "demo.orders.join" "demo.orders.csv_to_json,demo.orders.sql_to_sql"
mentdb
0
both flows have ended OK since the arming
admin
etl mark_done "demo.orders.sql_to_sql"; etl join_ready "demo.orders.join" "demo.orders.csv_to_json,demo.orders.sql_to_sql"
mentdb
1
etl join_ready <prefix> <wait_for> <pid>
Description
The delay condition of step 1 of a flow with wait_for (the stack gives [PID]): when the join is ready its pass is recorded with this pid, so the next run needs new completions while a retry of the same stack entry passes again
The delay condition of step 1 of a flow with wait_for (the stack gives [PID]): when the join is ready its pass is recorded with this pid, so the next run needs new completions while a retry of the same stack entry passes again
Parameters
prefix: The prefix of the flow that waits - String - required
wait_for: The prefixes of the flows waited for, separated by commas - String - required
pid: The pid of the stack entry ([PID]) - String - required
prefix: The prefix of the flow that waits - String - required
wait_for: The prefixes of the flows waited for, separated by commas - String - required
pid: The pid of the stack entry ([PID]) - String - required
admin
etl join_ready "demo.orders.join" "demo.orders.csv_to_json,demo.orders.sql_to_sql" "x"
mentdb
1
another entry y: it waits for new completions
admin
etl join_ready "demo.orders.join" "demo.orders.csv_to_json,demo.orders.sql_to_sql" "y"
mentdb
0
etl error_mark <prefix> <flow_pid> <pid>
Description
Used by the error links of a generated flow: 1 only for the first error of the run and at the last attempt of the stack entry (a step that will be tried again gives 0)
Used by the error links of a generated flow: 1 only for the first error of the run and at the last attempt of the stack entry (a step that will be tried again gives 0)
Parameters
prefix: The prefix of the flow - String - required
flow_pid: The pid of the run ([FLOW_PID]) - String - required
pid: The pid of the stack entry ([PID]); empty: taken as the last attempt - String - required
prefix: The prefix of the flow - String - required
flow_pid: The pid of the run ([FLOW_PID]) - String - required
pid: The pid of the stack entry ([PID]); empty: taken as the last attempt - String - required
admin
etl error_mark "demo.orders.join" "w" ""
mentdb
1
etl marks <prefix>
Description
The marks of a flow: {prefix, start, start_pid, done, armed, join, join_pid, error, error_pid} (dates of the database with milliseconds, null when none; a pid only when the mark has one). Needs the grant of the flow (or sys)
The marks of a flow: {prefix, start, start_pid, done, armed, join, join_pid, error, error_pid} (dates of the database with milliseconds, null when none; a pid only when the mark has one). Needs the grant of the flow (or sys)
Parameters
prefix: The prefix of the flow - String - required
prefix: The prefix of the flow - String - required
admin
etl marks "demo.orders.join"
mentdb
{"prefix": "demo.orders.join", "start": "2026-11-19 13:11:57.463", "start_pid": "w", "done": null, "armed": "2026-11-19 13:11:57.464", "join": "2026-11-19 13:11:57.467", "join_pid": "x", "error": "2026-11-19 13:11:57.468", "error_pid": "w"}
etl mark_reset <prefix>
Description
Remove every mark of a flow (start, done, armed, join, error): its joins start again from now. Returns 1
Remove every mark of a flow (start, done, armed, join, error): its joins start again from now. Returns 1
Parameters
prefix: The prefix of the flow - String - required
prefix: The prefix of the flow - String - required
admin
etl mark_reset "demo.orders.join"
mentdb
1
