MentDB Weak 4.3.0: 800 new functions for AI, standards and data integration @ Server @ MentDB Weak

This release is a major extension of MQL: about 800 new functions for integration, standards, AI, machine learning and science.
All the existing scripts keep working, except for the functions listed in the upgrade notes.

What's new

• Features: a new page lists everything MentDB can do, in 20 categories with 927 tags and a search. It is built at startup and reaches every function.
• Use Cases: now 1070 real examples for 91 profiles. Under each function, the MQL Functions reference shows the use cases that use it and the functions often used with it.
• AI: llm ask, chat, extract (JSON checked against a schema, from text, PDF or images), classify, summarize, translate, embed, chunk, with Claude, OpenAI, Mistral, Ollama, Voyage and compatible services. vector cosine and vector search for semantic search.
• Machine learning: ml train, ml fit, ml model (about 60 algorithms), ml automl, PMML export, data preparation, metrics, clustering, anomalies, recommendation, pattern mining, data streams, and time series (auto_arima, sarimax, ets, garch, var).
• Data formats: CSV and Excel to JSON rows, nested JSON to table, XML (XSD, XSLT), YAML, fixed width files with EBCDIC, Parquet read and write.
• Documents: Word templates with docx fill (fields, loops, conditions, images), docx create, Markdown conversion, 23 image functions (resize, crop, EXIF, watermark, duplicates), QR codes and 12 barcode types, created and read.
• Connectors: rest request and rest paginate (retries, OAuth2, multipart), SQL bulk insert and upsert, table sync, S3, Redis, Kafka, MQTT, AMQP, LDAP and Active Directory (read only).
• Business standards:
  • E-invoicing: Factur-X, CII, UBL, EN 16931 checks, invoice PDF.
  • Banking: SEPA credit transfers and direct debits, camt.052, camt.053, camt.054, CFONB 120, MT940, automatic reconciliation.
  • Accounting: FEC.
  • EDI: EDIFACT and X12, with CONTRL and 997 acknowledgements.
  • Healthcare: HL7 v2 with MLLP, FHIR REST.
• Security: PDF signature (PAdES) and XML signature (XML-DSig, XAdES) with verification, PGP, JWT, AES-256-GCM, HMAC, TOTP and HOTP, EPC payment QR codes, PEM and server certificate reading.
• Mail and calendars: read and write EML files, read Outlook MSG files, extract attachments safely, create and read iCalendar and vCard.
• Network checks: dns lookup, tcp check, tls certificate (days before expiry), http check, IP and CIDR calculation, IPv6 included.
• Data toolkit: 31 array functions, 24 new string functions in the style of PHP (format, number_format, natural_sort, query_parse...), url functions, new regex functions with a time limit.
• Analytics: data profiling, in-memory tables (table join, pivot, window), record linkage, data drift, data privacy, text mining, process mining, A/B tests, cohorts, RFM, marketing attribution.
• Science and operations:
  • Statistics: survival analysis, GLM, mixed models, GAM, causal inference, Bayesian inference with NUTS, multiple imputation, meta-analysis.
  • Laboratory and quality: method validation, control charts, Gage R&R, design of experiments, chemometrics.
  • Decisions: linear programming, supply chain, queueing, routing, scheduling, project planning, maintenance, finance, actuarial science, surveys, geospatial.
• Calculator of the editor: new functions in the ML, Stat, Num, DSP, Math and Vector pages.
• Administration: the AI models screen lists the ml models, the sketches and dl n_bayesian.

Upgrade notes

• From 4.2.3: copy the new src, core, docs and MentDB_Weak_Editor_GPLv3/core folders, and the start scripts of the bin folder, then restart.
• In the lib folder:
  • Add liblinear-2.44.jar, libsvm-3.35.jar, spmf-java21.jar, zxing-core-3.5.3.jar, commonmark-0.24.0.jar, commonmark-ext-gfm-tables-0.24.0.jar, kafka-clients-3.9.1.jar, lz4-java-1.8.0.jar, snappy-java-1.1.10.7.jar, zstd-jni-1.5.6-6.jar, org.eclipse.paho.client.mqttv3-1.2.5.jar and amqp-client-5.21.0.jar.
  • Delete encog-core-3.4.jar.
• Before starting, run python3 tools/upgrade_check.py with the folder of your server: it lists the missing libraries.
• Merge the three scripts of the AI models screen: app.100.template.admin.actions.SYS.ai_models.source.exe, app.100.template.admin.actions.SYS.ai_models.delete.exe and app.100.template.admin.lib.help.init_lang.exe.
• Before any upgrade, back up the cert and conf folders, and never delete the data folder.
• Behavior changes:
  • dl bayesian, dl csv and dl img are removed: use ml train (mlp_classifier, mlp_regressor, naive_bayes...) and ml prep tfidf. The demonstration scripts demo.dl.cat_dog* are removed too.
  • New function names are checked first: no existing MQL function has been replaced by a new one.
• Good to know:
  • The text sent to llm functions goes to the chosen provider, except with a local Ollama.
  • pdf verify_signatures and xml verify_signature return valid: true only when the document is intact and the signer is in your trust store. MentDB produces advanced signatures, not qualified eIDAS signatures.
  • edifact, x12, hl7 and fhir work at the level of the standard: check them against the guide of each partner. They are not certified medical interfaces.