System administrator & DevOps

Health report in JSON for the monitoring

Every minute the supervision tool pulls one JSON document: CPU of the system and of the JVM, heap usage, RAM really available (cache excluded), swap, and an OK/WARN status above 85 % CPU or 90 % RAM.

#1. Identity and CPU load;
json load "rx_health" "{}";
json iobject "rx_health" / "ts" (date systimestamp) STR;
json iobject "rx_health" / "mentdb" (concat (name) " v" (version)) STR;
json iobject "rx_health" / "cpu_sys" (metric current cpu system) NUM;
json iobject "rx_health" / "cpu_jvm" (metric current cpu jvm) NUM;
#2. JVM heap: used/committed/max in MB;
-> "[jvm]" (metric current mem jvm);
json iobject "rx_health" / "heap_used_pct" (math round (* 100 (/ (atom get [jvm] 1 "/") (atom get [jvm] 3 "/"))) 1) NUM;
#3. RAM really available, free RAM and swap;
-> "[total]" (metric total mem);
json iobject "rx_health" / "ram_used_pct" (math round (* 100 (/ (- [total] (metric current available mem)) [total])) 1) NUM;
json iobject "rx_health" / "ram_free_mb" (metric current free mem) NUM;
json iobject "rx_health" / "ram_used_with_cache_mb" (metric current used mem) NUM;
json iobject "rx_health" / "swap_used_mb" (metric current used swap mem) NUM;
json iobject "rx_health" / "swap_free_mb" (metric current free swap mem) NUM;
json iobject "rx_health" / "swap_total_mb" (metric total swap mem) NUM;
#4. Verdict: WARN above 85 percent CPU or 90 percent RAM;
json iobject "rx_health" / "status" (if (or (> (json select "rx_health" /cpu_sys) 85) (> (json select "rx_health" /ram_used_pct) 90)) {"WARN"} {"OK"}) STR;
json doc "rx_health"

{
  "ram_used_with_cache_mb": 18304.0,
  "cpu_sys": 11.0,
  "ram_free_mb": 127.0,
  "swap_total_mb": 2048.0,
  "ram_used_pct": 73.2,
  "heap_used_pct": 15.2,
  "swap_used_mb": 1149.0,
  "ts": "2026-11-01 16:27:23",
  "mentdb": "mona payet v4.2.3",
  "cpu_jvm": 0.5,
  "swap_free_mb": 898.0,
  "status": "OK"
}

When will the disk be full?

The data disk grows by about 1.8 GB per day. The script reads the root file system from the metrics, keeps a 10 % safety margin and gives the number of days and the date when the margin is reached, with an alert below 30 days.

#1. Root file system from the metrics, sizes in MB;
json load "rx_disk" (metric file system roots);
-> "[mount]" (json select "rx_disk" "/[0]/absolutePath");
-> "[free]" (json select "rx_disk" "/[0]/freeSpace");
-> "[total]" (json select "rx_disk" "/[0]/totalSpace");
-> "[used_pct]" (math round (* 100 (/ (- [total] [free]) [total])) 1);
#2. Growth of 1.8 GB per day, keep a 10 percent safety margin;
-> "[usable]" (- [free] (* 0.10 [total]));
-> "[days]" (math floor (/ [usable] 1800));
#3. Date when the margin is reached;
-> "[limit]" (date add (date sysdate) "DAY" [days]);
json unload "rx_disk";
concat "{\"mount\": \"" [mount] "\", \"used_pct\": " [used_pct] ", \"free_gb\": " (math round (/ [free] 1024) 1) ", \"days_left\": " [days] ", \"margin_reached_on\": \"" [limit] "\", \"alert\": " (if (< [days] 30) {1} {0}) "}"

{"mount": "/", "used_pct": 68.5, "free_gb": 145.2, "days_left": 56, "margin_reached_on": "2026-12-27", "alert": 0}

Server inventory and security checks

Before an audit, the administrator needs the OS, CPUs, Java runtime, time zone, MentDB build and mode, and the days left on the TLS certificate. Two checks are added without printing the account: MentDB must not run as root and must be started from its folder.

#1. Operating system and Java runtime;
json load "rx_inv" "{}";
json iobject "rx_inv" / "os" (concat (os name) " " (os version) " " (os arch)) STR;
json iobject "rx_inv" / "os_type" (os type) STR;
json iobject "rx_inv" / "kernel" (concat (metric system name) " " (metric system version) " " (metric system architecture)) STR;
json iobject "rx_inv" / "cpus" (metric system nb processor) INT;
json iobject "rx_inv" / "java" (concat (metric java version) " " (metric java vendor)) STR;
json iobject "rx_inv" / "tz_lang" (concat (os user_timezone) " " (os user_lang)) STR;
#2. Security checks without printing the account: not root, started from the MentDB folder;
json iobject "rx_inv" / "runs_as_root" (if (or (string equals_ignore_case (os user_name) "root") (string contains (os user_home) "/root")) {1} {0}) INT;
json iobject "rx_inv" / "workdir_is_mentdb" (string contains (os user_dir) "MentDB") INT;
#3. MentDB build, mode and size;
json iobject "rx_inv" / "mentdb" (concat (version) " - " (ai firstname) " " (ai lastname)) STR;
json iobject "rx_inv" / "mode" (if (parameter exist "SERVER_MODE") {parameter get value "SERVER_MODE"} {"?"}) STR;
json iobject "rx_inv" / "mode_doc" (parameter get description "SERVER_MODE") STR;
json iobject "rx_inv" / "mode_private" (parameter is_private "SERVER_MODE") INT;
json iobject "rx_inv" / "mode_restore" (parameter generate update "SERVER_MODE") STR;
json iobject "rx_inv" / "mql_functions" (function count) INT;
json iobject "rx_inv" / "java_lines" (src count) INT;
#4. Days before the TLS certificate expires;
json load "rx_cert" (config cert);
json iobject "rx_inv" / "cert_days_left" (json select "rx_cert" "/[1]/days_left") INT;
json unload "rx_cert";
json doc "rx_inv"

{
  "workdir_is_mentdb": 1,
  "os": "Mac OS X 14.4 aarch64",
  "cpus": 11,
  "mode_restore": "parameter update \"SERVER_MODE\" \"DEV\";",
  "kernel": "Mac OS X 14.4 aarch64",
  "tz_lang": "Indian/Reunion fr",
  "mode_doc": "Mode of the server (DEV, TEST, PROD...), shown in the administration.",
  "mode_private": 0,
  "cert_days_left": 1819,
  "mql_functions": 1606,
  "mode": "DEV",
  "java": "21.0.13 Eclipse Adoptium",
  "os_type": "Mac OS X",
  "runs_as_root": 0,
  "mentdb": "4.2.3 - mona payet",
  "java_lines": 798979
}

Sessions and errors of the day

Morning check: how many sessions are open and against which capacity, how many come from the MQL editor, the oldest one, the web socket sessions, the exceeded sessions, and the scripts that ended in error in the log today.

#1. Open sessions: how many come from the MQL editor, the oldest one;
json load "rx_sess" (sessions);
-> "[n]" (json count "rx_sess" /);
-> "[mql]" 0;
-> "[oldest]" 0;
for (-> "[i]" 0) (< [i] [n]) (++ "[i]") {
	if (equal (json select "rx_sess" (concat "/[" [i] "]/origin")) "MQL") {++ "[mql]"};
	-> "[oldest]" (math max [oldest] (json select "rx_sess" (concat "/[" [i] "]/used")));
};
#2. Session capacity from the metrics, without the editor marker;
json load "rx_ms" (string substring (metric sessions) 39);
#3. Scripts in error in the log today;
json load "rx_ko" (string substring (log search "ko" null null null null (concat (date sysdate) " 00:00:00") (concat (date sysdate) " 23:59:59") DESC 1 50) 39);
json load "rx_ws" (@sessions);
concat "{\"me\": \"" (who) "\", \"my_commands\": " (cmdid) ", \"java_pid\": " (pid) ", \"open\": " (count sessions) ", \"capacity\": \"" (json select "rx_ms" "/data/[0]/open_session") "\", \"from_editor\": " [mql] ", \"websocket\": \"" (@count sessions) "/" (json count "rx_ws" /) "\"" ", \"oldest_s\": " [oldest] ", \"exceeded\": " (exceeded sessions) ", \"ws_exceeded\": " (@exceeded sessions) ", \"log_id\": " (log current id) ", \"errors_today\": " (json count "rx_ko" /data) ", \"log_retention_days\": " (log retention day) ", \"archive\": \"" (log archive path) " every " (log archive size) "\"}"

{"me": "admin", "my_commands": 5, "java_pid": 80950, "open": 2, "capacity": "2/100", "from_editor": 1, "websocket": "0/0", "oldest_s": 3760, "exceeded": 0, "ws_exceeded": 0, "log_id": 0, "errors_today": 0, "log_retention_days": 50, "archive": "archives/logs every 30"}

Maintenance window: lock, change the config, reopen

A change request asks to keep 90 days of logs. The administrator closes the web services, web apps, jobs and mails, changes and reloads the config, purges the logs, closes the editor sessions left open by other users, then reopens and traces the window.

#1. Close web services, web apps, jobs and mails, keep the MQL editor open;
maintenance set 0 1 1 1 0;
maintenance set_lock "MAIL" 1;
log write "Maintenance window opened" "OK" "change" "CHG-2026-118";
#2. Keep 90 days of logs instead of 50, apply it and purge;
config set "LOG" "RETENTION_DAYS" "90";
config reload;
log remove;
#3. Close the editor sessions left open by other users;
json load "rx_sess" (sessions);
for (-> "[i]" 0) (< [i] (json count "rx_sess" /)) (++ "[i]") {
	-> "[id]" (json select "rx_sess" (concat "/[" [i] "]/id"));
	if (and (equal (json select "rx_sess" (concat "/[" [i] "]/origin")) "MQL") (not (equal [id] (sid)))) {kill [id]};
};
reset exceeded sessions;
#4. Restart the scheduler, release the stack locks, reopen and trace;
job scheduler restart;
stack reset;
maintenance set 0 0 0 0 0;
maintenance set_lock "MAIL" 0;
log trace "Maintenance window closed";
json unload "rx_sess";
maintenance get

Morning check line from the system statistics

`metric system` returns a full text report. The script keeps only the runtime figures (Java, CPUs, JVM heap, RAM, root file system), leaves the identity fields out and builds one line for the morning check with a disk verdict.

#1. Full system statistics in one call;
-> "[sys]" (metric system);
#2. Keep only the runtime lines, no identity fields;
-> "[java]" (string regexp_extract [sys] "Java: v_([^\n]+)" 1);
-> "[os]" (string regexp_extract [sys] "nb_proc: ([0-9]+)" 1);
-> "[heap]" (string regexp_extract [sys] "Used jvm mem: ([0-9]+)/([0-9]+)/([0-9]+)M" 0);
-> "[ram]" (string regexp_extract [sys] "Used mem: ([0-9]+)/([0-9]+)M" 0);
-> "[root]" (string regexp_extract [sys] "\[/\] ([0-9]+)%" 1);
#3. One line for the morning check;
concat "MentDB " (version) " on Java " [java] " | " [os] " CPUs | " [heap] " | " [ram] " | root fs " [root] "% used" (if (> [root] 85) {" -> CLEAN THE DISK"} {" -> ok"})

MentDB 4.2.3 on Java 21.0.13, Eclipse Adoptium | 11 CPUs | Used jvm mem: 167/512/2048M | Used mem: 13109/18432M | root fs 68% used -> ok

Do the logs cover the metric window?

When an incident is investigated, the metrics (31 days) and the logs must cover the same period. The script reads the 31 days of the metrics, counts the business days and checks the log retention against the window.

#1. The 31 days kept by the metrics;
-> "[days]" (metric date);
-> "[first]" (string regexp_extract [days] "[0-9]{4}-[0-9]{2}-[0-9]{2}" 0);
-> "[last]" (string regexp_extract [days] "([0-9]{4}-[0-9]{2}-[0-9]{2})\]" 1);
#2. Working days and weekend days in the window;
-> "[business]" (date nb_business_days [first] [last]);
-> "[span]" (+ (date nb_day [first] [last]) 1);
#3. Do the logs cover the same window?;
-> "[ret]" (log retention day);
concat "metric window " [first] " -> " [last] " (" [span] " days, " [business] " business days, starts on a " (date dayname [first] "en") ") | log retention " [ret] " days: " (if (>= [ret] [span]) {"the logs cover the whole metric window"} {"WARNING the logs are purged before the metrics"})

metric window 2026-10-02 -> 2026-11-01 (31 days, 21 business days, starts on a Friday) | log retention 50 days: the logs cover the whole metric window

CMDB record with a hashed host id

Each server must be registered in the CMDB without exposing its hostname in reports. The script builds a stable host id (SHA-256 of hostname, arch and CPU count, 16 characters) and an inventory record in JSON.

#1. Stable id of the host for the CMDB, the hostname itself is not printed;
-> "[host_id]" (string left (string sha_256 (concat (os hostname) "|" (os arch) "|" (metric system nb processor))) 16);
#2. Inventory record;
json load "rx_p4h_ci" "{}";
json iobject "rx_p4h_ci" / "host_id" [host_id] STR;
json iobject "rx_p4h_ci" / "hostname_length" (string length (os hostname)) INT;
json iobject "rx_p4h_ci" / "platform" (concat (os type) " " (os arch)) STR;
json iobject "rx_p4h_ci" / "ram_gb" (math round (/ (metric total mem) 1024) 1) NUM;
json iobject "rx_p4h_ci" / "swap_gb" (math round (/ (metric total swap mem) 1024) 1) NUM;
json iobject "rx_p4h_ci" / "java" (metric java version) STR;
json iobject "rx_p4h_ci" / "mentdb" (version) STR;
json iobject "rx_p4h_ci" / "seen_at" (date to_iso (date systimestamp)) STR;
json doc "rx_p4h_ci"

{
  "java": "21.0.13",
  "hostname_length": 26,
  "host_id": "1581bc9d9e69bbd7",
  "platform": "Mac OS X aarch64",
  "ram_gb": 18.0,
  "swap_gb": 2.0,
  "mentdb": "4.2.3",
  "seen_at": "2026-11-01T18:30:36+04:00"
}

Trace a check in the log and find it back

An audit needs to prove that the memory check ran. A temporary check script with the activity log on writes its result with a client key and a unique reference; the script then finds the entry back with `log search` and deletes the check script.

#1. A check script with the activity log on: log write only writes from such a script;
script create exe "tuto.p4h.memcheck" true 1
	(param
		(var "[ref]" {true} "Reference of the run" is_null:false is_empty:false "run-1")
	;)
	"Memory check traced in the log"
{
	-> "[free_pct]" (math round (* 100 (/ (metric current available mem) (metric total mem))) 1);
	-> "[status]" (if (< [free_pct] 10) {"KO"} {"OK"});
	log write (concat "Memory check: " [free_pct] "% available") [status] "tuto_p4h_check" [ref];
	[status];
} "OK or KO";
#2. Run it with a unique reference;
-> "[ref]" (concat "run-" (date current_ms));
-> "[status]" (execute "tuto.p4h.memcheck.exe" "[ref]" [ref]);
#3. Find the entry back by its client key and value, like an auditor;
json load "rx_p4h_log" (string substring (log search (string lower [status]) "tuto.p4h.memcheck.exe" "tuto_p4h_check" [ref] null (concat (date sysdate) " 00:00:00") (concat (date sysdate) " 23:59:59") DESC 1 10) 39);
script delete "tuto.p4h.memcheck.exe";
concat "status " [status] " | entries found: " (json count "rx_p4h_log" /data) " | message: " (json select "rx_p4h_log" "/data/[0]/msg") " | script: " (json select "rx_p4h_log" "/data/[0]/script")

status OK | entries found: 1 | message: Memory check: 28.9% available | script: tuto.p4h.memcheck.exe

Peak or sustained CPU load?

One CPU reading says little. The script samples the system CPU 12 times every 500 ms, then compares the mean, the 95th percentile and the max to tell a sustained load from short peaks, with the share of the MentDB JVM.

#1. Sample the system CPU 12 times, every 500 ms;
json load "rx_p4h_cpu" "[]";
for (-> "[i]" 0) (< [i] 12) (++ "[i]") {
	json iarray "rx_p4h_cpu" / (metric current cpu system) NUM;
	wait 500;
};
-> "[s]" (json doc "rx_p4h_cpu");
#2. Peak or sustained load? mean, 95th percentile, max;
-> "[mean]" (math round (stat mean [s]) 1);
-> "[p95]" (math round (stat percentile [s] 95) 1);
-> "[max]" (stat max [s]);
-> "[jvm]" (metric current cpu jvm);
concat "cpu system over 6 s: mean " [mean] "%, p95 " [p95] "%, max " [max] "% | MentDB JVM share now " [jvm] "% | " (case (> [mean] 80) {"SUSTAINED load: add CPUs or move jobs"} (> [p95] 80) {"short peaks only"} {"normal"})

cpu system over 6 s: mean 8.1%, p95 11.9%, max 12% | MentDB JVM share now 0.1% | normal