Profiles / Platform & operations
SA

System administrator & DevOps

Health checks, metrics, logs, OS facts and sessions: monitor a MentDB server from MQL.

17 recipes on this page · open in the interactive view

Recipes
  1. Health report in JSON for the monitoring
  2. When will the disk be full?
  3. Server inventory and security checks
  4. Sessions and errors of the day
  5. Maintenance window: lock, change the config, reopen
  6. Morning check line from the system statistics
  7. Do the logs cover the metric window?
  8. CMDB record with a hashed host id
  9. Trace a check in the log and find it back
  10. Peak or sustained CPU load?
  11. Publish the counters of an application as a Prometheus /metrics page
  12. Read the disk space of a node exporter and draw the free space of each disk
  13. Keep the daily export of a configuration in a Git repository and see what changed
  14. Warn the team on Slack and Teams when disks are almost full
  15. Read an nginx access log: server errors, robots and slow requests
  16. Send the orders of the day to Grafana or Datadog as OpenTelemetry metrics
  17. Send the depth of the work queues to a StatsD or Datadog agent
executed on MentDB

Health report in JSON for the monitoring

Every minute the supervision tool pulls one JSON document: CPU of the system and of the JVM, heap usage, RAM really available (cache excluded), swap, and an OK/WARN status above 85 % CPU or 90 % RAM.

Pipeline
Script
admin
#1. Identity and CPU load;
json load "rx_health" "{}";
json iobject "rx_health" / "ts" (date systimestamp) STR;
json iobject "rx_health" / "mentdb" (concat (name) " v" (version)) STR;
json iobject "rx_health" / "cpu_sys" (metric current cpu system) NUM;
json iobject "rx_health" / "cpu_jvm" (metric current cpu jvm) NUM;
#2. JVM heap: used/committed/max in MB;
-> "[jvm]" (metric current mem jvm);
json iobject "rx_health" / "heap_used_pct" (math round (* 100 (/ (atom get [jvm] 1 "/") (atom get [jvm] 3 "/"))) 1) NUM;
#3. RAM really available, free RAM and swap;
-> "[total]" (metric total mem);
json iobject "rx_health" / "ram_used_pct" (math round (* 100 (/ (- [total] (metric current available mem)) [total])) 1) NUM;
json iobject "rx_health" / "ram_free_mb" (metric current free mem) NUM;
json iobject "rx_health" / "ram_used_with_cache_mb" (metric current used mem) NUM;
json iobject "rx_health" / "swap_used_mb" (metric current used swap mem) NUM;
json iobject "rx_health" / "swap_free_mb" (metric current free swap mem) NUM;
json iobject "rx_health" / "swap_total_mb" (metric total swap mem) NUM;
#4. Verdict: WARN above 85 percent CPU or 90 percent RAM;
json iobject "rx_health" / "status" (if (or (> (json select "rx_health" /cpu_sys) 85) (> (json select "rx_health" /ram_used_pct) 90)) {"WARN"} {"OK"}) STR;
json doc "rx_health"
Result
mentdb
{
  "ram_used_with_cache_mb": 18304.0,
  "cpu_sys": 11.0,
  "ram_free_mb": 127.0,
  "swap_total_mb": 2048.0,
  "ram_used_pct": 73.2,
  "heap_used_pct": 15.2,
  "swap_used_mb": 1149.0,
  "ts": "2026-11-01 16:27:23",
  "mentdb": "mona payet v4.2.3",
  "cpu_jvm": 0.5,
  "swap_free_mb": 898.0,
  "status": "OK"
}
Go further
  • Store each report with json iarray in a daily array and plot the CPU with plot line
  • Send it to the supervision every minute: a job add on a script that returns json doc
executed on MentDB

When will the disk be full?

The data disk grows by about 1.8 GB per day. The script reads the root file system from the metrics, keeps a 10 % safety margin and gives the number of days and the date when the margin is reached, with an alert below 30 days.

Pipeline
Script
admin
#1. Root file system from the metrics, sizes in MB;
json load "rx_disk" (metric file system roots);
-> "[mount]" (json select "rx_disk" "/[0]/absolutePath");
-> "[free]" (json select "rx_disk" "/[0]/freeSpace");
-> "[total]" (json select "rx_disk" "/[0]/totalSpace");
-> "[used_pct]" (math round (* 100 (/ (- [total] [free]) [total])) 1);
#2. Growth of 1.8 GB per day, keep a 10 percent safety margin;
-> "[usable]" (- [free] (* 0.10 [total]));
-> "[days]" (math floor (/ [usable] 1800));
#3. Date when the margin is reached;
-> "[limit]" (date add (date sysdate) "DAY" [days]);
json unload "rx_disk";
concat "{\"mount\": \"" [mount] "\", \"used_pct\": " [used_pct] ", \"free_gb\": " (math round (/ [free] 1024) 1) ", \"days_left\": " [days] ", \"margin_reached_on\": \"" [limit] "\", \"alert\": " (if (< [days] 30) {1} {0}) "}"
Result
mentdb
{"mount": "/", "used_pct": 68.5, "free_gb": 145.2, "days_left": 56, "margin_reached_on": "2026-12-27", "alert": 0}
Go further
  • Measure the real growth: save freeSpace every night and fit it with stat linear_regression
  • Several mounts: loop on json count of the array
executed on MentDB

Server inventory and security checks

Before an audit, the administrator needs the OS, CPUs, Java runtime, time zone, MentDB build and mode, and the days left on the TLS certificate. Two checks are added without printing the account: MentDB must not run as root and must be started from its folder.

Pipeline
Script
admin
#1. Operating system and Java runtime;
json load "rx_inv" "{}";
json iobject "rx_inv" / "os" (concat (os name) " " (os version) " " (os arch)) STR;
json iobject "rx_inv" / "os_type" (os type) STR;
json iobject "rx_inv" / "kernel" (concat (metric system name) " " (metric system version) " " (metric system architecture)) STR;
json iobject "rx_inv" / "cpus" (metric system nb processor) INT;
json iobject "rx_inv" / "java" (concat (metric java version) " " (metric java vendor)) STR;
json iobject "rx_inv" / "tz_lang" (concat (os user_timezone) " " (os user_lang)) STR;
#2. Security checks without printing the account: not root, started from the MentDB folder;
json iobject "rx_inv" / "runs_as_root" (if (or (string equals_ignore_case (os user_name) "root") (string contains (os user_home) "/root")) {1} {0}) INT;
json iobject "rx_inv" / "workdir_is_mentdb" (string contains (os user_dir) "MentDB") INT;
#3. MentDB build, mode and size;
json iobject "rx_inv" / "mentdb" (concat (version) " - " (ai firstname) " " (ai lastname)) STR;
json iobject "rx_inv" / "mode" (if (parameter exist "SERVER_MODE") {parameter get value "SERVER_MODE"} {"?"}) STR;
json iobject "rx_inv" / "mode_doc" (parameter get description "SERVER_MODE") STR;
json iobject "rx_inv" / "mode_private" (parameter is_private "SERVER_MODE") INT;
json iobject "rx_inv" / "mode_restore" (parameter generate update "SERVER_MODE") STR;
json iobject "rx_inv" / "mql_functions" (function count) INT;
json iobject "rx_inv" / "java_lines" (src count) INT;
#4. Days before the TLS certificate expires;
json load "rx_cert" (config cert);
json iobject "rx_inv" / "cert_days_left" (json select "rx_cert" "/[1]/days_left") INT;
json unload "rx_cert";
json doc "rx_inv"
Result
mentdb
{
  "workdir_is_mentdb": 1,
  "os": "Mac OS X 14.4 aarch64",
  "cpus": 11,
  "mode_restore": "parameter update \"SERVER_MODE\" \"DEV\";",
  "kernel": "Mac OS X 14.4 aarch64",
  "tz_lang": "Indian/Reunion fr",
  "mode_doc": "Mode of the server (DEV, TEST, PROD...), shown in the administration.",
  "mode_private": 0,
  "cert_days_left": 1819,
  "mql_functions": 1606,
  "mode": "DEV",
  "java": "21.0.13 Eclipse Adoptium",
  "os_type": "Mac OS X",
  "runs_as_root": 0,
  "mentdb": "4.2.3 - mona payet",
  "java_lines": 798979
}
Go further
  • Read the whole configuration with config show: secret values are masked
  • Alert when cert_days_left is below 30 and renew the certificate
executed on MentDB

Sessions and errors of the day

Morning check: how many sessions are open and against which capacity, how many come from the MQL editor, the oldest one, the web socket sessions, the exceeded sessions, and the scripts that ended in error in the log today.

Pipeline
Script
admin
#1. Open sessions: how many come from the MQL editor, the oldest one;
json load "rx_sess" (sessions);
-> "[n]" (json count "rx_sess" /);
-> "[mql]" 0;
-> "[oldest]" 0;
for (-> "[i]" 0) (< [i] [n]) (++ "[i]") {
	if (equal (json select "rx_sess" (concat "/[" [i] "]/origin")) "MQL") {++ "[mql]"};
	-> "[oldest]" (math max [oldest] (json select "rx_sess" (concat "/[" [i] "]/used")));
};
#2. Session capacity from the metrics, without the editor marker;
json load "rx_ms" (string substring (metric sessions) 39);
#3. Scripts in error in the log today;
json load "rx_ko" (string substring (log search "ko" null null null null (concat (date sysdate) " 00:00:00") (concat (date sysdate) " 23:59:59") DESC 1 50) 39);
json load "rx_ws" (@sessions);
concat "{\"me\": \"" (who) "\", \"my_commands\": " (cmdid) ", \"java_pid\": " (pid) ", \"open\": " (count sessions) ", \"capacity\": \"" (json select "rx_ms" "/data/[0]/open_session") "\", \"from_editor\": " [mql] ", \"websocket\": \"" (@count sessions) "/" (json count "rx_ws" /) "\"" ", \"oldest_s\": " [oldest] ", \"exceeded\": " (exceeded sessions) ", \"ws_exceeded\": " (@exceeded sessions) ", \"log_id\": " (log current id) ", \"errors_today\": " (json count "rx_ko" /data) ", \"log_retention_days\": " (log retention day) ", \"archive\": \"" (log archive path) " every " (log archive size) "\"}"
Result
mentdb
{"me": "admin", "my_commands": 5, "java_pid": 80950, "open": 2, "capacity": "2/100", "from_editor": 1, "websocket": "0/0", "oldest_s": 3760, "exceeded": 0, "ws_exceeded": 0, "log_id": 0, "errors_today": 0, "log_retention_days": 50, "archive": "archives/logs every 30"}
Go further
needs your systems

Maintenance window: lock, change the config, reopen

A change request asks to keep 90 days of logs. The administrator closes the web services, web apps, jobs and mails, changes and reloads the config, purges the logs, closes the editor sessions left open by other users, then reopens and traces the window.

Pipeline
Script
admin
#1. Close web services, web apps, jobs and mails, keep the MQL editor open;
maintenance set 0 1 1 1 0;
maintenance set_lock "MAIL" 1;
log write "Maintenance window opened" "OK" "change" "CHG-2026-118";
#2. Keep 90 days of logs instead of 50, apply it and purge;
config set "LOG" "RETENTION_DAYS" "90";
config reload;
log remove;
#3. Close the editor sessions left open by other users;
json load "rx_sess" (sessions);
for (-> "[i]" 0) (< [i] (json count "rx_sess" /)) (++ "[i]") {
	-> "[id]" (json select "rx_sess" (concat "/[" [i] "]/id"));
	if (and (equal (json select "rx_sess" (concat "/[" [i] "]/origin")) "MQL") (not (equal [id] (sid)))) {kill [id]};
};
reset exceeded sessions;
#4. Restart the scheduler, release the stack locks, reopen and trace;
job scheduler restart;
stack reset;
maintenance set 0 0 0 0 0;
maintenance set_lock "MAIL" 0;
log trace "Maintenance window closed";
json unload "rx_sess";
maintenance get
Result
This recipe uses your own connections (database, SFTP, SMTP...). Create them with the CM functions, then run it on your server.
Go further
executed on MentDB

Morning check line from the system statistics

`metric system` returns a full text report. The script keeps only the runtime figures (Java, CPUs, JVM heap, RAM, root file system), leaves the identity fields out and builds one line for the morning check with a disk verdict.

Pipeline
Script
admin
#1. Full system statistics in one call;
-> "[sys]" (metric system);
#2. Keep only the runtime lines, no identity fields;
-> "[java]" (string regexp_extract [sys] "Java: v_([^\n]+)" 1);
-> "[os]" (string regexp_extract [sys] "nb_proc: ([0-9]+)" 1);
-> "[heap]" (string regexp_extract [sys] "Used jvm mem: ([0-9]+)/([0-9]+)/([0-9]+)M" 0);
-> "[ram]" (string regexp_extract [sys] "Used mem: ([0-9]+)/([0-9]+)M" 0);
-> "[root]" (string regexp_extract [sys] "\[/\] ([0-9]+)%" 1);
#3. One line for the morning check;
concat "MentDB " (version) " on Java " [java] " | " [os] " CPUs | " [heap] " | " [ram] " | root fs " [root] "% used" (if (> [root] 85) {" -> CLEAN THE DISK"} {" -> ok"})
Result
mentdb
MentDB 4.2.3 on Java 21.0.13, Eclipse Adoptium | 11 CPUs | Used jvm mem: 167/512/2048M | Used mem: 13109/18432M | root fs 68% used -> ok
Go further
  • Send the line to the team chat from a job every morning.
  • Store the figures in a table to follow the heap and the disk over months.
  • Get each value directly with metric current mem jvm or metric file system roots when you need numbers.
executed on MentDB

Do the logs cover the metric window?

When an incident is investigated, the metrics (31 days) and the logs must cover the same period. The script reads the 31 days of the metrics, counts the business days and checks the log retention against the window.

Pipeline
Script
admin
#1. The 31 days kept by the metrics;
-> "[days]" (metric date);
-> "[first]" (string regexp_extract [days] "[0-9]{4}-[0-9]{2}-[0-9]{2}" 0);
-> "[last]" (string regexp_extract [days] "([0-9]{4}-[0-9]{2}-[0-9]{2})\]" 1);
#2. Working days and weekend days in the window;
-> "[business]" (date nb_business_days [first] [last]);
-> "[span]" (+ (date nb_day [first] [last]) 1);
#3. Do the logs cover the same window?;
-> "[ret]" (log retention day);
concat "metric window " [first] " -> " [last] " (" [span] " days, " [business] " business days, starts on a " (date dayname [first] "en") ") | log retention " [ret] " days: " (if (>= [ret] [span]) {"the logs cover the whole metric window"} {"WARNING the logs are purged before the metrics"})
Result
mentdb
metric window 2026-10-02 -> 2026-11-01 (31 days, 21 business days, starts on a Friday) | log retention 50 days: the logs cover the whole metric window
Go further
  • Raise the retention in the LOG section of the configuration when the logs are shorter than the metrics.
  • Archive the logs before the purge (log archive path).
  • Cross the busiest day of metric sessions with log search on that day.
executed on MentDB

CMDB record with a hashed host id

Each server must be registered in the CMDB without exposing its hostname in reports. The script builds a stable host id (SHA-256 of hostname, arch and CPU count, 16 characters) and an inventory record in JSON.

Pipeline
Script
admin
#1. Stable id of the host for the CMDB, the hostname itself is not printed;
-> "[host_id]" (string left (string sha_256 (concat (os hostname) "|" (os arch) "|" (metric system nb processor))) 16);
#2. Inventory record;
json load "rx_p4h_ci" "{}";
json iobject "rx_p4h_ci" / "host_id" [host_id] STR;
json iobject "rx_p4h_ci" / "hostname_length" (string length (os hostname)) INT;
json iobject "rx_p4h_ci" / "platform" (concat (os type) " " (os arch)) STR;
json iobject "rx_p4h_ci" / "ram_gb" (math round (/ (metric total mem) 1024) 1) NUM;
json iobject "rx_p4h_ci" / "swap_gb" (math round (/ (metric total swap mem) 1024) 1) NUM;
json iobject "rx_p4h_ci" / "java" (metric java version) STR;
json iobject "rx_p4h_ci" / "mentdb" (version) STR;
json iobject "rx_p4h_ci" / "seen_at" (date to_iso (date systimestamp)) STR;
json doc "rx_p4h_ci"
Result
mentdb
{
  "java": "21.0.13",
  "hostname_length": 26,
  "host_id": "1581bc9d9e69bbd7",
  "platform": "Mac OS X aarch64",
  "ram_gb": 18.0,
  "swap_gb": 2.0,
  "mentdb": "4.2.3",
  "seen_at": "2026-11-01T18:30:36+04:00"
}
Go further
  • Post the record to the CMDB API with rest https.
  • Compare the host id with yesterday's one to detect a cloned or moved server.
  • Add the certificate expiry from config cert.
executed on MentDB

Trace a check in the log and find it back

An audit needs to prove that the memory check ran. A temporary check script with the activity log on writes its result with a client key and a unique reference; the script then finds the entry back with `log search` and deletes the check script.

Pipeline
Script
admin
#1. A check script with the activity log on: log write only writes from such a script;
script create exe "tuto.p4h.memcheck" true 1
	(param
		(var "[ref]" {true} "Reference of the run" is_null:false is_empty:false "run-1")
	;)
	"Memory check traced in the log"
{
	-> "[free_pct]" (math round (* 100 (/ (metric current available mem) (metric total mem))) 1);
	-> "[status]" (if (< [free_pct] 10) {"KO"} {"OK"});
	log write (concat "Memory check: " [free_pct] "% available") [status] "tuto_p4h_check" [ref];
	[status];
} "OK or KO";
#2. Run it with a unique reference;
-> "[ref]" (concat "run-" (date current_ms));
-> "[status]" (execute "tuto.p4h.memcheck.exe" "[ref]" [ref]);
#3. Find the entry back by its client key and value, like an auditor;
json load "rx_p4h_log" (string substring (log search (string lower [status]) "tuto.p4h.memcheck.exe" "tuto_p4h_check" [ref] null (concat (date sysdate) " 00:00:00") (concat (date sysdate) " 23:59:59") DESC 1 10) 39);
script delete "tuto.p4h.memcheck.exe";
concat "status " [status] " | entries found: " (json count "rx_p4h_log" /data) " | message: " (json select "rx_p4h_log" "/data/[0]/msg") " | script: " (json select "rx_p4h_log" "/data/[0]/script")
Result
mentdb
status OK | entries found: 1 | message: Memory check: 28.9% available | script: tuto.p4h.memcheck.exe
Go further
  • log write writes only when the script was created with activateLog true: in the editor it returns 1 but writes nothing.
  • Schedule the check with job add and search the KO entries every morning.
  • Use log show_time with the pid to measure the duration of a run.
executed on MentDB

Peak or sustained CPU load?

One CPU reading says little. The script samples the system CPU 12 times every 500 ms, then compares the mean, the 95th percentile and the max to tell a sustained load from short peaks, with the share of the MentDB JVM.

Pipeline
Script
admin
#1. Sample the system CPU 12 times, every 500 ms;
json load "rx_p4h_cpu" "[]";
for (-> "[i]" 0) (< [i] 12) (++ "[i]") {
	json iarray "rx_p4h_cpu" / (metric current cpu system) NUM;
	wait 500;
};
-> "[s]" (json doc "rx_p4h_cpu");
#2. Peak or sustained load? mean, 95th percentile, max;
-> "[mean]" (math round (stat mean [s]) 1);
-> "[p95]" (math round (stat percentile [s] 95) 1);
-> "[max]" (stat max [s]);
-> "[jvm]" (metric current cpu jvm);
concat "cpu system over 6 s: mean " [mean] "%, p95 " [p95] "%, max " [max] "% | MentDB JVM share now " [jvm] "% | " (case (> [mean] 80) {"SUSTAINED load: add CPUs or move jobs"} (> [p95] 80) {"short peaks only"} {"normal"})
Result
mentdb
cpu system over 6 s: mean 8.1%, p95 11.9%, max 12% | MentDB JVM share now 0.1% | normal
Go further
executed on MentDB

Publish the counters of an application as a Prometheus /metrics page

Prometheus, Grafana Agent and most monitoring tools read the text format of a /metrics page. The counters are kept as JSON (from SQL or a script), prometheus format writes the page, and a web service returns it with the content type text/plain; version=0.0.4. The page is read back here to check it, as Prometheus would.

Pipeline
Script
admin
-> "[metrics]" "[{\"name\": \"orders_received_total\", \"type\": \"counter\", \"help\": \"Orders received since the start.\", \"samples\": [{\"labels\": {\"channel\": \"web\"}, \"value\": 1027}, {\"labels\": {\"channel\": \"shop\"}, \"value\": 412}]}, {\"name\": \"orders_failed_total\", \"type\": \"counter\", \"help\": \"Orders refused by the payment.\", \"samples\": [{\"labels\": {\"channel\": \"web\"}, \"value\": 9}, {\"labels\": {\"channel\": \"shop\"}, \"value\": 2}]}, {\"name\": \"queue_waiting\", \"type\": \"gauge\", \"help\": \"Messages waiting in the queue.\", \"value\": 14}, {\"name\": \"order_seconds\", \"type\": \"histogram\", \"help\": \"Time to process an order.\", \"samples\": [{\"buckets\": {\"0.1\": 820, \"0.5\": 1300, \"2\": 1431, \"+Inf\": 1439}, \"sum\": 402.7}]}]";
-> "[page]" (prometheus format [metrics]);
json load "rec_pm" (prometheus parse [page] "{\"flat\": true}");
-> "[lines]" (json count "rec_pm" "/");
-> "[web]" (json select "rec_pm" "/[0]/value");
-> "[failed]" (json select "rec_pm" "/[2]/value");
json unload "rec_pm";
-> "[om]" (prometheus format [metrics] "{\"format\": \"openmetrics\"}");
-> "[res]" (concat [lines] " samples | web failure rate " (math round (* 100 (/ [failed] [web])) 2) " % | OpenMetrics ends with " (string right [om] 6) (string char 10) [page]);
[res]
Result
mentdb
11 samples | web failure rate 0.88 % | OpenMetrics ends with # EOF

# HELP orders_received_total Orders received since the start.
# TYPE orders_received_total counter
orders_received_total{channel="web"} 1027
orders_received_total{channel="shop"} 412
# HELP orders_failed_total Orders refused by the payment.
# TYPE orders_failed_total counter
orders_failed_total{channel="web"} 9
orders_failed_total{channel="shop"} 2
# HELP queue_waiting Messages waiting in the queue.
# TYPE queue_waiting gauge
queue_waiting 14
# HELP order_seconds Time to process an order.
# TYPE order_seconds histogram
order_seconds_bucket{le="0.1"} 820
order_seconds_bucket{le="0.5"} 1300
order_seconds_bucket{le="2.0"} 1431
order_seconds_bucket{le="+Inf"} 1439
order_seconds_count 1439
order_seconds_sum 402.7
Go further
  • The histogram gives the 0.1, 0.5 and 2 second buckets: Prometheus computes the quantiles from them (histogram_quantile).
  • Use the openmetrics format for a scraper that asks for application/openmetrics-text; it ends with # EOF.
executed on MentDB

Read the disk space of a node exporter and draw the free space of each disk

A node exporter (or any Prometheus exporter) gives the state of a machine as text. Here the text was saved from a scrape, so it runs without the exporter; on site, rest request on http://host:9100/metrics gives the same text. prometheus parse turns it into JSON, the free share of each disk is computed and drawn as horizontal bars.

Pipeline
Script
admin
-> "[scrape]" "# HELP node_filesystem_avail_bytes Free space.\n# TYPE node_filesystem_avail_bytes gauge\nnode_filesystem_avail_bytes{mountpoint=\"/\"} 2100000000 1762585200000\nnode_filesystem_avail_bytes{mountpoint=\"/data\"} 45000000000 1762585200000\n# HELP node_filesystem_size_bytes Size.\n# TYPE node_filesystem_size_bytes gauge\nnode_filesystem_size_bytes{mountpoint=\"/\"} 20000000000 1762585200000\nnode_filesystem_size_bytes{mountpoint=\"/data\"} 50000000000 1762585200000";
json load "rec_ne" (prometheus parse [scrape]);
-> "[root]" (math round (* 100 (/ (json select "rec_ne" "/[0]/samples/[0]/value") (json select "rec_ne" "/[1]/samples/[0]/value"))) 1);
-> "[data]" (math round (* 100 (/ (json select "rec_ne" "/[0]/samples/[1]/value") (json select "rec_ne" "/[1]/samples/[1]/value"))) 1);
json unload "rec_ne";
-> "[chart]" (concat "{\"type\": \"bar\", \"horizontal\": true, \"title\": \"Free disk space\", \"categories\": [\"/\", \"/data\"], \"values\": [" [root] ", " [data] "], \"number_format\": {\"suffix\": \" %\"}, \"data_labels\": true, \"y_min\": 0, \"y_max\": 100}");
-> "[svg]" (chart svg [chart] "{\"width\": 400, \"height\": 160}");
-> "[res]" (concat "free on /: " [root] " % | free on /data: " [data] " % | SVG of " (string length [svg]) " characters, starts with " (string left [svg] 40));
[res]
Result
mentdb
free on /: 10.5 % | free on /data: 90 % | SVG of 1472 characters, starts with <svg xmlns="http://www.w3.org/2000/svg"
Go further
  • Only 10.5 % is free on /: a scheduled script can send this chart by mail when a disk goes under a threshold.
  • prometheus parse with flat gives one row per line, ready for sql insert or a table of all the disks.
executed on MentDB

Keep the daily export of a configuration in a Git repository and see what changed

Each night the parameters of an application are exported to a CSV file. Committed in a Git repository, every change gets a date, an author and a message, and git diff tells exactly which line changed between two nights; a tag marks the version that went to production. JGit is inside MentDB: no git program is needed, no hook of the repository is ever run, and the same commit (same content, author and date) gives the same id as the git command. With a remote (git clone, git push with token), the history can be kept on a Git server.

Pipeline
Script
admin
if (equal (file exist "tmp/rec_ops_git") "1") {file delete "tmp/rec_ops_git";};
if (equal (file exist "tmp/rec_ops_git") "0") {file mkdir "tmp/rec_ops_git";};
-> "[repo]" "tmp/rec_ops_git/config";
git init [repo];
csv write "tmp/rec_ops_git/config/settings.csv" "[{\"key\": \"smtp.host\", \"value\": \"mail.example.com\"}, {\"key\": \"backup.hour\", \"value\": \"02:00\"}]";
git add [repo] "settings.csv";
git commit [repo] "Export of 2026-11-09" "{\"author_name\": \"Ops robot\", \"author_email\": \"ops@example.com\", \"date\": \"2026-11-09T02:00:00Z\"}";
csv write "tmp/rec_ops_git/config/settings.csv" "[{\"key\": \"smtp.host\", \"value\": \"mail2.example.com\"}, {\"key\": \"backup.hour\", \"value\": \"02:00\"}, {\"key\": \"backup.keep_days\", \"value\": \"30\"}]";
json load "rg_st" (git status [repo]);
git add [repo] ".";
git commit [repo] "Export of 2026-11-10" "{\"author_name\": \"Ops robot\", \"author_email\": \"ops@example.com\", \"date\": \"2026-11-10T02:00:00Z\"}";
-> "[diff]" (git diff [repo] "{\"from\": \"HEAD~1\", \"to\": \"HEAD\", \"context\": 0}");
git tag [repo] "prod-2026-11-10";
json load "rg_log" (git log [repo] "{\"path\": \"settings.csv\"}");
-> "[res]" (concat "modified before the commit: " (json select "rg_st" "/modified/[0]") " | commits: " (json select "rg_log" "/[1]/message") ", then " (json select "rg_log" "/[0]/message") " (" (json select "rg_log" "/[0]/short_id") ") | diff: " (string char 10) [diff]);
json unload "rg_st"; json unload "rg_log";
if (equal (file exist "tmp/rec_ops_git") "1") {file delete "tmp/rec_ops_git";};
[res]
Result
mentdb
modified before the commit: settings.csv | commits: Export of 2026-11-09, then Export of 2026-11-10 (f06d0a4) | diff: 
diff --git a/settings.csv b/settings.csv
index bf5ba97..b18810d 100644
--- a/settings.csv
+++ b/settings.csv
@@ -2 +2 @@
-smtp.host,mail.example.com
+smtp.host,mail2.example.com
@@ -3,0 +4 @@
+backup.keep_days,30
Go further
  • git diff without parameters shows what is not added yet; {"cached": true} what is added and not committed.
  • A token is never written in a URL: git push "repo" "{\"token\": \"git_token\"}" reads it from a secret (secret set).
executed on MentDB

Warn the team on Slack and Teams when disks are almost full

A scheduled script reads the disk usage of the servers (here JSON rows; in real life sql to json or a metrics API) and keeps the servers over 90 %. One message lists them as fields, with a button to the dashboard. The same parameters build a Slack Block Kit message and a Teams Adaptive Card. With dry_run the functions send nothing and need no secret: they give the exact JSON that would be sent and its size, so the message can be checked here. In production, replace dry_run by webhook_url, the name of a secret (secret set) of the server that holds the webhook URL.

Pipeline
Script
admin
-> "[disks]" "[{\"host\": \"db-01\", \"used_pct\": 93, \"free_gb\": 14}, {\"host\": \"web-01\", \"used_pct\": 61, \"free_gb\": 80}, {\"host\": \"files-02\", \"used_pct\": 97, \"free_gb\": 3}]";
json load "rx_d" [disks];
-> "[f]" "";
-> "[k]" 0;
for (-> "[i]" 0) (< [i] (json count "rx_d" "/")) (++ "[i]") {
  -> "[u]" (json select "rx_d" (concat "/[" [i] "]/used_pct"));
  if (> [u] 90) {
    if (> [k] 0) {-> "[f]" (concat [f] ", ");};
    -> "[f]" (concat [f] "\"" (json select "rx_d" (concat "/[" [i] "]/host")) "\": \"" [u] " % (" (json select "rx_d" (concat "/[" [i] "]/free_gb")) " GB free)\"");
    -> "[k]" (+ [k] 1);
  };
};
json unload "rx_d";
-> "[params]" (concat "{\"title\": \"Disks almost full\", \"fields\": {" [f] "}, \"link_url\": \"https://grafana.example.org/d/disks\", \"link_text\": \"Open the dashboard\", \"dry_run\": true}");
-> "[text]" (concat [k] " servers are over 90 %");
json load "rx_s" (notify slack [text] [params]);
json load "rx_t" (notify teams [text] [params]);
-> "[res]" (concat "alerts: " [k] " | slack blocks: " (json count "rx_s" "/payload/blocks") ", first " (json select "rx_s" "/payload/blocks/[0]/type") ", fields " (json count "rx_s" "/payload/blocks/[2]/fields") ", " (json select "rx_s" "/bytes") " bytes | teams facts: " (json count "rx_t" "/payload/attachments/[0]/content/body/[2]/facts") ", " (json select "rx_t" "/bytes") " bytes of 28672");
json unload "rx_s"; json unload "rx_t";
[res]
Result
mentdb
alerts: 2 | slack blocks: 4, first header, fields 2, 575 bytes | teams facts: 2, 674 bytes of 28672
Go further
  • Put the webhook URL in a secret (secret set) (for example SLACK_WEBHOOK_URL) and give {"webhook_url": "slack_webhook_url"}: the URL holds a secret, so it is refused in clear and never appears in a result or an error.
  • Teams: create the webhook with the Workflows app of Teams. The old Office 365 connector URLs (webhook.office.com) stopped working in May 2026 and are refused with an explanation.
  • A rate limit (HTTP 429) is an error that says how long to wait: in a loop over many alerts, catch it and stop, rather than sending again at once.
executed on MentDB

Read an nginx access log: server errors, robots and slow requests

The access log of a web server says who called what, the answer and the time it took. log parse detects the nginx format, gives typed fields (status and request_time are numbers, the date is ISO 8601 with its offset) and reports the lines it cannot read instead of failing. With useragent true, each row also says whether the visitor is a robot and which browser and device it uses. Here the log is a text; in production, log parse_file reads the log file itself as a stream, also a rotated .gz file.

Pipeline
Script
admin
-> "[nl]" (string char 10);
-> "[log]" (concat "203.0.113.9 - - [03/Nov/2026:10:00:00 +0100] \"GET / HTTP/2.0\" 200 5120 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36\" \"-\" rt=0.012" [nl] "66.249.66.1 - - [03/Nov/2026:10:00:02 +0100] \"GET /robots.txt HTTP/1.1\" 200 68 \"-\" \"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\" \"-\" rt=0.001" [nl] "198.51.100.4 - - [03/Nov/2026:10:00:05 +0100] \"POST /api/orders HTTP/2.0\" 502 157 \"https://shop.example.com/cart\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 17_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Mobile/15E148 Safari/604.1\" \"-\" rt=2.481" [nl] "203.0.113.9 - - [03/Nov/2026:10:00:09 +0100] \"GET /api/orders/42 HTTP/2.0\" 404 153 \"-\" \"curl/8.4.0\" \"-\" rt=0.004" [nl] "\x16\x03\x01 garbage from a scanner" [nl] "198.51.100.4 - - [03/Nov/2026:10:01:00 +0100] \"POST /api/orders HTTP/2.0\" 201 310 \"https://shop.example.com/cart\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 17_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Mobile/15E148 Safari/604.1\" \"-\" rt=1.210");
json load "rw_l" (log parse [log] "{\"useragent\": true, \"fields\": [\"timestamp\", \"client\", \"method\", \"path\", \"status\", \"request_time\", \"ua\"]}");
-> "[err]" 0; -> "[bots]" 0; -> "[slow]" "";
for (-> "[i]" 0) (< [i] (json count "rw_l" "/rows")) (++ "[i]") {
  if (> (json select "rw_l" (concat "/rows/[" [i] "]/status")) 499) {-> "[err]" (+ [err] 1);};
  if (equal (json select "rw_l" (concat "/rows/[" [i] "]/ua/is_bot")) "true") {-> "[bots]" (+ [bots] 1);};
  if (> (json select "rw_l" (concat "/rows/[" [i] "]/request_time")) 1) {-> "[slow]" (concat [slow] " " (json select "rw_l" (concat "/rows/[" [i] "]/path")) " (" (json select "rw_l" (concat "/rows/[" [i] "]/request_time")) " s, " (json select "rw_l" (concat "/rows/[" [i] "]/ua/os")) ")");};
};
-> "[res]" (concat "format " (json select "rw_l" "/format") " | rows " (json select "rw_l" "/parsed") ", unreadable " (json select "rw_l" "/unparsed") " (line " (json select "rw_l" "/unparsed_lines/[0]/line") ") | 5xx: " [err] " | robots: " [bots] " | slow:" [slow] " | first at " (json select "rw_l" "/rows/[0]/timestamp"));
json unload "rw_l";
[res]
Result
mentdb
format nginx | rows 5, unreadable 1 (line 5) | 5xx: 1 | robots: 2 | slow: /api/orders (2.481 s, iOS) /api/orders (1.21 s, iOS) | first at 2026-11-03T10:00:00+01:00
Go further
  • robots and HTTP libraries (curl, python-requests) are both bots: filter on ua/bot_name to keep only the search engines.
  • For an Apache log or an IIS log the same script works: the format is detected. A format of your own is read with a grok pattern, see the next recipe.
  • rdap ip on the address of an unknown client gives the owner and the country of its network (it asks the registries over the Internet).
executed on MentDB

Send the orders of the day to Grafana or Datadog as OpenTelemetry metrics

A scheduled script reads the orders (here JSON rows; in real life sql to json) and publishes two business metrics next to the technical ones: a counter of orders and a histogram of the amounts, built from the raw amounts (the buckets, the sum, the minimum and the maximum are computed). otel metrics turns them into an OTLP request in JSON, the format that every OpenTelemetry collector, Grafana Alloy, Datadog and the other backends read. With dry_run it sends nothing and gives the exact request, so the result can be checked here; in production give the endpoint of the collector, usually http://127.0.0.1:4318.

Pipeline
Script
admin
-> "[orders]" "[{\"shop\": \"Paris\", \"amount\": 120.5}, {\"shop\": \"Lyon\", \"amount\": 42}, {\"shop\": \"Paris\", \"amount\": 310}, {\"shop\": \"Lyon\", \"amount\": 75.25}, {\"shop\": \"Paris\", \"amount\": 18}]";
json load "rx_o" [orders];
-> "[n]" (json count "rx_o" "/");
-> "[vals]" "";
for (-> "[i]" 0) (< [i] [n]) (++ "[i]") {
  if (> [i] 0) {-> "[vals]" (concat [vals] ", ");};
  -> "[vals]" (concat [vals] (json select "rx_o" (concat "/[" [i] "]/amount")));
};
json unload "rx_o";
-> "[metrics]" (concat "[{\"name\": \"shop.orders\", \"type\": \"counter\", \"unit\": \"{order}\", \"value\": " [n] ", \"start_time\": \"2026-11-11T00:00:00Z\", \"time\": \"2026-11-11T18:00:00Z\"}, {\"name\": \"shop.order.amount\", \"type\": \"histogram\", \"unit\": \"EUR\", \"bounds\": [50, 100, 250], \"values\": [" [vals] "], \"time\": \"2026-11-11T18:00:00Z\"}]");
json load "rx_m" (otel metrics "" [metrics] "{\"dry_run\": true, \"resource\": {\"service.name\": \"shop-backend\"}}");
-> "[h]" "/payload/resourceMetrics/[0]/scopeMetrics/[0]/metrics/[1]/histogram/dataPoints/[0]";
-> "[res]" (concat "data points: " (json select "rx_m" "/items") " | orders: " (json select "rx_m" "/payload/resourceMetrics/[0]/scopeMetrics/[0]/metrics/[0]/sum/dataPoints/[0]/asInt") " | amounts up to 50, 100, 250 and above: " (json select "rx_m" (concat [h] "/bucketCounts/[0]")) " " (json select "rx_m" (concat [h] "/bucketCounts/[1]")) " " (json select "rx_m" (concat [h] "/bucketCounts/[2]")) " " (json select "rx_m" (concat [h] "/bucketCounts/[3]")) " | sum " (json select "rx_m" (concat [h] "/sum")) ", max " (json select "rx_m" (concat [h] "/max")) " EUR");
json unload "rx_m";
[res]
Result
mentdb
data points: 2 | orders: 5 | amounts up to 50, 100, 250 and above: 2 1 1 1 | sum 565.75, max 310 EUR
Go further
  • Production: replace "" by the endpoint (http://127.0.0.1:4318 for a local collector) and remove dry_run. For a SaaS backend, put its headers in a secret (secret set) (for example GRAFANA_OTLP_HEADERS=Authorization=Basic%20...) and give {"secret_headers": "grafana_otlp_headers"}: the key never appears in a script or a result.
  • A counter is cumulative: send the total since start_time (here midnight) at each run, and the backend computes the rate. For the change since the last run, use "temporality": "delta".
  • StatsD agents (Datadog, Telegraf) take the same measures with statsd send.
executed on MentDB

Send the depth of the work queues to a StatsD or Datadog agent

Each minute a script reads the depth of the work queues and the age of their oldest message, and sends them as gauges tagged with the name of the queue to the local agent (the Datadog agent, Telegraf or the Prometheus statsd_exporter listen on UDP port 8125). statsd send packs the lines in datagrams that never exceed max_packet bytes. With dry_run nothing is sent and the datagrams are given, so the lines can be checked here.

Pipeline
Script
admin
-> "[queues]" "[{\"queue\": \"orders\", \"depth\": 17, \"oldest_s\": 42.5}, {\"queue\": \"invoices\", \"depth\": 0, \"oldest_s\": 0}, {\"queue\": \"emails\", \"depth\": 250, \"oldest_s\": 610}]";
json load "rx_q" [queues];
-> "[m]" "";
for (-> "[i]" 0) (< [i] (json count "rx_q" "/")) (++ "[i]") {
  -> "[name]" (json select "rx_q" (concat "/[" [i] "]/queue"));
  if (> [i] 0) {-> "[m]" (concat [m] ", ");};
  -> "[m]" (concat [m] "{\"name\": \"queue.depth\", \"type\": \"gauge\", \"value\": " (json select "rx_q" (concat "/[" [i] "]/depth")) ", \"tags\": {\"queue\": \"" [name] "\"}}, {\"name\": \"queue.oldest_seconds\", \"type\": \"gauge\", \"value\": " (json select "rx_q" (concat "/[" [i] "]/oldest_s")) ", \"tags\": {\"queue\": \"" [name] "\"}}");
};
json unload "rx_q";
json load "rx_s" (statsd send "127.0.0.1" "8125" (concat "[" [m] "]") "{\"flavor\": \"dogstatsd\", \"prefix\": \"mentdb.\", \"tags\": {\"env\": \"prod\"}, \"max_packet\": 128, \"dry_run\": true}");
-> "[res]" (concat "lines: " (json select "rx_s" "/lines") " | datagrams of 128 bytes at most: " (json select "rx_s" "/packets") " | first line: " (string substr (json select "rx_s" "/payloads/[0]") 0 46));
json unload "rx_s";
[res]
Result
mentdb
lines: 6 | datagrams of 128 bytes at most: 3 | first line: mentdb.queue.depth:17|g|#queue:orders,env:prod
Go further
  • Production: remove dry_run. UDP has no answer, so a stopped agent is not an error: watch the agent itself.
  • Without Datadog, use the default flavor etsy (no tags): put the queue in the name instead, for example queue.orders.depth.
  • For a collector that speaks OpenTelemetry, send the same gauges with otel metrics and the attribute queue.

Also in Platform & operations

AA
Automation architect
Stack with priorities and limits, cron jobs, file watchers, retries: run and watch the background work.
11 recipes
AD
Application developer
Web apps, SCRUD generation, virtual hosts, users and groups, script versioning.
10 recipes
CH
Cluster & high-availability engineer
Cluster nodes, signals, failover and replication of scripts between servers.
10 recipes