MentDB WeakMQL Recipes
← innov-ai.com All profiles Portal
Profiles / Engineering & data
DS

Data scientist

Regression, correlation, feature scaling and drift tests.

10 recipes on this page · open in the interactive view

Recipes
  1. Sales forecast from ad spend
  2. Log transform and standardization
  3. Data drift between training and production
  4. Histogram of a feature with its normal fit
  5. Price of flats by multiple linear regression
  6. Activation functions of a neural network, side by side
  7. Least squares by QR decomposition, then ridge regression
  8. Glorot uniform initialisation of a dense layer
  9. Regression through the origin: heating energy of a building
  10. Multivariate Gaussian log-likelihood with an LU decomposition
executed on MentDB

Sales forecast from ad spend

Weekly ad spend and sales are available for ten weeks. The script measures the correlation, fits a line and forecasts the sales for a new budget.

Pipeline
01 · correlatePearson and Spearmanstat correlation→02 · fitLeast squares linestat fit→03 · forecastPredict at 85 kEURmath eval
Script
admin
#1. Weekly ad spend in kEUR and sales in kEUR;
-> "[ad]" "[12, 18, 25, 31, 40, 46, 52, 60, 67, 75]";
-> "[sales]" "[210, 245, 290, 318, 372, 398, 441, 470, 512, 548]";
#2. Strength of the link: Pearson and Spearman;
-> "[r]" (stat correlation [ad] [sales]);
json load "rx_ds_sp" (stat spearman [ad] [sales]);
#3. Least squares line and forecast for a budget of 85 kEUR;
json load "rx_ds_fit" (stat fit [ad] [sales] "linear");
-> "[pred]" (math eval (json select "rx_ds_fit" /expression) "{\"x\": 85}");
concat "r = " (math round [r] 4) ", rho = " (math round (json select "rx_ds_sp" /rho) 3) ", sales = " (math round (json select "rx_ds_fit" /a) 1) " + " (math round (json select "rx_ds_fit" /b) 3) " * ad, R2 = " (math round (json select "rx_ds_fit" /r2) 4) ", forecast at 85 kEUR: " (math round [pred] 0) " kEUR"
Result
mentdb
r = 0.9989, rho = 1, sales = 151.5 + 5.374 * ad, R2 = 0.9978, forecast at 85 kEUR: 608 kEUR
Go further
  • Get the confidence interval of the slope with pa rl_slope_confidence_interval
  • Try a log model with stat fit "log" for diminishing returns
  • Fit several features with pa rm_load (multiple regression)
executed on MentDB

Log transform and standardization

Incomes are strongly right-skewed. The script applies a log transform, standardizes the values with z-scores and checks the skewness and the histogram.

Pipeline
01 · transformln of each valuemath calc→02 · scalez-scoresstat zscore→03 · shapeSkewness before and afterstat skewness→04 · bins4 classesstat histogram
Script
admin
#1. Customer incomes in kEUR, right-skewed;
-> "[inc]" "[18, 22, 25, 27, 29, 31, 34, 38, 42, 47, 55, 68, 85, 120, 190]";
json load "rx_ds_inc" [inc];
#2. Log transform of each value;
-> "[lg]" "["; -> "[sep]" "";
json parse_array "rx_ds_inc" "/" "[v]" {
  -> "[lg]" (concat [lg] [sep] (math calc "ln([v])"));
  -> "[sep]" ",";
};
-> "[lg]" (concat [lg] "]");
#3. Standardize and check the shape;
-> "[z]" (stat zscore [lg]);
concat "skewness raw = " (math round (stat skewness [inc]) 2) ", after log = " (math round (stat skewness [lg]) 2) ", z-scores from " (math round (stat min [z]) 2) " to " (math round (stat max [z]) 2) ", histogram: " (stat histogram [z] 4)
Result
mentdb
skewness raw = 2.13, after log = 0.89, z-scores from -1.36 to 2.23, histogram: [
  {"from": -1.3589523213190358, "to": -0.4625482543642111, "count": 6},
  {"from": -0.4625482543642111, "to": 0.43385581259061357, "count": 5},
  {"from": 0.43385581259061357, "to": 1.3302598795454381, "count": 2},
  {"from": 1.3302598795454381, "to": 2.226663946500263, "count": 2}
]
Go further
executed on MentDB

Data drift between training and production

A model feature is compared between the training set and last week production data. The script runs a two-sample Kolmogorov-Smirnov test and measures the shift in standard deviations.

Pipeline
01 · testTwo-sample KS teststat ks_2→02 · shiftMean shift in sdstat mean→03 · decideRetrain alarmif
Script
admin
#1. A feature in the training set and in last week production data;
-> "[train]" "[0.42, 0.51, 0.38, 0.47, 0.55, 0.44, 0.49, 0.40, 0.53, 0.46, 0.50, 0.43]";
-> "[prod]" "[0.58, 0.62, 0.49, 0.66, 0.71, 0.55, 0.60, 0.68, 0.57, 0.63, 0.59, 0.65]";
#2. Two-sample Kolmogorov-Smirnov test of the distributions;
json load "rx_ds_ks" (stat ks_2 [train] [prod]);
#3. Size of the shift in standard deviations of the training data;
-> "[shift]" (math calc (concat "(" (stat mean [prod]) "-" (stat mean [train]) ")/" (stat stddev [train])));
-> "[p]" (json select "rx_ds_ks" /p_value);
concat "KS D = " (math round (json select "rx_ds_ks" /d) 3) ", p = " (math round_sig [p] 3) ", mean shift = " (math round [shift] 2) " sd -> " (if (< [p] 0.01) {"data drift: retrain the model"} {"no drift"})
Result
mentdb
KS D = 0.833, p = 0.000204, mean shift = 2.76 sd -> data drift: retrain the model
Go further
executed on MentDB

Histogram of a feature with its normal fit

Before modelling, check the shape of a feature: the script draws the histogram of 1000 session times and overlays the normal density with the same mean and standard deviation, scaled to the counts of each class.

Pipeline
01 · dataSimulated featurestat dist_random→02 · widthClass widthstat max→03 · histHistogramplot histogram→04 · overlayNormal density on topplot add
Script
admin
#1. A feature of 1000 users: session time in min, simulated normal with mean 50 and sd 8;
-> "[x]" (stat dist_random "normal" "[50, 8]" 1000);
-> "[m]" (stat mean [x]); -> "[s]" (stat stddev [x]);
#2. Class width of a 25 bin histogram;
-> "[w]" (math calc (concat "(" (stat max [x]) "-" (stat min [x]) ")/25"));
#3. Histogram, then overlay the normal density with the same mean and sd, scaled to counts;
plot histogram [x] 25;
plot add (plot function "1000*[w]*exp(-(x-[m])^2/(2*[s]^2))/([s]*sqrt(2*pi))" "x" 20 80)
Result
mentdb
The chart is shown in the Plot tab of the editor.
Go further
  • Test the normality with stat shapiro before using a z-score.
  • A skewed feature (lognormal) shows the gap at once; try stat dist_random "lognormal".
executed on MentDB

Price of flats by multiple linear regression

Ten flats were sold with their surface, number of rooms and age. Fit price = b0 + b1 surface + b2 rooms + b3 age by ordinary least squares, compute the t statistic of each coefficient, the adjusted R2 and the residual standard error, and estimate the price of a new flat.

Pipeline
01 · fitOLS on three featurespa rm load_from_json→02 · coefficientsb and their standard errorspa rm estimate_regression_parameters_standard_errors→03 · qualityAdjusted R2pa rm calculate_adjusted_r_squared→04 · predictPrice of a new flatpa rm predict
Script
admin
#1. Sold flats: surface m2, rooms, age in years, and price in kEUR;
-> "[X]" "[[45, 2, 30], [62, 3, 12], [80, 4, 5], [38, 1, 40], [95, 4, 2], [70, 3, 20], [55, 2, 8], [110, 5, 15], [66, 3, 35], [84, 4, 10]]";
-> "[y]" "[152, 228, 310, 118, 372, 241, 205, 398, 205, 315]";
#2. Ordinary least squares: price = b0 + b1 surface + b2 rooms + b3 age;
pa rm load_from_json "rx_flats" [X] [y];
json load "rx_b" (pa rm estimate_regression_parameters "rx_flats");
json load "rx_se" (pa rm estimate_regression_parameters_standard_errors "rx_flats");
#3. t statistics of the coefficients;
-> "[out]" "";
json load "rx_lbl" "[\"intercept\", \"surface\", \"rooms\", \"age\"]";
for (-> "[k]" 0) (< [k] 4) (++ "[k]") {
  -> "[b]" (json select "rx_b" (concat "/[" [k] "]"));
  -> "[out]" (concat [out] (json select "rx_lbl" (concat "/[" [k] "]")) " " (math round [b] 3) " (t = " (math round (/ [b] (json select "rx_se" (concat "/[" [k] "]"))) 2) ") | ");
};
#4. Fit quality and price estimate of a 75 m2 flat, 3 rooms, 10 years old;
-> "[r2a]" (pa rm calculate_adjusted_r_squared "rx_flats");
-> "[s]" (pa rm estimate_regression_standard_error "rx_flats");
-> "[p]" (pa rm predict "rx_flats" "[75, 3, 10]");
pa rm close "rx_flats";
concat [out] "adjusted R2 = " (math round [r2a] 4) ", residual std error = " (math round [s] 1) " kEUR | 75 m2, 3 rooms, 10 years: " (math round [p] 1) " kEUR"
Result
mentdb
intercept 26.475 (t = 1.67) | surface 3.793 (t = 7.37) | rooms -4.71 (t = -0.49) | age -1.404 (t = -4.99) | adjusted R2 = 0.9912, residual std error = 8.6 kEUR | 75 m2, 3 rooms, 10 years: 282.8 kEUR
Go further
executed on MentDB

Activation functions of a neural network, side by side

Before choosing the activation of a hidden layer, the data scientist plots sigmoid, tanh, softplus and ELU on [-4, 4]. Softplus uses log1p(exp z) and ELU uses expm1(z), the numerically safe forms used by deep learning libraries.

Pipeline
01 · sigmoid1 / (1 + exp(-z))math exp→02 · tanhHyperbolic tangentmath tanh→03 · softpluslog1p(exp z)math log1p→04 · ELUexpm1(z) for z < 0math expm1→05 · plotFour curvesplot chart
Script
admin
#1. Grid of pre-activations, step 0.5;
-> "[x]" ""; -> "[sig]" ""; -> "[th]" ""; -> "[sp]" ""; -> "[elu]" "";
for (-> "[i]" -8) (<= [i] 8) (++ "[i]") {
  -> "[z]" (/ [i] 2);
  -> "[x]" (concat [x] "," [z]);
  -> "[sig]" (concat [sig] "," (/ 1 (+ 1 (math exp (- 0 [z])))));
  -> "[th]" (concat [th] "," (math tanh [z]));
  -> "[sp]" (concat [sp] "," (math log1p (math exp [z])));
  -> "[elu]" (concat [elu] "," (if (> [z] 0) {[z];} {math expm1 [z];}));
};
#2. One chart, four series;
plot chart (concat "{\"title\": \"Activation functions\", \"x_label\": \"z\", \"y_label\": \"f(z)\", \"series\": [{\"name\": \"sigmoid\", \"x\": [" (string substr [x] 1) "], \"y\": [" (string substr [sig] 1) "]}, {\"name\": \"tanh\", \"x\": [" (string substr [x] 1) "], \"y\": [" (string substr [th] 1) "]}, {\"name\": \"softplus\", \"x\": [" (string substr [x] 1) "], \"y\": [" (string substr [sp] 1) "]}, {\"name\": \"ELU\", \"x\": [" (string substr [x] 1) "], \"y\": [" (string substr [elu] 1) "]}]}")
Result
mentdb
The chart is shown in the Plot tab of the editor.
Go further
  • Add the derivatives to see where the gradients vanish.
  • Check the identity tanh(z) = 2 sigmoid(2z) - 1 with math calc.
executed on MentDB

Least squares by QR decomposition, then ridge regression

With a small, slightly noisy dataset, the script fits y = b0 + b1 x through the QR decomposition (the stable way used by statistical software) and then the ridge solution (X'X + lambda D)^-1 X'y, where the diagonal matrix D leaves the intercept unpenalised.

Pipeline
01 · QRX = Q.Rmatrix qr→02 · OLSSolve R1 b = Q1' ymatrix solve→03 · penaltyDiagonal matrix lambda . Dmatrix diag→04 · ridgeNormal equations with penaltymatrix inverse
Script
admin
#1. Design matrix with a column of ones, and the target;
-> "[X]" "[[1, 1], [1, 2], [1, 3], [1, 4], [1, 5], [1, 6]]";
-> "[y]" "[2.3, 4.1, 6.2, 7.8, 10.3, 11.9]";
#2. OLS through QR: keep the first 2 rows of R and of Q'y;
json load "rx_p4g_qr" (matrix qr [X]);
-> "[qty]" (matrix mul (matrix transpose (json select "rx_p4g_qr" /Q)) [y]);
json load "rx_p4g_qty" [qty];
-> "[R1]" (concat "[" (json select "rx_p4g_qr" "/R[0]") ", " (json select "rx_p4g_qr" "/R[1]") "]");
-> "[ols]" (matrix solve [R1] (concat "[" (json select "rx_p4g_qty" "/[0]") ", " (json select "rx_p4g_qty" "/[1]") "]"));
json unload "rx_p4g_qr"; json unload "rx_p4g_qty";
#3. Ridge with lambda = 5 on the slope only;
-> "[Xt]" (matrix transpose [X]);
-> "[A]" (matrix add (matrix mul [Xt] [X]) (matrix diag "[0, 5]"));
-> "[ridge]" (matrix mul (matrix inverse [A]) (matrix mul [Xt] [y]));
concat "OLS by QR: " [ols] [_n_] "ridge lambda 5: " [ridge] [_n_] "check with matrix solve: " (matrix solve [X] [y])
Result
mentdb
OLS by QR: [0.2799999999999971,1.9485714285714293]
ridge lambda 5: [1.7955555555555591,1.5155555555555553]
check with matrix solve: [0.2800000000000079,1.9485714285714266]
Go further
  • Choose lambda by leave-one-out cross validation in a for loop.
  • Compare the condition numbers of X'X and X'X + lambda D with matrix cond.
executed on MentDB

Glorot uniform initialisation of a dense layer

A dense layer has 64 inputs and 32 outputs. The script draws the weight matrix uniformly in [-L, L] with L = sqrt(6/(fan_in + fan_out)) from a random matrix, and checks the bounds, the mean and the variance against the theoretical 2/(fan_in + fan_out).

Pipeline
01 · randomUniform matrix in [0, 1[matrix random→02 · scale2U - 1 then times Lmatrix scalar_mul→03 · shiftSubtract a matrix of onesmatrix sub→04 · checksMin, max, mean, variancestat pvariance
Script
admin
#1. Glorot limit for 64 -> 32;
-> "[fin]" 64; -> "[fout]" 32;
-> "[L]" (math sqrt (/ 6 (+ [fin] [fout])));
#2. W = L . (2U - 1);
-> "[U]" (matrix random [fin] [fout]);
-> "[W]" (matrix scalar_mul (matrix sub (matrix scalar_mul [U] 2) (matrix ones [fin] [fout])) [L]);
#3. Flatten and check the draw;
-> "[w]" (concat "[" (string regexp_replace [W] "[\[\]]" "") "]");
-> "[var]" (stat pvariance [w]);
-> "[target]" (/ 2 (+ [fin] [fout]));
concat "shape " (matrix size [W]) ", limit L = " [L] [_n_] "inside [-L, L]: " (and (>= (stat min [w]) (- 0 [L])) (<= (stat max [w]) [L])) ", mean close to 0: " (< (math abs (stat mean [w])) 0.02) [_n_] "target variance = " (math round [target] 5) ", draw within 10 %: " (< (math abs (- [var] [target])) (* 0.1 [target]))
Result
mentdb
shape [64,32], limit L = 0.25
inside [-L, L]: 1, mean close to 0: 1
target variance = 0.02083, draw within 10 %: 1
Go further
executed on MentDB

Regression through the origin: heating energy of a building

Heating energy is physically zero with zero degree-days and zero occupancy, so the energy model is fitted without intercept: kWh = a . HDD + b . occupied hours. The script compares it with the model with intercept and forecasts a cold week.

Pipeline
01 · data8 weeks of meter datapa rm load_from_json→02 · no interceptForce the originpa rm set_no_intercept→03 · fitCoefficients and R2pa rm estimate_regression_parameters→04 · forecastCold weekpa rm predict
Script
admin
#1. Weekly heating degree-days and occupied hours, energy in kWh;
-> "[x]" "[[45, 50], [62, 55], [80, 52], [95, 60], [110, 58], [70, 40], [55, 62], [88, 45]]";
-> "[y]" "[1510, 1980, 2390, 2840, 3190, 2050, 1850, 2550]";
#2. Model with intercept;
pa rm load_from_json "rx_p4g_e1" [x] [y];
-> "[p1]" (string regexp_replace (pa rm estimate_regression_parameters "rx_p4g_e1") "\s+" "");
#3. Model through the origin;
pa rm load_from_json "rx_p4g_e0" [x] [y];
pa rm set_no_intercept "rx_p4g_e0" true;
-> "[p0]" (string regexp_replace (pa rm estimate_regression_parameters "rx_p4g_e0") "\s+" "");
-> "[se]" (string regexp_replace (pa rm estimate_regression_parameters_standard_errors "rx_p4g_e0") "\s+" "");
-> "[f]" (pa rm predict "rx_p4g_e0" "[120, 55]");
-> "[r]" (concat "with intercept [b0,a,b]: " [p1] [_n_] "through origin [a,b]: " [p0] ", std errors " [se] [_n_] "forecast HDD 120, 55 h: " (math round [f] 0) " kWh");
pa rm close "rx_p4g_e1"; pa rm close "rx_p4g_e0";
[r]
Result
mentdb
with intercept [b0,a,b]: [-6.7634917036253785,24.941766035780343,7.877581710857582]
through origin [a,b]: [24.927410991925406,7.771949937735065], std errors [0.10516038164074848,0.1547319810680258]
forecast HDD 120, 55 h: 3419 kWh
Go further
  • Test whether the intercept is significant with its t ratio from the standard errors.
  • Add the solar gains as a third variable.
executed on MentDB

Multivariate Gaussian log-likelihood with an LU decomposition

An anomaly detector scores a new sensor vector with the log density of a 3D Gaussian fitted on normal data. The log-determinant of the covariance comes from the diagonal of U in the LU decomposition, the quadratic form from the inverse, and 2 pi from the constants.

Pipeline
01 · decomposeP.Sigma = L.Umatrix lu→02 · log detSum of ln U_iimath log→03 · Mahalanobisd' Sigma^-1 dvector dot→04 · constant2 piconstant math_two_pi→05 · scorelog pdfmath calc
Script
admin
#1. Mean and covariance of temperature, vibration, current;
-> "[mu]" "[60, 2.0, 12]";
-> "[S]" "[[4, 0.3, 1.2], [0.3, 0.09, 0.15], [1.2, 0.15, 1.0]]";
-> "[xv]" "[64.5, 2.6, 13.9]";
#2. log det Sigma from the diagonal of U;
json load "rx_p4g_lu" (matrix lu [S]);
-> "[ld]" 0;
for (-> "[i]" 0) (< [i] 3) (++ "[i]") {
  -> "[ld]" (+ [ld] (math log (math abs (json select "rx_p4g_lu" (concat "/U[" [i] "][" [i] "]")))));
};
json unload "rx_p4g_lu";
#3. Squared Mahalanobis distance;
-> "[d]" (vector sub [xv] [mu]);
-> "[m2]" (vector dot [d] (matrix mul (matrix inverse [S]) [d]));
#4. log pdf = -1/2 . m2 + log det + k ln 2pi;
-> "[tpi]" (string regexp_replace (constant math_two_pi) "^\((\S+) .*$" "$1");
-> "[lp]" (math calc "-0.5*([m2]+[ld]+3*ln([tpi]))");
concat "det from LU = " (math round (math exp [ld]) 6) ", matrix det = " (math round (matrix det [S]) 6) [_n_] "Mahalanobis^2 = " (math round [m2] 3) ", log pdf = " (math round [lp] 3) [_n_] "anomaly at 99 %: " (> [m2] (stat dist_inv "chi2" "[3]" 0.99))
Result
mentdb
det from LU = 0.1584, matrix det = 0.1584
Mahalanobis^2 = 6.254, log pdf = -4.963
anomaly at 99 %: 0
Go further

Also in Engineering & data

TR
Telecom & radar engineer
Link budgets in dB, radar equation, detection thresholds and spectra of modulated signals.
10 recipes
CE
Control engineer
Transfer functions, margins, PID and state feedback: complex numbers, CAS, ODE and matrices.
10 recipes
RE
Robotics engineer
Kinematics, Jacobians and trajectories: matrices, vectors, non-linear systems and splines.
10 recipes
EE
Electrical engineer
Impedances, three-phase power and cable sizing with complex numbers and units.
10 recipes
ES
Electronics & signal engineer
Filter design, spectra, peak detection and SNR on sampled signals.
10 recipes
IS
IoT & sensor engineer
Sensor calibration, smoothing, drift detection and measurement uncertainty.
10 recipes
MC
Mechanical & civil engineer
Deflection, stresses, safety factors and concrete strength statistics.
10 recipes
IQ
Industrial quality engineer
SPC control limits, process capability and tests on machines.
10 recipes
SB
Statistician & biostatistician
Confidence intervals, sample size, ANOVA, chi-square and non-parametric tests.
10 recipes
DQ
Data quality analyst
Outliers, format checks, normalization and duplicate detection.
10 recipes