API reference

Reports

vetted.report.validate_risk_model(pnl, var_99, var_975=None, es_975=None, actual_pnl=None, risk_theoretical_pnl=None, pit=None, scale=None, significance=0.05, missing='raise', title='Risk model validation', n_sims=2000, seed=0)

Run every applicable backtest on a VaR / ES model.

Parameters:

Name	Type	Description	Default
pnl		Daily hypothetical P&L (gain positive).	required
var_99		99% one-day VaR forecast for each day (positive loss amount).	required
var_975		97.5% VaR, needed for the desk backtest and the ES tests.	None
es_975		97.5% Expected Shortfall forecast (positive loss amount).	None
actual_pnl		Actual P&L, if the traffic light should count against it too (MAR32.5 takes the greater count).	None
risk_theoretical_pnl		The risk model's own P&L, for the PLA test.	None
pit		The model's forecast CDF at the realised P&L, for the Du-Escanciano unconditional and conditional tests.	None
scale		A volatility forecast, for the standardised exceedance residual test.	None
significance	float	Level for the statistical tests.	0.05
missing	str	How statistical tests treat gaps ("raise" or "drop"). The regulatory counts always treat a gap as an exception (MAR32.5).	'raise'
n_sims	int	Monte Carlo draws for the VaR tests' p-values and the bootstrap of the exceedance residual test.	2000
seed	int	Random seed for those draws.	0

vetted.report.validate_strategy(returns, trials=None, sharpe_variance=None, periods=252, significance=0.05, n_blocks=16, title='Strategy validation')

Run every overfitting check on a backtested strategy.

Parameters:

Name	Type	Description	Default
returns		Returns of the selected strategy.	required
trials		Every strategy that was tried, as a (T, N) array or DataFrame of returns (including the selected one), or the number tried. Without it, only the probabilistic Sharpe ratio runs, which does not correct for selection.	None
sharpe_variance	Optional[float]	When trials is a number, the variance of the tried strategies' annualised Sharpe ratios (required then).	None
periods	int	Periods per year.	252

The strategy passes when the deflated Sharpe ratio is significant and PBO is at most 0.5.

vetted.report.Report dataclass

A set of test results with one overall verdict.

Running ten tests at 5% each would flag a correct model about 40% of the time, so the verdict does not simply ask whether any test rejected. The p-values of the statistical tests are adjusted together with Holm's method (holm_p), and a test counts against the model only if its adjusted p-value is below significance. Tests that report a zone are outside that family: red (or an ineligible desk) counts against the model, amber is flagged but does not fail it. Tests without a p-value or zone (PBO) count whenever they reject.

For a strategy, passing means showing skill: a test named in pass_requires_rejection (the deflated Sharpe ratio) counts against the strategy when it does not reject its null.

adjusted_p property

Holm-adjusted p-values by test name (for display).

rejected property

Tests that count against the model (see the class docstring).

flagged property

Amber zones. MAR32.11: amber outcomes "could result from either accurate or inaccurate models"; a correct 99% VaR lands in the amber zone about 11% of the time over 250 days. They are reported, not counted as failures.

passed property

True when at least one test ran and nothing counts against the model. Amber zones do not fail a model; check flagged.

holm_p()

Holm-adjusted p-value for each result, by position (nan for results outside the family).

vetted._result.TestResult dataclass

Outcome of a statistical test.

Attributes:

Name	Type	Description
name	str	Short name of the test.
statistic	float	The test statistic.
p_value	float	p-value under the null hypothesis (nan when the test has no p-value, for example a regulatory zone).
reject	bool	Whether the null is rejected at significance.
significance	float	Level the decision was taken at.
null	str	The null hypothesis, in words.
reference	str	Where the test is defined.
detail	dict[str, Any]	Anything else the test computed (counts, components, zones).

VaR

vetted.var

Backtests for Value-at-Risk forecasts.

Every function takes the realised P&L and the VaR that was forecast for each day (a positive loss amount, see vetted._inputs), plus the VaR tail probability alpha (0.01 for a 99% VaR).

exceptions(pnl, var, missing='raise')

Boolean array, True on days where the loss exceeded the VaR (pnl < -var). See vetted._inputs.align for missing.

kupiec(pnl, var, alpha, significance=0.05, missing='raise', pvalue='asymptotic', n_sims=10000, seed=0)

Kupiec (1995) proportion-of-failures test.

H0: the probability of an exception is alpha. Two-sided: too many exceptions and too few both reject. LR ~ chi2(1), or a Monte Carlo p-value with pvalue="monte_carlo".

Reference: Kupiec, P. (1995). Techniques for verifying the accuracy of risk measurement models. Journal of Derivatives, 3(2), 73-84.

christoffersen(pnl, var, alpha, significance=0.05, missing='raise', pvalue='asymptotic', n_sims=10000, seed=0)

Christoffersen (1998) independence and conditional coverage tests.

statistic and p_value are for conditional coverage, LR_cc = LR_uc + LR_ind ~ chi2(2): exceptions happen with probability alpha and do not cluster. The independence part alone (first-order Markov alternative, LR_ind ~ chi2(1)) is in detail. The decomposition matches R rugarch VaRTest.

Reference: Christoffersen, P. (1998). Evaluating interval forecasts. International Economic Review, 39(4), 841-862.

dynamic_quantile(pnl, var, alpha, lags=4, significance=0.05, missing='raise', squared_pnl_lag=False, pvalue='asymptotic', n_sims=5000, seed=0)

Engle and Manganelli (2004) dynamic quantile (DQ) test.

Regresses the demeaned hit sequence Hit_t = 1{exception} - alpha on a constant, lags of its own lags and the VaR forecast itself. Under a correct model none of these predict the next hit, so DQ = b' X'X b / (alpha (1 - alpha)) ~ chi2(lags + 2), with the degrees of freedom reduced by one when the VaR is constant (it is then collinear with the intercept).

Catches what the Christoffersen test cannot: clustering beyond one day and exceptions that depend on the level of the forecast.

squared_pnl_lag=True adds yesterday's squared P&L as a regressor (df = lags + 3), the variant implemented in the R package GAS (BacktestVaR).

With few expected exceptions (a 99% VaR over one year expects 2.5) the chi-squared p-value is oversized: in vetted's simulations it rejected a correct model 6.4% of the time at 5% over 250 days (tools/simulations.py). Use pvalue="monte_carlo" there; validate_risk_model does.

Reference: Engle, R. F. and Manganelli, S. (2004). CAViaR: conditional autoregressive value at risk by regression quantiles. Journal of Business and Economic Statistics, 22(4), 367-381.

exception_durations(pnl, var, alpha, missing='raise')

Days between exceptions, and how they compare with the geometric distribution a correct model implies (mean 1 / alpha). Descriptive; for a formal test use christoffersen or dynamic_quantile.

quantile_loss(pnl, var, alpha)

Per-day quantile (tick) loss of a VaR forecast, the strictly consistent scoring function for a quantile. Lower is better; compare models with vetted.compare.diebold_mariano.

Expected Shortfall

vetted.es

Backtests for Expected Shortfall forecasts.

Expected Shortfall replaced VaR as the capital measure under FRTB, but it is not elicitable on its own (Gneiting 2011), so there is no single loss function that backtests it the way the quantile loss backtests VaR. This module collects the tests that work anyway:

• acerbi_szekely: the Z1 and Z2 statistics, with the fixed Z2 traffic light the paper shows is stable across distributions, and exact Monte Carlo p-values when you can simulate from your model.
• conditional_calibration: Nolde and Ziegel (2017), a deterministic test of VaR and ES jointly, built on their identification function.
• exceedance_residuals: McNeil and Frey (2000), bootstrap test that the losses beyond VaR average the forecast ES.
• du_escanciano: unconditional and conditional (clustering) tests from the probability integral transforms of the P&L.

Sign conventions as in vetted._inputs: P&L positive for a gain, VaR and ES positive loss amounts, alpha the tail probability (0.025 for the FRTB 97.5% ES).

acerbi_szekely(pnl, var, es, alpha=0.025, sampler=None, n_sims=10000, seed=0, significance=0.05, missing='raise')

Acerbi and Szekely (2014) tests of Expected Shortfall.

Z2 = sum_t P&L_t 1{exception_t} / (T alpha ES_t) + 1 tests the frequency and the size of tail losses together; Z1 averages P&L_t / ES_t over the exceptions only and so tests size given the VaR. Both are 0 in expectation under a correct model and negative when risk is underestimated. statistic is Z2; Z1 is in detail.

Without a sampler, the decision uses the paper's fixed Z2 thresholds (amber below -0.70, red below -1.8), which hold across tail shapes for T = 250 and alpha = 2.5%, and p_value is nan. For other sample sizes the thresholds are scaled by sqrt(250 / T). That scaling is vetted's, not the paper's: unscaled, the 5% threshold would almost never fire on 1000 days (a false alarm rate near 0% instead of 5%). Scaled, the amber threshold keeps a false alarm rate of about 4-6.5% from T = 60 to 2500 in simulations. The red (0.01%) threshold is only approximate away from T = 250, because the far tail of Z2 is skewed: its false alarm rate ranged from about 0.001% to 0.1% over the same span. For an exact level, pass a sampler. With a sampler(rng, n) returning an (n, T) array of P&L paths drawn from the model's own daily predictive distributions, p-values for Z1 and Z2 are exact Monte Carlo p-values (one-sided, small when risk is underestimated).

Reference: Acerbi, C. and Szekely, B. (2014). Backtesting expected shortfall. Risk, December 2014.

conditional_calibration(pnl, var, es, alpha=0.025, scale=None, significance=0.05, missing='raise')

Nolde and Ziegel (2017) conditional calibration test of VaR and ES.

Uses the identification function of the (VaR, ES) pair, V1 = alpha - 1{exception} and V2 = ES_r - VaR_r + 1{exception} (VaR_r - r) / alpha (in return space, where VaR_r = -var and ES_r = -es), which has mean zero when both forecasts are right. statistic and p_value are the two-sided Wald test of the "simple" version. With a volatility forecast scale, the "general" versions are added to detail. The two-sided tests follow the R package esback (Bayer and Dimitriadis) exactly.

One-sided tests (detail["p_simple_one_sided"] and ["p_general_one_sided"]) are the paper's: they reject only when risk is underestimated, that is when there are too many exceptions or the losses beyond VaR are larger than the ES. esback's one-sided p-values use the opposite sign for V1, so they also reject a model that is too conservative; they are reported as p_simple_one_sided_esback and p_general_one_sided_esback.

Use the two-sided test with care on short samples: it is asymptotic, and on a correct Student-t(5) model it rejected 24% of the time at 250 days, 17% at 500 and 13% at 1000 at the 5% level in vetted's simulations (300 paths each, tools/simulations.py). validate_risk_model uses the paper's one-sided version.

Reference: Nolde, N. and Ziegel, J. F. (2017). Elicitability and backtesting: perspectives for banking regulation. Annals of Applied Statistics, 11(4), 1833-1874.

exceedance_residuals(pnl, var, es, scale=None, n_boot=10000, seed=0, significance=0.05, missing='raise', min_exceptions=5)

McNeil and Frey (2000) exceedance residual test.

On exception days the realised return minus the forecast ES (in return space) should average zero. The statistic is the t-ratio of those residuals (divided by scale if a volatility forecast is given) and the one-sided p-value, small when losses beyond VaR are worse than the ES said, comes from a centred bootstrap, as in esback's er_backtest. The two-sided p-value is in detail.

The bootstrap needs a handful of exceptions to mean anything: with two, every resample reproduces the observed t-ratio or is degenerate, and a correct model is rejected whenever that ratio is negative (esback has the same behaviour). Below min_exceptions the test raises ValueError.

Reference: McNeil, A. J. and Frey, R. (2000). Estimation of tail-related risk measures for heteroscedastic financial time series: an extreme value approach. Journal of Empirical Finance, 7(3-4), 271-300.

du_escanciano(pit, alpha=0.025, lags=5, significance=0.05, pvalue='asymptotic', n_sims=10000, seed=0)

Du and Escanciano (2017) backtests from probability integral transforms.

pit is u_t = F_t(P&L_t), the model's own forecast CDF evaluated at the realised P&L (small u is a large loss). The cumulative violation H_t = (alpha - u_t) 1{u_t <= alpha} / alpha has mean alpha / 2 and variance alpha (1/3 - alpha/4) under a correct model, whatever the distribution.

statistic and p_value are the unconditional test (one-sided, rejecting when the tail is heavier than forecast). The conditional test in detail is a Box-Pierce statistic on the first lags autocorrelations of H_t - alpha/2. Its null is E[H_t | past] = alpha/2, so it rejects clustering of tail losses but also a miscalibrated tail (a model that is too conservative has few tail events and fails it too).

pvalue="monte_carlo" simulates both statistics under the null (independent uniform PITs), which keeps its level at any sample size. The asymptotic conditional test is oversized: in vetted's simulations it rejected a correct model 12.8% of the time at 250 days and 9.6% at 1000, at a nominal 5% (tools/simulations.py).

Reference: Du, Z. and Escanciano, J. C. (2017). Backtesting expected shortfall: accounting for tail risk. Management Science, 63(4), 940-958.

fz0_loss(pnl, var, es, alpha=0.025)

Per-day FZ0 loss of a joint (VaR, ES) forecast.

VaR and ES are not separately elicitable for comparison purposes but the pair is (Fissler and Ziegel 2016). FZ0 is the zero-degree homogeneous member of that family, recommended by Patton, Ziegel and Chen (2019) because loss differences do not depend on the scale of the P&L. Lower is better; compare two models with vetted.compare.diebold_mariano.

In return space with v = -var, e = -es (both negative): L = -1{r <= v} (v - r) / (alpha e) + v / e + log(-e) - 1.

Reference: Patton, A. J., Ziegel, J. F. and Chen, R. (2019). Dynamic semiparametric models for expected shortfall (and value-at-risk). Journal of Econometrics, 211(2), 388-413.

Basel FRTB

vetted.frtb

Regulatory backtesting under the Basel market risk framework (FRTB).

Implements the tests in chapter MAR32 of the Basel Framework ("Internal models approach: backtesting and P&L attribution test requirements"), in force from 1 January 2023, and the binomial logic of MAR99 that sets the backtesting zones:

• traffic_light: bank-wide VaR backtesting zone and multiplier (MAR32.5, MAR32.9 Table 1), counting a missing P&L or VaR as an exception, and taking the greater of the actual-P&L and hypothetical-P&L counts.
• binomial_zones: the same green / amber / red cut-offs for any sample size and VaR level, from the cumulative binomial probabilities (MAR99).
• desk_backtest: trading desk eligibility, at most 12 exceptions at the 99th and 30 at the 97.5th percentile over 250 days (MAR32.18-32.19).
• pla_test: the P&L attribution test, Spearman correlation and the Kolmogorov-Smirnov distance between risk-theoretical and hypothetical P&L, and the resulting PLA zone (MAR32.34-32.42).

Source: https://www.bis.org/basel_framework/chapter/MAR/32.htm

binomial_zones(n=250, alpha=0.01, green=0.95, red=0.9999)

Green, amber and red exception counts for n days at VaR tail probability alpha.

A count k is green while the probability of seeing k or fewer exceptions from a correct model is below green (95%), red once it reaches red (99.99%), and amber in between, the rule MAR99 uses to set Table 1. For n = 250 and alpha = 1% this reproduces Table 1 exactly: green 0-4, amber 5-9, red 10 or more.

traffic_light(var, hypothetical_pnl, actual_pnl=None, alpha=0.01, missing='exception', window=250)

Bank-wide VaR backtesting zone (MAR32.5 and MAR32.9).

Exceptions against hypothetical and actual P&L are counted separately and the greater count is used (MAR32.5); a day on which the P&L or the VaR is missing counts as an exception (missing="exception", the default). The zone comes from binomial_zones for the sample size and level given, and the capital multiplier from Table 1 when the window is the standard 250 days at the 99th percentile.

window: the regulatory count uses the most recent 12 months, so by default only the last 250 days are counted. Pass window=None to use the whole series (the zones then come from the binomial rule for that length).

statistic is the exception count; reject is True outside the green zone.

desk_backtest(var_99, var_975, hypothetical_pnl, actual_pnl=None, missing='exception', window=250)

Trading desk backtesting requirement (MAR32.18-32.19).

A desk stays on the internal models approach only if, over the most recent 250 days (window), it has at most 12 exceptions of its 99th percentile VaR and at most 30 of its 97.5th percentile VaR, counting against actual and hypothetical P&L separately and taking the greater. reject is True when the desk fails and must move to the standardised approach (detail["zone"] is "ineligible"). statistic is the 99% count; both counts are in detail.

pla_zone(spearman, ks)

PLA zone from the two test metrics (MAR32.42 Table 2): green if the correlation is above 0.80 and the KS distance below 0.09; red if the correlation is below 0.70 or the KS distance above 0.12; amber otherwise. The boundaries themselves (0.80, 0.09) fall in amber.

pla_test(risk_theoretical_pnl, hypothetical_pnl)

P&L attribution test (MAR32.34-32.42).

Spearman rank correlation between risk-theoretical P&L (RTPL, the P&L the risk model's pricing produces) and hypothetical P&L (HPL, front office pricing), and the Kolmogorov-Smirnov distance between their empirical distributions. Green needs correlation above 0.80 and KS below 0.09; red is correlation below 0.70 or KS above 0.12; anything else is amber (Table 2). A red desk must use the standardised approach (MAR32.43); an amber desk pays a capital surcharge (MAR32.44).

statistic is the Spearman correlation, the KS distance is in detail, and reject is True outside the green zone. MAR32.35 asks for the most recent 250 days.

Model comparison

vetted.compare

Compare two risk models on the same days.

A backtest says whether one model is acceptable; a comparison says which of two is better. Score each model with a strictly consistent loss (vetted.var.quantile_loss for VaR, vetted.es.fz0_loss for VaR and ES together) and test whether the mean loss difference is zero.

diebold_mariano(loss_a, loss_b, horizon=1, alternative='two-sided', varestimator='acf', significance=0.05)

Diebold-Mariano test of equal predictive accuracy, with the Harvey, Leybourne and Newbold (1997) small-sample correction and Student-t p-values.

loss_a and loss_b are the per-day losses of two models. The statistic is negative when model A has the lower loss. alternative="less" tests that A is better. The long-run variance uses autocovariances up to horizon - 1, unweighted ("acf") or with Bartlett weights ("bartlett"). This matches R forecast::dm.test.

References: Diebold, F. X. and Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business and Economic Statistics, 13(3), 253-263. Harvey, D., Leybourne, S. and Newbold, P. (1997). Testing the equality of prediction mean squared errors. International Journal of Forecasting, 13(2), 281-291.

Overfitting

vetted.overfitting

Is a backtest evidence of skill, or the best of many lucky tries?

• probabilistic_sharpe: probability the true Sharpe ratio exceeds a benchmark, allowing for sample length, skewness and fat tails.
• expected_max_sharpe: the Sharpe ratio the best of N skill-less strategies shows on average (the "false strategy theorem").
• deflated_sharpe: the probabilistic Sharpe ratio against that hurdle.
• min_track_record: how long a track record has to be before its Sharpe ratio is significant.
• pbo: probability of backtest overfitting by combinatorially symmetric cross-validation.
• haircut_sharpe: the Sharpe ratio after a multiple-testing correction (Bonferroni, Holm or Benjamini-Hochberg-Yekutieli).
• effective_trials: how many independent trials a set of correlated strategies amounts to.

Sharpe ratios are annualised with periods (252 for daily data) at the interface and computed per period inside, as the formulas require. Skewness and kurtosis are the population moments used by Bailey and Lopez de Prado and by R PerformanceAnalytics ProbSharpeRatio.

sharpe_std_error(sr, skew, kurt, n)

Standard error of a per-period Sharpe ratio for non-normal returns (Mertens 2002; Bailey and Lopez de Prado 2012, eq. 3). kurt is raw kurtosis (3 for a normal distribution).

probabilistic_sharpe(returns, benchmark=0.0, periods=252)

Probability that the true Sharpe ratio exceeds benchmark (annualised).

Reference: Bailey, D. H. and Lopez de Prado, M. (2012). The Sharpe ratio efficient frontier. Journal of Risk, 15(2), 3-44.

expected_max_sharpe(trials, sharpe_variance)

Expected maximum Sharpe ratio of trials independent strategies whose true Sharpe ratio is zero, when their estimated Sharpe ratios have variance sharpe_variance. Units follow the input: pass the variance of annualised Sharpe ratios to get an annualised hurdle.

E[max] ~ sqrt(V) ((1 - gamma) Z^-1(1 - 1/N) + gamma Z^-1(1 - 1/(N e))), with gamma the Euler-Mascheroni constant.

Reference: Bailey, D. H. and Lopez de Prado, M. (2014). The deflated Sharpe ratio: correcting for selection bias, backtest overfitting and non-normality. Journal of Portfolio Management, 40(5), 94-107.

deflated_sharpe(returns, trials, sharpe_variance=None, periods=252, significance=0.05, independent_trials=None)

Deflated Sharpe ratio: the probability the selected strategy's true Sharpe ratio is above zero, allowing for how many strategies were tried and for non-normal returns.

trials is either the number of strategies tried, with sharpe_variance the variance of their annualised Sharpe ratios, or a (T, N) array or DataFrame holding every tried strategy's returns, from which both are measured.

Correlated trials: measuring the Sharpe variance on the trials already reflects their correlation (correlated strategies have similar Sharpe ratios, so the variance is smaller and so is the hurdle). Do not also shrink the count to an effective number of trials: that discounts the correlation twice, and in simulations with pairwise correlation 0.7 it raised the false positive rate from about 3.5% to 23% at a nominal 5%. independent_trials is for the Bailey and Lopez de Prado recipe where both the count and sharpe_variance come from clusters of strategies, not from the raw trials.

statistic is the DSR (a probability); reject means the Sharpe ratio is significant after deflation (DSR > 1 - significance). The hurdle and the undeflated PSR are in detail.

Reference: Bailey and Lopez de Prado (2014), Journal of Portfolio Management 40(5).

min_track_record(returns, benchmark=0.0, confidence=0.95, periods=252)

Minimum number of observations for the Sharpe ratio to be significantly above benchmark (annualised) at confidence. inf if the observed Sharpe ratio does not exceed the benchmark.

Reference: Bailey and Lopez de Prado (2012), Journal of Risk 15(2).

pbo(trials, n_blocks=16, metric=None, rank_denominator='n+1')

Probability of backtest overfitting (CSCV).

Splits the (T, N) matrix of every configuration's returns into n_blocks equal blocks and, for every way of choosing half of them as in-sample, picks the configuration with the best in-sample metric (Sharpe by default) and records its relative rank out of sample. PBO is the share of splits in which the in-sample winner ranks at or below the out-of-sample median. Near 0: selection found something real. Near 0.5 or above: the selection process is overfitting.

metric(M) receives a (t, N) block and returns N scores. Rows beyond a multiple of n_blocks are dropped from the end.

rank_denominator: the relative rank is rank / (N + 1) in Bailey et al. (2017); the R package pbo divides by N. Pass "n" to reproduce it.

Reference: Bailey, D. H., Borwein, J., Lopez de Prado, M. and Zhu, Q. J. (2017). The probability of backtest overfitting. Journal of Computational Finance, 20(4), 39-69.

haircut_sharpe(sharpe, observations, trials, method='holm', periods=252)

Sharpe ratio after correcting its p-value for multiple testing.

The annualised sharpe over observations periods is converted to a t-statistic, t = SR sqrt(observations / periods); its two-sided p-value is adjusted for the number of strategies tried and converted back. With trials as a count, only "bonferroni" is possible; pass the annualised Sharpe ratios of every trial (same length of history, including this strategy, which is added if it is missing) to use "holm" or "bhy", which are less conservative.

Returns the haircut Sharpe ratio, the haircut (share of the Sharpe ratio lost), and the single and adjusted p-values.

Reference: Harvey, C. R. and Liu, Y. (2015). Backtesting. Journal of Portfolio Management, 42(1), 13-28.

effective_trials(trials)

Effective number of independent strategies among correlated ones.

Uses the eigenvalues of the correlation matrix of the trials' returns: N_eff = (sum lambda)^2 / sum lambda^2, which is N for uncorrelated strategies and 1 when they are all the same strategy.

This is the participation ratio of the spectrum, a descriptive measure of how many distinct bets a search made. It is biased low when the history is short relative to the number of strategies (100 independent series of 250 days give about 71), and it is not an input to deflated_sharpe when the Sharpe variance is measured on the same trials (see that function).

Conformal VaR

vetted.conformal

VaR with a coverage guarantee, from conformal prediction.

Historical-simulation VaR takes an empirical quantile and hopes. Split conformal prediction takes the same quantile with a (n + 1) correction and gets a finite-sample guarantee: if the calibration P&L and the next day are exchangeable, the next loss exceeds the VaR with probability at most alpha, with no distributional assumption. Markets are not exchangeable for long, so adaptive_var adds Adaptive Conformal Inference, which steers the level online and guarantees the long-run exception rate converges to alpha even when volatility regimes change.

Both accept an optional scale (a volatility forecast for each day, from GARCH, EWMA or anything else), in which case conformal calibration is done on standardised losses and the result is a conformalised version of that model: its shape, with coverage that is guaranteed rather than assumed.

conformal_quantile(scores, alpha)

The ceil((n + 1)(1 - alpha))-th smallest of n scores, the quantile that makes split conformal valid in finite samples. inf when n is too small for the level (fewer than 1 / alpha - 1 scores).

conformal_var(calibration_pnl, alpha=0.01, scale=None, next_scale=1.0)

One-day VaR (positive loss amount) with P(loss > VaR) <= alpha when the calibration days and the next day are exchangeable.

With a volatility forecast scale for the calibration days and next_scale for the next day, calibration uses standardised losses -pnl / scale and the VaR is their conformal quantile times next_scale.

Reference: Vovk, V., Gammerman, A. and Shafer, G. (2005). Algorithmic Learning in a Random World. Springer. Lei, J. et al. (2018). Distribution-free predictive inference for regression. JASA 113(523).

adaptive_var(pnl, alpha=0.01, window=250, gamma=0.005, scale=None, alpha_start=None, unbounded='max_loss')

Out-of-sample VaR forecasts that adapt to regime changes, with a long-run coverage guarantee.

For each day t >= window the VaR is the conformal quantile of the previous window (standardised) losses at level alpha_t, using only data before t. After seeing day t, the level moves by alpha_{t+1} = alpha_t + gamma (alpha - err_t), err_t = 1{exception}. With unbounded="inf", whatever the data do, the realised exception rate over T days is within (max(alpha_1, 1 - alpha_1) + gamma) / (gamma T) of alpha.

When alpha_t falls below 1 / (window + 1), the conformal quantile is infinite: no loss in the window is large enough. unbounded="inf" keeps the infinite forecast, which is what the guarantee assumes. unbounded="max_loss" (the default) uses the largest loss in the window instead, so every forecast is a usable number; the guarantee then holds only if no exception falls on such a capped day, which the result reports as guarantee_holds (with capped_days and capped_exceptions). In a steady sell-off the capped VaR can be breached repeatedly and the guarantee is lost; check the flag.

Returns the VaR path (NaN for the first window days), the exception indicators, the level path and the realised exception rate.

Reference: Gibbs, I. and Candes, E. (2021). Adaptive conformal inference under distribution shift. NeurIPS 34.

Metrics

vetted.metrics.compute(metric, convention, r, periods_per_year)

vetted.metrics.conventions_for(metric)

vetted.metrics.metrics()

vetted.testing.cases(metric, convention, skip_undefined=True)