Skip to main content
Back

RELIA Toolkit Makes AI Release Decisions Replayable and Auditable

A trending open-source toolkit brings reproducible release gates, A/B ship decisions, and offline-verifiable records to real-time ML decision systems.

Repo, Full Access
  • Trending repo ai-decision-system packages release and A/B ship gates for real-time AI decision models.
  • Three workflows: RELIA reliability gates, expgate A/B decisions, and a typed Jev decision model.
  • RELIA enforces nine release gates on worst-of-five seeds, blocking any failure with reproducible synthetic benchmarks.
  • Experiment gate emits SHIP, HOLD, ROLLBACK, or INVALID with CUPED and sequential monitoring support built in.
  • Every decision produces an offline-verifiable JSON record with SHA-256 digest and replay-based verification CLI.
  • Repository is MIT-licensed at root; RELIA itself remains an unreleased candidate pending Apache-2.0 clearance.

The GitHub project ai-decision-system has appeared on the platform’s trending charts with three Python workflows for recording and reviewing AI-related release decisions. Its two gating tools cover synthetic commerce models and live A/B tests, while a third workflow produces typed decisions from local models or the OpenAI Decisions API.

Its scope is release governance: separating evaluation data, measuring calibration, restricting model inputs, enforcing decision constraints, and preserving the evidence behind a ship, hold, or rollback call. These controls make approvals easier to reproduce and audit. Broader claims about safety, fairness, compliance, and commercial performance still require evidence from the deployment environment.

Three workflows, separate contracts

A workspace.json manifest treats each workflow as an independent project with its own tests and CI commands, as documented in the workspace guide.

Workflow Purpose Runtime
RELIA Evaluates release controls for synthetic commerce offers, retrieval, and ranking models. Pinned Python 3.13 environment with a lockfile.
Experiment gate Turns aggregate A/B test results into SHIP, HOLD, ROLLBACK, or INVALID. Python standard library only.
Jev decision model Returns typed decisions from a local model or the OpenAI Decisions API. Local or API-backed execution.

The Jev workflow gives downstream code a stable set of typed fields and removes the need to parse free-form model responses. Its OpenAI-backed path requires network access and credentials, while local execution keeps generation on the machine.

RELIA tests release assumptions

RELIA is a reference implementation for models that make real-time commercial decisions, including offer selection and candidate ranking. It targets evaluation leakage, weak calibration, unauthorized feature access, decisions outside declared constraints, and regressions introduced by new data or retraining.

  • Data separation: Time- and group-based partitions include an explicit assertion that detects leakage between training and evaluation data.
  • Model comparison: Five fixed training seeds are used with a regularized logistic baseline and a gradient-boosted candidate.
  • Calibration: Isotonic fitting maps scores to probabilities on a held-out partition. Reports include log loss, Brier score, expected calibration error, worst-slice calibration error, and a Murphy decomposition of probability quality.
  • Feature controls: An allowlisted record view counts every attempted read outside the approved input set. Access to labels, identifiers, or sensitive proxies can therefore block release.
  • Decision stability: At least 99.5% of near-equivalent pairs must receive the same decision when only prohibited fields and a sub-cent basket perturbation change.

Nine release gates are evaluated against the worst result across the five seeds, reducing the chance that a favorable random initialization hides an unstable candidate. Any failed gate blocks promotion.

Experiment gate controls live promotion

The experiment gate consumes an aggregate summary exported by an experimentation platform and applies four outcomes in a fixed order. The ordering matters because data-validity and harm checks take precedence over evidence of improvement.

  1. INVALID when a chi-square sample-ratio-mismatch test finds that observed traffic allocation differs from the expected split. The default srm_alpha is 0.001.
  2. ROLLBACK when the primary metric is significantly worse, or when a guardrail is significantly worse and exceeds its permitted degradation margin.
  3. SHIP when the primary metric is significantly better and every guardrail satisfies its non-inferiority requirement.
  4. HOLD when none of the preceding conditions applies.

Its metric layer supports proportions, means, and ratios. Ratio metrics use a delta-method standard error, which approximates uncertainty for a quotient from the uncertainty of its numerator and denominator. The package also implements CUPED, which uses pre-experiment information to reduce variance, and confidence sequences whose bounds remain valid when teams inspect an experiment repeatedly.

The repository reports 2,000 simulated A/A tests evaluated at 20 evenly spaced checkpoints. Reapplying fixed-horizon thresholds at every checkpoint shipped 19.8% of tests and rolled back 22.0%, despite a nominal 5% error rate. Setting policy.sequential enables repeated-look bounds intended to control that inflation, with a corresponding reduction in the ability to detect real effects at the same sample size.

Records survive the original run

Each experiment-gate evaluation writes a sorted decision.json and a readable DECISION.md. The JSON record contains a schema version, producer tag, full parsed input, and lowercase SHA-256 digest of the input’s canonical JSON encoding. Canonical encoding fixes serialization details such as key order, allowing another machine to hash the same logical input consistently.

The experiment-gate-verify command recomputes the digest and reruns the evaluator without network access. It can confirm that the stored decision still follows from the stored input or identify a mismatch after code, policy, or data changes.

Because the record embeds the complete parsed input, teams should check exported summaries for secrets, personal data, and regulated fields before retaining or sharing artifacts. RELIA adds a NIST AI Risk Management Framework crosswalk that maps its controls to specific framework subcategories.

Run the gates from source

The experiment gate can be installed from its source directory while its PyPI release remains pending:

git clone https://github.com/BichengWang/ai-decision-system.git
cd ai-decision-system/ai-workflows/experiment-gate
python -m pip install .

experiment-gate --input examples/offer-ranker-v2.json --out review-run --require-ship
experiment-gate-verify review-run/decision.json

In CI, --require-ship allows the command to reject outcomes that do not authorize promotion, while the verification command checks the resulting artifact independently.

RELIA uses uv to recreate its frozen environment before running the benchmark command documented in its README:

cd ../ai-decision-reliability
uv sync --frozen

The project says fresh runs under pinned local environments should produce byte-identical artifacts. API-backed model generation introduces an external service dependency, so teams using Jev should distinguish reproducibility of the stored record from reproducibility of the remote model response.

Limits and license status

RELIA’s synthetic benchmark cannot establish production performance, legal compliance, fairness across real populations, or consumer benefit. Its nine gates are proposed engineering thresholds, not government or industry standards, and deployment teams must set margins and guardrails that match their own risks.

The repository carries an MIT license at its root, while the RELIA documentation says that subproject remains an unreleased candidate with no license grant and a planned Apache 2.0 release after rights are cleared. That mismatch should be resolved before copying, modifying, or redistributing RELIA code.

Teams operating ranking, pricing, offer, or experimentation systems can use the repository as a reference for deterministic evaluations, explicit promotion policies, CI-enforced outcomes, and review artifacts that remain testable after the original release decision.