Rizzo Flow Brings Typed AI Decisions Offline With 2.8x Faster Throughput
Rizzo Flow is an open-source, local-first alternative to TypeSafe's Jev that returns typed decisions from an LLM by reading answer-letter logits instead of generating tokens.
Takeaways
- Rizzo Flow returns typed LLM decisions by reading answer-letter logits, generating zero tokens per call.
- HTTP interface mirrors TypeSafe's Jev API, so existing clients can switch by changing one URL.
- Fine-tuned 4B model hits 0.648 accuracy on typed-decisions vs 0.727 for hosted Jev, with 6x lower KL.
- Roughly 50 ms per decision at Q8_0 on an RTX 5060 Ti using 5.6 GiB of GPU memory.
- Runs on llama.cpp across CUDA, Metal, Vulkan, ROCm, SYCL or CPU; weights on Hugging Face, Apache-2.0.
- Weak spot: under-uses abstention on missing-evidence inputs; probabilities need temperature scaling for your domain.
Rizzo Flow is a new open-source, local-first implementation of the programming model behind TypeSafe’s hosted Jev service. It runs an open-weight model on local hardware and returns typed decisions by inspecting logits for candidate answers, eliminating model-generated JSON and its parsing failures.
TypeSafe describes Jev as a System One model: a function receives unstructured state and returns judgments such as booleans, choices, rubric scores, numbers, and associated probabilities. Rizzo Flow mirrors the TypeSafe HTTP interface, so an existing client may require only a base URL change to target localhost. Its llama.cpp runtime supports Apple, NVIDIA, AMD, and Intel GPUs, along with CPU-only execution.
A forward pass becomes a decision
Rizzo Flow converts each question into multiple choice, with every candidate mapped to one uppercase letter. At request time, it verifies that each letter occupies exactly one tokenizer token. The alphabet therefore limits each question to 26 candidate slots.
The shared state appears first in the prompt and is processed once. Its key-value cache, which stores the transformer’s reusable attention state, then feeds every question. The runtime evaluates those question branches together in an unpadded micro-batch, avoiding duplicate work on the shared context.
At each answer position, the runtime reads only the logits assigned to allowed letters. It does not sample tokens or enter a decoding loop. A Python layer normalizes those scores into probabilities and maps the candidates to the requested output type. The project’s “zero-token” description refers to the absence of generated output tokens; the model still performs the forward pass needed to calculate logits.