hanzo-ml-research

Native Hanzo AI/ML research and systems-design skill for LLMs, encoders, decision models, training, inference, kernels, quantization, retrieval/reranking, safety, evaluation, and scientific benchmarking. Use Hanzo's Rust stack (hanzo-ml, hanzo-nn, hanzo-train, hanzo-transformers, hanzo-kernel/DSL, hanzo-research, engine, and decision/Kai) by default; search current literature when needed; record falsifiable experiments and artifacts through the unified /v1/research plane.

Hanzo ML Research

Use this skill for machine learning, LLM, encoder, decision-model, retrieval, training, inference, kernel, quantization, safety, evaluation, or AI architecture work in the Hanzo stack.

The default operating principle is:

Research in the same stack that ships.

Exploratory scripts may be used for disposable analysis, but canonical results, model code, training, evaluation, latency measurements, and production paths should be implemented and reproduced in native Rust on Hanzo's ML/runtime stack unless the task explicitly requires another environment.

Core stack

Prefer the repository/workspace already associated with the task. The important Hanzo components are:

When a local workspace exposes additional Hanzo DSL, CUDA, ROCm, Metal, Vulkan, distributed-training, or model crates, inspect and reuse them rather than creating parallel implementations.

Research plane: /v1/research

Canonical R&D evidence should be recorded through hanzo-research, which writes to the unified research plane under /v1/research.

Rust:

use hanzo_research::{Research, Verdict};

let r = Research::from_env();

let mut exp = r
    .experiment("pooling", "j6-logmeanexp", "override-detection")
    .metric("recall_at_99_5_specificity")
    .hypothesis("localized pooling recovers procedural override evidence hidden by mean pooling")
    .predict("LogMeanExp improves held-out override recall without increasing benign FP beyond the preregistered gate");

exp.log("frozen encoder; identical split, seed, optimizer, and token cache across arms");
exp.conclude(
    Verdict::Proven,
    "LogMeanExp cleared the preregistered gate",
    Some(0.87),
)?;

Environment:

Important native endpoints include:

Use hanzo-research APIs rather than hand-rolling HTTP when working in Rust. The SDK automatically captures repository, git, host, and resolved Hanzo crate provenance.

A refutation is a valid research result. Do not rewrite failed hypotheses into successes after the run.

Research workflow

For substantial research work, follow this order.

1. Establish the question

Write a falsifiable hypothesis before looking at the decisive result.

Bad:

Try some pooling methods and see what works.

Good:

Frozen P0 token states already encode procedural-override evidence, but global mean pooling dilutes short localized cues. A learned linear token scorer with normalized LogSumExp pooling should improve family-held-out override recall at fixed benign specificity.

Specify:

2. Inspect the actual Hanzo implementation

Before proposing a replacement:

Do not recommend rebuilding components that already exist.

3. Search current research

For state-of-the-art or rapidly changing ML topics, search current primary sources before making SOTA claims.

Prefer:

  1. peer-reviewed paper or official arXiv manuscript
  2. authors' official repository
  3. model/system technical report
  4. official benchmark repository
  5. independent reproduction

Use external research to generate hypotheses and baselines. Reimplement important mechanisms in the Hanzo stack before treating them as production candidates.

Good areas to scan when relevant:

4. Build the smallest controlled ablation

Change one causal factor at a time.

Examples:

Keep identical where possible:

If one arm uses extra capacity, report it as a capacity ablation, not a clean mechanism ablation.

5. Assert implementation invariants

Before benchmarking, add tests that make silent confounds fail loudly.

Typical invariants:

6. Measure the right decomposition

Do not collapse a pipeline failure into one metric.

For retrieval/reranking, separate:

For classification/safety, separate:

For model adaptation, separate:

For latency, separate:

7. Use parent-relative controls when preserving competence

When adding a protection or adaptation mechanism, compare against both:

This distinguishes retention from causal prevention of degradation.

When continuation matters, prefer full warm resume:

8. Seal the result

Once the preregistered endpoint is reached:

Rust-native implementation rules

Tensor graph first

Training-time operations return tensors. Do not call to_scalar, to_vec, or equivalent host extraction before the loss/backward boundary.

Bad:

let score: f32 = pooled.to_scalar()?;

inside a trainable forward path.

Good:

let pooled: Tensor = pool.forward(&scores, &mask)?;
let loss: Tensor = objective.forward(&pooled, &labels)?;
loss.backward()?;

Convert to host scalars only for logging/evaluation.

GPU hot path

Avoid:

Reference implementations may be simple/ragged first. Optimize only after correctness is established.

Numerical policy

Keep numerically sensitive reductions in F32 unless a validated lower-precision path is equivalent.

Examples:

Use BF16/FP16/FP8/NVFP4 where supported and validated, but preserve a higher-precision reference.

Model-design principles

Prefer modular architecture

Separate:

Do not force one expensive path on every decision.

Large option sets

For hundreds to millions of options, prefer a cascade:

StateHandle
   ↓
semantic / memory retrieval
   ↓
top-K
   ↓
late interaction
   ↓
smaller K
   ↓
cross-attention / decision scorer
   ↓
joint verifier

Measure retrieval recall separately from decision accuracy.

Dynamic options

Prefer semantic option representations with optional learned/memory residuals so unseen tools/actions can participate without retraining the full model.

Set semantics

If option order is semantically irrelevant, architecture and tests should enforce permutation invariance/equivariance rather than relying only on shuffled training examples.

Calibration

For decision systems, accuracy alone is insufficient.

Report:

Avoid quantization or objective changes that improve speed/accuracy while silently destroying probability quality.

Safety/control-plane heads

A specialized safety head should be isolated from unrelated decision types. Require a feature-off/non-target invariance test.

For token-localized safety evidence, test global and localized pooling under the same scorer capacity. Prefer semantic labels and family-held-out negatives over keyword rules.

LLM and inference research

When optimizing generative models, inspect:

Benchmark on the actual hardware/backend targeted for deployment.

Do not extrapolate CUDA results to ROCm/Metal/Vulkan without measurement.

Kernel and DSL research

For a kernel change:

  1. preserve a numerically trusted reference
  2. state expected bottleneck: memory bandwidth, arithmetic, occupancy, launch overhead, synchronization, etc.
  3. test dominant real shapes, not only toy shapes
  4. test cold and warm
  5. record backend/device/compiler versions
  6. compare generated DSL and hand-written implementations where relevant
  7. record a refutation when the generated or hand kernel loses

Use hanzo-research so kernel evidence accumulates into the same research corpus as model-level experiments.

Benchmark integrity

Never claim SOTA from mismatched protocols.

For competitor comparisons, match as closely as possible:

If protocols differ, say so explicitly.

Prefer paired comparisons on the same examples:

Output style for research tasks

When reporting a result, lead with:

  1. what changed
  2. what the result supports
  3. what it does not yet establish
  4. next decisive experiment

Use causal language proportional to the design.

Prefer:

Persistent first-order gradient conflict is strongly disfavored under this matched regime.

over:

We proved gradients never conflict.

Prefer:

The result falsifies the architectural capacity-ceiling hypothesis.

over:

The architecture is optimal.

Production promotion gates

A research mechanism is not production-ready only because it improves one metric.

Before promotion require, as applicable:

Default decision rule

When choosing between two mechanisms with similar quality, prefer the one that is:

  1. simpler
  2. easier to falsify
  3. cheaper at runtime
  4. easier to reproduce
  5. more native to the Hanzo stack
  6. less coupled to benchmark-specific heuristics

The goal is not merely to win a benchmark. The goal is to produce a reproducible mechanism that becomes a durable primitive in Hanzo ML.