CUSTOM RL ENVIRONMENTS
What kind of gym
does your domain need?
What benchmark should
you target?
Sales Gym proved the pattern: a reproducible, verifiable environment that trains agents to manage something real. The same architecture applies to any domain — support, operations, finance, healthcare, logistics, code.
Every SpaceSwordAI gym follows the same ten principles. They make environments trainable, verifiable, and honest — regardless of domain.
Portfolio-first
One episode contains many concurrent work items and shared resources. The agent manages a pipeline, not a single case.
Progression-first
Reward verified movement through domain milestones — not messages sent, clicks made, or tickets touched.
Throughput-aware
Reward quality-adjusted advances per finite capacity, accounting for opportunity cost.
State-authoritative
The hidden simulator, not the agent, determines outcomes. Policy sees a permitted view only.
No reward hacking
Stage inflation, fabricated evidence, and needless escalation yield zero or negative reward.
Transfer before claims
Report held-out and external results. Never imply uplift from simulator-only numbers.
Gyms are not one-size-fits-all. The right environment matches your task, your evaluation target, and your deployment loop. Start with these four questions.
1 · What is the task?
A single decision (classify, route, approve) or a long-horizon process (manage a queue, run a territory, resolve a case)? Long-horizon tasks need portfolios, milestones, and shared resources. Single decisions need a classifier, not a gym.
2 · What does "good" mean?
Define the verified outcome: a signed deal, a resolved ticket, a clean hand-off, a delivered unit. Reward must attach to evidence-backed outcomes, not activity. If you cannot define good, you cannot train for it.
3 · What is the benchmark target?
Frozen splits with held-out families. You need train/eval/test territories whose difficulty cannot leak into training. The benchmark is what makes improvement measurable — and honest.
4 · What are the resources?
Finite capacity creates the real problem. Agents must schedule, prioritise, and accept opportunity cost. No shared resources, no portfolio decision — just autocomplete.
Every process domain has a buying journey — a sequence of verifiable stages between "identified" and "done". We model it as milestones with required evidence at each step. This is the pattern behind Sales Gym's 8-stage pipeline:
Your domain gets its own milestone set. Support: triaged → reproduced → diagnosed → fix-verified → closed. Logistics: booked → picked → routed → delivered → confirmed. The pattern is the same: each stage requires immutable evidence, and CRM-style self-declaration never counts.
The heart of the Sales Gym design is the private verifier. The policy never sees verifier internals, reward coefficients, or hidden state. Milestones advance only when immutable evidence — authored by the world, not the agent — supports the transition.
Evidence ontology is domain-specific: for sales it is buyer reply, meeting acceptance, budget confirmation, authorised sponsor, technical success, procurement acceptance, contract signature. For support it is reproduction steps, diagnostics, confirmation. Every evidence record carries source, timestamp, claim, proof reference, and an unforgeable author.
Anti-exploit defences are designed in, not bolted on: unauthorised commitments, fabricated evidence, stage cycling, premature disqualification, and unnecessary escalation all yield zero or negative reward.
A gym is only as good as the worlds it generates. Sales Gym ships ten scenario families, each targeting a specific failure mode: false champion, budget freeze, misfit customer, procurement trap, incumbent fortress, vanishing sponsor, territory triage, shared resource collision, false progression, timing and inaction.
Failure-first worlds
Curate scenarios that break naive agents: the champion who cannot buy, the deal that should not close, the stage that the CRM lies about.
Curriculum path
Single case → small allocation → full portfolio → noisy observations → asynchronous shocks → unfamiliar configurations.
Oracle-first mining
Sample feasible evidence paths and construct worlds that provably support them. Solvability is known by construction.
Hypothesis-first mining
Propose a precise failure hypothesis, build a world, test a target model, and minimise confirmed failures to their smallest cause.
Splits that hold
Split by templates, graphs, and mechanisms — not random seeds. Hold out at least one family. Hash and version every split.
Contamination audit
External benchmarks stay in a read-only workspace. Every split is hashed and versioned. Nothing leaks into training.
The benchmark is the contract between you and your models. Sales Gym's ESPB freezes manifests, seeds, and scoring before any training run, and reports uncertainty by territory, not by action. The same discipline applies to any gym.
Required baselines: random valid action, priority heuristics, expected-value policy, base model prompting, SFT, and the trained policy. Report median, mean, confidence interval, failure modes, and compute cost. Never call an internal benchmark an external one.
A custom gym is not a research deliverable — it is the seed of a deployment loop. Agents trained in your gym deploy through the standard SpaceSwordAI stack: Sokka for routing and runtime control, Haiku Guard to inspect every action, FireAgent to isolate execution. funcroute keeps the edge open source.
Sales Gym is our reference build of this pattern. See how it works.
Environment design
Domain model, milestone pipeline, evidence ontology, resource pool, event tape, and observation permissions.
World generation
Procedural generator with provenance-tagged synthetic data, feasible-path construction, and controlled distractors.
Reward + verifier
Potential-based shaping, private verifier, anti-exploit penalties, and counterfactual reward testing.
Benchmark + splits
Frozen manifests, held-out families, baselines, scoring CLI, and bootstrap confidence intervals.
Training wiring
GRPO / agentic RL adapters, rollout cloning, token alignment, and TRL / verl integration.
Deployment path
Wiring into Sokka, Haiku Guard, and FireAgent so trained agents run with control and isolation.
Tell us the domain.
We will design the gym, the benchmark target, and the deployment loop with you.