RL ENVIRONMENT FOR MODEL STUDIOS
Train and benchmark
tool-using models on
enterprise selling.
A finite-horizon, partially-observable multi-agent reinforcement learning (RL) environment for B2B enterprise sales portfolio management — the sales equivalent of a competitive game environment, but with real economic structure. 10–100 concurrent opportunities, shared seller/SE/legal/management capacity, and a private verifier that scores only immutable buyer evidence. Built for model studios that need a defensible agentic benchmark — not a chatbot demo.
Environment
Finite-horizon partially-observable semi-Markov decision process with asynchronous multi-agent responses. Formal RL spec: E = (S, O, A, T, Ω, R, H, C).
Episode
One territory, 10–100 concurrent opportunities, shared capacity budget. 30 / 90 / 180-day horizons. Debug mode with 3–5 opportunities.
Milestones
8 verified stages: Identified → Engaged → Qualified → Validated → Mobilised → Commercial → Commit → Closed. Disqualification is a legitimate terminal.
Evidence Ontology
11 evidence types, each with source_actor, timestamp, claim, proof_ref, verified_by. Milestones advance only on evidence the agent cannot forge.
Tool API
18 typed actions with JSON-Schema (OpenAI-compatible) exports. Each returns status, state_version, time_cost, resource_cost, event_ids.
State Model
Append-only events. Immutable IDs, versioning, provenance. Materialised views derived by replay. Deterministic replay per seed.
Confidentiality
Policy sees portfolio summary, CRM fields, verified milestone, tools. Never sees hidden budget truth, motives, competitor actions, verifier scores, or reward coefficients.
Determinism
Same seed + same actions → identical state and reward. Pinned event tape. Exact clone() for group rollouts.
Agentic reinforcement learning needs more than a chat harness. You need an environment where the model's decisions have economic consequences, where success cannot be gamed, and where the benchmark means something to your customers.
Sales Gym is that environment. It converts enterprise selling — a messy human process — into a verifiable, reproducible RL problem: allocate finite capacity across a portfolio to maximise evidence-backed, quality-adjusted progression per constrained hour.
The result is a benchmark you can train on, publish on, and defend — because the scoring is independent of the policy.
Opportunities progress through 8 verified milestones. Only immutable buyer evidence advances milestones — CRM edits, stage inflation, and fabricated claims yield zero reward. Each milestone has an anti-gaming condition.
11 Evidence Types
Buyer reply, meeting acceptance, budget confirmation, authorised sponsor, technical success, mutual action plan, security/legal/procurement approval, contract signature, disqualification reason.
18 Tool Actions
Research, contact, schedule, evaluate, business case, proposal, negotiate, escalate. Each consumes time and resources with explicit opportunity cost.
Shared Resources
Seller, SE, legal, procurement, management capacity with booking lead times, deadlines, and preemption costs.
Anti-Gaming Conditions
Stage cycling, CRM-only edits, fabricated evidence, unauthorised discounts, premature disqualification — all zero or negative reward.
Progression-first and potential-based. Reward attaches to verified buyer progress, not activity. The policy never observes reward coefficients or verifier internals — the terminal score is authoritative and independent.
Potential-based shaping with time discount: r_progress = γΔt · Φ(St+1) − Φ(St). Weights configurable via RewardConfig. Anti-exploit policies (spam_all, inflate_stages, always_discount, cycle_stages) verified not to beat a legitimate reference policy.
Frozen, hashed, versioned train/eval/held-out splits across scenario families and generative templates — not random seeds. Baseline suite and scoring CLI included. Confidence intervals bootstrapped by territory, not by action.
Required baselines: random valid action, CRM-stage priority, expected-value heuristic, earliest-deadline, greedy single-opportunity, base model prompting, SFT, and GRPO-trained. Primary metric: verified value-weighted advances per 100 constrained seller hours.
Wired for the reinforcement learning loop you already run. Framework-neutral rollout adapter, independent group clones per world seed, token-aligned trajectory metadata, and pinned configs for TRL and verl.
Rollout Adapter
Framework-neutral. Drives a model client through tool-call episodes; returns prompt tokens, generated tokens, action masks, tool events, group IDs, reward.
Independent Clones
Same world_seed + territory_id + event tape per group; separate mutable state, isolated SQLite / copy-on-write per rollout.
TRL + verl
Pinned releases documented. LoRA config for Qwen2.5-1.5B-Instruct included. FastAPI server for remote evaluation.
The Gym is built on rigorous RL environment design and answers specific research hypotheses — and to keep you from over-claiming. Five research hypotheses are pre-registered in the environment spec:
Scientific Protocol
Freeze manifests before training. Matched seeds and event tapes. Bootstrap by territory. Report median, mean, CI, failure modes, compute cost.
Ablations Defined
No throughput reward, no shared resources, no hidden buyer state, no curriculum, no failure mining, no terminal economics.
Contamination Audit
Splits by templates, graphs, mechanisms. External benchmarks in read-only workspace. Hashed, versioned, auditable.
Installable Package
spaceswordai-sales-gym on PyPI. Pure Python, framework-neutral. Local execution and optional HTTP service.
World Generators
Procedural territory generator with provenance-tagged synthetic data. No scraped personal data; deterministic per seed.
Private Verifier
MilestoneVerifier with evidence proofs. Independent of the policy. Terminal score is authoritative.
ESPB CLI
Frozen splits, scoring, baseline suite, bootstrap CIs. The same numbers run on your cluster and ours.
Training Configs
GRPO + LoRA for Qwen2.5-1.5B. TRL and verl adapters. Rollout recorder with token alignment.
Documentation
Architecture, reward design, tool contracts, research manifest, licensing, audit logs. Build spec v2.0.
Everything a researcher needs to evaluate the environment before committing compute is documented: architecture, reward design, tool contracts, benchmark protocol, and research manifest. No black boxes.
Architecture Spec
Entity model, state transitions, confidentiality boundary, scheduler rules, observation permissions. Build spec v2.0.
Reward Design Doc
Potential-based shaping, milestone weights, anti-exploit verification, counterfactual reward testing methodology.
Tool Contracts
18 tools with parameters, side effects, failure modes, resource costs, and JSON-Schema exports (OpenAI-compatible).
Benchmark Protocol
ESPB frozen splits, required baselines, primary metrics, ablation list, bootstrap CI method, contamination audit.
Research Manifest
Pinned external references (CRMArena-Pro, TRL, verl, SCUBA, SalesLLM), licence checks, and reproducibility notes.
Ethics & Scope
Synthetic provenance-tagged data only. No scraped personal data. Transfer-before-claims principle documented.
What is Sales Gym?
A reproducible reinforcement learning environment for training and benchmarking tool-using agents on enterprise sales portfolio management. Agentic RL with economic structure, not a chatbot demo.
What makes it a good RL benchmark?
Frozen hashed train/eval/held-out splits, a private verifier independent of the policy, deterministic replay per seed, shared-resource contention, and anti-exploit verified reward design.
Can I wire it into my training loop?
Yes. Framework-neutral rollout adapter, independent group clones per world seed, GRPO + LoRA configs for Qwen2.5-1.5B, and pinned TRL / verl integration paths.
What models does it support?
Any model that emits structured tool calls. The tool API exports JSON Schema in OpenAI-compatible format; the environment is provider-neutral.
Is it reproducible?
Same seed + same actions → identical state and reward. Append-only event log, pinned event tape, exact clone() for group rollouts.
What is the benchmark protocol?
Freeze manifests before training, matched seeds and event tapes, bootstrap confidence intervals by territory, report median/mean/CI/failure modes/compute cost. ESPB protocol documented in the build spec.
Sales Gym is released under MIT for the core environment. ESPB benchmark splits, world generators, and the private verifier are available under a research license for model studios.
Request access to get the full build spec, benchmark manifest, and a walkthrough of how it wires into your training loop.