Fullsend

Appearance

66. Behaviour tests with Gherkin and pluggable drivers ​

Date: 2026-06-07

Status ​

Accepted

Context ​

Fullsend needs end-to-end tests that validate deterministic platform behaviour — dispatch routing, harness loading, schema validation, post-scripts, token scoping, sandbox policy, and SCM mutations — without depending on LLM output. This is distinct from admin install e2e (ADR 0040) and from LLM/instruction testing (testing-agents.md).

Runtime selection is shared with production via defaults.runtime in org config.yaml (runtimes.md). Harness definitions remain as in ADR 0024. Per-repo install mode (ADR 0033) is the behaviour v1 default; per-org install is deferred.

Decision ​

  • Add behaviour tests under e2e/behaviour/ using godog and portable Gherkin feature files.
  • Exercise real SCM + real CI through driver interfaces (scm.Driver, ci.Driver, install.Driver); v1 implementations target GitHub and GitHub Actions.
  • Substitute inference with a dummy runtime (runtime: dummy in per-repo config, or defaults.runtime: dummy for per-org) that executes scripted operations in the real OpenShell sandbox and emits behaviour-results.json.
  • Select backends via runner env (BEHAVIOUR_SCM, BEHAVIOUR_CI, BEHAVIOUR_INSTALL_MODE); feature files stay install-mode agnostic. v1 runs per-repo against the halfsend org pool; the suite provisions fullsend via fullsend github setup rather than requiring pre-installed orgs.
  • Use compatibility tags (@skip:*, @requires:*) to filter scenarios for future backends; tags do not select configuration.

Consequences ​

  • Behaviour tests can pass while prompt quality regresses; LLM evals remain necessary for instruction coverage.
  • Behaviour orgs are provisioned at suite start with --runtime dummy; production orgs must not use dummy unintentionally.
  • Adding GitLab or Tekton requires new drivers and runner env values, not feature file rewrites.
  • Dummy runtime op vocabulary stays minimal; new ops require runtime + docs updates when scenarios need them.
  • Behaviour tests depend on live external infrastructure: GitHub API, GitHub Actions runners, GCP WIF/mint, and the shared halfsend org pool. Transient outages, API rate limits, or pool org state corruption can fail the suite; CI distinguishes infrastructure failures from regressions via workflow logs and artifact inspection, but there is no offline fallback.
  • Behaviour tests share the halfsend org pool and lock mechanism with admin e2e tests (e2e.yml runs both jobs). Lock hold time scales with scenario count; pool size was doubled to absorb the additional load and can be increased again if contention appears.