diff --git a/.github/release.yml b/.github/release.yml new file mode 100644 index 000000000..aecad37b8 --- /dev/null +++ b/.github/release.yml @@ -0,0 +1,33 @@ +changelog: + exclude: + labels: + - skip-changelog + - dependencies + authors: + - dependabot + categories: + - title: Features + labels: + - feat + - enhancement + - title: Fixes + labels: + - fix + - bug + - title: Security + labels: + - security + - title: Documentation + labels: + - docs + - title: CI/CD + labels: + - ci + - devops + - title: Maintenance + labels: + - chore + - refactor + - title: Other + labels: + - "*" diff --git a/.github/workflows/main-branch-flow.md b/.github/workflows/main-branch-flow.md new file mode 100644 index 000000000..4e21a33c8 --- /dev/null +++ b/.github/workflows/main-branch-flow.md @@ -0,0 +1,265 @@ +# Main Branch Delivery Flows + +This document explains what runs when code is proposed to `dev`/`main`, merged to `main`, and released. + +Use this with: + +- [`docs/ci-map.md`](../../docs/ci-map.md) +- [`docs/pr-workflow.md`](../../docs/pr-workflow.md) +- [`docs/release-process.md`](../../docs/release-process.md) + +## Event Summary + +| Event | Main workflows | +| --- | --- | +| PR activity (`pull_request_target`) | `pr-intake-checks.yml`, `pr-labeler.yml`, `pr-auto-response.yml` | +| PR activity (`pull_request`) | `ci-run.yml`, `sec-audit.yml`, plus path-scoped workflows | +| Push to `dev`/`main` | `ci-run.yml`, `sec-audit.yml`, plus path-scoped workflows | +| Tag push (`v*`) | `pub-release.yml` publish mode, `pub-docker-img.yml` publish job | +| Scheduled/manual | `pub-release.yml` verification mode, `sec-codeql.yml`, `feature-matrix.yml`, `test-fuzz.yml`, `pr-check-stale.yml`, `pr-check-status.yml`, `ci-queue-hygiene.yml`, `sync-contributors.yml`, `test-benchmarks.yml`, `test-e2e.yml` | + +## Runtime and Docker Matrix + +Observed averages below are from recent completed runs (sampled from GitHub Actions on February 17, 2026). Values are directional, not SLA. + +| Workflow | Typical trigger in main flow | Avg runtime | Docker build? | Docker run? | Docker push? | +| --- | --- | ---:| --- | --- | --- | +| `pr-intake-checks.yml` | PR open/update (`pull_request_target`) | 14.5s | No | No | No | +| `pr-labeler.yml` | PR open/update (`pull_request_target`) | 53.7s | No | No | No | +| `pr-auto-response.yml` | PR/issue automation | 24.3s | No | No | No | +| `ci-run.yml` | PR + push to `dev`/`main` | 74.7s | No | No | No | +| `sec-audit.yml` | PR + push to `dev`/`main` | 127.2s | No | No | No | +| `workflow-sanity.yml` | Workflow-file changes | 34.2s | No | No | No | +| `pr-label-policy-check.yml` | Label policy/automation changes | 14.7s | No | No | No | +| `pub-docker-img.yml` (`pull_request`) | Docker build-input PR changes | 240.4s | Yes | Yes | No | +| `pub-docker-img.yml` (`push`) | tag push `v*` | 139.9s | Yes | No | Yes | +| `pub-release.yml` | Tag push `v*` (publish) + manual/scheduled verification (no publish) | N/A in recent sample | No | No | No | + +Notes: + +1. `pub-docker-img.yml` is the only workflow in the main PR/push path that builds Docker images. +2. Container runtime verification (`docker run`) occurs in PR smoke only. +3. Container registry push occurs on tag pushes (`v*`) only. +4. `ci-run.yml` "Build (Smoke)" builds Rust binaries, not Docker images. + +## Step-By-Step + +### 1) PR from branch in this repository -> `dev` + +1. Contributor opens or updates PR against `dev`. +2. `pull_request_target` automation runs (typical runtime): + - `pr-intake-checks.yml` posts intake warnings/errors. + - `pr-labeler.yml` sets size/risk/scope labels. + - `pr-auto-response.yml` runs first-interaction and label routes. +3. `pull_request` CI workflows start: + - `ci-run.yml` + - `feature-matrix.yml` (Rust/workflow path scope) + - `sec-audit.yml` + - `sec-codeql.yml` (if Rust/codeql paths changed) + - path-scoped workflows if matching files changed: + - `pub-docker-img.yml` (Docker build-input paths only) + - `docs-deploy.yml` (docs + README markdown paths; deploy contract guard enforces promotion + rollback ref policy) + - `workflow-sanity.yml` (workflow files only) + - `pr-label-policy-check.yml` (label-policy files only) + - `ci-change-audit.yml` (CI/security path changes) + - `ci-provider-connectivity.yml` (probe config/script/workflow changes) + - `ci-reproducible-build.yml` (Rust/build reproducibility paths) +4. In `ci-run.yml`, `changes` computes: + - `docs_only` + - `docs_changed` + - `rust_changed` + - `workflow_changed` +5. `build` runs for Rust-impacting changes. +6. On PRs, full lint/test/docs checks run by default for Rust-impacting changes: + - `lint` + - strict lint delta gate (inside `lint` job) + - `test` + - `flake-probe` (single-retry telemetry; optional block via `CI_BLOCK_ON_FLAKE_SUSPECTED`) + - `docs-quality` +7. If root license files (`LICENSE-APACHE`, `LICENSE-MIT`) changed, `license-file-owner-guard` allows only PR author `willsarg`. +8. `lint-feedback` posts actionable comment if lint/docs gates fail. +9. `CI Required Gate` aggregates results to final pass/fail. +10. Maintainer merges PR once checks and review policy are satisfied. +11. Merge emits a `push` event on `dev` (see scenario 4). + +### 2) PR from fork -> `dev` + +1. External contributor opens PR from `fork/` into `zeroclaw:dev`. +2. Immediately on `opened`: + - `pull_request_target` workflows start with base-repo context and base-repo token: + - `pr-intake-checks.yml` + - `pr-labeler.yml` + - `pr-auto-response.yml` + - `pull_request` workflows are queued for the fork head commit: + - `ci-run.yml` + - `sec-audit.yml` + - path-scoped workflows (`pub-docker-img.yml`, `workflow-sanity.yml`, `pr-label-policy-check.yml`) if changed files match. +3. Fork-specific permission behavior in `pull_request` workflows: + - token is restricted (read-focused), so jobs that try to write PR comments/status extras can be limited. + - secrets from the base repo are not exposed to fork PR `pull_request` jobs. +4. Approval gate possibility: + - if Actions settings require maintainer approval for fork workflows, the `pull_request` run stays in `action_required`/waiting state until approved. +5. Event fan-out after labeling: + - manual label changes emit `labeled`/`unlabeled` events. + - those events retrigger only label-driven `pull_request_target` automation (`pr-auto-response.yml`); `pr-labeler.yml` now runs only on PR lifecycle events (`opened`/`reopened`/`synchronize`/`ready_for_review`) to reduce churn. +6. When contributor pushes new commits to fork branch (`synchronize`): + - reruns: `pr-intake-checks.yml`, `pr-labeler.yml`, `ci-run.yml`, `sec-audit.yml`, and matching path-scoped PR workflows. + - does not rerun `pr-auto-response.yml` unless label/open events occur. +7. `ci-run.yml` execution details for fork PR: + - `changes` computes `docs_only`, `docs_changed`, `rust_changed`, `workflow_changed`. + - `build` runs for Rust-impacting changes. + - `lint` (includes strict delta gate), `test`, and `docs-quality` run on PRs for Rust/docs-impacting changes without maintainer labels. + - `CI Required Gate` emits final pass/fail for the PR head. +8. Fork PR merge blockers to check first when diagnosing stalls: + - run approval pending for fork workflows. + - `license-file-owner-guard` failing when root license files are modified by non-owner PR author. + - `CI Required Gate` failure caused by upstream jobs. + - repeated `pull_request_target` reruns from label churn causing noisy signals. +9. After merge, normal `push` workflows on `dev` execute (scenario 4). + +### 3) PR to `main` (direct or from `dev`) + +1. Contributor or maintainer opens PR with base `main`. +2. `ci-run.yml` and `sec-audit.yml` run on the PR, plus any path-scoped workflows. +3. Maintainer merges PR once checks and review policy pass. +4. Merge emits a `push` event on `main`. + +### 4) Push/Merge Queue to `dev` or `main` (including after merge) + +1. Commit reaches `dev` or `main` (usually from a merged PR), or merge queue creates a `merge_group` validation commit. +2. `ci-run.yml` runs on `push` and `merge_group`. +3. `feature-matrix.yml` runs on `push` to `dev` for Rust/workflow paths and on `merge_group`. +4. `sec-audit.yml` runs on `push` and `merge_group`. +5. `sec-codeql.yml` runs on `push`/`merge_group` when Rust/codeql paths change (path-scoped on push). +6. `ci-supply-chain-provenance.yml` runs on push when Rust/build provenance paths change. +7. Path-filtered workflows run only if touched files match their filters. +8. In `ci-run.yml`, push/merge-group behavior differs from PR behavior: + - Rust path: `lint` (with strict delta gate), `test`, `build`, and binary-size regression (PR-only) are expected. + - Docs/non-rust paths: fast-path behavior applies. +9. `CI Required Gate` computes overall push/merge-group result. + +## Docker Publish Logic + +Workflow: `.github/workflows/pub-docker-img.yml` + +### PR behavior + +1. Triggered on `pull_request` to `dev` or `main` when Docker build-input paths change. +2. Runs `PR Docker Smoke` job: + - Builds local smoke image with Buildx builder. + - Verifies container with `docker run ... --version`. +3. Typical runtime in recent sample: ~240.4s. +4. No registry push happens on PR events. + +### Push behavior + +1. `publish` job runs on tag pushes `v*` only. +2. Workflow trigger includes semantic version tag pushes (`v*`) only. +3. Login to `ghcr.io` uses `${{ github.actor }}` and `${{ secrets.GITHUB_TOKEN }}`. +4. Tag computation includes semantic tag from pushed git tag (`vX.Y.Z`) + SHA tag (`sha-<12>`) + `latest`. +5. Multi-platform publish is used for tag pushes (`linux/amd64,linux/arm64`). +6. `scripts/ci/ghcr_publish_contract_guard.py` validates anonymous pullability and digest parity across `vX.Y.Z`, `sha-<12>`, and `latest`, then emits rollback candidate mapping evidence. +7. A pre-push Trivy gate scans the release-candidate image (`CRITICAL` blocks publish, `HIGH` is advisory). +8. After push, Trivy scans are emitted for version, SHA, and latest references. +9. `scripts/ci/ghcr_vulnerability_gate.py` validates Trivy JSON outputs against `.github/release/ghcr-vulnerability-policy.json` and emits audit-event evidence. +10. Typical runtime in recent sample: ~139.9s. +11. Result: pushed image tags under `ghcr.io//` with publish-contract + vulnerability-gate + scan artifacts. + +Important: Docker publish now requires a `v*` tag push; regular `dev`/`main` branch pushes do not publish images. + +## Release Logic + +Workflow: `.github/workflows/pub-release.yml` + +1. Trigger modes: + - Tag push `v*` -> publish mode. + - Manual dispatch -> verification-only or publish mode (input-driven). + - Weekly schedule -> verification-only mode. +2. `prepare` resolves release context (`release_ref`, `release_tag`, publish/draft mode) and runs `scripts/ci/release_trigger_guard.py`. + - publish mode enforces actor authorization, stable annotated tag policy, `origin/main` ancestry, and `release_tag` == `Cargo.toml` version at the tag commit. + - trigger provenance is emitted as `release-trigger-guard` artifacts. +3. `build-release` builds matrix artifacts across Linux/macOS/Windows targets. +4. `verify-artifacts` runs `scripts/ci/release_artifact_guard.py` against `.github/release/release-artifact-contract.json` in verify-stage mode (archive contract required; manifest/SBOM/notice checks intentionally skipped) and uploads `release-artifact-guard-verify` evidence. +5. In publish mode, workflow generates SBOM (`CycloneDX` + `SPDX`), `SHA256SUMS`, and a checksum provenance statement (`zeroclaw.sha256sums.intoto.json`) plus audit-event envelope. +6. In publish mode, after manifest generation, workflow reruns `release_artifact_guard.py` in full-contract mode and emits `release-artifact-guard.publish.json` plus `audit-event-release-artifact-guard-publish.json`. +7. In publish mode, workflow keyless-signs release artifacts and composes a supply-chain release-notes preface via `release_notes_with_supply_chain_refs.py`. +8. In publish mode, workflow verifies GHCR release-tag availability. +9. In publish mode, workflow creates/updates the GitHub Release for the resolved tag and commit-ish, combining generated supply-chain preface with GitHub auto-generated commit notes. + +Pre-release path: + +1. Pre-release tags (`vX.Y.Z-alpha.N`, `vX.Y.Z-beta.N`, `vX.Y.Z-rc.N`) trigger `.github/workflows/pub-prerelease.yml`. +2. `scripts/ci/prerelease_guard.py` enforces stage progression, `origin/main` ancestry, and Cargo version/tag alignment. +3. In publish mode, prerelease assets are attached to a GitHub prerelease for the stage tag. + +Canary policy lane: + +1. `.github/workflows/ci-canary-gate.yml` runs weekly or manually. +2. `scripts/ci/canary_guard.py` evaluates metrics against `.github/release/canary-policy.json`. +3. Decision output is explicit (`promote`, `hold`, `abort`) with auditable artifacts and optional dispatch signal. + +## Merge/Policy Notes + +1. Workflow-file changes (`.github/workflows/**`) are validated through `pr-intake-checks.yml`, `ci-change-audit.yml`, and `CI Required Gate` without a dedicated owner-approval gate. +2. PR lint/test strictness runs by default for Rust-impacting changes; no maintainer label is required. +3. `pr-intake-checks.yml` now blocks PRs missing a Linear issue key (`RMN-*`, `CDV-*`, `COM-*`) to keep execution mapped to Linear. +4. `sec-audit.yml` runs on PR/push/merge queue (`merge_group`), plus scheduled weekly. +5. `ci-change-audit.yml` enforces pinned `uses:` references for CI/security workflow changes. +6. `sec-audit.yml` includes deny policy hygiene checks (`deny_policy_guard.py`) before cargo-deny. +7. `sec-audit.yml` includes gitleaks allowlist governance checks (`secrets_governance_guard.py`) against `.github/security/gitleaks-allowlist-governance.json`. +8. `ci-reproducible-build.yml` and `ci-supply-chain-provenance.yml` provide scheduled supply-chain assurance signals outside release-only windows. +9. Some workflows are operational and non-merge-path (`pr-check-stale`, `pr-check-status`, `sync-contributors`, etc.). +10. Workflow-specific JavaScript helpers are organized under `.github/workflows/scripts/`. +11. `ci-run.yml` includes cache partitioning (`prefix-key`) across lint/test/build/flake-probe lanes to reduce cache contention. +12. `ci-rollback.yml` provides a guarded rollback planning lane (scheduled dry-run + manual execute controls) with audit artifacts. +13. `ci-queue-hygiene.yml` periodically deduplicates superseded queued runs for lightweight PR automation workflows to reduce queue pressure. + +## Mermaid Diagrams + +### PR to Dev + +```mermaid +flowchart TD + A["PR opened or updated -> dev"] --> B["pull_request_target lane"] + B --> B1["pr-intake-checks.yml"] + B --> B2["pr-labeler.yml"] + B --> B3["pr-auto-response.yml"] + A --> C["pull_request CI lane"] + C --> C1["ci-run.yml"] + C --> C2["sec-audit.yml"] + C --> C3["pub-docker-img.yml (if Docker paths changed)"] + C --> C4["workflow-sanity.yml (if workflow files changed)"] + C --> C5["pr-label-policy-check.yml (if policy files changed)"] + C1 --> D["CI Required Gate"] + D --> E{"Checks + review policy pass?"} + E -->|No| F["PR stays open"] + E -->|Yes| G["Merge PR"] + G --> H["push event on dev"] +``` + +### Main Delivery and Release + +```mermaid +flowchart TD + D0["Commit reaches dev"] --> B0["ci-run.yml"] + D0 --> C0["sec-audit.yml"] + PRM["PR to main"] --> QM["ci-run.yml + sec-audit.yml (+ path-scoped)"] + QM --> M["Merge to main"] + M --> A["Commit reaches main"] + A --> B["ci-run.yml"] + A --> C["sec-audit.yml"] + A --> D["path-scoped workflows (if matched)"] + T["Tag push v*"] --> R["pub-release.yml"] + W["Manual/Scheduled release verify"] --> R + T --> DP["pub-docker-img.yml publish job"] + R --> R1["Artifacts + SBOM + checksums + signatures + GitHub Release"] + W --> R2["Verification build only (no GitHub Release publish)"] + DP --> P1["Push ghcr image tags (version + sha + latest)"] +``` + +## Quick Troubleshooting + +1. Unexpected skipped jobs: inspect `scripts/ci/detect_change_scope.sh` outputs. +2. CI/CD-change PR blocked: verify `@chumyin` approved review is present. +3. Fork PR appears stalled: check whether Actions run approval is pending. +4. Docker not published: confirm a `v*` tag was pushed to the intended commit. diff --git a/.github/workflows/test-coverage.yml b/.github/workflows/test-coverage.yml new file mode 100644 index 000000000..d7c899636 --- /dev/null +++ b/.github/workflows/test-coverage.yml @@ -0,0 +1,106 @@ +name: Test Coverage + +on: + push: + branches: [dev, main] + paths: + - "Cargo.toml" + - "Cargo.lock" + - "src/**" + - "crates/**" + - "tests/**" + - ".github/workflows/test-coverage.yml" + pull_request: + branches: [dev, main] + paths: + - "Cargo.toml" + - "Cargo.lock" + - "src/**" + - "crates/**" + - "tests/**" + - ".github/workflows/test-coverage.yml" + workflow_dispatch: + +concurrency: + group: test-coverage-${{ github.event.pull_request.number || github.ref || github.run_id }} + cancel-in-progress: true + +permissions: + contents: read + +env: + GIT_CONFIG_COUNT: "1" + GIT_CONFIG_KEY_0: core.hooksPath + GIT_CONFIG_VALUE_0: /dev/null + CARGO_TERM_COLOR: always + +jobs: + coverage: + name: Coverage (non-blocking) + runs-on: [self-hosted, Linux, X64, aws-india, blacksmith-2vcpu-ubuntu-2404, hetzner] + timeout-minutes: 90 + env: + CARGO_HOME: ${{ github.workspace }}/.ci-rust/${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}/cargo + RUSTUP_HOME: ${{ github.workspace }}/.ci-rust/${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}/rustup + CARGO_TARGET_DIR: ${{ github.workspace }}/.ci-rust/${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}/target + steps: + - name: Checkout + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + + - name: Self-heal Rust toolchain cache + shell: bash + run: ./scripts/ci/self_heal_rust_toolchain.sh 1.92.0 + + - uses: dtolnay/rust-toolchain@631a55b12751854ce901bb631d5902ceb48146f7 # stable + with: + toolchain: 1.92.0 + components: llvm-tools-preview + + - id: rust-cache + uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v3 + with: + prefix-key: test-coverage + cache-bin: false + + - name: Install cargo-llvm-cov + shell: bash + run: cargo install cargo-llvm-cov --locked --version 0.6.16 + + - name: Run coverage (non-blocking) + id: cov + shell: bash + run: | + set -euo pipefail + mkdir -p artifacts + set +e + cargo llvm-cov --workspace --all-features --lcov --output-path artifacts/lcov.info + status=$? + set -e + + if [ "$status" -eq 0 ]; then + echo "coverage_ok=true" >> "$GITHUB_OUTPUT" + else + echo "coverage_ok=false" >> "$GITHUB_OUTPUT" + echo "::warning::Coverage generation failed (non-blocking)." + fi + + - name: Publish coverage summary + if: always() + shell: bash + run: | + set -euo pipefail + { + echo "### Coverage Lane (non-blocking)" + echo "- Coverage generation success: \`${{ steps.cov.outputs.coverage_ok || 'false' }}\`" + echo "- rust-cache hit: \`${{ steps.rust-cache.outputs.cache-hit || 'unknown' }}\`" + echo "- Artifact: \`artifacts/lcov.info\` (when available)" + } >> "$GITHUB_STEP_SUMMARY" + + - name: Upload coverage artifact + if: always() + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 + with: + name: coverage-lcov + path: artifacts/lcov.info + if-no-files-found: ignore + retention-days: 14 diff --git a/README.md b/README.md index 91ddc7fc7..044185093 100644 --- a/README.md +++ b/README.md @@ -81,6 +81,62 @@ Use this board for important notices (breaking changes, security advisories, mai - **Fully swappable:** core systems are traits (providers, channels, tools, memory, tunnels). - **No lock-in:** OpenAI-compatible provider support + pluggable custom endpoints. +## Quick Start + +### Option 1: Homebrew (macOS/Linuxbrew) + +```bash +brew install zeroclaw +``` + +### Option 2: Clone + Bootstrap + +```bash +git clone https://github.com/zeroclaw-labs/zeroclaw.git +cd zeroclaw +./bootstrap.sh +``` + +On Windows PowerShell: + +```powershell +git clone https://github.com/zeroclaw-labs/zeroclaw.git +cd zeroclaw +.\bootstrap.ps1 +``` + +> **Note:** Source builds require ~2GB RAM and ~6GB disk. For resource-constrained systems, use `./bootstrap.sh --prefer-prebuilt` to download a pre-built binary instead. + +### Option 3: Cargo Install + +```bash +cargo install zeroclaw +``` + +### First Run + +```bash +# Start the gateway (serves the Web Dashboard API/UI) +zeroclaw gateway + +# Open the dashboard URL shown in startup logs +# (default: http://127.0.0.1:3000/) + +# Or chat directly +zeroclaw chat "Hello!" +``` + +For detailed setup options, see [docs/one-click-bootstrap.md](docs/one-click-bootstrap.md). + +### Installation Docs (Canonical Source) + +Use repository docs as the source of truth for install/setup instructions: + +- [README Quick Start](#quick-start) +- [docs/one-click-bootstrap.md](docs/one-click-bootstrap.md) +- [docs/getting-started/README.md](docs/getting-started/README.md) + +Issue comments can provide context, but they are not canonical installation documentation. ## Benchmark Snapshot (ZeroClaw vs OpenClaw, Reproducible) Local machine quick benchmark (macOS arm64, Feb 2026) normalized for 0.8GHz edge hardware. diff --git a/bootstrap.ps1 b/bootstrap.ps1 new file mode 100644 index 000000000..73571d11e --- /dev/null +++ b/bootstrap.ps1 @@ -0,0 +1,214 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Windows bootstrap entrypoint for ZeroClaw. + +.DESCRIPTION + Provides the core bootstrap flow for native Windows: + - optional Rust toolchain install + - optional prebuilt binary install + - source build + cargo install fallback + - optional onboarding + + This script is intentionally scoped to Windows and does not replace + Docker/bootstrap.sh flows for Linux/macOS. +#> + +[CmdletBinding()] +param( + [switch]$InstallRust, + [switch]$PreferPrebuilt, + [switch]$PrebuiltOnly, + [switch]$ForceSourceBuild, + [switch]$SkipBuild, + [switch]$SkipInstall, + [switch]$Onboard, + [switch]$InteractiveOnboard, + [string]$ApiKey = "", + [string]$Provider = "openrouter", + [string]$Model = "" +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = "Stop" + +function Write-Info { + param([string]$Message) + Write-Host "==> $Message" +} + +function Write-Warn { + param([string]$Message) + Write-Warning $Message +} + +function Ensure-RustToolchain { + if (Get-Command cargo -ErrorAction SilentlyContinue) { + Write-Info "cargo is already available." + return + } + + if (-not $InstallRust) { + throw "cargo is not installed. Re-run with -InstallRust or install Rust manually from https://rustup.rs/" + } + + Write-Info "Installing Rust toolchain via rustup-init.exe" + $tempDir = Join-Path $env:TEMP "zeroclaw-bootstrap-rustup" + New-Item -ItemType Directory -Path $tempDir -Force | Out-Null + $rustupExe = Join-Path $tempDir "rustup-init.exe" + Invoke-WebRequest -Uri "https://win.rustup.rs/x86_64" -OutFile $rustupExe + & $rustupExe -y --profile minimal --default-toolchain stable + + $cargoBin = Join-Path $env:USERPROFILE ".cargo\bin" + if (-not ($env:Path -split ";" | Where-Object { $_ -eq $cargoBin })) { + $env:Path = "$cargoBin;$env:Path" + } + + if (-not (Get-Command cargo -ErrorAction SilentlyContinue)) { + throw "Rust installation did not expose cargo in PATH. Open a new shell and retry." + } +} + +function Install-PrebuiltBinary { + $target = "x86_64-pc-windows-msvc" + $url = "https://github.com/zeroclaw-labs/zeroclaw/releases/latest/download/zeroclaw-$target.zip" + $tempDir = Join-Path $env:TEMP ("zeroclaw-prebuilt-" + [guid]::NewGuid().ToString("N")) + New-Item -ItemType Directory -Path $tempDir -Force | Out-Null + $archivePath = Join-Path $tempDir "zeroclaw-$target.zip" + $extractDir = Join-Path $tempDir "extract" + New-Item -ItemType Directory -Path $extractDir -Force | Out-Null + + try { + Write-Info "Downloading prebuilt binary: $url" + Invoke-WebRequest -Uri $url -OutFile $archivePath + Expand-Archive -Path $archivePath -DestinationPath $extractDir -Force + + $binary = Get-ChildItem -Path $extractDir -Recurse -Filter "zeroclaw.exe" | Select-Object -First 1 + if (-not $binary) { + throw "Downloaded archive does not contain zeroclaw.exe" + } + + $installDir = Join-Path $env:USERPROFILE ".cargo\bin" + New-Item -ItemType Directory -Path $installDir -Force | Out-Null + $dest = Join-Path $installDir "zeroclaw.exe" + Copy-Item -Path $binary.FullName -Destination $dest -Force + Write-Info "Installed prebuilt binary to $dest" + return $true + } + catch { + Write-Warn "Prebuilt install failed: $($_.Exception.Message)" + return $false + } + finally { + Remove-Item -Path $tempDir -Recurse -Force -ErrorAction SilentlyContinue + } +} + +function Invoke-SourceBuildInstall { + param( + [string]$RepoRoot + ) + + if (-not $SkipBuild) { + Write-Info "Running cargo build --release --locked" + & cargo build --release --locked + } + else { + Write-Info "Skipping build (-SkipBuild)" + } + + if (-not $SkipInstall) { + Write-Info "Running cargo install --path . --force --locked" + & cargo install --path . --force --locked + } + else { + Write-Info "Skipping cargo install (-SkipInstall)" + } +} + +function Resolve-ZeroClawBinary { + $cargoBin = Join-Path $env:USERPROFILE ".cargo\bin\zeroclaw.exe" + if (Test-Path $cargoBin) { + return $cargoBin + } + + $fromPath = Get-Command zeroclaw -ErrorAction SilentlyContinue + if ($fromPath) { + return $fromPath.Source + } + + return $null +} + +function Run-Onboarding { + param( + [string]$BinaryPath + ) + + if (-not $BinaryPath) { + throw "Onboarding requested but zeroclaw binary is not available." + } + + if ($InteractiveOnboard) { + Write-Info "Running interactive onboarding" + & $BinaryPath onboard --interactive + return + } + + $resolvedApiKey = $ApiKey + if (-not $resolvedApiKey) { + $resolvedApiKey = $env:ZEROCLAW_API_KEY + } + + if (-not $resolvedApiKey) { + throw "Onboarding requires -ApiKey (or ZEROCLAW_API_KEY) unless using -InteractiveOnboard." + } + + $cmd = @("onboard", "--api-key", $resolvedApiKey, "--provider", $Provider) + if ($Model) { + $cmd += @("--model", $Model) + } + Write-Info "Running onboarding with provider '$Provider'" + & $BinaryPath @cmd +} + +if ($IsLinux -or $IsMacOS) { + throw "bootstrap.ps1 is for Windows. Use ./bootstrap.sh on Linux/macOS." +} + +if ($PrebuiltOnly -and $ForceSourceBuild) { + throw "-PrebuiltOnly cannot be combined with -ForceSourceBuild." +} + +if ($InteractiveOnboard) { + $Onboard = $true +} + +$repoRoot = Split-Path -Parent $PSCommandPath +Set-Location $repoRoot + +Ensure-RustToolchain + +$didPrebuiltInstall = $false +if (($PreferPrebuilt -or $PrebuiltOnly) -and -not $ForceSourceBuild) { + $didPrebuiltInstall = Install-PrebuiltBinary + if ($PrebuiltOnly -and -not $didPrebuiltInstall) { + throw "Prebuilt-only mode requested but prebuilt install failed." + } +} + +if (-not $didPrebuiltInstall -and -not $PrebuiltOnly) { + Invoke-SourceBuildInstall -RepoRoot $repoRoot +} + +$zeroclawBin = Resolve-ZeroClawBinary +if (-not $zeroclawBin) { + throw "ZeroClaw binary was not found after bootstrap." +} + +Write-Info "ZeroClaw bootstrap completed." +Write-Info "Binary: $zeroclawBin" + +if ($Onboard) { + Run-Onboarding -BinaryPath $zeroclawBin +} diff --git a/docs/ci-map.md b/docs/ci-map.md index 5edd7c76f..48dcbc4d1 100644 --- a/docs/ci-map.md +++ b/docs/ci-map.md @@ -14,6 +14,7 @@ Merge-blocking checks should stay small and deterministic. Optional checks are u - Purpose: Rust validation (`cargo fmt --all -- --check`, `cargo clippy --locked --all-targets -- -D clippy::correctness`, strict delta lint gate on changed Rust lines, `test`, release build smoke) + docs quality checks when docs change (`markdownlint` blocks only issues on changed lines; link check scans only links added on changed lines) - Additional behavior: for Rust-impacting PRs and pushes, `CI Required Gate` requires `lint` + `test` + `restricted-hermetic` + `build` (no PR build-only bypass) - Additional behavior: includes `Restricted Hermetic Validation` lane (`./scripts/ci/restricted_profile.sh`) that runs a capability-aware subset with isolated `HOME`/workspace/config roots and no external provider credentials + - Additional behavior: PRs with Rust changes run a binary-size regression guard versus base commit (`check_binary_size_regression.sh`, default max increase 10%) - Additional behavior: rust-cache is partitioned per job role via `prefix-key` to reduce cache churn across lint/test/build/flake-probe lanes - Additional behavior: emits `test-flake-probe` artifact from single-retry probe when tests fail; optional blocking can be enabled with repository variable `CI_BLOCK_ON_FLAKE_SUSPECTED=true` - Additional behavior: PRs that change `.github/workflows/**` require at least one approving review from a login in `WORKFLOW_OWNER_LOGINS` (repository variable fallback: `theonlyhennygod,willsarg`) @@ -42,10 +43,12 @@ Merge-blocking checks should stay small and deterministic. Optional checks are u - Additional behavior: owner routing + escalation policy is documented in `docs/operations/nightly-all-features-runbook.md` - `.github/workflows/sec-audit.yml` (`Security Audit`) - Purpose: dependency advisories (`rustsec/audit-check`, pinned SHA), policy/license checks (`cargo deny`), gitleaks-based secrets governance (allowlist policy metadata + expiry guard), and SBOM snapshot artifacts (`CycloneDX` + `SPDX`) +- `.github/workflows/test-coverage.yml` (`Test Coverage`) + - Purpose: non-blocking coverage lane using `cargo-llvm-cov` with `lcov` artifact upload for trend tracking before hard-gating coverage - `.github/workflows/sec-codeql.yml` (`CodeQL Analysis`) - Purpose: static analysis for security findings on PR/push (Rust/codeql paths) plus scheduled/manual runs - `.github/workflows/ci-change-audit.yml` (`CI/CD Change Audit`) - - Purpose: machine-auditable diff report for CI/security workflow changes (line churn, new `uses:` references, unpinned action-policy violations, pipe-to-shell policy violations, broad `permissions: write-all` grants, new `pull_request_target` trigger introductions, new secret references) + - Purpose: machine-auditable diff report for CI/security workflow changes (line churn, new `uses:` references, unpinned action-policy violations, pipe-to-shell policy violations, broad `permissions: write-all` grants, unsafe workflow-script JS execution patterns, new `pull_request_target` trigger introductions, new secret references) - `.github/workflows/ci-provider-connectivity.yml` (`CI Provider Connectivity`) - Purpose: scheduled/manual/provider-list probe matrix with downloadable JSON/Markdown artifacts for provider endpoint reachability - `.github/workflows/ci-reproducible-build.yml` (`CI Reproducible Build`) @@ -97,6 +100,7 @@ Merge-blocking checks should stay small and deterministic. Optional checks are u - `Nightly All-Features`: daily schedule and manual dispatch - `Release`: tag push (`v*`), weekly schedule (verification-only), manual dispatch (verification or publish) - `Security Audit`: push to `dev` and `main`, PRs to `dev` and `main`, weekly schedule +- `Test Coverage`: push/PR on Rust paths to `dev` and `main`, manual dispatch - `Sec Vorpal Reviewdog`: manual dispatch only - `Workflow Sanity`: PR/push when `.github/workflows/**`, `.github/*.yml`, or `.github/*.yaml` change - `Dependabot`: all update PRs target `main` (not `dev`) @@ -118,7 +122,7 @@ Merge-blocking checks should stay small and deterministic. Optional checks are u 6. PR intake failures: inspect `.github/workflows/pr-intake-checks.yml` sticky comment and run logs. If intake policy changed recently, trigger a fresh `pull_request_target` event (for example close/reopen PR) because `Re-run jobs` can reuse the original workflow snapshot. 7. Label policy parity failures: inspect `.github/workflows/pr-label-policy-check.yml`. 8. Docs failures in CI: inspect `docs-quality` job logs in `.github/workflows/ci-run.yml`. -9. Strict delta lint failures in CI: inspect `lint-strict-delta` job logs and compare with `BASE_SHA` diff scope. +9. Strict delta lint failures in CI: inspect the `lint` job logs (`Run strict lint delta gate` step) and compare with `BASE_SHA` diff scope. ## Maintenance Rules diff --git a/docs/one-click-bootstrap.md b/docs/one-click-bootstrap.md index f2d8ddb37..8bf64269e 100644 --- a/docs/one-click-bootstrap.md +++ b/docs/one-click-bootstrap.md @@ -2,7 +2,7 @@ This page defines the fastest supported path to install and initialize ZeroClaw. -Last verified: **February 20, 2026**. +Last verified: **March 5, 2026**. ## Option 0: Homebrew (macOS/Linuxbrew) @@ -18,6 +18,14 @@ cd zeroclaw ./bootstrap.sh ``` +Windows PowerShell equivalent: + +```powershell +git clone https://github.com/zeroclaw-labs/zeroclaw.git +cd zeroclaw +.\bootstrap.ps1 +``` + What it does by default: 1. `cargo build --release --locked` @@ -65,6 +73,7 @@ Notes: - `--prefer-prebuilt` tries release binary download first, then falls back to source build. - `--prebuilt-only` disables source fallback. - `--force-source-build` disables pre-built flow entirely. +- On Windows, use `bootstrap.ps1` (`-InstallRust`, `-PreferPrebuilt`, `-PrebuiltOnly`, `-ForceSourceBuild`). ## Option B: Remote one-liner diff --git a/docs/operations/branch-protection.md b/docs/operations/branch-protection.md new file mode 100644 index 000000000..30242d269 --- /dev/null +++ b/docs/operations/branch-protection.md @@ -0,0 +1,61 @@ +# Branch Protection Baseline + +This document is the repository-side baseline for branch protection on `dev` and `main`. +It should be updated whenever branch/ruleset policy changes. + +## Baseline Date + +- Baseline updated: 2026-03-05 (UTC) + +## Protected Branches + +- `dev` +- `main` + +## Required Checks + +Required check names are versioned in [required-check-mapping.md](./required-check-mapping.md). +At minimum, protect both branches with: + +- `CI Required Gate` +- `Security Audit` +- `Feature Matrix Summary` +- `Workflow Sanity` + +## Required Branch Rules + +- Require a pull request before merging. +- Require status checks before merging. +- Require at least one approving review. +- Require CODEOWNERS review for protected paths. +- Dismiss stale approvals on new commits. +- Restrict force-pushes. +- Restrict bypass access to org owners/admins only. + +## Export Procedure + +Export live policy snapshots whenever branch protection changes: + +```bash +mkdir -p docs/operations/branch-protection +gh api repos/zeroclaw-labs/zeroclaw/branches/dev/protection \ + > docs/operations/branch-protection/dev-protection.json +gh api repos/zeroclaw-labs/zeroclaw/branches/main/protection \ + > docs/operations/branch-protection/main-protection.json +``` + +If your org uses repository rulesets, also export: + +```bash +gh api repos/zeroclaw-labs/zeroclaw/rulesets \ + > docs/operations/branch-protection/rulesets.json +``` + +## Validation Checklist + +After updating branch protection: + +1. Confirm required check names exactly match [required-check-mapping.md](./required-check-mapping.md). +2. Confirm merge queue compatibility for required workflows (`merge_group` on merge-critical workflows). +3. Confirm direct pushes are blocked for non-admin users. +4. Commit updated JSON snapshots under `docs/operations/branch-protection/`. diff --git a/docs/operations/branch-protection/README.md b/docs/operations/branch-protection/README.md new file mode 100644 index 000000000..a272c6c06 --- /dev/null +++ b/docs/operations/branch-protection/README.md @@ -0,0 +1,10 @@ +# Branch Protection Snapshots + +This directory stores exported branch protection and ruleset JSON snapshots: + +- `dev-protection.json` +- `main-protection.json` +- `rulesets.json` (when repository rulesets are enabled) + +Generate snapshots with the commands documented in +[../branch-protection.md](../branch-protection.md). diff --git a/docs/operations/ci-cd-blacksmith-optimization-report-2026-03-05.md b/docs/operations/ci-cd-blacksmith-optimization-report-2026-03-05.md new file mode 100644 index 000000000..9ba7361de --- /dev/null +++ b/docs/operations/ci-cd-blacksmith-optimization-report-2026-03-05.md @@ -0,0 +1,122 @@ +# CI/CD + Blacksmith Optimization Report + +Date: 2026-03-05 (UTC) + +## Scope + +This report summarizes repository changes applied to implement the CI/CD hardening +and performance plan across security, signal quality, and runtime throughput. + +## Implemented Changes + +### 1) Release supply-chain hardening + +- `pub-release.yml` already installs Syft via pinned installer script with checksum validation (`scripts/ci/install_syft.sh`). +- No remote `curl | sh` Syft install path remains in release workflow. + +### 2) Container vulnerability gate before push + +- Added pre-push Trivy gate in `.github/workflows/pub-docker-img.yml`. +- New behavior: + - build local release-candidate image (`linux/amd64`) + - block publish on `CRITICAL` findings + - report `HIGH` findings as advisory warnings +- Post-push Trivy evidence collection remains for release/sha/latest parity and audit artifacts. +- Updated policy: + - `.github/release/ghcr-vulnerability-policy.json` now blocks `CRITICAL` only. + - `docs/operations/ghcr-vulnerability-policy.md` updated accordingly. + +### 3) `pull_request_target` safety contract enforcement + +- Added explicit safety contract in `docs/actions-source-policy.md`. +- Extended `scripts/ci/ci_change_audit.py` policy checks to block newly introduced unsafe workflow-script JS patterns in `.github/workflows/scripts/**`: + - `eval(...)` + - `Function(...)` + - `vm.runInContext/runInNewContext/runInThisContext/new vm.Script` + - dynamic `child_process` execution APIs +- Added/updated tests in `scripts/ci/tests/test_ci_scripts.py`. + +### 4) Branch protection baseline documentation + +- Added `docs/operations/branch-protection.md` with: + - protected branch baseline (`dev`, `main`) + - required checks and branch rules + - export commands for live policy snapshots +- Added snapshot directory doc: + - `docs/operations/branch-protection/README.md` +- Linked baseline in: + - `docs/pr-workflow.md` + - `docs/operations/required-check-mapping.md` + +### 5) PR lint/test defaults and CI signal quality + +- `ci-run.yml` already runs lint/test/build by default for Rust-impacting PRs (no `ci:full` label requirement). +- Updated stale workflow docs (`.github/workflows/main-branch-flow.md`) to reflect actual behavior. + +### 6) Binary size guardrails (PR + release parity) + +- Added Windows binary size enforcement in `.github/workflows/pub-release.yml`. +- Added PR binary-size regression job in `.github/workflows/ci-run.yml`: + - compares PR head binary vs base SHA binary + - default max allowed increase: `10%` + - fails PR merge gate when threshold is exceeded +- Added helper script: + - `scripts/ci/check_binary_size_regression.sh` + +### 7) Blacksmith throughput and cache contention + +- Heavy CI jobs continue to run on Blacksmith-tagged runners. +- Scoped Docker build cache keys added in `.github/workflows/pub-docker-img.yml`: + - separate scopes for PR smoke and release publish paths + - reduced cache contention across event types. + +### 8) CI telemetry improvements + +- Added per-job telemetry summaries in `ci-run.yml` for lint/test/build/binary-size lanes: + - rust-cache hit/miss output + - job duration (seconds) +- Added binary-size regression summary output to step summary. + +### 9) Coverage follow-up (non-blocking) + +- Added non-blocking coverage workflow: + - `.github/workflows/test-coverage.yml` + - uses `cargo-llvm-cov` and uploads `lcov.info` + - does not gate merge by default. + +### 10) Developer experience follow-up + +- Added Windows bootstrap entrypoint: + - `bootstrap.ps1` +- Updated setup docs: + - `README.md` + - `docs/one-click-bootstrap.md` +- Added release note category config: + - `.github/release.yml` +- Updated release docs: + - `docs/release-process.md` + +## Validation Performed + +- Targeted CI policy tests: + - `python3 -m unittest -k ci_change_audit scripts.ci.tests.test_ci_scripts` + - result: pass (8 tests) + - note: executed with hooks disabled via: + - `GIT_CONFIG_COUNT=1` + - `GIT_CONFIG_KEY_0=core.hooksPath` + - `GIT_CONFIG_VALUE_0=/dev/null` +- Script syntax checks: + - `bash -n scripts/ci/check_binary_size_regression.sh` (pass) + - `python3 -m py_compile scripts/ci/ci_change_audit.py scripts/ci/ghcr_vulnerability_gate.py` (pass) +- Diff hygiene: + - `git diff --check` (pass) + +## Known Follow-up + +- Live branch protection JSON export is documented but not committed in this change set + because local `gh` authentication token is currently invalid. + After re-authentication, run export commands in `docs/operations/branch-protection.md` + and commit: + - `docs/operations/branch-protection/dev-protection.json` + - `docs/operations/branch-protection/main-protection.json` + - `docs/operations/branch-protection/rulesets.json` (if applicable) diff --git a/docs/operations/required-check-mapping.md b/docs/operations/required-check-mapping.md index fe4aba9a7..4eebb7e02 100644 --- a/docs/operations/required-check-mapping.md +++ b/docs/operations/required-check-mapping.md @@ -34,6 +34,7 @@ Feature matrix lane check names (informational, non-required): 2. Enumerate check/job names and compare to this mapping: - `gh run view --repo zeroclaw-labs/zeroclaw --json jobs --jq '.jobs[].name'` 3. If any merge-critical check name changed, update this file before changing branch protection policy. +4. Export and commit branch/ruleset snapshots as documented in `docs/operations/branch-protection.md`. ## Notes diff --git a/docs/pr-workflow.md b/docs/pr-workflow.md index 7f21b7269..31e0262d1 100644 --- a/docs/pr-workflow.md +++ b/docs/pr-workflow.md @@ -107,6 +107,7 @@ Maintain these branch protection rules on `dev` and `main`: - Restrict force-push on protected branches. - Route normal contributor PRs to `main` by default (`dev` is optional for dedicated integration batching). - Allow direct merges to `main` once required checks and review policy pass. +- Keep live export snapshots and policy baseline versioned in `docs/operations/branch-protection.md`. --- diff --git a/docs/release-process.md b/docs/release-process.md index feceea01b..1ca008c47 100644 --- a/docs/release-process.md +++ b/docs/release-process.md @@ -2,7 +2,7 @@ This runbook defines the maintainers' standard release flow. -Last verified: **February 25, 2026**. +Last verified: **March 5, 2026**. ## Release Goals @@ -44,6 +44,7 @@ Publish-mode guardrails: - Trigger provenance is recorded in `release-trigger-guard.json` and `audit-event-release-trigger-guard.json`. - Multi-arch artifact contract is enforced by `.github/release/release-artifact-contract.json` through `release_artifact_guard.py`. - Release notes include a generated supply-chain evidence preface (`release-notes-supply-chain.md`) plus GitHub-generated commit-window notes. +- GitHub release note categories are defined in `.github/release.yml` to keep generated notes grouped by label intent. ## Maintainer Procedure