CI failure triage and fix

Started when a GitHub Actions workflow fails on a watched branch or pull request, or by hand with a run id. The run is diagnosed from its logs and the diagnosis can be posted on the pull request; in fix mode a code-fixable failure gets a draft pull request into the branch that failed, checked against the failing step’s own command when there is one and revised while its CI fails. Where an admin allows it, a pull request’s fix can instead be pushed onto its own branch. Never changes workflow files and never merges.

Template	ci-triage-and-fix
Works on	A git repository, cloned into an isolated Docker sandbox
Where a person acts	Nobody inside the run. The pull request it opens is merged by a person, outside it.
Waits on systems	CI result, up to 4h

The workflow

Drawn from the template spec at build time. Bookkeeping nodes that only record status are left out; amber nodes are where a person acts.

flowchart TD
  n_triage["Triage"]
  n_routeTriage{"Route triage?"}
  n_implement["Implement"]
  n_checkSkipped{"Check skipped?"}
  n_checkImplemented{"Check implemented?"}
  n_skipped(["Skipped"])
  n_reportDiagnosis["Report diagnosis"]
  n_reportFixFailed["Report fix failed"]
  n_checkChanged{"Check changed?"}
  n_diagnosed(["Done<br/><small>diagnosed</small>"])
  n_fixFailed(["Failed<br/><small>fix failed</small>"])
  n_runLint["Run lint"]
  n_runTypecheck["Run typecheck"]
  n_runTests["Run tests"]
  n_verifyFix["Verify fix"]
  n_storeVerification{"Store verification?"}
  n_routeDelivery{"Route delivery?"}
  n_pushFix["Push fix"]
  n_openPR["Open PR"]
  n_checkPushed{"Check pushed?"}
  n_reportFix["Report fix"]
  n_reportPushed["Report pushed"]
  n_loadCiWaitConfig["Load CI wait config"]
  n_watchPushedCi["Watch pushed CI"]
  n_routeCiWait{"Route CI wait?"}
  n_checkPushedCi{"Check pushed CI?"}
  n_pollForCI["Poll for CI"]
  n_waitForCI[/"Wait for CI<br/><small>waits up to 4h: CI result</small>"/]
  n_checkPushRetries{"Check push retries?"}
  n_finishPushed["Finish pushed"]
  n_checkCI{"Check CI?"}
  n_terminateCITimedOut(["Timed out<br/><small>ci timed out</small>"])
  n_reportPushFailed["Report push failed"]
  n_fetchPushedLogs["Fetch pushed logs"]
  n_pushed(["Done<br/><small>pushed</small>"])
  n_done(["Done"])
  n_checkCILimit{"Check CI limit?"}
  n_finishPushFailed["Finish push failed"]
  n_pushRetryFix["Push retry fix"]
  n_terminateCIFailed(["Failed<br/><small>ci failed</small>"])
  n_fetchLogs["Fetch logs"]
  n_pushFailed(["Failed<br/><small>push failed</small>"])
  n_checkPushRetryFix{"Check push retry fix?"}
  n_ciFix["CI fix"]
  n_pushFixAgain["Push fix again"]
  n_checkCiFixChanged{"Check CI fix changed?"}
  n_checkPushedAgain{"Check pushed again?"}
  n_repushAfterFix["Repush after fix"]
  n_triage --> n_routeTriage
  n_routeTriage -->|"yes"| n_implement
  n_routeTriage -->|"no"| n_checkSkipped
  n_implement --> n_checkImplemented
  n_checkSkipped -->|"yes"| n_skipped
  n_checkSkipped -->|"no"| n_reportDiagnosis
  n_checkImplemented -->|"yes"| n_reportFixFailed
  n_checkImplemented -->|"no"| n_checkChanged
  n_reportDiagnosis --> n_diagnosed
  n_reportFixFailed --> n_fixFailed
  n_checkChanged -->|"yes"| n_reportDiagnosis
  n_checkChanged -->|"no"| n_runLint
  n_runLint --> n_runTypecheck
  n_runTypecheck --> n_runTests
  n_runTests --> n_verifyFix
  n_verifyFix --> n_storeVerification
  n_storeVerification -->|"yes"| n_routeDelivery
  n_storeVerification -->|"no"| n_routeDelivery
  n_routeDelivery -->|"yes"| n_pushFix
  n_routeDelivery -->|"no"| n_openPR
  n_pushFix --> n_checkPushed
  n_openPR --> n_reportFix
  n_checkPushed -->|"yes"| n_reportPushed
  n_checkPushed -->|"no"| n_openPR
  n_reportFix --> n_loadCiWaitConfig
  n_reportPushed --> n_watchPushedCi
  n_loadCiWaitConfig --> n_routeCiWait
  n_watchPushedCi --> n_checkPushedCi
  n_routeCiWait -->|"yes"| n_pollForCI
  n_routeCiWait -->|"no"| n_waitForCI
  n_checkPushedCi -->|"yes"| n_checkPushRetries
  n_checkPushedCi -->|"no"| n_finishPushed
  n_pollForCI --> n_checkCI
  n_waitForCI -->|"received"| n_checkCI
  n_waitForCI -->|"timed out"| n_terminateCITimedOut
  n_checkPushRetries -->|"yes"| n_reportPushFailed
  n_checkPushRetries -->|"no"| n_fetchPushedLogs
  n_finishPushed --> n_pushed
  n_checkCI -->|"yes"| n_done
  n_checkCI -->|"no"| n_checkCILimit
  n_reportPushFailed --> n_finishPushFailed
  n_fetchPushedLogs --> n_pushRetryFix
  n_checkCILimit -->|"yes"| n_terminateCIFailed
  n_checkCILimit -->|"no"| n_fetchLogs
  n_finishPushFailed --> n_pushFailed
  n_pushRetryFix --> n_checkPushRetryFix
  n_fetchLogs --> n_ciFix
  n_checkPushRetryFix -->|"yes"| n_reportPushFailed
  n_checkPushRetryFix -->|"no"| n_pushFixAgain
  n_ciFix --> n_checkCiFixChanged
  n_pushFixAgain --> n_checkPushedAgain
  n_checkCiFixChanged -->|"yes"| n_checkCILimit
  n_checkCiFixChanged -->|"no"| n_repushAfterFix
  n_checkPushedAgain -->|"yes"| n_reportPushed
  n_checkPushedAgain -->|"no"| n_reportPushFailed
  n_repushAfterFix --> n_loadCiWaitConfig
  class n_waitForCI wait

Maturity: built from the flagship steps

Ships as a built-in template and reuses the engineering steps the flagship flow runs. This particular arrangement of them is not separately exercised end to end against a real repository. See product overview §8.

Use it

Every built-in template is seeded into the workflow library and can be cloned, edited on the visual canvas, and versioned — see architecture for the node types. The spec lives in packages/shared/src/workflow/templates/ciTriageAndFix.ts.