Troubleshooting

Common failures, what causes them, and what to do. If you are looking for day-to-day operational procedures instead, see the runbook.

Install

What is my controller Deployment called?

Two, depending on how you installed. Every kubectl command in these docs assumes one of them, so check before you copy one:

Install method	Deployment name
kubectl apply -f .../deploy/orka.yaml	orka-controller-manager
helm install orka ...	orka-controller

The Helm name is <release-name>-controller, so it changes if you name the release something other than orka. When in doubt:

kubectl -n orka-system get deploy

Helm refuses to render

The chart validates its inputs before producing any manifests, so a bad install fails at helm install rather than at 3 a.m. The message names the value it wants. The common ones:

Message mentions	What it wants
agentExecutionSnapshot.existingSecret	Immutable once installed. Keep the name the live controller mounts. See below.
watchNamespace	Must be set, and must equal the release namespace.
providerProxy	When enabled, provide your gateway endpoint and egress rules. It can stay disabled during installation. See Provider proxy.
upstreamBaseURL	Your gateway's HTTP(S) URL, without credentials, a query, or a fragment.
image	Use a valid tag or SHA256 digest. Runtime overrides need a full registry/repository reference.
replicas / leaderElect	Must be 1 and true. The controller is a single writer.
caBundle or caInjectionAnnotations	You set webhooks.tls.existingSecret, so the chart needs the CA that signed it. Leave both empty to let the controller issue its own certificate.
mode	Only harness-v2 is supported.

The snapshot key

controller.agentExecutionSnapshot encrypts stored agent execution records. A fresh install generates the key into a Secret named <release>-agent-execution-snapshot and reuses it on every upgrade, so you normally never touch it. To bring your own, create a Secret containing either 32 raw bytes or their base64 encoding before installing, and pass its name as existingSecret:

kubectl -n orka-system create secret generic orka-agent-snapshot-key \
  --from-literal=key="$(openssl rand -base64 32)"

Do not rotate this casually

The Secret name, the item key, and the key material must stay the same for the life of the release. Changing any of them makes every retained snapshot permanently unreadable.

The chart guards only two of those three. On upgrade it compares the Secret name and item key against the live Deployment and fails if either changed. That includes switching between a generated Secret and your own. It cannot see the key material, so replacing the bytes under the same name and key passes the guard silently — and the controller then restarts unable to read any snapshot it wrote before. The generated Secret is kept on helm uninstall and reused by a reinstall under the same release name for the same reason. Treat the material as immutable yourself, and never use helm upgrade --force.

the SQLite dataset requires an existing key but snapshot key Secret ... has no "key" item

The controller-generated snapshot key is gone, but the SQLite dataset it belongs to already completed key bootstrap or was adopted from an older layout during an upgrade. The dataset records that history itself, so this applies even if every ordinary record has since been deleted. The controller refuses to mint a new key that could not read existing records. Restore the Secret from backup. If the data is expendable, delete the data volume and restart the controller to start over. See Dataset identity and key bootstrap.

synchronize SQLite dataset metadata: checkpoint incomplete

At startup the controller forces a full SQLite checkpoint so dataset metadata is durable before it loads the key. The checkpoint could not finish because another process holds the database open, for example a sqlite3 shell or a backup job reading the data volume. Stop that process and restart the controller. Nothing is lost: committed metadata is kept, and startup resumes from it.

invalid SQLite dataset identity

The dataset metadata row inside the database is malformed. If the row is missing, the message is SQLite dataset identity must contain exactly one record instead. Either usually means the database was edited outside Orka. The controller never regenerates this row, because it decides whether a new snapshot key may be minted. Restore the data volume and its key from backup rather than editing the row by hand.

unsupported SQLite schema version N

The database's schema version marker is one this controller does not support. Usually a newer Orka release wrote the database and the controller image was then downgraded. Migration is forward-only, so run the release that wrote the database. Rolling back further needs a coordinated restore of the database, its original key, configuration, and Kubernetes state; see Automatic pre-upgrade backup. The controller leaves the database unchanged.

The controller crashes immediately

Check the logs first. The Deployment name depends on how you installed, so discover it rather than guessing:

# The two installs label the controller differently, so try both.
# Helm sets app.kubernetes.io/component=controller; the release manifest
# sets control-plane=controller-manager and no component label at all.
CONTROLLER="$(kubectl -n orka-system get deploy \
  -l app.kubernetes.io/component=controller \
  -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)"
CONTROLLER="${CONTROLLER:-$(kubectl -n orka-system get deploy \
  -l control-plane=controller-manager \
  -o jsonpath='{.items[0].metadata.name}')}"

kubectl -n orka-system logs "deploy/$CONTROLLER" --previous

That is orka-controller for a Helm release named orka, and orka-controller-manager for the release manifest, as the table above shows.

unable to resolve ACP runtime images

At startup the controller resolves the coding-agent runtime image tags to digests by asking the registry, ghcr.io for the default images, over HTTPS. If the controller Pod cannot reach it, through a proxy or an egress policy, the controller stays up but every coding-agent runtime is unavailable until it restarts: AI and container Tasks work, agent Tasks fail with an unavailable runtime. Either allow that access, or pin the runtime images to digests so no lookup is needed, or set the images to empty strings to run without coding agents, then restart the controller:

helm upgrade orka orka/orka --namespace orka-system --reuse-values \
  --set-string controller.acpRuntime.codexImage= \
  --set-string controller.acpRuntime.claudeImage= \
  --set-string controller.acpRuntime.copilotImage= \
  --set-string controller.acpRuntime.opencodeImage=

See Image overrides.

--watch-namespace is required; controller modes cannot use a cluster-wide watch

Orka watches exactly one namespace. Set controller.watchNamespace to the release namespace.

controller-mode namespace claim failed or namespace "...": unsupported execution mode "..."

The namespace already carries an orka.ai/controller-mode label other than harness-v2. The controller claims an unlabeled namespace on its own, but it never takes over a namespace claimed for another mode, and the label cannot be changed once set. Install into a new namespace. If you run the controller with --claim-namespace-mode=false, label the namespace yourself before starting it:

kubectl label namespace orka-system orka.ai/controller-mode=harness-v2

Do not relabel a namespace another installation owns.

unable to read controller-mode namespace

The namespace does not exist yet, or the controller's ServiceAccount cannot read it.

Tasks

My Task stays Pending

In order of likelihood:

1. Wrong namespace. Orka only sees its watch namespace. A Task in default is never reconciled and produces no error — it just sits there.

   kubectl get task '<name>' -A          # where did it actually land?
   kubectl -n orka-system get tasks    # where Orka is looking

2. The Agent or Provider is missing, or is in a different namespace. Check kubectl -n orka-system describe task <name> for the reason.

3. Concurrency limit reached. If maxTasksPerNamespace is set, new Tasks queue. kubectl -n orka-system get tasks shows how many are Running.

4. No pool capacity for a type: agent Task. kubectl -n orka-system get runtimepools — a pool that is not Serving and Accepting will not take new sessions.

A type: agent Task fails immediately

The runtime image was not configured. If you did not set controller.acpRuntime.codexImage (or claude/copilot/opencode), that runtime is unavailable and Tasks asking for it fail rather than falling back to another. This is intentional.

The provider proxy is not reachable or not ready. Built-in coding agents need the optional provider proxy enabled and connected to your gateway. Check that gateway's readiness and network access, then check Orka's proxy:

kubectl -n orka-system get deploy -l app.kubernetes.io/component=provider-auth-proxy

See Provider proxy.

Codex with allowBash: false. The Codex CLI has no reliable shell-disable mode, so Orka fails fast instead of pretending the restriction is enforced. Set defaultAllowBash: true.

OpenCode without contextWindow and maxTokens. Both are required and must be set on the Agent.

go: command not found or read-only filesystem errors

The Pod filesystem is read-only outside /tmp, /home/worker, and /workspace, and bash -lc breaks official language images. Both are covered in Container tasks.

A pool will not replace or scale down

A pool drains before it is replaced: in-flight sessions finish, new ones are refused. If it is stuck, something is still holding a session. Check kubectl -n orka-system describe runtimepool <name> for the sessions it is waiting on. Deleting the Pods directly does not help — the controller will wait for the drain regardless.

API access

I get 403 from the API

Namespace mismatch. The API serves exactly one namespace. Asking for another returns namespace not allowed, with the requested and allowed namespaces in the controller log.

The token has no namespace. With enforceNamespaceIsolation on (the default), authenticated callers must carry a namespace. A token from a ServiceAccount in the wrong namespace, or a user token with no namespace at all, is rejected.

The caller lacks the requested permission. Kubernetes TokenReview callers need RBAC permission for each protected API operation, checked through SubjectAccessReview. This includes Agent creation, chat, approval decisions, and stored records such as sessions and memories. Nested chat tools need their own resource permissions. A successful authentication check does not grant access to those actions. OIDC and transaction tokens retain their separate policies; see API authorization.

The Helm chart creates a client ServiceAccount and namespace-scoped Role. Raw-manifest and Kustomize installs do not create a client ServiceAccount. Current-source bundles include orka-api-viewer-role and orka-api-editor-role; for a client that can create Tasks and use chat, bind the editor role:

kubectl -n orka-system create serviceaccount orka-client

kubectl -n orka-system create rolebinding orka-client \
  --clusterrole=orka-api-editor-role --serviceaccount=orka-system:orka-client

Use orka-api-viewer-role for read-only access. The editor also permits Agent, Tool, and Provider changes, memory review/apply, and security actions. Use a narrower Role from the permission inventory when that exceeds the client's needs. Both helper roles keep Secrets and workspace-class use separate; nested tools that run Kubernetes workloads need explicit workload grants as well.

If a bundle does not include these helper roles, apply config/rbac/api_viewer_role.yaml and config/rbac/api_editor_role.yaml from a checkout containing the authorization update. Files applied directly use the unprefixed names api-viewer-role and api-editor-role; use those names in the RoleBinding instead.

GatewayClass reads require a separate cluster-scoped grant. If the client needs them, bind the installed GatewayClass viewer role:

kubectl create clusterrolebinding orka-client-gatewayclass-viewer \
  --clusterrole=orka-gatewayclass-viewer-role --serviceaccount=orka-system:orka-client

Sessions, chats, memories, and other stored records use virtual RBAC resources without CRDs. Their grants are included in the API helper roles.

kubectl create token orka-client fails

Add -n orka-system. The ServiceAccount lives in the watch namespace. If it is still not found, create it and its RBAC roles; raw-manifest and Kustomize installs do not create it.

Browser requests fail with a CORS error

ORKA_CORS_ALLOWED_ORIGINS controls the allowed origins and defaults to *. If you have narrowed it, add your origin. See Configuration.

Upgrades

Custom resources disappear or the controller reports unknown fields

Helm does not create or update CRDs during helm upgrade — a Helm behavior, not an Orka one. Apply the CRDs from the target chart yourself first. See Upgrading.

The chart refuses to upgrade

Several values are immutable for the life of a release: controller.mode, controller.watchNamespace, the snapshot key Secret and item key, the release fullname, and the ACP runtime namespace. Changing any of them means a new release, not an upgrade.

Gateways

Gateway state lives in the controller's SQLite store, and its failure modes are specific enough to have their own page — including the two ways to corrupt a backup. See Gateways.

Getting more detail

kubectl -n orka-system logs "deploy/$CONTROLLER" -f   # $CONTROLLER from above
kubectl -n orka-system describe task '<name>'
kubectl -n orka-system get events --sort-by=.lastTimestamp
orka task events '<name>'
orka task trace '<name>'

Durable per-Task history is in execution events, which survive Pod restarts and are the right thing to read when the logs have rotated away.

Set controller.logLevel=debug for more detail. If you think you have found a bug, open an issue.