Skip to main content
This guide walks you through deploying the Qodo platform on Kubernetes using the Helm chart. Before installing, prepare the cluster and external dependencies. For more information, see Infrastructure requirements.
PostgreSQL prerequisite correction: the shipped base values expect an external PostgreSQL server. The bundled server is opt-in: set postgres.enabled: true and global.externalDatabase.host: qodo-postgres. The generated prerequisite paragraph below describes an earlier default. See Infrastructure requirements.
To collect application metrics with your monitoring stack, follow the configuration and verification steps. For more information, see Monitor Qodo with Prometheus and Grafana. For a complete list of Helm values and configuration options, see the Helm chart configuration reference. This chart installs the Qodo platform into a single Kubernetes namespace: the web portal, the API, the code-review agents, the code-intelligence engine, the git gateway, and their supporting datastores. Configuration is two values files: The complete list of values you can set is Helm chart configuration reference, with a ready-to-edit skeleton in values-customer-template.yaml. Both are generated from the chart, so they cannot drift from what it actually accepts.

Where the chart comes from

Two options. They install the same chart; pick one and follow it through, since they differ in one prerequisite. Option B, once per machine:
Same credentials as the image registry — one licence covers the chart and the images. Then use the oci:// URL wherever this page says ./qodo-onprem, and skip Secret 1 in step 4: a chart pulled this way carries your licence, and the chart turns it into the image-pull Secrets every workload needs. Creating qodo-pull as well does no harm, it is simply unused. To see available versions: helm show chart oci://artifacts-self-hosted.qodo.ai/codium-stack/qodo-onprem/qodo-onprem.

1. Prerequisites

Before working through this section, run ./qodo-check.sh -n <namespace> -f <your-values.yaml> and read READINESS.md: they verify these prerequisites for you and cover the backup, rollback and test-account items this guide does not. Run the same command again after installing, before opening traffic — it detects that Qodo is now installed and adds the post-installation checks.
  • Kubernetes 1.25+ and Helm 3, with a namespace you can install into.
  • An ingress controller (any: ingress-nginx, GCE, Traefik, …) and a DNS record you control — see step 2.
  • PostgreSQL. Nothing to prepare by default: the chart runs Postgres 16 with pgvector in-cluster, creates its own databases and generates its own admin password. To use your own database server instead — PostgreSQL 15 or 16 with the pgvector extension available — set externalDatabase.enabled: true and see step 5.
  • A StorageClass that provisions PersistentVolumes (ReadWriteOnce is enough). Postgres, Redis, RabbitMQ and the Qodo Git repo cache each claim one. If your cluster has no dynamic provisioning at all, see Clusters without persistent volumes.
  • Credentials Qodo issued you for the image registry.
  • One identity provider for single sign-on (step 3): commercial Entra, GCC High (or another sovereign Microsoft cloud), Ping Identity, Okta, or Google Workspace. Each is a different global.entra entity — do not mix their fields.
  • An LLM endpoint: an OpenAI-compatible URL and API key. It may be your own gateway; no outbound access to api.openai.com is required if you provide one.

2. DNS and TLS

Pick a base domain, e.g. qodo.example.com. The chart derives these public hosts by default: One wildcard DNS record *.<baseDomain> pointing at your ingress controller covers all defaults, and one wildcard certificate covers them in TLS. Set a complete hostname override when an existing certificate or Kubernetes Gateway listener only covers another DNS level. Every generated Ingress route, OIDC issuer, portal URL, platform URL and CORS origin follows the same override. The six resolved hostnames must remain distinct because they route / to different services. Two things are easy to get wrong here:
  • The resolved auth hostname must resolve from inside the cluster too. Pods validate sign-on tokens against that public OIDC issuer, so a record that only exists on your corporate DNS, or a split-horizon setup that resolves it differently inside, will fail after login rather than at install time.
  • The webhook host must be reachable by whatever calls it. If your git provider is cloud-hosted, git.<baseDomain> needs to be reachable from the internet. You can serve webhooks on the base domain itself instead of a subdomain — set global.ingress.gitHostname — but the host you choose is also the URL handed to the provider when hooks are registered, so it must resolve and be covered by the certificate from the caller’s point of view.
  • The SDK host defaults to sdk.<baseDomain>. If your DNS or certificate policy requires a different hostname, set global.ingress.sdkHostname; both the QAR Ingress route and the platform’s DYNACONF_SDK__BASE_URL are derived from that one value.

3. Identity provider (Entra, GCC High, Ping, Okta, or Google Workspace)

global.entra is one provider. Choose one entity and use only that entity’s fields: Do not mix them: commercial Entra needs tenantId and must not set a sovereign or generic OIDC issuerUrl. GCC High, Ping Identity, Okta, and Google Workspace use generic-oidc, while their issuers and mapping choices remain provider-specific. Confirm <issuerUrl>/.well-known/openid-configuration for every generic OIDC provider. Register both callback URIs on every application (Secret contract is the same):
Do not reuse the same provider name when changing connector types: Zitadel refuses to replace an existing native Entra provider with a generic OIDC provider of the same name.

Commercial Entra (type: entra)

In Microsoft Entra admin center → App registrations → New registration:
  1. Note the Application (client) ID and Directory (tenant) ID.
  2. Create a client secret and copy its value.
  3. Add both Redirect URIs above (type Web).
Register both. The sign-on service uses the first one; the second is required by its API. Registering only /idps/callback looks correct and fails at login with AADSTS50011: The redirect URI ... does not match the redirect URIs configured for the application.
If you manage app registrations with the Azure CLI, note that az ad app update --web-redirect-uris replaces the entire list — read the existing URIs first and write them all back, or you will silently drop them. Define an Entra app role with the case-sensitive value Qodo-Platform-Admin; users assigned that role become Qodo organization owners, while users with any other or missing role remain regular users. If your app registration already uses another administrator role value, set global.entra.adminRoleName to match it exactly.

GCC High (type: generic-oidc)

A sovereign Microsoft tenant cannot use the native entra connector: that path always derives login.microsoftonline.com. Use generic OIDC and the exact GCC High issuer. Keep userClaimsSource: id_token — GCC High tokens are Entra-shaped (given_name / family_name). Leave tenantId empty.
Confirm discovery at that issuer uses GCC High authorization, token, and JWKS hosts (login.microsoftonline.us), not commercial login.microsoftonline.com.

Ping Identity (type: generic-oidc)

The Qodo on-prem portal supports PingOne through Zitadel’s existing generic OIDC connector; no Ping-specific runtime component is required. Other Ping products must expose equivalent standard OIDC endpoints and claims. In PingOne, create and enable an OIDC web application. Use response type Code, grant type Authorization Code, and token endpoint authentication Client Secret Basic. Grant the intended users or groups access, allow the openid profile email scopes, include email, given_name, and family_name in the ID token, and register both callback URIs listed above. Use the environment’s exact issuer URL and leave tenantId empty:

Google Workspace (type: generic-oidc)

Fresh installations only. This procedure configures Google Workspace for a new Qodo and Zitadel installation. Switching an existing installation from another identity provider is an identity migration and account-linking project; that migration is not covered here.
Create the OAuth application in a Google Cloud project owned by the Google Workspace organization that should access Qodo:
  1. Open Google Cloud Console → Google Auth Platform → Audience and confirm the user type is Internal. This is the access boundary: do not rely on the optional hd request parameter or claim.
  2. Open Google Auth Platform → Clients, create an OAuth 2.0 Client ID, and choose Web application.
  3. Add both resolved callback URIs:
  4. Copy the public client ID. Deliver the client secret through the customer-approved Kubernetes secret manager; never place it in this values file.
  5. In Admin Console → Security → Access and data control → API controls → Third-party app access, a Workspace administrator locates the OAuth app by its client ID and sets an access status permitted by the customer’s policy. This is not domain-wide delegation, and the Admin Console does not create the OAuth client.
Google’s issuer is global and exact: https://accounts.google.com. Leave tenantId empty and map claims from the ID token:
Before opening Qodo to users, test one managed Workspace account twice and one account outside the Workspace:
  1. The managed account must reach Qodo.
  2. A fresh second login must reuse the same Qodo/Zitadel identity.
  3. The outside account must be rejected by Google before Qodo creates a user.
Troubleshooting:
  • missing provisioning Secret: create the Secret in the Helm release namespace, ensure its name exactly matches global.entra.existingSecret, and ensure its key is exactly IDP_CLIENT_SECRET.
  • invalid_client: verify that the client ID and Kubernetes Secret belong to the same Google OAuth client without printing the secret.
  • redirect mismatch: compare both registered callback URIs exactly.
  • an outside account reaches Qodo: stop rollout and change the Google OAuth audience to Internal.
  • a duplicate user appears on the second login: stop rollout and investigate issuer/subject identity continuity.
  • administrator controls are missing after promotion: sign out and obtain a fresh session, then verify the promoted email and workspace ID onprem-workspace.

Okta (type: generic-oidc)

Same generic OIDC connector as GCC High, different entity: Okta org authorization-server ID tokens typically have name and email, not given_name / family_name. With mapping on, Zitadel Login V1 only reads those given/family claims and shows a first/last-name form. Set userClaimsSource to userinfo so Zitadel uses the userinfo endpoint. Pin a qodo-platform image that honors IDP_USER_CLAIMS_SOURCE. Do not set tenantId.
https://<org>.okta.com is the org authorization server. A custom authorization server uses a different issuer, for example https://<org>.okta.com/oauth2/default — pick one and use it consistently.

Bootstrap a generic OIDC administrator

The Ping and Google configurations above do not provide the Entra application-role claim. Their users therefore start as regular members. After the intended owner signs in once, promote only that user:
Sign out and sign in again to obtain a fresh session with administrator access.

Multiple identity providers

Repeat the registration steps for every entity (commercial Entra, GCC High, Ping Identity, Okta). Give every provider a unique, stable display name, register both callback URIs in every application, and use the same administrator app-role value across tenants: Qodo has one global.entra.adminRoleName setting for the shared installation. Multi-IdP credentials use a different, secret-only contract. Create one Kubernetes Secret whose DYNACONF_ZITADEL_PROVISION__EXTERNAL_IDPS key contains the complete Dynaconf TOML list. Every { ... } entry must stay on one logical line:
List entries are per entity, same rules as the table above: commercial Entra uses type="entra" (no user_claims_source); GCC High uses type="generic-oidc" with the login.microsoftonline.us issuer (source defaults to id_token); Ping Identity uses type="generic-oidc" with its environment issuer and user_claims_source="id_token"; Okta uses type="generic-oidc" with the Okta issuer and user_claims_source="userinfo". global.entra.userClaimsSource is only wired for the single-provider Job when type is generic-oidc (IDP_USER_CLAIMS_SOURCE). Native entra does not get that env. Prefer creating that value through your Vault, External Secrets, CSI, or other approved secret manager so the completed list never enters shell history. The list must include every provider the provisioner should manage. Omitted providers remain in Zitadel until an operator removes them manually. Provider names are Zitadel’s idempotency keys: renaming one creates another provider. Select list mode in values and leave the legacy fields empty:
Updating only this Secret does not start a new provisioning Job. After adding or removing a provider, or after rotating a client secret through Vault, External Secrets, CSI, or another secret manager, wait for the Kubernetes Secret to contain the new list and then re-run the same helm upgrade --install command used for the installation. The new Helm release revision starts the provisioner, applies the provider changes in Zitadel, and rolls the platform to read the refreshed configuration. The rotation is not active until that upgrade completes. The chart refuses list mode together with any legacy name, clientId, tenantId, or issuerUrl. On a one-to-many upgrade, remove those four fields from your customer values file, keep the existing provider’s name and credentials in the new list, and do not delete qodo-zitadel-generated. The provisioner preserves that Secret, clears the old direct-provider pin, and the chart rolls the platform after provisioning so the provider picker takes effect.

4. Create the Secrets

Create the Secrets required by the infrastructure choices below. Credentials never belong in a values file.
On option B (the Qodo registry), skip Secret 1. A chart pulled with your licence already carries the registry credential and creates the pull Secrets itself.
Use Vault, External Secrets, CSI, or another approved secret manager instead when available. Whichever method you use, the resulting Secret name must match global.entra.existingSecret and its key must be IDP_CLIENT_SECRET. The command above intentionally fails if the Secret already exists; update an existing Secret through your approved secret manager instead of storing its manifest in a client-side-apply annotation. Qodo Telemetry is optional and disabled by default. Qodo supplies the signed offline identity tool with its verification and enrollment guide. After Qodo approves your public identity, create the customer-owned signing Secret from the three files. This step is the same for direct and Replicated installs:
The chart mounts the files read-only. Keep the private key out of Helm values. Allow outbound HTTPS to https://listener.qodo.ai for network sends, then opt in:
automatic sends at startup, on a bounded heartbeat, and after meaningful health changes. on-demand sends only through the local operator API. file makes no receiver requests and allows a local unsigned JSON report download; it needs only a valid public installation ID. Missing or invalid identity does not block the product install. Air-gapped installations should keep telemetry disabled unless an approved local file-only procedure is in place. To inspect the exact bounded payload, last attempt, and last successful send, port-forward the sender locally. These endpoints have no Service or Ingress and reject non-loopback clients:
Do not pre-create qodo-db-admin on the bundled-PostgreSQL path. The chart renders that Secret as part of the Helm release; an existing Secret without Helm ownership metadata makes install fail with invalid ownership metadata. The exception is the fully pre-provisioned mode global.generatedSecrets.render=false, where you must create every Secret in the generated-credentials inventory before install.
Everything else the platform needs is generated on first install and kept across upgrades. On platform 2.182.0 and later, Zitadel provisioning also creates one scim-<provider-slug> machine user per configured provider and checkpoints its one-time PAT as SCIM_PAT_<PROVIDER_SLUG> in qodo-zitadel-generated. Keep that Secret backed up and access-restricted; runtime workloads consume only its OIDC keys, not those PATs.

5. Database

The default is an in-cluster database: Postgres 16 with pgvector, on a persistent volume sized by global.storage.sizes.postgres (20Gi), creating the four databases it needs. You supply the three routing values below; with global.generatedSecrets.render=true, Helm creates qodo-db-admin itself:
Both halves are needed: postgres.enabled runs the server, and global.externalDatabase.* is how every component finds it — including the migration Jobs, which is why existingSecret cannot be empty. The key is named externalDatabase because it is “where the database is”; enabled: true under it is what switches to a server of your own.

Only if you bring your own database server

Set externalDatabase.enabled: true with host, existingSecret (the qodo-db-admin Secret from step 4) and — importantly — adminUser:
The install creates its own databases, so that user needs CREATEDB. require prevents plaintext connections, but does not authenticate the database certificate or hostname; verified TLS modes and custom database CA plumbing are not yet exposed by this chart. Set adminUser explicitly. It defaults to postgres, which most managed database services do not offer — on Google Cloud SQL the equivalent is the user you created with the cloudsqlsuperuser role, on Amazon RDS the master user. Leaving the default in place fails part-way through the install, in a setup job, rather than up front. Use an empty database or instance. The install creates databases named zitadel, litegit, rag_db, qodo_merge and qodo_rules; if objects with those names already exist and are owned by a different user, the setup jobs will fail on permissions. pgvector must be creatable, not merely installed. Two of those databases (rag_db and qodo_rules) hold embeddings and need the vector extension inside them — extensions are per-database, so one does not cover the other. Managed services usually make creating this particular extension superuser-only: on Amazon RDS it needs rds_superuser, which the install user should not have. The extension being available on the server is not enough. If your user cannot create it, have an admin run this once, before installing. The databases must be owned by the configured adminUser, not by the privileged operator running these commands (replace qodo_admin below with that exact role name):
The install detects this and stops with the same instructions rather than failing later inside a migration. Note the bundled connection pooler (advanced.components.pgbouncer.enabled) is for the in-cluster database only and is rejected alongside externalDatabase.enabled: true — front a managed database with its own pooler (RDS Proxy, Cloud SQL connectors) instead.

Clusters without persistent volumes

If your cluster cannot provision PersistentVolumes, add qodo-onprem/values-emptydir.yaml as an extra -f, after your own values file. Note the path: it ships inside the chart directory, unlike onprem-base.yaml at the top level.
It moves all four datastores onto node-local emptyDir storage, so the install needs no StorageClass and creates no PVCs.
This destroys data. An emptyDir lives and dies with its pod. Any restart, eviction, node drain, or helm upgrade that rolls a pod wipes that component — the databases included. Verified, not theoretical: deleting the Postgres pod took the database count from 10 back to 5 and dropped a test table. Use it for a short-lived evaluation, never for an environment whose data matters, and expect to re-run onboarding after any restart.
Switching to real volumes later means a reinstall, not an upgrade: drop the extra -f and reinstall into a clean namespace. Pass it on EVERY upgrade, not just the install. Values files are not remembered between helm upgrade runs, so omitting this -f later asks the chart to move the datastores back onto PersistentVolumes. That upgrade fails part-way, and the failure is not self-healing:
Kubernetes refuses the two StatefulSets (their volume claims are immutable), but everything else in that run is already patched — including qodo-gitway, which is left mounting a qodo-gitway-repo-store PVC that can never bind on a cluster with no provisioner. The pod goes Pending with pod has unbound immediate PersistentVolumeClaims, and re-running the correct three-file command does not fix it: Helm compares its own last-good manifest with the new one, both emptyDir-shaped, so it sees nothing to change and never removes the stray volume. Recover by deleting the deployment and its orphaned claim, then upgrading again with all three files:
Nothing durable is lost — that volume is the repository cache, which is emptyDir on this path and re-clones on demand.

Only if you bring your own Redis

The chart runs Redis in-cluster by default, without a password, and nothing needs configuring. To point at a managed instance (ElastiCache, Memorystore):
If that instance requires a password, create a Secret and name it — no extra -f layer, no other change:
Every component that talks to Redis picks it up from there. Any password works — @, /, :, # and spaces included; each component URL-encodes it.

6. Write your values file

If this bundle contains a values-example.yaml, start from that — it is a worked file for your environment, with your hostnames and endpoints already filled in and the placeholders you must supply marked. Otherwise copy values-customer-template.yaml and uncomment what you need. A minimal file:
That is a complete file — the database is in-cluster and needs nothing. To use your own database server instead, add the block from step 5:

Rules embedding dimensions

The standard setup remains unchanged, with a default rule embedding dimension of 1024. Only override it for a model that requires different settings, using the same configuration on the API and event consumer. For example, native Nomic v1 produces 768 values and does not accept a request dimension parameter:
The migration Job inherits these three settings from platform.env. ConfigMap or Secret references are supported on fresh installs when the referenced resources are available. On upgrades, use literal value entries for these settings: the pre-upgrade migration runs before Helm updates ConfigMaps and Secrets, so the chart rejects valueFrom entries to prevent stale settings. Register the selected model in your LLM gateway and use platform images that support configurable rule embedding dimensions. Set the dimension before the dimension migration first runs. Changing it later does not resize the database automatically; existing incompatible vectors need a migration and regeneration plan. Back up the database before changing models.

PR history embedding dimensions

PR history stores embeddings in rag_db with a default dimension of 2000. Only override it for an embedding model that returns a different size, for example Amazon Titan Text Embeddings V2, which returns at most 1024 values. Set the same literal on all three engine workloads, and keep the embedding request size equal to it:
The engine migration Job creates the rag_db vector columns from the engine-indexer value, even when the indexer itself is disabled. The chart rejects values that differ across the three workloads (an unset value counts as the default 2000), values that are not a positive whole number, and valueFrom entries, because the migration runs before Helm updates ConfigMaps and Secrets. Set the dimension before PR history is first indexed. The migration only creates missing tables, so changing the value later does not resize them. To change it on an existing install, scale the three engine workloads to zero, then in rag_db empty the PR history tables and resize both vector columns before upgrading:
PR history is then re-indexed from scratch.

Optional per-user GitHub repository access

Account linking, access calculation and enforcement are separate steps. Use a Qodo-provided Platform release supporting on-prem account linking and configure the customer OAuth App first. The installed GitHub App must also have the permissions needed to read repository and organization membership. Start calculation without changing visibility:
This starts the internal access worker and enables the API and Git event consumer publishers against the existing message broker and database; it creates no new database or migration. With a compatible Platform release, both account links and installation-created/repositories-added events trigger calculation. For a self-hosted GitHub server, ensure the API’s configured GitHub host matches the linked accounts and installations. Existing links need a supported backfill or another link/sign-in trigger; starting a consumer alone does not compute them. Once the worker is ready and a linked user’s access map is verified, set global.repositoryAccess.filteringEnabled: true. Unlinked users, including administrators, cannot see private repositories on the configured GitHub host. Disabling enforcement restores unfiltered reads. Queue naming and rollout settings are documented under global.repositoryAccess in CUSTOMER-VALUES.md.

7. Install

Option A — from this bundle:
Option B — from the Qodo registry (after helm registry login, section 1). Fetch and unpack first, because onprem-base.yaml lives inside the chart and -f only reads local files:
Install from the unpacked directory, not the oci:// URL directly: the copy you pulled carries your licence, which is what produces the image-pull Secrets. Pulling again for an upgrade is the same two commands with a new --version. Order matters: the second -f wins where the two overlap. $NAMESPACE is the one you set in step 4 — the Secrets have to live in the same namespace as the release. Any namespace works; pick whatever fits your cluster conventions. The release name, though, must stay qodo. Resource names are fixed to a qodo- prefix in this packaging, so another release name yields qodo-* resources anyway and two releases in one namespace would collide. First install takes several minutes: setup jobs prepare the databases and configure sign-on before the applications become ready.

Pulling the images from your own registry

Skip this unless you mirror images internally. By default every image comes from Qodo’s registry using the pull Secret from step 4. It takes one value. Mirror the images listed under “Images in this chart version” in Helm chart configuration reference — keeping the repository path and tag identical, only the registry host changes — then set:
Do not hand-write the image list: it is generated from a full render of this chart version, so it includes the ones referenced only from inside templates and cannot drift from the pins.
Then confirm nothing was missed — this must print nothing:
Run that check. A partial mirror still installs cleanly: Helm accepts it, and the images you missed fail minutes later as ImagePullBackOff, in a pod event that does not mention your values file. Your pull Secret (global.pullSecret, default qodo-pull) must carry credentials for your registry once you do this.

Confining Qodo to particular nodes

Skip this unless Qodo has to run on specific nodes. By default every pod is pinned to x86-64 nodes — the only architecture the images are built for — and placed anywhere else your scheduler chooses. Three values apply to every workload, the setup Jobs included:
A nodeSelector is an AND of its entries, so adding one label narrows placement rather than replacing the architecture pin. A toleration only permits scheduling onto a tainted node — it does not keep pods off the other nodes, so pair it with the selector. If the setup Jobs need to land somewhere the applications do not, add global.jobScheduling.nodeSelector / .tolerations; those are applied on top of the values above rather than instead of them. Check placement before opening traffic — every pod should be on a node you expect, and none should be stuck Pending:
A pod that stays Pending with didn't match Pod's node affinity/selector in kubectl describe means no node carries every label you set.

Clusters that require pods to declare their IDs

Skip this unless your cluster rejects pods that do not name a user or group — most commonly OpenShift, where a restricted SCC admits a pod only if the IDs it names fall inside the range assigned to your namespace. Two values apply to every workload, including the setup Jobs that run before the applications start:
Read the admissible range from your own namespace rather than copying the numbers above:
The two are separate values because Kubernetes splits the fields: fsGroup and supplementalGroups are Pod-only, while allowPrivilegeEscalation, readOnlyRootFilesystem and capabilities are container-only. Setting either in the wrong place is rejected by the API server. Both are empty by default, and a component that ships its own setting keeps it. On OpenShift you will also be publishing through Routes rather than an Ingress — see global.route in Helm chart configuration reference.

8. Verify

Then browse to the resolved portal URL (https://app.<baseDomain> by default, or https://<global.ingress.appHostname> when overridden), choose SSO (Single Sign-On), and sign in with your Entra ID account. Your user is created automatically on first login.

Troubleshooting

Pods stuck in ImagePullBackOff — the qodo-pull Secret is missing, misnamed, or holds the wrong credentials. Confirm with kubectl describe pod <pod> -n <namespace> and check the name is exactly qodo-pull. The ingress never gets an address — with the GCE ingress controller, ask Qodo for the additional values layer it requires; container-native load balancing needs annotations this chart does not apply by default. On other controllers, check that global.ingress.className matches an IngressClass that exists (kubectl get ingressclass). Setup jobs fail with a permissions or role error — almost always global.externalDatabase.adminUser: see step 5. Login fails with AADSTS50011 — a Redirect URI is missing from the app registration: see step 3, and note that both are required. The portal loads but stays on a spinner — check that the pods are ready and that the resolved portal API hostname (portal-api.<baseDomain> or global.ingress.portalApiHostname) resolves and is covered by the certificate; the portal calls it from the browser. A review pod restarts every few minutes and reviews never finish — read the restart reason before treating it as a crash or a failed health check, because those look identical from kubectl get pods:
Reason: OOMKilled with Exit Code: 137 means the container hit its memory limit, not a bug. The queue worker reserves 10Gi per replica, while the HTTP agentic-review endpoint requests 4Gi with an 8Gi limit. Very large diffs or a long global.merge.settings metadata list can still exceed those limits. If both entry points serve reviews in your installation, raise both aliases so the outcome does not depend on which one handled the review:
Each container reserves its requests.memory, so multiply by replicaCount before raising it. Six queue-worker containers at 10Gi reserve 60Gi; with the default 1Gi analytics sidecar in every pod, the six replicas reserve 66Gi of schedulable memory. Pods stay Pending if the nodes cannot satisfy that. A pod is Pending with unbound immediate PersistentVolumeClaims — on a cluster without volume provisioning, this is usually an upgrade that omitted values-emptydir.yaml; see “Clusters without persistent volumes” in step 5 for the recovery, which Helm cannot perform on its own. Something else — collect kubectl get pods -n <namespace>, kubectl describe ingress -n <namespace>, and the logs of the failing pod, and contact Qodo support.

Upgrades

Replace onprem-base.yaml with the new version, keep your own values file, and re-run the same helm upgrade --install command. Your file is never overwritten by an upgrade. Database migrations gate upgrades. After the RabbitMQ topology check, db-init and the enabled application migration Jobs run as pre-upgrade hooks. A failed hook stops the upgrade before Helm updates application Deployments. Existing pods remain on the previous version; inspect the failed Job and correct the cause before retrying. These hooks use the existing database and credential Secrets; do not rotate or rename those Secrets in the same upgrade. On a first install, migration Jobs remain ordinary resources because generated Secrets and bundled PostgreSQL are not yet available. If an upgrade is intended to enable bundled PostgreSQL for the first time or repair an unavailable database, the pre-upgrade migrations cannot run against it. Do not bypass the hooks with --no-hooks: that can roll new application images onto an unmigrated database. Restore database availability first, or contact Qodo support for a staged migration and data-transfer plan. Switching to a newly enabled bundled PostgreSQL Service in one upgrade is rejected before any resources change. Offline helm template --is-upgrade cannot verify that Service; use helm upgrade --dry-run=server to preview an existing installation. The revisioned qodo-merge-migrate-* Job applies PR-Agent database changes. The review worker and agentic endpoint wait in wait-for-schema until their image’s Alembic head matches the qodo_merge database; the init container reads alembic_version directly (it never invokes Alembic’s database-creating startup path) and never runs migrations. Use --wait --wait-for-jobs --timeout 30m for upgrades so Helm reports a failed migration or a pod stuck waiting for it. Inspect the migration Job logs before retrying; changing the Deployment image alone does not start the Job. Upgrading into PR Agent 3.49.0 requires a review write pause. Its one-time 070_backfill_dropped_findings migration moves existing suppressed findings into a separate table. Before starting this upgrade, gate webhook and direct review intake, suspend scheduled producers, and drain accepted review and agentic work while LiteGit remains reachable. Record the replica counts, then stop both review writers, even if the RabbitMQ topology is already current:
Wait for both Deployments’ old Pods, including terminating Pods, to disappear before running Helm. Keep the authoritative replica counts in the supplied values; Helm restores target writers after the schema hooks. The topology hook alone is not a writer-stop mechanism when conversion is a no-op. Do not use a normal overlapping review-worker rollout for this schema transition. Migration 070 preserves suppressed rows without a workspace in findings. If a legacy-data import assigns their workspace IDs after 070, a reviewed forward reconciliation must move the newly eligible rows before accepting the upgrade; rerunning Helm at the same Alembic head does not repeat the backfill. Validate original finding IDs and payloads across both findings and dropped_findings, and require no eligible suppressed row left in findings. Do not delete retained rows or stamp the schema backwards to force a rerun. Application Deployments roll one pod at a time without adding surge capacity (maxSurge: 0, maxUnavailable: 1). A multi-replica service keeps at least replicas - 1 ready while its new pods start; a single-replica service still has a brief interruption. QAR keeps its separate zero-unavailable, one-surge strategy and schema-aware startup gate. Old and new pods can overlap during a rolling upgrade, so image and database changes must remain backward-compatible through that interval. Helm reporting deployed does not by itself prove every rollout or migration finished; verify the Deployment and Job statuses. Before application workloads change, the chart checks the three persisted LiteGit workflow queues. RabbitMQ cannot change queue type or delivery arguments in place, while the independently released Platform, Engine and qodo-git images do not all adopt the bounded quorum contract in the same release. The chart therefore carries the topology supported by each pinned client: it migrates a queue to quorum only when that consumer supports bounded redelivery, and keeps an older consumer’s queue classic until its image moves. The default global.rabbitmqMigration.mode: automatic runs a guarded pre-upgrade hook. Accepted reviews still call LiteGit, so the hook drains active workflow queues while LiteGit remains available. For a busy installation, pause public webhook intake and scheduled producers first; internal LiteGit APIs must remain reachable until accepted reviews finish. Confirm both ready/in-flight deliveries and active review runs have settled. For an upgrade from a chart whose qodo-revision-roller Role cannot list pods, have the namespace administrator add apiGroups: [""], resources: ["pods"], verbs: ["list"] to that Role before migrating. The hook checks this permission before changing replicas; it needs to observe terminating pods, too. Fresh installs receive this read permission from the chart automatically. After draining, the hook stops LiteGit, waits until its pods are gone, and checks for deliveries that raced shutdown using two clean broker observations ten seconds apart after a ten-second statistics settling window. A race aborts before deleting any queue and restores the old replicas. It then stops affected consumers and replaces only queues RabbitMQ confirms are empty and unused. Compatible queues and existing dead letters are preserved. After conversion, incompatible old consumers stay paused: Helm applies their target images and configured replica counts after the schema migration Jobs. By default the publisher also waits for that apply. To resume webhook intake earlier, set global.rabbitmqMigration.resumePublisherAfterMigration: true only after rehearsing that the currently installed gateway and its dependencies remain compatible throughout every pending database migration. With this setting, the hook reads back the restored queue bindings, restores only the gateway’s original replicas and waits for its Deployment to be ready. Incompatible old consumers remain at zero. New events accumulate in the verified queues until Helm starts the target consumers after the schema Jobs. An originally stopped or absent gateway stays stopped. Before pausing, the hook snapshots all relevant bindings and requires an unscoped or pass-through route covering incoming events. Missing optional subscriptions, narrower custom routes or an oversized routing snapshot defer early recovery to the normal target apply without failing the upgrade. Reverification compares all saved bindings, including unchanged queues. A small qodo.ai/rabbitmq-publisher-recovery Deployment annotation retains the original positive publisher replica count and routing snapshot across failed hooks. A retry can recover that publisher even when topology is already current, without restarting old consumers. Successful recovery clears the annotation; stale markers from a different workload revision/image are discarded without scaling. Preserve the marker during forward repair; clear it deliberately if an operator intends to keep a previously interrupted publisher stopped. Ordinary compatible-topology upgrades without a recovery marker do not scale workloads. The opt-in adds its bounded 180-second publisher-readiness allowance to the hook Job deadline; retain the documented 30-minute Helm timeout for default settings, and account for a longer configured drain allowance in the maintenance budget. With this option, a Helm pre-upgrade Job runs after the schema Jobs and stamps only zero-replica consumers with no active or terminating Pods with the target revision and CA-bundle checksum. Helm then applies the target configuration and restores their replicas. The post-upgrade revision hook skips these fresh consumers, allowing the backlog to drain without a second restart. Running consumers retain their normal revision rollout. A preparation marker on Deployment metadata identifies its Helm upgrade attempt. Successful post-upgrade handling clears that marker without restarting fresh Pods; after an interrupted apply or rollback, a later attempt refreshes those consumers even if their version stamps already match the target. Preserve that marker through forward repair. This early-resumption option requires Helm; for Argo CD, keep it false and keep intake gated through the rollout. This option does not upgrade the gateway early or make database migrations reversible. If a later migration fails, retain the backlog and complete forward recovery; do not restart incompatible consumers. Providers receive a retryable failure while the publisher is stopped for queue conversion; ensure their retry window or a durable upstream buffer covers that interval. An independently managed ingress maintenance gate must be reopened explicitly after publisher recovery if early intake is intended. Keeping that gate closed prevents early intake regardless of this setting. Already-compatible installations skip the topology maintenance entirely. If draining exceeds global.rabbitmqMigration.drainTimeoutSeconds (default 600), the upgrade fails with LiteGit still available and no queue deleted. Failures before topology changes restore any recorded replicas. After the first queue deletion, failures keep incompatible old consumers paused. The publisher stays stopped until every converted queue and its bindings verify; when early resumption is enabled it may already be running if readiness or a later schema Job fails. Inspect publisher health and the retained Job, preserve queued work and complete the target upgrade; do not start old consumers against converted queues or roll back database revisions:
Clusters that require an operator-controlled maintenance procedure may set global.rabbitmqMigration.mode: validate. That mode is read-only and blocks the upgrade while any legacy queue remains; use it only with a migration plan provided by Qodo.

Argo CD / ApplicationSet (external PostgreSQL)

Argo CD runs helm template, not helm upgrade: it cannot use Kubernetes lookup to reuse live credentials, and .Release.Revision does not advance. Use the packaged values-argocd.yaml as the last values layer after onprem-base.yaml and your environment values. It selects global.deploymentMode: argocd and global.generatedSecrets.render: false. Do not put credentials in any values file or in Git. This mode requires a reachable external PostgreSQL server; bundled qodo-postgres is deliberately rejected because its pod starts after the database setup Jobs on a first sync. The external Secret controller (for example, External Secrets or a sealed- Secret workflow) must reconcile these fixed names and keys in the target namespace before Argo CD syncs Qodo. They are absent from the Argo CD chart render; changing their names in values does not repoint the consumers. The Platform bearer must equal qodo-svc-onprem- plus the preserved raw key; the allowlist must contain that bearer; LiteGit, Gitway, Engine, QAR, both review workers, and Platform read it from the same qodo-qodo-platform-api-key Secret. The TOML key must match it. The packaged preflight checks these invariants without displaying any credential:
Python 3.11+ is required. This checks the internal inventory above. The ordinary install prerequisites still apply separately: the image-pull Secret (qodo-pull on direct-registry installs), qodo-openai when using the bundled LLM gateway, the configured identity-provider Secret, and TLS as appropriate. qodo-engine-secrets-toml contains no credential and remains chart-rendered. qodo-zitadel-generated is written by the provisioner at runtime, not on a fresh first sync. An existing install must preserve it as well. The handoff preflight checks every required identity key used by Platform; an unrelated nonempty Secret does not pass. An empty IdP pin is valid for local login or a provider picker, but its key must remain present. For a Helm-to-Argo CD handoff:
  1. Pin the release name to qodo, the same namespace and customer values, and a reviewed chart version. Keep the values order onprem-base.yaml, customer file, values-argocd.yaml. If the customer file lives in another Git repository, use Argo CD multi-source $values/... for that middle entry. Keep the Argo CD Application name qodo too, or configure a separate Argo CD tracking label key: Argo CD’s default app.kubernetes.io/instance rewrite can otherwise break this chart’s selectors.
  2. Back up and import the existing bytes of every Secret above into the external Secret source, including Zitadel’s master key, LiteGit’s vault master key, and the database password. Preserve qodo-zitadel-generated. Never regenerate those keys against existing databases. Keep backup/export material only in approved encrypted Secret storage, never a values file, shell transcript, PR, or temporary render.
  3. Have the external controller reconcile first. Run the preflight above with --require-zitadel-generated; verify its output is green, and verify the old and newly managed Secret data are byte-identical in your Secret system. Do not delete and recreate a live Secret to transfer its ownership. If only the pr-agent TOML check fails, reconstruct that file from the existing Platform bearer in the Secret source; do not rotate the bearer or its allowlist to match a stale TOML copy.
  4. Freeze direct Helm upgrades. Start the ApplicationSet with automated pruning and self-heal off, inspect argocd app diff, and confirm Qodo’s credential Secrets are not in its desired manifest. Do not run helm uninstall: that deletes other live resources and is not an Argo CD adoption step. The old Helm release record can remain while Argo CD owns future reconciliation. Only one controller should write the workloads.
  5. Sync in a maintenance window if the chart or images move. Argo CD waits for the chart’s ordered Jobs: wave -3 prepares static prerequisites/RBAC and waits for bundled RabbitMQ and Redis to be healthy; wave -2 checks RabbitMQ topology and initializes databases; wave -1 runs migrations; wave 0 updates workloads and provisions Zitadel; PostSync stamps the revision. The RabbitMQ Job checks the running broker on both install and upgrade. Absent queues on a fresh broker need no migration; a missing publisher alone does not prove retained queues are absent. Completed Jobs have stable names for identical renders and remain present; a chart, values, or queue-script change gives them new names and reruns them. Replicated pull-credential rotation alone does not change those names. Use a full Application sync, not selective resource sync: Argo CD skips hooks during selective sync, including this chart’s PostSync revision roll. Also disable ApplyOutOfSyncOnly=true: the topology Job can pause a workload whose manifest was unchanged at the start of the sync. Later waves must apply every Deployment to restore its configured replicas; argocd-upgrade-check.py pre rejects this selective-apply policy.
  6. Check every migration Job and Deployment is healthy before enabling automatic sync/pruning. If you enable pruning, use PruneLast=true and keep the external Secret resources in their own Application/Secret controller with a deliberate retention policy. Never let pruning remove a credential the Qodo chart no longer renders. Before enabling self-heal, tell the Application to ignore the annotations that Qodo’s provision/revision Jobs maintain (including preparation markers retained across a Helm handoff), and set RespectIgnoreDifferences=true:
    Without that policy, Argo CD can undo the Jobs’ pod restarts and repeatedly report drift. This Application policy belongs in the ApplicationSet, not in Helm values; validate it in the actual generated Application before turning on self-heal.
The first Argo CD sync also runs the PostSync revision roller. Newly installed Deployments have no revision stamp, so this causes one additional rollout after initial readiness. Wait for those rollouts to settle before acceptance checks. Later identical syncs retain the stamp and do not repeat that rollout. Set Platform rule embedding model, dimension and send-dimensions settings with literal platform.env.*.value entries in Argo CD mode. Their migration runs before workload ConfigMap and Secret updates, so valueFrom cannot safely supply these non-secret settings. Embedding API credentials remain Secret refs. If Secret bytes are unchanged, ownership handoff alone needs no pod restart. If an external Secret legitimately changes, environment variables and subPath TOML mounts do not refresh in running pods: coordinate a rollout of every consumer after updating the Secret. A changed Platform bearer affects Platform, LiteGit, Gitway, QAR, Engine and review workers together. The PostSync revision roller handles chart-version/CA changes, not external Secret rotation. Zitadel master, LiteGit vault and database credentials must not be rotated as a side effect of migration. The qodo-revision ConfigMap reports QODO_DEPLOYMENT_MODE=argocd, the actual chart version, and QODO_HELM_REVISION=0 (Argo CD has no Helm release revision). Use Argo CD’s Application history for sync revisions. The external credential source, Argo CD ApplicationSet policy, Secret retention, and tracking-label choice cannot be supplied by chart values alone. A direct Helm install remains the default; do not add the Argo CD layer to one. For a first Argo CD install, provision Secrets and the external PostgreSQL server before the first sync, then run the preflight without --require-zitadel-generated.

Check before and after every Argo CD sync

verify-external-secrets.py checks the Secret inventory; it cannot see whether the servers still accept those values. argocd-upgrade-check.py checks the live state around a sync and never prints a credential:
Run pre after changing the chart version and values and refreshing the Application. It reports NO-GO when:
  • an operation is running or queued, or automated sync is on;
  • the Application’s comparison is stale or failed, or its desired state contains a credential Secret;
  • a credential Secret is missing, or one that the Application tracks lacks Prune=false;
  • a running pod’s environment or mounted pr-agent TOML differs from its Secret;
  • RabbitMQ or the bundled LLM gateway rejects the Secret’s value;
  • Zitadel logs master-key decrypt errors or database login failures;
  • any of these checks cannot run (missing access, unreadable container, no running pods).
Before a first install nothing runs yet: use verify-external-secrets.py as the gate, and run pre --first-install, which skips the checks against running pods and servers. Run post after the sync. It repeats the Secret, running-pod and server checks. It also requires the sync to have succeeded at the expected chart version (or commit), every setup and migration Job of that revision to have succeeded, and every Deployment and StatefulSet to be fully rolled out.

Moving an Argo CD install off generatedSecrets.render: true

If Argo CD ever synced this chart with generatedSecrets.render: true, each render wrote new random internal credentials into the Secrets. Several things keep this hidden:
  • The diff doesn’t show it. ignoreDifferences on Secret data hides the change from the diff but not from the sync.
  • RespectIgnoreDifferences doesn’t stop it when it covers only /data, because the chart renders stringData.
  • The servers keep the old values. RabbitMQ keeps the password it stored on first boot. Zitadel’s master key encrypts its stored keys, and LiteGit’s vault key encrypts provider credentials, so the old values stay authoritative there.
  • Running pods keep working until they restart. An upgrade restarts all of them, and then RabbitMQ logins, Zitadel and Git integrations fail together.
Before the first sync with values-argocd.yaml:
  1. Turn off automated sync and make sure no operation is running.
  2. Run the pre check. If secrets.match_running_pods, rabbitmq.accepts_secret or zitadel.credentials fails, the Secrets no longer hold the values the servers use.
  3. Recover the original bytes from your own backup, or from the newest Helm release record in the namespace (sh.helm.release.v1.qodo.v<N>, written by the last direct helm install or helm upgrade). Put them back into the six generated credential Secrets and qodo-pr-agent-secrets-toml. Restore Zitadel’s master key and LiteGit’s vault key exactly: regenerating them loses the data they protect.
  4. The database setup Job resets the zitadel role’s password whenever it runs. If the role rejects the restored ZITADEL_DB_PASSWORD, set it from a database admin session with ALTER ROLE zitadel PASSWORD. If RabbitMQ rejects every candidate, set the broker to the Secret’s value with rabbitmqctl change_password qodo instead of generating a new one.
  5. Annotate the Secrets argocd.argoproj.io/sync-options: Prune=false and restart the Qodo Deployments so every pod reads the restored values. Repeat the pre check until it reports GO.
A rollback needs the same care: restore the database backup and the same Secret bytes, then sync the older version with values-argocd.yaml. Re-rendering an older chart with render: true writes yet another random set. Rolling back from 7.x to 6.x also needs a RabbitMQ step, because the 7.x topology migration converts the event queues on the broker’s own volume and a database restore does not touch it. The 6.x consumers then fail with PRECONDITION_FAILED - inequivalent arg. Before syncing the older version:
  1. Scale qodo-litegit (the publisher) to 0.
  2. Wait until vcs-events.pr-agent-pro-__pass_through__ and vcs-events.qodo-platform-rules-consumer show 0 messages (rabbitmqctl list_queues name messages).
  3. Delete those two queues with rabbitmqctl delete_queue.
The 6.x consumers recreate them as classic queues when they start. Before retrying the upgrade after a rollback, delete the setup and migration Jobs of the version you rolled back from. Their names depend only on the chart and values, so a retry with the same chart reuses the completed Jobs, and the database and queue migrations would not run against the restored state. The pre check fails with jobs.will_run in this case and prints the exact kubectl delete job command.

Argo CD operations during an upgrade

  • Argo CD runs one operation per Application at a time. While a sync waits for an unhealthy resource, later sync requests queue behind it. Terminate the operation and wait for a terminal phase before syncing again.
  • Argo CD before 3.3.15, 3.4.9 and 3.5.3 can replay the previous result, without applying anything, when a sync is started by writing the Application’s operation field directly (scripts, kubectl patch). Syncs started from the Argo CD UI or CLI are not affected. This is argoproj/argo-cd#26530, fixed by #28778. After any scripted sync, check that the operation’s start time changed.
  • Do not use “apply only” (skip hooks) or a selective sync to get past a failing step. Both skip the RabbitMQ topology migration and the revision roll, so the workloads move to the new version without them.
  • If the RabbitMQ topology Job reports rejected user 'qodo' (HTTP 401), the password in qodo-rabbitmq-password does not match the broker. Fix the credential as described above; syncing again cannot.
  • Any chart or values change gives every setup and migration Job a new name, so all of them run again. The previous revision’s Jobs then show as “requires pruning”. Remove them with a pruning sync (PruneLast=true) once every tracked credential Secret carries Prune=false, or delete them by hand.

Azure SAML with Login V1

For a SAML enterprise application without Graph User.Read, see SAML-LOGIN-V1.md. It includes the platform release prerequisite, provider Secret, values overlay, claims mapping and Azure callback settings.