PostgreSQL prerequisite correction: the shipped base values expect an
external PostgreSQL server. The bundled server is opt-in: set
postgres.enabled: true and global.externalDatabase.host: qodo-postgres.
The generated prerequisite paragraph below describes an earlier default.
See Infrastructure requirements.
The complete list of values you can set is
Helm chart configuration reference, with a ready-to-edit skeleton in
values-customer-template.yaml. Both are
generated from the chart, so they cannot drift from what it actually accepts.
Where the chart comes from
Two options. They install the same chart; pick one and follow it through, since they differ in one prerequisite.
Option B, once per machine:
oci:// URL wherever this page says ./qodo-onprem, and
skip Secret 1 in step 4: a chart pulled this way carries your licence, and
the chart turns it into the image-pull Secrets every workload needs. Creating
qodo-pull as well does no harm, it is simply unused.
To see available versions: helm show chart oci://artifacts-self-hosted.qodo.ai/codium-stack/qodo-onprem/qodo-onprem.
1. Prerequisites
Before working through this section, run./qodo-check.sh -n <namespace> -f <your-values.yaml> and read READINESS.md: they verify these prerequisites
for you and cover the backup, rollback and test-account items this guide does
not. Run the same command again after installing, before opening traffic — it
detects that Qodo is now installed and adds the post-installation checks.
- Kubernetes 1.25+ and Helm 3, with a namespace you can install into.
- An ingress controller (any: ingress-nginx, GCE, Traefik, …) and a DNS record you control — see step 2.
- PostgreSQL. Nothing to prepare by default: the chart runs Postgres 16
with
pgvectorin-cluster, creates its own databases and generates its own admin password. To use your own database server instead — PostgreSQL 15 or 16 with thepgvectorextension available — setexternalDatabase.enabled: trueand see step 5. - A StorageClass that provisions PersistentVolumes (
ReadWriteOnceis enough). Postgres, Redis, RabbitMQ and the Qodo Git repo cache each claim one. If your cluster has no dynamic provisioning at all, see Clusters without persistent volumes. - Credentials Qodo issued you for the image registry.
- One identity provider for single sign-on (step 3): commercial Entra,
GCC High (or another sovereign Microsoft cloud), Ping Identity,
Okta, or Google Workspace. Each is a different
global.entraentity — do not mix their fields. - An LLM endpoint: an OpenAI-compatible URL and API key. It may be your own
gateway; no outbound access to
api.openai.comis required if you provide one.
2. DNS and TLS
Pick a base domain, e.g.qodo.example.com. The chart derives these public
hosts by default:
One wildcard DNS record
*.<baseDomain> pointing at your ingress controller
covers all defaults, and one wildcard certificate covers them in TLS. Set a
complete hostname override when an existing certificate or Kubernetes Gateway
listener only covers another DNS level. Every generated Ingress route, OIDC
issuer, portal URL, platform URL and CORS origin follows the same override.
The six resolved hostnames must remain distinct because they route / to
different services.
Two things are easy to get wrong here:
- The resolved auth hostname must resolve from inside the cluster too. Pods validate sign-on tokens against that public OIDC issuer, so a record that only exists on your corporate DNS, or a split-horizon setup that resolves it differently inside, will fail after login rather than at install time.
- The webhook host must be reachable by whatever calls it. If your git
provider is cloud-hosted,
git.<baseDomain>needs to be reachable from the internet. You can serve webhooks on the base domain itself instead of a subdomain — setglobal.ingress.gitHostname— but the host you choose is also the URL handed to the provider when hooks are registered, so it must resolve and be covered by the certificate from the caller’s point of view. - The SDK host defaults to
sdk.<baseDomain>. If your DNS or certificate policy requires a different hostname, setglobal.ingress.sdkHostname; both the QAR Ingress route and the platform’sDYNACONF_SDK__BASE_URLare derived from that one value.
3. Identity provider (Entra, GCC High, Ping, Okta, or Google Workspace)
global.entra is one provider. Choose one entity and use only that
entity’s fields:
Do not mix them: commercial Entra needs
tenantId and must not set a sovereign
or generic OIDC issuerUrl. GCC High, Ping Identity, Okta, and Google
Workspace use generic-oidc, while their issuers and mapping choices remain
provider-specific. Confirm <issuerUrl>/.well-known/openid-configuration for
every generic OIDC provider. Register both callback URIs on every
application (Secret contract is the same):
name when changing connector types: Zitadel
refuses to replace an existing native Entra provider with a generic OIDC
provider of the same name.
Commercial Entra (type: entra)
In Microsoft Entra admin center → App registrations → New registration:
- Note the Application (client) ID and Directory (tenant) ID.
- Create a client secret and copy its value.
- Add both Redirect URIs above (type Web).
Register both. The sign-on service uses the first one; the second is required by its API. Registering onlyIf you manage app registrations with the Azure CLI, note that/idps/callbacklooks correct and fails at login withAADSTS50011: The redirect URI ... does not match the redirect URIs configured for the application.
az ad app update --web-redirect-uris replaces the entire list — read the
existing URIs first and write them all back, or you will silently drop them.
Define an Entra app role with the case-sensitive value
Qodo-Platform-Admin; users assigned that role become Qodo organization
owners, while users with any other or missing role remain regular users. If
your app registration already uses another administrator role value, set
global.entra.adminRoleName to match it exactly.
GCC High (type: generic-oidc)
A sovereign Microsoft tenant cannot use the native entra connector: that
path always derives login.microsoftonline.com. Use generic OIDC and the
exact GCC High issuer. Keep userClaimsSource: id_token — GCC High tokens
are Entra-shaped (given_name / family_name). Leave tenantId empty.
login.microsoftonline.us), not commercial login.microsoftonline.com.
Ping Identity (type: generic-oidc)
The Qodo on-prem portal supports PingOne through Zitadel’s existing generic
OIDC connector; no Ping-specific runtime component is required. Other Ping
products must expose equivalent standard OIDC endpoints and claims.
In PingOne, create and enable an OIDC web application. Use response type
Code, grant type Authorization Code, and token endpoint authentication
Client Secret Basic. Grant the intended users or groups access, allow the
openid profile email scopes, include email, given_name, and family_name
in the ID token, and register both callback URIs listed above.
Use the environment’s exact issuer URL and leave tenantId empty:
Google Workspace (type: generic-oidc)
Fresh installations only. This procedure configures Google Workspace for a new Qodo and Zitadel installation. Switching an existing installation from another identity provider is an identity migration and account-linking project; that migration is not covered here.Create the OAuth application in a Google Cloud project owned by the Google Workspace organization that should access Qodo:
-
Open Google Cloud Console → Google Auth Platform → Audience and confirm
the user type is Internal. This is the access boundary: do not rely on
the optional
hdrequest parameter or claim. - Open Google Auth Platform → Clients, create an OAuth 2.0 Client ID, and choose Web application.
-
Add both resolved callback URIs:
- Copy the public client ID. Deliver the client secret through the customer-approved Kubernetes secret manager; never place it in this values file.
- In Admin Console → Security → Access and data control → API controls → Third-party app access, a Workspace administrator locates the OAuth app by its client ID and sets an access status permitted by the customer’s policy. This is not domain-wide delegation, and the Admin Console does not create the OAuth client.
https://accounts.google.com. Leave
tenantId empty and map claims from the ID token:
- The managed account must reach Qodo.
- A fresh second login must reuse the same Qodo/Zitadel identity.
- The outside account must be rejected by Google before Qodo creates a user.
- missing provisioning Secret: create the Secret in the Helm release namespace,
ensure its name exactly matches
global.entra.existingSecret, and ensure its key is exactlyIDP_CLIENT_SECRET. invalid_client: verify that the client ID and Kubernetes Secret belong to the same Google OAuth client without printing the secret.- redirect mismatch: compare both registered callback URIs exactly.
- an outside account reaches Qodo: stop rollout and change the Google OAuth audience to Internal.
- a duplicate user appears on the second login: stop rollout and investigate issuer/subject identity continuity.
- administrator controls are missing after promotion: sign out and obtain a
fresh session, then verify the promoted email and workspace ID
onprem-workspace.
Okta (type: generic-oidc)
Same generic OIDC connector as GCC High, different entity: Okta org
authorization-server ID tokens typically have name and email, not
given_name / family_name. With mapping on, Zitadel Login V1 only reads
those given/family claims and shows a first/last-name form. Set
userClaimsSource to userinfo so Zitadel uses the userinfo endpoint.
Pin a qodo-platform image that honors
IDP_USER_CLAIMS_SOURCE. Do not set tenantId.
https://<org>.okta.com is the org authorization server. A custom
authorization server uses a different issuer, for example
https://<org>.okta.com/oauth2/default — pick one and use it consistently.
Bootstrap a generic OIDC administrator
The Ping and Google configurations above do not provide the Entra application-role claim. Their users therefore start as regular members. After the intended owner signs in once, promote only that user:Multiple identity providers
Repeat the registration steps for every entity (commercial Entra, GCC High, Ping Identity, Okta). Give every provider a unique, stable display name, register both callback URIs in every application, and use the same administrator app-role value across tenants: Qodo has oneglobal.entra.adminRoleName setting for the shared installation.
Multi-IdP credentials use a different, secret-only contract. Create one
Kubernetes Secret whose
DYNACONF_ZITADEL_PROVISION__EXTERNAL_IDPS key contains the complete Dynaconf
TOML list. Every { ... } entry must stay on one logical line:
type="entra" (no user_claims_source); GCC High uses
type="generic-oidc" with the login.microsoftonline.us issuer (source
defaults to id_token); Ping Identity uses type="generic-oidc" with its
environment issuer and user_claims_source="id_token"; Okta uses
type="generic-oidc" with the Okta issuer and user_claims_source="userinfo".
global.entra.userClaimsSource is only wired for the single-provider Job when
type is generic-oidc
(IDP_USER_CLAIMS_SOURCE). Native entra does not get that env.
Prefer creating that value through your Vault, External Secrets, CSI, or other
approved secret manager so the completed list never enters shell history. The
list must include every provider the provisioner should manage. Omitted
providers remain in Zitadel until an operator removes them manually. Provider
names are Zitadel’s idempotency keys: renaming one creates another provider.
Select list mode in values and leave the legacy fields empty:
helm upgrade --install
command used for the installation. The new Helm release revision starts the
provisioner, applies the provider changes in Zitadel, and rolls the platform to
read the refreshed configuration. The rotation is not active until that upgrade
completes.
The chart refuses list mode together with any legacy name, clientId,
tenantId, or issuerUrl. On a one-to-many upgrade, remove those four fields
from your customer values file, keep the existing provider’s name and
credentials in the new list, and do not delete qodo-zitadel-generated. The
provisioner preserves that Secret, clears the old direct-provider pin, and the
chart rolls the platform after provisioning so the provider picker takes
effect.
4. Create the Secrets
Create the Secrets required by the infrastructure choices below. Credentials never belong in a values file.On option B (the Qodo registry), skip Secret 1. A chart pulled with your licence already carries the registry credential and creates the pull Secrets itself.
global.entra.existingSecret and its key must be IDP_CLIENT_SECRET. The
command above intentionally fails if the Secret already exists; update an
existing Secret through your approved secret manager instead of storing its
manifest in a client-side-apply annotation.
Qodo Telemetry is optional and disabled by default. Qodo supplies the signed
offline identity tool with its verification and enrollment guide. After Qodo
approves your public identity, create the customer-owned signing Secret from
the three files. This step is the same for direct and Replicated installs:
https://listener.qodo.ai for network sends, then opt in:
automatic sends at startup, on a bounded heartbeat, and after meaningful
health changes. on-demand sends only through the local operator API. file
makes no receiver requests and allows a local unsigned JSON report download;
it needs only a valid public installation ID. Missing or invalid identity
does not block the product install. Air-gapped installations should keep
telemetry disabled unless an approved local file-only procedure is in place.
To inspect the exact bounded payload, last attempt, and last successful send,
port-forward the sender locally. These endpoints have no Service or Ingress and
reject non-loopback clients:
Do not pre-createEverything else the platform needs is generated on first install and kept across upgrades. On platform 2.182.0 and later, Zitadel provisioning also creates oneqodo-db-adminon the bundled-PostgreSQL path. The chart renders that Secret as part of the Helm release; an existing Secret without Helm ownership metadata makes install fail withinvalid ownership metadata. The exception is the fully pre-provisioned modeglobal.generatedSecrets.render=false, where you must create every Secret in the generated-credentials inventory before install.
scim-<provider-slug> machine user per configured provider and
checkpoints its one-time PAT as SCIM_PAT_<PROVIDER_SLUG> in
qodo-zitadel-generated. Keep that Secret backed up and access-restricted;
runtime workloads consume only its OIDC keys, not those PATs.
5. Database
The default is an in-cluster database: Postgres 16 withpgvector, on a
persistent volume sized by global.storage.sizes.postgres (20Gi), creating the
four databases it needs. You supply the three routing values below; with
global.generatedSecrets.render=true, Helm creates qodo-db-admin itself:
postgres.enabled runs the server, and
global.externalDatabase.* is how every component finds it — including the
migration Jobs, which is why existingSecret cannot be empty. The key is named
externalDatabase because it is “where the database is”; enabled: true under
it is what switches to a server of your own.
Only if you bring your own database server
SetexternalDatabase.enabled: true with host, existingSecret (the
qodo-db-admin Secret from step 4) and — importantly — adminUser:
require prevents plaintext connections, but does not authenticate the
database certificate or hostname; verified TLS modes and custom database CA
plumbing are not yet exposed by this chart.
Set adminUser explicitly. It defaults to postgres, which most managed
database services do not offer — on Google Cloud SQL the equivalent is the
user you created with the cloudsqlsuperuser role, on Amazon RDS the master
user. Leaving the default in place fails part-way through the install, in a
setup job, rather than up front.
Use an empty database or instance. The install creates databases named
zitadel, litegit, rag_db, qodo_merge and qodo_rules; if objects with
those names already exist and are owned by a different user, the setup jobs
will fail on permissions.
pgvector must be creatable, not merely installed. Two of those databases
(rag_db and qodo_rules) hold embeddings and need the vector extension
inside them — extensions are per-database, so one does not cover the other.
Managed services usually make creating this particular extension
superuser-only: on Amazon RDS it needs rds_superuser, which the install
user should not have. The extension being available on the server is not
enough. If your user cannot create it, have an admin run this once, before
installing. The databases must be owned by the configured adminUser, not by
the privileged operator running these commands (replace qodo_admin below
with that exact role name):
advanced.components.pgbouncer.enabled) is for the in-cluster database only
and is rejected alongside externalDatabase.enabled: true — front a managed
database with its own pooler (RDS Proxy, Cloud SQL connectors) instead.
Clusters without persistent volumes
If your cluster cannot provision PersistentVolumes, addqodo-onprem/values-emptydir.yaml as an extra -f, after your own values
file. Note the path: it ships inside the chart directory, unlike
onprem-base.yaml at the top level.
emptyDir storage, so the install
needs no StorageClass and creates no PVCs.
This destroys data. AnSwitching to real volumes later means a reinstall, not an upgrade: drop the extraemptyDirlives and dies with its pod. Any restart, eviction, node drain, orhelm upgradethat rolls a pod wipes that component — the databases included. Verified, not theoretical: deleting the Postgres pod took the database count from 10 back to 5 and dropped a test table. Use it for a short-lived evaluation, never for an environment whose data matters, and expect to re-run onboarding after any restart.
-f and reinstall into a clean namespace.
Pass it on EVERY upgrade, not just the install. Values files are not
remembered between helm upgrade runs, so omitting this -f later asks the
chart to move the datastores back onto PersistentVolumes. That upgrade fails
part-way, and the failure is not self-healing:
qodo-gitway,
which is left mounting a qodo-gitway-repo-store PVC that can never bind on a
cluster with no provisioner. The pod goes Pending with pod has unbound immediate PersistentVolumeClaims, and re-running the correct three-file
command does not fix it: Helm compares its own last-good manifest with the
new one, both emptyDir-shaped, so it sees nothing to change and never removes
the stray volume. Recover by deleting the deployment and its orphaned claim,
then upgrading again with all three files:
emptyDir on this path and re-clones on demand.
Only if you bring your own Redis
The chart runs Redis in-cluster by default, without a password, and nothing needs configuring. To point at a managed instance (ElastiCache, Memorystore):-f layer, no other change:
@, /, :, # and spaces included; each component URL-encodes it.
6. Write your values file
If this bundle contains avalues-example.yaml, start from that — it is a
worked file for your environment, with your hostnames and endpoints already
filled in and the placeholders you must supply marked. Otherwise copy
values-customer-template.yaml and uncomment what you need. A minimal file:
Rules embedding dimensions
The standard setup remains unchanged, with a default rule embedding dimension of 1024. Only override it for a model that requires different settings, using the same configuration on the API and event consumer. For example, native Nomic v1 produces 768 values and does not accept a request dimension parameter:platform.env. ConfigMap
or Secret references are supported on fresh installs when the referenced
resources are available. On upgrades, use literal value entries for these
settings: the pre-upgrade migration runs before Helm updates ConfigMaps and
Secrets, so the chart rejects valueFrom entries to prevent stale settings.
Register the selected model in your LLM gateway and use platform images that
support configurable rule embedding dimensions.
Set the dimension before the dimension migration first runs. Changing it later
does not resize the database automatically; existing incompatible vectors need
a migration and regeneration plan. Back up the database before changing models.
PR history embedding dimensions
PR history stores embeddings inrag_db with a default dimension of 2000. Only
override it for an embedding model that returns a different size, for example
Amazon Titan Text Embeddings V2, which returns at most 1024 values. Set the same
literal on all three engine workloads, and keep the embedding request size
equal to it:
rag_db vector columns from the
engine-indexer value, even when the indexer itself is disabled. The chart
rejects values that differ across the three workloads (an unset value counts as
the default 2000), values that are not a positive whole number, and valueFrom
entries, because the migration runs before Helm updates ConfigMaps and Secrets.
Set the dimension before PR history is first indexed. The migration only
creates missing tables, so changing the value later does not resize them.
To change it on an existing install, scale the three engine workloads to zero,
then in rag_db empty the PR history tables and resize both vector columns
before upgrading:
Optional per-user GitHub repository access
Account linking, access calculation and enforcement are separate steps. Use a Qodo-provided Platform release supporting on-prem account linking and configure the customer OAuth App first. The installed GitHub App must also have the permissions needed to read repository and organization membership. Start calculation without changing visibility:global.repositoryAccess.filteringEnabled: true. Unlinked users, including
administrators, cannot see private repositories on the configured GitHub host.
Disabling enforcement restores unfiltered reads. Queue naming and rollout
settings are documented under global.repositoryAccess in CUSTOMER-VALUES.md.
7. Install
Option A — from this bundle:helm registry login, section 1).
Fetch and unpack first, because onprem-base.yaml lives inside the chart and
-f only reads local files:
oci:// URL directly: the
copy you pulled carries your licence, which is what produces the image-pull
Secrets. Pulling again for an upgrade is the same two commands with a new
--version.
Order matters: the second -f wins where the two overlap.
$NAMESPACE is the one you set in step 4 — the Secrets have to live in the
same namespace as the release. Any namespace works; pick whatever fits your
cluster conventions.
The release name, though, must stay qodo. Resource names are fixed to
a qodo- prefix in this packaging, so another release name yields qodo-*
resources anyway and two releases in one namespace would collide.
First install takes several minutes: setup jobs prepare the databases and
configure sign-on before the applications become ready.
Pulling the images from your own registry
Skip this unless you mirror images internally. By default every image comes from Qodo’s registry using the pull Secret from step 4. It takes one value. Mirror the images listed under “Images in this chart version” in Helm chart configuration reference — keeping the repository path and tag identical, only the registry host changes — then set:ImagePullBackOff, in a pod event
that does not mention your values file.
Your pull Secret (global.pullSecret, default qodo-pull) must carry
credentials for your registry once you do this.
Confining Qodo to particular nodes
Skip this unless Qodo has to run on specific nodes. By default every pod is pinned to x86-64 nodes — the only architecture the images are built for — and placed anywhere else your scheduler chooses. Three values apply to every workload, the setup Jobs included:nodeSelector is an AND of its entries, so adding one label narrows
placement rather than replacing the architecture pin. A toleration only permits
scheduling onto a tainted node — it does not keep pods off the other nodes, so
pair it with the selector.
If the setup Jobs need to land somewhere the applications do not, add
global.jobScheduling.nodeSelector / .tolerations; those are applied on top
of the values above rather than instead of them.
Check placement before opening traffic — every pod should be on a node you
expect, and none should be stuck Pending:
Pending with didn't match Pod's node affinity/selector in
kubectl describe means no node carries every label you set.
Clusters that require pods to declare their IDs
Skip this unless your cluster rejects pods that do not name a user or group — most commonly OpenShift, where a restricted SCC admits a pod only if the IDs it names fall inside the range assigned to your namespace. Two values apply to every workload, including the setup Jobs that run before the applications start:fsGroup and
supplementalGroups are Pod-only, while allowPrivilegeEscalation,
readOnlyRootFilesystem and capabilities are container-only. Setting either
in the wrong place is rejected by the API server.
Both are empty by default, and a component that ships its own setting keeps it.
On OpenShift you will also be publishing through Routes rather than an Ingress
— see global.route in Helm chart configuration reference.
8. Verify
https://app.<baseDomain> by default,
or https://<global.ingress.appHostname> when overridden), choose SSO
(Single Sign-On), and sign in with your Entra ID account. Your user is
created automatically on first login.
Troubleshooting
Pods stuck inImagePullBackOff — the qodo-pull Secret is missing,
misnamed, or holds the wrong credentials. Confirm with
kubectl describe pod <pod> -n <namespace> and check the name is exactly qodo-pull.
The ingress never gets an address — with the GCE ingress controller, ask
Qodo for the additional values layer it requires; container-native load
balancing needs annotations this chart does not apply by default. On other
controllers, check that global.ingress.className matches an IngressClass that
exists (kubectl get ingressclass).
Setup jobs fail with a permissions or role error — almost always
global.externalDatabase.adminUser: see step 5.
Login fails with AADSTS50011 — a Redirect URI is missing from the app
registration: see step 3, and note that both are required.
The portal loads but stays on a spinner — check that the pods are ready and
that the resolved portal API hostname (portal-api.<baseDomain> or
global.ingress.portalApiHostname) resolves and is covered by the certificate;
the portal calls it from the browser.
A review pod restarts every few minutes and reviews never finish — read the
restart reason before treating it as a crash or a failed health check, because
those look identical from kubectl get pods:
Reason: OOMKilled with Exit Code: 137 means the container hit its memory
limit, not a bug. The queue worker reserves 10Gi per replica, while the HTTP
agentic-review endpoint requests 4Gi with an 8Gi limit. Very large diffs or a
long global.merge.settings metadata list can still exceed those limits. If
both entry points serve reviews in your installation, raise both aliases so
the outcome does not depend on which one handled the review:
requests.memory, so multiply by replicaCount
before raising it. Six queue-worker containers at 10Gi reserve 60Gi; with the
default 1Gi analytics sidecar in every pod, the six replicas reserve 66Gi of
schedulable memory. Pods stay Pending if the nodes cannot satisfy that.
A pod is Pending with unbound immediate PersistentVolumeClaims — on a
cluster without volume provisioning, this is usually an upgrade that omitted
values-emptydir.yaml; see “Clusters without persistent volumes” in step 5 for
the recovery, which Helm cannot perform on its own.
Something else — collect kubectl get pods -n <namespace>,
kubectl describe ingress -n <namespace>, and the logs of the failing pod, and
contact Qodo support.
Upgrades
Replaceonprem-base.yaml with the new version, keep your own values file, and
re-run the same helm upgrade --install command. Your file is never overwritten
by an upgrade.
Database migrations gate upgrades. After the RabbitMQ topology check,
db-init and the enabled application migration Jobs run as pre-upgrade
hooks. A failed hook stops the upgrade before Helm updates application
Deployments. Existing pods remain on the previous version; inspect the failed
Job and correct the cause before retrying. These hooks use the existing
database and credential Secrets; do not rotate or rename those Secrets in the
same upgrade. On a first install, migration Jobs remain ordinary resources
because generated Secrets and bundled PostgreSQL are not yet available.
If an upgrade is intended to enable bundled PostgreSQL for the first time or
repair an unavailable database, the pre-upgrade migrations cannot run against
it. Do not bypass the hooks with --no-hooks: that can roll new application
images onto an unmigrated database. Restore database availability first, or
contact Qodo support for a staged migration and data-transfer plan. Switching
to a newly enabled bundled PostgreSQL Service in one upgrade is rejected before
any resources change. Offline helm template --is-upgrade cannot verify that
Service; use helm upgrade --dry-run=server to preview an existing installation.
The revisioned qodo-merge-migrate-* Job applies PR-Agent database changes.
The review worker and agentic endpoint wait in wait-for-schema until their
image’s Alembic head matches the qodo_merge database; the init container
reads alembic_version directly (it never invokes Alembic’s database-creating
startup path) and never runs migrations. Use
--wait --wait-for-jobs --timeout 30m for upgrades so Helm reports a failed
migration or a pod stuck waiting for it. Inspect the migration Job logs before
retrying; changing the Deployment image alone does not start the Job.
Upgrading into PR Agent 3.49.0 requires a review write pause. Its one-time
070_backfill_dropped_findings migration moves existing suppressed findings
into a separate table. Before starting this upgrade, gate webhook and direct
review intake, suspend scheduled producers, and drain accepted review and
agentic work while LiteGit remains reachable. Record the replica counts, then
stop both review writers, even if the RabbitMQ topology is already current:
findings.
If a legacy-data import assigns their workspace IDs after 070, a reviewed
forward reconciliation must move the newly eligible rows before accepting the
upgrade; rerunning Helm at the same Alembic head does not repeat the backfill.
Validate original finding IDs and payloads across both findings and
dropped_findings, and require no eligible suppressed row left in findings.
Do not delete retained rows or stamp the schema backwards to force a rerun.
Application Deployments roll one pod at a time without adding surge capacity
(maxSurge: 0, maxUnavailable: 1). A multi-replica service keeps at least
replicas - 1 ready while its new pods start; a single-replica service still
has a brief interruption. QAR keeps its separate zero-unavailable, one-surge
strategy and schema-aware startup gate. Old and new pods can overlap during a
rolling upgrade, so image and database changes must remain backward-compatible
through that interval. Helm reporting deployed does not by itself prove every
rollout or migration finished; verify the Deployment and Job statuses.
Before application workloads change, the chart checks the three persisted
LiteGit workflow queues. RabbitMQ cannot change queue type or delivery
arguments in place, while the independently released Platform, Engine and
qodo-git images do not all adopt the bounded quorum contract in the same
release. The chart therefore carries the topology supported by each pinned
client: it migrates a queue to quorum only when that consumer supports bounded
redelivery, and keeps an older consumer’s queue classic until its image moves.
The default global.rabbitmqMigration.mode: automatic runs a guarded
pre-upgrade hook. Accepted reviews still call LiteGit, so the hook drains
active workflow queues while LiteGit remains available. For a busy
installation, pause public webhook intake and scheduled producers first;
internal LiteGit APIs must remain reachable until accepted reviews finish.
Confirm both ready/in-flight deliveries and active review runs have settled.
For an upgrade from a chart whose qodo-revision-roller Role cannot list pods,
have the namespace administrator add apiGroups: [""], resources: ["pods"],
verbs: ["list"] to that Role before migrating. The hook checks this permission
before changing replicas; it needs to observe terminating pods, too. Fresh
installs receive this read permission from the chart automatically.
After draining, the hook stops LiteGit, waits until its pods are gone, and
checks for deliveries that raced shutdown using two clean broker observations
ten seconds apart after a ten-second statistics settling window. A race
aborts before deleting any queue and restores the old
replicas. It then stops affected consumers and replaces only queues RabbitMQ
confirms are empty and unused. Compatible queues and existing dead letters
are preserved. After conversion, incompatible old consumers stay paused:
Helm applies their target images and configured replica counts after the
schema migration Jobs. By default the publisher also waits for that apply.
To resume webhook intake earlier, set
global.rabbitmqMigration.resumePublisherAfterMigration: true only after
rehearsing that the currently installed gateway and its dependencies remain
compatible throughout every pending database migration. With this setting,
the hook reads back the restored queue bindings, restores only the gateway’s
original replicas and waits for its Deployment to be ready. Incompatible old
consumers remain at zero. New events accumulate in the verified queues until
Helm starts the target consumers after the schema Jobs. An originally stopped
or absent gateway stays stopped. Before pausing, the hook snapshots all relevant
bindings and requires an unscoped or pass-through route covering incoming events.
Missing optional subscriptions, narrower custom routes or an oversized routing
snapshot defer early recovery to the normal target apply without failing the
upgrade. Reverification compares all saved bindings, including unchanged queues.
A small qodo.ai/rabbitmq-publisher-recovery Deployment annotation retains the
original positive publisher replica count and routing snapshot across failed
hooks. A retry can recover that publisher even when topology is already current,
without restarting old consumers. Successful recovery clears the annotation;
stale markers from a different workload revision/image are discarded without
scaling. Preserve the marker during forward repair; clear it deliberately if
an operator intends to keep a previously interrupted publisher stopped. Ordinary
compatible-topology upgrades without a recovery marker do not scale workloads.
The opt-in adds its bounded 180-second publisher-readiness allowance to the hook
Job deadline; retain the documented 30-minute Helm timeout for default settings,
and account for a longer configured drain allowance in the maintenance budget.
With this option, a Helm pre-upgrade Job runs after the schema Jobs and stamps
only zero-replica consumers with no active or terminating Pods with the target
revision and CA-bundle checksum. Helm then applies the target configuration and restores their replicas.
The post-upgrade revision hook skips these fresh consumers, allowing the backlog
to drain without a second restart. Running consumers retain their normal revision
rollout. A preparation marker on Deployment metadata identifies its Helm upgrade
attempt. Successful post-upgrade handling clears that marker without restarting
fresh Pods; after an interrupted apply or rollback, a later attempt refreshes
those consumers even if their version stamps already match the target. Preserve
that marker through forward repair. This early-resumption option requires Helm; for Argo CD, keep it false
and keep intake gated through the rollout.
This option does not upgrade the gateway early
or make database migrations reversible. If a later migration fails, retain the
backlog and complete forward recovery; do not restart incompatible consumers.
Providers receive a retryable failure while the publisher is stopped for queue
conversion; ensure their retry window or a durable upstream buffer covers that
interval. An independently managed ingress maintenance gate must be reopened
explicitly after publisher recovery if early intake is intended. Keeping that
gate closed prevents early intake regardless of this setting. Already-compatible
installations skip the topology maintenance entirely.
If draining exceeds global.rabbitmqMigration.drainTimeoutSeconds (default
600), the upgrade fails with LiteGit still available and no queue deleted.
Failures before topology changes restore any recorded replicas. After the
first queue deletion, failures keep incompatible old consumers paused. The
publisher stays stopped until every converted queue and its bindings verify;
when early resumption is enabled it may already be running if readiness or a
later schema Job fails. Inspect publisher health and the retained Job, preserve
queued work and complete the target upgrade; do not start old consumers against
converted queues or roll back database revisions:
global.rabbitmqMigration.mode: validate. That mode is read-only and blocks
the upgrade while any legacy queue remains; use it only with a migration plan
provided by Qodo.
Argo CD / ApplicationSet (external PostgreSQL)
Argo CD runshelm template, not helm upgrade: it cannot use Kubernetes
lookup to reuse live credentials, and .Release.Revision does not advance.
Use the packaged values-argocd.yaml as the last values layer after
onprem-base.yaml and your environment values. It selects
global.deploymentMode: argocd and global.generatedSecrets.render: false.
Do not put credentials in any values file or in Git. This mode requires a
reachable external PostgreSQL server; bundled qodo-postgres is deliberately
rejected because its pod starts after the database setup Jobs on a first sync.
The external Secret controller (for example, External Secrets or a sealed-
Secret workflow) must reconcile these fixed names and keys in the target
namespace before Argo CD syncs Qodo. They are absent from the Argo CD
chart render; changing their names in values does not repoint the consumers.
The Platform bearer must equal
qodo-svc-onprem- plus the preserved raw key;
the allowlist must contain that bearer; LiteGit, Gitway, Engine, QAR, both
review workers, and Platform read it from the same
qodo-qodo-platform-api-key Secret. The TOML key must match it. The packaged
preflight checks these invariants without displaying any credential:
qodo-pull on direct-registry installs), qodo-openai when using the bundled
LLM gateway, the configured identity-provider Secret, and TLS as appropriate.
qodo-engine-secrets-toml contains no credential and remains chart-rendered.
qodo-zitadel-generated is written by the provisioner at runtime, not on a
fresh first sync. An existing install must preserve it as well. The handoff
preflight checks every required identity key used by Platform; an unrelated
nonempty Secret does not pass. An empty IdP pin is valid for local login or a
provider picker, but its key must remain present.
For a Helm-to-Argo CD handoff:
-
Pin the release name to
qodo, the same namespace and customer values, and a reviewed chart version. Keep the values orderonprem-base.yaml, customer file,values-argocd.yaml. If the customer file lives in another Git repository, use Argo CD multi-source$values/...for that middle entry. Keep the Argo CD Application nameqodotoo, or configure a separate Argo CD tracking label key: Argo CD’s defaultapp.kubernetes.io/instancerewrite can otherwise break this chart’s selectors. -
Back up and import the existing bytes of every Secret above into the
external Secret source, including Zitadel’s master key, LiteGit’s vault
master key, and the database password. Preserve
qodo-zitadel-generated. Never regenerate those keys against existing databases. Keep backup/export material only in approved encrypted Secret storage, never a values file, shell transcript, PR, or temporary render. -
Have the external controller reconcile first. Run the preflight above with
--require-zitadel-generated; verify its output is green, and verify the old and newly managed Secret data are byte-identical in your Secret system. Do not delete and recreate a live Secret to transfer its ownership. If only the pr-agent TOML check fails, reconstruct that file from the existing Platform bearer in the Secret source; do not rotate the bearer or its allowlist to match a stale TOML copy. -
Freeze direct Helm upgrades. Start the ApplicationSet with automated
pruning and self-heal off, inspect
argocd app diff, and confirm Qodo’s credential Secrets are not in its desired manifest. Do not runhelm uninstall: that deletes other live resources and is not an Argo CD adoption step. The old Helm release record can remain while Argo CD owns future reconciliation. Only one controller should write the workloads. -
Sync in a maintenance window if the chart or images move. Argo CD waits for
the chart’s ordered Jobs: wave -3 prepares static prerequisites/RBAC and
waits for bundled RabbitMQ and Redis to be healthy;
wave -2 checks RabbitMQ topology and initializes databases; wave -1 runs
migrations; wave 0 updates workloads and provisions Zitadel; PostSync
stamps the revision. The RabbitMQ Job checks the running broker on both
install and upgrade. Absent queues on a fresh broker need no migration;
a missing publisher alone does not prove retained queues are absent.
Completed Jobs have stable names for identical renders and remain present;
a chart, values, or queue-script change gives them new names and reruns them.
Replicated pull-credential rotation alone does not change those names.
Use a full Application sync, not selective resource sync: Argo CD skips
hooks during selective sync, including this chart’s PostSync revision roll.
Also disable
ApplyOutOfSyncOnly=true: the topology Job can pause a workload whose manifest was unchanged at the start of the sync. Later waves must apply every Deployment to restore its configured replicas;argocd-upgrade-check.py prerejects this selective-apply policy. -
Check every migration Job and Deployment is healthy before enabling
automatic sync/pruning. If you enable pruning, use
PruneLast=trueand keep the external Secret resources in their own Application/Secret controller with a deliberate retention policy. Never let pruning remove a credential the Qodo chart no longer renders. Before enabling self-heal, tell the Application to ignore the annotations that Qodo’s provision/revision Jobs maintain (including preparation markers retained across a Helm handoff), and setRespectIgnoreDifferences=true:Without that policy, Argo CD can undo the Jobs’ pod restarts and repeatedly report drift. This Application policy belongs in the ApplicationSet, not in Helm values; validate it in the actual generated Application before turning on self-heal.
platform.env.*.value entries in Argo CD mode. Their migration runs
before workload ConfigMap and Secret updates, so valueFrom cannot safely
supply these non-secret settings. Embedding API credentials remain Secret refs.
If Secret bytes are unchanged, ownership handoff alone needs no pod
restart. If an external Secret legitimately changes, environment variables
and subPath TOML mounts do not refresh in running pods: coordinate a rollout
of every consumer after updating the Secret. A changed Platform bearer affects
Platform, LiteGit, Gitway, QAR, Engine and review workers together. The
PostSync revision roller handles chart-version/CA changes, not external
Secret rotation. Zitadel master, LiteGit vault and database credentials must
not be rotated as a side effect of migration.
The qodo-revision ConfigMap reports QODO_DEPLOYMENT_MODE=argocd, the
actual chart version, and QODO_HELM_REVISION=0 (Argo CD has no Helm release
revision). Use Argo CD’s Application history for sync revisions.
The external credential source, Argo CD ApplicationSet policy, Secret
retention, and tracking-label choice cannot be supplied by chart values
alone. A direct Helm install remains the default; do not add the Argo CD layer
to one. For a first Argo CD install, provision Secrets and the external
PostgreSQL server before the first sync, then run the preflight without
--require-zitadel-generated.
Check before and after every Argo CD sync
verify-external-secrets.py checks the Secret inventory; it cannot see whether
the servers still accept those values. argocd-upgrade-check.py checks the
live state around a sync and never prints a credential:
pre after changing the chart version and values and refreshing the
Application. It reports NO-GO when:
- an operation is running or queued, or automated sync is on;
- the Application’s comparison is stale or failed, or its desired state contains a credential Secret;
- a credential Secret is missing, or one that the Application tracks lacks
Prune=false; - a running pod’s environment or mounted pr-agent TOML differs from its Secret;
- RabbitMQ or the bundled LLM gateway rejects the Secret’s value;
- Zitadel logs master-key decrypt errors or database login failures;
- any of these checks cannot run (missing access, unreadable container, no running pods).
verify-external-secrets.py as
the gate, and run pre --first-install, which skips the checks against
running pods and servers.
Run post after the sync. It repeats the Secret, running-pod and server
checks. It also requires the sync to have succeeded at the expected chart
version (or commit), every setup and migration Job of that revision to have
succeeded, and every Deployment and StatefulSet to be fully rolled out.
Moving an Argo CD install off generatedSecrets.render: true
If Argo CD ever synced this chart with generatedSecrets.render: true, each
render wrote new random internal credentials into the Secrets. Several things
keep this hidden:
- The diff doesn’t show it.
ignoreDifferenceson Secret data hides the change from the diff but not from the sync. RespectIgnoreDifferencesdoesn’t stop it when it covers only/data, because the chart rendersstringData.- The servers keep the old values. RabbitMQ keeps the password it stored on first boot. Zitadel’s master key encrypts its stored keys, and LiteGit’s vault key encrypts provider credentials, so the old values stay authoritative there.
- Running pods keep working until they restart. An upgrade restarts all of them, and then RabbitMQ logins, Zitadel and Git integrations fail together.
values-argocd.yaml:
- Turn off automated sync and make sure no operation is running.
- Run the
precheck. Ifsecrets.match_running_pods,rabbitmq.accepts_secretorzitadel.credentialsfails, the Secrets no longer hold the values the servers use. - Recover the original bytes from your own backup, or from the newest Helm
release record in the namespace (
sh.helm.release.v1.qodo.v<N>, written by the last directhelm installorhelm upgrade). Put them back into the six generated credential Secrets andqodo-pr-agent-secrets-toml. Restore Zitadel’s master key and LiteGit’s vault key exactly: regenerating them loses the data they protect. - The database setup Job resets the
zitadelrole’s password whenever it runs. If the role rejects the restoredZITADEL_DB_PASSWORD, set it from a database admin session withALTER ROLE zitadel PASSWORD. If RabbitMQ rejects every candidate, set the broker to the Secret’s value withrabbitmqctl change_password qodoinstead of generating a new one. - Annotate the Secrets
argocd.argoproj.io/sync-options: Prune=falseand restart the Qodo Deployments so every pod reads the restored values. Repeat theprecheck until it reports GO.
values-argocd.yaml.
Re-rendering an older chart with render: true writes yet another random set.
Rolling back from 7.x to 6.x also needs a RabbitMQ step, because the 7.x
topology migration converts the event queues on the broker’s own volume and a
database restore does not touch it. The 6.x consumers then fail with
PRECONDITION_FAILED - inequivalent arg. Before syncing the older version:
- Scale
qodo-litegit(the publisher) to 0. - Wait until
vcs-events.pr-agent-pro-__pass_through__andvcs-events.qodo-platform-rules-consumershow 0 messages (rabbitmqctl list_queues name messages). - Delete those two queues with
rabbitmqctl delete_queue.
pre check fails with jobs.will_run in this case and prints the exact
kubectl delete job command.
Argo CD operations during an upgrade
- Argo CD runs one operation per Application at a time. While a sync waits for an unhealthy resource, later sync requests queue behind it. Terminate the operation and wait for a terminal phase before syncing again.
- Argo CD before 3.3.15, 3.4.9 and 3.5.3 can replay the previous result,
without applying anything, when a sync is started by writing the
Application’s
operationfield directly (scripts,kubectl patch). Syncs started from the Argo CD UI or CLI are not affected. This is argoproj/argo-cd#26530, fixed by #28778. After any scripted sync, check that the operation’s start time changed. - Do not use “apply only” (skip hooks) or a selective sync to get past a failing step. Both skip the RabbitMQ topology migration and the revision roll, so the workloads move to the new version without them.
- If the RabbitMQ topology Job reports
rejected user 'qodo' (HTTP 401), the password inqodo-rabbitmq-passworddoes not match the broker. Fix the credential as described above; syncing again cannot. - Any chart or values change gives every setup and migration Job a new name,
so all of them run again. The previous revision’s Jobs then show as
“requires pruning”. Remove them with a pruning sync (
PruneLast=true) once every tracked credential Secret carriesPrune=false, or delete them by hand.
Azure SAML with Login V1
For a SAML enterprise application without GraphUser.Read, see
SAML-LOGIN-V1.md. It includes the platform release prerequisite,
provider Secret, values overlay, claims mapping and Azure callback settings.