Installing and configuring ROH on Kubernetes with Helm
The Orchestration Hub runs the integration and scan jobs scheduled by RegScale. It comprises an application tier, four queue-specific Celery worker pools, a scheduler, and a database migration job. This guide covers the procedure from an empty namespace to a verified deployment.
The chart README documents every configurable value and serves as the reference companion to this document.
synqly.enabled: falseROH requires a dedicated SQL database and Redis instance. It does not share those provisioned for the core RegScale chart.
Two Redis settings are determined when the instance is created and cannot be changed afterwards. Confirm both before provisioning.
Cluster mode must be disabled. The Kombu library used by Celery does not support Redis OSS cluster mode. The AWS ElastiCache console enables cluster mode by default. Changing this setting later requires a new instance. Redis Enterprise clustering is supported; the restriction applies only to Redis OSS cluster mode.
Authentication requires transit encryption. Where AUTH is enabled, the endpoint accepts TLS connections only and the connection string must use the rediss:// scheme. On ElastiCache these two settings are linked. The TLS port is provider-specific: 6380 on Azure Cache for Redis, 6379 on ElastiCache and Memorystore. The examples below use 6379; substitute the port your provider serves TLS on.
A separate RegScale CLI deployment is not required. The CLI is included in the worker image and is invoked by the workers.
ROH stores every timestamp in UTC and renders it in UTC. That holds only if the clocks underneath it agree, and two of them are yours to set.
Do not set TZ on the ROH containers. The images run UTC, and the application writes each timestamp in UTC explicitly rather than reading the container's clock. Setting TZ in a pod spec or a cluster-wide mutating policy does not change what ROH records; it changes only how the container's own log lines are rendered, so the logs and the stored data stop agreeing for anyone reading them together.
Provision the SQL Server host in UTC. SQL Server takes its clock from the host operating system and has no per-session time zone, so this is settled once, where the instance is created, and no connection string corrects it afterwards.
| Service | What to do |
|---|---|
| Azure SQL Database, Azure SQL Managed Instance | Nothing. There is no time zone setting; the service runs UTC |
| RDS for SQL Server | Keep the default UTC time zone. It is chosen when the instance is created and cannot be changed afterwards |
| Cloud SQL for SQL Server | Keep the instance at its UTC default |
| SQL Server you run yourself, including this chart's in-cluster option | Set the host operating system to UTC |
Confirm it before installing, and again after a database migration or a failover:
A non-zero result does not corrupt the timestamps ROH writes itself, because the application supplies those values rather than asking the database for them. It affects the columns whose defaults the database fills, and it makes the database's own logs, and any query you write against GETDATE(), disagree with everything else. Correct it at the host rather than working around it in queries.
Keep the node clocks synchronized. The workers, the scheduler and the database compare timestamps against one another, so they need a common time source. Managed Kubernetes services synchronize node clocks by default; if you run your own nodes, use NTP and treat drift as an alertable condition.
The default worker scaling mode is KEDA queue-depth autoscaling, which allows worker pools to scale to zero while idle. KEDA is a cluster-wide prerequisite and is not installed by this chart:
KEDA is not required if workers.scaling.mode is set to static.
Running on unsupported Kubernetes version "1.31". KEDA 2.20 is supported and tested on Kubernetes "1.33" or higher. Review the KEDA compatibility matrix against your cluster version and either select a KEDA release that supports it, or set workers.scaling.mode: static, which removes the KEDA dependency.The supplied overlays assume cluster components that the chart does not install. Installation fails if the required custom resource definitions are absent, and an Ingress resource is never reconciled if its controller is absent.
| Overlay | Also needs |
|---|---|
values-aks.yaml | An nginx ingress controller, and the Prometheus Operator CRDs, because the overlay enables serviceMonitor, alertRules, and dashboards |
values-eks.yaml | The AWS Load Balancer Controller, required by className: alb, and the Prometheus Operator CRDs |
values-gke.yaml | No additional ingress component; GCE L7 is built in. Uses GCP Managed Prometheus (PodMonitoring) rather than the Prometheus Operator |
To install without Prometheus, disable the monitoring options:
You should see both charts:
The core chart's app version is not the core version that ROH needs. Set the core image from Version pairing.
helm search lists only regscale/regscale. The Orchestration Hub entry appears once the chart reaches the repository index, so the ROH install commands below cannot run before then.Images come from public Docker Hub (orchestration-hub-app and orchestration-hub-worker), so no registry credentials are required. To use a private mirror, point global.imageRegistry at it. It also covers the core proxy image (nginxinc/nginx-unprivileged).
Synqly Embedded is different. Its images come from quay.io/synqly and need a pull credential from Synqly. The Synqly subchart does not read global.imageRegistry, and it pulls with imagePullPolicy: Always. To mirror them, set synqly.embedded.registry and synqly.managementUI.registry (for example registry.example.com/synqly). See Synqly Embedded.
ROH reads five values from a Kubernetes Secret. Create the Secret directly rather than supplying these values with --set, which records them in shell history and in the Helm release metadata. Every pod reads these values by environment-variable name. The following key names are required exactly as shown.
| Key | What it is |
|---|---|
SECRET_KEY | Signs ROH session tokens. Must be at least 32 characters |
SECRETS_ENCRYPTION_KEY | Encrypts stored integration credentials. Must be exactly 32 characters. See the note below |
DATABASE_URL | Full connection string, pymssql form |
REDIS_URL | Full connection string, including the password. Use the rediss:// scheme where the endpoint requires TLS, which includes any ElastiCache instance with AUTH enabled |
METRICS_TOKEN | Restricts access to the /metrics endpoint |
This key decrypts credentials already stored in the ROH database. Once ROH has encrypted data with it, replacing the value renders that data permanently unreadable. Running helm upgrade with a new value does not re-encrypt existing records.
Generate this value once and retain a backup outside the cluster. Loss of this key constitutes unrecoverable loss of all stored integration credentials. The chart does not generate this value automatically.
openssl rand -hex 16 produces 16 random bytes rendered as 32 hexadecimal characters, which is the exact length SECRETS_ENCRYPTION_KEY requires. SECRET_KEY has a 32-character minimum rather than a fixed length, so -hex 32 is correct there.
Note the following requirements.
ROH_BROKER_URL or ROH_RESULT_BACKEND. The chart derives both values from REDIS_URL. Setting them independently can cause the application tier and the workers to address different Redis instances.REDIS_PASSWORD key containing the password in plain form. The Redis subchart and the KEDA trigger read this value directly and cannot extract it from REDIS_URL.enableTLS setting from the scheme in REDIS_URL. Specifying redis:// for a TLS-only endpoint results in a cleartext connection attempt and disables TLS for KEDA. ElastiCache AUTH requires transit encryption, so AWS deployments require rediss://. The converse also applies: the in-cluster Redis subchart serves cleartext, so an in-cluster deployment uses redis://:PASSWORD@<release>-redis-master:6379/0. The example above shows the managed form.The following four settings are independent. Any combination is supported.
queueLength is the queued-message count per replica that triggers a scale-up. min: 0 means the pool costs nothing while idle, at the price of a cold start on the next job.
cli_tasks jobs can run up to four hours, and the chart sets a matching termination grace period so scaling drains workers rather than killing them mid-job. Do not lower gracePeriod below your longest expected job.| Cloud | Overlay | Identity | Ingress | Redis | Metrics |
|---|---|---|---|---|---|
| Azure AKS | values-aks.yaml | Workload Identity | nginx | in-cluster Bitnami | Prometheus Operator |
| AWS EKS | values-eks.yaml | IRSA | ALB | managed (ElastiCache) | Prometheus Operator |
| GCP GKE | values-gke.yaml | Workload Identity | GCE L7 | managed (Memorystore) | GCP Managed Prometheus |
The overlays do not configure application storage. ROH maintains no persistent state beyond its database and Redis instance. The AKS overlay declares a PersistentVolumeClaim, which belongs to the in-cluster Redis subchart rather than to ROH.
Record the configuration in a values file rather than a sequence of --set flags. The same file is required for every subsequent upgrade.
regscaleUrl and prefixNamespace are both required. regscaleUrl specifies the address at which ROH reaches core RegScale. Core must see each login through it as HTTPS: use core's TLS ingress, or set coreProxy.enabled: true with core's in-cluster Service. See Make core see ROH logins as HTTPS. prefixNamespace namespaces this deployment's Redis keys and must be unique per deployment: two ROH installs sharing a Redis with the same prefix will consume each other's jobs.
Fully commented example values files are published for the two most common platforms. Each one is a complete starting configuration: download it, replace the REPLACE- placeholders, and install.
Install with the downloaded file:
Using the in-chart overlays instead. The chart also contains a minimal overlay per cloud, including one for GKE. These overlays are inside the chart archive, and helm -f reads only the local filesystem, so -f values-gke.yaml fails with no such file or directory until the chart is extracted.
That leaves a regscale-orchestration-hub/ directory containing all three overlays. Install, layering the cloud overlay under your own values:
Retain this directory. The same overlay path is required for every upgrade.
CrashLoopBackOff with Error: Can't drop privilege as nonroot user.The app container runs supervisord, which manages nginx and gunicorn as separate processes and must start as root to setuid down to the service account. The chart's default settings for this tier are runAsNonRoot: true, runAsUser: 1001, readOnlyRootFilesystem: true, and capabilities.drop: [ALL], none of which permit that operation.
Workers and the scheduler are unaffected: they run Celery directly as an unprivileged user and stay fully hardened. Only the app tier needs this, so add it to your values file:
The capabilities: {drop: []} entry is required. Helm replaces list values rather than merging them, but only where the key is supplied. If the key is omitted, the default drop: [ALL] remains in effect and removes the SETUID and SETGID capabilities required by supervisord and the CHOWN capability required by nginx. The resulting failure does not obviously indicate this setting as the cause.
Setting runAsNonRoot: false at both levels is also deliberate. The container value is what Kubernetes enforces, but leaving runAsNonRoot: true on the pod while running as UID 0 is a contradiction that reads as a mistake to anyone auditing the manifest, and to some admission policies.
supervisord configuration or removal of supervisord. Until that change is released, the chart's hardened defaults and the image's requirements are incompatible, and this override is the supported resolution.The chart validates the supplied values at render time and reports a specific error if a required value is missing, preventing a partially installed release.
alembic upgrade head); only after it completes do the app, workers, and scheduler start. That is why the database must be reachable before you install.Create the database once, after the SQL Server pod is Ready:
Substitute the database.incluster.saPassword value for $SA_PASSWORD, then restart the tiers:
alembic upgrade head runs at application startup and builds the schema.
CrashLoopBackOff. SQL Server reports an absent database as error 18456 state 38, with the message text Login failed for user 'sa'. This error against a correct SA password indicates that the database does not exist.1. The release installed.
A status of deployed confirms that the pre-install migration hook completed successfully. Helm does not report deployed if the hook fails.
hook-delete-policy: hook-succeeded, so Helm removes it on completion. kubectl get jobs -n regscale therefore returns No resources found, which indicates success. A Job that remains visible indicates that the migration failed.If the release is not deployed, do not continue; the remaining steps depend on a successful migration. Resource names follow <release>-<chart>-<component>, which is why the commands below use label selectors instead.
2. All pods are Running.
priority pool maintains a running replica while idle, because it is configured with min: 1. The remaining three pools scale to zero until work is queued. This is the expected state and does not indicate a failed deployment.3. The application tier responds.
4. Autoscaling is wired up (KEDA mode):
A value of READY=False generally indicates that KEDA cannot reach Redis. Verify the Redis credentials in the Secret.
5. End to end. Run a job from the RegScale Orchestration Hub interface and watch a worker pool wake up:
cli_tasks pod should appear within about 30 seconds (the KEDA polling interval), run the job, and disappear after the cooldown period (300 seconds by default).Both charts install into the same cluster, and typically the same namespace. They are separate releases with separate databases. ROH does not write to the core RegScale database, and core RegScale does not read the ROH database.
Core RegScale first, ROH second. ROH's migration hook runs at install time and its app delegates authentication to core, so a ROH install against a missing core gives you a running deployment that cannot log anyone in.
Each ROH version works with core from the version below onward. Set core's image tag to that version or later. The core chart's default image can be older.
| ROH | Earliest core | Why |
|---|---|---|
| 6.34.3.1 (chart 0.3.0) | 6.34.3.0 | ROH calls GET /api/v3/authentication/claims, which core serves from 6.34.3.0 |
ROH also needs core's Client API feature flag, FeatureFlag__ClientApi=true. The core chart 3.1.16 sets it by default through config.featureFlags.ClientApi. With an older core chart, set it yourself:
The core chart 3.1.15 pins core 6.31.1.3. With it, every ROH login fails with 401 "Could not establish the tenant and application scope". Override the core image:
Both were hit during verification, and both block you before ROH is even attempted.
storage.persistence.accessModes to ReadWriteMany by default. On AWS this requires EFS, as EBS supports ReadWriteOnce only. Without it the pod never schedules: FailedScheduling: pod has unbound immediate PersistentVolumeClaims.Provision EFS and its CSI driver, then configure the core storage class to use it. The procedure is documented in aws-setup, which uses eksctl for the IRSA configuration. Alternatively, where a single writer is acceptable, override the access mode:
app.resources.limits.memory: 2Gi. Core applies several hundred migrations on first install and loops: Applying 353 pending migrations... Out of memory. Starting migration process...Increase the limit for the first installation. A limit of 8Gi completed without restarts. The limit may be reduced after installation completes, as the elevated memory requirement applies to the initial schema build rather than to steady-state operation.
| Core RegScale | ROH | |
|---|---|---|
| Chart | regscale/regscale | regscale/regscale-orchestration-hub |
| Service | <release>-svc | <release>-regscale-orchestration-hub-svc |
| Database | its own | its own, separate |
| Redis | not required | required |
| Secret keys | jwtSecretKey, encryptionKey, sqlConnectionString | SECRET_KEY, SECRETS_ENCRYPTION_KEY, DATABASE_URL, REDIS_URL, METRICS_TOKEN |
The following procedure installs both charts into one namespace. ROH reaches core through the chart's core proxy, coreProxy in Make core see ROH logins as HTTPS. To add Synqly Embedded, follow Synqly Embedded for step 6 instead.
Helm parses --set values, and a comma is its list separator. A real RDS endpoint is written host,1433, so --set secrets.sqlConnectionString="Server=db...rds.amazonaws.com,1433;Database=regscale;..." renders as SQLConn: "Server=db...rds.amazonaws.com". All content following the first comma is discarded without warning, removing the database name and credentials. Core RegScale then fails to reach its database, and the Helm output provides no indication of the cause.
A pre-created Secret avoids this entirely, which is why the example above uses one. --set-file and escaping every comma as \, both also work, but the Secret keeps the credential out of your shell history too.
Core's Secret is consumed with envFrom, so the key names matter: JWTSecretKey, EncryptionKey, SQLConn.
Core 6.34 sets its login cookie only over HTTPS. When core does not see a login request as HTTPS, it returns 500 after the password check, and ROH shows "Incorrect username or password". So the plain in-cluster Service, http://regscale-svc..., fails every ROH login. Core takes the scheme from the X-Forwarded-Proto header. Use one of these:
| Wiring | Values | When |
|---|---|---|
| TLS ingress in front of core | regscaleUrl: https://regscale.example.com | Core already has a TLS ingress that ROH can reach. The ingress controller sets X-Forwarded-Proto: https |
| Core proxy in the ROH chart | regscaleUrl: http://<core release>-svc.<core namespace>.svc.cluster.local and coreProxy.enabled: true | ROH reaches core inside the cluster. The cloud overlays turn it on |
With coreProxy.enabled, the chart runs two nginx pods with a PodDisruptionBudget. They set X-Forwarded-Proto: https and forward to regscaleUrl, and ROH calls them instead of core. Notes:
regscaleUrl must be core's full Service name, ending in .svc.cluster.local. nginx resolves it through the cluster DNS (coreProxy.dnsResolver) without search domains. It resolves it per request, so it follows a re-created core Service.regscaleUrl is https://: core already sees HTTPS.coreProxy.clientMaxBodySize (100m, as core's default upload limit), and waits coreProxy.readTimeout (600s) for core.networkPolicy.enabled, the ROH NetworkPolicy allows port 8080 to the proxy pods.A fix in core, for example no Secure-only cookie on an API login, would make the proxy unnecessary.
Set ForwardedHeaders:KnownProxies or ForwardedHeaders:KnownIPNetworks on core. Without one, core trusts X-Forwarded-Proto from any pod in the cluster, so any pod can make a plain-HTTP request look like HTTPS. List the addresses of the proxy pods or the ingress controller. Pod addresses change, so a network range is usually easier. In the core chart, pass them as environment variables:
Do not copy a range from an example. The pod range depends on the cluster:
az aks show -g <group> -n <cluster> --query networkProfile.podCidr or gcloud container clusters describe <cluster> --format='value(clusterIpv4Cidr)'.kubectl get pods -n regscale -l app.kubernetes.io/component=core-proxy -o wide.A range that misses the proxy pods makes core ignore X-Forwarded-Proto, and every ROH login fails with 500. A range that covers every pod gives no protection. The variable name ForwardedHeaders__KnownIPNetworks__0 has not been confirmed against core's source, so after you set it, sign in to ROH and confirm that the login works.
regscaleUrl rightThis value is the only connection between the two releases and a frequent source of configuration errors. The core chart names its Service <release>-svc. Confirm the Service name, which varies with the release and namespace names:
With coreProxy.enabled, regscaleUrl stays core's Service. The proxy's own Service, <roh release>-regscale-orchestration-hub-core-proxy, is in the ROH namespace, and the chart points ROH at it.
Then confirm that a login through regscaleUrl works. Sign in to ROH as a core user. The user needs the core Administrator role and a membership in a core app. The seeded core user admin does not work for ROH: core gives it only GlobalAdmin and AppAdmin, and ROH answers 403.
If ROH says "Incorrect username or password", read the core log. ROH shows that message for any core 400 or 500.
A 500 with an antiforgery or cookie error means core did not see the request as HTTPS. Where core RegScale runs outside the cluster, use its external https:// URL.
regscale and ROH in roh, the URL remains http://regscale-svc.regscale.svc.cluster.local, because cross-namespace service DNS includes the namespace. Create the ROH Secret in the ROH namespace, and confirm that no NetworkPolicy blocks traffic between the two.Synqly Embedded runs the connectors that ROH calls. The ROH chart installs it as the subchart synqly/embedded 0.1.166, in the same namespace as ROH, when synqly.enabled is true. That is the default. With Synqly on, the chart sets SYNQLY_BASE_URL to http://embedded.<namespace>.svc.cluster.local:8000 in the ROH ConfigMap, and the ROH NetworkPolicy opens egress on port 8000.
To install ROH without Synqly, set synqly.enabled: false. The release then holds no Synqly resource, and the ROH ConfigMap holds no SYNQLY_BASE_URL.
embedded and the Secret name synqly-vault-access. A second ROH release with Synqly on must go in another namespace.global.synqlyLicense holds a license issued by Synqly. Upgrade the chart before then, or set the license.organizationID and organizationFullName, only when it first creates its organization. Changing them later means recreating the Synqly database. So the chart stops before it creates anything unless global.organizationAdmin.default.secretRef is set (or inviteMembers with SMTP, which sends the invitation), and unless organizationID, organizationFullName and apiAddress are set and not REPLACE-.global.imageRegistry. Set synqly.embedded.registry and synqly.managementUI.registry for a mirror.| What | |
|---|---|
| PostgreSQL | A database for Synqly. PostgreSQL 17 was tested |
| HashiCorp Vault | A Transit secrets engine, and an AppRole whose policy can read and write it. See Synqly's Vault guide |
| Registry access | A quay.io pull credential from Synqly, or a mirror of quay.io/synqly |
| Hostnames and a TLS certificate | One hostname for the Synqly API and one for the management UI, and one certificate for both |
The values file holds no credential. Create every Secret in the release namespace first. These names match values-synqly.yaml.
Helm stores every value, and every rendered manifest, in the release record. So values-synqly.yaml reads each credential from a Secret that you create, through the subchart's secretRef settings. For example, it reads the Vault AppRole from the Secret synqly-vault-access through appRole.secretRef, and the subchart then renders no Secret from values.
The chart stops if Vault AppRole or token auth is on without secretRef. It also stops if one of these is set as a plain value, at the top level global: or under synqly.global::
dbPassword, root.credentials.rootToken, root.credentials.tokenSigningKey, smtpPassword, and organizationAdmin.default.credentials.password (the subchart passes that password as a container argument)appRole.roleId, appRole.secretId and token.tokenValueaws.secretAccessKey, the Azure Key Vault clientSecret, the AWS KMS user secretAccessKey, and the GCP KMS keyJSONsynqly.pods.synqlyRegistryCredentials.password and synqly.managementUI.uiCookieEncryptionKeyPlaceholders such as <DB_PASSWORD>, the subchart defaults, pass.
Get the values file from the chart:
Edit values-synqly.yaml: replace each REPLACE- value, and set vaultURL. The chart stops while a REPLACE- value is left. Then install ROH with the cloud overlay you use, your values, and values-synqly.yaml:
Leave out the overlay line only if you installed without one. Keep --timeout 20m on every install and upgrade with Synqly. Helm waits for the allow-list Job within --timeout, 5 minutes by default, and the Job can wait 15 minutes for Synqly Embedded.
Call the Synqly API from the ROH pod on port 8000. The command must print the URL and 200.
Then open https://<uiAddress> and sign in with the user in the Secret synqly-admin.
With Synqly on, the chart sets these on ROH. They match the SaaS configuration.
| Variable | Set from | Default |
|---|---|---|
SYNQLY_BASE_URL | the subchart's Service | http://embedded.<namespace>.svc.cluster.local:8000 |
SYNQLY_ALLOW_INSECURE_HTTP | fixed | true. ISL refuses cleartext http to a host that is not loopback, and Embedded serves http inside the cluster |
SYNQLY_ORG_ID | global.organizationID | |
SYNQLY_ADMIN_MEMBER_NAME, SYNQLY_ADMIN_SECRET | synqlyIntegration.onboardingMember.secretRef when it is set (see The onboarding Member); otherwise global.organizationAdmin.default.secretRef (Secret synqly-admin) | app container only |
SYNQLY_CATALOG_MEMBER_NAME | synqlyIntegration.catalogMember.name | regscale-catalog |
SYNQLY_CATALOG_MEMBER_SECRET | synqlyIntegration.catalogMember.secretRef (Secret synqly-catalog-member) | app container only |
SYNQLY_TIER | synqlyIntegration.tier | nonprod; prod for production tenants |
SYNQLY_ENVIRONMENT | synqlyIntegration.environment | test; prod for the Synqly prod environment |
SYNQLY_ONBOARD_ON_LOGIN | synqlyIntegration.onboardOnLogin | true |
ISL_ENABLED | synqlyIntegration.islEnabled | true |
ISL_SERVE_AUTOSTART | synqlyIntegration.islServeAutostart | true |
ISL_MINT_WORKER_CONCURRENCY | synqlyIntegration.islMintWorkerConcurrency | 2 |
Two more apply with or without Synqly: ROH_PUBLIC_URL from publicUrl (or the first app.ingress host), and GUNICORN_WORKERS from app.gunicornWorkers (default 2).
The workers and Beat get no Synqly logon. ROH also strips the admin and catalog secrets from every job's environment.
The catalog member. ROH lists ISL commands as the read-only Synqly member regscale-catalog, which isl manage bootstrap creates with the secret that you choose. Until it exists, ROH starts but lists no ISL category. After the install, run bootstrap once against Embedded with the same secret as the Secret synqly-catalog-member, from a machine with the isl CLI from an rs-isl release:
isl binary, so ISL_SERVE_AUTOSTART starts no ISL host in them. It takes effect with an image built with ISL.ROH onboards a tenant with a Synqly account and an account token. By default it logs on to Synqly as the default admin to do so, so a leaked ROH app pod credential reaches every tenant account in the Synqly installation. Set synqlyIntegration.onboardingMember.secretRef and ROH logs on instead as a dedicated service Member, isl-onboarder by default, on a Role with only the account-provisioner permission set. That Member can create an account and its token, and nothing else: it lists nothing and makes no Role and no Member.
It needs an ISL release that has REG-33941 (rs-isl 1.1.0.0 or later). An older ISL makes a Role and a Member for each tenant, which this Member cannot do: ISL answers 403 Forbidden, and ROH logs one error that the Member lacks the administrator permissions that this ISL version needs. Keep the default admin until the image bundles a newer ISL.
Create the Secret before the install or upgrade. Synqly does not generate member secrets, so choose one of 8 to 71 characters, and keep it:
SYNQLY_ADMIN_MEMBER_NAME and SYNQLY_ADMIN_SECRET), and it then does not read the Secret synqly-admin. Workers and Beat get neither.isl-onboarder unless nameKey or name says otherwise. The Role is isl-onboarder unless roleName says otherwise.synqlyIntegration.onboardingMember.secret or .password is set, because Helm stores every value in the release record. The release holds the Secret's name only.<release>-regscale-orchestration-hub-synqly-onboarder runs as a Helm hook. It logs on as the default admin, makes the Role and the Member if they do not exist, and then logs on as the Member with the secret of your Secret. On a second upgrade it changes nothing. It never widens or narrows an existing Role or Member and never rotates a secret: a Role on another permission set, a Member bound to other Roles, or a Member whose secret differs stops the Job with the reason. If Synqly refuses to create the Role or the Member, the Job fails with Synqly's answer in its log, and helm upgrade reports the failure. It needs the default admin Secret, so the chart requires global.organizationAdmin.default.secretRef while the Job is on. To make the Role and Member yourself, set synqlyIntegration.onboardingMember.job.enabled: false.After each install and upgrade, the chart runs the Job <release>-regscale-orchestration-hub-synqly-allow-lists as a Helm hook. It waits for Synqly Embedded to be ready, then sets the allow lists of the templates regscale-<category>, the same way isl manage allow-lists does. So the Embedded deployment offers only the providers that ROH supports. The lists come from files/synqly-allow-lists.json, a copy of rs-isl .github/synqly/allow-lists.json.
global.organizationAdmin.default.secretRef, so the chart requires that Secret while the Job is on. It revokes its session at the end.test and prod lists; a missing template is created. A category left out is not touched. A provider other than * that Synqly does not offer stops the Job before any change.synqlyAllowLists.lists in the same shape. To manage the lists yourself, set synqlyAllowLists.enabled: false.If the Job fails, or does not finish within helm --timeout, Helm marks the release failed. The Job can run for synqlyAllowLists.readyTimeoutSeconds plus 300 seconds, 15 minutes by default, so pass a longer --timeout, for example --timeout 20m. Read its log:
values-synqly.yaml turns on the management UI and serves it through the TLS ingress at synqly.managementUI.uiAddress. The chart sets the UI's APP_URL to https://<uiAddress>, and the UI's server calls that URL for its own API. So the UI needs the TLS ingress, and that hostname must resolve from inside the cluster too.
The ingress targets an nginx ingress controller. For another controller, set global.ingress.ingressClassName, and add any annotations under global.ingress.annotations.
secretRefA release that was installed with the AppRole in its values owns the Secret synqly-vault-access, and the release record holds the AppRole. With secretRef, the subchart no longer renders that Secret, so the upgrade would delete it, and Synqly would lose access to Vault. Tell Helm to keep the Secret first. Then upgrade with values-synqly.yaml, and set the old values to null:
The null values matter with --reuse-values or --reset-then-reuse-values, which carry the old values over. The new revision then holds no AppRole. Older revisions in the release history still do. Rotate the secret ID in Vault, then update the Secret synqly-vault-access.
If this installation ran a ROH release before 6.34.3.0, read Upgrading from a release before 6.34.3.0 before you run the upgrade.
Upgrades drain rather than interrupt: running jobs finish before their worker exits, so a rollout can take as long as your longest in-flight job. Reuse the same values files every time, and never change SECRETS_ENCRYPTION_KEY.
Upgrading to chart 0.3.3. The chart moves Synqly Embedded from 0.1.162 to 0.1.166 (images embedded-2026.10.22).
--timeout 20m from Install, including -f values-synqly.yaml. --reset-then-reuse-values (Helm 3.14 or later) also works.--reuse-values. It keeps the old image tag, embedded-2026.10.06, and runs it with the new subchart's arguments, a combination that was not tested.embedded-2026.10.22 images first. Otherwise the new pods stay in ImagePullBackOff while the old pod keeps serving.global.apiAddress is now also the https:// base of provider webhook, Shared Signals and JWKS URLs. Make sure that your providers can reach it.embedded has a startup probe. On every container start, liveness checks wait up to 10 minutes for it to pass. The first start needs this time for database migrations.Upgrading to chart 0.3.0. Two changes:
--reuse-values, stops before it changes anything, with a message that names global.organizationAdmin. To keep ROH without Synqly, add --set synqly.enabled=false or put it in my-values.yaml. To add Synqly, follow Synqly Embedded, and add -f values-synqly.yaml to every upgrade.coreProxy.enabled: true). ROH then calls the proxy, and the proxy forwards to your regscaleUrl. regscaleUrl stays core's address, and must be core's full Service name (the chart stops otherwise). The proxy does nothing for an https:// URL. Set ForwardedHeaders on core as in Make core see ROH logins as HTTPS. To keep ROH calling core directly, set coreProxy.enabled: false.helm rollback reverts the pods but not the database migration. If an upgrade included a schema change, roll the schema back before rolling back the release.This section applies only to an installation that ran a ROH release before 6.34.3.0. A new installation, or one that started on 6.34.3.0 or later, is not affected.
Before 6.34.3.0, two clocks filled some of the timestamps that ROH stored.
The clock of the SQL Server host filled the creation and change times of analytics dashboards and their widgets. These values are in the columns created_at and updated_at of the tables dashboards and dashboard_widgets. If that clock used a time zone other than UTC, those older values are in local time.
The clock of the ROH container filled the next scheduled run of each job (jobs.next_execution_at) and the planned start time of each scheduled execution (executions.scheduled_at). If TZ was set on the ROH containers, those older values are in local time. From 6.34.3.0, ROH replaces jobs.next_execution_at with a UTC value when the job next runs, or when you edit or resume the job. Older values in executions.scheduled_at keep their local time.
While the application ran, ROH wrote all other timestamps in UTC in those releases. Everything runs in UTC describes the clocks involved.
ROH treats the values in dashboards, dashboard_widgets and executions.scheduled_at as UTC. For an older row that holds a local time, ROH therefore shows the wrong time. ROH cannot detect or correct these rows.
ROH itself never wrote rows to audit_logs before 6.34.3.0. If another system writes to orca.audit_logs and your SQL Server uses a time zone other than UTC, those rows can hold local time. ROH treats every value in audit_logs.timestamp as UTC, so it shows the wrong time for such a row. ROH cannot detect or correct these rows. To find out if this applies, run SELECT COUNT(*) FROM orca.audit_logs;. A result of 0 means that nothing wrote to the table, and this paragraph does not apply. orca is the default schema. If you set DATABASE_SCHEMA, use that schema instead.
Before you upgrade, check both clocks. To check the SQL Server clock, which affects the dashboard columns, run this query on the SQL Server:
A different result means that older dashboard rows can hold local time. This query returns the time zone name, not the current offset. A host in a zone such as London shows an offset of 0 in winter, but it wrote local time in summer.
CURRENT_TIMEZONE() exists only on SQL Server 2022 and later, Azure SQL Database and Azure SQL Managed Instance. On an earlier version the query fails. In that case, read the time zone of the host instead: the instance's time zone setting on RDS for SQL Server or Cloud SQL for SQL Server, or the host operating system's time zone on a server you run yourself.
To check the container clock, which affects the job schedule columns, look for a TZ variable on the running ROH pods:
No output means that TZ is not set in the pod specs. A value other than UTC or Etc/UTC means that older schedule values can hold local time. Also look for TZ in your values files, because a variable that a pod loads with envFrom does not show in this output.
After the upgrade, read the version endpoint:
The fields database_utc_offset_minutes and container_utc_offset_minutes must both be 0. A request made in the first seconds after a pod starts can return null for the database field, because the database probe that runs when the pod starts has not finished. Repeat the request once a minute. If the value is still null after five minutes, ROH cannot read the database clock: look for an error that starts with Database clock in the app logs. A null container field means that ROH cannot read the container clock, and the app logs show Container clock probe failed. These fields report the current offset, so a host in a zone such as London also shows 0 in winter. The checks before the upgrade are the ones that settle the time zone.
On a default install the images are public and need no credentials. This usually means global.imageRegistry points at a mirror that lacks them, or a pinned image.tag does not exist. For a Synqly pod, check the Secret synqly-image-pull-secret and synqly.embedded.registry: the Synqly subchart ignores global.imageRegistry. Check what the chart is actually requesting:
Almost always the database: wrong DATABASE_URL, the database does not exist, a firewall blocks the cluster's egress, or the user lacks schema-creation rights. Resolve the cause, then run helm uninstall roh and reinstall. A failed migration leaves the release in a state that an upgrade does not repair.
KEDA authenticates to Redis separately from the application, so check its own wiring first:
No TriggerAuthentication at all means the chart decided no Redis password was in play. With managed Redis it renders one when either redis.managed.url carries a password or you set secrets.existingSecret; the object then reads the raw password from that secret's REDIS_PASSWORD key, so the key has to be there.
A TriggerAuthentication that exists but still gives READY=False is usually the password itself, or the scheme: redis:// against a TLS-only endpoint renders enableTLS: "false" and the connection fails.
The readiness probe currently checks only that the process answers, not that the database is reachable, so a pod can be marked Ready with a broken database connection. Check the logs directly:
Expected. The endpoint requires the METRICS_TOKEN from your secret as a bearer token. Configure your scraper with it, or use the chart's ServiceMonitor/PodMonitor options, which wire the token automatically.
Every pod logs alembic upgrade head at startup in addition to the migration Job. This has no effect, because Alembic performs no action when the schema is current. It is a known inefficiency tracked against the application image.
One resource survives: the ServiceAccount is a Helm hook annotated before-hook-creation with no hook-succeeded, so helm uninstall does not remove it. All other resources are removed: the ConfigMap and chart-created Secret are standard resources that uninstallation deletes, and the migration hook's ConfigMap and Secret copies are deleted automatically once the migration completes.
Deleting the namespace is the simplest way to leave nothing behind:
Neither command affects a managed SQL database or managed Redis instance. Data in those services is retained, which permits a reinstallation to reuse it, provided the original SECRETS_ENCRYPTION_KEY value is available.
ReadWriteMany volume on EKS; note its IRSA steps use eksctl, which this guide does not otherwise require