Sizing, scaling, upgrades and operational runbooks for a self-hosted deployment
This guide covers operating the Orchestration Hub after installation: sizing, scaling, upgrading, and resolving common failures. Complete the Orchestration Hub installation guide first.
Audience: platform and infrastructure engineers who own the cluster hosting the Orchestration Hub.
| Section | |
|---|---|
| 1. Resource sizing | CPU and memory settings |
| 2. Scaling | Worker replicas, autoscaling, job splitting |
| 3. Versions | Which version you are running, and compatibility |
| 4. Upgrading | Procedure, timings, rollback |
| 5. Troubleshooting | Diagnosis and resolution by symptom |
Plan capacity properly. When running a job execution that will synchronize more than 100,000 records from an integration, use the Parallel Job Splitting feature and/or modify the frequency of scheduled jobs to decrease the size of each execution. Increasing workers will not decrease the job's execution time. It may over-saturate the RegScale instance, further increasing duration.
Size the RegScale database for both write throughput and storage. Storage exhaustion affects every user of the RegScale instance, not only Orchestration Hub's jobs.
The chart's default cli_tasks worker settings are the recommended configuration. No action is required to adopt them:
Limits and requests differ deliberately: the limit is the ceiling a job may reach, and the request is what is reserved for scheduling.
Leave the application tier, the scheduler and the coordinator, priority and celery worker pools at chart defaults. They are not the constraint during ingestion.
Worker CPU stays within a 4 core limit. Memory tracks the size of the synchronization, and is lower when a job is split.
Peak worker memory by volume:
| Records per sync | Un-split | Split |
|---|---|---|
| 5,000 | 450 MiB | 360 MiB |
| 10,000 | 580 MiB | 420 MiB |
| 20,000 | 690 MiB | 420 MiB |
| 50,000 | 1,330 MiB | 810 MiB |
| 100,000 | did not complete | 1,270 MiB |
To size your own deployment, find your largest routine synchronization in the table, take the figure for the mode you run it in, and add headroom. The chart's 4 GiB limit covers every value in the table, which is why it is the recommended starting point.
| If | Then |
|---|---|
| Synchronizations stay at or below 20,000 records | 1 CPU / 1 GiB is sufficient. Reduce only if cluster capacity requires it; a 690 MiB peak against a 1 GiB limit leaves little margin |
| Synchronizations exceed 20,000 records, or the workload is mixed or growing | Keep the chart defaults |
| Synchronizations exceed 100,000 records | Keep the chart defaults and split the job |
| Workers are terminated during large jobs | Raise the memory limit. See 5.1 |
These conditions raise the figures a deployment can sustain:
| Condition | Effect |
|---|---|
| A faster RegScale database tier | Larger single jobs complete, and additional replicas contribute before contention sets in |
| More database storage and a shorter retention period | More ingestion before cleanup is required |
| A connector that transforms rather than forwards data | Worker CPU begins to matter, and a larger CPU limit pays off |
Measure your own peak usage before changing the default:
Each worker pod runs one job at a time, so the replica count is the number of synchronizations writing to RegScale concurrently. Two replicas is a sensible floor.
Additional replicas do not make an individual synchronization faster. While the database is the constraint, they add contention.
Worker autoscaling is driven by job queue depth. Approximately one worker starts per queueLength queued items, up to max:
queueLength. Raising max alone has no effect: at queueLength: 4, a ten-part job requests about three workers whether max is 4 or 20.min: 1 for a queue where the cold-start delay on the first job matters.gracePeriod below your longest expected job. Scale-down drains a worker rather than interrupting it. A shorter value terminates work in progress.Splitting divides one synchronization into independently processed parts. Use it to complete a load a single job cannot, not to make a load faster.
| Records per sync | |
|---|---|
| Below 10,000 | Do not split. Splitting is slower than a single job at this volume |
| 20,000 to 50,000 | Split on a busy RegScale instance; a lightly loaded one handles this range unsplit |
| Above 50,000 | Split |
| Above 100,000 | Always split. Unsplit synchronizations do not complete at this size |
| Version | Identifies |
|---|---|
| Chart version | The chart packaging |
appVersion | The Orchestration Hub release the chart deploys |
List both before upgrading:
image.tag defaults to empty and resolves to the chart's appVersion, so upgrading the chart upgrades the application. There is no second version to set.
Setting image.tag explicitly overrides this. The pinned tag persists across later chart upgrades and will hold the application back, so clear it once a chart carrying the fix is available.
The web interface is built into the application image. There is no separate interface version.
The Orchestration Hub delegates authentication to the core RegScale application and calls its API to read and write records, so the two are coupled at runtime. Follow these practices to keep them aligned:
version must match the APP VERSION of the chart release you deployed. If it does not, either the rollout has not finished or image.tag is pinned.
No outage window is required. No component that serves users goes offline during an upgrade.
Take a database backup or snapshot, and confirm it completed. This is your rollback mechanism. See 4.4.
Then:
helm search repo for the application version the target chart carries.helm get values roh --namespace regscale > values-backup.yaml.Set --timeout 30m. Helm's 5 minute default applies to the schema migration, which is allowed 30 minutes. A migration taking longer than 5 minutes will report a timeout on an upgrade that is succeeding.
Do not use --atomic. It rolls the release back on failure or timeout. The migration has already committed by that point, and an automatic rollback produces the failure in 4.4.
If the command reports a timeout, check the migration before concluding it failed:
The schema migration runs first, as a pre-upgrade hook. The upgrade does not proceed unless it succeeds. Workloads then roll:
| Component | Interruption | Notes |
|---|---|---|
| Application (API and interface) | None | A replacement replica passes its readiness probe before a current one is removed |
cli_tasks and coordinator workers | None | Running jobs finish. Each is allowed 4h15m to drain |
priority and celery workers | None | 2 minute drain |
| Scheduler | Brief pause in job dispatch | Replaced rather than rolled, because two schedulers would dispatch every job twice. Scheduled jobs start late; they are not skipped or duplicated |
| Migration job | Not user-facing | 30 minute limit |
With --wait, the command also waits for the worker drain, which can extend it by up to 4h15m. Upgrade when the job queue is quiet if you need it to finish promptly.
Rolling back requires restoring the database as well as reverting the chart. helm rollback reverts the Kubernetes objects only. Where the upgrade applied a schema migration, the database is then ahead of the restored image and the pods do not start, reporting Can't locate revision identified by <revision>.
To roll back:
helm rollback to the previous revision.Restore the database and revert the chart together. Doing either alone leaves them out of step.
A rollback without a database restore is safe only if the upgrade applied no migration. If you cannot confirm that, restore the database.
Prefer rolling forward. Where the goal is to fix a problem rather than return to a specific version, a later release avoids the database restore entirely.
curl -fsS https://roh.example.com/api/v1/version returns the expected application version.kubectl get pods --namespace regscale shows all pods running and ready.Upgrade one minor chart version at a time. A single large jump applies every accumulated migration in one 30 minute window, with nothing to fall back to except the database snapshot.
Resource names carry the Helm release name, so these commands select by label. Substitute your namespace for regscale.
Check the RESTARTS column first when a job has failed with no apparent error. A worker that restarted during a job did not complete it.
Cause. The worker exceeded its memory limit and was terminated.
Confirm it. Check the pod directly; the Orchestration Hub job record does not reflect a kernel termination:
Reason: OOMKilled with Exit Code: 137 confirms it.
Fix. Raise the memory limit for the affected queue:
Splitting the job is the alternative, and reduces peak memory per worker.
Raise the limit before re-running. Interrupted parts are re-dispatched a limited number of times. Parts that exhaust those attempts do not recover when the parent job is re-run.
Cause. The autoscaling threshold, not the replica maximum, governs how many workers start.
Confirm.
KEDA must be installed, and READY and ACTIVE should both be True for a queue with work waiting.
Fix. Lower queueLength, per 2.2. Raising max alone will not change the behavior. Confirm the database can absorb the added concurrency first.
Cause. A split job's status reflects the parent job's completion. Individual parts can fail independently.
Confirm. Compare the record count in RegScale against the source system, then check the per-part results in the job's execution.
Fix. Re-run with fewer, larger parts. Verify the record count after any synchronization that matters.
Cause. RegScale's database applies a time limit to a single write transaction. The threshold depends on database load, so the same job size can succeed on a quiet database and fail on a busy one.
Confirm. The job runs for several minutes, then fails against the write to RegScale rather than the read from the source. Smaller synchronizations of the same type succeed.
Fix. Keep a single unsplit synchronization to approximately 20,000 records on a busy RegScale instance, and split above that.
Cause. A single task that runs beyond approximately 90 minutes is recycled and restarted, so a synchronization that cannot finish inside that window restarts indefinitely and cannot complete as a single unit. Additional CPU or memory does not extend the window.
Confirm. The job's start time moves at roughly 90 minute intervals while the worker shows no memory pressure and the database shows no strain.
Fix. Split the job so each part completes inside the window. Increasing worker resources does not help.
The cause is usually an Orchestration Hub job consuming the shared database's storage.
Cause. The RegScale database has reached its storage limit. Logins fail and API calls time out for every user of the instance. The Orchestration Hub shares this database with RegScale, so sustained ingestion consumes storage RegScale depends on.
Confirm. Check consumed size against the configured maximum using your database platform's tooling. Do not diagnose from application logs; a log buffer can retain storage errors from an earlier incident.
Fix. Raise the storage allocation first, then reduce consumption. A database at its limit can be too full to delete from.
Prevent.
Cancel it through the Orchestration Hub, in the interface or the API:
Canceling a split job also cancels its parts. Deleting the job cancels any running execution.
Then confirm database write activity has stopped. Cancellation is not instantaneous; work already submitted continues until those transactions complete, which can take minutes on a large job. Watch database activity rather than the job's status.
Do not delete worker pods to stop a job. Terminated work is re-dispatched, so this restarts the work rather than ending it, and risks the permanent part loss in 5.1.
Cause. An execution was left running after an interruption, such as a worker being replaced mid-job.
Fix. The Orchestration Hub reconciles stuck executions hourly, resolving those running more than four hours or pending more than one. To act sooner, cancel the execution per 5.7 and re-run it.
Before re-running, rule out the two causes that will otherwise recur: worker memory (5.1) and the 90 minute task window (5.5).