Background Maintenance

OutlabsAuth periodically cleans up expired tokens, aggregates optional activity metrics, and syncs Redis-backed API-key usage. In production, keep that work out of FastAPI workers.

The scheduler and executor may share a machine or run separately. A locally supervised worker is a normal production choice. Put it in the cloud only when availability or network access requires that placement. The executor needs direct access to the host Postgres database and, when configured, Redis.

Choose an entry point

CLI

Use the CLI from Cron, a systemd timer, a Kubernetes CronJob, or another one-shot runner:

export DATABASE_URL='postgresql+asyncpg://auth_worker:...@db-host/app'
export OUTLABS_AUTH_SCHEMA='outlabs_auth'
export SECRET_KEY='...'

# Required only when this host uses Redis-backed auth features:
export REDIS_URL='redis://cache-host:6379/0'
export OUTLABS_AUTH_REDIS_KEY_PREFIX='myapp:production'

outlabs-auth run-maintenance

The command initializes OutlabsAuth with background loops disabled, runs one cycle, and prints a typed JSON report. It exits 0 only when ok=true, exits 1 when a configured step is missing or reports errors, and retains Click's exit 2 for missing required configuration. Keep secrets in the executor environment or secret store—never in command arguments or schedule payloads.

The CLI constructs the standard SimpleRBAC maintenance configuration. If the host has custom feature flags or service wiring, use the programmatic entry point so the worker and API share the same auth factory.

Programmatic

Call the deterministic one-shot API from a queue task or host-owned worker:

async def run_auth_maintenance():
    auth = build_auth(background_job_mode="disabled")
    try:
        await auth.initialize()
        report = await auth.run_maintenance_once()
        if not report.ok:
            raise RuntimeError(
                f"auth maintenance incomplete: "
                f"missing={report.missing_steps!r} "
                f"errors={report.reported_errors}"
            )
        return report
    finally:
        await auth.shutdown()

build_auth() is host code. It should select the same preset, schema, Redis prefix, and feature flags as the API. It must not start embedded loops.

Understand one cycle

run_maintenance_once() runs the applicable steps in this order:

1. expired and revoked refresh-token cleanup, when enabled;
2. activity aggregation, when activity tracking is enabled;
3. API-key usage sync, when the API-key service and Redis are available.

The returned immutable MaintenanceReport makes operational success explicit:

Redis-enabled Auth expects api_key_usage_sync, so unavailable configured Redis is reported as missing instead of looking like an empty success. run_background_jobs_once() remains available as a backward-compatible raw dictionary.

Delivery and retry behavior

A cycle is not one transaction across all three steps. Token cleanup and activity sync commit independently, while API-key sync has its own durable batch/receipt flow. A later failure can leave earlier work committed.

Treat delivery as at least once:

• retry a failed invocation instead of trying to roll back the whole cycle;
• require report.ok instead of interpreting a missing result key as a successful zero-count run;
• record the exit status and typed report fields;
• keep host wrappers idempotent and tolerant of partial progress.

The built-in operations are retry-safe. API-key usage sync stages Redis counters and records a database receipt so a retry does not double-apply a committed batch.

Production safety contract

Before activation, verify that:

• exactly one logical scheduler clock owns this database and environment;
• every API replica uses background_job_mode="disabled";
• the executor uses restricted runtime credentials, not migration-owner credentials;
• the executor can reach Postgres and the host Redis, when Redis is enabled;
• secrets live in the executor environment, never in a schedule manifest;
• the schedule starts paused, forbids overlap, and uses queue concurrency one unless the host has proved parallel runs safe;
• scheduler lag, worker availability, job outcomes, and result counts are monitored independently of the API /health endpoint.

Do not rely on the one-clock rule alone for correctness. Schedulers and queues can redeliver work, so execution must retain at-least-once semantics.

Choose the cadence

The host owns the cadence. There is no universal five-minute interval.

The one-shot call runs every enabled, applicable step on every invocation. token_cleanup_interval_hours, activity_sync_interval, and api_key_usage_sync_interval control embedded loops; they do not skip work inside run_maintenance_once().

Prevent accidental production execution

Pausing a schedule prevents new scheduled occurrences; it does not make the worker configuration safe.

• Use distinct database credentials, Redis prefixes, task namespaces, and queues for development, staging, and production.
• Never fall back to a production URL when a local environment variable or dotenv file is missing.
• Require an explicit environment label and fail startup when it conflicts with the selected queue or expected database identity.
• Log the environment, database host/name, schema, Redis prefix, and queue at startup with credentials redacted.
• Inspect and drain stale queued jobs before attaching a worker to production.

A local worker can intentionally serve production. The boundary is its explicit configuration and restricted credentials, not whether it runs in a container or in the cloud.

Optional TaskQ pattern

TaskQ is one possible host scheduler; it is not an OutlabsAuth dependency. Register a host task that calls run_maintenance_once(), converts report.ok=false into a retry, then starts with a paused source manifest:

version: 1
namespace: myapp
source: api-deployment
schedules:
  auth-maintenance:
    display_name: OutlabsAuth deterministic maintenance
    task: myapp.auth.maintenance
    queue: auth_maintenance
    interval_seconds: 300
    catchup: fire_once
    overlap: forbid
    max_lateness_seconds: 900
    state: paused
    payload:
      mode: all

The task name and payload belong to the host adapter; they are not built into OutlabsAuth. The scheduler only enqueues. A supervised worker—local or remote—executes the task and needs Postgres/Redis connectivity.

Activating an interval schedule is normally from now: the first occurrence becomes due after one full interval. Use an explicit one-shot job when you need an immediate canary.

Activate safely

Roll back

Use a stop-first rollback:

1. stop the scheduler clock;
2. pause the schedule;
3. let an in-flight job finish or drain it, then stop the worker;
4. keep API maintenance disabled and use manual one-shot runs if necessary.

Only restore background_job_mode="embedded" temporarily when the host is provably single-process. It is unsafe in a multi-replica API because every replica can become a scheduler.

Test the host integration

• Run the one-shot entry point against representative Postgres and Redis and assert the exact expected steps.
• Repeat it and simulate a retry after partial completion.
• Prove one scheduled occurrence creates one worker job.
• Prove API startup does not create a second maintenance owner.
• Rehearse pause, drain, and manual one-shot rollback before production.

Related

• Configuration
• Deployment
• Testing
• Observability
• TaskQ documentation