Backup, Recovery, and Upgrades

About 5 min

Backup, Recovery, and Upgrades

1. What to Back Up

Object	Contents
Platform PostgreSQL	Users, conversations, model settings, approvals, audits, usage, knowledge bases, extracted attachment text, workspace, managed database passwords, API tokens, and external service settings
`backend-data`	Conversation images, exports, and possibly automatically generated keys
`cli-config`	Database Context configuration
`.env`, overrides, and related keys	Passwords, encryption keys, sign-in signing keys, metrics tokens, and other secrets
Images and deployment materials	Images, templates, release notes, and reverse proxy settings required to restore the same version

Encrypt keys and store them separately. Copy backups off site and regularly rehearse recovery. If backend-data contains automatically generated keys, treat it as sensitive material too.

Platform backups do not include business data in TimechoDB/IoTDB. Restoring the platform does not undo writes already executed in business databases. Database administrators must back up those databases separately.

2. Back Up the Bundled Platform Database

Validation scope: Backup and recovery of the bundled PostgreSQL have been rehearsed in an internal test environment. External PostgreSQL backup commands and scheduled backup jobs have not been verified. Perform a backup/recovery rehearsal in the target environment before deployment. See Upgrade and Rollback for upgrade validation requirements.

Routine online backup

Run in the same Bash session. The destination must be writable and have sufficient space:

set -euo pipefail
umask 077
cd /opt/timecho-agent/deploy
B="/var/backups/timecho-agent/$(date +%F-%H%M%S)"
mkdir -p "$B"
docker compose exec -T postgres sh -c 'pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Fc' > "$B/db.dump"
docker compose exec -T backend tar czf - -C /app/data . > "$B/backend-data.tgz"
docker compose exec -T backend tar czf - -C /app/home/.config/timecho-cli . > "$B/cli-config.tgz"
docker compose exec -T postgres pg_restore --list < "$B/db.dump" > "$B/db-manifest.txt"
tar tzf "$B/backend-data.tgz" > /dev/null
tar tzf "$B/cli-config.tgz" > /dev/null
(cd "$B" && sha256sum db.dump backend-data.tgz cli-config.tgz > SHA256SUMS)

Securely back up .env, overrides, keys, and deployment materials separately. Copy completed backups off site and regularly rehearse recovery.

Quiesced backup before an upgrade

An online database backup and the subsequently archived attachments do not form an instantaneous snapshot. Before an upgrade, or when a strict recovery point is required, ask users to pause usage, leave PostgreSQL running, and stop the frontend and backend before backing up:

set -euo pipefail
umask 077
cd /opt/timecho-agent/deploy
B="/var/backups/timecho-agent/maintenance-$(date +%F-%H%M%S)"
mkdir -p "$B"
docker compose stop frontend backend
docker compose exec -T postgres sh -c 'pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Fc' > "$B/db.dump"
docker compose run --rm -T --no-deps --entrypoint tar backend czf - -C /app/data . > "$B/backend-data.tgz"
docker compose run --rm -T --no-deps --entrypoint tar backend czf - -C /app/home/.config/timecho-cli . > "$B/cli-config.tgz"
docker compose exec -T postgres pg_restore --list < "$B/db.dump" > "$B/db-manifest.txt"
tar tzf "$B/backend-data.tgz" > /dev/null
tar tzf "$B/cli-config.tgz" > /dev/null
(cd "$B" && sha256sum db.dump backend-data.tgz cli-config.tgz > SHA256SUMS)

After backup:

Retain materials: Save the keys, .env, overrides, and deployment files for the same recovery point separately.
Validation failure: Keep logs, resolve the problem, and do not proceed with an upgrade.
Backup only: After success, run docker compose up -d and verify the service.
Proceeding with an upgrade: Leave the application stopped.

External PostgreSQL: Ask the database administrator to back it up according to their procedures. Still retain attachments, Contexts, and keys. Do not use commands targeting the bundled postgres container.

3. Recovery Example

Recovery overwrites target platform data. Confirm the recovery point, target environment, and how to preserve current data before proceeding.

Prerequisites:

Images and configuration for the same version are ready, and the keys match the backup.
Use a new environment or prepared empty volumes.
Target attachment and CLI configuration volumes must be empty. Do not extract into volumes that are in use or already contain data, which could mix files from different recovery points.

Bundled PostgreSQL example:

set -euo pipefail
cd /opt/timecho-agent/deploy
B="/var/backups/timecho-agent/<CONFIRMED_BACKUP_DIRECTORY>"
(cd "$B" && sha256sum -c SHA256SUMS)
docker compose config -q
docker compose down
docker compose up -d --wait postgres

# Deletes and recreates this project's platform database; confirm the target first
docker compose exec -T postgres sh -c 'dropdb -U "$POSTGRES_USER" --if-exists "$POSTGRES_DB" && createdb -U "$POSTGRES_USER" "$POSTGRES_DB"'
docker compose exec -T postgres sh -c 'pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --no-owner --exit-on-error' < "$B/db.dump"

# Target volumes must be empty; one-off containers extract files without starting the backend
docker compose run --rm -T --no-deps --entrypoint tar backend xzf - -C /app/data < "$B/backend-data.tgz"
docker compose run --rm -T --no-deps --entrypoint tar backend xzf - -C /app/home/.config/timecho-cli < "$B/cli-config.tgz"
docker compose up -d
docker compose ps

External platform database: Have the database administrator restore it into an empty database first, then restore both application volumes and start the services. Do not use docker compose down -v for routine shutdown, because it removes volumes.

After recovery, verify:

Existing accounts can sign in, and conversations, knowledge bases, attachments, and workspace files are complete.
Model keys, managed database passwords, and external service authorization can be decrypted.
Database connections work and a real read-only query succeeds.
API token and external tool configurations and calls work, if used.

Recovery boundary: Platform data created after the backup is not automatically recovered. Pending approval requests do not resume.

4. Upgrade and Rollback

Before upgrading:

Read the target version's release notes.
Verify upgrade and recovery in a test environment.
Check old and new Compose templates and PostgreSQL major versions.
Notify users to stop writes and arrange a maintenance window.
Back up platform PostgreSQL, backend-data, cli-config, .env, keys, and deployment materials.
Retain old images, configuration, and data volumes.
Ensure two backend instances will not use the same platform database simultaneously.

Upgrade within the same PostgreSQL major version

cd /opt/timecho-agent/deploy

docker compose stop frontend backend
docker load -i /path/to/timecho-agent-<NEW_VERSION>-images.tar.gz
# Update .env and Compose templates according to the release notes
docker compose config -q
docker compose up -d
docker compose logs --tail=200 backend
docker compose ps

After startup, verify in order:

Backend and frontend status.
/api/v1/system/info or the system status endpoint.
Administrator sign-in.
Model connectivity.
Contexts and read-only queries.
Knowledge bases, conversations, attachments, and workspace files.
Approvals, audits, and API tokens.

Upgrade PostgreSQL 16 to 18

A PostgreSQL major-version upgrade requires export and recovery. Replacing the image while reusing the old data volume is insufficient.

Stop the frontend and backend, retaining the old PostgreSQL service.
Export the old database with pg_dump.
Create a new empty PostgreSQL 18 database.
Import the backup with pg_restore.
Restore backend-data, cli-config, and keys from the same recovery point.
Update Compose and start the new services.
Verify sign-in, models, Contexts, knowledge bases, attachments, and audits.
Retire the old environment only after verification succeeds.

Retain old images, configuration, and data volumes before upgrading. Do not mount a PostgreSQL 16 data directory directly into a PostgreSQL 18 image.

Rollback

After application database migrations, do not rely on the new version to downgrade automatically. Use:

Pre-upgrade images.
Pre-upgrade Compose templates and .env.
A complete pre-upgrade database backup.
Pre-upgrade backend-data and cli-config.
Encryption keys matching the backup.

Preserve data created after the upgrade before restoring according to Recovery Example. After rollback, verify administrator sign-in, read-only database queries, model calls, knowledge bases, and attachments.

Upgrades and rollbacks interrupt conversations and may invalidate pending approvals. Notify users when the service is available again.

5. Cold Standby Failover After Host Failure

Prerequisite: Only one backend can use a platform database; concurrent replicas are unsupported. If an external platform database is still healthy, first determine whether only the application host needs switching. Do not restore an older backup solely to perform failover.

Confirm the original backend is stopped or isolated.
Restore configuration and data for the same version on the standby host.
Complete the checks in Recovery Example, then switch the access entry point.
When the original host recovers, leave its old backend stopped.

If startup still fails: If the original backend is confirmed stopped or isolated but the system still reports another instance using the platform database, ask a database administrator to investigate or contact the Timecho team. Resolve the issue before starting the new instance.