Routine Operations
Routine Operations
1. Routine Checks
Check more than whether containers are running: also inspect logs, disk, memory, model services, database execution channels, and backups. Service control commands are in Service Start and Stop Commands.
| Item | What to check |
|---|---|
| Services | postgres, backend, and frontend are running and healthy |
| Backend logs | No recurring exceptions, repeated restarts, database connection failures, or model invocation failures |
| Database and model | A user can complete a real read-only query and a model connectivity test |
| Disk and memory | The Docker data directory has enough free space and memory usage is not continuously increasing |
| Backups | The latest backup succeeded, is readable, and has been copied to another machine or storage system |
| Logs and approvals | Failed requests, abnormal sign-ins, timeouts, and pending approvals have been checked |
Basic checks:
cd /opt/timecho-agent/deploy
docker compose ps
docker compose logs --since 1h backend
docker compose logs --since 1h frontend
docker stats --no-stream
df -h / /var/lib/dockerPay particular attention to whether:
- All three services are healthy.
- The backend repeatedly logs errors or restarts.
- PostgreSQL has connection failures or insufficient disk space.
- The model service reports authentication, timeout, or quota errors.
- The Docker data directory has adequate free space.
- The latest backup succeeded and can be read.
A healthy service does not establish that model and database execution channels work. Also verify:
- As an administrator, inspect
/api/v1/system/statusfor self-check results. - Check
/actuator/health/readinessover a controlled internal network or local port. - Complete a real read-only query as a regular user.
- Run a connectivity test on the model configuration page.
- Verify that the user is bound to the correct database Context.
- Check recent approval and audit records for anomalies.
If a problem is found, record its time, user, conversation title, error ID, and relevant logs, then follow Handle User-Reported Errors.
2. Handle User-Reported Errors
Before troubleshooting, record:
- Platform version and deployment method.
- Time of occurrence.
- User account, conversation title, and reproduction steps.
- Interface message, error ID, and expected result.
- Whether any database write has already occurred.
- Related backend or frontend logs.
| Problem | Check first |
|---|---|
| Page does not open or API returns 502 | Network entry, reverse proxy, frontend, and backend |
| Immediately redirected to sign-in after signing in | HTTPS, cookies, Set-Cookie, and reverse proxy settings |
| Cannot query a database | Context existence, user binding, database address, port, permissions, and password |
| Model timeout or failed response | Connectivity, API key, gateway quota, timeouts, and streaming configuration |
| Knowledge base returns no results | Document indexing, text layer, keywords, and knowledge base/agent bindings |
| Upload fails | Format, file size, total request size, and reverse proxy body-size limit |
| Response arrives all at once or disconnects | Proxy buffering, compression, read timeout, and streaming settings |
| API token call fails | Expiration or revocation, account status, request headers, and role permissions |
| Model keys or database passwords cannot be decrypted | Whether TIMECHO_SECRET_KEY was changed or lost |
| Write or delete operation times out | Check whether target data already changed before deciding to retry |
View recent backend logs:
cd /opt/timecho-agent/deploy
docker compose logs --since 1h backendAfter editing .env, validate the configuration and recreate or start services:
docker compose config -q
docker compose up -d
docker compose psDo not immediately resubmit a write, update, or delete operation after a timeout. Verify the actual database state first and retain the necessary logs.
If the problem persists, prepare sanitized version information, error IDs, reproduction steps, and logs as described in Get Support, then contact the appropriate administrator or the Timecho team.
3. Logs, Monitoring, and Cleanup
Logs
View the last hour:
cd /opt/timecho-agent/deploy
docker compose logs --since 1h backend
docker compose logs --since 1h frontendView a specific number of recent lines:
docker compose logs --tail=200 backendFollow backend logs:
docker compose logs -f backendDo not leave debug logging enabled in production. Apply your organization's access-control and sanitization requirements to logs containing user information, database addresses, or error context.
Resources and monitoring
Check resources and disk space regularly:
docker stats --no-stream
df -h / /var/lib/docker
docker system dfMetrics are disabled by default. When enabling Prometheus metrics:
- Use a separate metrics scraping token.
- Do not substitute an administrator sign-in token.
- Allow access to the endpoint only from the monitoring network.
- Restrict the sources that can reach it.
- Configure scraping as described in Enable Metrics Monitoring.
Cleanup
| Data | Recommendation |
|---|---|
| Audit records | Archive or clean up according to the organization's retention policy |
| Deleted conversations | Inspect trash and expired data regularly |
| Attachments and exports | Download required deliverables, then remove old files |
| Docker logs | Configure rotation to prevent disk exhaustion |
| Backups | Remove expired backups according to policy and retain off-site copies |
| Knowledge bases and shared memory | Authorized administrators should maintain them regularly and remove outdated content |
Confirm the scope and backup status before cleanup. Do not run unverified docker compose down -v or Docker volume cleanup commands that might remove platform databases, attachments, or Context configurations.
4. Recover a Forgotten Administrator Password
Prefer resetting or unlocking the account with a backup administrator account.
If no administrator can sign in:
- Prepare a temporary random password satisfying the password policy.
- Add
TIMECHO_SECURITY_ADMIN_RECOVERY_PASSWORDto the backend configuration. - Validate the configuration and recreate the backend service.
- Sign in to the administrator account with the temporary password.
- Immediately set a new permanent password.
- Remove the temporary recovery configuration.
- Recreate the backend service again.
- Confirm that logs no longer warn about a remaining recovery password and that previous sign-in tokens are invalid.
After configuration changes:
cd /opt/timecho-agent/deploy
docker compose config -q
docker compose up -d
docker compose ps
docker compose logs --tail=200 backendUse the recovery password only for controlled recovery. Do not put it in ordinary documents, chat history, or ticket descriptions. After recovery, review administrator accounts, the backup administrator, API tokens, and audit records.