Administration & operations

Operator console#

Use More → Admin when the connected account has the onchain admin role, or open the administrator route. Merely viewing that route does not grant permission. Administration uses the owner passkey and a funded administrator account; it is not an arcade-key permission.

The UI provides tournament creation, independent game/market pause and resume controls, and indexed integrity signals. Tournament creation requires its contract role and validates closing time, capacity, entry fee and funded prize. Signals are diagnostic and do not apply automatic sanctions.

Service responsibilities#

Production runs the web app, relayer, Envio, Hasura, PostgreSQL, Caddy and separate RPC gateways on the VPS. Caddy manages HTTPS. Internal services and database credentials are not public browser endpoints.

Gameplay commands take priority over payment and historical work. Sponsorship's daily ceiling is disabled, while balance reservations, gas-price limits, abuse quotas and market solvency constraints remain enforced. Monitor relayer funding and /api/health alongside queue, receipt and payment-worker errors.

Backup and deployment#

Before a compatible release, record the active commit and images, create a private database/configuration backup and wait for active games to finish before restarting their services. Backups use seven-day retention with a protected offsite copy; that copy depends on the destination workstation being available.

Service tests belong to a private, disposable VPS environment with separate chain, accounts, database and network. Never attach that environment to production financial state. Preserve the unique relayer nonce journal across application versions.

Recovery and rollback#

Recover already signed transactions from their persisted exact bytes and receipts. Do not delete the journal or reset the sender nonce to make a stuck queue disappear. Likewise, reconcile a restored database with the chain before allowing new writes.

Use the repository's compatible V4 rollback procedure with an existing release. It verifies immutable contract bindings; an older V3-only binary is not a valid rollback after accepting V4 transactions. Application rollback does not reverse blockchain transactions.

The operations runbook contains the maintained commands. Secrets, backup contents and operator credentials are intentionally excluded from this public guide.

Contract-level limits#

Contracts are immutable and sealed. Replacing an immutable treasury requires redeployment. Financial debts reserved for rejected recipients cannot be reclaimed as treasury surplus. Keep all supported legacy withdrawals, claims and result references available during an upgrade. The isolated Interlude lab uses its own deployment manifest and operator-hosted node. Do not route V4 jobs or financial settlement through that node. Its delegation lifetime and any operator restart need separate monitoring and validation.

Continuous delegation and historical recovery#

The rooms pressure and lifecycle keys are distinct from the financial relayer sender. Load their private read-only files through ROOMS_PRESSURE_KEY_FILE and ROOMS_LIFECYCLE_KEY_FILE; never put them in a manifest or client bundle. Enabling Chaos requires a sealed finance configuration and an enabled lifecycle worker.

il_engine_jobs binds the engine sender nonce to app and delegation epoch. relay_jobs remains the unique financial journal and records exact finance app and round action. il_lifecycle_jobs records the operator's signed raw transactions before broadcast. Do not run another deployer command with that operator account while lifecycle processing owns its nonce.

Current v3 arenas remain delegated continuously. Their observed expiresAt = 0 does not require periodic renewal. The operator policy blocks automatic closures before signature, including age-based rotation and node replacement. Points, the next match and result publication use the existing live session. Publication and financial reconciliation run independently of movement commands.

Do not call undelegate, forceClose, closeEngine or an equivalent helper as a routine recovery step. A missing response retains its original journal until reconciled. An exceptional closure requires an explicit operational decision and preserves every affected result. Only delegations already closed may proceed through the actual stakeUnlockAt, release and exact result-root verification. Historical deployments keep their original recovery procedure and financial generation; they are never silently treated as a current v3 session.

Keep rooms tables, accepted finance manifests and payment workers when rolling the UI back. Restoring a pre-rooms relayer binary would strand or misinterpret signed jobs. Do not delete pending transactions or reset nonces. Preserve V4 and earlier vault references, even after closing a rooms deployment. See the Chaos operations record.

State transport and diagnostics#

ROOMS_STATE_STREAM_ENABLED=true enables the applied-event adapter on the server and advertises it to new browser sessions. Set it to false and restart the relayer to return to serialized polling while retaining the lobby, receipt and session-recovery fixes. This changes no contract or financial rule. An open tab can refresh after the flag changes.

The additive il_presence, il_result_pending and il_rpc_diagnostics tables separate heartbeats, durable result processing and private measurements from the lobby document. Never drop the pending-result table during rollback: retained results must be reconciled before removing their jobs. Diagnostics aggregate counts per second, distinguish network calls from locally blocked attempts and retain data for seven days. They are not an independent source of scores, grants or payouts.

Profile engine traffic by method and component, not just the total HTTP count. PONGIT's own account/IP guards are separate from an upstream quota. A 429 alone does not establish that quota. Include UTC times, method, response id and measured traffic windows when reporting an upstream rejection.

Early payment audit#

New testnet markets accept the first published Interlude result without the challenge wait. The relayer persists acceptance metadata, hub challenges, availability disputes, resolutions, unwinds and result-hash corrections in PostgreSQL. Inspect il_settlement_challenge_report and il_settlement_checks; join market payment receipts to verify actual transfers. Console events use rooms-settlement-audit and rooms-accepted-result-corrected. A challenge is not necessarily proven fraud. The audit records canonical and orphaned observations and does not issue compensation.

Each market and queued transaction keeps its financial generation. The worker captures newly published results while the delegation stays open. Publication on Monad is distinct from the end of contestability. An exceptional migration must preserve unsettled results and pending payments. Existing financial contracts and withdrawal rights remain available separately. See the repository's docs/EARLY_TESTNET_PAYMENTS.md for deployment and rollback procedures.