> For the complete documentation index, see [llms.txt](https://suigar.gitbook.io/suigar-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://suigar.gitbook.io/suigar-docs/game-onboarding/mainnet-backend-incident-2026-05-23.md).

# Mainnet backend incident

## Cause

The VPS had a password-enabled Unix account named `github` that was created for a private CI runner on 2026-04-28. Host SSH password authentication was effectively enabled by cloud-init, so that account was reachable from the public internet.

On 2026-05-23, source `179.133.60.100` tried several SSH usernames, then got an accepted password login for `github` at 16:43:20 UTC. The attacker changed the `github` password at 16:43:33 UTC and installed cron persistence under `/home/github/.configrc7`.

This was not caused by GitHub Actions executing repository code. The weakness was the public SSH password path for a manually created runner account.

## Containment Applied

* Removed the `github` Unix account and quarantined its home directory.
* Removed repo-level private CI runners from GitHub.
* Disabled SSH password and keyboard-interactive login.
* Restricted SSH login to the root key path.
* Added fail2ban for SSH.
* Persistently blocked `179.133.60.100`.
* Blocked external access to published Postgres ports on the VPS public interface.
* Quarantined runner and malware artifacts under the incident directory on the VPS.

## Project Controls

* Workflows must use GitHub-hosted runners.
* Formatting checks run read-only and no longer push commits from CI.
* CI blocks runner-selection references under `.github`.
* Backend deploys reuse an existing immutable ECR commit tag instead of failing reruns.
* Backend healthchecks allow enough startup time for mainnet recovery before Traefik removes the route.
* Mainnet recovery flags can disable non-critical live workers and expensive reads:
  * `TELEGRAM_BOT_ENABLED=false`
  * `TELEGRAM_NOTIFICATION_SOURCE_ENABLED=false`
  * `DISCORD_BOT_ENABLED=false`
  * `X_TWEET_RELAY_ENABLED=false`
  * `DAILY_STREAK_LIVE_SCAN_DISABLED=true`
  * `ADMIN_ANALYTICS_REFRESH_DISABLED=true`
  * `STATS_READ_MODEL_REFRESH_DISABLED=true`
  * `LEADERBOARD_REFRESH_DISABLED=true`
  * `WEEKLY_CONTEST_STARTUP_SYNC_DISABLED=true`
  * `WEEKLY_CONTEST_GROWTH_HINT_DISABLED=true`
  * `STATS_POOL_APYS_DISABLED=true`
  * `INDEXER_DISABLED=true`
  * `LIQUIDITY_REBALANCE_ENABLED=false`

## Performance Recovery

After the initial containment, mainnet backend latency was still degraded. The high CPU process named `MainThread` was the production Node backend (`node dist/src/main`), not a runner process. CPU profiles showed the backend burning time in Prisma/GC through Telegram notification-source scans and analytics refresh workers, then in Sui RPC protobuf parsing through live on-chain polling. Postgres also had long-running admin analytics CTEs.

Temporary live recovery on 2026-05-24:

* Disabled Telegram notification-source cron methods in the running Node process.
* Disabled weekly contest scheduled refresh and growth-hint paths in the running Node process.
* Disabled admin analytics, stats read-model, and game leaderboard refresh workers in the running Node process.
* Disabled indexer listeners and the liquidity rebalance loop in the running container.
* Patched the same compiled files inside the current container so a restart of that container keeps the recovery behavior until the next image deploy.
* Added `LIQUIDITY_REBALANCE_ENABLED=false` and longer idle sleep to `/root/main-backend-runtime.env` for future container recreates.
* Snapshotted the patched container as local image `suigar-backend:incident-20260524` on the VPS.
* Cancelled and then terminated the stuck Postgres analytics backend connection.
* Verified post-recovery state: backend healthy, Postgres `0 active|0 idle_xact`, no runner processes, no sockets from `179.133.60.100`, and backend Docker CPU down from sustained multi-core saturation to roughly 24% on the cooldown sample.

## Follow-Up

* Rebuild the VPS from a clean image when there is a maintenance window.
* Deploy a fresh backend image from the hardened repository changes; the current compiled-container hotfix is for live recovery only.
* Rotate `VPS_SSH_KEY` and the root authorized deploy key after the rebuild.
* Rotate app secrets that could be exposed if later evidence shows root access. Current evidence shows access as the unprivileged `github` user only.
* Keep database ports unavailable from the public internet; use SSH tunneling or a private network for direct database administration.