Deploy the Central Hub

The Central Hub is the cloud-hosted side of FLIP — it manages projects, users, the federated learning server, and the public UI. It runs in AWS and is provisioned with Terraform/OpenTofu plus Ansible. This guide covers a standalone Central Hub deployment; for trust-side deployment see Deploy a FLIP node on-prem and Deploy a FLIP node in a TRE.

Architecture 

The Central Hub stack runs in a custom VPC with public and private subnets across two AZs:

flip-ui — static assets served from S3 behind CloudFront at the canonical subdomain (stag.flip.aicentre.co.uk / app.flip.aicentre.co.uk). CloudFront also forwards /api/* to the ALB.
flip-api — the central application API.
fl-api and fl-server — the federated learning control plane. The FL server accepts outbound gRPC connections from trust-side FL clients via a Network Load Balancer (NLB).
PostgreSQL (RDS) — managed database, private subnet.
Cognito — user authentication (TOTP MFA enforced).
SES — transactional email (invites, password reset, access requests).
Secrets Manager — AES key, database password, internal service key hash.

Operator access is via AWS Systems Manager (SSM) Session Manager — port 22 is not open on any security group.

Prerequisites 

AWS CLI configured with SSO access — see deploy/README.md.
Terraform >= 1.13.1 (or OpenTofu).
Python 3.12+ with UV.
GitHub CLI — needed to authenticate against GitHub Container Registry for image pulls.
SSH key pair at ~/.ssh/host-aws — uploaded to AWS and used as the identity file for the SSM ProxyCommand-based SSH config.
Environment file — .env.stag (staging) or .env.production (production) in the project root.
AWS Session Manager plugin — required for ssh flip and make forward-trust.

AWS profile aliases (prod, stag, dev) should be configured in ~/.aws/config so the Makefile guards can verify the active profile against the chosen environment.

Required IAM permissions 

The operator role used to provision infrastructure needs the following managed policies (or equivalent custom permissions):

AmazonEC2FullAccess
AmazonRDSFullAccess
CloudWatchLogsFullAccess
SecretsManagerReadWrite
IAMFullAccess
ElasticLoadBalancingFullAccess
AmazonSESFullAccess (optional, for email functionality)

Deployed EC2 instances themselves use scoped least-privilege roles rather than these broad permissions — see deploy/providers/AWS/iam_ecs.tf for the exact policy attachments.

Full-stack deployment 

The complete pipeline is wrapped behind a single Make target:

cd deploy/providers/AWS
make full-deploy PROD=stag    # staging
# OR
make full-deploy PROD=true    # production

This runs, in order:

github-login — GitHub CLI auth (for GHCR image pulls).
aws-login — AWS SSO auth for the selected profile.
init — initialise Terraform with the environment-specific S3 backend.
import-persistent — import existing persistent resources (Cognito, S3, Secrets) to prevent replacement.
plan and apply — apply infrastructure changes.
update-env — refresh the root env file with Terraform outputs.
ssh-config — write SSH config blocks with SSM ProxyCommand.
ansible-init — configure EC2 instances with Docker, CloudWatch, and FL assets.
deploy-centralhub — deploy hub services via Docker Compose / ECS.
deploy-trust — deploy any AWS-hosted trust services (skip when only using on-prem trusts).
status — comprehensive health checks.

The PROD variable selects the environment file (stag → .env.stag, true → .env.production) and is mapped onto TF_VAR_environment (stag or prod) so Terraform can gate prod-only RDS hardening (deletion protection, final snapshot).

Subsequent UI-only deploys do not need Terraform:

make deploy-ui PROD=stag

This rebuilds the UI from the working tree, regenerates window.js, syncs to S3, and invalidates CloudFront. There is no legacy EC2 UI container; CloudFront is the only supported UI path.

Step-by-step deployment 

For debugging or selective steps:

export PROD=stag    # or: export PROD=true

make github-login
make aws-login
make create-backend                          # one-off bootstrap of the Terraform state bucket
make init
make import-persistent
make generate-internal-service-key           # fl-server → flip-api key
make plan
make apply
make ssh-config
make ansible-init
make deploy-centralhub
make register-trusts                         # register trusts on the hub (after deploy-centralhub seeds the FL kit-slot pool)
make deploy-trust
make status

Service authentication 

The hub uses three separate authentication mechanisms (see System administration for full details):

Trust API keys — minted by the register_trust service when a trust is registered. The hub stores only the SHA-256 hash in the api_key_hash column of the trust table; the plaintext is written once into that trust’s kit file (trust/.env.<CODE>.<env>). Trusts are registered with make register-trust KIT=<CODE> (or make register-trusts for the shipped dev roster).
Internal service key — single hub-internal key for fl-server → flip-api calls. Generated with make generate-internal-service-key.
Trust-internal service keys — per-trust shared secret used inside each trust for trust-api / imaging-api / fl-client → imaging-api / data-access-api calls. The hub never sees these. Minted by register_trust alongside the trust API key and written into the trust’s kit file.

make generate-internal-service-key populates the active env file (.env.stag or .env.production) and preserves any keys that already exist; make register-trusts writes the per-trust keys into the kit files.

Applying schema changes 

The hub has no migration framework (no Alembic). On startup the entrypoint runs seed_essential_data.py, which calls SQLModel.metadata.create_all() — this only creates missing tables. It never alters an existing table, so a release that adds a non-nullable column, changes a column type, or drops/renames a column is not applied to a database that already has those tables; the new code then fails at runtime against the old schema.

The Central Hub database is treated as recreatable: it holds platform state (projects, queries, FL job / metrics / audit rows, the trust registry), not a system of record that must be migrated in place. To apply a schema-changing release, recreate the database so the entrypoint rebuilds it.

Development (docker-compose, disposable volume):

make down
docker compose -f deploy/compose.development.yml down -v   # drop the postgres volume
make up                                                    # entrypoint reruns create_all + seeders

Staging (RDS is not deletion-protected):

cd deploy/providers/AWS
make destroy PROD=stag && make full-deploy PROD=stag

Production (RDS has deletion protection + a final snapshot):

Take a manual RDS snapshot first.
Connect to the database (SSM port-forward) and reset the schema so the entrypoint can rebuild it — do not delete the RDS instance (deletion protection blocks it, and recreating churns the endpoint and secrets):
```
DROP SCHEMA public CASCADE;
CREATE SCHEMA public;
```
Force a new flip-api deploy so the entrypoint reruns create_all and the seeders.
Re-register the trusts (make register-trusts) and redistribute the refreshed kit files — the trust table and FL kit-slot pool come back empty.

Warning

Recreation is destructive: every project, cohort query, model, FL result, and trust registration is lost. Confirm the database genuinely holds no data that must be preserved before recreating a production hub. If in-place preservation is ever required, a real migration (Alembic or hand-written ALTER / backfill SQL) must be introduced instead.

FL image compatibility on upgrade 

The hub and the trust-side FL base images (flip-fl-base for NVFLARE, flip-fl-base-flower for Flower) share a wire contract for training metrics and logs. The metrics and logs endpoints now require an fl_client_name field, so the hub and the FL images must be upgraded together:

An old FL base image that omits fl_client_name is rejected (HTTP 422) by the new hub. FL clients historically swallow that failure, so training appears to complete while the metrics chart and logs stay empty.
Deploy the hub and bump the trust-side FL image tag in the same maintenance window; do not run training in the gap.

This pairs with the flip-fl-base follow-up that adds the trust-internal service-key header to the flip client wrappers (see the Trust-internal Service Authentication section in the repo-root CLAUDE.md).

Email setup 

FLIP uses SES for transactional email and Cognito for authentication. Before the first deploy you must:

Verify the sender identity — Terraform creates the SES identity; click the verification link in the email that arrives at the verified address.
Confirm the identity status shows Verified in the SES console.

If SES is still in the sandbox, request production access from the SES console or only send to verified destination addresses for testing.

Status check 

After deployment:

make status

This validates Terraform state and outputs, VPC and subnet configuration, EC2 health, RDS connectivity, Secrets Manager access, S3 buckets, Cognito user pool, Docker container status, public endpoint availability, SSH connectivity, and CloudWatch logging.

SSH access via SSM 

After make ssh-config writes the SSM-based SSH configuration to ~/.ssh/config, the hub and any cloud-hosted trust are reachable directly:

ssh flip          # Central Hub
ssh flip-trust    # cloud trust (if deployed)

Trust web UIs (XNAT, Orthanc, swagger docs, Grafana) are reachable via SSM port forwarding:

make forward-trust

This prints the local URLs to paste into your browser. Press Ctrl+C to close all forwards.

Destroy infrastructure 

make destroy

The destroy target preserves Cognito, Secrets Manager, and the application S3 bucket. In prod, the RDS instance has deletion protection enabled and a final snapshot is taken before deletion is allowed — staging stays disposable.

Troubleshooting 

Run make status first; it auto-diagnoses AWS resource health, network connectivity, endpoint availability, container status, and resource usage.

For known failure modes — Terraform state drift, ECS service errors, CloudFront cache invalidation, RDS connectivity, SSM Session Manager issues — see deploy/providers/AWS/TROUBLESHOOTING.md.