diff --git a/.goosehints b/.goosehints deleted file mode 100644 index 01cc646..0000000 --- a/.goosehints +++ /dev/null @@ -1,201 +0,0 @@ -# .goosehints for dumpnet-argo - -This is a personal single-node Talos/Kubernetes cluster on AWS EC2, managed via -ArgoCD GitOps, migrating services off a legacy NixOS box (dumpnet.chat / -git.keane.sh). Below are conventions and hard-won lessons — follow them -before improvising a new pattern. - -## Core architecture - -- Single EC2 node (t3.medium), Talos Linux, one EIP, one cluster — this will - **never** be multi-node. Don't suggest multi-node solutions (NLB, EFS for - RWX, etc.) as the default; hostPath/local-disk patterns are correct here. -- Terraform (`terraform/`) owns all AWS infra: EC2, EIP, subnet, security - group, Talos bootstrap, Route53, S3, IAM, Secrets Manager, SES. -- ArgoCD App-of-Apps owns all Kubernetes-side resources. Root app is - `apps/apps.yaml`, which points at the `apps/` directory itself (self-managing, - only needs one manual `kubectl apply` ever, via `make bootstrap`). -- Groups under `apps/`: `cluster.yaml` (platform infra), `data.yaml` (shared - stateful stuff like Postgres), `services.yaml` (user-facing apps), `mcp.yaml` - (MCP servers). Manifests live in matching `manifests//` dirs. Add new - groups here rather than growing one flat directory. -- One global `values.yaml` at repo root holds `clusterName`, `domain`, - `repoURL`, `certEmail`, `awsRegion`, `registry.*`. Charts read it via the - multi-source `$values/values.yaml` pattern. `repoURL` inside ArgoCD - `Application` specs themselves can NOT be templated (Helm doesn't touch - those fields) — document this when it matters, don't try to fix it. - -## Secrets - -- **Exactly one** AWS Secrets Manager secret: `dumpnet` (nested JSON by - service, e.g. `cluster`, `postgres`, `kan`, `forgejo`, `ses`, - `tailscale`, `mcp_auth_proxy`). Never create a second Secrets Manager - secret — Secrets Manager billing is per-secret and that was an explicit - decision to avoid. -- External Secrets Operator (ESO) is the only way secrets get into the - cluster. `ClusterSecretStore` auths via the EC2 instance IAM role — no - static credentials anywhere. -- Sensitive local files (`controlplane.yaml`, `worker.yaml`, `talosconfig`, - `terraform.tfstate`) are gitignored entirely — never SOPS-encrypted in-repo. - Decrypt-to-`/tmp` was rejected in favor of "never committed at all." - `terraform.tfvars` is intentionally NOT gitignored (values aren't - sensitive) — don't re-add it to `.gitignore`. -- When adding a new service's secrets: add the key(s) under the service's - namespace in the `dumpnet` JSON blob (via `aws secretsmanager - get-secret-value` → merge in Python → `put-secret-value`), then add an - `ExternalSecret` template in that chart's `templates/` dir referencing - `property: .`. - -## ArgoCD / Helm patterns (learned the hard way) - -- **Use charts as-is.** Don't split an app into multiple ArgoCD Applications - just because another unrelated chart happens to be split that way (this - was explicitly called out as "cargo culting" — cert-manager's - ClusterIssuer split is justified due to CRD ordering; ESO's original split - was not and got merged back into one app). -- Extra resources (ClusterSecretStore, Issuers, Namespaces, ExternalSecrets) - belong in that chart's own `templates/` directory, not a second - Application, unless there's a *proven* cross-app ordering problem. -- Any namespace that needs `hostNetwork`, `hostPath`, or other privileged - pod behavior (ingress-nginx, fluent-bit, tailscale, mcp-auth-proxy) needs a - `Namespace` manifest in that chart's `templates/` with: - ```yaml - metadata: - labels: - pod-security.kubernetes.io/enforce: privileged - pod-security.kubernetes.io/audit: privileged - pod-security.kubernetes.io/warn: privileged - ``` - Do **not** create these namespaces via Terraform `kubernetes_namespace` — - that was tried and reverted; it causes import/ordering pain on rebuilds. - ArgoCD-managed namespace manifests are the correct pattern. -- ESO `ExternalSecret` `.target.template` uses its own Go-template syntax - that Helm will try to consume if you're not careful. To keep a value - literal for ESO to interpolate at runtime while still letting Helm - interpolate `.Values.*`, use the `{{ "{{" }}` / backtick escaping pattern - already present in `charts/*/templates/*.yaml` — copy that pattern rather - than reinventing it. -- Watch out for fields ESO/operators add as defaults after creation - (`mergePolicy`, `engineVersion`, etc.) causing perpetual ArgoCD - OutOfSync — either match them explicitly in the manifest or use - `ignoreDifferences` in the Application spec. - -## Storage - -- Single shared hostPath-backed PV (`appdata`, ReadWriteMany, `Retain` reclaim - policy) at `/var/local/appdata` on the node. Services claim subpaths via - their own PVCs bound to that PV — this is the deliberate "Unraid appdata" - equivalent for a single-node cluster. Don't provision per-service EBS - volumes or suggest EFS/CSI drivers unless the user explicitly wants to - outgrow single-node. -- PV `capacity` on hostPath volumes is a label, not an enforced quota — don't - be alarmed if it's set higher than the actual disk size, but don't blow - past real disk space either. - -## Networking / TLS - -- ingress-nginx runs as a DaemonSet in `hostNetwork` mode (binds 80/443 - directly on the node) — no LoadBalancer/NLB. This was a deliberate - cost-saving choice for single-node. -- cert-manager + Let's Encrypt (`letsencrypt-prod` ClusterIssuer, HTTP01 via - ingress-nginx) for all public hostnames. A separate `selfsigned` Issuer - handles internal webhook certs (e.g. ESO) — never try to get Let's Encrypt - certs for internal `*.svc.cluster.local` names, it will always fail with - "domain needs a public suffix." -- Tailscale operator is used for private/internal access only (e.g. DBeaver - → Postgres), annotated per-Service with `tailscale.com/expose: "true"` and - `tailscale.com/hostname: dumpnet-`. `ingressClass` is disabled on - the operator so it doesn't try to manage public ingresses too. -- DNS records are Terraform-managed (`dns_records` list in - `terraform.tfvars`) — add new subdomains there, not manually in Route53 - console. - -## Container images - -- Prefer official upstream images. When a custom image must be built (e.g. - `mcp-auth-proxy`), it lives under `images//Dockerfile` in this same - repo — no separate repo for a thin wrapper Dockerfile. -- Use multi-stage builds; final stage should be `scratch` or - `distroless` for Go binaries — keep images small, self-hosted Forgejo has a - registry size limit (`nginx.clientMaxBodySize` + Forgejo's own - `packages.MAX_BLOB_SIZE`, both had to be raised from defaults on the - NixOS box). -- Custom images push to the **Forgejo container registry** - (`forge.keane.sh/ian/`), not ECR. ECR was set up once and - deliberately removed — an EC2/Talos node can't easily pull from ECR without - baking a system extension into a custom AMI via Image Factory, which is - more infra than it's worth for a personal cluster. Forgejo + the node's - default pull path works fine and stays consistent with "no unnecessary - centralized services." -- `registry.host` / `registry.user` are values in the root `values.yaml` — - reference them, don't hardcode `forge.keane.sh`/`ian` in new charts. - -## Databases - -- One shared Postgres in the `data` group/namespace (`postgres`), used by - multiple services (kan, future services). Don't spin up a dedicated - Postgres per service. -- New service databases: connect via DBeaver (Tailscale-exposed - `dumpnet-postgres:5432`) and `CREATE DATABASE ;` manually — there's - no automated database-per-service provisioning yet. Password comes from - `postgres.password` in the `dumpnet` secret (same user/password works for - every DB on the instance, it's one Postgres server). -- Migration Jobs (e.g. `kan-migrate`) should NOT use ArgoCD `PreSync` hooks — - hooks and sync-wave ordering don't compose the way you'd expect ExternalSecret - creation vs. hook timing. Use plain `Job` resources with `sync-wave` - annotations instead (ExternalSecret at wave `-1`, migration Job at wave - `1`), and set `ttlSecondsAfterFinished` so completed jobs clean themselves up. - -## Email - -- Outbound email (magic links etc.) goes through SES SMTP - (`email-smtp.us-east-1.amazonaws.com`), domain `dumpnet.chat` (verified via - Terraform-managed DKIM/TXT records). `keane.sh` is reserved for the - Fastmail alias — don't touch its DNS or try to send from `@keane.sh`. -- SES starts in sandbox mode — verify individual recipient addresses via - `aws ses verify-email-identity` until production access is granted. -- Check upstream app source (not assumptions) for the actual expected env - var names before wiring up SMTP — e.g. kan uses `SMTP_HOST` / - `SMTP_PORT` / `SMTP_USER` / `SMTP_PASSWORD` / `SMTP_SECURE` / - `EMAIL_FROM`, not `EMAIL_SERVER_*`. Wrong var names fail silently/weirdly - (e.g. connects to `127.0.0.1:465` when nothing is configured) rather than - erroring clearly. - -## Operational workflow - -- `make apply` is a **two-phase** terraform apply (first targets - `talos_cluster_kubeconfig.this` and its deps, then a full apply) — the - Terraform `helm`/`kubernetes` providers can't initialize before the - cluster exists. Don't try to collapse this into one `terraform apply`. -- After any fresh `apply`: `make post-apply` (waits for node, imports - ingress-nginx namespace state if needed, runs `make bootstrap`, builds/pushes - any custom images). Then DNS/cert issuance takes a few minutes. -- `make clean` before `make destroy` when doing a full from-zero rebuild — - force-deletes the Secrets Manager secret (avoids the 30-day recovery - window blocking recreation) and clears stale kubectl/talosctl contexts. - It does NOT touch Route53 (Terraform-managed, fine to destroy/recreate). -- After any pod-affecting change to an `ExternalSecret`, the **existing** - pod does not pick up new secret data automatically — force ESO to - re-sync (`kubectl annotate externalsecret -n force-sync=$(date - +%s) --overwrite`) and then delete/restart the pod. `kubectl rollout - restart` alone is not reliably enough — prefer `kubectl delete pod` for a - clean restart, especially if the deployment spec itself changed and - ArgoCD hasn't synced yet (check `kubectl get deployment ... -o - jsonpath='{.spec.template.spec.containers[0].args}'` to confirm before - assuming a restart will fix anything). -- When something is "OutOfSync" in ArgoCD for no apparent reason, actually - look at the diff (`kubectl describe app -n argocd`) rather than - guessing — several past sessions burned significant time on guessed fixes - (mergePolicy, engineVersion, type placement) before checking the real - diff. -- Prefer diagnosing root cause over reflexive retries. This repo's history - includes multiple incidents where the same command was rerun repeatedly - without new information — check logs/state first, form a hypothesis, - test it once. - -## Documentation - -- Keep `README.md` up to date for: first-time setup, secrets approach, - day-to-day ops, Makefile reference, forking/multi-environment notes. If a - new operational gotcha is discovered (like the two-phase apply, or - `make clean`), add it to the README, not just this file. diff --git a/charts/repertory-api/Chart.yaml b/charts/repertory-api/Chart.yaml deleted file mode 100644 index 04d7f88..0000000 --- a/charts/repertory-api/Chart.yaml +++ /dev/null @@ -1,5 +0,0 @@ -apiVersion: v2 -name: repertory-api -description: Repertory API (Flask backend for repertory) -type: application -version: 0.1.0 diff --git a/charts/repertory-api/templates/external-secret.yaml b/charts/repertory-api/templates/external-secret.yaml deleted file mode 100644 index 1c946b8..0000000 --- a/charts/repertory-api/templates/external-secret.yaml +++ /dev/null @@ -1,28 +0,0 @@ -apiVersion: external-secrets.io/v1 -kind: ExternalSecret -metadata: - name: repertory-api-secrets - namespace: repertory-api - annotations: - argocd.argoproj.io/sync-wave: "-1" -spec: - refreshInterval: 1h - secretStoreRef: - name: aws-secrets-manager - kind: ClusterSecretStore - target: - name: repertory-api-secrets - template: - data: - API_KEY: "{{ `{{ .api_key }}` }}" - DATABASE_URL: "postgresql://postgres:{{ `{{ .postgres_password }}` }}@postgres.postgres.svc.cluster.local:5432/repertory" - CORS_ORIGINS: "{{ .Values.corsOrigins }}" - data: - - secretKey: api_key - remoteRef: - key: dumpnet - property: repertory_api.api_key - - secretKey: postgres_password - remoteRef: - key: dumpnet - property: postgres.password diff --git a/charts/repertory-api/templates/namespace.yaml b/charts/repertory-api/templates/namespace.yaml deleted file mode 100644 index 8e49d9d..0000000 --- a/charts/repertory-api/templates/namespace.yaml +++ /dev/null @@ -1,4 +0,0 @@ -apiVersion: v1 -kind: Namespace -metadata: - name: repertory-api diff --git a/charts/repertory-api/templates/registry-secret.yaml b/charts/repertory-api/templates/registry-secret.yaml deleted file mode 100644 index dbe8852..0000000 --- a/charts/repertory-api/templates/registry-secret.yaml +++ /dev/null @@ -1,27 +0,0 @@ -{{- $host := .Values.registry.host }} -{{- $user := .Values.registry.user }} -apiVersion: external-secrets.io/v1 -kind: ExternalSecret -metadata: - name: forgejo-registry - namespace: repertory-api - annotations: - argocd.argoproj.io/sync-wave: "-1" -spec: - refreshInterval: 1h - secretStoreRef: - name: aws-secrets-manager - kind: ClusterSecretStore - target: - name: forgejo-registry - template: - engineVersion: v2 - mergePolicy: Replace - type: kubernetes.io/dockerconfigjson - data: - .dockerconfigjson: '{"auths":{"{{ $host }}":{"username":"{{ $user }}","password":"{{ "{{" }} .registry_token {{ "}}" }}","auth":"{{ "{{" }} printf "{{ $user }}:%s" .registry_token | b64enc {{ "}}" }}"}}}' - data: - - secretKey: registry_token - remoteRef: - key: dumpnet - property: forgejo.registry_token diff --git a/charts/repertory-api/templates/repertory-api.yaml b/charts/repertory-api/templates/repertory-api.yaml deleted file mode 100644 index fec3b4e..0000000 --- a/charts/repertory-api/templates/repertory-api.yaml +++ /dev/null @@ -1,96 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: repertory-api - namespace: repertory-api -spec: - replicas: 1 - selector: - matchLabels: - app: repertory-api - template: - metadata: - labels: - app: repertory-api - spec: - imagePullSecrets: - - name: forgejo-registry - containers: - - name: repertory-api - image: "{{ .Values.registry.host }}/{{ .Values.registry.user }}/repertory-api:latest" - ports: - - containerPort: 5000 - envFrom: - - secretRef: - name: repertory-api-secrets - readinessProbe: - httpGet: - path: /health - port: 5000 - initialDelaySeconds: 5 - periodSeconds: 10 - livenessProbe: - httpGet: - path: /health - port: 5000 - initialDelaySeconds: 10 - periodSeconds: 20 ---- -apiVersion: v1 -kind: Service -metadata: - name: repertory-api - namespace: repertory-api -spec: - selector: - app: repertory-api - ports: - - port: 80 - targetPort: 5000 ---- -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: repertory-api - namespace: repertory-api - annotations: - cert-manager.io/cluster-issuer: letsencrypt-prod -spec: - ingressClassName: nginx - tls: - - hosts: - - repertory-api.{{ .Values.domain }} - secretName: repertory-api-tls - rules: - - host: repertory-api.{{ .Values.domain }} - http: - paths: - - path: / - pathType: Prefix - backend: - service: - name: repertory-api - port: - number: 80 ---- -apiVersion: batch/v1 -kind: Job -metadata: - name: repertory-api-migrate - namespace: repertory-api - annotations: - argocd.argoproj.io/sync-wave: "1" -spec: - ttlSecondsAfterFinished: 120 - template: - spec: - restartPolicy: Never - imagePullSecrets: - - name: forgejo-registry - containers: - - name: migrate - image: "{{ .Values.registry.host }}/{{ .Values.registry.user }}/repertory-api:latest" - command: ["uv", "run", "alembic", "upgrade", "head"] - envFrom: - - secretRef: - name: repertory-api-secrets diff --git a/charts/repertory-api/values.yaml b/charts/repertory-api/values.yaml deleted file mode 100644 index 0a95ca1..0000000 --- a/charts/repertory-api/values.yaml +++ /dev/null @@ -1,5 +0,0 @@ -domain: dumpnet.chat -registry: - host: forge.keane.sh - user: ian -corsOrigins: "*" diff --git a/manifests/services/repertory-api.yaml b/manifests/services/repertory-api.yaml deleted file mode 100644 index 3b84a86..0000000 --- a/manifests/services/repertory-api.yaml +++ /dev/null @@ -1,33 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: Application -metadata: - name: repertory-api - namespace: argocd -spec: - project: default - sources: - - repoURL: https://forge.keane.sh/ian/dumpnet-argo.git - targetRevision: HEAD - path: charts/repertory-api - helm: - valueFiles: - - $values/values.yaml - - repoURL: https://forge.keane.sh/ian/dumpnet-argo.git - targetRevision: HEAD - ref: values - destination: - server: https://kubernetes.default.svc - namespace: repertory-api - syncPolicy: - automated: - prune: true - selfHeal: true - syncOptions: - - CreateNamespace=true - ignoreDifferences: - - group: external-secrets.io - kind: ExternalSecret - jsonPointers: - - /spec/target/template/mergePolicy - - /spec/target/template/engineVersion - - /spec/target/type diff --git a/terraform/terraform.tfvars b/terraform/terraform.tfvars index 439106f..a700577 100644 --- a/terraform/terraform.tfvars +++ b/terraform/terraform.tfvars @@ -1,2 +1,2 @@ hosted_zone_id = "Z068835512G0ZQJ9SJGOI" -dns_records = ["argocd", "todo", "git-mcp", "git-mcp-oauth", "repertory-api"] +dns_records = ["argocd", "todo", "git-mcp", "git-mcp-oauth"]