Add goosehints

.goosehints

@@ -0,0 +1,201 @@
+# .goosehints for dumpnet-argo
+
+This is a personal single-node Talos/Kubernetes cluster on AWS EC2, managed via
+ArgoCD GitOps, migrating services off a legacy NixOS box (dumpnet.chat /
+git.keane.sh). Below are conventions and hard-won lessons — follow them
+before improvising a new pattern.
+
+## Core architecture
+
+- Single EC2 node (t3.medium), Talos Linux, one EIP, one cluster — this will
+  **never** be multi-node. Don't suggest multi-node solutions (NLB, EFS for
+  RWX, etc.) as the default; hostPath/local-disk patterns are correct here.
+- Terraform (`terraform/`) owns all AWS infra: EC2, EIP, subnet, security
+  group, Talos bootstrap, Route53, S3, IAM, Secrets Manager, SES.
+- ArgoCD App-of-Apps owns all Kubernetes-side resources. Root app is
+  `apps/apps.yaml`, which points at the `apps/` directory itself (self-managing,
+  only needs one manual `kubectl apply` ever, via `make bootstrap`).
+- Groups under `apps/`: `cluster.yaml` (platform infra), `data.yaml` (shared
+  stateful stuff like Postgres), `services.yaml` (user-facing apps), `mcp.yaml`
+  (MCP servers). Manifests live in matching `manifests/<group>/` dirs. Add new
+  groups here rather than growing one flat directory.
+- One global `values.yaml` at repo root holds `clusterName`, `domain`,
+  `repoURL`, `certEmail`, `awsRegion`, `registry.*`. Charts read it via the
+  multi-source `$values/values.yaml` pattern. `repoURL` inside ArgoCD
+  `Application` specs themselves can NOT be templated (Helm doesn't touch
+  those fields) — document this when it matters, don't try to fix it.
+
+## Secrets
+
+- **Exactly one** AWS Secrets Manager secret: `dumpnet` (nested JSON by
+  service, e.g. `cluster`, `postgres`, `kan`, `forgejo`, `ses`,
+  `tailscale`, `mcp_auth_proxy`). Never create a second Secrets Manager
+  secret — Secrets Manager billing is per-secret and that was an explicit
+  decision to avoid.
+- External Secrets Operator (ESO) is the only way secrets get into the
+  cluster. `ClusterSecretStore` auths via the EC2 instance IAM role — no
+  static credentials anywhere.
+- Sensitive local files (`controlplane.yaml`, `worker.yaml`, `talosconfig`,
+  `terraform.tfstate`) are gitignored entirely — never SOPS-encrypted in-repo.
+  Decrypt-to-`/tmp` was rejected in favor of "never committed at all."
+  `terraform.tfvars` is intentionally NOT gitignored (values aren't
+  sensitive) — don't re-add it to `.gitignore`.
+- When adding a new service's secrets: add the key(s) under the service's
+  namespace in the `dumpnet` JSON blob (via `aws secretsmanager
+  get-secret-value` → merge in Python → `put-secret-value`), then add an
+  `ExternalSecret` template in that chart's `templates/` dir referencing
+  `property: <service>.<key>`.
+
+## ArgoCD / Helm patterns (learned the hard way)
+
+- **Use charts as-is.** Don't split an app into multiple ArgoCD Applications
+  just because another unrelated chart happens to be split that way (this
+  was explicitly called out as "cargo culting" — cert-manager's
+  ClusterIssuer split is justified due to CRD ordering; ESO's original split
+  was not and got merged back into one app).
+- Extra resources (ClusterSecretStore, Issuers, Namespaces, ExternalSecrets)
+  belong in that chart's own `templates/` directory, not a second
+  Application, unless there's a *proven* cross-app ordering problem.
+- Any namespace that needs `hostNetwork`, `hostPath`, or other privileged
+  pod behavior (ingress-nginx, fluent-bit, tailscale, mcp-auth-proxy) needs a
+  `Namespace` manifest in that chart's `templates/` with:
+  ```yaml
+  metadata:
+    labels:
+      pod-security.kubernetes.io/enforce: privileged
+      pod-security.kubernetes.io/audit: privileged
+      pod-security.kubernetes.io/warn: privileged
+  ```
+  Do **not** create these namespaces via Terraform `kubernetes_namespace` —
+  that was tried and reverted; it causes import/ordering pain on rebuilds.
+  ArgoCD-managed namespace manifests are the correct pattern.
+- ESO `ExternalSecret` `.target.template` uses its own Go-template syntax
+  that Helm will try to consume if you're not careful. To keep a value
+  literal for ESO to interpolate at runtime while still letting Helm
+  interpolate `.Values.*`, use the `{{ "{{" }}` / backtick escaping pattern
+  already present in `charts/*/templates/*.yaml` — copy that pattern rather
+  than reinventing it.
+- Watch out for fields ESO/operators add as defaults after creation
+  (`mergePolicy`, `engineVersion`, etc.) causing perpetual ArgoCD
+  OutOfSync — either match them explicitly in the manifest or use
+  `ignoreDifferences` in the Application spec.
+
+## Storage
+
+- Single shared hostPath-backed PV (`appdata`, ReadWriteMany, `Retain` reclaim
+  policy) at `/var/local/appdata` on the node. Services claim subpaths via
+  their own PVCs bound to that PV — this is the deliberate "Unraid appdata"
+  equivalent for a single-node cluster. Don't provision per-service EBS
+  volumes or suggest EFS/CSI drivers unless the user explicitly wants to
+  outgrow single-node.
+- PV `capacity` on hostPath volumes is a label, not an enforced quota — don't
+  be alarmed if it's set higher than the actual disk size, but don't blow
+  past real disk space either.
+
+## Networking / TLS
+
+- ingress-nginx runs as a DaemonSet in `hostNetwork` mode (binds 80/443
+  directly on the node) — no LoadBalancer/NLB. This was a deliberate
+  cost-saving choice for single-node.
+- cert-manager + Let's Encrypt (`letsencrypt-prod` ClusterIssuer, HTTP01 via
+  ingress-nginx) for all public hostnames. A separate `selfsigned` Issuer
+  handles internal webhook certs (e.g. ESO) — never try to get Let's Encrypt
+  certs for internal `*.svc.cluster.local` names, it will always fail with
+  "domain needs a public suffix."
+- Tailscale operator is used for private/internal access only (e.g. DBeaver
+  → Postgres), annotated per-Service with `tailscale.com/expose: "true"` and
+  `tailscale.com/hostname: dumpnet-<service>`. `ingressClass` is disabled on
+  the operator so it doesn't try to manage public ingresses too.
+- DNS records are Terraform-managed (`dns_records` list in
+  `terraform.tfvars`) — add new subdomains there, not manually in Route53
+  console.
+
+## Container images
+
+- Prefer official upstream images. When a custom image must be built (e.g.
+  `mcp-auth-proxy`), it lives under `images/<name>/Dockerfile` in this same
+  repo — no separate repo for a thin wrapper Dockerfile.
+- Use multi-stage builds; final stage should be `scratch` or
+  `distroless` for Go binaries — keep images small, self-hosted Forgejo has a
+  registry size limit (`nginx.clientMaxBodySize` + Forgejo's own
+  `packages.MAX_BLOB_SIZE`, both had to be raised from defaults on the
+  NixOS box).
+- Custom images push to the **Forgejo container registry**
+  (`forge.keane.sh/ian/<image>`), not ECR. ECR was set up once and
+  deliberately removed — an EC2/Talos node can't easily pull from ECR without
+  baking a system extension into a custom AMI via Image Factory, which is
+  more infra than it's worth for a personal cluster. Forgejo + the node's
+  default pull path works fine and stays consistent with "no unnecessary
+  centralized services."
+- `registry.host` / `registry.user` are values in the root `values.yaml` —
+  reference them, don't hardcode `forge.keane.sh`/`ian` in new charts.
+
+## Databases
+
+- One shared Postgres in the `data` group/namespace (`postgres`), used by
+  multiple services (kan, future services). Don't spin up a dedicated
+  Postgres per service.
+- New service databases: connect via DBeaver (Tailscale-exposed
+  `dumpnet-postgres:5432`) and `CREATE DATABASE <name>;` manually — there's
+  no automated database-per-service provisioning yet. Password comes from
+  `postgres.password` in the `dumpnet` secret (same user/password works for
+  every DB on the instance, it's one Postgres server).
+- Migration Jobs (e.g. `kan-migrate`) should NOT use ArgoCD `PreSync` hooks —
+  hooks and sync-wave ordering don't compose the way you'd expect ExternalSecret
+  creation vs. hook timing. Use plain `Job` resources with `sync-wave`
+  annotations instead (ExternalSecret at wave `-1`, migration Job at wave
+  `1`), and set `ttlSecondsAfterFinished` so completed jobs clean themselves up.
+
+## Email
+
+- Outbound email (magic links etc.) goes through SES SMTP
+  (`email-smtp.us-east-1.amazonaws.com`), domain `dumpnet.chat` (verified via
+  Terraform-managed DKIM/TXT records). `keane.sh` is reserved for the
+  Fastmail alias — don't touch its DNS or try to send from `@keane.sh`.
+- SES starts in sandbox mode — verify individual recipient addresses via
+  `aws ses verify-email-identity` until production access is granted.
+- Check upstream app source (not assumptions) for the actual expected env
+  var names before wiring up SMTP — e.g. kan uses `SMTP_HOST` /
+  `SMTP_PORT` / `SMTP_USER` / `SMTP_PASSWORD` / `SMTP_SECURE` /
+  `EMAIL_FROM`, not `EMAIL_SERVER_*`. Wrong var names fail silently/weirdly
+  (e.g. connects to `127.0.0.1:465` when nothing is configured) rather than
+  erroring clearly.
+
+## Operational workflow
+
+- `make apply` is a **two-phase** terraform apply (first targets
+  `talos_cluster_kubeconfig.this` and its deps, then a full apply) — the
+  Terraform `helm`/`kubernetes` providers can't initialize before the
+  cluster exists. Don't try to collapse this into one `terraform apply`.
+- After any fresh `apply`: `make post-apply` (waits for node, imports
+  ingress-nginx namespace state if needed, runs `make bootstrap`, builds/pushes
+  any custom images). Then DNS/cert issuance takes a few minutes.
+- `make clean` before `make destroy` when doing a full from-zero rebuild —
+  force-deletes the Secrets Manager secret (avoids the 30-day recovery
+  window blocking recreation) and clears stale kubectl/talosctl contexts.
+  It does NOT touch Route53 (Terraform-managed, fine to destroy/recreate).
+- After any pod-affecting change to an `ExternalSecret`, the **existing**
+  pod does not pick up new secret data automatically — force ESO to
+  re-sync (`kubectl annotate externalsecret <name> -n <ns> force-sync=$(date
+  +%s) --overwrite`) and then delete/restart the pod. `kubectl rollout
+  restart` alone is not reliably enough — prefer `kubectl delete pod` for a
+  clean restart, especially if the deployment spec itself changed and
+  ArgoCD hasn't synced yet (check `kubectl get deployment ... -o
+  jsonpath='{.spec.template.spec.containers[0].args}'` to confirm before
+  assuming a restart will fix anything).
+- When something is "OutOfSync" in ArgoCD for no apparent reason, actually
+  look at the diff (`kubectl describe app <name> -n argocd`) rather than
+  guessing — several past sessions burned significant time on guessed fixes
+  (mergePolicy, engineVersion, type placement) before checking the real
+  diff.
+- Prefer diagnosing root cause over reflexive retries. This repo's history
+  includes multiple incidents where the same command was rerun repeatedly
+  without new information — check logs/state first, form a hypothesis,
+  test it once.
+
+## Documentation
+
+- Keep `README.md` up to date for: first-time setup, secrets approach,
+  day-to-day ops, Makefile reference, forking/multi-environment notes. If a
+  new operational gotcha is discovered (like the two-phase apply, or
+  `make clean`), add it to the README, not just this file.

charts/repertory-api/Chart.yaml

@@ -0,0 +1,5 @@
+apiVersion: v2
+name: repertory-api
+description: Repertory API (Flask backend for repertory)
+type: application
+version: 0.1.0

charts/repertory-api/templates/external-secret.yaml

@@ -0,0 +1,28 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: repertory-api-secrets
+  namespace: repertory-api
+  annotations:
+    argocd.argoproj.io/sync-wave: "-1"
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: aws-secrets-manager
+    kind: ClusterSecretStore
+  target:
+    name: repertory-api-secrets
+    template:
+      data:
+        API_KEY: "{{ `{{ .api_key }}` }}"
+        DATABASE_URL: "postgresql://postgres:{{ `{{ .postgres_password }}` }}@postgres.postgres.svc.cluster.local:5432/repertory"
+        CORS_ORIGINS: "{{ .Values.corsOrigins }}"
+  data:
+    - secretKey: api_key
+      remoteRef:
+        key: dumpnet
+        property: repertory_api.api_key
+    - secretKey: postgres_password
+      remoteRef:
+        key: dumpnet
+        property: postgres.password

charts/repertory-api/templates/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: repertory-api

charts/repertory-api/templates/registry-secret.yaml

@@ -0,0 +1,27 @@
+{{- $host := .Values.registry.host }}
+{{- $user := .Values.registry.user }}
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: forgejo-registry
+  namespace: repertory-api
+  annotations:
+    argocd.argoproj.io/sync-wave: "-1"
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: aws-secrets-manager
+    kind: ClusterSecretStore
+  target:
+    name: forgejo-registry
+    template:
+      engineVersion: v2
+      mergePolicy: Replace
+      type: kubernetes.io/dockerconfigjson
+      data:
+        .dockerconfigjson: '{"auths":{"{{ $host }}":{"username":"{{ $user }}","password":"{{ "{{" }} .registry_token {{ "}}" }}","auth":"{{ "{{" }} printf "{{ $user }}:%s" .registry_token | b64enc {{ "}}" }}"}}}'
+  data:
+    - secretKey: registry_token
+      remoteRef:
+        key: dumpnet
+        property: forgejo.registry_token

charts/repertory-api/templates/repertory-api.yaml

@@ -0,0 +1,96 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: repertory-api
+  namespace: repertory-api
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: repertory-api
+  template:
+    metadata:
+      labels:
+        app: repertory-api
+    spec:
+      imagePullSecrets:
+        - name: forgejo-registry
+      containers:
+        - name: repertory-api
+          image: "{{ .Values.registry.host }}/{{ .Values.registry.user }}/repertory-api:latest"
+          ports:
+            - containerPort: 5000
+          envFrom:
+            - secretRef:
+                name: repertory-api-secrets
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: 5000
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: 5000
+            initialDelaySeconds: 10
+            periodSeconds: 20
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: repertory-api
+  namespace: repertory-api
+spec:
+  selector:
+    app: repertory-api
+  ports:
+    - port: 80
+      targetPort: 5000
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: repertory-api
+  namespace: repertory-api
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+spec:
+  ingressClassName: nginx
+  tls:
+    - hosts:
+        - repertory-api.{{ .Values.domain }}
+      secretName: repertory-api-tls
+  rules:
+    - host: repertory-api.{{ .Values.domain }}
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: repertory-api
+                port:
+                  number: 80
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: repertory-api-migrate
+  namespace: repertory-api
+  annotations:
+    argocd.argoproj.io/sync-wave: "1"
+spec:
+  ttlSecondsAfterFinished: 120
+  template:
+    spec:
+      restartPolicy: Never
+      imagePullSecrets:
+        - name: forgejo-registry
+      containers:
+        - name: migrate
+          image: "{{ .Values.registry.host }}/{{ .Values.registry.user }}/repertory-api:latest"
+          command: ["uv", "run", "alembic", "upgrade", "head"]
+          envFrom:
+            - secretRef:
+                name: repertory-api-secrets

charts/repertory-api/values.yaml

@@ -0,0 +1,5 @@
+domain: dumpnet.chat
+registry:
+  host: forge.keane.sh
+  user: ian
+corsOrigins: "*"

manifests/services/repertory-api.yaml

@@ -0,0 +1,33 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: repertory-api
+  namespace: argocd
+spec:
+  project: default
+  sources:
+    - repoURL: https://forge.keane.sh/ian/dumpnet-argo.git
+      targetRevision: HEAD
+      path: charts/repertory-api
+      helm:
+        valueFiles:
+          - $values/values.yaml
+    - repoURL: https://forge.keane.sh/ian/dumpnet-argo.git
+      targetRevision: HEAD
+      ref: values
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: repertory-api
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+  ignoreDifferences:
+    - group: external-secrets.io
+      kind: ExternalSecret
+      jsonPointers:
+        - /spec/target/template/mergePolicy
+        - /spec/target/template/engineVersion
+        - /spec/target/type

terraform/terraform.tfvars

@@ -1,2 +1,2 @@
 hosted_zone_id = "Z068835512G0ZQJ9SJGOI"
-dns_records    = ["argocd", "todo", "git-mcp", "git-mcp-oauth"]
+dns_records    = ["argocd", "todo", "git-mcp", "git-mcp-oauth", "repertory-api"]