.goosehints

@@ -1,201 +0,0 @@
-# .goosehints for dumpnet-argo
-
-This is a personal single-node Talos/Kubernetes cluster on AWS EC2, managed via
-ArgoCD GitOps, migrating services off a legacy NixOS box (dumpnet.chat /
-git.keane.sh). Below are conventions and hard-won lessons — follow them
-before improvising a new pattern.
-
-## Core architecture
-
-- Single EC2 node (t3.medium), Talos Linux, one EIP, one cluster — this will
-  **never** be multi-node. Don't suggest multi-node solutions (NLB, EFS for
-  RWX, etc.) as the default; hostPath/local-disk patterns are correct here.
-- Terraform (`terraform/`) owns all AWS infra: EC2, EIP, subnet, security
-  group, Talos bootstrap, Route53, S3, IAM, Secrets Manager, SES.
-- ArgoCD App-of-Apps owns all Kubernetes-side resources. Root app is
-  `apps/apps.yaml`, which points at the `apps/` directory itself (self-managing,
-  only needs one manual `kubectl apply` ever, via `make bootstrap`).
-- Groups under `apps/`: `cluster.yaml` (platform infra), `data.yaml` (shared
-  stateful stuff like Postgres), `services.yaml` (user-facing apps), `mcp.yaml`
-  (MCP servers). Manifests live in matching `manifests/<group>/` dirs. Add new
-  groups here rather than growing one flat directory.
-- One global `values.yaml` at repo root holds `clusterName`, `domain`,
-  `repoURL`, `certEmail`, `awsRegion`, `registry.*`. Charts read it via the
-  multi-source `$values/values.yaml` pattern. `repoURL` inside ArgoCD
-  `Application` specs themselves can NOT be templated (Helm doesn't touch
-  those fields) — document this when it matters, don't try to fix it.
-
-## Secrets
-
-- **Exactly one** AWS Secrets Manager secret: `dumpnet` (nested JSON by
-  service, e.g. `cluster`, `postgres`, `kan`, `forgejo`, `ses`,
-  `tailscale`, `mcp_auth_proxy`). Never create a second Secrets Manager
-  secret — Secrets Manager billing is per-secret and that was an explicit
-  decision to avoid.
-- External Secrets Operator (ESO) is the only way secrets get into the
-  cluster. `ClusterSecretStore` auths via the EC2 instance IAM role — no
-  static credentials anywhere.
-- Sensitive local files (`controlplane.yaml`, `worker.yaml`, `talosconfig`,
-  `terraform.tfstate`) are gitignored entirely — never SOPS-encrypted in-repo.
-  Decrypt-to-`/tmp` was rejected in favor of "never committed at all."
-  `terraform.tfvars` is intentionally NOT gitignored (values aren't
-  sensitive) — don't re-add it to `.gitignore`.
-- When adding a new service's secrets: add the key(s) under the service's
-  namespace in the `dumpnet` JSON blob (via `aws secretsmanager
-  get-secret-value` → merge in Python → `put-secret-value`), then add an
-  `ExternalSecret` template in that chart's `templates/` dir referencing
-  `property: <service>.<key>`.
-
-## ArgoCD / Helm patterns (learned the hard way)
-
-- **Use charts as-is.** Don't split an app into multiple ArgoCD Applications
-  just because another unrelated chart happens to be split that way (this
-  was explicitly called out as "cargo culting" — cert-manager's
-  ClusterIssuer split is justified due to CRD ordering; ESO's original split
-  was not and got merged back into one app).
-- Extra resources (ClusterSecretStore, Issuers, Namespaces, ExternalSecrets)
-  belong in that chart's own `templates/` directory, not a second
-  Application, unless there's a *proven* cross-app ordering problem.
-- Any namespace that needs `hostNetwork`, `hostPath`, or other privileged
-  pod behavior (ingress-nginx, fluent-bit, tailscale, mcp-auth-proxy) needs a
-  `Namespace` manifest in that chart's `templates/` with:
-  ```yaml
-  metadata:
-    labels:
-      pod-security.kubernetes.io/enforce: privileged
-      pod-security.kubernetes.io/audit: privileged
-      pod-security.kubernetes.io/warn: privileged
-  ```
-  Do **not** create these namespaces via Terraform `kubernetes_namespace` —
-  that was tried and reverted; it causes import/ordering pain on rebuilds.
-  ArgoCD-managed namespace manifests are the correct pattern.
-- ESO `ExternalSecret` `.target.template` uses its own Go-template syntax
-  that Helm will try to consume if you're not careful. To keep a value
-  literal for ESO to interpolate at runtime while still letting Helm
-  interpolate `.Values.*`, use the `{{ "{{" }}` / backtick escaping pattern
-  already present in `charts/*/templates/*.yaml` — copy that pattern rather
-  than reinventing it.
-- Watch out for fields ESO/operators add as defaults after creation
-  (`mergePolicy`, `engineVersion`, etc.) causing perpetual ArgoCD
-  OutOfSync — either match them explicitly in the manifest or use
-  `ignoreDifferences` in the Application spec.
-
-## Storage
-
-- Single shared hostPath-backed PV (`appdata`, ReadWriteMany, `Retain` reclaim
-  policy) at `/var/local/appdata` on the node. Services claim subpaths via
-  their own PVCs bound to that PV — this is the deliberate "Unraid appdata"
-  equivalent for a single-node cluster. Don't provision per-service EBS
-  volumes or suggest EFS/CSI drivers unless the user explicitly wants to
-  outgrow single-node.
-- PV `capacity` on hostPath volumes is a label, not an enforced quota — don't
-  be alarmed if it's set higher than the actual disk size, but don't blow
-  past real disk space either.
-
-## Networking / TLS
-
-- ingress-nginx runs as a DaemonSet in `hostNetwork` mode (binds 80/443
-  directly on the node) — no LoadBalancer/NLB. This was a deliberate
-  cost-saving choice for single-node.
-- cert-manager + Let's Encrypt (`letsencrypt-prod` ClusterIssuer, HTTP01 via
-  ingress-nginx) for all public hostnames. A separate `selfsigned` Issuer
-  handles internal webhook certs (e.g. ESO) — never try to get Let's Encrypt
-  certs for internal `*.svc.cluster.local` names, it will always fail with
-  "domain needs a public suffix."
-- Tailscale operator is used for private/internal access only (e.g. DBeaver
-  → Postgres), annotated per-Service with `tailscale.com/expose: "true"` and
-  `tailscale.com/hostname: dumpnet-<service>`. `ingressClass` is disabled on
-  the operator so it doesn't try to manage public ingresses too.
-- DNS records are Terraform-managed (`dns_records` list in
-  `terraform.tfvars`) — add new subdomains there, not manually in Route53
-  console.
-
-## Container images
-
-- Prefer official upstream images. When a custom image must be built (e.g.
-  `mcp-auth-proxy`), it lives under `images/<name>/Dockerfile` in this same
-  repo — no separate repo for a thin wrapper Dockerfile.
-- Use multi-stage builds; final stage should be `scratch` or
-  `distroless` for Go binaries — keep images small, self-hosted Forgejo has a
-  registry size limit (`nginx.clientMaxBodySize` + Forgejo's own
-  `packages.MAX_BLOB_SIZE`, both had to be raised from defaults on the
-  NixOS box).
-- Custom images push to the **Forgejo container registry**
-  (`forge.keane.sh/ian/<image>`), not ECR. ECR was set up once and
-  deliberately removed — an EC2/Talos node can't easily pull from ECR without
-  baking a system extension into a custom AMI via Image Factory, which is
-  more infra than it's worth for a personal cluster. Forgejo + the node's
-  default pull path works fine and stays consistent with "no unnecessary
-  centralized services."
-- `registry.host` / `registry.user` are values in the root `values.yaml` —
-  reference them, don't hardcode `forge.keane.sh`/`ian` in new charts.
-
-## Databases
-
-- One shared Postgres in the `data` group/namespace (`postgres`), used by
-  multiple services (kan, future services). Don't spin up a dedicated
-  Postgres per service.
-- New service databases: connect via DBeaver (Tailscale-exposed
-  `dumpnet-postgres:5432`) and `CREATE DATABASE <name>;` manually — there's
-  no automated database-per-service provisioning yet. Password comes from
-  `postgres.password` in the `dumpnet` secret (same user/password works for
-  every DB on the instance, it's one Postgres server).
-- Migration Jobs (e.g. `kan-migrate`) should NOT use ArgoCD `PreSync` hooks —
-  hooks and sync-wave ordering don't compose the way you'd expect ExternalSecret
-  creation vs. hook timing. Use plain `Job` resources with `sync-wave`
-  annotations instead (ExternalSecret at wave `-1`, migration Job at wave
-  `1`), and set `ttlSecondsAfterFinished` so completed jobs clean themselves up.
-
-## Email
-
-- Outbound email (magic links etc.) goes through SES SMTP
-  (`email-smtp.us-east-1.amazonaws.com`), domain `dumpnet.chat` (verified via
-  Terraform-managed DKIM/TXT records). `keane.sh` is reserved for the
-  Fastmail alias — don't touch its DNS or try to send from `@keane.sh`.
-- SES starts in sandbox mode — verify individual recipient addresses via
-  `aws ses verify-email-identity` until production access is granted.
-- Check upstream app source (not assumptions) for the actual expected env
-  var names before wiring up SMTP — e.g. kan uses `SMTP_HOST` /
-  `SMTP_PORT` / `SMTP_USER` / `SMTP_PASSWORD` / `SMTP_SECURE` /
-  `EMAIL_FROM`, not `EMAIL_SERVER_*`. Wrong var names fail silently/weirdly
-  (e.g. connects to `127.0.0.1:465` when nothing is configured) rather than
-  erroring clearly.
-
-## Operational workflow
-
-- `make apply` is a **two-phase** terraform apply (first targets
-  `talos_cluster_kubeconfig.this` and its deps, then a full apply) — the
-  Terraform `helm`/`kubernetes` providers can't initialize before the
-  cluster exists. Don't try to collapse this into one `terraform apply`.
-- After any fresh `apply`: `make post-apply` (waits for node, imports
-  ingress-nginx namespace state if needed, runs `make bootstrap`, builds/pushes
-  any custom images). Then DNS/cert issuance takes a few minutes.
-- `make clean` before `make destroy` when doing a full from-zero rebuild —
-  force-deletes the Secrets Manager secret (avoids the 30-day recovery
-  window blocking recreation) and clears stale kubectl/talosctl contexts.
-  It does NOT touch Route53 (Terraform-managed, fine to destroy/recreate).
-- After any pod-affecting change to an `ExternalSecret`, the **existing**
-  pod does not pick up new secret data automatically — force ESO to
-  re-sync (`kubectl annotate externalsecret <name> -n <ns> force-sync=$(date
-  +%s) --overwrite`) and then delete/restart the pod. `kubectl rollout
-  restart` alone is not reliably enough — prefer `kubectl delete pod` for a
-  clean restart, especially if the deployment spec itself changed and
-  ArgoCD hasn't synced yet (check `kubectl get deployment ... -o
-  jsonpath='{.spec.template.spec.containers[0].args}'` to confirm before
-  assuming a restart will fix anything).
-- When something is "OutOfSync" in ArgoCD for no apparent reason, actually
-  look at the diff (`kubectl describe app <name> -n argocd`) rather than
-  guessing — several past sessions burned significant time on guessed fixes
-  (mergePolicy, engineVersion, type placement) before checking the real
-  diff.
-- Prefer diagnosing root cause over reflexive retries. This repo's history
-  includes multiple incidents where the same command was rerun repeatedly
-  without new information — check logs/state first, form a hypothesis,
-  test it once.
-
-## Documentation
-
-- Keep `README.md` up to date for: first-time setup, secrets approach,
-  day-to-day ops, Makefile reference, forking/multi-environment notes. If a
-  new operational gotcha is discovered (like the two-phase apply, or
-  `make clean`), add it to the README, not just this file.

charts/repertory-api/Chart.yaml

@@ -1,5 +0,0 @@
-apiVersion: v2
-name: repertory-api
-description: Repertory API (Flask backend for repertory)
-type: application
-version: 0.1.0

charts/repertory-api/templates/external-secret.yaml

@@ -1,28 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: repertory-api-secrets
-  namespace: repertory-api
-  annotations:
-    argocd.argoproj.io/sync-wave: "-1"
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    name: aws-secrets-manager
-    kind: ClusterSecretStore
-  target:
-    name: repertory-api-secrets
-    template:
-      data:
-        API_KEY: "{{ `{{ .api_key }}` }}"
-        DATABASE_URL: "postgresql://postgres:{{ `{{ .postgres_password }}` }}@postgres.postgres.svc.cluster.local:5432/repertory"
-        CORS_ORIGINS: "{{ .Values.corsOrigins }}"
-  data:
-    - secretKey: api_key
-      remoteRef:
-        key: dumpnet
-        property: repertory_api.api_key
-    - secretKey: postgres_password
-      remoteRef:
-        key: dumpnet
-        property: postgres.password

charts/repertory-api/templates/namespace.yaml

@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: repertory-api

charts/repertory-api/templates/registry-secret.yaml

@@ -1,27 +0,0 @@
-{{- $host := .Values.registry.host }}
-{{- $user := .Values.registry.user }}
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: forgejo-registry
-  namespace: repertory-api
-  annotations:
-    argocd.argoproj.io/sync-wave: "-1"
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    name: aws-secrets-manager
-    kind: ClusterSecretStore
-  target:
-    name: forgejo-registry
-    template:
-      engineVersion: v2
-      mergePolicy: Replace
-      type: kubernetes.io/dockerconfigjson
-      data:
-        .dockerconfigjson: '{"auths":{"{{ $host }}":{"username":"{{ $user }}","password":"{{ "{{" }} .registry_token {{ "}}" }}","auth":"{{ "{{" }} printf "{{ $user }}:%s" .registry_token | b64enc {{ "}}" }}"}}}'
-  data:
-    - secretKey: registry_token
-      remoteRef:
-        key: dumpnet
-        property: forgejo.registry_token

charts/repertory-api/templates/repertory-api.yaml

@@ -1,96 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: repertory-api
-  namespace: repertory-api
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: repertory-api
-  template:
-    metadata:
-      labels:
-        app: repertory-api
-    spec:
-      imagePullSecrets:
-        - name: forgejo-registry
-      containers:
-        - name: repertory-api
-          image: "{{ .Values.registry.host }}/{{ .Values.registry.user }}/repertory-api:latest"
-          ports:
-            - containerPort: 5000
-          envFrom:
-            - secretRef:
-                name: repertory-api-secrets
-          readinessProbe:
-            httpGet:
-              path: /health
-              port: 5000
-            initialDelaySeconds: 5
-            periodSeconds: 10
-          livenessProbe:
-            httpGet:
-              path: /health
-              port: 5000
-            initialDelaySeconds: 10
-            periodSeconds: 20
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: repertory-api
-  namespace: repertory-api
-spec:
-  selector:
-    app: repertory-api
-  ports:
-    - port: 80
-      targetPort: 5000
----
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: repertory-api
-  namespace: repertory-api
-  annotations:
-    cert-manager.io/cluster-issuer: letsencrypt-prod
-spec:
-  ingressClassName: nginx
-  tls:
-    - hosts:
-        - repertory-api.{{ .Values.domain }}
-      secretName: repertory-api-tls
-  rules:
-    - host: repertory-api.{{ .Values.domain }}
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: repertory-api
-                port:
-                  number: 80
----
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: repertory-api-migrate
-  namespace: repertory-api
-  annotations:
-    argocd.argoproj.io/sync-wave: "1"
-spec:
-  ttlSecondsAfterFinished: 120
-  template:
-    spec:
-      restartPolicy: Never
-      imagePullSecrets:
-        - name: forgejo-registry
-      containers:
-        - name: migrate
-          image: "{{ .Values.registry.host }}/{{ .Values.registry.user }}/repertory-api:latest"
-          command: ["uv", "run", "alembic", "upgrade", "head"]
-          envFrom:
-            - secretRef:
-                name: repertory-api-secrets

charts/repertory-api/values.yaml

@@ -1,5 +0,0 @@
-domain: dumpnet.chat
-registry:
-  host: forge.keane.sh
-  user: ian
-corsOrigins: "*"

manifests/services/repertory-api.yaml

@@ -1,33 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: repertory-api
-  namespace: argocd
-spec:
-  project: default
-  sources:
-    - repoURL: https://forge.keane.sh/ian/dumpnet-argo.git
-      targetRevision: HEAD
-      path: charts/repertory-api
-      helm:
-        valueFiles:
-          - $values/values.yaml
-    - repoURL: https://forge.keane.sh/ian/dumpnet-argo.git
-      targetRevision: HEAD
-      ref: values
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: repertory-api
-  syncPolicy:
-    automated:
-      prune: true
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true
-  ignoreDifferences:
-    - group: external-secrets.io
-      kind: ExternalSecret
-      jsonPointers:
-        - /spec/target/template/mergePolicy
-        - /spec/target/template/engineVersion
-        - /spec/target/type

terraform/terraform.tfvars

@@ -1,2 +1,2 @@
 hosted_zone_id = "Z068835512G0ZQJ9SJGOI"
-dns_records    = ["argocd", "todo", "git-mcp", "git-mcp-oauth", "repertory-api"]
+dns_records    = ["argocd", "todo", "git-mcp", "git-mcp-oauth"]