Fix mistakes identified via LLM analysis #11

Merged

steffo merged 4 commits from fix/llm-analysis-experiment into main 2026-08-12 21:41:02 +00:00

Conversation 0 Commits 4 Files changed 8 +8 -8

steffo commented 2026-08-12 16:42:08 +00:00

Owner

Copy link

This was my first attempt at using a LLM as an analysis tool to identify mistakes, then fixing them myself manually.

Thoughts

It did give some results, as you can see from the commits in this PR.

The whole pass expended 1.58€ of the free credit included as part of my JetBrains All Products pack.

I feel would be very much not worth it if it wasn't included for free or if I had to spend more to get an IDE with it included.

This is reinforcing my convinction that the only things LLMs are good at are highlighting design issues in existing tooling.

LLM info

Prompt

The prompt was:

What would you improve in this k3s infrastructure managed via Flux CD?

External calls

No requests to external calls or websites were made.

Full response

Claude Agent output

I'll explore the repository structure first to understand what's here.

Let me read through the manifests.

Now the workloads themselves.

Let me verify a few specific things I noticed.

The setter-marker mismatch is confirmed. A couple more quick checks:

I've read through the whole repo. Overall this is a well-organized Flux setup — the ArtifactGenerator monorepo split is a clever use of Flux 2.8, the naming convention is consistent, and the inline comments explaining why (rate limits, OIDC linking, dead services) are better than most infra repos. Here's what I'd change, roughly in priority order.

Actual bugs

1. Three image-automation setters point at ImagePolicies that don't exist. The marker name must match the ImagePolicy's metadata.name, and these don't:

Marker	File	Actual policy name
starshard-synapse:livekit	applications/starshard-synapse/synapse-livekit-jwt.Deployment.yaml:22	synapse-livekit-jwt
steffo-porobot:discord	applications/steffo-porobot/porobot-discord.Deployment.yaml:21	porobot-discord
steffo-porobot:telegram	applications/steffo-porobot/porobot-telegram.Deployment.yaml:21	porobot-telegram

Those three images are silently frozen. The lk-jwt-service one is the worst: it's pinned to :latest, so it's both un-updated by Flux and un-reproducible — :latest defaults to imagePullPolicy: Always, so the running version changes on every pod restart with nothing in git recording it. (Commit 8ce8699 "Fix porobot-discord ImagePolicy" looks like an attempt at this that fixed the policy side but not the marker.)

2. Namespaces in metadata that contradict the directory:

• applications/steffo-porobot/imgproxy.Service.yaml:5 → starshard-porobot
• applications/starshard-soju/sojuauth.ImagePolicy.yaml:5 and sojuauth-redis.ImagePolicy.yaml:5 → starshard-sojuauth

These are currently harmless because kustomization.yaml's namespace: field overrides them, but that means the repo silently disagrees with itself and you can't trust metadata.namespace when reading a file. I'd either fix them or drop metadata.namespace from resources entirely and let the kustomization be the single source.

3. letsencrypt-production-tlsserver reuses the shortlived issuer's ACME account key (resources/letsencrypt-production-tlsserver.ClusterIssuer.yaml:15, same in the staging variant). Reusing one ACME account across two issuers mostly works, but it's clearly copy-paste rather than intent, and the two issuers can thrash the account registration if their email/config ever diverge. Give each its own privateKeySecretRef.

4. livekit.HelmRelease.yaml:24 pins version: "*". Any chart release ships straight to prod unreviewed. Everything else in the repo uses a bounded range (^1.20.2, ^84.5.0) — this should too.

Biggest real risks

5. No backups anywhere. Six CNPG Clusters with no .spec.backup — no WAL archiving, no PITR, no ScheduledBackup. Every PVC is local-path, i.e. a directory on the single node's disk. Synapse's media, GoToSocial's uploads, Forgejo's 10Gi of repos, Tranquil's blobs, and all six databases live on one disk with no copy. One disk failure or one bad force: true reconcile and it's gone. This is the single highest-value thing to add: CNPG barman-cloud object storage backups pointed at any off-node S3-compatible target, plus something like VolSync or a restic CronJob for the PVCs.

6. Secrets are entirely outside GitOps. .gitignore excludes *.Secret.yaml, and no kustomization references one — so they're applied by hand and exist nowhere but the live cluster and your laptop. You can't rebuild this cluster from the repo. SOPS with age is the natural fit (Flux decrypts natively via spec.decryption) and would let the Secret files live in the kustomizations alongside everything else.

7. Flux's git remote is inside the cluster Flux manages. flux.FluxInstance.yaml syncs from g.starshard.space, which is the Forgejo running in starshard-forgejo. If the cluster is down or Forgejo is broken, Flux can't reconcile its way out. A push-mirror to an external forge, used as a fallback source, removes the circularity.

8. No resource requests or limits on anything — not one Deployment, not one CNPG Cluster. On a single node this matters more than on a big cluster: every pod is BestEffort, so when the node hits memory pressure the kubelet's eviction ordering is essentially arbitrary. Synapse or Postgres can be killed to make room for a Discord bot. Even rough requests would establish a sane eviction order and give Grafana something to show.

9. No liveness or readiness probes on anything. This also quietly undermines your Kustomizations: they all set wait: true, but without a readinessProbe a container counts as Ready the moment it starts, so Flux reports a successful reconcile for an app that's crash-looping on a bad config. Readiness probes would make wait: true mean what you want it to mean, and would stop Traefik routing to pods that haven't finished starting.

10. No alerting on Flux failures. notification-controller is deployed but there are zero Provider/Alert resources, so a failed reconcile is only visible if you go looking. You already run Synapse and a Discord bot — a Provider pointing at either, with an Alert on all Kustomization/HelmRelease failures, is maybe 20 lines.

Reliability and operability

11. ConfigMap changes don't restart pods. Editing synapse.ConfigMap.yaml's homeserver.yml updates the ConfigMap and nothing else — the pod keeps running the old config until you manually restart it. Same for all the envFrom ConfigMaps. Either switch to kustomize configMapGenerator (the hash suffix forces a rollout, and it's the GitOps-native answer) or deploy Stakater Reloader.

12. Cert renewal doesn't reach soju. soju.Deployment.yaml mounts the i-cert secret directly — a cert owned by the sojuauth Ingress. When cert-manager rotates it, soju keeps serving the old cert until someone restarts it. The same applies to LiveKit's TURN listener using livekit-cert. I suspect this is exactly why sojuauth uses the tlsserver profile while everything else uses shortlived — a 6-day cert would break weekly. Reloader annotations would let you move soju to shortlived too. Also worth noting the ownership coupling: deleting or renaming the sojuauth Ingress silently breaks soju's TLS.

13. force: true on all eleven app Kustomizations. This tells kustomize-controller to delete-and-recreate any resource it can't patch due to an immutable field. On a Service that's a brief outage; on a PVC backed by local-path, it's data loss. I'd remove it and handle immutable-field conflicts deliberately when they come up.

14. Prometheus has no persistent storage. You set grafana.persistence in kube-prometheus-stack.HelmRelease.yaml but not prometheus.prometheusSpec.storageSpec, so Prometheus history lives in an emptyDir and vanishes on every restart. Alertmanager is also deployed with the chart's default null receiver, so nothing it fires goes anywhere.

15. Nothing app-level is actually monitored. No ServiceMonitor or PodMonitor exists. imgproxy goes out of its way to expose a Prometheus port (IMGPROXY_PROMETHEUS_BIND, service port 8081) that nothing scrapes, and CNPG's monitoring.enablePodMonitor: true is unset on all six clusters — so you have no Postgres metrics for the databases you have no backups of.

16. steffo-www's Postgres gets 256M. That's shared with WAL; Postgres will fill it. It's also the only storage value not in Gi.

17. No CI validation. Every bug in the first section — the marker mismatches, the namespace typos — is something a kustomize build + kubeconform check on push would have caught. You're already running Forgejo, so Forgejo Actions is right there. flux build/flux diff on PRs would go further.

Security hardening

18. No securityContext anywhere — no runAsNonRoot, readOnlyRootFilesystem, allowPrivilegeEscalation: false, or capability drops, and no Pod Security Standards labels on the namespaces. Adding pod-security.kubernetes.io/enforce: baseline (or restricted where it fits) to the Namespace manifests is a cheap first step.

19. No NetworkPolicies. k3s enforces them out of the box, but there are none, so any compromised pod can reach every Postgres in every namespace. A default-deny per namespace plus explicit allows would be a meaningful improvement given the per-app namespace layout you already have. Note FluxInstance.cluster.networkPolicy: true only protects flux-system.

The client-IP FIXME

gotosocial.ConfigMap.yaml disables rate limiting entirely with GTS_ADVANCED_RATE_LIMIT_REQUESTS: "0" and a FIXME about real IPs — and controllers/traefik/traefik.HelmChartConfig.yaml is an empty valuesContent: "" left over from the fight in commits b495748/7f5649b. The missing piece is that k3s's ServiceLB SNATs the source address, so Traefik never sees the real client. Two changes:

# traefik.HelmChartConfig.yaml
service:
  spec:
    externalTrafficPolicy: Local

then tell GoToSocial to trust Traefik's X-Forwarded-For — the same thing Forgejo already does with REVERSE_PROXY_TRUSTED_PROXIES: 10.42.0.0/16. GoToSocial's equivalent is GTS_TRUSTED_PROXIES set to the pod CIDR; worth confirming the exact key against the GoToSocial docs for your version before applying. With real IPs arriving you can restore rate limiting and drop GTS_LOG_LEVEL: debug (commit 3a5ffbb) back to info.

Housekeeping

• applications/starshard-prosody is a namespace and nothing else, yet carries a full Kustomization and ArtifactGenerator entry. steffo-nortrom and steffo-threadkiller are annotated "it's dead" in their own ImageRepositories but still deployed. Either finish or delete.
• Nine ImageUpdateAutomations clone the repo on 5m intervals and all push to main; that's a few thousand clones a day against Forgejo and occasional non-fast-forward retries. Consolidation isn't clean (IUA only reads ImagePolicies in its own namespace), but the 5m interval on apps whose ImageRepository only scans every 24h is pure waste — match those to 24h.
• controllers reconciles at 24h and resources at 12h, so a ClusterIssuer edit can sit unapplied for half a day. A Receiver + Forgejo webhook would make all of it push-driven and let you keep the long intervals as a safety net.
• .idea/ is committed, but workspace.xml is correctly excluded by .idea/.gitignore, so that's fine as-is.

If you want, I can start on any of these — I'd suggest the four bugs in the first section first, since they're small and self-contained, then the CNPG backups.

This was my first attempt at using a LLM as an analysis tool to identify mistakes, then fixing them myself manually. ## Thoughts It did give some results, as you can see from the commits in this PR. The whole pass expended 1.58€ of the free credit included as part of my JetBrains All Products pack. I feel would be very much **not worth it** if it wasn't included for free or if I had to spend more to get an IDE with it included. This is reinforcing my convinction that the only things LLMs are good at are highlighting design issues in existing tooling. ## LLM info ### Prompt The prompt was: ``` What would you improve in this k3s infrastructure managed via Flux CD? ``` ### External calls No requests to external calls or websites were made. ### Full response <details> <summary>Claude Agent output</summary> I'll explore the repository structure first to understand what's here. Let me read through the manifests. Now the workloads themselves. Let me verify a few specific things I noticed. The setter-marker mismatch is confirmed. A couple more quick checks: I've read through the whole repo. Overall this is a well-organized Flux setup — the `ArtifactGenerator` monorepo split is a clever use of Flux 2.8, the naming convention is consistent, and the inline comments explaining *why* (rate limits, OIDC linking, dead services) are better than most infra repos. Here's what I'd change, roughly in priority order. ## Actual bugs **1. Three image-automation setters point at ImagePolicies that don't exist.** The marker name must match the ImagePolicy's `metadata.name`, and these don't: | Marker | File | Actual policy name | |---|---|---| | `starshard-synapse:livekit` | `applications/starshard-synapse/synapse-livekit-jwt.Deployment.yaml:22` | `synapse-livekit-jwt` | | `steffo-porobot:discord` | `applications/steffo-porobot/porobot-discord.Deployment.yaml:21` | `porobot-discord` | | `steffo-porobot:telegram` | `applications/steffo-porobot/porobot-telegram.Deployment.yaml:21` | `porobot-telegram` | Those three images are silently frozen. The lk-jwt-service one is the worst: it's pinned to `:latest`, so it's both un-updated by Flux *and* un-reproducible — `:latest` defaults to `imagePullPolicy: Always`, so the running version changes on every pod restart with nothing in git recording it. (Commit 8ce8699 "Fix `porobot-discord` `ImagePolicy`" looks like an attempt at this that fixed the policy side but not the marker.) **2. Namespaces in metadata that contradict the directory:** - `applications/steffo-porobot/imgproxy.Service.yaml:5` → `starshard-porobot` - `applications/starshard-soju/sojuauth.ImagePolicy.yaml:5` and `sojuauth-redis.ImagePolicy.yaml:5` → `starshard-sojuauth` These are currently harmless because `kustomization.yaml`'s `namespace:` field overrides them, but that means the repo silently disagrees with itself and you can't trust `metadata.namespace` when reading a file. I'd either fix them or drop `metadata.namespace` from resources entirely and let the kustomization be the single source. **3. `letsencrypt-production-tlsserver` reuses the shortlived issuer's ACME account key** (`resources/letsencrypt-production-tlsserver.ClusterIssuer.yaml:15`, same in the staging variant). Reusing one ACME account across two issuers mostly works, but it's clearly copy-paste rather than intent, and the two issuers can thrash the account registration if their email/config ever diverge. Give each its own `privateKeySecretRef`. **4. `livekit.HelmRelease.yaml:24` pins `version: "*"`.** Any chart release ships straight to prod unreviewed. Everything else in the repo uses a bounded range (`^1.20.2`, `^84.5.0`) — this should too. ## Biggest real risks **5. No backups anywhere.** Six CNPG `Cluster`s with no `.spec.backup` — no WAL archiving, no PITR, no `ScheduledBackup`. Every PVC is `local-path`, i.e. a directory on the single node's disk. Synapse's media, GoToSocial's uploads, Forgejo's 10Gi of repos, Tranquil's blobs, and all six databases live on one disk with no copy. One disk failure or one bad `force: true` reconcile and it's gone. This is the single highest-value thing to add: CNPG barman-cloud object storage backups pointed at any off-node S3-compatible target, plus something like VolSync or a restic CronJob for the PVCs. **6. Secrets are entirely outside GitOps.** `.gitignore` excludes `*.Secret.yaml`, and no kustomization references one — so they're applied by hand and exist nowhere but the live cluster and your laptop. You can't rebuild this cluster from the repo. SOPS with age is the natural fit (Flux decrypts natively via `spec.decryption`) and would let the Secret files live in the kustomizations alongside everything else. **7. Flux's git remote is inside the cluster Flux manages.** `flux.FluxInstance.yaml` syncs from `g.starshard.space`, which is the Forgejo running in `starshard-forgejo`. If the cluster is down or Forgejo is broken, Flux can't reconcile its way out. A push-mirror to an external forge, used as a fallback source, removes the circularity. **8. No resource requests or limits on anything** — not one Deployment, not one CNPG Cluster. On a single node this matters more than on a big cluster: every pod is BestEffort, so when the node hits memory pressure the kubelet's eviction ordering is essentially arbitrary. Synapse or Postgres can be killed to make room for a Discord bot. Even rough requests would establish a sane eviction order and give Grafana something to show. **9. No liveness or readiness probes on anything.** This also quietly undermines your Kustomizations: they all set `wait: true`, but without a readinessProbe a container counts as Ready the moment it starts, so Flux reports a successful reconcile for an app that's crash-looping on a bad config. Readiness probes would make `wait: true` mean what you want it to mean, and would stop Traefik routing to pods that haven't finished starting. **10. No alerting on Flux failures.** `notification-controller` is deployed but there are zero `Provider`/`Alert` resources, so a failed reconcile is only visible if you go looking. You already run Synapse and a Discord bot — a `Provider` pointing at either, with an `Alert` on all `Kustomization`/`HelmRelease` failures, is maybe 20 lines. ## Reliability and operability **11. ConfigMap changes don't restart pods.** Editing `synapse.ConfigMap.yaml`'s `homeserver.yml` updates the ConfigMap and nothing else — the pod keeps running the old config until you manually restart it. Same for all the `envFrom` ConfigMaps. Either switch to kustomize `configMapGenerator` (the hash suffix forces a rollout, and it's the GitOps-native answer) or deploy Stakater Reloader. **12. Cert renewal doesn't reach `soju`.** `soju.Deployment.yaml` mounts the `i-cert` secret directly — a cert owned by the `sojuauth` Ingress. When cert-manager rotates it, soju keeps serving the old cert until someone restarts it. The same applies to LiveKit's TURN listener using `livekit-cert`. I suspect this is exactly why sojuauth uses the `tlsserver` profile while everything else uses `shortlived` — a 6-day cert would break weekly. Reloader annotations would let you move soju to `shortlived` too. Also worth noting the ownership coupling: deleting or renaming the `sojuauth` Ingress silently breaks soju's TLS. **13. `force: true` on all eleven app Kustomizations.** This tells kustomize-controller to delete-and-recreate any resource it can't patch due to an immutable field. On a Service that's a brief outage; on a PVC backed by `local-path`, it's data loss. I'd remove it and handle immutable-field conflicts deliberately when they come up. **14. Prometheus has no persistent storage.** You set `grafana.persistence` in `kube-prometheus-stack.HelmRelease.yaml` but not `prometheus.prometheusSpec.storageSpec`, so Prometheus history lives in an emptyDir and vanishes on every restart. Alertmanager is also deployed with the chart's default null receiver, so nothing it fires goes anywhere. **15. Nothing app-level is actually monitored.** No `ServiceMonitor` or `PodMonitor` exists. imgproxy goes out of its way to expose a Prometheus port (`IMGPROXY_PROMETHEUS_BIND`, service port 8081) that nothing scrapes, and CNPG's `monitoring.enablePodMonitor: true` is unset on all six clusters — so you have no Postgres metrics for the databases you have no backups of. **16. `steffo-www`'s Postgres gets 256M.** That's shared with WAL; Postgres will fill it. It's also the only storage value not in `Gi`. **17. No CI validation.** Every bug in the first section — the marker mismatches, the namespace typos — is something a `kustomize build` + `kubeconform` check on push would have caught. You're already running Forgejo, so Forgejo Actions is right there. `flux build`/`flux diff` on PRs would go further. ## Security hardening **18. No `securityContext` anywhere** — no `runAsNonRoot`, `readOnlyRootFilesystem`, `allowPrivilegeEscalation: false`, or capability drops, and no Pod Security Standards labels on the namespaces. Adding `pod-security.kubernetes.io/enforce: baseline` (or `restricted` where it fits) to the Namespace manifests is a cheap first step. **19. No NetworkPolicies.** k3s enforces them out of the box, but there are none, so any compromised pod can reach every Postgres in every namespace. A default-deny per namespace plus explicit allows would be a meaningful improvement given the per-app namespace layout you already have. Note `FluxInstance.cluster.networkPolicy: true` only protects `flux-system`. ## The client-IP FIXME `gotosocial.ConfigMap.yaml` disables rate limiting entirely with `GTS_ADVANCED_RATE_LIMIT_REQUESTS: "0"` and a FIXME about real IPs — and `controllers/traefik/traefik.HelmChartConfig.yaml` is an empty `valuesContent: ""` left over from the fight in commits b495748/7f5649b. The missing piece is that k3s's ServiceLB SNATs the source address, so Traefik never sees the real client. Two changes: ```yaml # traefik.HelmChartConfig.yaml service: spec: externalTrafficPolicy: Local ``` then tell GoToSocial to trust Traefik's `X-Forwarded-For` — the same thing Forgejo already does with `REVERSE_PROXY_TRUSTED_PROXIES: 10.42.0.0/16`. GoToSocial's equivalent is `GTS_TRUSTED_PROXIES` set to the pod CIDR; worth confirming the exact key against the GoToSocial docs for your version before applying. With real IPs arriving you can restore rate limiting and drop `GTS_LOG_LEVEL: debug` (commit 3a5ffbb) back to `info`. ## Housekeeping - `applications/starshard-prosody` is a namespace and nothing else, yet carries a full Kustomization and ArtifactGenerator entry. `steffo-nortrom` and `steffo-threadkiller` are annotated "it's dead" in their own ImageRepositories but still deployed. Either finish or delete. - Nine `ImageUpdateAutomation`s clone the repo on 5m intervals and all push to `main`; that's a few thousand clones a day against Forgejo and occasional non-fast-forward retries. Consolidation isn't clean (IUA only reads ImagePolicies in its own namespace), but the 5m interval on apps whose ImageRepository only scans every 24h is pure waste — match those to 24h. - `controllers` reconciles at 24h and `resources` at 12h, so a ClusterIssuer edit can sit unapplied for half a day. A `Receiver` + Forgejo webhook would make all of it push-driven and let you keep the long intervals as a safety net. - `.idea/` is committed, but `workspace.xml` is correctly excluded by `.idea/.gitignore`, so that's fine as-is. If you want, I can start on any of these — I'd suggest the four bugs in the first section first, since they're small and self-contained, then the CNPG backups. </details>

steffo added 1 commit 2026-08-12 16:42:08 +00:00

Fix mismatches in $imagepolicy names ... 77d28bee5a

See the merge commit for comments on this LLM pass.

Assisted-by: Claude Agent (Claude Opus 5) (via Jetbrains AI Assistant)

steffo added 3 commits 2026-08-12 17:13:35 +00:00

Fix mismatches in namespace names ... 59d643f763

See the merge commit for comments on this LLM pass.

Assisted-by: Claude Agent (Claude Opus 5) (via Jetbrains AI Assistant)

Fix mismatch in privateKeySecretRef key name ... 9f0eb317d2

See the merge commit for comments on this LLM pass.

Assisted-by: Claude Agent (Claude Opus 5) (via Jetbrains AI Assistant)

Pin livekit helm chart version to 1.* ... 454d30598e

This is pretty much useless but sure, why not.

See the merge commit for comments on this LLM pass.

Assisted-by: Claude Agent (Claude Opus 5) (via Jetbrains AI Assistant)

steffo changed title from WIP: Fix mistakes identified via LLM analysis to Fix mistakes identified via LLM analysis 2026-08-12 21:40:19 +00:00

steffo merged commit ae7a6de08b into main 2026-08-12 21:41:02 +00:00

steffo deleted branch fix/llm-analysis-experiment 2026-08-12 21:41:03 +00:00

steffo referenced this pull request from a commit 2026-08-12 21:41:04 +00:00

Merge: Fix mistakes identified via LLM analysis (#11)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

starshard/infrastructure!11

No description provided.