turingpi/CLAUDE.md

CLAUDE.md

This file provides guidance to LLM agents like Claude, codex or cursor-cli when working with code in this repository.

Repository Overview

This repository contains Kubernetes configuration files for a K3s homelab cluster running on TuringPi hardware. It includes Helm charts, values files, and manifests for deploying various self-hosted applications.

Cluster Architecture

Hardware Setup

• turing1: Control plane + worker (master node, IP: 192.168.222.237)
• turing2: Worker node (currently SchedulingDisabled)
• turing3: Worker node (also serves as NFS server at turing3.lan)
• turing4: Worker node
• beelink: Additional worker node

Core Infrastructure

• K3s version: v1.31.6+k3s1
• Storage: NFS-backed persistent volumes from turing3.lan:/mnt/ssd
• Load Balancer: MetalLB for bare metal LoadBalancer services
• SSL: cert-manager with Let's Encrypt (staging/production cluster issuers)
• Ingress: Traefik (K3s default) with LAN-only restrictions

Application Stack

Media Services

• Plex: kube-plex (Kubernetes-native with dynamic transcoding pods)
• Jellyfin: Alternative media server
• Sonarr/Radarr: TV/Movie management (Bananaspliff charts)
• Prowlarr: Indexer management (custom chart)
• Transmission: BitTorrent client with OpenVPN
• FlareSolverr: Captcha solver service

Other Applications

• Actual Budget: Personal finance (custom chart: my-actual-server/)
• Home Assistant Voice LLMs: AI voice integration (custom chart)
• Ollama: Local LLM inference
• Prometheus: Monitoring stack
• PostgreSQL: Database backend

Common Helm Operations

Repository Management

# Key repositories used
helm repo add metallb https://metallb.github.io/metallb
helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner
helm repo add jetstack https://charts.jetstack.io
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add bananaspliff https://bananaspliff.github.io/geek-charts
helm repo add k8s-at-home https://k8s-at-home.com/charts/
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add jellyfin https://jellyfin.github.io/jellyfin-helm
helm repo add ollama-helm https://otwld.github.io/ollama-helm/
helm repo update

Application Deployment Pattern

# Standard deployment with values file
helm upgrade <release-name> <chart> -f <app>_values.yaml -i

# Examples from history:
helm upgrade actual my-actual-server -f actual_values.yaml -i  
helm upgrade plex kube-plex/charts/kube-plex --values plex_values.yml
helm upgrade radarr bananaspliff/radarr -f radarr_values.yaml
helm upgrade sonarr bananaspliff/sonarr -f sonarr_values.yaml
helm upgrade prowlarr prowlarr -f prowlarr_values.yml

Development Workflow

# Chart development
helm create <chart-name>
helm lint <chart-path>
helm template <chart> -f <values> | vim -

# Values inspection
helm show values <chart> > <app>_values.yaml
helm get values <release-name>
helm get manifest <release-name>

File Structure Patterns

• <app>_values.yaml - Helm values overrides for each application
• Custom charts in subdirectories (my-actual-server/, home-assistant-voice-llms/, prowlarr/)
• *_persistent_volume.yml - PV definitions for applications requiring storage
• Infrastructure manifests: metallb.yml, ingress.yaml, cluster-issuer-*.yaml

Storage Configuration

• NFS Server: turing3.lan serving /mnt/ssd
• StorageClass: nfs-client (via nfs-subdir-external-provisioner)
• Access Mode: ReadWriteMany for shared media access
• PVC Pattern: Applications create their own PVCs or reference pre-existing ones

Network Setup

• Pod Network: Cluster subnet requires allowlisting in Plex for transcoding
• Ingress: LAN-only access enforced via limit_ingress_to_lan.yaml
• Load Balancer: MetalLB provides external IPs for services
• DNS: .lan domain for internal services

Kube-Plex Specifics

The kube-plex/ directory contains a Go application that replaces the standard Plex transcoder:

• Creates Kubernetes pods for each transcode job
• Requires AMD64 nodes (configured via nodeSelector)
• Mounts shared NFS volumes for media access
• Environment variables: DATA_PVC, CONFIG_PVC, TRANSCODE_PVC, PMS_IMAGE, PMS_INTERNAL_ADDRESS

ArgoCD Migration Guidelines

Critical Configuration Preservation Rules

When migrating applications from direct Helm deployment to ArgoCD GitOps:

1. NEVER change working configurations - If an application was working with specific values files and service configurations, maintain those exact settings
2. NO PATCHES OR WORKAROUNDS - Do not create additional PVCs, port forwards, or other patches to "fix" configuration mismatches
3. Root Cause Analysis Required - Always identify why a configuration that worked before is now failing after ArgoCD migration
4. Preserve Original Service Ports - Applications should maintain their original service port configurations (e.g., Radarr:7878, Sonarr:8989)
5. Respect Existing PVC Structure - Use the existing PVC and storage configuration patterns, don't create compatibility layers

Troubleshooting Process

1. Compare working Helm values with ArgoCD application configuration
2. Verify that Helm chart sources and versions match original deployments
3. Ensure service definitions in ArgoCD match original working services
4. Check that ingress configurations align with actual service ports
5. Fix the root configuration issue, don't patch around it

Common Migration Pitfalls

• Creating unnecessary "myvolume" PVCs for chart compatibility - instead fix the chart values
• Changing service ports to match ingress instead of fixing ingress to match services
• Using generic chart defaults instead of preserving working custom configurations

ArgoCD GitOps Implementation Lessons

Successfully Migrated Applications

• Plex: Multi-source setup (kube-plex chart + turingpi values)
• Radarr: Bananaspliff chart with custom volume configuration
• Sonarr: Bananaspliff chart with custom volume configuration

Critical ArgoCD Configuration Patterns

Multi-Source Application Structure

For applications requiring custom values from Git repository:

spec:
  project: default
  sources:
    - repoURL: https://chart-repository.com/charts
      chart: app-name
      targetRevision: "*"
      ref: charts
      helm:
        releaseName: app-name
        valueFiles:
          - $values/helm-values/app_values.yaml
    - repoURL: http://gitea-http.gitea.svc.cluster.local:3000/admin/turingpi.git
      targetRevision: HEAD
      ref: values

CRITICAL: Never use both source: and sources: sections - this creates conflicts where source: overrides sources: configuration.

Image Auto-Update Configuration

For applications using "latest" tags with automatic updates:

metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: app=registry/image:latest
    argocd-image-updater.argoproj.io/app.update-strategy: digest
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: master
    argocd-image-updater.argoproj.io/git-commit-user: argocd-image-updater
    argocd-image-updater.argoproj.io/git-commit-email: argocd@turing.lan

Chart-Specific Configuration Requirements

Bananaspliff Charts (Radarr/Sonarr)

These charts require specific volume configuration syntax:

# CORRECT: Direct volumes/volumeMounts (working configuration)
volumes:
  - name: "plex-data"
    persistentVolumeClaim:
      claimName: "plex-data"

volumeMounts:
  - name: "plex-data"
    mountPath: "/config" 
    subPath: "configs/app-name"
  - name: "plex-data"
    mountPath: "/nfs"

# INCORRECT: persistence syntax (doesn't work with these charts)  
persistence:
  config:
    enabled: true
    existingClaim: "plex-data"
    subPath: "configs/app-name"

Service Port Configuration

Bananaspliff charts use simple service structure:

service:
  type: ClusterIP
  port: 7878  # Direct port specification

Ingress Controller Configuration

K3s Default Setup

• Default Controller: Traefik (not Nginx)
• Ingress Class: Use kubernetes.io/ingress.class: traefik
• Common Mistake: Using nginx class when Traefik is the active controller

Verification Commands

# Check active ingress controller
kubectl get pods -A | grep traefik
kubectl get pods -A | grep ingress

# Verify ingress class in configurations
kubectl get ingress -o yaml | grep "ingress.class"

Pre-Migration Checklist

Before migrating any application to ArgoCD:

1. Capture Current Configuration:

   helm get values <release-name> > original_values.yaml
   helm get manifest <release-name> > original_manifest.yaml
   

2. Document Volume Mounts:

   kubectl describe deployment <app-name> | grep -A 10 "Mounts:"
   

3. Verify Service Configuration:

   kubectl get svc <app-name> -o yaml
   

4. Test Chart Template Locally:

   helm template <app-name> <chart> --values <values-file> | grep -A 20 "kind: Service"
   

Post-Migration Verification

1. Check Volume Mounts Match Original:

   kubectl describe pod -l app=<app-name> | grep -A 10 "Mounts:"
   

2. Verify Service Ports:

   kubectl get svc <app-name>
   

3. Test Application Access:

   curl -I http://<app-domain>
   

4. Confirm Configuration Persistence:

   kubectl exec -it deployment/<app-name> -- ls -la /config
   

ArgoCD Image-Updater Multi-Source Application Issues

Problem: Credential Errors with Multi-Source Applications

When using ArgoCD image-updater with multi-source applications (chart from external repo + values from Git), the image-updater may fail with credential errors like:

Could not update application spec: could not get creds for repo 'https://chart-repository.com': credentials for 'https://chart-repository.com' are not configured in Argo CD settings

Root Cause Analysis

1. Multi-Source Confusion: Image-updater tries to write back changes to the chart repository instead of the values repository
2. Git Write-Back Limitations: The git write-back method doesn't handle multi-source applications properly
3. Repository Credentials: External chart repositories (like Bananaspliff) don't have write credentials configured

Solution: Use ArgoCD API Write-Back Method

Instead of using git write-back method, use the argocd API method for multi-source applications:

metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: app=registry/image:latest
    argocd-image-updater.argoproj.io/app.update-strategy: digest
    argocd-image-updater.argoproj.io/write-back-method: argocd  # Use ArgoCD API instead of git
    argocd-image-updater.argoproj.io/write-back-target: http://git-repo.local/values.git  # Optional: specify target repo

Implementation Steps

1. Update Image-Updater Configuration:

   kubectl patch configmap argocd-image-updater-config -n argocd --patch '{"data":{"git.user":"argocd-image-updater","git.email":"argocd@turing.lan"}}'
   

2. Change Application Write-Back Method:

   kubectl patch application <app-name> -n argocd --type='merge' --patch='{"metadata":{"annotations":{"argocd-image-updater.argoproj.io/write-back-method":"argocd"}}}'
   

3. Restart Image-Updater:

   kubectl rollout restart deployment argocd-image-updater -n argocd
   

Verification Commands

# Check image-updater logs for success
kubectl logs -n argocd deployment/argocd-image-updater --tail=20

# Look for these success indicators:
# - "Successfully updated the live application spec" 
# - "Processing results: applications=X images_considered=X images_skipped=0 images_updated=X errors=0"

# Verify applications remain healthy
argocd app list

# Check that pods are updated with new images
kubectl get pods -l app=<app-name>

Key Learnings

• ArgoCD API Method: Works better than Git write-back for multi-source applications
• No Repository Credentials Needed: ArgoCD API method doesn't require external repository write credentials
• Application Spec Updates: Changes are applied directly to ArgoCD application specs, not Git files
• Multi-Source Compatibility: This approach handles complex application configurations properly