turingpi/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Repository Overview

This repository contains Kubernetes configuration files for a K3s homelab cluster running on TuringPi hardware. It includes Helm charts, values files, and manifests for deploying various self-hosted applications.

## Cluster Architecture

### Hardware Setup
- **turing1**: Control plane + worker (master node, IP: 192.168.222.237)
- **turing2**: Worker node (currently SchedulingDisabled)  
- **turing3**: Worker node (also serves as NFS server at turing3.lan)
- **turing4**: Worker node
- **beelink**: Additional worker node

### Core Infrastructure
- **K3s version**: v1.31.6+k3s1
- **Storage**: NFS-backed persistent volumes from turing3.lan:/mnt/ssd
- **Load Balancer**: MetalLB for bare metal LoadBalancer services
- **SSL**: cert-manager with Let's Encrypt (staging/production cluster issuers)
- **Ingress**: Traefik (K3s default) with LAN-only restrictions

## Application Stack

### Media Services
- **Plex**: kube-plex (Kubernetes-native with dynamic transcoding pods)
- **Jellyfin**: Alternative media server
- **Sonarr/Radarr**: TV/Movie management (Bananaspliff charts)
- **Prowlarr**: Indexer management (custom chart)
- **Transmission**: BitTorrent client with OpenVPN
- **FlareSolverr**: Captcha solver service

### Other Applications
- **Actual Budget**: Personal finance (custom chart: my-actual-server/)
- **Home Assistant Voice LLMs**: AI voice integration (custom chart)
- **Ollama**: Local LLM inference
- **Prometheus**: Monitoring stack
- **PostgreSQL**: Database backend

## Common Helm Operations

### Repository Management
```bash
# Key repositories used
helm repo add metallb https://metallb.github.io/metallb
helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner
helm repo add jetstack https://charts.jetstack.io
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add bananaspliff https://bananaspliff.github.io/geek-charts
helm repo add k8s-at-home https://k8s-at-home.com/charts/
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add jellyfin https://jellyfin.github.io/jellyfin-helm
helm repo add ollama-helm https://otwld.github.io/ollama-helm/
helm repo update
```

### Application Deployment Pattern
```bash
# Standard deployment with values file
helm upgrade <release-name> <chart> -f <app>_values.yaml -i

# Examples from history:
helm upgrade actual my-actual-server -f actual_values.yaml -i  
helm upgrade plex kube-plex/charts/kube-plex --values plex_values.yml
helm upgrade radarr bananaspliff/radarr -f radarr_values.yaml
helm upgrade sonarr bananaspliff/sonarr -f sonarr_values.yaml
helm upgrade prowlarr prowlarr -f prowlarr_values.yml
```

### Development Workflow
```bash
# Chart development
helm create <chart-name>
helm lint <chart-path>
helm template <chart> -f <values> | vim -

# Values inspection
helm show values <chart> > <app>_values.yaml
helm get values <release-name>
helm get manifest <release-name>
```

## File Structure Patterns

- `<app>_values.yaml` - Helm values overrides for each application
- Custom charts in subdirectories (my-actual-server/, home-assistant-voice-llms/, prowlarr/)
- `*_persistent_volume.yml` - PV definitions for applications requiring storage
- Infrastructure manifests: metallb.yml, ingress.yaml, cluster-issuer-*.yaml

## Storage Configuration

- **NFS Server**: turing3.lan serving /mnt/ssd
- **StorageClass**: nfs-client (via nfs-subdir-external-provisioner)
- **Access Mode**: ReadWriteMany for shared media access
- **PVC Pattern**: Applications create their own PVCs or reference pre-existing ones

## Network Setup

- **Pod Network**: Cluster subnet requires allowlisting in Plex for transcoding
- **Ingress**: LAN-only access enforced via limit_ingress_to_lan.yaml
- **Load Balancer**: MetalLB provides external IPs for services
- **DNS**: .lan domain for internal services

## Kube-Plex Specifics

The kube-plex/ directory contains a Go application that replaces the standard Plex transcoder:
- Creates Kubernetes pods for each transcode job
- Requires AMD64 nodes (configured via nodeSelector)
- Mounts shared NFS volumes for media access
- Environment variables: DATA_PVC, CONFIG_PVC, TRANSCODE_PVC, PMS_IMAGE, PMS_INTERNAL_ADDRESS

## ArgoCD Migration Guidelines

### Critical Configuration Preservation Rules
When migrating applications from direct Helm deployment to ArgoCD GitOps:

1. **NEVER change working configurations** - If an application was working with specific values files and service configurations, maintain those exact settings
2. **NO PATCHES OR WORKAROUNDS** - Do not create additional PVCs, port forwards, or other patches to "fix" configuration mismatches
3. **Root Cause Analysis Required** - Always identify why a configuration that worked before is now failing after ArgoCD migration
4. **Preserve Original Service Ports** - Applications should maintain their original service port configurations (e.g., Radarr:7878, Sonarr:8989)
5. **Respect Existing PVC Structure** - Use the existing PVC and storage configuration patterns, don't create compatibility layers

### Troubleshooting Process
1. Compare working Helm values with ArgoCD application configuration
2. Verify that Helm chart sources and versions match original deployments
3. Ensure service definitions in ArgoCD match original working services
4. Check that ingress configurations align with actual service ports
5. Fix the root configuration issue, don't patch around it

### Common Migration Pitfalls
- Creating unnecessary "myvolume" PVCs for chart compatibility - instead fix the chart values
- Changing service ports to match ingress instead of fixing ingress to match services
- Using generic chart defaults instead of preserving working custom configurations

## ArgoCD GitOps Implementation Lessons

### Successfully Migrated Applications
- **Plex**: Multi-source setup (kube-plex chart + turingpi values)
- **Radarr**: Bananaspliff chart with custom volume configuration  
- **Sonarr**: Bananaspliff chart with custom volume configuration

### Critical ArgoCD Configuration Patterns

#### Multi-Source Application Structure
For applications requiring custom values from Git repository:
```yaml
spec:
  project: default
  sources:
    - repoURL: https://chart-repository.com/charts
      chart: app-name
      targetRevision: "*"
      ref: charts
      helm:
        releaseName: app-name
        valueFiles:
          - $values/helm-values/app_values.yaml
    - repoURL: http://gitea-http.gitea.svc.cluster.local:3000/admin/turingpi.git
      targetRevision: HEAD
      ref: values
```

**CRITICAL**: Never use both `source:` and `sources:` sections - this creates conflicts where `source:` overrides `sources:` configuration.

#### Image Auto-Update Configuration  
For applications using "latest" tags with automatic updates:
```yaml
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: app=registry/image:latest
    argocd-image-updater.argoproj.io/app.update-strategy: digest
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: master
    argocd-image-updater.argoproj.io/git-commit-user: argocd-image-updater
    argocd-image-updater.argoproj.io/git-commit-email: argocd@turing.lan
```

### Chart-Specific Configuration Requirements

#### Bananaspliff Charts (Radarr/Sonarr)
These charts require specific volume configuration syntax:
```yaml
# CORRECT: Direct volumes/volumeMounts (working configuration)
volumes:
  - name: "plex-data"
    persistentVolumeClaim:
      claimName: "plex-data"

volumeMounts:
  - name: "plex-data"
    mountPath: "/config" 
    subPath: "configs/app-name"
  - name: "plex-data"
    mountPath: "/nfs"

# INCORRECT: persistence syntax (doesn't work with these charts)  
persistence:
  config:
    enabled: true
    existingClaim: "plex-data"
    subPath: "configs/app-name"
```

#### Service Port Configuration
Bananaspliff charts use simple service structure:
```yaml
service:
  type: ClusterIP
  port: 7878  # Direct port specification
```

### Ingress Controller Configuration

#### K3s Default Setup
- **Default Controller**: Traefik (not Nginx)
- **Ingress Class**: Use `kubernetes.io/ingress.class: traefik` 
- **Common Mistake**: Using `nginx` class when Traefik is the active controller

#### Verification Commands
```bash
# Check active ingress controller
kubectl get pods -A | grep traefik
kubectl get pods -A | grep ingress

# Verify ingress class in configurations
kubectl get ingress -o yaml | grep "ingress.class"
```

### Pre-Migration Checklist
Before migrating any application to ArgoCD:

1. **Capture Current Configuration**:
   ```bash
   helm get values <release-name> > original_values.yaml
   helm get manifest <release-name> > original_manifest.yaml
   ```

2. **Document Volume Mounts**:
   ```bash
   kubectl describe deployment <app-name> | grep -A 10 "Mounts:"
   ```

3. **Verify Service Configuration**:
   ```bash
   kubectl get svc <app-name> -o yaml
   ```

4. **Test Chart Template Locally**:
   ```bash
   helm template <app-name> <chart> --values <values-file> | grep -A 20 "kind: Service"
   ```

### Post-Migration Verification

1. **Check Volume Mounts Match Original**:
   ```bash
   kubectl describe pod -l app=<app-name> | grep -A 10 "Mounts:"
   ```

2. **Verify Service Ports**:
   ```bash
   kubectl get svc <app-name>
   ```

3. **Test Application Access**:
   ```bash
   curl -I http://<app-domain>
   ```

4. **Confirm Configuration Persistence**:
   ```bash
   kubectl exec -it deployment/<app-name> -- ls -la /config
   ```

### ArgoCD Image-Updater Multi-Source Application Issues

#### Problem: Credential Errors with Multi-Source Applications
When using ArgoCD image-updater with multi-source applications (chart from external repo + values from Git), the image-updater may fail with credential errors like:
```
Could not update application spec: could not get creds for repo 'https://chart-repository.com': credentials for 'https://chart-repository.com' are not configured in Argo CD settings
```

#### Root Cause Analysis
1. **Multi-Source Confusion**: Image-updater tries to write back changes to the chart repository instead of the values repository
2. **Git Write-Back Limitations**: The `git` write-back method doesn't handle multi-source applications properly
3. **Repository Credentials**: External chart repositories (like Bananaspliff) don't have write credentials configured

#### Solution: Use ArgoCD API Write-Back Method
Instead of using `git` write-back method, use the `argocd` API method for multi-source applications:

```yaml
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: app=registry/image:latest
    argocd-image-updater.argoproj.io/app.update-strategy: digest
    argocd-image-updater.argoproj.io/write-back-method: argocd  # Use ArgoCD API instead of git
    argocd-image-updater.argoproj.io/write-back-target: http://git-repo.local/values.git  # Optional: specify target repo
```

#### Implementation Steps
1. **Update Image-Updater Configuration**:
   ```bash
   kubectl patch configmap argocd-image-updater-config -n argocd --patch '{"data":{"git.user":"argocd-image-updater","git.email":"argocd@turing.lan"}}'
   ```

2. **Change Application Write-Back Method**:
   ```bash
   kubectl patch application <app-name> -n argocd --type='merge' --patch='{"metadata":{"annotations":{"argocd-image-updater.argoproj.io/write-back-method":"argocd"}}}'
   ```

3. **Restart Image-Updater**:
   ```bash
   kubectl rollout restart deployment argocd-image-updater -n argocd
   ```

#### Verification Commands
```bash
# Check image-updater logs for success
kubectl logs -n argocd deployment/argocd-image-updater --tail=20

# Look for these success indicators:
# - "Successfully updated the live application spec" 
# - "Processing results: applications=X images_considered=X images_skipped=0 images_updated=X errors=0"

# Verify applications remain healthy
argocd app list

# Check that pods are updated with new images
kubectl get pods -l app=<app-name>
```

#### Key Learnings
- **ArgoCD API Method**: Works better than Git write-back for multi-source applications
- **No Repository Credentials Needed**: ArgoCD API method doesn't require external repository write credentials
- **Application Spec Updates**: Changes are applied directly to ArgoCD application specs, not Git files
- **Multi-Source Compatibility**: This approach handles complex application configurations properly