Building a Production-Ready GitOps Platform on Talos Kubernetes

The Challenge

Production-grade Kubernetes environments require rigorous solutions to several fundamental problems. The primary challenge lies in reconciling the transparency of GitOps with the confidentiality required for credentials. Storing sensitive data in public repositories exposes systems to immediate compromise, yet maintaining secrets outside of version control breaks the "single source of truth" paradigm essential to GitOps.

Furthermore, managing shared resources across distinct applications without sacrificing isolation presents complex networking and access control hurdles. The objective was to architect a platform that enables continuous deployment and efficient resource utilization while enforcing strict security boundaries.

Architecture Overview

Infrastructure Topology

Multi-Namespace Architecture

The cluster employs a multi-namespace architecture to enforce logical separation of concerns. This design pattern allows for granular resource quotas, network policies, and access controls while enabling efficient resource sharing where appropriate.

Infrastructure Namespace (data)

The data namespace functions as a shared service layer. By centralizing stateful workloads here, the architecture reduces the operational overhead of managing redundant database instances for every microservice.

Application Namespace (chatbot)

Application namespaces host the stateless microservices associated with specific business domains. Each service maintains its own deployment lifecycle, with configuration injected at runtime via Kubernetes ConfigMaps and Secrets.

Secret Management with Sealed Secrets

The Problem

Standard Kubernetes Secrets are merely base64-encoded strings, offering no encryption at rest within the configuration files. This limitation effectively prevents the commitment of secret manifests to version control systems, creating a divergence between the desired state (Git) and the actual state (Cluster).

The Solution

Bitnami Sealed Secrets implements asymmetric cryptography to secure credentials. The cluster maintains a private key known only to the Sealed Secrets controller, while a public key is distributed to developers. This allows secrets to be encrypted offline ("sealed") and safely committed to public repositories.

Implementation Details

# Install Sealed Secrets controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/\
  releases/download/v0.26.0/controller.yaml

# Fetch the public key
kubeseal --fetch-cert > sealed-secrets-cert.pem

# Create and encrypt a secret
kubectl create secret generic my-secret \
  --namespace=my-namespace \
  --from-literal=password=super-secret \
  --dry-run=client -o yaml | \
kubeseal --format yaml --cert sealed-secrets-cert.pem \
  > my-sealed-secret.yaml

# Commit the encrypted version
git add my-sealed-secret.yaml
git commit -m "Add encrypted secret"
git push

GitOps with ArgoCD

Repository Structure

App-of-Apps Pattern

The "App of Apps" pattern is utilized to manage the application landscape declaratively. A root "meta-application" tracks a directory of Application manifests, allowing the entire cluster state to be bootstrapped from a single entry point. This approach ensures scalability and consistent policy enforcement across all deployed workloads.

Automated Sync Configuration

Each application defines strict synchronization policies to ensure the cluster state always converges with the Git repository.

Cross-Namespace Communication

Reliable inter-service communication is established using Kubernetes native DNS service discovery. By addressing services via their Fully Qualified Domain Names (FQDNs), the architecture decouples networking from IP addressing, allowing for dynamic pod rescheduling without service interruption.

Service Mesh Architecture

Operational Constraints & Resolutions

Constraint 1: Sealed Secret Ownership

The Sealed Secrets controller employs strict ownership checks to prevent accidental overwrites of existing resources. Secrets created manually prior to the GitOps implementation will block the synchronization process.

# Purge unmanaged secrets
kubectl delete secret postgres-secret -n data
kubectl delete secret chatbot-secret -n chatbot

# Force controller cache refresh
kubectl delete pod -n kube-system \
  -l name=sealed-secrets-controller

Policy: All secrets in managed namespaces must be originated via SealedSecret manifests.

Constraint 2: Automated Repository Authentication

Automated GitOps agents (ArgoCD) require non-interactive authentication methods. SSH keys protected by passphrases will cause the synchronization loop to fail, as the agent cannot provide the decryption password at runtime.

# Generate passphrase-less ed25519 key
ssh-keygen -t ed25519 -C "argocd@homelab" \
  -f ~/.ssh/argocd_nopass -N "" 

Constraint 3: Distributed Development State

Insight: GitOps fundamentally decouples the development environment from the runtime environment. Modifications made directly to the cluster (via kubectl) are ephemeral and will be overwritten by the controller. All persistent changes must be committed to the remote Git repository, which serves as the sole authority for cluster state.

Performance & Efficiency

Resource Optimization

The shared services architectural pattern yields measurable efficiency gains compared to isolated deployments. By consolidating stateful services, we eliminate the overhead of redundant runtime environments and caching layers.

Sync Latency

Security Posture

Operational Metrics

Roadmap

Conclusion

This architecture validates that enterprise-grade reliability and security patterns are attainable in compact Kubernetes environments. By leveraging Talos Linux for immutable infrastructure, ArgoCD for declarative state management, and Sealed Secrets for encrypted configuration, the platform achieves a high degree of automation and security.

The transition to GitOps fundamentally reframes infrastructure management as a software engineering discipline. It enables the application of rigorous version control, peer review, and automated testing to the infrastructure itself, resulting in a system that is audit-friendly, resilient to drift, and rapid to recover.

Appendix: Deployment Reference

The following procedure illustrates the standard operating procedure for onboarding a new microservice:

# 1. Clone repository
git clone git@github.com:T4Bu/homelab-gitops.git
cd homelab-gitops

# 2. Define Application Manifests
mkdir -p apps/newapp
vim apps/newapp/deployment.yaml

# 3. Generate Encrypted Secret
./scripts/create-sealed-secret.sh app-secret newapp \
  API_KEY=secret123 \
  DATABASE_URL=postgresql://...

# 4. Register Application with ArgoCD
cat > argocd-apps/newapp.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: newapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: git@github.com:T4Bu/homelab-gitops.git
    targetRevision: HEAD
    path: apps/newapp
  destination:
    server: https://kubernetes.default.svc
    namespace: newapp
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
EOF

# 5. Commit to Trigger Deployment
git add apps/newapp/ argocd-apps/newapp.yaml
git commit -m "feat: onboard newapp service"
git push

# 6. Verify Convergence
argocd app get newapp --watch

# 7. Validation
kubectl get all -n newapp