Skip to Content
OperationsRBAC

RBAC (Role-Based Access Control)

Overview

The homelab implements a unified RBAC system that automatically synchronizes permissions across both ArgoCD and Kubernetes. A single User CRD serves as the source of truth, with the RBAC Operator automating RoleBinding creation and ArgoCD policy synchronization.

Key Features:

  • Single Source of Truth: User CRDs (zengarden.space/v1) define all user access
  • Automated Synchronization: RBAC Operator manages both Kubernetes RoleBindings and ArgoCD policies
  • Four-Tier Role Model: Hierarchical roles from developer to cluster admin
  • Dynamic Namespace Discovery: Automatically discovers application namespaces from ArgoCD Applications
  • ClusterRole Annotations: Namespace configuration stored in ClusterRole annotations (no manual labeling needed)

Architecture

┌─────────────────────────────────────────────────────────────┐
│                        User CRD                             │
│  apiVersion: zengarden.space/v1                             │
│  kind: User                                                 │
│  spec:                                                      │
│    email: [email protected]                                  │
│    roles: [app-developer, platform-operator, system-admin]  │
│    enabled: true                                            │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│                   RBAC Operator                             │
│  (StatefulSet in rbac-system namespace)                     │
│  ┌────────────────┐  ┌─────────────────────────────┐       │
│  │ Shell-Operator │  │ Python Service              │       │
│  │ - Watches:     │◄─┤ - ClusterRole discovery     │       │
│  │   • Users      │  │ - Namespace expansion       │       │
│  │   • ClusterRoles│ │ - RoleBinding creation     │       │
│  │   • ArgoCD Apps│  │ - ArgoCD RBAC sync         │       │
│  └────────────────┘  └─────────────────────────────┘       │
└─────────────────────┬───────────────────────────────────────┘
                      │
        ┌─────────────┼─────────────┐
        │             │             │
        ▼             ▼             ▼
┌──────────────┐ ┌────────────┐ ┌─────────────┐
│ Kubernetes   │ │ ArgoCD     │ │ Dynamic NS  │
│ RoleBindings │ │ RBAC CM    │ │ Discovery   │
│ (per NS)     │ │ (synced)   │ │ (@argocd)   │
└──────────────┘ └────────────┘ └─────────────┘

Role Hierarchy

Application Developer → Platform Operator → System Administrator → Cluster Admin

Each role has minimum necessary permissions with clear escalation paths for exceptional needs.

Namespace Organization

System Namespaces (Infrastructure)

Managed by system administrators only:

  • cert-manager - TLS certificate management
  • cnpg-system - CloudNativePG operator
  • external-dns - DNS automation
  • external-tunnel - External connectivity
  • ingress-nginx - Ingress controller
  • metallb-system - Load balancer
  • secrets-system - External Secrets Operator
  • cilium-secrets - Cilium network policy secrets
  • integrations - External service integrations (1Password, OAuth, etc.)
  • secrets - Derived secrets storage
  • kube-system, kube-public, kube-node-lease - Kubernetes core

Platform Namespaces

Managed by platform operators:

  • argocd - GitOps platform
  • gitea - Git repository
  • metabase - Analytics platform
  • victoria-metrics - Monitoring and metrics

Application Namespaces

Managed by application developers:

  • homelab-docs - Documentation site
  • dev-retroboard, dev-retroboard-api - Development environment
  • prod-retroboard, prod-retroboard-api - Production environment
  • temperature-monitor, temperature-monitor-api - IoT monitoring
  • secret-editor - Secret management UI

RBAC System Architecture

The RBAC system is centralized in system/helmfile/rbac-system:

rbac-system/
├── cluster-roles/          # All ClusterRoles and static bindings
│   └── manifests/
│       └── templates/
│           ├── rbac-app-developer.yaml
│           ├── rbac-platform-operator.yaml
│           ├── rbac-system-admin.yaml
│           └── rbac-cluster-admin.yaml
├── rbac-crds/             # User CRD definition
│   └── templates/
│       └── user-crd.yaml
├── rbac-operator/         # Automated RBAC management
│   ├── files/
│   │   └── rbac-service.py   # Sync logic
│   └── templates/
└── env.yaml               # Cluster admin email configuration

How It Works

  1. Admin creates User CRD specifying email and roles
  2. RBAC Operator discovers namespaces:
    • Reads ClusterRole zengarden.space/namespaces annotations
    • Expands @argocd token to all ArgoCD Application namespaces
  3. Operator creates resources:
    • Kubernetes: RoleBindings in appropriate namespaces
    • ArgoCD: Updates argocd-rbac-cm ConfigMap with user assignments
  4. Automatic reconciliation every 5 minutes + on-demand

User Management

Creating a User

Basic user with single role:

apiVersion: zengarden.space/v1
kind: User
metadata:
  name: john-doe
spec:
  email: [email protected]  # Must match OIDC email
  roles:
    - app-developer
  enabled: true
kubectl apply -f user.yaml

This single resource automatically:

  • Creates Kubernetes RoleBindings in all appropriate namespaces
  • Updates ArgoCD RBAC policy (argocd-rbac-cm)
  • Keeps permissions synchronized across both systems

User with Multiple Roles

apiVersion: zengarden.space/v1
kind: User
metadata:
  name: platform-admin
spec:
  email: [email protected]
  roles:
    - platform-operator
    - system-admin
  enabled: true

This creates RoleBindings in:

  • Application namespaces (from platform-operator role)
  • Platform namespaces (from platform-operator role)
  • System namespaces (from system-admin role)

Note: For ArgoCD, only the highest role (system-admin) is assigned to avoid permission conflicts.

Disabling a User

Disable access while preserving the User resource:

kubectl patch user john-doe --type=merge -p '{"spec":{"enabled":false}}'

Or edit the User directly:

spec:
  enabled: false  # Removes all RoleBindings and ArgoCD access

Managing Users

# List all users
kubectl get users
 
# Get user details and reconciliation status
kubectl get user john-doe -o yaml
 
# Check user status
kubectl get user john-doe -o jsonpath='{.status.conditions[*].message}'
# Output: Successfully created 8 RoleBindings
 
# Update user roles
kubectl edit user john-doe
 
# Delete user (removes all RoleBindings)
kubectl delete user john-doe

ArgoCD RBAC

ArgoCD RBAC is managed automatically by the RBAC Operator via the argocd-rbac-cm ConfigMap. Users assigned to roles in User CRDs automatically receive matching ArgoCD permissions.

Configuration: system/helmfile/rbac-system/rbac-operator/files/rbac-service.py (lines 374-455)

Projects

apps Project

Purpose: Application deployments managed by developers

Configuration: platform/helmfile/argocd/charts/argocd-config/templates/project-apps.yaml

Allowed Namespaces:

  • All application namespaces listed above
  • secrets namespace (for DerivedSecret CRDs only)

Allowed Resources:

  • Core: Service, ConfigMap, Secret, ServiceAccount, PVC
  • Workloads: Deployment, StatefulSet, DaemonSet, Job, CronJob
  • Networking: Ingress, NetworkPolicy
  • Autoscaling: HorizontalPodAutoscaler
  • Policy: PodDisruptionBudget
  • CRDs: ExternalSecret, CNPG Cluster/Database, VMServiceScrape, DerivedSecret
  • RBAC: Role, RoleBinding (for job watchers)

Cluster Resources: None (namespace-scoped only)

Source Repositories: Internal Gitea only

default Project

Purpose: Platform and system components (not managed via ArgoCD projects)

Roles

Application Developer (role:app-developer)

Permissions:

  • ✅ View and sync applications in apps project
  • ✅ View logs and exec into pods
  • ✅ Perform application actions (restart, rollback)
  • ✅ Override sync parameters (for debugging)

Restrictions:

  • ❌ Cannot create or delete applications
  • ❌ Cannot access default project (platform/system apps)
  • ❌ Cannot modify ArgoCD configuration

Use Case: Day-to-day application development, debugging, and deployments

Configuration: Auto-generated by RBAC Operator

# Can work with apps in the 'apps' project only
p, role:app-developer, applications, get, apps/*, allow
p, role:app-developer, applications, sync, apps/*, allow
p, role:app-developer, applications, override, apps/*, allow
p, role:app-developer, applications, action/*, apps/*, allow
p, role:app-developer, logs, get, apps/*, allow
p, role:app-developer, exec, create, apps/*, allow

Note: Users should only be assigned ONE ArgoCD role. The operator automatically assigns the highest role from the User CRD.

Platform Operator (role:platform-operator)

Permissions:

  • ✅ All Application Developer permissions
  • ✅ Full management of apps project applications
  • ✅ Create and delete applications in apps project
  • ✅ View default project applications (read-only)
  • ✅ Manage ArgoCD projects and repositories
  • ✅ View and troubleshoot platform services

Restrictions:

  • ❌ Cannot sync or delete default project apps
  • ❌ Cannot delete critical projects (default, apps)
  • ❌ Limited cluster-level access

Use Case: Managing application lifecycle, onboarding new services, platform troubleshooting

Configuration: Auto-generated by RBAC Operator

# Full access to apps project
p, role:platform-operator, applications, *, apps/*, allow
p, role:platform-operator, logs, get, */*, allow
p, role:platform-operator, exec, create, */*, allow
 
# Can view default project apps (but not modify)
p, role:platform-operator, applications, get, default/*, allow
 
# Can manage projects and repositories
p, role:platform-operator, projects, get, *, allow
p, role:platform-operator, projects, create, *, allow
p, role:platform-operator, projects, update, *, allow
p, role:platform-operator, repositories, get, *, allow
p, role:platform-operator, repositories, create, *, allow
p, role:platform-operator, repositories, update, *, allow

System Administrator (role:system-admin)

Permissions:

  • ✅ Full access to all ArgoCD projects
  • ✅ Manage all applications including platform/system
  • ✅ Manage ArgoCD configuration (projects, repos, certificates, GPG keys)
  • ✅ Account management

Restrictions:

  • ❌ Cannot delete Kubernetes clusters from ArgoCD

Use Case: Infrastructure management, system upgrades, security patching

Configuration: Auto-generated by RBAC Operator

# Full access to all projects and ArgoCD management
p, role:system-admin, applications, *, */*, allow
p, role:system-admin, logs, *, */*, allow
p, role:system-admin, exec, *, */*, allow
p, role:system-admin, projects, *, *, allow
p, role:system-admin, repositories, *, *, allow
p, role:system-admin, certificates, *, *, allow
p, role:system-admin, gpgkeys, *, *, allow
p, role:system-admin, accounts, get, *, allow
p, role:system-admin, accounts, update, *, allow

Cluster Admin (role:cluster-admin)

Permissions:

  • ✅ Unrestricted access to everything in ArgoCD

Restrictions: None

Use Case: Break-glass only, emergency access, should be audited

Configuration: Auto-generated by RBAC Operator

# Break-glass full access
p, role:cluster-admin, *, *, *, allow

Kubernetes RBAC

ClusterRoles

homelab:app-developer

Scope: Application namespaces only (via RoleBindings)

Permissions:

  • Read pods, logs, events
  • Exec into pods for debugging
  • Read deployments, services, configmaps, secrets
  • Read ingresses, jobs, cronjobs, HPA

Restrictions:

  • No write access to any resources (GitOps only)
  • No access to platform or system namespaces

Configuration: system/helmfile/rbac-system/cluster-roles/manifests/templates/rbac-app-developer.yaml

ClusterRole Annotations:

annotations:
  zengarden.space/role: app-developer
  zengarden.space/namespaces: "@argocd"

The @argocd token automatically expands to all namespaces where ArgoCD Applications are deployed. The RBAC Operator creates RoleBindings dynamically.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: homelab:app-developer
rules:
  # View and debug pods
  - apiGroups: [""]
    resources: ["pods", "pods/log", "pods/status"]
    verbs: ["get", "list", "watch"]
 
  # Exec into pods for debugging
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]
 
  # View deployments, services, configmaps
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
    verbs: ["get", "list", "watch"]
 
  - apiGroups: [""]
    resources: ["services", "endpoints", "configmaps"]
    verbs: ["get", "list", "watch"]
 
  # View secrets (read-only, for troubleshooting)
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]

homelab:platform-operator

Scope: Application + Platform namespaces (via RoleBindings)

Permissions:

  • Full CRUD on application workloads (Deployments, StatefulSets, etc.)
  • Full CRUD on services, configmaps, ingresses
  • Read-only access to secrets (write via secret-editor tool only)
  • Manage ArgoCD Applications and AppProjects
  • View CRDs and cluster resources

Restrictions:

  • Cannot write secrets directly (use secret-editor tool)
  • Cannot modify system namespaces
  • Limited cluster-level access

Configuration: system/helmfile/rbac-system/cluster-roles/manifests/templates/rbac-platform-operator.yaml

ClusterRole Annotations:

annotations:
  zengarden.space/role: platform-operator
  zengarden.space/namespaces: "@argocd,argocd,gitea,metabase,victoria-metrics"

Combines dynamic discovery (@argocd = all application namespaces) with static platform namespaces.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: homelab:platform-operator
rules:
  # Full access to application resources (excluding secrets write)
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
    verbs: ["*"]
 
  - apiGroups: [""]
    resources: ["pods", "pods/log", "pods/status", "pods/exec"]
    verbs: ["*"]
 
  # Read secrets, limited write via secret-editor only
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]
 
  # Can manage ArgoCD Applications
  - apiGroups: ["argoproj.io"]
    resources: ["applications", "applicationsets", "appprojects"]
    verbs: ["*"]

homelab:system-admin

Scope: Multiple namespaces (via RoleBindings in application, platform, and system namespaces)

Permissions:

  • Full access to all namespaced resources
  • Manage CRDs, ClusterRoles (limited), namespaces, nodes
  • Full access to operator CRDs (cert-manager, ESO, CNPG, MetalLB, Cilium)
  • Full access to integrations namespace (external credentials)

Restrictions:

  • Cannot delete critical ClusterRoles or ClusterRoleBindings

Configuration: system/helmfile/rbac-system/cluster-roles/manifests/templates/rbac-system-admin.yaml

ClusterRole Annotations:

annotations:
  zengarden.space/role: system-admin
  zengarden.space/namespaces: "@argocd,argocd,gitea,metabase,victoria-metrics,cert-manager,secrets-system,metallb-system,ingress-nginx,external-dns,external-tunnel,cnpg-system,cilium-secrets,integrations,secrets"

Includes all application (@argocd), platform, and system namespaces.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: homelab:system-admin
rules:
  # Full access to all namespaced resources
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
 
  # Cert-manager resources
  - apiGroups: ["cert-manager.io"]
    resources: ["*"]
    verbs: ["*"]
 
  # External Secrets Operator
  - apiGroups: ["external-secrets.io"]
    resources: ["*"]
    verbs: ["*"]
 
  # CNPG (CloudNativePG)
  - apiGroups: ["postgresql.cnpg.io"]
    resources: ["*"]
    verbs: ["*"]
 
  # MetalLB
  - apiGroups: ["metallb.io"]
    resources: ["*"]
    verbs: ["*"]
 
  # Cilium Network Policies
  - apiGroups: ["cilium.io"]
    resources: ["*"]
    verbs: ["*"]

homelab:cluster-admin

Scope: Cluster-wide (ClusterRoleBinding)

Permissions:

  • Unrestricted access to everything
  • Non-resource URLs (API server endpoints)

Restrictions: None (break-glass role)

Configuration: system/helmfile/rbac-system/cluster-roles/manifests/templates/rbac-cluster-admin.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: homelab:cluster-admin
  labels:
    rbac.homelab/break-glass: "true"
rules:
  # Unrestricted access to everything
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
 
  # Non-resource URLs (API server endpoints)
  - nonResourceURLs: ["*"]
    verbs: ["*"]

Secret Access Strategy

integrations Namespace

Purpose: External service credentials (1Password, Google OAuth, Cloudflare API tokens)

Access Control:

  • Application Developer: No access
  • Platform Operator: Read-only (for troubleshooting)
  • System Administrator: Full access
  • Cluster Admin: Full access

secrets Namespace

Purpose: Derived secrets for applications (DerivedSecrets operator)

Access Control:

  • Applications: Via ServiceAccounts only
  • Application Developer: No direct access (view via ArgoCD)
  • Platform Operator: Read-only
  • System Administrator: Full access
  • Human users: Via secret-editor tool only

Application Namespaces

Purpose: Application-specific secrets (database credentials, API keys)

Access Control:

  • Application Developer: Read-only (for troubleshooting)
  • Platform Operator: Read-only (write via secret-editor)
  • System Administrator: Full access

User Assignment

Unified User Management

Single Source of Truth: User CRDs automatically manage both Kubernetes and ArgoCD access.

Create a User CRD:

apiVersion: zengarden.space/v1
kind: User
metadata:
  name: john-doe
spec:
  email: [email protected]
  roles:
    - app-developer    # Can have multiple roles for K8s namespaces
  enabled: true
kubectl apply -f user.yaml

This automatically:

  1. Creates Kubernetes RoleBindings in appropriate namespaces
  2. Updates ArgoCD argocd-rbac-cm ConfigMap
  3. Grants matching permissions in both systems

Managing Users

# List all users
kubectl get users
 
# Get user details and status
kubectl get user john-doe -o yaml
 
# Disable user (removes all access)
kubectl patch user john-doe --type=merge -p '{"spec":{"enabled":false}}'
 
# Update user roles
kubectl edit user john-doe
 
# Delete user (removes all RoleBindings)
kubectl delete user john-doe

Cluster Admin Assignment

The cluster admin email is configured in system/helmfile/rbac-system/env.yaml:

clusterAdminEmail: [email protected]

This creates a static ClusterRoleBinding with unrestricted cluster access.

See RBAC Operator for detailed documentation on User CRD management.

Deployment

Deploy RBAC System

# Deploy system components (includes RBAC system)
cd /Users/oleksiyp/dev/homelab/system/helmfile
helmfile sync

This deploys:

  1. ClusterRoles with namespace annotations
  2. User CRD definition
  3. RBAC Operator for automated management

No manual namespace labeling required - the operator handles everything automatically.

Move Existing Apps to apps Project

# Move each application to the apps project
argocd app set hemelab.dev.retroboard --project apps --grpc-web
argocd app set hemelab.dev.retroboard-api --project apps --grpc-web
argocd app set hemelab.prod.retroboard --project apps --grpc-web
argocd app set hemelab.prod.retroboard-api --project apps --grpc-web
 
# Verify project assignment
argocd app list --grpc-web

Monitoring

Check Operator Status

# Check operator pod
kubectl get pods -n rbac-system -l app.kubernetes.io/name=rbac-operator
 
# Expected output:
# NAME              READY   STATUS    RESTARTS   AGE
# rbac-operator-0   2/2     Running   0          5m

View Operator Logs

# View RBAC service logs (main reconciliation logic)
kubectl logs -n rbac-system -l app.kubernetes.io/name=rbac-operator -c rbac-service -f
 
# View shell-operator logs (watch events)
kubectl logs -n rbac-system -l app.kubernetes.io/name=rbac-operator -c operator -f

Check Reconciliation Status

# Check all users
kubectl get users
 
# Example output:
# NAME         EMAIL                   ROLES                  ENABLED   AGE
# john-doe     [email protected]    ["app-developer"]      true      10m
# admin-user   [email protected]       ["system-admin"]       true      5m
 
# Check user status details
kubectl get user john-doe -o yaml | grep -A 10 status:

Force Reconciliation

To trigger immediate reconciliation:

# Restart the operator
kubectl rollout restart statefulset/rbac-operator -n rbac-system
 
# Or add annotation to user
kubectl annotate user john-doe reconcile=now-$(date +%s)

Verification

Verify ArgoCD Projects

# List all projects
argocd proj list --grpc-web
 
# Get apps project details
argocd proj get apps --grpc-web
 
# List applications by project
argocd app list --project apps --grpc-web

Verify ArgoCD RBAC

# Test permissions as different roles
argocd account can-i sync applications 'apps/*'
argocd account can-i delete applications 'default/*'
argocd account can-i create projects '*'

Verify Kubernetes RBAC

# Check User CRDs
kubectl get users
 
# Check user status
kubectl get user john-doe -o yaml
 
# Check operator-managed RoleBindings
kubectl get rolebindings -A -l app.kubernetes.io/managed-by=rbac-operator
 
# Check ClusterRoles
kubectl get clusterroles | grep homelab
 
# Test permissions (as specific user via impersonation)
kubectl auth can-i get pods -n dev-retroboard [email protected]
kubectl auth can-i delete secrets -n integrations [email protected]
kubectl auth can-i create clusterroles [email protected]

Verify ArgoCD RBAC ConfigMap

# Check ArgoCD RBAC ConfigMap
kubectl get configmap argocd-rbac-cm -n argocd -o yaml
 
# Verify user assignments
kubectl get configmap argocd-rbac-cm -n argocd -o jsonpath='{.data.policy\.csv}' | grep "^g,"
 
# Check scopes configuration
kubectl get configmap argocd-rbac-cm -n argocd -o jsonpath='{.data.scopes}'

Best Practices

Principle of Least Privilege

Users should be assigned the minimum role needed for their daily work. Escalation should be temporary and audited.

GitOps First

All changes should go through Git commits and ArgoCD sync. Direct kubectl access is for emergencies and debugging only.

Secret Segregation

  • Never commit secrets to Git
  • Use External Secrets Operator for external credentials
  • Use DerivedSecrets for application secrets
  • Access secrets via secret-editor tool, not direct kubectl

Regular Audits

Review role assignments and permissions quarterly:

# List all users
kubectl get users
 
# Check who has elevated roles
kubectl get users -o jsonpath='{range .items[?(@.spec.roles[*]=="system-admin")]}{.metadata.name}: {.spec.email}{"\n"}{end}'
 
# Review RoleBindings
kubectl get rolebindings -A -l app.kubernetes.io/managed-by=rbac-operator

Questions to answer:

  • Are users still in the appropriate role?
  • Are there unused or disabled users?
  • Are RBAC policies still aligned with responsibilities?

User Naming Convention

Use lowercase, hyphenated names matching email prefix:

# Good
name: john-doe
email: [email protected]
 
# Bad
name: JohnDoe
email: [email protected]

Role Assignment Strategy

Follow the principle of least privilege:

  1. Start with app-developer for most users
  2. Promote to platform-operator only when needed
  3. Reserve system-admin for infrastructure team
  4. Never assign cluster-admin via User CRD (use static ClusterRoleBinding)

Break-glass Process

Cluster Admin access should be:

  • Time-limited (request → approve → revoke)
  • Audited (all actions logged)
  • Justified (documented reason for escalation)

Troubleshooting

RoleBindings Not Created

Check User status:

kubectl get user john-doe -o yaml

Look for error in status.conditions:

status:
  conditions:
  - type: Ready
    status: "False"
    reason: ReconciliationFailed
    message: "ClusterRole homelab:app-developer not found"

Verify ClusterRoles exist:

kubectl get clusterrole homelab:app-developer
kubectl get clusterrole homelab:platform-operator
kubectl get clusterrole homelab:system-admin

If missing, deploy RBAC system:

cd ~/dev/homelab/system/helmfile
helmfile sync

Verify ArgoCD Applications exist:

kubectl get applications -A

Operator Not Starting

Check pod status:

kubectl get pods -n rbac-system -l app.kubernetes.io/name=rbac-operator

Check logs:

# Shell-operator logs
kubectl logs -n rbac-system rbac-operator-0 -c operator
 
# Python service logs
kubectl logs -n rbac-system rbac-operator-0 -c rbac-service

Common issues:

  • Missing ServiceAccount permissions
  • Failed to install Python dependencies
  • Cannot connect to Kubernetes API

”Permission Denied” in ArgoCD

Check:

  1. User CRD exists and is enabled
  2. ArgoCD RBAC ConfigMap has user assignment (kubectl get cm argocd-rbac-cm -n argocd)
  3. Application’s project assignment (argocd app get <app> --grpc-web)
  4. User email matches OIDC token (userIdKey: email in dex config)

Debug:

# Check current user
argocd account get-user-info --grpc-web
 
# Check what user can do
argocd account can-i sync applications 'apps/*' --grpc-web
 
# Verify user in ArgoCD RBAC policy
kubectl get cm argocd-rbac-cm -n argocd -o jsonpath='{.data.policy\.csv}' | grep [email protected]

“Forbidden” in kubectl

Check:

  1. RoleBinding exists in the target namespace
  2. User is in the subjects list
  3. ClusterRole has the required permissions

Debug:

# Check if user can perform action
kubectl auth can-i get pods -n dev-retroboard [email protected]
 
# List user's permissions
kubectl auth can-i --list [email protected]
 
# Describe RoleBinding
kubectl describe rolebinding homelab:app-developer -n dev-retroboard

Applications Not Syncing

Check:

  1. Application is in the correct project (argocd app get <app>)
  2. Project allows the destination namespace
  3. Project allows the source repository
  4. User has sync permission for that project

Debug:

# Get application details
argocd app get hemelab.dev.retroboard --grpc-web
 
# Get project details
argocd proj get apps --grpc-web
 
# Check sync errors
argocd app sync hemelab.dev.retroboard --dry-run --grpc-web

Project Namespace Not Allowed

Error: application destination spec is not permitted in project

Solution: Add namespace to project destinations in project-apps.yaml:

destinations:
  - namespace: new-app-namespace
    server: {{ .Values.application.destinationServer }}

Then redeploy:

cd /Users/oleksiyp/dev/homelab/platform/helmfile/argocd
helmfile sync

Security Considerations

RBAC Is Part of Defense-in-Depth

RBAC is Layer 5 in the homelab’s security model (see Security). It works together with:

  • Layer 1: Network segmentation (firewall)
  • Layer 2: Cluster network (Cilium, eBPF)
  • Layer 3: Ingress security (TLS, WAF)
  • Layer 4: Network policies (pod-to-pod restrictions)
  • Layer 5: RBAC (user permissions)
  • Layer 6: Application security (non-root containers)
  • Layer 7: Audit & monitoring (logging, alerting)

Audit Logging

All RBAC decisions are logged to Kubernetes audit logs:

# What's logged:
# - Who (user, service account)
# - What (resource, verb)
# - When (timestamp)
# - Result (allowed, denied)

Review audit logs regularly:

# Find denied requests
kubectl logs -n kube-system -l component=kube-apiserver | grep 'Forbidden'
 
# Find specific user's actions
kubectl logs -n kube-system -l component=kube-apiserver | grep '[email protected]'

Google OAuth Integration

Users authenticate via Google OAuth (configured in K3s):

kube-apiserver-arg:
  - "oidc-issuer-url=https://accounts.google.com"
  - "oidc-client-id=<google-client-id>.apps.googleusercontent.com"
  - "oidc-username-claim=email"
  - "oidc-groups-claim=groups"

ArgoCD also uses Google OAuth with OIDC (configured in platform/helmfile/argocd/helmfile.yaml.gotmpl):

dex.config: |
  connectors:
    - type: oidc
      id: google
      name: Google
      config:
        issuer: https://accounts.google.com
        clientID: $argocd-google-oauth:clientId
        clientSecret: $argocd-google-oauth:clientSecret
        # Use email claim as username for RBAC matching
        userIdKey: email

The userIdKey: email configuration ensures ArgoCD uses the user’s email address (not numeric Google ID) for RBAC policy matching.

RBAC System Components

The unified RBAC system (system/helmfile/rbac-system) consists of:

1. ClusterRoles (cluster-roles/)

  • Defines all 4 ClusterRoles with permissions
  • Uses annotations to specify target namespaces
  • Supports @argocd token for dynamic namespace discovery
  • No manual namespace labeling required

2. User CRD (rbac-crds/)

  • Custom resource for defining users and their roles
  • Single source of truth for access control
  • Supports multiple roles per user (for Kubernetes)
  • Enable/disable user access with enabled field

3. RBAC Operator (rbac-operator/)

  • Watches User CRDs, ClusterRoles, and ArgoCD Applications
  • Automatically creates/updates Kubernetes RoleBindings
  • Syncs ArgoCD RBAC ConfigMap (argocd-rbac-cm)
  • Reconciles every 5 minutes + on-demand
  • Assigns highest role to users in ArgoCD (avoiding conflicts)
  • Security - Full security architecture and defense-in-depth
  • Deployment - Initial setup instructions
  • Monitoring - Security event detection and alerting
  • Maintenance - Regular RBAC audits and system maintenance

Conclusion

The unified RBAC system provides:

  1. Single Source of Truth: User CRDs define all access across Kubernetes and ArgoCD
  2. Full Automation: RoleBindings and ArgoCD policies managed automatically
  3. Dynamic Discovery: Namespaces discovered from ArgoCD Applications (no manual labeling)
  4. Granular Permissions: Four-tier role model with clear escalation paths
  5. Namespace Segregation: Application, platform, and system isolation
  6. Audit Trail: All changes tracked via User status and operator logs
  7. GitOps Friendly: User CRDs can be version controlled and managed via GitOps

Key Benefits:

  • Reduces operational overhead: Eliminates manual RoleBinding configuration
  • Ensures consistency: Same permissions across all matching namespaces
  • Scales automatically: New applications get RoleBindings automatically
  • Maintains security: Principle of least privilege enforced through role hierarchy

RBAC is layered with other security controls to provide defense-in-depth protection for the homelab infrastructure.

RBAC is not just about permissions—it’s about accountability, auditability, and limiting blast radius through automation.

OverviewMaintenance