April 3, 2026 argocd kubernetes aws gitops debugging

ArgoCD, Private GitHub Repos, and the AWS SSO Kubeconfig Problem

Two problems blocked our ArgoCD bootstrap: AWS SSO's login session config prevented kubectl from authenticating, and ArgoCD couldn't pull our private mypie-infra repo. Here is how we solved both.

After ArgoCD was installed via Helm, we tried to apply our root-app.yaml to bootstrap the App-of-Apps pattern. Two separate issues came up in sequence — one about kubectl itself not being able to talk to the cluster, and one about ArgoCD not being able to read from our private GitHub repository.

Problem 1: AWS SSO Kubeconfig Exec Plugin Failure

We use AWS IAM Identity Center (SSO) for authentication. The login flow is aws login (not aws sso login — Identity Center uses a different command). This command stores a short-lived token in ~/.aws/login/cache/*.json.

Our kubeconfig used the aws eks get-token exec plugin, which is the standard EKS setup:

users:
- name: arn:aws:eks:eu-central-1:<account-id>:cluster/mypie-eks-staging
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: aws
      args:
      - eks
      - get-token
      - --cluster-name
      - mypie-eks-staging
      - --region
      - eu-central-1

Every kubectl command was failing:

error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1beta1"

And even after installing the right aws cli version, aws eks get-token was returning an expired/invalid token. The reason: aws login stores the session under a [login_session] profile in ~/.aws/config, and this profile takes precedence over environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN) when the aws CLI is invoked by the exec plugin.

In other words: even if you exported your credentials into the shell environment, every time kubectl called aws eks get-token, the AWS CLI read ~/.aws/config, found the [login_session] profile, used that (which may be expired or point to a different assume-role chain), and ignored your environment variables.

The Fix: Static Token Patch

We bypassed the exec plugin entirely by extracting the token ourselves and patching the kubeconfig with a static bearer token. This script ran before every kubectl session:

import json, glob, subprocess, yaml, re

# Step 1: Read the cached access token from aws login
cache_files = glob.glob('/home/<user>/.aws/login/cache/*.json')
credentials = json.load(open(cache_files[0]))['accessToken']

env = {
    'AWS_ACCESS_KEY_ID':     credentials['accessKeyId'],
    'AWS_SECRET_ACCESS_KEY': credentials['secretAccessKey'],
    'AWS_SESSION_TOKEN':     credentials['sessionToken'],
    'AWS_DEFAULT_REGION':    'eu-central-1',
    # Crucially: ignore the ~/.aws/config to prevent login_session override
    'AWS_CONFIG_FILE':       '/dev/null',
    'HOME': '/home/<user>',
    'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
}

# Step 2: Call get-token with credentials from env only (no config file)
result = subprocess.run(
    ['aws', 'eks', 'get-token',
     '--cluster-name', 'mypie-eks-staging',
     '--region', 'eu-central-1'],
    env=env, capture_output=True, text=True
)

token = json.loads(result.stdout)['status']['token']

# Step 3: Patch kubeconfig — replace exec block with static token
kube_data = yaml.safe_load(open('/home/<user>/.kube/config').read())
for user in kube_data.get('users', []):
    if 'staging' in user.get('name', ''):
        user['user'] = {'token': token}

with open('/home/<user>/.kube/config', 'w') as f:
    yaml.dump(kube_data, f, default_flow_style=False)

print("Kubeconfig patched with static token")

The key insight is AWS_CONFIG_FILE=/dev/null — this tells the AWS CLI in that subprocess to use no config file, so it reads credentials exclusively from environment variables. After patching, kubectl commands worked for the ~15-minute lifetime of the SSO token.

For production use: Set up a dedicated kubeconfig for CI/CD using an IRSA-based service account or a long-lived IAM user with EKS API access, rather than relying on SSO tokens.

Problem 2: ArgoCD “Repository Not Found”

With kubectl working, we applied the root app:

kubectl apply -f k8s/argocd/apps/root-app.yaml

ArgoCD accepted the resource but all applications immediately showed an error in the ArgoCD UI and via kubectl get applications:

ComparisonError: rpc error: code = Unknown
desc = repository not found or not accessible

The mypie-infra repository is private on GitHub. ArgoCD needed credentials to clone it.

Creating the Repository Secret

ArgoCD uses Kubernetes secrets with a specific label to discover repository credentials. We created the secret directly (we later templated this into a sealed-secret for GitOps safety):

kubectl create secret generic argocd-repo-secret \
  --namespace argocd \
  --from-literal=type=git \
  --from-literal=url=https://github.com/<org>/mypie-infra \
  --from-literal=username=<github-username> \
  --from-literal=password=<github-personal-access-token>

# The label is required — without it ArgoCD won't use the secret
kubectl label secret argocd-repo-secret \
  -n argocd \
  argocd.argoproj.io/secret-type=repository

The PAT needs the repo scope (read access to private repositories). After applying the label, ArgoCD picked up the secret within a few seconds and retried the repo connection. The error cleared and applications transitioned to their correct sync states.

Alternative: SSH Key

If you prefer SSH over HTTPS:

# Generate a deploy key (no passphrase for automated use)
ssh-keygen -t ed25519 -f /tmp/argocd-deploy-key -N ""

# Add the public key as a deploy key in GitHub repo settings
cat /tmp/argocd-deploy-key.pub

# Create the ArgoCD secret
kubectl create secret generic argocd-repo-secret \
  --namespace argocd \
  --from-literal=type=git \
  --from-literal=url=git@github.com:<org>/mypie-infra.git \
  --from-file=sshPrivateKey=/tmp/argocd-deploy-key

kubectl label secret argocd-repo-secret \
  -n argocd \
  argocd.argoproj.io/secret-type=repository

The Token Expiry Problem

The SSO token from aws login expires every ~15 minutes. This means the kubeconfig patch needs to be re-run each session. A practical approach for a workstation:

# Add to .bashrc or .zshrc
function kube-refresh-staging() {
  python3 /path/to/patch-kubeconfig.py
  echo "Kubeconfig refreshed"
}

# Or as an alias that auto-refreshes before any kubectl command
# (more complex, requires shell hooks)

For CI/CD environments, the proper solution is a Kubernetes service account with a long-lived token or IRSA with an assume-role that has eks:DescribeCluster + in-cluster RBAC bindings — avoiding SSO entirely.