Skip to Content
Tuan Tran
GitOps & Progressive Delivery

Lesson 09 — Infrastructure GitOps & Platform Bootstrap

Terraform, Argo CD App-of-Apps, ApplicationSet, controller ownership.

Learning objectives

  • Separate Day-0, Day-1, and Day-2 platform responsibilities: Break the bootstrap chicken-and-egg cycle by provisioning GKE and IAM with Terraform, seeding Argo CD safely, and handing ongoing reconciliation to GitOps.
  • Enforce single authoritative ownership across control planes: Define clear boundaries between Terraform, Argo CD, Config Sync, Config Connector, and ESO to prevent drift tug-of-war and field-management conflicts.
  • Structure multi-component delivery with App-of-Apps and ApplicationSet: Compare bootstrapping patterns and centralized versus per-cluster topologies, protecting root inputs and repository credentials.
  • Sequence dependencies with sync waves and readiness gates: Distinguish CRD registration from controller readiness, ordering namespaces, CRDs, operators, and workloads without mistaking sync waves for external health checks.
  • Plan disaster recovery and safe cluster decommissioning: Protect stateful data and shared CRDs from unintended pruning, maintain break-glass bootstrap procedures, and tear down external resources before destroying cluster infrastructure.

Day 0, Day 1, and Day 2

Day 0: Terraform provisions the VPC, subnets, GKE cluster, IAM, and other infrastructure dependencies. Day 1: Bootstrap Argo CD, controllers, and CRDs before deploying workloads. Day 2: Handle upgrades, drift reconciliation, incident response, backups, and teardown.

flowchart TD
  A[Terraform / GKE] --> B[Bootstrap Argo CD]
  B --> C[Root Application]
  C --> D[Platform Controllers]
  D --> E[Applications]

The chicken-and-egg problem

Argo CD cannot sync itself before it exists in the cluster. Use Helm or a pipeline for the initial installation, then hand ownership over to GitOps. Do not remove bootstrap dependencies until you confirm that Argo CD manages itself reliably.

See Helm, Kustomize, and Argo CD for manifest generation, Helm release history, and the delivery handoff to Argo CD.

For environment layout, image digest promotion, and production rollback by Git revert, see GitOps Repository Design and Environment Promotion.

Reconciliation ownership

Each field should have one authoritative owner. Terraform owns the cloud resources it provisions. Argo CD owns the Kubernetes desired state. Controllers such as Config Connector manage cloud resources declared through custom resources (CRs). Avoid having Terraform and Config Connector modify the same resource.

Sync waves

argocd.argoproj.io/sync-wave determines the sync order for resources within a single Application. It does not guarantee readiness across separate Applications.

Production checklist

  • Pin Helm chart versions, images, and controller versions.
  • Keep repositories with elevated privileges separate from application repositories.
  • Define rollback procedures and break-glass access.
  • Test disaster recovery from a fresh cluster.
  • During teardown, account for finalizers, PVCs, and data outside the cluster.

Official: Argo CD · Config Connector.

Deep dive: bootstrapping a cluster without circular dependencies

Requiring Argo CD to install itself before anything has bootstrapped it is an anti-pattern. Separate the process into three phases:

  1. Provision: Terraform creates the VPC, subnets, IAM, GKE, and baseline DNS and KMS configuration.
  2. Bootstrap: A pipeline or controlled procedure installs Argo CD and provisions the minimum credentials and repository access.
  3. Reconcile: Argo CD installs controllers, monitoring, secrets integrations, and workloads from Git.
flowchart TD
 TF[Terraform] --> GKE[GKE Cluster]
 GKE --> Boot[Bootstrap controller]
 Boot --> Argo[Argo CD root app]
 Argo --> C[CRDs + operators]
 C --> P[Platform workloads]
 P --> A[Application workloads]

Dependency management

If you apply an ExternalSecret before its CRD exists, the Kubernetes API returns a NoMatch or UnknownKind error. If the External Secrets Operator (ESO) CRD exists but its controller is not ready, the API may accept the object without the controller reconciling it. Distinguish API registration from controller readiness. Sync waves can order operations, but they do not replace readiness checks or retry logic.

Example ownership matrix

ResourceAuthoritative ownerAvoid sharing ownership with
GKE control plane/VPCTerraformConfig Connector on the same fields
Platform namespace/policiesPlatform GitOpsApplication repo
Application image and rollout specApplication Argo CDManual kubectl patch
ReplicasHPA (when used)Static replica count in Git
ExternalSecret target SecretESOManually authored Secret

Bootstrap recovery runbook

First check the Terraform state and GKE credentials. Install or restore the pinned version of Argo CD, then verify repository connectivity and destination credentials. Sync CRDs and operators in dependency order, and verify workload health. Do not use restored GitOps configuration to delete production storage before the disaster recovery process has established who owns the stateful data.

Practice: Create a kind cluster, install Argo CD with a pinned Helm chart version, and create a root Application that manages only one lab namespace. Pause Argo CD, edit a live ConfigMap, restart Argo CD, and watch it reconcile the drift.

Deep dive: GitOps on Day 0, Day 1, and Day 2

A platform's GitOps model often has multiple control planes. Terraform manages cloud infrastructure; Argo CD or Config Sync applies Kubernetes desired state; and Config Connector can manage Google Cloud resources through CRDs. The question is not which tool is best, but which tool owns each resource and in what order to bootstrap them.

Breaking the chicken-and-egg cycle

Before Argo CD can install platform controllers, you need a Kubernetes cluster, Argo CD instances, and credentials. You cannot put all of Day 0 under Argo CD's control before Argo CD exists. One possible sequence:

flowchart TD
  subgraph Day0["Day-0: Provisioning"]
    T1["Terraform"] --> T2["VPC / subnets / IAM"]
    T2 --> T3["GKE cluster"]
    T3 --> T4["Bootstrap access"]
  end
  subgraph Day1["Day-1: Bootstrap"]
    B1["Bootstrap Helm / Kustomize"] --> B2["Argo CD + core policies + CRDs"]
  end
  subgraph Day2["Day-2: Operations"]
    R1["Argo CD reconciles platform components and applications"]
  end
  T4 --> B1
  B2 --> R1

Use Terraform or a controlled bootstrap script to install Argo CD initially. When handing off to GitOps, avoid letting Terraform continue to control Kubernetes resources that Argo CD also owns. If Terraform must keep managing the Helm release, document that ownership and do not let Argo CD manage the same release state.

Model dependencies as a clear directed acyclic graph

flowchart TD
  A[GKE + IAM] --> B[Argo CD bootstrap]
  B --> C[Network/Storage baseline]
  C --> D[CRDs and Operators]
  D --> E[External Secrets / Certificates]
  E --> F[Namespaces and Policies]
  F --> G[Application workloads]
  G --> H[End-to-end smoke tests]

Sync waves order operations by phase and wave, but they do not make external dependencies ready. For example, creating an ExternalSecret before the ESO controller has Secret Manager permissions can cause the workload's Secret mount to fail.

Resources that need deletion safeguards

Argo CD's automated pruning can delete resources that are no longer in Git. For StatefulSets and PVCs, Google Cloud resources managed by Config Connector, and CRDs shared by multiple Applications, deletion policy directly affects the data lifecycle. Set prune: false for components critical to bootstrap or use a separate decommissioning process where appropriate. Neither is an absolute safeguard. In particular, account for cascading deletion and finalizers.

Disaster recovery for the GitOps control plane

  1. Restore the GKE infrastructure with Terraform.
  2. Create bootstrap identities and repository credentials through a secure recovery channel.
  3. Install the approved version of Argo CD Core and any required components.
  4. Sync platform prerequisites first: CRDs, admission controls, CSI, and ESO.
  5. Restore persistent data from backups; Git cannot supply it.
  6. Bring applications online and verify APIs, metrics, and transactions from outside the cluster.

A repository containing every desired YAML manifest is not a database backup. Account for secret encryption, Git history, and external resources.

Production GitOps ownership table

ResourceOwnerFailure if duplicated
Google Cloud VPC/IAMTerraformDrift tug-of-war / permission changes
Kubernetes CRDsPlatform GitOpsUpgrade compatibility issues
Application Deployment imageApp GitOpsUnexpected rollback/image churn
Deployment replicasHPA if enabledScaling conflict
Database recordsDatabase system + backupsIrrecoverable state if deleted
Kubernetes Pod statusKubelet/controllersDo not set by Git

Lab

In a kind or GKE test cluster, bootstrap Argo CD from manifests or Helm. Use App-of-Apps or ApplicationSet to manage namespaces, operators, and apps. Try syncing a CR before its CRD exists, observe the error, and fix the dependency chain. Simulate drift on a ConfigMap and record its owner and the rollback procedure. Do not test pruning on a production database.

Sources: Argo CD cluster bootstrapping, Sync waves, Terraform state.

Production walkthrough for a new GKE cluster

A Git pull request can initiate the entire process. An authenticated runner still needs to apply the initial infrastructure plan and install or connect the first reconciler. For a new production cluster, record an observable outcome for each stage before moving on.

StageWorkEvidence
Day 0 — provisionApply the approved Terraform plan for the GCP project, VPC, IAM, and GKE. Keep the state backend available independently of the workload cluster.State is readable by the recovery runner, the Kubernetes API is reachable, and the bootstrap identity can perform its intended operations.
Day 1 — seed GitOpsInstall Argo CD or register the workload cluster with an existing management instance. Configure repository credentials, destination access, and an AppProject.Argo CD can fetch the pinned source and authenticate to the intended destination.
Day 1 — establish the platformSync namespaces, policies, monitoring, CRDs, operators, and their required identities in dependency order.Discovery, controller readiness, and a representative reconciliation all succeed.
Day 1 — enroll workloadsEnable the ApplicationSet generator inputs or directory of child Applications after prerequisites pass.Generated Applications target the expected cluster and workloads pass smoke and compliance checks.
Day 2 — operateUpgrade GKE and add-ons, rotate credentials, handle drift and incidents, restore data, and decommission clusters.Release, rollback, and recovery procedures have been exercised with the identities and versions they will use.

Terraform continues to own infrastructure changes such as GKE upgrades during Day 2. Argo CD and operators often participate in both bootstrapping and ongoing operations. Provision GKE resources with Terraform

flowchart TD
  PR[New cluster pull request] --> PLAN[Terraform plan]
  PLAN --> REVIEW{Approved?}
  REVIEW -->|Yes| APPLY[Apply infrastructure]
  REVIEW -->|No| PR
  APPLY --> ACCESS[Verify cluster and bootstrap access]
  ACCESS --> TOPOLOGY{Delivery topology}
  TOPOLOGY -->|Centralized| REGISTER[Register workload cluster]
  TOPOLOGY -->|Per cluster| INSTALL[Install local Argo CD]
  REGISTER --> PLATFORM[Generate or sync platform Applications]
  INSTALL --> PLATFORM
  PLATFORM --> GATES[Verify CRDs, controllers and platform outputs]
  GATES --> WORKLOADS[Enroll workload Applications]
  WORKLOADS --> VERIFY[Verify health, compliance and smoke tests]

Provisioning and reconciliation boundaries

ConcernTerraformConfig Connector
Desired configurationHCL, modules, and provider configurationKubernetes CR specs, usually delivered from Git
Tracked stateBackend state maps Terraform addresses to provider resourcesKubernetes CRs and status plus the live Google Cloud API
ExecutionPlan, review, and apply through a workflowController reconciliation with resource-specific behavior
DependenciesTerraform graph and provider operationsReferences, reconciliation, and readiness gates
DriftObserved when the workflow refreshes state and plans or appliesDetected during reconciliation for supported resources
Initial platformProject, shared network, IAM foundation, GKE, and independently bootstrapped state storageApplication-scoped cloud resources after Kubernetes, the controller, and Google Cloud access are established

For example, keep a shared Cloud SQL instance under Terraform while an application-specific Storage bucket is managed by Config Connector. Argo CD or Config Sync can deliver the StorageBucket CR while Config Connector reconciles the cloud bucket. Record both ownership layers. If Terraform and Config Connector target the same bucket settings, each can interpret the other's writes as drift. Moving ownership requires an explicit handoff with the cloud resource identity, retention policy, and adoption behavior verified first. Terraform state · Config Connector reconciliation

Repository layout and access

platform/
├── infrastructure/terraform/
│   ├── modules/{network,gke,iam}/
│   └── environments/{dev,staging,prod}/
├── bootstrap/
│   ├── argocd-values.yaml
│   ├── install-argocd.sh
│   └── root-application.yaml
└── gitops/
    ├── bootstrap/children/
    ├── clusters/{dev,staging,prod}/
    ├── platform/{argocd,external-secrets,monitoring,policies,config-connector}/
    ├── applicationsets/microservices.yaml
    └── workloads/{payment,order,inventory}/

These paths show review boundaries. Separate repositories are also appropriate when infrastructure operators, platform administrators, and application teams have distinct access. Restrict changes to bootstrap roots, projects, and cluster selectors to platform administrators. Supply repository credentials through a recovery-accessible secure channel.

For the first install, point the bootstrap runner at the intended cluster, configure the Argo Helm repository, and set ARGOCD_CHART_VERSION to the exact chart version approved for that cluster. Run these examples from the directory containing the values and root manifests.

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update argo
helm upgrade --install argocd argo/argo-cd \
  --namespace argocd --create-namespace \
  --version "${ARGOCD_CHART_VERSION:?Set the approved chart version}" \
  --values argocd-values.yaml --wait --timeout 10m

kubectl wait --for=condition=Established --timeout=120s \
  crd/applications.argoproj.io crd/applicationsets.argoproj.io

Verify repository and destination access, and create the authorized AppProject before seeding the root with kubectl apply -f root-application.yaml. The chart version and Argo CD application version are separate release identifiers. Retain the chart, values, and corresponding controller version for audit and recovery. Argo CD declarative setup

App of Apps and ApplicationSet

In App of Apps, a root Application sync creates child Application objects. Argo CD then reconciles each child's own source and destination. For this example, assume platform-admin permits the example repository, the in-cluster destinations and the required resource kinds. Replace the example URL with your protected repository. Commit the child manifest under gitops/bootstrap/children.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: platform-bootstrap
  namespace: argocd
spec:
  project: platform-admin
  source:
    repoURL: https://github.com/example/platform.git
    targetRevision: main
    path: gitops/bootstrap/children
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: false
      selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: argocd
spec:
  project: platform-admin
  source:
    repoURL: https://github.com/example/platform.git
    targetRevision: main
    path: gitops/platform/external-secrets
  destination:
    server: https://kubernetes.default.svc
    namespace: external-secrets
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

Save the two documents as separate files and apply only the root during seeding. This child requires a manual sync or a separate automation decision. If a baseline controller owns the namespace, provision it through that controller and omit CreateNamespace=true here. An administrator who can change the root's inputs can influence child sources, destinations, and project selection. Protect those inputs accordingly. Root pruning and deletion of a child Application are separate operations. A child's resource finalizer determines whether deleting that child also deletes its deployed resources. Argo CD cluster bootstrapping

PatternHow Applications appearTypical use
App of AppsApplication Controller applies explicitly listed child Application manifestsA curated set of platform components with independently configured child Applications
ApplicationSetApplicationSet Controller renders Applications from a template and generator parametersLabel-based fleet enrollment and repeated cluster or environment deployments

Argo CD recommends ApplicationSet and cluster labels for multi-cluster bootstrapping. In a centralized deployment, keep cluster registrations in the management instance and enroll targets only after their connectivity and permissions are verified. A GKE Fleet membership does not create an Argo CD cluster registration. Fleet and Config Sync · Argo CD cluster bootstrapping

TopologyFailure domainOperational trade-off
CentralizedManagement-plane outage can interrupt reconciliation across many clusters while existing workloads may continue runningFewer installations, concentrated destination access, and a requirement for network reachability to remote APIs
Per clusterLocal GitOps outages are largely isolated to one targetMore instances and upgrades, with local deployment credentials and stronger separation between environments

Gate each dependency at the appropriate layer

CheckpointExample evidenceWhat it establishes
CRD submittedkubectl get crd externalsecrets.external-secrets.ioThe definition object exists
CRD establishedkubectl wait --for=condition=Established --timeout=120s crd/externalsecrets.external-secrets.ioThe CRD is established. Confirm the desired group/version is listed in API discovery.
Controller and webhooks readyESO Deployment rollout succeeds and a representative API write passes admissionThe software can run and the required admission endpoint works
Custom resource readykubectl -n payments wait --for=condition=Ready --timeout=120s externalsecret/payment-dbESO reports successful reconciliation. Verify the expected Secret exists without printing its contents.
Workload readyDeployment rollout and API smoke test passThe consumer can run with the resulting configuration

Within one Application, a namespace can use wave -2, CRDs and operators wave -1, custom resources wave 0, and workloads wave 1. Apply the wave through the annotation below. Argo CD uses sync phase and wave ordering and evaluates health before progressing. During pruning, higher waves are processed first.

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

Across independently syncing Applications, a parent wave orders its child objects. To use parent waves as readiness gates, configure and verify an Application health assessment that reflects child readiness and ensure the relevant children actually sync. Alternatively, use staged enrollment or an external pipeline gate with direct API checks. An established ESO CRD can accept ExternalSecret objects before the operator can process them. Provider IAM, a ready SecretStore, and a successful target Secret reconciliation are separate prerequisites. Sync phases and waves · Argo CD resource health

Self-managed Argo CD and recovery

A self-management Application can point to gitops/platform/argocd using platform-admin and the local argocd namespace. Start with manifests matching the live installation and a protected, reviewable revision. Argo CD's documented self-management setup requires Server-Side Apply.

syncPolicy:
  syncOptions:
    - ServerSideApply=true

Begin with a reviewed manual sync by omitting syncPolicy.automated. (Explicit automated.enabled: false requires Argo CD 3.1+; on older versions, the API server prunes the unknown enabled field and leaves any remaining automated block active.) Compare rendered resources with the installed version, especially the CRDs, repository access, RBAC, and immutable fields. Pause any former recurring Helm or Terraform writer during the handoff. Keep Helm release records and account for what the old system could still remove. Running helm uninstall on the original release can delete resources the new Application depends on. Once the handoff is verified, enable automated sync if appropriate to the recovery plan. Argo CD self-management

During an upgrade, Kubernetes retains Application CRs while controllers restart. A broken repo-server image, revoked credentials, incompatible CRD, or RBAC regression can prevent the controller from reading or applying the repair. Keep a tested bootstrap runner and break-glass identity outside Argo CD. In a recovery drill, freeze the bad revision, restore approved manifests and credentials with that runner, verify source and destination access, and resume reconciliation from a known-good commit. Restore persistent data from its own backup and verify workload behavior before declaring recovery complete. Argo CD disaster recovery

Coexistence with Config Sync and ESO

One ownership split gives Config Sync namespaces, quotas, shared NetworkPolicies, RBAC, and baseline policy. Argo CD manages application Deployments, Services, Ingress, HPAs, and application CRs. Config Connector reconciles selected Google Cloud resources from their CRs. ESO writes the target Secrets referenced by workloads. A single namespace can contain resources delivered by multiple reconcilers.

For example, define Namespace/payments and its environment=production label in one baseline source. A second source declaring environment=staging creates competing intent. Depending on field management and write method, the overlap can cause conflicts or repeated updates. Inspect Git sources, controller logs and managedFields, then remove the duplicate declaration through a tested handoff. Account for tracking and prune behavior before moving a live object between tools. Config Sync supports Git, OCI, and Helm sources. Its sync-object deletion propagation also warrants review during a handoff. Config Sync overview · Managing Config Sync objects

Decommission while the reconcilers still work

For a cluster such as gke-staging-02, review the Kubernetes and Google Cloud inventories before approving infrastructure removal.

  1. Inventory Applications, ApplicationSets, sync objects, operators, CRs, finalizers, PVs and PVCs, backing disks, shared DNS, and cloud resources. Identify each resource's full cloud identity and current lifecycle owner.
  2. Freeze new deployment to the destination. Inspect what removing a generator match would do before changing registration labels. Generated Applications can be deleted depending on policy, and their finalizers can trigger workload deletion.
  3. Classify and back up data. Record delete, migrate, or retain for each stateful resource and verify restore procedures. Check storage reclaim policies and any provider-level deletion protection.
  4. Detach or remove workloads under the chosen Application deletion and prune policies. Keep the destination reachable while Argo CD needs it for cleanup. Verify the results in the target cluster.
  5. Resolve external resources while Config Connector and other operators still have credentials and network access. For a retained Config Connector resource, apply and verify cnrm.cloud.google.com/deletion-policy: abandon on its CR before removing that CR. Verify the cloud resource still exists and record its next owner. For approved deletion, confirm both the CR's finalizer completion and the intended outcome in the cloud API.
  6. Remove platform components in dependency order after dependent CRs and finalizers are resolved. Deleting a CRD also deletes its instances, so review all namespaces and any shared ownership before removing one. Remove cluster registration after remote cleanup finishes.
  7. Review the Terraform destroy plan and apply only within the approved scope. Check that shared networks, IAM bindings, state storage, and retained data survive, and record the resulting inventory.

Deleting GKE first can strand a Config Connector-managed Cloud SQL instance by removing the controller and its CR state before either deletion or handoff completes. A successful Terraform destroy only verifies operations in its own managed scope. Managing and deleting Config Connector resources · ApplicationSet resource modification policies

Self-check questions

  1. Why does an Application fail to start delivery in a fresh cluster? Answer. The Application CRD enables API admission for that type, and the Argo CD controller turns an accepted Application into reconciliation work. Bootstrap must provide both.
  2. Can Config Sync and Argo CD safely maintain different values for the same namespace label? Answer. Give that field one authoritative desired value and owner. SSA conflict detection and force behavior vary by writer and cannot reconcile opposing intent.
  3. What can happen when GKE is destroyed before Config Connector cleanup? Answer. External resources may remain unmanaged. Resolve retention, transfer, or deletion while the controller and CR still exist.
  4. Does a healthy root Application prove a dependent workload can use its database credentials? Answer. Verify child health, the ExternalSecret Ready condition, and an application-level connection check.
  5. Which controller owns the desired state, and which owns the actual state in your bootstrap topology?
  6. What failure modes can occur even if a Kubernetes resource reports Ready, and what would you verify before deploying to production?

Continue with Lesson 10 — Production Reliability and Progressive Delivery for rolling update capacity, readiness, disruption budgets, and canary or blue-green delivery with metric-driven analysis.

On this page