EKS is easy to stand up and easy to get subtly wrong: a cluster left on an unsupported version, a node group pinned to one AZ, a pod handed the node's broad role. These are the everyday ways EKS automation becomes fragile or insecure.
Walk this list once now, then use it as a review checklist before shipping any code that provisions, scales, or secures a cluster.
Wait on every async operation. Use cluster_active, nodegroup_active, and addon_active waiters (or waitUntilClusterActive and friends) rather than assuming a create returned means ready.
Order teardown correctly. Delete node groups, Fargate profiles, and add-ons before the cluster; DeleteCluster fails while they exist, and waiting on nodegroup_deleted avoids the race.
Poll updates by id, not by guess. Async updates return an update id; check DescribeUpdate for Successful/Failed instead of sleeping a fixed time.
Rely on built-in retries. Both SDKs retry throttling and transient errors with backoff; tune max_attempts rather than reimplementing retries.
Wait for the cluster instead of guessing it finished.
Upgrade one minor version at a time. EKS supports single-step control-plane upgrades (1.29 -> 1.30); skipping versions is not allowed, so stay current enough to hop one at a time.
Stay on a supported version. Kubernetes minors leave standard support on a schedule; falling behind risks forced upgrades and extended-support charges.
Upgrade add-ons alongside the control plane. After a version bump, update vpc-cni, coredns, and kube-proxy to a compatible build via DescribeAddonVersions + UpdateAddon.
Roll node groups after the control plane. Bring workers to the new version with UpdateNodegroupVersion, which drains gradually; the control plane can lead nodes by one minor, not more.
Test upgrades in non-prod first. Run the same SDK upgrade path against a staging cluster before production.
Update the control plane, then bring add-ons to a compatible version.
Give pods AWS access with IRSA, not the node role. A per-service-account role scopes credentials to one workload instead of sharing the node's broad role.
Register the OIDC provider once per cluster. All IRSA roles reuse a single CreateOpenIDConnectProvider registration of the cluster issuer.
Pin the trust policy to one service account. Use StringEquals on sub (system:serviceaccount:ns:name) and aud = sts.amazonaws.com; never a wildcard.
One narrow role per workload. Roles are free; a role per service account keeps least privilege instead of a shared god-role.
Consider EKS Pod Identity for scale. For many clusters, CreatePodIdentityAssociation avoids per-cluster OIDC and per-role trust wiring and is easier to automate.
Read the issuer, register it once, then reuse it for every IRSA role.
Use access entries, not aws-auth. Manage cluster access from the EKS SDK with CreateAccessEntry + AssociateAccessPolicy instead of the fragile, kubectl-edited ConfigMap.
Move clusters to authenticationMode API. Once every mapping is an access entry, switch off CONFIG_MAP so aws-auth cannot silently grant access - remembering the switch is one-way.
Grant least-privilege access policies. Prefer AmazonEKSEditPolicy or AmazonEKSViewPolicy scoped to namespaces over cluster-wide AmazonEKSClusterAdminPolicy.
Scope endpoints deliberately. Enable endpointPrivateAccess and restrict publicAccessCidrs so the API server is not reachable from anywhere.
Enable control-plane logging and encrypt secrets. Turn on the relevant logging types and use a KMS key for envelope encryption of Kubernetes secrets.
Why wait on EKS operations instead of assuming they finished?
Cluster, node group, and add-on operations are asynchronous and take minutes. A returned create call only means "accepted." Use the *_active waiters so you act on a ready resource, not a pending one.
Can I skip Kubernetes versions when upgrading?
No. EKS control-plane upgrades are one minor version at a time (for example 1.29 to 1.30). Stay current enough that you can always hop a single version.
Should I upgrade add-ons when I upgrade the cluster?
Yes. After a control-plane version bump, update vpc-cni, coredns, and kube-proxy to a compatible build via DescribeAddonVersions and UpdateAddon, or you risk incompatibility.
Do I set desiredSize on a node group?
Only if no autoscaler manages the group. With Cluster Autoscaler or Karpenter running, set minSize and maxSize and let the controller own desiredSize to avoid a tug-of-war.
Why IRSA instead of the node role for pod credentials?
The node role is shared by every pod on the node, which violates least privilege. IRSA gives each service account its own scoped, short-lived role so a compromise is contained.
How many OIDC providers do I need?
One per cluster. Register the cluster's issuer once with CreateOpenIDConnectProvider; every IRSA role reuses it, differing only in the trust policy's sub condition.
Should I still use the aws-auth ConfigMap?
Prefer access entries. They put access decisions on the EKS SDK, avoid the fragile ConfigMap, and support least-privilege access policies. Migrate existing mappings and move to authenticationMode: API.
Which access policy should most principals get?
The least-privilege one that still works - typically AmazonEKSEditPolicy or AmazonEKSViewPolicy scoped to specific namespaces, not cluster-wide AmazonEKSClusterAdminPolicy.
How do I keep the API server from being publicly reachable?
Enable endpointPrivateAccess, and if public access is needed, restrict publicAccessCidrs to known ranges rather than leaving it open to the internet.
What is the biggest hidden EKS cost?
Idle control planes. A cluster bills per hour even with no workloads, so ephemeral and test clusters left running are a common surprise. Tear them down and sweep by tag and age.
Should I write my own retry logic for EKS calls?
Usually no. Both SDKs retry throttling and transient errors with backoff. Tune retry mode and max attempts instead of reimplementing it.
What is the safe order to delete a cluster?
Remove node groups, Fargate profiles, and add-ons first (waiting on their deletion), then call DeleteCluster. It fails while dependents exist.