DevOps interview questions for experienced

Keep the bucket private and put CloudFront in front of it. Use Origin Access Control (OAC) so only CloudFront can read the objects.

Steps:

• Create the bucket with Block Public Access on. Do not turn on S3 static website hosting, since that endpoint is public only.
• Create a CloudFront distribution with the bucket as origin and attach an OAC.
• Add a bucket policy that allows cloudfront.amazonaws.com to s3:GetObject, and limit it with AWS:SourceArn to your distribution.
• Attach an ACM certificate (issued in us-east-1) and redirect HTTP to HTTPS.
• Point Route 53 at the distribution with an alias record.
• You can also add AWS WAF and security headers (through a response headers policy). Use CloudFront signed URLs or cookies for private content.

If one site has public and private content, serve public paths openly. Protect the private path pattern with a trusted key group, so only signed requests work.

Deploy with aws s3 sync ./dist s3://my-site-bucket --delete and invalidate the cache with aws cloudfront create-invalidation --distribution-id <id> --paths "/*".

What they are checking: Whether you know the private bucket plus OAC pattern rather than a public bucket

The load balancer spreads requests across healthy instances. The Auto Scaling Group (ASG) changes how many instances run, based on metrics.

Flow when the system slows:

• Load rises, a CloudWatch metric such as average CPU or RequestCountPerTarget crosses the target.
• A scaling policy (target tracking or step) raises desired capacity.
• The ASG launches instances from the launch template in several AZs.
• Once an instance passes the target group health check, the ALB starts sending it traffic.
• When load falls, the ASG scales in; connection draining (deregistration delay) lets in-flight requests finish.

Points interviewers like:

• Turn on ELB health checks on the ASG, so unhealthy instances are replaced. EC2 status checks alone are not enough.
• Set a health check grace period longer than app start time.
• Keep instances stateless (sessions in Redis or DynamoDB, files in S3 or EFS).
• If the database is the bottleneck, scaling the web tier will not help; use read replicas, caching or connection pooling.

What they are checking: Whether you understand the scale-out loop end to end, including health checks and statelessness

CPU-threshold target tracking reacts only after the load has arrived. New instances need minutes to start and warm up. So a sharp spike can overload the fleet first.

Ways to cope:

• Scheduled scaling: raise min and desired capacity before known peaks (sale, campaign, office hours).
• Predictive scaling: learns daily and weekly patterns and scales ahead.
• Better metrics: scale on ALB RequestCountPerTarget, queue depth, or latency rather than CPU alone.
• Lower the target (for example 50 percent CPU) to keep spare room. Use shorter cooldowns or instance warm-up settings.
• Warm pools or pre-baked AMIs so new instances join quickly.
• Absorb bursts: CloudFront caching, a queue between tiers, and rate limiting at the edge.
• Keep min capacity across at least two AZs so one AZ failure is also covered.

aws autoscaling put-scheduled-update-group-action \
  --auto-scaling-group-name web-asg --scheduled-action-name morning \
  --recurrence "0 8 * * *" --min-size 4 --desired-capacity 6

What they are checking: Whether you know scaling is reactive and can combine scheduled, predictive and metric choices

Use an S3 Lifecycle configuration with an expiration rule on the bucket or on a prefix. S3 deletes eligible objects in the background, free of charge.

• Filter by prefix or tag, for example logs/.
• Set Expiration to the number of days after creation.
• For versioned buckets, also expire old versions and clean up expired delete markers. Otherwise old versions keep costing money.
• Add AbortIncompleteMultipartUpload to remove stuck uploads.
• To control growth rather than delete, transition to cheaper classes first (Standard-IA, Glacier) and expire later; use Intelligent-Tiering when access is unpredictable.

{"Rules":[{"ID":"expire-logs","Status":"Enabled",
 "Filter":{"Prefix":"logs/"},
 "Expiration":{"Days":10},
 "NoncurrentVersionExpiration":{"NoncurrentDays":1},
 "AbortIncompleteMultipartUpload":{"DaysAfterInitiation":7}}]}

aws s3api put-bucket-lifecycle-configuration --bucket my-logs --lifecycle-configuration file://lifecycle.json

Monitor size with S3 Storage Lens or CloudWatch BucketSizeBytes, and set a budget alert.

What they are checking: Whether you know lifecycle rules, including versioning and multipart cleanup

Control how fast the queue drives the function. This stops it from using up account concurrency or overloading other systems.

Run a Transfer Family SFTP server with service-managed users. Script user creation with a Terraform module or the CLI. Each client then gets its own separate home directory.

A cold start is the extra delay when Lambda must create a new execution environment. It downloads the code, starts the runtime, and runs your initialization code before handling the request. Warm invocations reuse an existing environment and skip this.

Set the timeouts based on how long your consumer needs. Send messages that keep failing to a dead-letter queue.

Use an ALB for the public entry point. Use service discovery inside the VPC for private traffic between services.

Use AWS Organizations to group accounts into organizational units (OUs). Apply guardrails in one place, and use IAM Identity Center for sign-in. Common accounts are management, security or log archive, shared services, and separate dev, staging and production accounts.

Use VPC endpoints so resources in private subnets reach AWS services over the AWS network. You need no NAT gateway and no public internet.

Keep the web, application and data tiers in different subnets across at least two AZs. Expose only the front door.

Put API Gateway in front, one or a few Lambda functions behind it, and DynamoDB for storage. There are no servers to patch, and cost follows usage.

Choose ECS when you want simple container management that fits AWS well. Choose EKS when you need Kubernetes itself.

ECS:

• No control plane fee, easy to learn, task definitions and services.
• Deep integration with IAM task roles, ALB, CloudWatch and Fargate.
• AWS-specific, so less portable.

Scale on the backlog per worker, not on raw CPU, so capacity follows the work waiting.

Amplify Hosting supports Next.js SSR natively: it builds the app and runs server-rendered routes on managed compute. It serves static files (_next/static, public files) through its own CloudFront distribution backed by S3. So you usually do not connect these yourself.

Plan the rollback before the change, because RDS cannot undo a bad write in place.

Use the ECS service's rolling update deployment controller together with the deployment circuit breaker and health checks.

A dead-letter queue (DLQ) is a separate queue for messages that the main queue could not process after a set number of tries. They then do not block the queue or loop forever.

Graviton instances (such as m7g, c7g, t4g) usually give better value for money. But the software must run on the arm64 architecture.

Layer controls so only intended traffic reaches the load balancer and the targets behind it.

Combine controls that stop bad settings before they happen with controls that find and fix them later.

Start by measuring (Cost Explorer, tags, Budgets), then act on the biggest items.

Build the new version into an AMI. Create a new launch template version that points to it. Then replace instances slowly.

No. A React single-page app compiles to static files, so EKS is heavy overkill. You pay about $0.10 per hour for the control plane plus worker nodes. You must also run the cluster.

With Fargate, AWS runs the compute for you (serverless containers); with the EC2 launch type you manage the instances in the cluster.

Keep secrets out of images and task definitions as plain text. Use IAM roles so no keys are stored.

Place an ALB in public subnets, tasks in private subnets, and register tasks through IP-type target groups.

Run them across Availability Zones, keep them private, encrypt them, and monitor failover.

Connect the two VPCs privately, then allow the traffic with routes and security groups.

They go through a NAT gateway placed in a public subnet. The private subnet route table sends 0.0.0.0/0 to the NAT gateway. The NAT gateway has an Elastic IP. It sits in a subnet whose route table points to the internet gateway.

Security groups are stateful firewalls attached to network interfaces (instances, load balancers); network ACLs (NACLs) are stateless firewalls attached to subnets.

Start with the address plan, then subnets, routing and connectivity.

Each service invokes Lambda in a different way, which affects retries and scaling.

Lambda behavior depends on how it is invoked, so plan timeouts, retries and idempotency together.

Recursive loop detection is a Lambda safeguard. It stops a function from calling itself again and again through AWS services. An example is a function triggered by an SQS queue or SNS topic that writes back to the same queue or topic. After about 16 recursive calls in a chain, Lambda blocks the next one. It sends a notice through the Health Dashboard and email.

Use Route 53 routing policies, or AWS Global Accelerator, in front of identical stacks in each region.

Define the configuration as an image plus a template, and copy or rebuild both in each region.

Cognito provides authentication, authorization and user management for web and mobile apps.

Staging should look like production in design, but not in size or uptime. This cuts cost without losing test value.

Amazon ECS (Elastic Container Service) is the AWS service that manages containers. It runs Docker containers as tasks and services on Fargate or EC2.

EBS has SSD volumes for most workloads and HDD volumes for throughput-heavy, infrequent access.

It is AWS's set of best practices for reviewing a workload, organised into six pillars:

An S3 access decision combines the caller's IAM policy, the bucket policy, and several guardrails. Work through each.

Find what changed with Cost Explorer, fix it, then add guardrails.

On a directory, execute (x) means you can traverse it: cd into it and access items inside by name. Read (r) means you can list its names.

• r only (e.g. chmod 400 dir): ls dir shows names. But you cannot cd into it, read file details, or open files inside.
• x only: you can cd and open a file if you know its exact name, but cannot list.
• r + x: the normal read-only access (r-x): list and enter.
• w + x: needed to create, delete or rename files in it.

So to cd into a folder you need x on it and on every parent directory. Read permission alone is not enough to enter or properly list contents.

What they are checking: Whether you understand that directory permissions differ from file permissions

Use groups and standard permissions first, and ACLs when the simple owner/group/others model is not enough.

# group-based: team A writes, team B reads
groupadd teama; groupadd teamb
chown -R deploy:teama /srv/app
chmod -R 2775 /srv/app          # setgid keeps the group on new files
setfacl -R -m g:teamb:rX /srv/app          # B: read + traverse
setfacl -R -d -m g:teamb:rX /srv/app       # default ACL for new files

# single user, read-only
setfacl -m u:alice:rX /srv/data
getfacl /srv/data

Capital X gives execute only on directories. So the user can enter the folder and read files, but cannot change them. Without execute on the directory, the user could not enter it even with read permission. Mind the ACL mask and parent directory traverse rights.

What they are checking: Whether you know groups, setgid and ACLs for multi-team access

set -e makes the script exit at once when a command fails (non-zero status). So errors are not ignored quietly.

The usual cause is a deleted file that a running process still holds open. The space is not freed until the process closes it.

Find out which resource is saturated first: CPU, memory, disk IO or network, then which process is responsible.

Read the unit status and the journal. Then check the command, user, paths and environment the unit uses.

Run the client in verbose mode to see which key is offered. Then check the key, the permissions and the server logs.

Create a .service file under /etc/systemd/system/ with [Unit], [Service] and [Install] sections, then reload and enable it.

Start the app under PM2 and save the process list. Then make the startup script, so systemd starts PM2 at boot.

Avoid storing changing state on the instances, and pick a shared or build-time approach.

The s is the setuid or setgid bit. The program runs with the permissions of its file owner (setuid) or group (setgid), not the user who starts it.

DNS servers are set in /etc/resolv.conf (on modern Ubuntu it is managed by systemd-resolved, check with resolvectl status). Local name overrides go in /etc/hosts.

Manage access in one place instead of copying keys by hand. Keep a second way in, so a lost key does not lock you out.

Check the kernel log for the out-of-memory killer, then fix the memory demand or limits.

Check inode exhaustion: a filesystem can run out of inodes (file entries) before it runs out of bytes.

Another process is already listening on that port. Find it, then decide whether to stop it or change the port.

Cron runs with a very small environment and no terminal. So most failures are path, permission or environment problems.

First stop the damage and check what changed. Then restore the right modes from a known good source, instead of guessing.

High wa means the CPUs are idle and waiting for storage. Find the device, then the process, then the reason.

Use logrotate to rotate, compress and expire logs automatically, and handle the live file correctly.

Grow the block device first, then the partition or logical volume, then the filesystem. You can do all this while it is mounted.

CPU use is the share of time the cores are busy. Load average counts tasks that are running or waiting to run. On Linux it also counts tasks stuck in uninterruptible IO wait.

Deny inbound traffic by default. Then open only the ports you need. Prefer the tool your distro ships with.

Use strace for system calls, lsof for open files and sockets, perf for CPU profiling, and tcpdump for network packets.

Fail fast, quote everything, validate input and clean up after yourself.

ulimit sets per-process limits such as open files. sysctl tunes kernel settings. For services, set the limit in the systemd unit.

Both are access control systems that limit what a process can do, even if it runs as root. SELinux is the default on RHEL-family distributions and AppArmor on Ubuntu and SUSE.

Attack both: smaller layers and better cache use.

Image size

• Use a slim, alpine or distroless base. Use multi-stage builds so compilers stay out of the final image.
• Combine related RUN steps and clean caches in the same layer (apt-get install --no-install-recommends then remove /var/lib/apt/lists/*).
• Use a .dockerignore to keep .git, node_modules and logs out of the context.

Build time

• Order instructions from least to most frequently changing: copy package.json or requirements.txt, install, then copy the source.
• Use BuildKit cache mounts (RUN --mount=type=cache,target=/root/.cache/pip pip install ...).
• In CI, reuse cache with --cache-from/--cache-to (registry or GitHub Actions cache), and build independent stages in parallel.
• Pin base images so the cache is not invalidated by surprise.

Measure with docker history and docker build --progress=plain to see which step is slow.

What they are checking: Whether you understand layer caching and can name concrete, practical optimisations.

Use a full Node image to install and build. Then use a slim or distroless image to run only the production output.

FROM node:22 AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci

FROM deps AS build
COPY . .
RUN npm run build && npm prune --omit=dev

FROM gcr.io/distroless/nodejs22-debian12
WORKDIR /app
ENV NODE_ENV=production
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
USER nonroot
EXPOSE 3000
CMD ["dist/server.js"]

• Build stage: node:22 (or -slim) has npm and build tooling.
• Runtime stage: node:22-alpine or distroless for a small, low-attack-surface image.
• The package files are copied first so npm ci stays cached.
• For Next.js, enable output: "standalone" and copy .next/standalone and .next/static.
• Add a .dockerignore for node_modules and .git.

What they are checking: Whether you can write a production-grade Dockerfile with caching, small runtime and non-root user.

• Minimal base images (slim, distroless) with fixed versions or digests. Use multi-stage builds so build tools are not shipped.
• Run as non-root: add USER app in the Dockerfile; in Kubernetes set runAsNonRoot: true.
• Scan images in CI (Trivy, Grype, ECR scanning) and fail on critical CVEs. Rebuild regularly for patches.
• No secrets in images or build args. Use runtime secrets or BuildKit secret mounts.
• Least privilege at runtime: read-only root filesystem, drop Linux capabilities, no --privileged, never mount /var/run/docker.sock.
• Trusted sources: pull from verified registries and sign images (cosign).
• Use a .dockerignore so .env and .git never enter the build context.

RUN addgroup -S app && adduser -S app -G app
USER app

What they are checking: Whether you apply defence in depth to the image and runtime, not just one fix.

A container exits when its main process (PID 1) ends, so find out why that process stopped.

Inside a container, localhost means the container itself, not the host or another container. So the app connects to nothing.

Two options:

Registry (best): push to Docker Hub, AWS ECR, GitHub Container Registry or a private registry. Then the other person pulls it.

docker tag myapp:1.0 registry.example.com/team/myapp:1.0
docker login registry.example.com
docker push registry.example.com/team/myapp:1.0

Running as non-root is the secure default, but it often breaks things written assuming root.

Use a multi-stage build: the SDK image compiles, and a small ASP.NET runtime image runs the published output.

Containerise each service separately and route to them through one entry point.

A 10-second stop usually means your app never got SIGTERM. Docker waited for the grace period and then sent SIGKILL.

Docker is a client that talks to a daemon. The daemon hands work to lower-level parts that use Linux kernel features to isolate processes.

Exit code 137 is 128 + 9, meaning the process was killed by SIGKILL. The most common reason is the kernel OOM killer, after the container hit its memory limit. But docker kill or a failed clean stop also give 137.

Never use ARG, ENV or COPY for secrets, because they stay in image layers and docker history. Use BuildKit secret mounts. They show the secret only during one RUN step and never write it to a layer.

latest is just a movable label, not the newest version by magic. It makes deployments non-reproducible: two hosts can pull different images under the same name, and rollback is unclear.

Scan images in CI, use small trusted bases, and rebuild regularly.

Use Docker Buildx (BuildKit) to build a multi-platform image and push it as one manifest list. Each machine then pulls the version for its own architecture.

Keep one base Compose file and layer environment-specific differences on top.

It means something already listens on that host port, either another container or a host process.

Find the owner:

docker ps --format 'table {{.Names}}\t{{.Ports}}'
sudo ss -ltnp | grep ':8080'

CI runners are usually clean machines with no local cache, so every build starts from zero. Fix that first, then optimise the Dockerfile.

Containers remove many differences, but not all. Compare what is still different.

The Docker socket is the API of the daemon, which runs as root. A container that can talk to it can start a privileged container that mounts the host filesystem. So access to the socket is the same as root on the host.

Match the tool to the scale and operational needs.

Helm is the package manager for Kubernetes. A chart is a versioned bundle of template manifests plus default settings. A release is one installed copy of a chart.

A chart contains Chart.yaml (metadata), values.yaml (defaults), and templates/ (Go-templated YAML). You override values per environment with -f values-prod.yaml or --set.

helm install web ./web -f values-prod.yaml -n prod
helm upgrade web ./web -f values-prod.yaml -n prod
helm rollback web 1 -n prod
helm list -n prod

Why it helps: one template works for dev, staging and prod. Releases have versions, history and rollback. You can install third-party software with one command.

What they are checking: Whether you have used Helm for templates and releases.

Path of a request: user -> DNS -> cloud load balancer -> ingress controller -> Service -> pod.

1. DNS resolves each domain to the same load balancer.
2. The load balancer (an ALB, or an NLB in front of NGINX) sends traffic to the ingress controller. With the AWS Load Balancer Controller, the ALB can send traffic straight to pod IPs (target-type: ip).
3. The Ingress rules send traffic to different Services by Host header and path.
4. The Service shares traffic across healthy pods (those that pass readiness probes).

spec:
  ingressClassName: alb
  tls:
    - hosts: [app.example.com, api.example.com]
      secretName: tls
  rules:
    - host: app.example.com
      http: {paths: [{path: /, pathType: Prefix, backend: {service: {name: web, port: {number: 80}}}}]}
    - host: api.example.com
      http: {paths: [{path: /, pathType: Prefix, backend: {service: {name: api, port: {number: 80}}}}]}

TLS certificates for several domains come from ACM (SNI) or cert-manager. One load balancer for many apps saves money. The ALB group.name annotation can join several Ingresses onto one ALB.

What they are checking: Whether you understand host and path routing from start to end, not only Service types.

Kubernetes is an open-source tool that runs and manages containers. You declare the state you want, for example three replicas of an image. Kubernetes keeps working to make the real state match it. It handles scheduling, scaling, self-healing and rollouts.

Key pieces:

• Control plane: API server, etcd, scheduler and controller manager.
• Nodes: run the kubelet and a container runtime.
• Objects: Pod, Deployment, Service, Ingress, ConfigMap, Secret, PersistentVolumeClaim.

What it gives you: automatic restarts of failed pods, rolling updates and rollbacks, scaling out, service discovery and load balancing. Your config is written as files that say what you want (declarative), which fits GitOps.

Managed offerings such as EKS, GKE and AKS run the control plane for you (the part that makes decisions).

What they are checking: Whether you can explain the problem Kubernetes solves and its core components.

A Deployment manages identical pods that keep no state. A StatefulSet manages pods that need identity and stable storage.

Use a StatefulSet for databases, Kafka, Elasticsearch and anything where each replica needs its own saved data or a stable hostname. Leader election and cluster membership are examples. Use a Deployment for web and API services. Apps like Airflow can use a StatefulSet for parts that keep local state, such as workers that need saved logs.

What they are checking: Whether you know when pod identity and per-pod storage matter.

Storage

• Install the EBS CSI driver for block volumes (RWO, one AZ) and the EFS CSI driver for shared storage (RWX).
• Create a StorageClass (for example gp3), then request storage with a PersistentVolumeClaim, or volumeClaimTemplates in a StatefulSet.
• Use EBS for databases and low-latency data, EFS for shared files. Set the reclaim policy, and take snapshots (VolumeSnapshot) or backups.

Secrets

• Keep secrets in AWS Secrets Manager or Parameter Store.
• Use the Secrets Store CSI driver with the AWS provider to mount them as files (or copy them into Kubernetes Secrets). You can also use External Secrets Operator.
• Grant access with IRSA or Pod Identity, limited to that one secret.

volumeClaimTemplates:
  - metadata: {name: data}
    spec:
      accessModes: [ReadWriteOnce]
      storageClassName: gp3
      resources: {requests: {storage: 20Gi}}

What they are checking: Whether you can match the right storage and the right AWS secret tool.

Access modes say how many nodes can mount a volume at the same time.

• ReadWriteOnce (RWO): read-write by a single node. Pods on that same node can still share it. It is typical for block storage such as AWS EBS, which is also tied to one availability zone.
• ReadWriteMany (RWX): read-write by many nodes at once. It needs a shared filesystem such as AWS EFS, NFS or CephFS.
• There are also ReadOnlyMany (ROX) and ReadWriteOncePod (one pod only).

Practical impact: a Deployment with several replicas on different nodes cannot all mount an RWO volume. So use RWX storage for shared uploads, or give each replica its own volume with a StatefulSet. Databases usually want RWO block storage because it is fast.

What they are checking: Whether you can match storage types to workloads and know how EBS and EFS differ.

Steps, in order:

It is possible, but often not the best choice. Managed services (RDS, Aurora) handle backups, patching, failover and upgrades for you. For most teams, a managed database is the safer default.

Use one standard chart and keep environment differences only in values.

A typical CI/CD and GitOps flow:

With the Amazon VPC CNI, every pod gets a real VPC IP address. The CNI attaches network interfaces (ENIs) to the node. It gives their extra private IPs to pods. So pods can be reached directly inside the VPC.

RBAC grants permissions to subjects (users, groups, ServiceAccounts) through roles and bindings. Permissions only add up. There are no deny rules.

NotReady means the control plane stopped getting healthy signals (heartbeats) from the kubelet. Start at the node and work outward.

OOMKilled (exit code 137) means the container used more memory than its limit, so the kernel killed it.

Confirm:

kubectl describe pod web-abc | grep -A5 'Last State'
kubectl top pod web-abc --containers

Follow the path from the Ingress down to the Pod and find where it breaks.

Start with kubectl describe hpa and kubectl get hpa; the Conditions and metrics columns show what is wrong.

Use a rolling update together with correct health checks and a clean shutdown.

Plan, test and roll out step by step, one minor version at a time.

Workloads on spot instances should keep no state, because AWS can take a spot instance back with a two-minute warning.

Use the same structure and automation everywhere, and make lower environments smaller.

Give each pod its own IAM role through its Kubernetes ServiceAccount. Do not use the node's role or fixed keys.

Do everything through code and reviewed pipelines, give only the access needed, and keep logs.

A DaemonSet makes sure one copy of a pod runs on every node (or on chosen nodes). When a node joins the cluster, the pod is added. When a node leaves, the pod is removed.

An operator is a custom controller that manages an application. It adds new object types to Kubernetes with a Custom Resource Definition (CRD). You write a custom object, for example kind: PostgresCluster. The operator's reconcile loop then creates and looks after the pods, volumes, backups and upgrades to match it.

Karpenter adds nodes to a Kubernetes cluster (it came from AWS). When pods are Pending, it starts EC2 instances sized to fit them, directly through the EC2 API. It removes nodes that are empty or can be merged.

Roll back with the Deployment revision history:

Follow the path step by step.

They use Service DNS names. Each Service gets a name of the form <service>.<namespace>.svc.cluster.local.

• Inside the same namespace, the short name orders works.
• Across namespaces use orders.shop or the full name orders.shop.svc.cluster.local.

Use NetworkPolicies. By default every pod can talk to every other pod. So start with a rule that denies all, then allow only what is needed.

Cordon the node so nothing new is placed on it. Then drain it to move Pods away safely. A PodDisruptionBudget (PDB) limits how many replicas can be down at once.

Helm packs manifests into a chart. Each install or upgrade makes a numbered release revision that you can roll back.

By default, Secrets are only base64-encoded, not encrypted. So they are not safe on their own.

Risks: anyone with get secret rights can read them. They sit unencrypted in etcd unless encryption at rest is on. They can also leak through Git, logs or env vars.

They control how much each namespace can use, so one team cannot take everything.

Init containers run and finish before the app containers start. Sidecars run next to the app for its whole life.

etcd holds all cluster state, so back it up and run several members that vote together (a quorum).

It finds subnets by tag, unless you set them yourself.

For most teams I choose Terraform, but the right answer depends on context.

Terraform strengths:

• Multi-cloud and many providers (Kubernetes, Cloudflare, Datadog, GitHub), one workflow.
• Easy-to-read HCL, many ready-made modules, and plan shows the exact changes.
• State gives you flexibility (import, move, run on one resource).
• Loops and expressions are more powerful (for_each, dynamic).

CloudFormation strengths:

• Made by AWS, with no state file to manage (AWS keeps it). It has automatic rollback on failure, drift detection (finding manual changes), and StackSets for many accounts.
• New AWS features often arrive first, and it works closely with Service Catalog and SAM.

Downsides: with Terraform you must secure and lock the state yourself. With CloudFormation you are limited to AWS, and templates can be long.

My choice: Terraform for multi-cloud or platform work. CloudFormation or CDK when the company uses only AWS and wants built-in rollback and control.

What they are checking: Whether you can compare tools by their good and bad points, not just name a favorite.

Do not copy the resource block 100 times. Use count or for_each, and build the name from the index or a key.

• Simple, identical servers: count = 100 with Name = "web-${count.index + 1}".
• Servers with set names from a list or map: use for_each with the names as keys. Then removing one does not affect the others.
• Use format("web-%03d", count.index + 1) for names with leading zeros.
• Put shared settings (AMI, type, subnet, tags) in variables or locals. Put per-instance changes in a map.
• Watch the API limits and quotas. In real work, a module or an Auto Scaling group is usually better than 100 separate instances.

resource "aws_instance" "web" {
  count         = 100
  ami           = var.ami_id
  instance_type = "t3.micro"
  tags = {
    Name = format("web-%03d", count.index + 1)
  }
}

What they are checking: Whether you use count or for_each, naming functions and variables instead of copy and paste, and think of ASGs.

Use a remote backend with state locking, so only one apply can change the state at a time. Run applies from one controlled place.

• Backend: S3 for storage. Terraform 1.10 and later can lock in S3 itself with use_lockfile = true. Older setups use a DynamoDB table with a LockID key.
• Enable bucket versioning and encryption, and limit access with IAM.
• Run plan on pull requests and apply from the CI pipeline after approval, not from laptops.
• Split the state by environment and part, so people rarely work on the same state.
• If a lock is stuck after a crashed run, check that nobody is running, then terraform force-unlock <LOCK_ID>.

terraform {
  backend "s3" {
    bucket       = "my-tfstate"
    key          = "prod/network/terraform.tfstate"
    region       = "ap-south-1"
    encrypt      = true
    use_lockfile = true
  }
}

What they are checking: Whether you understand locking and have worked in a team where applies at the same time would be dangerous.

A dynamic block builds repeated inner blocks inside a resource from a list or map. You do not write each one by hand.

Use it when the number of inner blocks changes. Examples are security group ingress rules, lifecycle_rule blocks on a bucket, or setting blocks on an Elastic Beanstalk environment.

• for_each gives the list or map.
• content { } defines the block content, using <label>.value and <label>.key.
• Do not overuse it. It makes code harder to read, so use plain blocks when the number is fixed.

resource "aws_security_group" "web" {
  name = "web"
  dynamic "ingress" {
    for_each = var.ports
    content {
      from_port   = ingress.value
      to_port     = ingress.value
      protocol    = "tcp"
      cidr_blocks = ["10.0.0.0/16"]
    }
  }
}

What they are checking: Whether you know the difference between loops on resources (count or for_each) and loops on inner blocks (dynamic).

Keep the logic in reusable modules. Give each environment its own config and its own state.

Common layout:

• modules/ holds reusable code (network, compute, database).
• envs/dev, envs/staging, envs/prod each call the modules with different variable values. Each environment has its own backend key.
• You can also use a separate AWS account for each environment to keep them apart.
• Version the modules with Git tags, so staging can try a new version before prod.
• Pipeline: a pull request runs plan for the environment it changes. Apply goes to dev first, then staging, then prod after approval.

Workspaces are a lighter option. But they share one backend config, and it is easy to apply to the wrong environment. So many teams use directories (or Terragrunt) for production. Each directory has its own state, so you can update staging without touching production.

What they are checking: Whether you keep state and variables apart for each environment and move changes up step by step.

It depends on whether you used count or for_each.

Terraform builds a dependency graph by itself from references. If one resource uses an attribute of another (for example subnet_id = aws_subnet.a.id), that is an automatic dependency. Terraform creates the subnet first and destroys it last.

The cleanest way is to remove the resource block from the code and apply. The plan then destroys only that resource, and the change is reviewed in Git.

Changes go through Git and a pipeline, never from a laptop.

Define tags once and use them everywhere. Do not repeat them on each resource.

No. One state for everything is risky. A mistake can hurt everything, plans are slow, teams block each other with locks, and one bad apply can damage all of it.

Split state by:

• environment (dev, staging, prod),
• layer or component (network, data, compute, apps),
• and ownership (team or service).

CloudFormation is the IaC service made by AWS. You write a template (YAML or JSON) that describes resources, and create a stack. CloudFormation then creates the resources and handles order, updates and rollback.

Answer in three parts: what you covered, examples, and what you learned. Sample answer:

Restore the last good copy first. Then check that it matches the real resources.

Use protection in Terraform and protection in AWS together.

Drift is when real infrastructure is different from the code or state. It usually comes from manual changes in the console.

Detect:

• terraform plan (or plan -refresh-only) shows differences.
• Schedule a nightly plan in CI that alerts on any diff.
• For CloudFormation, use stack drift detection.

A provider is the plugin that talks to an API. Declare it in required_providers with a version rule, and set it up in a provider block.

First, read the plan. Terraform marks the setting that forces the rebuild with # forces replacement. Then decide if the rebuild is really needed.

Do not force anything yet. First find out who or what holds the lock.

Terraform saves each resource in state as soon as it is created. So after a half-finished apply, the state is usually correct for what worked. Recovery is mostly "fix the cause and run again".

Treat it as leaked. Change the secret first, then clean up.

Immediate steps:

1. Change the credential at the source (RDS master password, API key).
2. If it was committed to Git, cleaning the history comes second. Changing the secret is what protects you.
3. Restrict who can read the state bucket and check access logs.

Use layers, from cheap and fast to slower and more realistic:

Terragrunt is a small wrapper around Terraform (and OpenTofu). It keeps configs DRY (no repeated code) across many environments and states.

Workspaces let one configuration keep several separate state files. You pick one with terraform workspace select. The default workspace is default, and you read the current name with terraform.workspace.

A module is a folder of Terraform files with inputs (variables), outputs and resources. You call it with a module block. A good module does one job, has a few well-named inputs and sensible defaults, and has documented outputs.

Both run commands, but at different places.

Pick the layer that fits how often the config changes.

Use the community module terraform-aws-modules/eks/aws, or build it from aws_eks_cluster, IAM roles, aws_eks_node_group (or Karpenter), and VPC subnets tagged for load balancers.

If you change a resource address, Terraform thinks the old one is deleted and a new one is needed. Tell it the object is the same. Then only the state changes, not the infrastructure.

Preferred way, which you can review: a moved block in code.

Upgrade on purpose, one small step at a time, and let plan show that it is safe.

The terraform state subcommands look at and edit the state without touching the infrastructure:

They move Terraform from a command on a laptop to a controlled platform with records of every action.

The first need is CIDR ranges that do not overlap. For many VPCs, the alternative is AWS Transit Gateway.

Use a public hosted zone for names on the internet. Use a private hosted zone for names that work only inside your VPCs.

Add an HTTP listener whose only action is a permanent redirect to HTTPS. For end-to-end TLS, encrypt the traffic again on the way to the backends.

Static routes are typed in by hand and never change on their own. Dynamic routing protocols let routers share routes and adjust when something fails.

A Layer 4 balancer routes by IP and port and does not read the content. A Layer 7 balancer reads the HTTP request, so it can route by host, path, header or cookie.

Start from the user and move outward, checking one layer at a time.

You publish DNS records so receivers can check the sender. The mail server also signs outgoing messages.

The pod subnets have no free IPs left. You need more address space, or you need to use it better.

Use country blocking at the edge or in a WAF, so blocked traffic never reaches the app.

Only the load balancer is public. The application instances stay in private subnets with no public IPs.

First connect the networks. Then expose services in a controlled way.

IPsec has two phases. In IKE phase 1, the two sides prove who they are and build a secure channel. In phase 2, they create the IPsec tunnel that carries the traffic.

A public IP is given automatically and can change. An Elastic IP is a fixed public IPv4 address that you keep until you release it.

Resolvers and clients keep the old answer in their cache until its TTL runs out, so the change is not instant everywhere. This is DNS caching. People often wrongly call it propagation.

A 502 Bad Gateway means the load balancer got a bad answer, or no answer, from a backend. A 504 Gateway Timeout means the backend did not answer within the idle timeout.

Renew and install the certificate right away. Then add automation and alerts, so it cannot expire without notice.

The likely cause is running out of source ports on the NAT device. Each connection through PAT uses its own source port on the shared public IP. Each destination IP and port pair can use only about 55,000 ports at once.

This pattern usually means a path MTU problem. MTU is the biggest packet size a link allows. Packets bigger than some link's MTU are dropped. The ICMP "fragmentation needed" message that should tell the sender never arrives, because ICMP is blocked.

Split the request time into DNS, connect, TLS, wait for the first byte and download. Then see which part is big.

Sample answer:

"A developer creates a feature branch from main and opens a pull request. The PR starts CI: install with cache, lint, unit tests, security scans, and a build that makes a Docker image. The image is tagged with the commit SHA and pushed to [registry]. Reviewers approve, and branch protection requires that all checks pass.

When the PR is merged to main, the pipeline deploys the same image to dev automatically and runs smoke tests (quick basic checks). Then it moves to staging for integration tests and QA. Production needs a manual approval. It then rolls out with [rolling/canary] and is watched with [monitoring]. It rolls back automatically if health checks fail.

Config for each environment comes from Helm values or a parameter store. Secrets come from [secrets manager]. Infrastructure changes follow a separate Terraform pipeline. We use version tags for releases and keep a changelog."

Change the tools to match your real experience, but keep the flow: branch, PR, CI, artifact, promote, approve, deploy, verify.

What they are checking: Whether you can explain the whole flow with approval steps, moving one artifact up the environments, and rollback.

First measure which steps are slow. Then fix the biggest one.

• Cache dependencies (npm, pip, Maven) and Docker layers (BuildKit cache, registry cache).
• Order the Dockerfile layers so steps that rarely change (dependency install) come before the source copy. Use multi-stage builds and small base images.
• Run in parallel: split lint, unit tests and builds into jobs that run at the same time. Split long test suites into parts.
• Build only what changed: path filters, monorepo tools, and finding which services changed.
• Faster runners: bigger or self-hosted machines with ready caches.
• Skip what is not needed: do not rebuild for docs-only changes. Reuse the artifact across environments.
• Faster deploys: pull smaller images, tune the health checks and rolling-update settings, and keep images already pulled on nodes.

Report the numbers before and after. This shows that you worked step by step.

What they are checking: Whether you measure first and know caching, parallel jobs and builds that run only for changes.

Sample flow:

1. A developer pushes a version tag (for example v1.8.0) or merges to main.
2. CI (GitHub Actions or Jenkins) runs tests, builds the image and scans it. It pushes the image to the registry with the version and SHA as tags.
3. CI updates the image tag in the config repo, for example in envs/staging/values.yaml. It does this with a pull request or a bot commit.
4. Argo CD watches the config repo. It builds the Helm chart with the values file for that environment and syncs the change to the staging cluster.
5. Smoke tests run. Moving to production means a reviewed change to envs/prod/values.yaml, which Argo CD then applies.
6. Rollback is a Git revert, or an Argo CD sync to the previous revision.

Per-environment values (values-dev.yaml, values-prod.yaml) hold the replicas, resources, ingress hosts and config. Secrets come from an external secrets operator. If you build the image once and move the same tag up, all environments stay consistent.

What they are checking: Whether you can explain how CI passes work to GitOps, and how environment config is kept apart from the chart.

Plan rollback before releasing.

Rollback:

• Keep the previous artifacts: image tags by SHA, Helm revisions, ECS task definition revisions.
• Kubernetes: kubectl rollout undo deployment/app or helm rollback app 12. GitOps: revert the commit. Blue/green: switch traffic back.
• Automate it with health checks and alarms that start the rollback.

Tracking releases: use version tags, release notes, deployment events in the pipeline, and a deploy marker on dashboards. Then an incident can be linked to a change.

Database migrations:

• Use migration tools that keep versions (Flyway, Liquibase, Alembic).
• Follow expand and contract. First add columns or tables in a way that works with old code. Then deploy the code. Remove the old structures in a later release.
• Take a backup or snapshot before risky migrations. Do not ship changes that delete data in the same release as the code.
• Move forward with a fix when a rollback would lose data.

What they are checking: Whether you can roll back safely and know that database schema changes are the hard part.

Order matters: build once, scan, and push only what passes.

A self-hosted runner is a machine you manage. It registers with the CI service and takes jobs from it.

Avoid kubeconfig files that live for a long time. Use a short-lived identity with only the access it needs.

• Rolling update: replace instances a few at a time (the Kubernetes default). It is simple and needs no extra capacity. But old and new versions run together, and a rollback is a second rollout.
• Blue/green: run a full new environment (green) next to the current one (blue) and test it. Then switch traffic...

Make things stable first. Investigate second.

Immediate:

1. Roll back to the last good version (redeploy the previous artifact, run helm rollback, or switch the blue-green router). Do not debug live while users are getting errors.
2. Tell people the status and keep a timeline.

Unstable tests (tests that fail sometimes and pass other times) destroy trust in the pipeline. Treat them as bugs, not noise.

Assume it is stolen the moment it appears, and act in this order:

Start with what Argo CD itself says.

Use OpenID Connect (OIDC). GitHub gives a short-lived signed token to each workflow run. AWS swaps it for temporary credentials by assuming an IAM role. No long-lived key is stored anywhere.

Use the expand and contract method. In rolling, blue-green or canary releases, both versions use the same database.

Servers that run workloads should be in private subnets. Only load balancers and entry points like a bastion are public. Pipelines reach the servers without opening SSH to the whole world.

Do not copy nearly identical charts for every service.

Keep the pipeline logic in one place and let each project give only the settings.

Multi-cluster:

• Register clusters in Argo CD (argocd cluster add <context>), which saves a service account credential for each cluster.
• Use one Application per workload per cluster, or an ApplicationSet with a cluster generator, which creates the applications for every cluster automatically.
• Choose one central Argo CD for all clusters (simple, one place of control). Or choose one Argo CD for each cluster (clear separation, more to maintain).

Use EventBridge. It receives CodePipeline state-change events and can run a Lambda function.

A common setup:

Locally (quick, for tests): zip the code and run aws lambda update-function-code --function-name my-fn --zip-file fileb://function.zip, or use SAM (sam build && sam deploy), Serverless Framework or CDK.

Yes. Many hosting platforms (like Amplify) can build for you. But if you run the build in your own CI, you get more control, and the host only serves the output.

A queue means there are more jobs than executors, or builds cannot find a matching agent.

Automate it with metric analysis, so nobody has to watch the dashboard.

A shared library is a Git repository of reusable Groovy pipeline code that many Jenkinsfiles load. It removes copy and paste and keeps pipelines consistent across dozens of services.

The pipeline can deploy to production, so attackers want it.

Use sed, yq or kustomize to replace the image tag with the build's tag. Then apply it or commit it.

Break the loop by making the CI trigger ignore the commit that CI itself makes.

GitFlow uses long-lived branches with strict roles. Trunk-based development keeps everyone adding small changes to one main branch.

All three move the branch pointer to a target commit. They differ in what happens to the index and the working directory.

Treat the secret as stolen. Rotate it first, then clean the history, then stop it from happening again.

The commits still exist in someone's clone and in the reflog. Find the last good commit and move the branch back to it.

Use the reflog. It records every place HEAD and branch tips have been, for about 90 days by default.

Revert the bad commit on main so history stays intact. Ship the revert through the normal pipeline. Then fix the problem properly in a new change.

Hooks are scripts in .git/hooks that Git runs at specific events, so you can enforce rules automatically.

Treat infrastructure like application code.

Integrate more often and make Git remember resolutions.

• Merge or rebase main into the branch daily so each conflict is small.
• Break the work into smaller PRs, hide unfinished features behind flags and merge them to main early.
• Enable rerere so Git replays conflict resolutions you already did:

Use git bisect, which binary searches the history, so about 8 steps cover 200 commits.

Usually it is long history, large binaries committed to Git, or both, and there are both client-side and repo-side fixes.

Copy the commit to the correct branch, then remove it from the wrong one, provided it is not pushed.

Use an interactive rebase to squash, reword and reorder commits on your own branch, then force push safely.

Make main protected so the only way in is a reviewed, passing pull request.

A submodule links another repository at a fixed commit inside yours. A subtree copies the other repository's content into your history.

git worktree lets you check out several branches at the same time. Each one is in its own directory, and they share one repository.

--force-with-lease overwrites the remote branch only if it still points where you last saw it. So you cannot silently destroy a teammate's new commits.

Neither is always better. A monorepo gives atomic cross-project changes, shared tooling and easy refactors. Many repos give clear ownership, independent releases and simpler access control.

Signing proves the commit really came from the key holder. Author name and email are easy to fake. Platforms show a Verified badge.

Create the hotfix branch from what is in production (the main branch or the release tag), not from develop. Then merge it back everywhere.

Use OIDC federation. The CI system issues a short-lived signed token. AWS trusts it. The pipeline swaps it for temporary credentials. No secret is stored in the CI.

Setup:

1. In IAM, create an OIDC identity provider for the CI issuer (for example token.actions.githubusercontent.com).
2. Create an IAM role. Its trust policy allows sts:AssumeRoleWithWebIdentity from that provider, with conditions on the token's sub (repo and branch) and aud.
3. Attach least-privilege permissions to the role.
4. The pipeline requests a token and assumes the role.

permissions:
  id-token: write
  contents: read
steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::111111111111:role/ci-deploy
      aws-region: ap-south-1

For a registry or cluster in a different account, create the role in that account (or chain: assume a role in the target account). Allow the role in the ECR repository policy. Always pin the sub condition. Otherwise any repo could assume the role. Credentials expire in about an hour.

What they are checking: Whether you can replace static keys with federation and know to scope the trust policy tightly

Start with the legal basics. Sign a Business Associate Agreement (BAA) with AWS. Use only HIPAA-eligible services for anything that touches PHI (protected health information). Then build the controls around that.

• Account structure: a separate AWS account for PHI workloads under AWS Organizations. Use service control policies to block risky actions.
• Network: private subnets for compute and databases. No public database endpoints. VPC endpoints for AWS services. A WAF and load balancer at the edge.
• Encryption: TLS in transit. KMS encryption at rest for S3, EBS, RDS and backups, with customer managed keys and key rotation.
• Access: least-privilege IAM roles, MFA, SSO, and no long-lived keys on servers.
• Audit and monitoring: CloudTrail in all regions with log file validation, AWS Config rules, GuardDuty and CloudWatch alarms. Keep logs in a locked-down bucket.
• Data handling: keep as little PHI as possible. Mask or de-identify it in non-production. Define how long to keep data. Test backups and disaster recovery.

Being HIPAA-eligible does not make an application compliant. It is a shared responsibility model. AWS secures the platform. You must set up and run your part correctly.

What they are checking: Whether you can combine compliance requirements with concrete AWS controls and the shared responsibility model

Native Kubernetes Secrets are only base64-encoded. They are not encrypted. Treat them as a way to deliver secrets, and keep the real source somewhere else.

Options, from basic to better:

• Turn on encryption at rest for etcd (EKS: envelope encryption with KMS). Lock down RBAC so only a few people can get secrets.
• External Secrets Operator or the Secrets Store CSI driver sync from AWS Secrets Manager, SSM Parameter Store or Vault into pods. Rotation happens in one place.
• Sealed Secrets or SOPS let you commit encrypted secrets to Git for GitOps.
• Workload identity (IRSA or EKS Pod Identity) lets pods read the secret store with IAM roles, not stored keys.

Good practice:

• Mount secrets as files or env vars only in the pods that need them.
• Never bake them into images or Helm values in plain text.
• Use separate namespaces and RBAC per team.
• Rotate regularly and audit access.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
spec:
  secretStoreRef: {name: aws-sm, kind: ClusterSecretStore}
  target: {name: db-creds}
  data:
    - secretKey: password
      remoteRef: {key: prod/db, property: password}

What they are checking: Whether you know Kubernetes Secrets are not truly secret and can use an external store with workload identity

Use defence in depth: private networking, tight rules at each layer, encryption everywhere, and logging.

• Topology: public subnets hold only the load balancer. App and database tiers sit in private subnets with no public IPs. Reach AWS services through VPC endpoints.
• Security groups: chain them. The database allows only the app group. The app allows only the load balancer group.
• Load balancer: use HTTPS listeners with ACM certificates. Redirect HTTP to HTTPS. Use a modern TLS policy. Use WAF for OWASP rules and rate limiting. Limit origin access to the CDN. Turn on access logs.
• Encryption: TLS in transit, KMS at rest for databases and S3.
• Access: no direct DB access. Use SSM or VPN, IAM auth, and secrets in Secrets Manager.
• Sensitive data: classify it (Macie) and keep as little as possible. Mask it in logs. Limit it by data residency. Keep audit trails (CloudTrail, Config).
• Compliance: map controls to the standard (PCI, HIPAA, SOC 2). Automate checks with Config and Security Hub.

What they are checking: Whether you layer controls from edge to data rather than relying on one

Secure the whole path: build, registry, runtime and the host.

Create a dedicated service account. Give it a narrow role on the bucket, not on the project.

Give access through audited, temporary, identity-based paths. Do not use open ports and shared keys.

They work together: detection, vulnerability checks and one combined view.

Reduce exposure right away. Then do the proper fix.

S3 has no real folders. It has only key prefixes. So limit access with prefixes in IAM policies or bucket policies.

Cover the cluster, the workloads, the network and the supply chain.

Grant only the permissions needed, to only the identities that need them, for only as long as needed.

An identity-based policy is attached to the caller. It says what the caller can do. A resource-based policy is attached to the resource. It says who may use it.

Combine Kubernetes and network-level controls.

Make sure the only way in is through the CDN or proxy. Also harden the instance itself.

Attach roles so applications get temporary credentials automatically, with no access keys in code.

EC2: create a role with the permissions, wrap it in an instance profile and attach to the instance. The SDK picks credentials from the instance metadata service. Enforce IMDSv2.

Set up the foundations first, because they are hard to change later.

Keep private resources (databases, internal APIs) in private subnets, and expose only a controlled entry point.

SOC 2 checks whether your controls meet the Trust Services Criteria (security, availability, confidentiality, processing integrity, privacy). It is about proving steady practice, not only having tools.

A common open source answer is Prometheus, Grafana, Loki and Alertmanager.

• Metrics: Node Exporter for hosts, kube-state-metrics and cAdvisor for Kubernetes, and client libraries for application metrics. Prometheus scrapes (pulls) them. On Kubernetes, use the kube-prometheus-stack Helm chart.
• Logs: Promtail or Grafana Alloy sends container and system logs to Loki. Query them with LogQL in Grafana.
• Dashboards: Grafana with Prometheus and Loki as data sources.
• Alerting: Prometheus alert rules fire to Alertmanager. It groups alerts, removes duplicates, and sends them to Slack, email or PagerDuty by severity and team.
• Good alerts: alert on symptoms (error rate, latency, pod restarts, pending pods, disk filling within 4 hours). Do not alert on every CPU spike. Each alert should link to a runbook.
• Managed option: CloudWatch with SNS, or Datadog, when you do not want to run the stack yourself.

- alert: HighErrorRate
  expr: sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
  for: 10m
  labels: {severity: page}

What they are checking: Whether you have really built an observability stack and know how alerts reach people.

First make things stable, then find the cause, then recover. Do not reboot blindly, because you may lose evidence.

• Check status: look at the EC2 console status checks. A failed system check points to the AWS host. Stop and start the instance to move it to new hardware. A failed instance check points to the OS (kernel panic, full disk, bad fstab).
• Metrics and logs: look at CloudWatch CPU, credit balance, disk and network graphs. Also read the console output and screenshot (aws ec2 get-console-output --instance-id i-0abc).
• If SSH fails: try SSM Session Manager first. If that fails, stop the instance and detach the root volume. Attach it to a healthy rescue instance and mount it. Then read /var/log, fix fstab, free disk space or fix sshd_config. Finally reattach the volume and start the instance.
• If it is just too small: stop it, change the instance type, and start it. Or snapshot the volume and launch a larger instance from it.
• Avoid a repeat: set alarms on CPU, disk and status checks. Use an Auto Scaling group with health checks so it heals itself. Hold a short review after the incident.

aws ec2 describe-instance-status --instance-ids i-0abc --include-all-instances

What they are checking: Whether you have a calm, ordered recovery process and know the detach-volume rescue trick.

Start with the user impact, then narrow down layer by layer using data, not guesses.

1. Scope it: which endpoint, since when, which users? Check recent deploys or config changes.
2. Golden signals (latency, traffic, errors, saturation) on a Grafana or CloudWatch dashboard.
3. Resources: CPU throttling, memory pressure and swap, disk I/O, network, connection limits.
4. Distributed tracing (OpenTelemetry with Jaeger or Tempo, or X-Ray): the slowest span shows which service or call is guilty.
5. Dependencies: slow SQL (slow query log, pg_stat_statements), locks, cache misses, third-party APIs, DNS.
6. Logs: correlate by request ID, look for timeouts and retries.
7. Fix and verify: add an index, cache, scale out (HPA), tune pool sizes, or roll back. Then confirm the latency graph recovered.

The key is to change one thing at a time and measure.

What they are checking: Whether you follow a structured, data driven approach and know tracing.

Aurora is AWS's cloud native engine (compatible with MySQL and PostgreSQL). Standard RDS runs the regular engine on EBS volumes.

Writer crash in Aurora: the cluster detects the failure and promotes a reader to writer automatically. It picks by failover priority tier (then size). Readers share storage, so no data copy is needed. The cluster endpoint then points at the new writer, so applications that reconnect keep working. The failed instance is replaced and joins again as a reader. Clients must retry dropped connections. They must use the cluster (writer) endpoint and the reader endpoint, not instance endpoints.

For RDS Multi-AZ, the standby is promoted and DNS flips. So keep the DNS TTL low and retry connections.

What they are checking: Whether you know the shared storage design and how failover is exposed to applications.

Look at what the database is doing. Then fix the query or the connection handling.

Make costs visible per team. Set budgets. Then act on the biggest items.

Run active copies in each region. Send users to the nearest healthy one. Replicate data.

Default to a managed database outside the cluster, and self-host only with a good reason.

Choose by data shape, consistency needs and access pattern.

Treat AI tools like any other powerful third party that can reach your code and systems.

Pick the service by how much control you need.

They measure different layers.

Alert from metrics. Then investigate with kubectl.

Check each hop between the two, starting at the backend.

Use the exit information Kubernetes already records.

HPA adds or removes pod replicas. VPA changes the CPU and memory requests of each pod.

Find the blocker. Decide whether to kill it. Stop it from happening again.

Most problems come from what an EC2 app assumes about its machine.

Find out which layer is still serving old content.

Separate the resolver, the DNS records and the network path.

Restore service first. Find the root cause second.

Retention is set per log group, and search uses Logs Insights or filter patterns.

• Retention: by default logs never expire, which costs money. Set a retention period in the console, with Terraform (retention_in_days), or the CLI:

Ship logs from every host and pod to one searchable store.

• Collect: Fluent Bit, Promtail or Grafana Alloy as a DaemonSet or agent, tagging logs with labels like namespace, pod and app. Prefer structured JSON logs with a request ID.
• Store and search: Loki or Elasticsearch/OpenSearch.

Instrument with OpenTelemetry, collect with a collector, and view in a tracing backend.

Check from outside the system, as a user would. Alert on failures.

Aurora Global Database spans multiple regions. It has one primary region and read-only secondary regions. It copies data at the storage level.

Put RDS in private subnets. Never expose it to the internet.

Combine automated backups, copies in another region and account, and tested restores.

Choose by recovery targets: how much data you can lose (RPO) and how fast you must be back (RTO).

DynamoDB queries must use a key, so indexes give you extra access patterns.

Describe the path from model to a secure, scalable endpoint.

Automate it from a template instead of clicking 50 times.

• Template: build one VM, install the OS and agents, generalise it (cloud-init or sysprep), and convert it to a template.
• Terraform with the vSphere provider:

Make the environments hard to confuse and hard to touch by accident.

Scale turns small issues into daily problems.

First find which role is making the call, and which policy is missing or blocking it.

Give a short map of what you built. Then name the tooling.

Start with a typical web application. Add each requirement as a layer.

Architecture

• Route 53 and CloudFront (with WAF) at the edge, caching static content.
• ALB across 3 AZs. Targets are in an Auto Scaling group (or ECS/EKS) in private subnets. Scale on CPU or request count.
• Aurora or RDS Multi-AZ for the database with read replicas, ElastiCache for hot data, S3 for objects.
• Stateless app tier, config in SSM/Secrets Manager, infrastructure in Terraform.

Recovery and DR

• Health checks, retries with backoff, timeouts, circuit breakers.
• For low RPO and RTO beyond one region: use Aurora Global Database or a cross-region read replica. Add S3 cross-region replication, infrastructure code ready for a second region, and Route 53 failover. Warm standby gives an RTO of minutes and an RPO of seconds.
• Regular restore tests and game days.

Cost

• Right-size instances.
• Use Savings Plans for the baseline.
• Use Spot for stateless or batch workers.
• Use S3 lifecycle tiers.
• Scale in at night.
• Keep the standby region small.

State the cost link: lower RTO/RPO costs more. Pick targets from the business impact.

What they are checking: Whether you can combine HA, scaling, DR and cost into one clear design and discuss the pros and cons

Three tiers: presentation, application and data. Each is in its own subnet or managed service.

• Edge:
  • A global external Application Load Balancer with a static IP.
  • Cloud CDN for static assets.
  • Cloud Armor for WAF and rate limits.
  • Managed SSL certificates.
• Web/app tier: a regional managed instance group (MIG) spread across zones, or Cloud Run/GKE. Use an autoscaling policy based on CPU use or load balancer requests per instance. Add autohealing health checks. Instances have no external IPs.
• Data tier: Cloud SQL with high availability (regional, standby in another zone) and read replicas, or Spanner/Firestore. Memorystore for caching. Cloud Storage for files.
• Network:
  • Custom-mode VPC with private subnets.
  • Cloud NAT for outbound traffic.
  • Firewall rules by network tag or service account (frontend can reach app, app can reach DB only).
  • Private Service Connect or private IP for Cloud SQL.
• Operations: Cloud Monitoring and Logging, instance templates built by CI, Terraform for everything.

Explain scaling: the stateless app tier scales horizontally. The database scales vertically, plus replicas.

What they are checking: Whether you can map the three-tier pattern to GCP services and explain autoscaling and network isolation

RPO (Recovery Point Objective) is how much data you can afford to lose, measured in time since the last good copy. RTO (Recovery Time Objective) is how long you can be down before service is back.

They come from the business impact. They then decide the strategy and the cost:

Plan for a single region failure on AWS:

• Copy data to another region (S3 CRR, Aurora Global Database, DynamoDB global tables, copied AMIs and ECR images).
• Keep infrastructure as code ready to deploy in the second region.
• Use Route 53 health checks and failover records.
• Write runbooks and test failovers regularly.

A DR plan that is never tested is a guess.

What they are checking: Whether you can define RPO/RTO precisely and match DR tiers to them

Use separate accounts (AWS) or projects/folders (GCP) as the main isolation boundary. They limit the damage from a problem, and they separate permissions and billing.

AWS structure with Organizations:

• Management account (billing and governance only).
• Security / log-archive accounts (CloudTrail, Config, GuardDuty aggregation).
• Shared services / network account (Transit Gateway, DNS, CI runners, ECR).
• Workload accounts per environment (dev, staging, prod), possibly per team or product.
• SCPs set guardrails per organisational unit. IAM Identity Center gives role-based access.

GCP equivalent: organization, folders per environment or team, one project per workload, Shared VPC from a host project, and org policies for guardrails.

Why clusters per environment: prod isolation, independent upgrades, separate quotas and cost reports. For thousands of machines, also use naming and labelling standards, IaC modules and pipelines per environment, and central monitoring. Avoid sharing databases across environments.

Cost of this: more accounts mean more overhead (networking, access). So automate with Terraform and landing-zone tooling (Control Tower).

What they are checking: Whether you understand account-level isolation, guardrails and the cost of operating many accounts

Match the method to how the data changes. Follow 3-2-1: three copies, two storage types, one off-site (another region or account).

• Databases: automated backups plus point-in-time recovery (RDS/Aurora, continuous transaction logs, restore to any second in the retention window). Add manual snapshots before risky changes. For copies with no downtime, use read replicas or snapshots instead of dumping the primary.
• Block storage: EBS snapshots (incremental) through AWS Backup or Data Lifecycle Manager on a schedule.
• Objects: S3 versioning plus cross-region replication. Object Lock protects against ransomware.
• Cross-region: copy snapshots to another region and, ideally, another account with a separate KMS key. Then a hacked account cannot delete them. AWS Backup supports cross-region and cross-account copy rules.
• Retention: lifecycle rules that move old backups to cheaper tiers.
• Verify: restore tests and alerts on failed jobs. A backup you have never restored is unproven.

Choose the frequency from the RPO. If you can lose 5 minutes, snapshots every night are not enough.

What they are checking: Whether you know PITR, snapshots, cross-account copies and the need to test restores

The queue is doing its job (it absorbs the burst). The fix is to control the consumption rate, not to speed up the consumers.

• Limit worker concurrency so total calls to the API stay under 20 per minute. A single worker with a rate limiter or token bucket, or a small fixed pool, enforces this. Do not autoscale consumers on queue depth blindly, because that makes the problem worse.
• Slow down in the worker: sleep or use a rate-limiting library. Respect 429 and Retry-After responses.
• Retry with exponential backoff and jitter (random delay). Send messages that keep failing to a dead-letter queue.
• Visibility timeout longer than the processing plus wait time, so messages are not delivered again while waiting.
• Idempotent processing (running it twice gives the same result), so retries are safe.
• Capacity maths: 2000 messages at 20 per minute takes about 100 minutes. If that is too slow, ask the API owner to raise the limit, batch requests, or put important messages in a separate queue.
• Alert on queue age (oldest message), not only depth.

What they are checking: Whether you throttle at the consumer and use backoff, DLQs and idempotency instead of just scaling out

Cross-AZ is simple and should be the default. Multi-region is a deliberate step up with real cost and complexity.

Cross-AZ: subnets and instances in 2 to 3 AZs, a load balancer across them, a Multi-AZ database, and stateless services.

Choose by data size, bandwidth and deadline.

Keep the cheap, stateless application tier apart from the expensive GPU workers. Let a queue connect them.

A 3 to 4 second response is impossible if each request pays the load time. So the model must already be loaded when requests arrive.

An API gateway is the single entry point for clients. It receives external requests, handles shared tasks, and sends each request to the right internal service.

Flow: client to gateway (HTTPS) to service (HTTP or gRPC over the internal network) and back.

Compute Engine and GKE need real subnets with IP ranges you plan up front. Serverless (Cloud Run, Cloud Functions) needs little, unless it must reach private resources.

At this size, full restores take days. So design around incremental, replicated and tiered protection and a realistic RTO.

An RTO and RPO of 1 hour is relaxed enough for pilot light or warm standby in a second region. You do not need to pay for active-active.

Typical flow: data lands in S3. Scheduled or event-driven ECS tasks transform it. Results load into a warehouse (Redshift, BigQuery or Snowflake).

Build it with Terraform, in this order.

Answer in three parts: how traffic flows, how the migration was done, and what changed. Use your own project. Here is a typical structure.

Structure: role and context, daily work, scale, impact, then the link to the new role.

Sample answer: "I am a [title] with [X] years of experience. I work at [company type], where I look after [environment, for example a multi-account AWS setup running Kubernetes]. My typical day covers three things. First, keeping systems healthy: I check dashboards and alerts, and handle incidents and tickets. Second, delivery: I maintain CI/CD pipelines, review Terraform changes, and support releases. Third, improvement work: cost savings, security fixes, and automation to remove manual steps. Recently I [achievement with a number, for example cut deployment time from 40 to 10 minutes]. I also [mentor juniors or write runbooks]. I want to take on [bigger scope the new role offers]."

Tips:

• Mention real tools (Terraform, Kubernetes, GitHub Actions, CloudWatch or Prometheus) and the scale (services, environments, traffic).
• Show ownership and results, not only tasks.
• Mention networking, DNS, certificates or monitoring only if the job needs them.

What they are checking: Whether your real responsibilities, scale and impact match the level you claim.

Group your answer by service area and say what you personally did.

Sample answer: "I have worked with AWS for [X] years. For compute I have run [EC2, ECS or EKS] workloads with autoscaling. For networking I designed VPCs with public and private subnets, NAT, security groups and Route 53. For storage and data I used S3, RDS and DynamoDB, with lifecycle rules and backups. For security I managed IAM roles and policies with least privilege, SSO and KMS. Everything is defined in Terraform, and deployed through CI/CD. For observability I set up CloudWatch metrics, alarms and logs. My own responsibilities are [for example on-call, cost reviews, building new environments]. One example: [project], where I [result]."

Prepare one deep story per area, because follow-up questions will test the details of what you claim.

What they are checking: Whether your depth matches your CV and you can back claims with examples.

State real numbers. Then explain how you keep the work manageable.

Sample answer: "I support [number] services across [number] AWS accounts and [environments]. That is about [number] servers or nodes and [number] databases, with roughly [traffic or users]. To manage this scale, everything is in Terraform. Config is standard through shared modules and images. Deployments run through pipelines. We use central monitoring and alerting. Patching and backups are automated. Runbooks cover the common failures. Standard tagging and cost reports keep spend visible."

Make sure the numbers are honest. Be ready to explain how you would handle growth.

What they are checking: Whether you have worked at real scale and rely on automation instead of manual work.

Be positive, honest and forward-looking. Never criticise your employer.

Structure: what you have gained, what you want next, why this company.

Sample answer: "I have learned a lot at [current company], especially [skill or achievement]. Now I want a role with [larger scale, more ownership, newer technology, or a platform engineering focus]. I cannot get that in my current position. This role matches because [specific reason from the job description]. I want to bring my experience in [your strength] and keep growing in [area]."

Avoid complaints about pay, your manager or colleagues. Avoid vague answers. Avoid reasons that suggest you leave quickly.

What they are checking: Whether your motivation is constructive and fits the role.

Use STAR: Situation, Task, Action, Result. Then add the lesson and how you stopped it from happening again.

Adopt it with evidence and shared ownership.

Use STAR. Stress planning, risk control and the result.

Cover the platform decisions, the delivery process and the results.

Describe the method: measure, find the biggest items, fix, and keep watching.

Give one or two real examples. Explain what you got out of them.