DevOps System Design Interview Questions and Answers

High availability (HA) means a system keeps working when a part fails, so downtime is small. It is usually given as uptime. 99.9% allows about 8.7 hours of downtime a year. 99.99% allows about 52 minutes.

Remove every single point of failure:

• Multiple AZs: run instances in at least two AZs behind a load balancer with health checks. Then traffic skips a failed zone.
• Auto Scaling group across AZs replaces failed instances automatically.
• Database: Multi-AZ RDS (synchronous standby, automatic failover) or Aurora with replicas.
• Stateless apps: keep sessions in Redis or a database, and files in S3. Then any instance can serve any request.
• Managed services that are already multi-AZ (S3, DynamoDB, SQS).
• Monitoring and automatic recovery (CloudWatch alarms, health checks).

HA protects against instance and AZ failure. A full region outage needs disaster recovery. That is a separate plan.

What they are checking: Whether you know HA means removing single points of failure and can name the AWS building blocks

For a purely static React build, S3 + CloudFront is usually the better choice. It is cheap and highly available. There are no servers to patch. It has a global CDN and almost nothing to scale. No load balancer is needed.

Reasons to run it in containers instead:

• Server-side rendering (Next.js) or a Node server that needs a runtime.
• The team already runs everything on one platform and wants one deployment and monitoring model.
• Same-origin routing to backend services, custom headers, or auth middleware at the app layer.
• Strict network requirements such as private-only access.

Cost view: a small static site on EKS pays for the control plane, nodes and a load balancer. S3 + CloudFront costs a few dollars. EKS is rarely cost-effective for a simple frontend.

Load balancers are needed in front of container or EC2 services (an ALB, or an Ingress on Kubernetes). They spread traffic and run health checks. Use an internal ALB for a private backend.

What they are checking: Whether you choose the simplest cost-effective hosting and justify containers only when needed

Layout in one VPC (for example 10.0.0.0/16), across two AZs.

• Public subnets: the Application Load Balancer and NAT gateways. Route table: 0.0.0.0/0 to the Internet Gateway.
• Private app subnets: EC2 instances (in an Auto Scaling group) running the backend. Route table: 0.0.0.0/0 to the NAT gateway. This lets them download updates and call external APIs. But nothing from the internet can reach them directly.
• Private data subnets: RDS (Multi-AZ), with no internet route.
• Frontend: host the React build on S3 + CloudFront. API calls go to the ALB, which passes them to the private backend.
• Security groups: the ALB allows 443 from the internet. The backend allows its port only from the ALB group. The database allows 5432/3306 only from the backend group.
• Access: SSM Session Manager instead of a public bastion.

What they are checking: Whether you understand subnet types, routing and why private tiers use a NAT gateway

A clear way to explain it is to follow the flow from commit to running traffic.

1. Terraform creates the VPC, EKS cluster, ECR, RDS and IAM roles, with state in S3.
2. A developer pushes code to GitHub.
3. GitHub Actions runs tests and linting. It builds the Docker image, scans it (Trivy), tags it with the commit SHA and pushes it to ECR. It signs in to AWS with OIDC, so no keys are stored.
4. Deployment: the pipeline updates the Kubernetes manifests or Helm values with the new tag and applies them. Or Argo CD syncs from Git, which is GitOps. A rolling update replaces pods, guarded by health checks and readiness probes.
5. Traffic: Route 53 sends users to the ALB (Ingress controller). It sends / to the frontend service and /api to the backend service. The backend talks to RDS in private subnets.
6. Operations: Prometheus and Grafana or CloudWatch for metrics, HPA for autoscaling.

What they are checking: Whether you can narrate a full delivery pipeline end to end and name the tools' roles

Start with a typical web application. Add each requirement as a layer.

Architecture

• Route 53 and CloudFront (with WAF) at the edge, caching static content.
• ALB across 3 AZs. Targets are in an Auto Scaling group (or ECS/EKS) in private subnets. Scale on CPU or request count.
• Aurora or RDS Multi-AZ for the database with read replicas, ElastiCache for hot data, S3 for objects.
• Stateless app tier, config in SSM/Secrets Manager, infrastructure in Terraform.

Recovery and DR

• Health checks, retries with backoff, timeouts, circuit breakers.
• For low RPO and RTO beyond one region: use Aurora Global Database or a cross-region read replica. Add S3 cross-region replication, infrastructure code ready for a second region, and Route 53 failover. Warm standby gives an RTO of minutes and an RPO of seconds.
• Regular restore tests and game days.

Cost

• Right-size instances.
• Use Savings Plans for the baseline.
• Use Spot for stateless or batch workers.
• Use S3 lifecycle tiers.
• Scale in at night.
• Keep the standby region small.

State the cost link: lower RTO/RPO costs more. Pick targets from the business impact.

What they are checking: Whether you can combine HA, scaling, DR and cost into one clear design and discuss the pros and cons

Three tiers: presentation, application and data. Each is in its own subnet or managed service.

• Edge:
  • A global external Application Load Balancer with a static IP.
  • Cloud CDN for static assets.
  • Cloud Armor for WAF and rate limits.
  • Managed SSL certificates.
• Web/app tier: a regional managed instance group (MIG) spread across zones, or Cloud Run/GKE. Use an autoscaling policy based on CPU use or load balancer requests per instance. Add autohealing health checks. Instances have no external IPs.
• Data tier: Cloud SQL with high availability (regional, standby in another zone) and read replicas, or Spanner/Firestore. Memorystore for caching. Cloud Storage for files.
• Network:
  • Custom-mode VPC with private subnets.
  • Cloud NAT for outbound traffic.
  • Firewall rules by network tag or service account (frontend can reach app, app can reach DB only).
  • Private Service Connect or private IP for Cloud SQL.
• Operations: Cloud Monitoring and Logging, instance templates built by CI, Terraform for everything.

Explain scaling: the stateless app tier scales horizontally. The database scales vertically, plus replicas.

What they are checking: Whether you can map the three-tier pattern to GCP services and explain autoscaling and network isolation

RPO (Recovery Point Objective) is how much data you can afford to lose, measured in time since the last good copy. RTO (Recovery Time Objective) is how long you can be down before service is back.

They come from the business impact. They then decide the strategy and the cost:

Plan for a single region failure on AWS:

• Copy data to another region (S3 CRR, Aurora Global Database, DynamoDB global tables, copied AMIs and ECR images).
• Keep infrastructure as code ready to deploy in the second region.
• Use Route 53 health checks and failover records.
• Write runbooks and test failovers regularly.

A DR plan that is never tested is a guess.

What they are checking: Whether you can define RPO/RTO precisely and match DR tiers to them

Use separate accounts (AWS) or projects/folders (GCP) as the main isolation boundary. They limit the damage from a problem, and they separate permissions and billing.

AWS structure with Organizations:

• Management account (billing and governance only).
• Security / log-archive accounts (CloudTrail, Config, GuardDuty aggregation).
• Shared services / network account (Transit Gateway, DNS, CI runners, ECR).
• Workload accounts per environment (dev, staging, prod), possibly per team or product.
• SCPs set guardrails per organisational unit. IAM Identity Center gives role-based access.

GCP equivalent: organization, folders per environment or team, one project per workload, Shared VPC from a host project, and org policies for guardrails.

Why clusters per environment: prod isolation, independent upgrades, separate quotas and cost reports. For thousands of machines, also use naming and labelling standards, IaC modules and pipelines per environment, and central monitoring. Avoid sharing databases across environments.

Cost of this: more accounts mean more overhead (networking, access). So automate with Terraform and landing-zone tooling (Control Tower).

What they are checking: Whether you understand account-level isolation, guardrails and the cost of operating many accounts

Match the method to how the data changes. Follow 3-2-1: three copies, two storage types, one off-site (another region or account).

• Databases: automated backups plus point-in-time recovery (RDS/Aurora, continuous transaction logs, restore to any second in the retention window). Add manual snapshots before risky changes. For copies with no downtime, use read replicas or snapshots instead of dumping the primary.
• Block storage: EBS snapshots (incremental) through AWS Backup or Data Lifecycle Manager on a schedule.
• Objects: S3 versioning plus cross-region replication. Object Lock protects against ransomware.
• Cross-region: copy snapshots to another region and, ideally, another account with a separate KMS key. Then a hacked account cannot delete them. AWS Backup supports cross-region and cross-account copy rules.
• Retention: lifecycle rules that move old backups to cheaper tiers.
• Verify: restore tests and alerts on failed jobs. A backup you have never restored is unproven.

Choose the frequency from the RPO. If you can lose 5 minutes, snapshots every night are not enough.

What they are checking: Whether you know PITR, snapshots, cross-account copies and the need to test restores

The queue is doing its job (it absorbs the burst). The fix is to control the consumption rate, not to speed up the consumers.

• Limit worker concurrency so total calls to the API stay under 20 per minute. A single worker with a rate limiter or token bucket, or a small fixed pool, enforces this. Do not autoscale consumers on queue depth blindly, because that makes the problem worse.
• Slow down in the worker: sleep or use a rate-limiting library. Respect 429 and Retry-After responses.
• Retry with exponential backoff and jitter (random delay). Send messages that keep failing to a dead-letter queue.
• Visibility timeout longer than the processing plus wait time, so messages are not delivered again while waiting.
• Idempotent processing (running it twice gives the same result), so retries are safe.
• Capacity maths: 2000 messages at 20 per minute takes about 100 minutes. If that is too slow, ask the API owner to raise the limit, batch requests, or put important messages in a separate queue.
• Alert on queue age (oldest message), not only depth.

What they are checking: Whether you throttle at the consumer and use backoff, DLQs and idempotency instead of just scaling out

Cross-AZ is simple and should be the default. Multi-region is a deliberate step up with real cost and complexity.

Cross-AZ: subnets and instances in 2 to 3 AZs, a load balancer across them, a Multi-AZ database, and stateless services.

Choose by data size, bandwidth and deadline.

Keep the cheap, stateless application tier apart from the expensive GPU workers. Let a queue connect them.

A 3 to 4 second response is impossible if each request pays the load time. So the model must already be loaded when requests arrive.

An API gateway is the single entry point for clients. It receives external requests, handles shared tasks, and sends each request to the right internal service.

Flow: client to gateway (HTTPS) to service (HTTP or gRPC over the internal network) and back.

Compute Engine and GKE need real subnets with IP ranges you plan up front. Serverless (Cloud Run, Cloud Functions) needs little, unless it must reach private resources.

At this size, full restores take days. So design around incremental, replicated and tiered protection and a realistic RTO.

An RTO and RPO of 1 hour is relaxed enough for pilot light or warm standby in a second region. You do not need to pay for active-active.

Typical flow: data lands in S3. Scheduled or event-driven ECS tasks transform it. Results load into a warehouse (Redshift, BigQuery or Snowflake).

Build it with Terraform, in this order.

Answer in three parts: how traffic flows, how the migration was done, and what changed. Use your own project. Here is a typical structure.