Microservices Architecture Patterns and Pitfalls

The conclusion up front: microservices are not a silver bullet. They are an expensive trade in which you exchange distributed-systems complexity for independent scaling and team autonomy. Get the decomposition right, and they can support engineering organizations with hundreds or thousands of people. Get it wrong, and you will miss the monolithic repository where a git push was all it took to deploy.

This article breaks down the core microservices patterns and lays out the most common pitfalls I have encountered.

1. The Essence and Cost of Microservices

The core definition of microservices has only two parts:

• Independent deployment: Each service can be built and released independently without holding up any other service.
• Loose coupling: Services interact through explicit interfaces (APIs or events); they share neither databases nor internal models.

What you gain is freedom to choose technology stacks, fault isolation, on-demand scaling, and clear team boundaries.

The costs are often invisible to newcomers and surface only six months after launch:

• Networks are unreliable: timeouts, retries, and cascading failures can happen at any time.
• Data consistency changes from "one transaction" to "eventual consistency across services."
• Debugging changes from "set a breakpoint" to "search the logs of ten services."
• CI/CD, configuration, service discovery, and observability all impose new infrastructure costs.

In short: a monolith puts complexity into code; microservices spread it across operations. If your team lacks a solid DevOps foundation, microservices will only amplify the disorder.

2. Service Boundaries: The Hardest Step

Service decomposition is the one microservices decision where a single mistake can leave you paying down debt for years. Follow three principles:

1. Split by business capability, not technical layer. Putting UserController / OrderController in separate services that call each other is an antipattern because every HTTP call upgrades "one in-memory method call" into "one network round trip."
2. One database per service (Database per service). A shared database creates hidden coupling, which means you have not truly separated the services.
3. Domain-driven design (DDD) is the foundational tool. Use a Bounded Context to define each boundary, with strong consistency inside a boundary and weak consistency across boundaries.

┌─────────────────────────────────────────────────────────────┐
│                         API Gateway                         │
└──────┬───────────┬───────────┬───────────┬───────────┬──────┘
       │           │           │           │           │
   ┌───▼────┐  ┌───▼────┐  ┌───▼────┐  ┌───▼────┐  ┌───▼────┐
   │  User  │  │ Product│  │  Order │  │Inventory│ │ Payment│
   │  DB A  │  │  DB B  │  │  DB C  │  │  DB D  │  │  DB E  │
   └────────┘  └────────┘  └────────┘  └────────┘  └────────┘

The trap is the nanoservice: splitting so finely that placing one order traverses eight services. The result is a request path five times slower, troubleshooting that requires people from five teams, and "autonomy" that turns into finger-pointing.

A practical threshold: a service should map to at least one complete business subdomain and be owned end to end by one 2-Pizza Team (6-8 people). Anything smaller should be merged back.

3. Interservice Communication: Synchronous or Asynchronous

The communication method determines the system's coupling and fault tolerance.

Practical recommendation: use synchronous calls for critical paths such as ordering and payment, and asynchronous events for supporting paths such as notifications, recommendations, reporting, and reconciliation. For example, "create order" synchronously returns a result to the user, while "send SMS notification," "update points," and "sync to the data warehouse" all use asynchronous events, with downstream services subscribing to the OrderCreated event.

Order API  ──synchronous──▶ Payment Service ──▶ Decrement Inventory
   │
   └──publish OrderCreated event──▶ [Kafka]
                                      ├──▶ Notification Service (SMS)
                                      ├──▶ Points Service
                                      └──▶ Risk Control Service

Synchronous calls must always include timeouts + circuit breakers + graceful degradation using tools such as Hystrix/Resilience4j/Sentinel. Otherwise, one slow downstream service can fill every upstream thread pool, causing a cascading failure within seconds.

4. API Gateway: A Unified Entry Point

Exposing dozens of service addresses to external clients is unacceptable. The API gateway is the single entry point for all external requests. Its core responsibilities are:

• Routing: Send /api/orders/* to the order service cluster.
• Authentication: Centralize JWT/Session validation so business services do not each reimplement it.
• Rate limiting: Protect the backend from sudden traffic spikes.
• Protocol translation: Use HTTP externally and gRPC internally.
• Aggregation: Fetch data from multiple services in parallel for one request, so the frontend makes only one call.

Common choices include Kong, APISIX, Envoy, Nginx + Lua, and Spring Cloud Gateway.

Pitfall: Do not put business logic in the gateway. The gateway should be a "dumb router." Once it grows "order validation logic," it becomes a second monolith that is even harder to test.

5. Service Discovery: Where Is the Service?

Service instances scale dynamically and their IP addresses constantly change, so callers cannot hard-code addresses. There are two mainstream approaches:

1. Server-side discovery: With Kubernetes Service / DNS, callers use a stable domain name such as order-svc.default.svc.cluster.local, and K8s handles load balancing automatically. It requires no extra dependency and is the preferred cloud-native option.
2. Client-side discovery: Consul / Eureka / Nacos acts as a service registry. Services register at startup, and callers fetch the instance list from the registry and perform load balancing themselves. This is flexible and works well in self-hosted data centers and heterogeneous language environments.

Service A starts ──registers──▶ Service Registry (Consul)
                                      ▲
Service B calls A ──gets instance list─┘ ──direct call─▶ Service A instances (1/2/3)

Pitfall: the service registry itself must be highly available. Eureka's AP mode permits inconsistent node data in exchange for availability, while Consul defaults to CP (strong consistency). Know which semantics you are using: AP may route calls to an instance that has already gone offline, with circuit breakers as the fallback; CP may reject registrations during a partition.

6. Distributed Transactions: Do Not Demand Strong Consistency Across Services

Strong consistency across services using 2PC/XA is effectively impractical in engineering: it locks resources, causes performance to collapse, and lets any failed node block the entire operation. Microservices accept eventual consistency by default, with three implementation patterns:

1. Saga orchestration: Break one long transaction into N local transactions + N compensating actions. Submit order → deduct inventory → debit balance → create order; if any step fails, execute the compensating actions in reverse order.
2. TCC (Try-Confirm-Cancel): Implement two phases in the business layer, with each action exposing three interfaces: "reserve + confirm + cancel." It provides better consistency but intrudes more deeply into business logic.
3. Event-driven processing + transactional outbox: In one local transaction, the service writes both business data and an event to an outbox table. A background worker delivers it to the MQ, and downstream consumers enforce idempotency.

Local transaction ┌─write business table
                  └─write outbox table ──worker──▶ MQ ──▶ downstream consumption (idempotent)

My recommended default is a transactional outbox + event-driven processing. It requires no additional coordinator, minimally intrudes on business logic, and is familiar to most engineers. The tradeoff is that you must design idempotency yourself: the same event may be delivered multiple times, so consumers must deduplicate it using a unique business key.

Hard-won lesson: Cross-service consistency is never a technical problem; it is a business problem. First ask whether the business can accept "the order succeeded, but the SMS arrived 30 seconds late." If it can, do not adopt TCC.

7. Observability: The Lifeline of Microservices

In a monolith, tail -f log.txt may be enough for debugging. In microservices, one request can traverse ten services, and without observability you are blind.

The three pillars are:

• Metrics: QPS, latency percentiles (p99), error rates, and JVM/Go runtime metrics. Prometheus + Grafana is the de facto standard.
• Logs: Centralize them in ELK / Loki, and every log entry must include a trace_id. Otherwise, troubleshooting becomes guesswork with grep.
• Distributed tracing (Trace): With Jaeger / Tempo / SkyWalking, one trace_id spans every service, and a waterfall view reveals the slow point.

Client ──trace_id=abc──▶ Gateway ──▶ OrderSvc ──▶ PaySvc
                                          └──▶ StockSvc (800ms slow)
       trace_id=abc links every span; the p99 anomaly points immediately to StockSvc

Alerts should target SLOs, such as "order-placement p99 remains below 500ms 99.9% of the time," instead of noisy alerts such as "CPU > 80%." Microservices without SLOs are a black box.

Summary

• Boundaries are the hardest microservices problem. Split by business capability, use DDD to define Bounded Contexts, give each service its own database, and avoid nanoservices at all costs.
• Communication: Use synchronous calls with circuit breakers for critical paths and asynchronous events for supporting paths; accept eventual consistency by default.
• Do not skimp on infrastructure: An API gateway, service discovery, unified tracing, centralized configuration, and CI/CD are the foundation. Miss one, and you cannot build reliably on top.
• Transactions: Strong consistency across services is an antipattern. Choose Saga / TCC / a transactional outbox according to consistency requirements, with event-driven processing as the first choice.
• Observability: Metrics + Logs + Trace must work together. Alert around SLOs, and propagate trace_id across the entire request path.

Key takeaway: Most teams should not adopt microservices: start with a monolith and split it only when the pain is real. Microservices solve a scale problem (organizational scale > technical scale), not a fashion problem. If a five-person team adopts microservices, it will multiply the monolith's complexity by the number of services, then divide it by the team size. Whether to split depends on whether your organization can support Conway's Law: if the service architecture grows before the organizational structure does, the result is a disaster.