Why an Enterprise API Becomes Slow as It Scales—and How to Diagnose It

An API that returned data in 80ms at launch, sailed through load testing, and shipped without complaint can still be timing out eighteen months later—not because anything obviously broke, but because traffic, data volume, and the number of things happening concurrently all grew past assumptions nobody wrote down. The frustrating part isn’t that it got slow. It’s that “it got slow” has at least eight plausible causes, and adding a cache or throwing more CPU at the box fixes maybe one of them.

Start with the question you’re actually trying to answer

Before touching any code, the diagnostic question is not “how do we make this faster” — it’s “where, specifically, is the time going, and does that change under load.” Those are different questions with different answers. A request that takes 200ms in isolation but 4 seconds under production concurrency has a completely different problem than a request that’s uniformly slow regardless of load. Conflating the two is how teams end up adding Redis in front of a query that was never the bottleneck.

This is where instrumentation earns its keep. If you don’t already have distributed tracing (Application Insights, OpenTelemetry with a backend like Jaeger or Tempo, or equivalent APM) wired into the API, database calls, and any outbound HTTP clients, that’s step zero — not a nice-to-have. Without it you’re reading logs and guessing at causality; with it you get an actual waterfall of where each request’s time was spent, per dependency, per percentile.

flowchart TD
    A[API reported as slow] --> B{Is it slow<br/>at all times or<br/>only under load?}
    B -->|Slow even at low load| C[Look at the request itself:<br/>query plan, serialization size,<br/>external call latency]
    B -->|Only slow under concurrency| D{Check resource<br/>saturation first}
    D --> E[DB connection pool:<br/>wait count / wait time]
    D --> F[Thread pool:<br/>queue length, worker starvation]
    D --> G[HTTP client pool:<br/>socket exhaustion, DNS]
    E --> H{Pool exhausted?}
    F --> I{Threads starved?}
    G --> J{Sockets exhausted?}
    H -->|Yes| K[Fix pool sizing / connection lifetime,<br/>find the query holding connections too long]
    I -->|Yes| L["Find sync-over-async calls:<br/>.Result, .Wait, GetAwaiter/GetResult"]
    J -->|Yes| M[Check HttpClient reuse<br/>and DNS/socket limits]
    C --> N{Check DB first:<br/>execution plan + query store}
    N -->|Missing/bad index,<br/>N+1 pattern| O[Fix query shape or index]
    N -->|Query is fine| P[Check serialization payload size<br/>and external dependency latency]
    K --> Q[Re-measure under<br/>the same load profile]
    L --> Q
    M --> Q
    O --> Q
    P --> Q
    Q --> R{Improved?}
    R -->|No| B
    R -->|Yes, but not enough| S[Consider caching—only for<br/>data that's provably safe to cache stale]

The order matters. Resource exhaustion (connection pools, thread pool, sockets) produces symptoms that look identical to “the query got slow” — rising p99 latency, timeouts, occasional 500s — but the fix is entirely different, and it’s usually cheaper to rule out than to assume. A five-minute look at connection pool metrics can save a day of index tuning that wouldn’t have moved the needle.

The classic first suspect: N+1 queries

This is the most common cause we see in ASP.NET Core / EF Core codebases that grew organically, and it’s the one that behaves most deceptively: it’s often fine in dev and staging (10 orders, 10 line items) and catastrophic in production (10,000 orders). The code didn’t change. The data did.

// PROBLEM: classic N+1. Looks innocent, generates 1 + N queries.
// For 5,000 orders this issues 5,001 round-trips to SQL Server.
public async Task<List<OrderSummaryDto>> GetOrderSummariesAsync()
{
    var orders = await _db.Orders
        .Where(o => o.Status == OrderStatus.Active)
        .ToListAsync();

    var result = new List<OrderSummaryDto>();
    foreach (var order in orders)
    {
        // Each access to a nav property that wasn't eagerly loaded
        // triggers a lazy-load query — one per order.
        var customer = order.Customer; // lazy load #1
        var lineItemCount = order.LineItems.Count; // lazy load #2

        result.Add(new OrderSummaryDto
        {
            OrderId = order.Id,
            CustomerName = customer.Name,
            LineItemCount = lineItemCount
        });
    }
    return result;
}

// FIX: project directly to the shape you need. EF Core translates this
// to a single SQL query with the necessary joins/aggregates — no lazy
// loading, no over-fetching whole entity graphs you'll discard anyway.
public async Task<List<OrderSummaryDto>> GetOrderSummariesAsync()
{
    return await _db.Orders
        .Where(o => o.Status == OrderStatus.Active)
        .Select(o => new OrderSummaryDto
        {
            OrderId = o.Id,
            CustomerName = o.Customer.Name,
            LineItemCount = o.LineItems.Count
        })
        .AsNoTracking()
        .ToListAsync();
}

Two things happen here that both matter: the query count drops from N+1 to 1, and .Select() projection means SQL Server only returns the columns actually used, instead of every column on Order, Customer, and the full LineItems collection. If you genuinely need full related entities (not just a couple of fields), .Include() with .AsSplitQuery() for multiple collection includes is usually the right middle ground — it avoids both the N+1 problem and the cartesian-product row explosion that a single .Include() with multiple collections can cause.

The reason this gets worse specifically under scale rather than being consistently bad: query count multiplies with row count, but round-trip latency to the database (even a fast one) is roughly constant per call — network hop, connection acquisition, query parse. At 10 rows, 11 round-trips is invisible. At 10,000 rows, it’s 10,001 round-trips competing for the same connection pool that every other concurrent request is also drawing from — which is exactly how an N+1 problem turns into connection pool exhaustion for completely unrelated endpoints.

The fastest way to confirm an N+1 problem in the field, before touching code, is EF Core’s logging (Microsoft.EntityFrameworkCore.Database.Command at Information level) or a SQL Server trace filtered to the app’s connection — if you see the same parameterized query text repeated hundreds of times in one request’s timeframe, you’ve found it. Don’t skip this confirmation step; “fixing” a query you assume is the N+1 offender, when it’s actually a different endpoint entirely, wastes a deploy cycle. Rough guide: confirming this pattern takes under an hour with query logging on; the fix itself is usually engineer-hours, not days.

Missing or wrong indexes — found with data, not guesses

Once query shape is confirmed reasonable, the next question is whether the database can execute it efficiently. “Add an index” is the reflexive answer, but adding indexes without evidence is its own failure mode — every index speeds up some reads and slows down every write to that table, and a table with a dozen speculative indexes can make bulk inserts and updates meaningfully slower.

The right sequence is: capture the actual execution plan for the slow query, and separately, ask the database itself what it thinks is missing, rather than trusting intuition about which column “should” be indexed.

-- 1. Look at what's actually expensive right now using Query Store
-- (SQL Server 2016+ — enable it once per database, then query it).
-- This finds queries with high total or average duration/logical reads
-- without needing to reproduce the slow request manually.
SELECT TOP 20
    qt.query_sql_text,
    rs.avg_duration / 1000.0 AS avg_duration_ms,
    rs.avg_logical_io_reads,
    rs.count_executions,
    rs.last_execution_time
FROM sys.query_store_query_text qt
JOIN sys.query_store_query q ON q.query_text_id = qt.query_text_id
JOIN sys.query_store_plan p ON p.query_id = q.query_id
JOIN sys.query_store_runtime_stats rs ON rs.plan_id = p.plan_id
WHERE rs.last_execution_time > DATEADD(HOUR, -2, GETUTCDATE())
ORDER BY rs.avg_duration DESC;

-- 2. For a specific query, ask SQL Server for missing index suggestions
-- tied to the actual execution plan (not a generic "index everything"
-- tool — this is scoped to plans the optimizer has actually compiled).
SELECT
    mid.statement AS table_name,
    migs.avg_user_impact,
    migs.user_seeks + migs.user_scans AS times_would_have_helped,
    'CREATE INDEX IX_' + OBJECT_NAME(mid.object_id) + '_suggested ON '
        + mid.statement + ' (' + ISNULL(mid.equality_columns, '')
        + CASE WHEN mid.inequality_columns IS NOT NULL
               THEN ',' + mid.inequality_columns ELSE '' END + ')'
        + ISNULL(' INCLUDE (' + mid.included_columns + ')', '') AS create_statement
FROM sys.dm_db_missing_index_details mid
JOIN sys.dm_db_missing_index_groups mig ON mig.index_handle = mid.index_handle
JOIN sys.dm_db_missing_index_group_stats migs ON migs.group_handle = mig.index_group_handle
ORDER BY migs.avg_user_impact * (migs.user_seeks + migs.user_scans) DESC;

Treat the missing-index DMV output as a hypothesis list, not a to-do list — it doesn’t account for write cost, doesn’t know about redundant existing indexes, and resets on server restart, so avg_user_impact from a system that’s only been up for an hour is noise. Cross-reference against the execution plan for the specific slow query: look for a table scan or a clustered index scan where you’d expect a seek, a high “estimated vs. actual rows” mismatch (stale statistics — sometimes the real fix is UPDATE STATISTICS, not a new index), or a key lookup happening for a huge row count (usually solved with an INCLUDE column, not a whole new index). Only after that correlation should an index change go out, and it should go through the same load-tested rollout as any other schema change, since it takes a full table lock (or an online rebuild, on supported editions) to build. Rough guide: an hour or two with Query Store to diagnose; the index change itself is quick, though a load-tested rollout on a large table can add a day.

Connection pool exhaustion — the “random” slowness

This is the cause we see most underdiagnosed, because the symptoms are misleading: intermittent timeouts, no obvious slow query in the logs, and requests that are fast in isolation but fail under real concurrency. It’s rarely one query being slow — it’s every request holding a database connection slightly too long, so the pool of available connections (default max pool size for SQL Server via ADO.NET is 100) runs dry, and everything else queues behind it.

Common root causes: DbContext instances (or raw connections) not being disposed promptly, long-running transactions that hold a connection open while doing unrelated work (like calling an external API mid-transaction), or a pool size left at the default while the app scaled its instance count without anyone revisiting it. The fix is rarely “raise Max Pool Size” — that just delays the wall you’ll hit, and a larger pool means more concurrent connections competing for the database’s own resources (CPU, lock manager, tempdb). The real fix is finding what’s holding connections too long and shortening that window — commit transactions before making outbound calls, scope DbContext per-request (the ASP.NET Core default) rather than per-application, and set an explicit, sane Connection Timeout so exhaustion fails fast and visibly instead of queuing silently.

The diagnostic signal: SqlConnection pool counters (via .NET EventCounters or dotnet-counters) showing NumberOfActiveConnectionPoolGroups growing or connection wait times climbing, correlated against traffic spikes — not against any single query’s duration. If pool exhaustion correlates with a specific endpoint’s traffic, that endpoint is the one holding connections too long, even if its own reported latency looks acceptable.

The same class of problem shows up on the outbound side with HttpClient: creating a new HttpClient per request (rather than using IHttpClientFactory) exhausts available sockets under load through a completely different mechanism — socket exhaustion from not reusing connections, compounded by each disposed HttpClient leaving a socket in TIME_WAIT. Same category of failure, same “looks random until you check pool metrics” diagnosis. Rough guide: a few hours to diagnose via pool counters; fixing every connection-holding code path can take days on a large codebase.

Synchronous blocking hiding inside async code

async/await throughout a codebase doesn’t guarantee non-blocking behavior — a single .Result or .Wait() call on a Task, buried in what looks like async code, blocks a thread-pool thread until that task completes. At low concurrency this is invisible: there are always spare thread-pool threads, so the block resolves almost immediately. Under real load, every request path that hits that line blocks a thread, the thread pool’s available worker count drops, .NET’s thread pool injector adds new threads only gradually (roughly one every couple hundred milliseconds once starvation is detected), and the queue behind it backs up — which looks exactly like generalized slowness across unrelated endpoints, not like one specific bug.

// PROBLEM: buried sync-over-async. Compiles fine, works in dev,
// starves the thread pool under concurrent load.
public IActionResult GetPricing(int productId)
{
    // .Result blocks the calling thread until the async call completes.
    var pricing = _pricingClient.GetPricingAsync(productId).Result;
    return Ok(pricing);
}

// FIX: async all the way up the call stack.
public async Task<IActionResult> GetPricing(int productId)
{
    var pricing = await _pricingClient.GetPricingAsync(productId);
    return Ok(pricing);
}

This is easy to introduce accidentally — a synchronous interface method calling into a newer async client, a background job scheduler that doesn’t support Task-returning delegates, a [HttpGet] action left non-async because “it’s a simple lookup.” The diagnostic tell is thread-pool starvation counters (ThreadPool.PendingWorkItemCount, or the .NET ThreadPoolQueueLength EventCounter) climbing under load while CPU utilization stays moderate — the machine isn’t busy, it’s just out of available threads to do the work with. That signature — high latency, low CPU — is one of the more specific diagnostic clues available and points away from “the query is slow” and toward “something is blocking.” Rough guide: diagnosing thread-pool starvation this way typically takes a few hours with the right counters; fixing a sync-over-async chain across a codebase this size can take engineer-days to weeks, depending on how deep the pattern goes.

External dependencies, timeouts, and the caching trap

An API that calls out to a partner service, a legacy SOAP endpoint, or another internal microservice inherits that dependency’s latency distribution — including its bad days. Without an explicit timeout, a HttpClient call can hang far longer than any reasonable SLA, tying up a thread-pool thread and a connection for the duration, and a burst of those simultaneously is enough to take down an otherwise healthy API. Explicit timeouts, paired with a circuit breaker (Polly’s CircuitBreakerPolicy is the standard choice in .NET) so repeated failures short-circuit to a fast failure instead of repeated slow ones, are close to mandatory for any synchronous outbound call in a request path.

Caching is the tool everyone reaches for next, and it’s the one most likely to be reached for too early. Caching is the right answer when data is read far more often than it changes and slightly-stale data is acceptable — reference data, pricing tables, permission sets refreshed every few minutes. It’s the wrong answer, or at least an incomplete one, when it’s used to paper over a query or dependency that’s fundamentally too slow for its actual usage pattern — the underlying problem is still there for the cache-miss path and for anything that can’t tolerate staleness, and now there’s a second problem: cache invalidation, and the specific failure mode of a “cache stampede” — a popular, expensive key expiring and dozens of concurrent requests all missing simultaneously and hammering the database at once. A short jittered expiry plus a request-coalescing pattern (or a background refresh-ahead) avoids that; caching everything indefinitely “to be safe” trades a performance problem for a data-correctness one. Rough guide: adding timeouts and a circuit breaker is usually an afternoon’s work per dependency; untangling an existing cache-as-band-aid setup properly is often engineer-days once invalidation and stampede protection are done right.

Two causes worth measuring even when they’re not the top suspect

Logging overhead. Verbose synchronous logging — writing to disk, or worse, a network log sink, on the hot path of every request — is a legitimate bottleneck at high request volume, not just a Trace-level annoyance. Structured logging frameworks (Serilog, Microsoft.Extensions.Logging with async sinks) should batch and flush off the request thread; if profiling shows meaningful time inside a logging call, that’s the same class of problem as a slow query, just less expected.

GC pressure. High-allocation code paths — building large intermediate collections, excessive string concatenation, boxing in hot loops — increase Gen0/Gen1 GC frequency, and under load, more concurrent allocation means more frequent collections, which shows up as intermittent latency spikes that don’t correlate with any single slow request. dotnet-counters or APM-reported GC pause time is the fastest way to confirm or rule this out; it’s rarely the primary cause of a slowdown but it compounds every other cause above by stealing CPU time exactly when the system is already under pressure.

Putting the causes and the diagnostics side by side

Key takeaways

• Diagnose before optimizing. A distributed trace or APM waterfall that shows where time is actually spent will save more engineering time than any specific fix — guessing wrong costs a full deploy-and-measure cycle.
• Separate “slow at any load” from “slow only under concurrency.” The first points at query shape, payload size, or a genuinely slow dependency. The second points at resource exhaustion — connection pools, thread pool, sockets — which behaves identically to a slow query but has a completely different fix.
• N+1 queries and over-fetched entity graphs are the most common root cause in ASP.NET Core / EF Core systems, and the most deceptive, because they’re invisible at low data volume and only bite in production.
• Add indexes based on execution plans and Query Store evidence, not intuition — every index has a write-side cost, and speculative indexing can make the system slower overall.
• Sync-over-async code (.Result, .Wait()) is one of the more specific diagnostic signatures available: high latency with moderate-to-low CPU usage under load almost always means thread-pool starvation, not a slow query.
• Timeouts and circuit breakers on outbound calls aren’t optional hardening — without them, one degraded downstream dependency can take the whole API down.
• Caching is a mitigation for acceptable staleness, not a fix for a fundamentally too-slow query — used as the latter, it hides the real problem and introduces stampede and staleness risk of its own.

Much of this diagnostic discipline matters just as much when the system in question is a modernized legacy platform or a multi-tenant SaaS backend — see our related notes on modernizing legacy .NET applications without a rewrite and designing a multi-tenant SaaS platform with ASP.NET Core, where connection pooling and tenant-aware caching carry even more weight. If your API’s performance problem has outgrown ad hoc troubleshooting, our engineering services team runs this exact diagnostic process as a structured performance assessment.