ishavasya

9. Concurrency, Parallelism, Cores, Threads, and Async in Software

May 12, 2026 · Updated May 23, 2026

What this note is really about

Words like:

  • concurrency
  • parallelism
  • thread
  • process
  • core
  • async
  • await
  • callback
  • thread pool

are often used as if they all mean roughly the same thing.

They do not.

They are related, but they describe different layers of the system:

  • hardware execution capability
  • operating-system scheduling
  • runtime behavior
  • application design
  • I/O waiting strategies

This note is about clearing those concepts up fundamentally.

The goal is not to memorize jargon.

The goal is to know:

  • what is physically happening
  • what is logically happening
  • which abstraction solves which problem
  • why modern software uses different tools for CPU work and I/O work

I will use C# examples where useful, because C# has clear language-level support for many of these ideas.

The first distinction: concurrency is not the same as parallelism

This is the most important distinction in the whole topic.

Concurrency

Concurrency means multiple tasks are in progress during overlapping periods of time.

That does not require them to be executing at the exact same instant.

A single CPU core can support concurrency by switching between tasks.

So concurrency is mainly about:

  • coordination
  • structure
  • dealing with multiple things at once conceptually

Parallelism

Parallelism means multiple tasks are actually executing at the same instant on different execution resources.

That usually means:

  • multiple CPU cores
  • or CPU plus GPU
  • or SIMD/vector execution inside one processor

So parallelism is mainly about:

  • simultaneous execution
  • throughput
  • speedup for suitable workloads

Short rule

You can have:

  • concurrency without parallelism
  • parallelism without interesting coordination problems
  • or both together

This one distinction already removes a large amount of confusion.

What a CPU core actually is

A CPU core is an independent execution engine inside the processor.

A modern CPU chip may contain multiple cores.

Each core can:

  • fetch instructions
  • decode instructions
  • execute operations
  • update registers and state

At a beginner level, the clean mental model is:

one core is one place where one instruction stream can make progress

That model is not the full microarchitectural truth, but it is the correct starting point.

Modern cores may internally do many advanced things:

  • pipelining
  • out-of-order execution
  • branch prediction
  • speculative execution
  • vector instructions

But none of that changes the main idea that a core is a unit of active execution capability.

One processor, many cores

Historically, performance was often increased by pushing clock speeds and microarchitecture harder on fewer cores.

Later, power and thermal limits made that harder, so multicore processors became the normal design.

If a CPU has:

  • 4 cores
  • 8 cores
  • 16 cores

it means it has multiple execution engines that can run separate work in parallel.

That does not mean every program automatically becomes that many times faster.

To benefit from multiple cores, the software must contain work that can proceed in parallel safely and efficiently.

Hardware threads and logical processors

Some CPUs support technologies like simultaneous multithreading, often called:

  • SMT
  • Hyper-Threading on some Intel systems

This allows one physical core to expose more than one logical processor to the operating system.

Important clarification:

a logical processor is not the same thing as a full extra physical core.

It is a way of improving utilization of a core’s internal resources by allowing multiple hardware execution contexts.

So when people say:

  • “my machine has 8 cores and 16 threads”

they often mean:

  • 8 physical cores
  • 16 logical processors visible to the OS

That is a hardware-level concept, not the same as software threads.

What a software thread is

A thread is an operating-system scheduled execution path within a process.

It has things like:

  • its own call stack
  • its own instruction pointer
  • its own CPU register context when running

Threads inside the same process usually share:

  • the same address space
  • the same heap
  • the same global state

That shared-memory aspect is both powerful and dangerous.

It makes communication fast, but it also creates risks such as:

  • race conditions
  • deadlocks
  • inconsistent state

Process vs thread

This distinction also matters.

Process

A process is a running program instance with its own address space and resources.

Thread

A thread is a path of execution inside a process.

A process can have:

  • one thread
  • many threads

Threads are usually lighter-weight than processes, but they are not free.

Creating many threads has costs:

  • memory for stacks
  • scheduling overhead
  • context switching
  • synchronization complexity

How threads relate to cores

Threads are software execution units managed by the operating system.

Cores are hardware execution units provided by the CPU.

The operating system scheduler maps runnable threads onto available logical processors.

So:

  • if you have more runnable threads than available execution resources, they take turns
  • if you have enough cores or logical processors, some threads can run truly in parallel

This is the clean relationship:

threads are scheduled onto cores

The thread does not “contain” the core, and the core does not “contain” the thread in a software sense.

The core executes whichever thread the scheduler currently runs there.

Time slicing and context switching

Suppose you have one core and three runnable threads.

They cannot all execute at the same instant.

So the operating system scheduler switches between them.

This is called time slicing.

When the CPU stops running one thread and starts another, it must save and restore execution context such as:

  • instruction pointer
  • registers
  • stack-related state

This is called a context switch.

Context switching is necessary, but it is not free.

Too many runnable threads can hurt performance rather than help it.

Concurrency on one core

This is worth stating explicitly.

Even with a single CPU core, a program can still be concurrent.

For example, one app may be:

  • handling user input
  • waiting for disk I/O
  • waiting for network responses
  • updating internal state

Those activities may overlap conceptually, even though only one thread at a time is executing on the core.

So concurrency is not “fake parallelism.”

It is a real and useful property of software structure.

Parallelism on many cores

If the machine has multiple available execution resources, then independent runnable threads may actually execute at the same time.

That is parallelism.

Examples:

  • processing multiple image tiles at once
  • running multiple simulation partitions
  • searching large datasets in chunks
  • doing per-request server work across many cores

Parallelism helps when:

  • the work can be split
  • the tasks do not block each other too much
  • synchronization overhead does not dominate

Parallelism does not automatically help if:

  • the workload is mostly waiting on I/O
  • the tasks constantly contend for the same lock
  • the work is inherently sequential

CPU-bound work vs I/O-bound work

This is another fundamental distinction.

CPU-bound work

The main limiting factor is computation.

Examples:

  • image processing
  • compression
  • encryption
  • simulation
  • parsing huge datasets

If CPU-bound work can be split, more cores may help.

I/O-bound work

The main limiting factor is waiting for external systems.

Examples:

  • database queries
  • HTTP calls
  • file reads
  • disk writes
  • waiting for user input

For I/O-bound work, more threads are often not the best answer.

What matters more is not blocking valuable threads while waiting.

This is where async programming becomes extremely important.

Why “more threads” is not the universal solution

Beginners often think:

“If one thread is good, many threads must be better.”

That is wrong.

More threads can create:

  • more scheduling overhead
  • more context switching
  • more memory usage
  • more lock contention
  • more debugging difficulty

If a workload is mostly waiting for I/O, a thread that just blocks is wasting a scarce execution resource.

So the correct question is not:

“How do I create more threads?”

It is:

“What kind of work is this, and what execution model fits it?”

The three broad tools

At a practical level, a lot of modern application design comes down to choosing among three broad execution ideas:

  1. synchronous sequential work
  2. parallel CPU work
  3. asynchronous I/O work

Each solves a different class of problem.

Synchronous sequential work

Good when:

  • the work is simple
  • ordering is strict
  • complexity would not be justified

Parallel CPU work

Good when:

  • the work is CPU-heavy
  • independent chunks can run simultaneously

Asynchronous I/O work

Good when:

  • the system spends much of its time waiting
  • you want high scalability without tying up threads unnecessarily

What callbacks are

Before async/await, many systems expressed asynchronous behavior using callbacks.

A callback is a function you provide so that some other component can call it later when work completes.

Example idea:

  • start network request now
  • when response arrives later, invoke this callback

Callbacks are not wrong.

They are one of the fundamental ways to represent deferred work.

But deeply nested callbacks can become hard to read and reason about.

That is one reason language-level async models became so valuable.

What async really means

async does not mean:

  • parallel
  • running on another thread automatically
  • faster by magic

This is one of the most important corrections to make.

At a fundamental level, async means:

the operation may complete later, and the current flow can be suspended without blocking a thread in the ordinary way.

That is especially useful for I/O-bound work.

So async is mainly about:

  • non-blocking waiting
  • scalable handling of many outstanding operations
  • keeping threads free while external work is in progress

C# async and await: what they actually do

In C#, async and await provide a structured way to write asynchronous code that looks more like ordinary sequential code.

Example:

public async Task<string> GetPageAsync(HttpClient client, string url)
{
    return await client.GetStringAsync(url);
}

This does not mean:

  • “start a new thread for this method”

It means:

  • begin an asynchronous operation
  • if it is not complete yet, return control to the caller
  • resume the method later when the awaited operation completes

The compiler transforms the method into a state machine behind the scenes.

That transformation is one of the key facts of C# async programming.

Await is about suspension, not about sleeping

Suppose you write:

var text = await client.GetStringAsync(url);

If the data is not ready yet:

  • the current method yields
  • the thread is free to do other work
  • the method continuation is scheduled for later

That is very different from:

Thread.Sleep(1000);

Thread.Sleep blocks a thread.

await on a proper asynchronous I/O operation usually does not block the thread while waiting.

This distinction is central.

Async does not create CPU speedup

Suppose a method performs heavy pure computation:

public int Compute()
{
    // expensive CPU work
}

Wrapping it in async does not make the computation cheaper.

If the work is CPU-bound, then:

  • either one core does it
  • or you parallelize it across multiple execution resources

Async is mostly about efficient waiting, not free computation.

This is why people say:

  • use parallelism for CPU-bound work
  • use async for I/O-bound work

That is not a perfect universal rule, but it is a very strong practical guideline.

The thread pool

Creating a brand new operating-system thread for every small task would be expensive and wasteful.

So modern runtimes maintain a thread pool:

a reusable set of worker threads that can execute queued work.

In .NET, the thread pool is foundational for many runtime services.

Benefits include:

  • lower thread creation cost
  • better reuse
  • centralized scheduling heuristics
  • better scalability for many short-lived work items

Thread pool threads are still real threads.

The point is that they are reused rather than created fresh every time.

C# Task is not the same thing as thread

This is critical.

A Task in C# represents an asynchronous operation or unit of work.

It is not simply “a thread object with a nicer name.”

A task may involve:

  • actual work running on a thread pool thread
  • an I/O operation that mostly waits without occupying a thread
  • a continuation that resumes later

So:

  • thread = execution resource managed by OS/runtime
  • task = logical operation abstraction

Confusing tasks and threads causes many design mistakes.

Task.Run and what it is for

In C#, Task.Run is commonly used to queue CPU-bound work to the thread pool.

Example:

var result = await Task.Run(() => ComputeExpensiveThing());

This means roughly:

  • place the computation on a thread pool thread
  • let it run there
  • await its completion

This can be useful when:

  • the work is CPU-bound
  • you intentionally want it off the current thread

It is not the correct way to make already asynchronous I/O APIs “more async.”

Bad pattern:

await Task.Run(() => File.ReadAllText(path));

If a proper asynchronous API exists, prefer that.

Example:

await File.ReadAllTextAsync(path);

That distinction matters because thread-pool offloading and true asynchronous I/O are different mechanisms.

Asynchronous I/O in modern systems

When a program requests network or disk I/O, the operation often involves:

  • the operating system
  • device drivers
  • hardware
  • external devices or remote systems

The request can be initiated, and then the thread does not need to sit there doing nothing until completion.

Instead, the system can be notified later that the operation has completed.

This is the deeper reason async scales well for I/O-heavy server software.

One thread can participate in managing many outstanding operations over time rather than blocking per request.

Why async matters so much on servers

Imagine a web API server handling thousands of requests, many of which:

  • query a database
  • call another HTTP service
  • read from storage

If every waiting operation blocked a thread, the server would need a huge number of threads just to sit idle waiting.

That would hurt:

  • memory usage
  • scheduling efficiency
  • scalability

Async lets the server initiate the external work and free the thread while waiting.

When the result arrives, execution resumes.

This is one of the main reasons modern server frameworks rely heavily on async.

Synchronization context and resuming

In some application models, especially UI frameworks, after an await the continuation may resume on a particular context such as the UI thread.

Why?

Because UI components often require single-threaded access.

In ASP.NET Core, there is generally no classic UI synchronization context in the same sense, so continuation behavior differs.

The beginner-level takeaway is:

after await, your method resumes later, but not always with the exact same threading assumptions you may casually imagine.

Understanding context and continuation behavior matters in advanced code.

Race conditions

Once multiple threads or concurrent flows interact with shared state, you can get race conditions.

A race condition occurs when the result depends on timing or interleaving in unsafe ways.

Example:

  • two threads increment the same variable without synchronization
  • one update gets lost

This is one of the main reasons concurrency is difficult.

Parallelism is not just a performance tool.

It is also a correctness challenge.

Locks and mutual exclusion

When shared state must be protected, one common tool is a lock.

In C#, this may look like:

private readonly object _gate = new();
 
lock (_gate)
{
    // protected critical section
}

This means only one thread at a time may execute that critical section for that lock object.

Locks can protect correctness, but overuse or bad design can create:

  • contention
  • reduced parallelism
  • deadlocks
  • latency spikes

So the real goal is not “use locks everywhere.”

It is:

  • minimize unsafe shared mutable state
  • synchronize carefully where needed

Deadlocks

A deadlock occurs when different execution flows wait on each other in a cycle and none can proceed.

Example shape:

  • thread A holds lock 1 and waits for lock 2
  • thread B holds lock 2 and waits for lock 1

Deadlocks can also occur in async code through bad blocking patterns or dependency cycles.

Concurrency bugs are often harder than ordinary logic bugs because they may:

  • depend on timing
  • reproduce rarely
  • disappear under debugging

That is why disciplined design matters so much here.

Async and deadlocks: a specific warning

In older .NET patterns, people sometimes caused deadlocks by mixing async code with blocking waits such as:

var result = SomeAsyncMethod().Result;

or:

SomeAsyncMethod().Wait();

These can be dangerous because they block a thread while waiting for an async continuation that may need that same execution context to resume.

The practical rule is:

do not block on async code unless you very carefully understand the context

Prefer async all the way when possible.

Thread pool starvation

The thread pool is useful, but it is not infinite magic.

If many thread pool threads become blocked for long periods, the system may struggle to get enough free workers to run queued work promptly.

This is sometimes called thread pool starvation.

It can happen when code:

  • blocks synchronously on I/O
  • uses .Result and .Wait() heavily
  • queues too much long-running blocking work

Symptoms may include:

  • delayed request handling
  • poor throughput
  • strange latency spikes

This is one reason good async usage improves server scalability.

Callbacks vs async/await

These solve related problems but have different ergonomics.

Callbacks

Good:

  • explicit low-level model of completion
  • natural for event-driven APIs

Weaknesses:

  • nesting complexity
  • harder error propagation
  • harder sequencing of many steps

Async/await

Good:

  • more readable control flow
  • easier composition
  • more natural error handling with exceptions
  • easier sequencing

Weaknesses:

  • can hide complexity if misunderstood
  • can encourage people to assume everything is thread-based or parallel when it is not

So async/await is not a new concept replacing deferred completion.

It is a better language-level way to express it in many cases.

Parallel loops and data parallelism in C#

For CPU-bound work, .NET offers tools such as:

  • Parallel.For
  • Parallel.ForEach
  • PLINQ
  • tasks used deliberately for parallel computation

Example:

Parallel.For(0, data.Length, i =>
{
    data[i] = Compute(data[i]);
});

This is about real parallel execution when possible.

It is not the same thing as asynchronous I/O.

The goal here is:

  • use multiple cores
  • increase throughput for CPU-heavy independent work

This can help greatly, but only if:

  • the work per item is large enough
  • the data can be processed independently
  • synchronization and memory contention stay manageable

Memory contention and false sharing

Parallel work is not only about CPU count.

Performance also depends on how threads interact with memory.

Problems include:

  • threads fighting over the same lock
  • heavy writes to shared structures
  • cache contention
  • false sharing

False sharing happens when separate threads modify different values that happen to live on the same cache line, causing unnecessary coherence traffic.

This is an advanced performance issue, but it matters because multicore performance is not just “more cores = more speed.”

Memory behavior matters too.

Throughput vs latency

Concurrency and parallelism are often used to improve one of two things:

Throughput

How much total work gets done over time.

Examples:

  • requests per second
  • files processed per minute

Latency

How long one operation takes.

Examples:

  • response time for one request
  • time to finish one computation

Some techniques improve throughput more than latency.

Some improve latency for certain tasks.

Some create tradeoffs.

So always ask:

“What metric am I actually trying to improve?”

Backpressure and throttling

If a system can start work faster than downstream systems can handle it, you need control mechanisms.

Important concepts include:

  • throttling
  • rate limiting
  • bounded concurrency
  • backpressure

These are not optional details in real systems.

They are part of making concurrency safe and stable.

API throttling: what it means

Suppose your application calls an external API.

That API may allow only:

  • a certain number of requests per second
  • a certain number of concurrent requests
  • a certain total quota per minute or day

If you ignore those limits, you may get:

  • 429 Too Many Requests
  • retries
  • bans
  • degraded performance

So API throttling is about respecting external system limits.

This often requires you to separate:

  • how many operations your app could launch
  • how many it should launch

That distinction is crucial in concurrent systems.

Bounded concurrency

One of the most practical patterns in modern software is bounded concurrency.

It means:

  • allow some parallel or concurrent work
  • but cap how much is in flight at once

Why?

Because “unlimited concurrency” often becomes resource exhaustion.

Resources that can be exhausted include:

  • threads
  • sockets
  • memory
  • database connections
  • API quotas
  • CPU

In C#, a common tool for bounding concurrency is SemaphoreSlim.

Example shape:

private readonly SemaphoreSlim _gate = new(10);
 
public async Task ProcessAsync(string item)
{
    await _gate.WaitAsync();
    try
    {
        await DoWorkAsync(item);
    }
    finally
    {
        _gate.Release();
    }
}

This means no more than 10 operations enter that region concurrently.

That is often much more correct than “fire everything at once.”

Rate limiting vs concurrency limiting

These are related but different.

Concurrency limiting

Controls how many operations are in progress at once.

Example:

  • at most 20 HTTP calls simultaneously

Rate limiting

Controls how many operations start over a time interval.

Example:

  • at most 100 requests per second

You may need one, the other, or both.

This matters a lot for external APIs.

Async streams and producer-consumer patterns

Real systems often involve flows of data over time, not just one request and one result.

Important patterns include:

  • producer-consumer queues
  • channels
  • pipelines
  • async streams

These help structure concurrency when data arrives incrementally.

In .NET, IAsyncEnumerable<T> can model asynchronous streams of values over time.

This is different from:

  • one synchronous list already in memory

It is useful when results are naturally produced gradually.

Cancellation

Concurrent and asynchronous systems need cancellation.

Why?

Because not all work should continue forever once started.

Examples:

  • user closes the page
  • request times out
  • system is shutting down
  • downstream dependency becomes unavailable

In .NET, CancellationToken is the standard mechanism for cooperative cancellation.

This matters because scalable software is not only about starting work.

It is also about stopping unnecessary work cleanly.

Timeouts, retries, and resilience

When many operations involve networks or remote systems, concurrency design must also include resilience.

Important concepts include:

  • timeout
  • retry
  • exponential backoff
  • circuit breaker
  • idempotency

Without these, concurrency can amplify failure instead of just amplifying throughput.

Example:

  • remote API slows down
  • app launches more waiting requests
  • queue grows
  • timeout storm begins

So good concurrency design includes operational discipline, not just language features.

UI apps vs server apps

Concurrency choices differ by environment.

UI applications

Goal:

  • keep the UI responsive

So long-running work should not block the UI thread.

Server applications

Goal:

  • scale efficiently under many simultaneous requests

So asynchronous I/O and careful resource management matter greatly.

Background processing systems

Goal:

  • maximize reliable throughput

So bounded work queues, retries, idempotency, and backpressure matter a lot.

Same core concepts, different priorities.

A clean C# mental map

Here is a compact practical map.

Thread

Low-level execution resource abstraction.

Use rarely in everyday application code unless you truly need explicit thread control.

Thread pool

Reusable worker threads managed by the runtime.

Very commonly used behind the scenes.

Task

Represents an asynchronous operation or queued unit of work.

This is a central modern abstraction in .NET.

async / await

Language features for composing asynchronous operations, especially I/O-bound workflows, without callback-heavy code.

Task.Run

Useful mainly for moving CPU-bound work onto the thread pool.

Not the general answer for all async code.

lock

Mutual exclusion for protecting shared state.

SemaphoreSlim

Useful for bounded concurrency and async-friendly gating.

Parallel.For / PLINQ

Parallel CPU-work tools, not async I/O tools.

CancellationToken

Cooperative cancellation mechanism.

This map alone prevents many category errors.

Common beginner confusions

1. “Async means multithreaded”

No.

Async mainly means non-blocking waiting and deferred continuation.

2. “Await starts a new thread”

No.

await suspends and resumes logical execution. It does not inherently create a thread.

3. “Task equals thread”

No.

A task is a logical operation abstraction.

4. “More threads always means more performance”

No.

It may mean more contention and overhead.

5. “Parallelism and concurrency are the same”

No.

Concurrency is about overlapping progress. Parallelism is about simultaneous execution.

6. “If code is I/O-bound, I should use Task.Run”

Usually no.

Prefer proper asynchronous I/O APIs.

7. “If async is good, unlimited concurrency must be better”

No.

Bounded concurrency is often essential.

8. “Using many cores automatically makes my program scale”

No.

The work must be parallelizable, and coordination costs must remain reasonable.

The most important conceptual thread

If you want one flow that ties the whole topic together, it is this:

CPU cores are hardware execution engines. Threads are software execution paths that the operating system schedules onto those cores. If more than one execution resource is available, threads may run in parallel; otherwise they take turns, which still allows concurrency. For CPU-bound work, performance may improve through parallel execution across multiple cores. For I/O-bound work, the key problem is usually not computation but waiting, so asynchronous programming avoids blocking threads unnecessarily. In C#, tasks represent logical operations, the thread pool provides reusable worker threads, async and await express non-blocking asynchronous workflows, callbacks are the lower-level completion idea that async often replaces ergonomically, and throttling or bounded concurrency prevents systems from overwhelming CPUs, thread pools, databases, or external APIs.

What you should come away knowing

You should leave this topic with these ideas clearly separated:

  1. Concurrency and parallelism are related but not the same.
  2. A core is a hardware execution engine.
  3. A thread is a software execution path scheduled onto available cores.
  4. Multiple runnable threads do not guarantee parallel execution.
  5. CPU-bound work and I/O-bound work need different strategies.
  6. Async is mainly about non-blocking waiting, not free speedup.
  7. A Task is not the same thing as a thread.
  8. The thread pool is for reusing worker threads efficiently.
  9. Shared mutable state creates race conditions and synchronization needs.
  10. Real systems need bounded concurrency, cancellation, timeouts, and throttling, not just raw parallelism.

Bottom line

Concurrency in software is about managing multiple in-progress activities. Parallelism is about actual simultaneous execution on hardware resources such as CPU cores. Threads connect software execution to those cores through OS scheduling, but tasks, async workflows, and callbacks are higher-level abstractions for structuring work. In C#, async and await are primarily for scalable asynchronous I/O, while thread-pool work items and parallel loops are more relevant to CPU-bound computation. Once you separate hardware execution, OS scheduling, runtime abstractions, and application design, the topic becomes much clearer and far less mystical.

Graph View