Thread Pool

complexitypopularity

Keep a fixed set of worker threads that pull tasks from a shared queue, so thread-creation cost is paid once and concurrency stays bounded, instead of spawning one new thread per task.

Real-world analogy

The barbershop with four chairs

A barbershop with four chairs does not hire a fifth barber every time a fifth customer walks in. Customers wait on the bench (the queue) until one of the four chairs frees up. The shop stays fast because it never pays to train and equip a new barber per customer, and it stays sane because four chairs is exactly as much simultaneous cutting as the shop can actually support.

The problem

ImageUploadHandler.onUpload() spins up a brand-new thread for every incoming image, so thumbnail generation never blocks the request. It works beautifully for ten uploads and catastrophically for ten thousand: each thread costs real memory and OS scheduling overhead to create, and nothing here puts a ceiling on how many can exist at once.

The system does not get slower because the work got harder - it gets slower, and eventually falls over, because of the overhead of constantly creating and tearing down the threads doing the work.

The solution

Create a fixed number of worker threads exactly once, when the pool starts up. Each worker loops forever: pull the next task off a shared queue, run it, loop back for the next one. Nothing about that loop involves creating or destroying a thread mid-flight.

Callers never touch a thread directly - they call submit(task), which drops the task on the queue and returns immediately. However many tasks arrive, however fast, the number of workers processing them never changes; the queue is what absorbs the difference between arrival rate and processing rate.

1. 1A request arrives that needs a thumbnail generated. The handler does not spawn a thread for it.
2. 2The pool never runs the task itself - it drops it on the shared queue and returns immediately.
3. 3One of a fixed four workers, currently idle, pulls the next task off the queue - blocking cheaply while the queue was empty.
4. 4The worker actually resizes the image. This can take a while; the other three workers are unaffected.
5. 5A hundred more uploads arrive in the same second. No new threads spawn - tasks simply queue up.
6. 6The queue absorbs the burst. Exactly four workers process it, at whatever pace four workers actually manage.
7. 7The moment this worker finishes task #1, it loops back and pulls the next one - the thread itself is never destroyed between tasks.

Structure

ThreadPool has both the workers and the queue, but callers only ever call submit() on the pool - nothing outside this diagram touches Worker or TaskQueue directly.

uses

Code

Same example three ways: bounded thumbnail generation behind a fixed worker pool.

// One thread per request, with no ceiling on how many can exist at once.

class ImageUploadHandler is

method onUpload(image) is

thread = new Thread(() => generateThumbnail(image))

thread.start()

// A thousand uploads in one second spawn a thousand threads.

// The OS scheduler, and then the machine, notice.

// Four workers, created once, handle any number of submitted tasks.

class ThreadPool {

constructor(size) {

this.queue = new BlockingQueue();

this.workers = Array.from({ length: size }, () => new Worker(this.queue).start());

}

 

submit(task) {

this.queue.enqueue(task); // returns instantly; a worker will get to it

}

}

const pool = new ThreadPool(4);

for (const image of uploadedImages) {

pool.submit(() => generateThumbnail(image)); // no new thread, ever

}

// What callers submit. The pool never knows what work actually happens.

interface Runnable is

method run()

 

// A thread-safe queue that lets idle workers block cheaply until work arrives.

class BlockingQueue is

field items: queue

field lock, notEmpty: condition

 

method enqueue(item) is

lock.acquire()

items.push(item)

notEmpty.signal()

lock.release()

 

method dequeue() is

lock.acquire()

while items.isEmpty() do

notEmpty.wait(lock)

item = items.pop()

lock.release()

return item

 

// A long-lived thread that never exits between tasks.

class Worker implements Thread is

field taskQueue: BlockingQueue

field running: boolean

 

method run() is

running = true

while running do

task = taskQueue.dequeue()

task.run()

 

method stop() is

running = false

 

// The public face callers use. It creates the workers once, up front.

class ThreadPool is

field taskQueue: BlockingQueue

field workers: list of Worker

 

constructor ThreadPool(size) is

taskQueue = new BlockingQueue()

workers = []

for i in range(size) do

worker = new Worker(taskQueue)

worker.start()

workers.add(worker)

 

method submit(task) is

taskQueue.enqueue(task)

 

method shutdown() is

foreach (worker in workers) do

worker.stop()

 

// Callers submit work; they never see a thread.

pool = new ThreadPool(4)

pool.submit(() => generateThumbnail(image))

When to use it

• Work arrives in short-lived units (a request, a job, a task) at a rate that can burst well past what should ever run simultaneously.
• Thread creation cost (or the cost of any other expensive-to-create worker) is measurably hurting throughput, and the work itself does not require an unbounded number of concurrent workers to make progress.

Pitfalls

• Wrong sizing for the workload. CPU-bound tasks want a pool close to the core count; I/O-bound tasks (mostly waiting on network or disk) tolerate, and often need, a much larger one. One "correct" pool size does not exist across workload types.
• Unbounded queue hiding overload. A queue with no size limit turns "too much work" into "memory grows until the process dies" instead of a visible, handleable backpressure signal.
• Blocking a worker forever. A task that never returns (a stuck network call with no timeout) permanently removes one worker from the pool. With a small pool, a handful of stuck tasks can stall everything behind them.

Don't confuse it with

• Producer-Consumer. A thread pool's workers are consumers pulling off a queue that callers produce onto - Thread Pool is a specific, bounded instance of the broader Producer-Consumer relationship, not a different pattern competing with it.
• Flyweight. Both save cost through reuse instead of repeated creation, but Flyweight shares immutable data across many logical objects; a pooled thread carries no shared state between the unrelated tasks it happens to run one after another.
• Async/await without a pool. Cooperative concurrency on a single thread (an event loop) solves a related problem - many pending operations without many OS threads - through an entirely different mechanism, not a fixed set of worker threads.

Check yourself

Question 1 of 5

A thousand upload requests arrive in one second. What does a correctly sized thread pool do?