Setting Ideal size of Thread Pool

how to decide thread pool size java
java high performance thread pool
thread pool size c#
thread pools help in
how thread pool reuse the threads
thread pool size mysql
thread pool size elasticsearch
runtime getruntime availableprocessors threadpool

What is the difference between-

newSingleThreadExecutor vs newFixedThreadPool(20)

in terms of Operating System and Programming point of view.

Whenever I am running my program using newSingleThreadExecutor my program works very well and end to end latency(95th percentile) comes around 5ms.

But as soon as I start running my program using-

newFixedThreadPool(20)

my program performance degrades and I start seeing end to end latency as 37ms.

So now I am trying to understand from architecture point of view what does number of threads means here? And how to decide what is the optimal number of threads I should choose?

And if I am using more number of threads then what will happen?

If anyone can explain me these simple things in a layman language then that will be very useful to me. Thanks for the help.

My machine config spec- I am running my program from Linux machine-

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping        : 7
cpu MHz         : 2599.999
cache size      : 20480 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm arat pln pts
bogomips        : 5199.99
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping        : 7
cpu MHz         : 2599.999
cache size      : 20480 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm arat pln pts
bogomips        : 5199.99
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Ok. Ideally assuming your threads do not have locking such that they do not block each other (independent of each other) and you can assume that the work load (processing) is same, then it turns out that, have a pool size of Runtime.getRuntime().availableProcessors() or availableProcessors() + 1 gives the best results.

But say, if threads interfere with each other or have I/O inlvolved, then Amadhal's law explains pretty well. From wiki,

Amdahl's law states that if P is the proportion of a program that can be made parallel (i.e., benefit from parallelization), and (1 − P) is the proportion that cannot be parallelized (remains serial), then the maximum speedup that can be achieved by using N processors is

In your case, based upon the number of cores available, and what work they precisely do (pure computation? I/O? hold locks? blocked for some resource? etc..), you need to come up with the solution based upon above parameters.

For example: Some months back I was involved with collecting data from numeral web-sites. My machine was 4-core and I had a pool size of 4. But because the operation was purely I/O and my net speed was decent, I realized that I had best performance with a pool size of 7. And that is because, the threads were not fighting for computational power, but for I/O. So I could leverage the fact that more threads can contest for core positively.

PS: I suggest, going through the chapter Performance from the book - Java Concurrency in Practice by Brian Goetz. It deals with such matters in detail.

How to set an ideal thread pool size, How to set an ideal thread pool size. by Anton Ilinchik - 2019-04-18. We all know that thread creation in Java is not free. The actual overhead varies across  What should be the size of thread pool. This concept applies to the fixed thread pool in which we have option to give number of threads and you should have many tasks to be executed. Ideally, there is no fixed number which is ideal to be used in deciding the number of threads pool. It all depends on the use-case of the java program.


So now I am trying to understand from architecture point of view what does number of threads means here?

Each thread has its own stack memory, program counter (like a pointer to what instruction executes next) and other local resources. Swapping them out hurts latency for a single task. The benefit is that while one thread is idle (usually when waiting for i/o) another thread can get work done. Also if there are multiple processors available, they can run in parallel if there is no resource and/or locking contention between the tasks.

And how to decide what is the optimal number of threads I should choose?

The trade-off between swap-price versus the opportunity to avoid idle time depends on the little details of what your task looks like (how much i/o, and when, with how much work between i/o, using how much memory to complete). Experimentation is always the key.

And if I am using more number of threads then what will happen?

There will usually be linear-ish growth in throughput at first, then a relative flat part, then a drop (which may be quite steep). Each system is different.

Tuning the Size of Your Thread Pool, Settings for the trivial experiment are: set “server thread pool size” to 1 Given the looping behavior of the client we might expect the average  My machine was 4-core and I had a pool size of 4. But because the operation was purely I/O and my net speed was decent, I realized that I had best performance with a pool size of 7. And that is because, the threads were not fighting for computational power, but for I/O. So I could leverage the fact that more threads can contest for core positively.


Looking at Amdahl’s law is fine, especially if you know exactly how big P and N are. Since this will never really happen, you could monitor the performance (which you should do anyway) and increase/decrease you thread pool size to optimize whatever performance metrics are important to you.

How To Determine Web Application Thread Pool Size, If a web application gets more requests than thread pool size, excess requests than one request concurrently we have to increase the number of threads. To find right thread pool size is to balance between throughput and  When a thread pool that is allowed to grow expands beyond the maximum size, the additional threads are not reused and are discarded from the pool after required work items are completed. If set to true , the number of threads can increase beyond the maximum size configured for the thread pool.


How to decide pool size for Thread Pools?, availableProcessors() + 1; This is ideal pool size, if your multithreaded task is kind of computation, where threads are not getting block, wait on I/O or some combination. But if your task also includes some kind of blocking or waiting time then there is one more way to determine pool size for such tasks. Settings for the trivial experiment are: set “server thread pool size” to 1 and have a single client repeatedly makes the request described above for 30 seconds. (Click on the image to enlarge it)


Thread Pools in Java, A thread pool reuses previously created threads to execute current tasks and offers The optimum size of the thread pool depends on the number of processors  Beginning with the .NET Framework 4, the default size of the thread pool for a process depends on several factors, such as the size of the virtual address space. A process can call the ThreadPool.GetMaxThreads method to determine the number of threads.


Finally Getting the Most out of the Java Thread Pool, Finally understanding how thread pools really work in Java can be the The number of threads can be controlled by setting the parameters: Because of this, it is best used to speed up work in cases when tasks can be  The limit of 25 threads per processor was back from .NET 1.1. It was increased in .NET 2, and now: Beginning with the .NET Framework version 4, the default size of the thread pool for a process depends on several factors, such as the size of the virtual address space. A process can call the GetMaxThreads method to determine the number of threads.