The idea: three ways to do more work per second
A pipelined core runs about one instruction per clock cycle. Clock rates stopped rising around 2005, so chips now get faster by doing several things at once. This page shows the three main ways. SIMD makes one instruction work on several data elements. Hardware multithreading keeps one core busy with a second thread while the first one waits. Multicore puts several whole cores on one chip.
Each one needs a different kind of parallelism in the program. SIMD needs data-level parallelism: the same operation on many elements. Multithreading and multicore need thread-level parallelism: several independent instruction streams.
SIMD: one instruction, many elements
SIMD stands for single instruction, multiple data. The core gets wide vector registers. A 128-bit register holds four 32-bit ints, each in its own lane. A vector add has four small adders, one per lane, that work at the same time. Harris & Harris call this packed arithmetic and show four 8-bit adds in one 32-bit register.
The loop c[i] = a[i] + b[i] over 4 elements takes 4 scalar instructions per element: two loads, an add and a store, so 16 in all (plus addi and bne for the loop itself). The vector version is vld, vld, addv, vst: 4 instructions for all 4 elements. Fewer instructions also means less fetch, decode and loop overhead (Demo 1: SIMD).
Real instruction sets: x86 SSE (128-bit), AVX (256-bit) and AVX-512, ARM NEON (128-bit) and SVE, RISC-V V. Compilers turn simple loops into vector code (auto-vectorization) when the elements don't depend on each other. GPUs push the same idea much further.
Hardware multithreading
A cache miss to main memory costs tens to hundreds of cycles. An in-order core can't run the instructions that need the loaded value, so its issue slots go empty. Even an out-of-order core runs out of independent work after a while.
A multithreaded core keeps the state of several threads in hardware: one thread context per thread, which is a PC and a full register file. The ALUs, caches and pipeline are shared. Switching threads means only choosing another PC to fetch from, so it takes no time, unlike an OS context switch, which saves registers to memory. When thread A stalls, the core issues thread B's instructions in the empty slots (Demo 3: multithreading). On the page that turns 22 cycles with 8 empty slots into 14 cycles with none. Thread A itself is not faster: multithreading raises throughput and utilization, not single-thread speed.
| Kind | When it switches | Example |
|---|---|---|
| Coarse-grained | On a long stall such as a cache miss. Refilling the pipeline costs a few cycles. | Intel Itanium 2 9000, Montecito (switch on event) |
| Fine-grained | Every cycle, round robin over the ready threads. Hides short stalls too, but one thread runs slower. | Sun UltraSPARC T1 (Niagara), GPUs |
| Simultaneous (SMT) | No switch: a superscalar core issues from several threads in the same cycle. | Intel Hyper-Threading, AMD Zen, IBM POWER |
The page uses the simplest policy: issue from A, and from B whenever A can't issue, with no switch cost. The OS sees each hardware thread as a logical CPU: a 4-core chip with 2-way SMT shows 8 CPUs.
Multicore and Amdahl's law
A multicore chip (a homogeneous multiprocessor) has several identical cores, each with its own pipeline and L1 caches, sharing the last-level cache and memory. The program must be split into threads; the OS runs one per core. The cores keep their caches consistent with a coherence protocol such as MESI.
Most programs have a serial part that only one core can do: reading the input, setting up, combining the results. If a fraction s of the work is serial, Amdahl's law gives the speedup on N cores:
With s = 20 %, 4 cores give 1 / (0.2 + 0.8 / 4) = 2.5×, and no number of cores gives more than 1 / s = 5× (Demo 4: 20% serial). With 40 % serial, 4 cores give only 1.8× (Demo 5: Amdahl's law).
Gustafson's law is the hopeful side: with more cores people run bigger problems, the parallel part grows and the serial fraction shrinks.
Which one helps which program
| Technique | Parallelism it needs | What it costs in hardware | Helps most |
|---|---|---|---|
| SIMD | The same operation on many elements | Wide registers and lane ALUs | Loops over arrays: images, audio, matrices |
| Multithreading | Several threads, often stalled | Extra PCs and register files (a few %) | Servers, databases: many cache misses |
| Multicore | Several threads, little serial work | Whole extra cores | Work that splits into big independent parts |
A modern chip uses all three at once: many cores, each with 2-way SMT and 256-bit or 512-bit vector units. See also Concurrency vs Parallelism for the software view of threads on cores, the MIPS pipeline and superscalar and out-of-order execution for what one core does, MESI and false sharing for how cores share cache lines, and From a Java Thread to a CPU Core for hyper-threading siblings as the OS sees them.
What the page leaves out
The SIMD scene skips loop overhead, alignment, a tail loop for lengths that are not a multiple of 4, masks and gather/scatter. The multithreading scene has one issue slot per cycle, a 0-cycle thread switch and no cache contention between the threads, which can cost real SMT cores some of the gain. The multicore scene assumes perfect load balance and no cost for starting threads, synchronizing or moving data between caches, all of which make the real speedup lower than Amdahl's law. Heterogeneous chips (big and little cores, GPUs) and memory bandwidth limits are left out.