Discuss the Memory Hierarchy in computer system with regard to Speed, Size and Cost.

The levels from top to bottom are the CPU registers, cache memory, main memory, secondary memory (magnetic disk, SSD) and tertiary memory (tape, optical disk).

Speed. Moving down the hierarchy, the access time increases. A disk is about a hundred thousand times slower than main memory.

Size. Moving down, the capacity increases by a large factor at each level.

Cost. Moving down, the cost per bit falls. Cache is costly per bit, so only a small amount is used.

Why it works

The hierarchy works because of the principle of locality of reference. Programs use the same data again soon (temporal locality) and use data near the data just used (spatial locality). So a small fast level can hold the data in use, and most accesses are served by it. When data is not found, a whole block is copied from the lower level to the upper level.

If the cache hit ratio is $h$, the cache time is $t_c$ and the main memory time is $t_m$, the average access time is

$$t_{avg} = h,t_c + (1-h),t_m$$

For $h = 0.95$, $t_c = 10$ ns and $t_m = 100$ ns, this is $0.95 \times 10 + 0.05 \times 100 = 14.5$ ns. This is near the speed of the cache, while the memory is as large as the main memory.

Data movement and the trade-off

The three requirements pull against each other. A fast memory needs costly technology, so it must be small. A large memory must use cheap technology, so it must be slow. The hierarchy solves this by placing a small amount of fast memory above a large amount of slow memory. The size of each level is about 100 to 1000 times that of the level above it. Each level holds a copy of the data of the level below that is likely to be used soon. A well chosen block size and a good replacement rule raise the hit ratio at every level.

Inclusion is another property. All data in the cache is also present in main memory, and all data in main memory is also present on the disk. So the lowest level always holds the complete set of data.

Result

The user gets a memory system with a speed close to the fastest level, a size equal to the largest level, and a cost near that of the cheapest level. Data moves between cache and main memory in blocks, and between main memory and disk in pages.