Design and explain the fundamental concept of computer memory hierarchy?

Design

The levels are arranged from the top (nearest the CPU) to the bottom:

1. CPU registers: smallest and fastest, inside the processor.
2. Cache memory (SRAM): small and very fast, in one or more levels (L1, L2, L3).
3. Main memory (DRAM): medium speed, a few GB.
4. Secondary memory (disk, SSD): large and slow, holds all programs and data.
5. Tertiary memory (tape, optical): very large, very slow, used for backup.

Going down, the capacity and access time increase and the cost per bit falls. Going up, the reverse is true.

Basic concept: locality of reference

The design works because programs show locality of reference. Temporal locality means that a recently used item is likely to be used again soon. Spatial locality means that items near a used item are likely to be used next. So data in use is kept in the upper levels, and data moves between levels in blocks. When the CPU finds the data in the upper level it is a hit, and otherwise it is a miss and a block is brought from the lower level.

Performance

For a two-level system with hit ratio $h$, the average access time is

$$t_{avg} = h,t_1 + (1-h),t_2$$

With $h = 0.95$, $t_1 = 10$ ns and $t_2 = 100$ ns, $t_{avg} = 14.5$ ns. This is close to the speed of the fast level, but the size is that of the large level.

Data movement

Between cache and main memory, data moves in blocks and is managed by hardware. Between main memory and disk, data moves in pages and is managed by the operating system through virtual memory. Registers are loaded by the program instructions.