When working with Oracle databases running on Exadata we have to understand the storage and disk architecture to be able to better deliver our work.

An Exadata Database Machine is a rack made of Database Servers, Storage Servers, Power Distribution Units and Switches.

The database and ASM instances run their processes and allocate their memory areas in the Database Servers. On the other hand we have the Storage Servers which provide the disks which combined mount the ASM diskgroups available to the databases to store data. The communication between Database Server and Storage Servers is made through Infiniband Switches until the X8 generation and starting with X8M that communication is made through RoCE (RDMA (Remote Direct Memory Access) over Converged Ethernet) Switches.

In this blog post we are going to explore the disk architecture and the relationship between the ASM Diskgroups and the Storage Server disks.

Storage Servers, depending on the generation of the hardware and/or customer choice, considering the latest generation (X9M) these are available:

For this blog post let's consider the Storage Server High Capacity since it is the one vastly used. It has twelve hard disk drives, four NVMe flash cards and twelve persistent memory cards.

Each of those we call PhysicalDisk which will then have a CellDisk created on it. CellDisks will have partitions to make GridDisks, FlashCache, FlashLog, PersistentMemoryCache, PersistentMemoryLog.

GridDisks are created by default on the CellDisks that are from the HDDs, FlashCache and FlashLog are created on the CellDisks that are from the flash cards and lastly PersistentMemoryCache and PersistentMemoryLog are created on top of the CellDisks that are from the pmem cards.

Running the command list diskmap in a Storage Server will show us exactly how the disks are organized:

Here another visualization of the disks and their health:

When checking from the ASM instance we can see the diskgroups and the space available:

To summarize...

• Storage Servers will be: Extreme Flash, High Capacity or Extended.
• Disk relationship will be: PhysicalDisk -> CellDisk -> GridDisk -> ASMDisk
• Flash disks will be used for: Smart Flash Cache and Smart Flash Log
• PMem Cards will be used for: PMem Cache, PMem Log
• Hard Disk Drives will be used for: GridDisks -> ASMDisks -> Diskgroups: DATA, RECO, SPARSE

I hope this helps you understand a bit better the Exadata disk architecture and relationship with the ASM instance and diskgroups.

Sincerely,

Franky