Why does the Exadata X5 and X6 storage nodes have only 4 Flashdisks per node?

Asset ID:	1-71-2139076.1
Update Date:	2016-06-02
Keywords:

Solution Type Technical Instruction Sure

Solution 2139076.1 : Why does the Exadata X5 and X6 storage nodes have only 4 Flashdisks per node?

Applies to:

Exadata X5-2 Hardware - Version All Versions and later
Oracle SuperCluster T5-8 Hardware - Version All Versions and later
Exadata X5-8 Hardware - Version All Versions and later
Oracle SuperCluster M6-32 Hardware - Version All Versions and later
Oracle SuperCluster M7 Hardware - Version All Versions and later
Information in this document applies to any platform.

Goal

The previous Exadata models have 16 Flash Modules/ASM FlashDisks, why do the Exadata X5 and X6 have only 4?

Solution

EXPLANATION

This is normal and expected. Previous Exadata models used 4 Flash "assemblies" which contained 4 individual "Flash Modules" (known as "FMOD's") on each Flash card, for a total of 16 Flash Modules/Flashdisks seen by the OS. In contrast, the newer Exadata Storage Servers used in X5-2, X4-8 (when equipped with X5-L storage nodes), X5-8, X6-2 and X6-8 and Supercluster (with Exadata X5-2 storage) use 4 Flash F160 (X5) or Flash F320 (X6) NVMe (Non-Volatile Memory Express) PCI-E Flash cards which have their NAND flash configured as a single module. Since these cards do not use multiple FMOD's they appear to the OS as 1 Flash disk per Flash card, or total of 4 Flash disks per server instead of 16 Flash disks per server seen in the previous models of Flash cards. Therefore the cell configuration layer also sees 1 physical disk and creates 1 lun on the whole card, total of 4 disks per Server, rather than 16 per server seen in the previous models of Exadata Storage Servers.

Example outputs

Exadata X2

NOTE - While this example is taken from an X2-2, the output for the X3 and X4 (which uses the F40 and F80 Flash respectively) looks much the same, just with more capacity (X3 = ~1488.75GB and X4 = ~2.91TB)

bash-4.1$ more flashcache-detail.out
name: xa02cel03_FLASHCACHE
cellDisk: FD_07_xa02cel03,FD_14_xa02cel03,FD_13_xa02cel03,FD_12_xa02cel03,FD_06_xa02cel03,FD_08_xa02cel03,FD_03_xa02cel03,FD_02_xa02cel03,FD_10 <----16
_xa02cel03,FD_00_xa02cel03,FD_04_xa02cel03,FD_05_xa02cel03,FD_01_xa02cel03,FD_15_xa02cel03,FD_11_xa02cel03,FD_09_xa02cel03      <------ASM flashdisks
creationTime: 2012-10-04T22:37:03-06:00
degradedCelldisks:
effectiveCacheSize: 364.75G
id: 4f89e90f-a95b-4229-bfc6-3d07555cbe35
size: 364.75G
status: normal
bash-4.1$

bash-4.1$ more physicaldisk-detail.out | egrep -i 'name|device|luns|makemodel|firmware'
[OUTPUT CUT FOR BREVITY]
name: FLASH_1_0 <------This is FMOD slot 0 on F20 card in PCI-E slot 0
deviceName: /dev/sdq
luns: 1_0
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_1_1 <-------slot 1
deviceName: /dev/sdr
luns: 1_1
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_1_2
deviceName: /dev/sds
luns: 1_2
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_1_3
deviceName: /dev/sdt
luns: 1_3
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_2_0 <------FMOD 0 on F20 card in PCI-E slot 2
deviceName: /dev/sdy
luns: 2_0
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_2_1
deviceName: /dev/sdz
luns: 2_1
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_2_2
deviceName: /dev/sdaa
luns: 2_2
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_2_3
deviceName: /dev/sdab
luns: 2_3
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_4_0
deviceName: /dev/sdm
luns: 4_0
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_4_1
deviceName: /dev/sdn
luns: 4_1
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_4_2
deviceName: /dev/sdo
luns: 4_2
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_4_3
deviceName: /dev/sdp
luns: 4_3
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_5_0
deviceName: /dev/sdu
luns: 5_0
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_5_1
deviceName: /dev/sdv
luns: 5_1
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_5_2
deviceName: /dev/sdw
luns: 5_2
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
name: FLASH_5_3
deviceName: /dev/sdx
luns: 5_3
makeModel: "Sun Flash Accelerator F20 PCIe Card"
physicalFirmware: D21Y
bash-4.1$om Word' to retain layout.]

Exadata X5 - F160 Flash

NOTE - While this output is taken an Exadata X5, the output from the Exadata X6's F320 also looks much the same but with higher capacity.

bash-4.1$ more flashcache-detail.out
name: nzepce301xm_FLASHCACHE
cellDisk: FD_03_nzepce301xm,FD_01_nzepce301xm,FD_02_nzepce301xm,FD_00_nzepce301xm <---4 ASM Flashdisks defined.
creationTime: 2015-06-12T17:29:36+00:00
degradedCelldisks:
effectiveCacheSize: 5.82122802734375T
id: da8dec9d-efc8-49d5-ba75-b88f62c7567d
size: 5.82122802734375T
status: normal

bash-4.1$ more physicaldisk-detail.out | egrep -i 'name|device|luns|makeModel|Firmware'
[OUTPUT CUT FOR BREVITY]
name: FLASH_1_1 <----F160 1.6TB Flash card in PCI-E Slot 1
deviceName: /dev/nvme3n1
luns: 1_1
makeModel: "Oracle Flash Accelerator F160 PCIe Card"
physicalFirmware: 8DV1RA12
name: FLASH_2_1 <-----Slot 2
deviceName: /dev/nvme2n1
luns: 2_1
makeModel: "Oracle Flash Accelerator F160 PCIe Card"
physicalFirmware: 8DV1RA12
name: FLASH_4_1 <-----Slot 4
deviceName: /dev/nvme0n1
luns: 4_1
makeModel: "Oracle Flash Accelerator F160 PCIe Card"
physicalFirmware: 8DV1RA12
name: FLASH_5_1 <-----Slot 5
deviceName: /dev/nvme1n1
luns: 5_1
makeModel: "Oracle Flash Accelerator F160 PCIe Card"
physicalFirmware: 8DV1RA12

bash-4.1$ egrep -i 'name|device|type|lun|size' celldisk-detail.out
[OUTPUT CUT FOR BREVITY]
name: FD_00_nzepce301xm
deviceName: /dev/nvme3n1
devicePartition: /dev/nvme3n1
diskType: FlashDisk
lun: 1_1
size: 1.455474853515625T
name: FD_01_nzepce301xm
deviceName: /dev/nvme2n1
devicePartition: /dev/nvme2n1
diskType: FlashDisk
lun: 2_1
size: 1.455474853515625T
name: FD_02_nzepce301xm
deviceName: /dev/nvme0n1
devicePartition: /dev/nvme0n1
diskType: FlashDisk
lun: 4_1
size: 1.455474853515625T
name: FD_03_nzepce301xm
deviceName: /dev/nvme1n1
devicePartition: /dev/nvme1n1
diskType: FlashDisk
lun: 5_1
size: 1.455474853515625T
bash-4.1$

Attachments

This solution has no attachment