Exadata X8M Smart Flash Cache : Write Back and Write Through mode Conversions

Exadata smart flash cache can be configured either in Write-Through mode or Write-Back mode, default being Write-Through mode.

Write-Through mode reads from flash cache but writes to ASM disk.

In Write-Back mode, both reads and writes are cached in flash cache, increasing performance.

No downtime is needed for this activity, but there will be decrease in performance while changing the mode as flash cache need to be dropped and recreated.

=> Create a file named xxlistcells on any of the cell storage server that contains names of all Exadata cells on separate lines

[celladmin@celstorage01 ~]$ cat xxlistcells
celstorage01
celstorage02
celstorage03
[celladmin@celstorage01 ~]$

=> Check the status of all physical disks, status should be Normal. Below command should not return any rows

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e list physicaldisk attributes name, status where status !=NORMAL
[celladmin@celstorage01 ~]$

=> Verify the flashcachemode on all cells, it should be Write-Through by default.

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e list cell attributes flashCacheMode detail
celstorage01: flashCacheMode:         WriteThrough
celstorage02: flashCacheMode:         WriteThrough
celstorage03: flashCacheMode:         WriteThrough
[celladmin@celstorage01 ~]$

=> Verify grid disk attribute cachingpolicy. As per Oracle recommendation we should not enable flashcache for RECO disk group. If caching policy for RECO disk group is set to DEFAULT then it used flashcahce, so we have to change it to NONE for RECO disk group, so that flashcache will not be used for any blocks stored on the RECO disk group

[celladmin@celstorage01 ~]$ dcli -g xxlistcells "cellcli -e list griddisk attributes name,cachingPolicy where name like \'RECO.*\'"
celstorage01: RECO_CD_00_celstorage01 default
celstorage01: RECO_CD_01_celstorage01 default
celstorage01: RECO_CD_02_celstorage01 default
celstorage01: RECO_CD_03_celstorage01 default
celstorage01: RECO_CD_04_celstorage01 default
celstorage01: RECO_CD_05_celstorage01 default
celstorage01: RECO_CD_06_celstorage01 default
celstorage01: RECO_CD_07_celstorage01 default
celstorage01: RECO_CD_08_celstorage01 default
celstorage01: RECO_CD_09_celstorage01 default
celstorage01: RECO_CD_10_celstorage01 default
celstorage01: RECO_CD_11_celstorage01 default
celstorage02: RECO_CD_00_celstorage02 default
celstorage02: RECO_CD_01_celstorage02 default
celstorage02: RECO_CD_02_celstorage02 default
celstorage02: RECO_CD_03_celstorage02 default
celstorage02: RECO_CD_04_celstorage02 default
celstorage02: RECO_CD_05_celstorage02 default
celstorage02: RECO_CD_06_celstorage02 default
celstorage02: RECO_CD_07_celstorage02 default
celstorage02: RECO_CD_08_celstorage02 default
celstorage02: RECO_CD_09_celstorage02 default
celstorage02: RECO_CD_10_celstorage02 default
celstorage02: RECO_CD_11_celstorage02 default
celstorage03: RECO_CD_00_celstorage03 default
celstorage03: RECO_CD_01_celstorage03 default
celstorage03: RECO_CD_02_celstorage03 default
celstorage03: RECO_CD_03_celstorage03 default
celstorage03: RECO_CD_04_celstorage03 default
celstorage03: RECO_CD_05_celstorage03 default
celstorage03: RECO_CD_06_celstorage03 default
celstorage03: RECO_CD_07_celstorage03 default
celstorage03: RECO_CD_08_celstorage03 default
celstorage03: RECO_CD_09_celstorage03 default
celstorage03: RECO_CD_10_celstorage03 default
celstorage03: RECO_CD_11_celstorage03 default
[celladmin@celstorage01 ~]$

=> Update the cachingPolicy attributes of disks in the RECO disk group to NONE.

[celladmin@celstorage01 ~]$ dcli -c celstorage01 cellcli -e alter griddisk RECO_CD_00_celstorage01,RECO_CD_01_celstorage01,RECO_CD_02_celstorage01,RECO_CD_03_celstorage01,RECO_CD_04_celstorage01,RECO_CD_05_celstorage01,RECO_CD_06_celstorage01,RECO_CD_07_celstorage01,RECO_CD_08_celstorage01,RECO_CD_09_celstorage01,RECO_CD_10_celstorage01,RECO_CD_11_celstorage01 cachingPolicy="none"
celstorage01: GridDisk RECO_CD_00_celstorage01 successfully altered
celstorage01: GridDisk RECO_CD_01_celstorage01 successfully altered
celstorage01: GridDisk RECO_CD_02_celstorage01 successfully altered
celstorage01: GridDisk RECO_CD_03_celstorage01 successfully altered
celstorage01: GridDisk RECO_CD_04_celstorage01 successfully altered
celstorage01: GridDisk RECO_CD_05_celstorage01 successfully altered
celstorage01: GridDisk RECO_CD_06_celstorage01 successfully altered
celstorage01: GridDisk RECO_CD_07_celstorage01 successfully altered
celstorage01: GridDisk RECO_CD_08_celstorage01 successfully altered
celstorage01: GridDisk RECO_CD_09_celstorage01 successfully altered
celstorage01: GridDisk RECO_CD_10_celstorage01 successfully altered
celstorage01: GridDisk RECO_CD_11_celstorage01 successfully altered

[celladmin@celstorage01 ~]$ dcli -c celstorage02 cellcli -e alter griddisk RECO_CD_00_celstorage02,RECO_CD_01_celstorage02,RECO_CD_02_celstorage02,RECO_CD_03_celstorage02,RECO_CD_04_celstorage02,RECO_CD_05_celstorage02,RECO_CD_06_celstorage02,RECO_CD_07_celstorage02,RECO_CD_08_celstorage02,RECO_CD_09_celstorage02,RECO_CD_10_celstorage02,RECO_CD_11_celstorage02 cachingPolicy="none"
celstorage02: GridDisk RECO_CD_00_celstorage02 successfully altered
celstorage02: GridDisk RECO_CD_01_celstorage02 successfully altered
celstorage02: GridDisk RECO_CD_02_celstorage02 successfully altered
celstorage02: GridDisk RECO_CD_03_celstorage02 successfully altered
celstorage02: GridDisk RECO_CD_04_celstorage02 successfully altered
celstorage02: GridDisk RECO_CD_05_celstorage02 successfully altered
celstorage02: GridDisk RECO_CD_06_celstorage02 successfully altered
celstorage02: GridDisk RECO_CD_07_celstorage02 successfully altered
celstorage02: GridDisk RECO_CD_08_celstorage02 successfully altered
celstorage02: GridDisk RECO_CD_09_celstorage02 successfully altered
celstorage02: GridDisk RECO_CD_10_celstorage02 successfully altered
celstorage02: GridDisk RECO_CD_11_celstorage02 successfully altered

[celladmin@celstorage01 ~]$ dcli -c celstorage03 cellcli -e alter griddisk RECO_CD_00_celstorage03,RECO_CD_01_celstorage03,RECO_CD_02_celstorage03,RECO_CD_03_celstorage03,RECO_CD_04_celstorage03,RECO_CD_05_celstorage03,RECO_CD_06_celstorage03,RECO_CD_07_celstorage03,RECO_CD_08_celstorage03,RECO_CD_09_celstorage03,RECO_CD_10_celstorage03,RECO_CD_11_celstorage03 cachingPolicy="none"
celstorage03: GridDisk RECO_CD_00_celstorage03 successfully altered
celstorage03: GridDisk RECO_CD_01_celstorage03 successfully altered
celstorage03: GridDisk RECO_CD_02_celstorage03 successfully altered
celstorage03: GridDisk RECO_CD_03_celstorage03 successfully altered
celstorage03: GridDisk RECO_CD_04_celstorage03 successfully altered
celstorage03: GridDisk RECO_CD_05_celstorage03 successfully altered
celstorage03: GridDisk RECO_CD_06_celstorage03 successfully altered
celstorage03: GridDisk RECO_CD_07_celstorage03 successfully altered
celstorage03: GridDisk RECO_CD_08_celstorage03 successfully altered
celstorage03: GridDisk RECO_CD_09_celstorage03 successfully altered
celstorage03: GridDisk RECO_CD_10_celstorage03 successfully altered
celstorage03: GridDisk RECO_CD_11_celstorage03 successfully altered
[celladmin@celstorage01 ~]$

=> Now verify the RECO disk group cachingpolicy. It should be NONE, now RECO disks will not use Write-Back mode post conversion.

[celladmin@celstorage01 ~]$ dcli -g xxlistcells "cellcli -e list griddisk attributes name,cachingPolicy where name like \'RECO.*\'"
celstorage01: RECO_CD_00_celstorage01 none
celstorage01: RECO_CD_01_celstorage01 none
celstorage01: RECO_CD_02_celstorage01 none
celstorage01: RECO_CD_03_celstorage01 none
celstorage01: RECO_CD_04_celstorage01 none
celstorage01: RECO_CD_05_celstorage01 none
celstorage01: RECO_CD_06_celstorage01 none
celstorage01: RECO_CD_07_celstorage01 none
celstorage01: RECO_CD_08_celstorage01 none
celstorage01: RECO_CD_09_celstorage01 none
celstorage01: RECO_CD_10_celstorage01 none
celstorage01: RECO_CD_11_celstorage01 none
celstorage02: RECO_CD_00_celstorage02 none
celstorage02: RECO_CD_01_celstorage02 none
celstorage02: RECO_CD_02_celstorage02 none
celstorage02: RECO_CD_03_celstorage02 none
celstorage02: RECO_CD_04_celstorage02 none
celstorage02: RECO_CD_05_celstorage02 none
celstorage02: RECO_CD_06_celstorage02 none
celstorage02: RECO_CD_07_celstorage02 none
celstorage02: RECO_CD_08_celstorage02 none
celstorage02: RECO_CD_09_celstorage02 none
celstorage02: RECO_CD_10_celstorage02 none
celstorage02: RECO_CD_11_celstorage02 none
celstorage03: RECO_CD_00_celstorage03 none
celstorage03: RECO_CD_01_celstorage03 none
celstorage03: RECO_CD_02_celstorage03 none
celstorage03: RECO_CD_03_celstorage03 none
celstorage03: RECO_CD_04_celstorage03 none
celstorage03: RECO_CD_05_celstorage03 none
celstorage03: RECO_CD_06_celstorage03 none
celstorage03: RECO_CD_07_celstorage03 none
celstorage03: RECO_CD_08_celstorage03 none
celstorage03: RECO_CD_09_celstorage03 none
celstorage03: RECO_CD_10_celstorage03 none
celstorage03: RECO_CD_11_celstorage03 none
[celladmin@celstorage01 ~]$

=> Drop Exadata Smart Flash Cache

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e drop flashcache
celstorage01: Flash cache celstorage01_FLASHCACHE successfully dropped
celstorage02: Flash cache celstorage02_FLASHCACHE successfully dropped
celstorage03: Flash cache celstorage03_FLASHCACHE successfully dropped
[celladmin@celstorage01 ~]$

=> Enable Write-Back Flash Cache 

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e alter cell flashCacheMode = WriteBack
celstorage01: Cell celstorage01 successfully altered
celstorage02: Cell celstorage02 successfully altered
celstorage03: Cell celstorage03 successfully altered
[celladmin@celstorage01 ~]$

=> Recreate Smart Flash Cache

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e create flashcache all
celstorage01: Flash cache celstorage01_FLASHCACHE successfully created
celstorage02: Flash cache celstorage02_FLASHCACHE successfully created
celstorage03: Flash cache celstorage03_FLASHCACHE successfully created
[celladmin@celstorage01 ~]$

=> Verify flashcachemode attribute

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e list cell attributes flashCacheMode detail
celstorage01: flashCacheMode:         WriteBack
celstorage02: flashCacheMode:         WriteBack
celstorage03: flashCacheMode:         WriteBack
[celladmin@celstorage01 ~]$

=> Verify that flash cache is being used 

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e list metriccurrent FC_BY_DIRTY
celstorage01: FC_BY_DIRTY FLASHCACHE 7.777 MB
celstorage02: FC_BY_DIRTY FLASHCACHE 8.820 MB
celstorage03: FC_BY_DIRTY FLASHCACHE 10.418 MB
[celladmin@celstorage01 ~]$

=> To change the mode from WriteBack to WriteThrough on Exadata X8M system with PMEM cache, then verify if the PMEM cache is in WriteBack mode, you first need to modify the PMEM cache to WriteThrough. 

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e list cell attributes pmemCacheMode detail
celstorage01: pmemCacheMode:          WriteThrough
celstorage02: pmemCacheMode:          WriteThrough
celstorage03: pmemCacheMode:          WriteThrough
[celladmin@celstorage01 ~]$

=> Check the status of all physical disks, status should be Normal. Below command should not return any rows

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e list physicaldisk attributes name, status where status !=NORMAL
[celladmin@celstorage01 ~]$

=> Flush the write back flash cache on all the cells 

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e alter flashcache all flush
celstorage01: Flash cache celstorage01_FLASHCACHE altered successfully
celstorage02: Flash cache celstorage02_FLASHCACHE altered successfully
celstorage03: Flash cache celstorage03_FLASHCACHE altered successfully
[celladmin@celstorage01 ~]$

=> Verify the flash cache dirty data 

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e list metriccurrent FC_BY_DIRTY
celstorage01: FC_BY_DIRTY FLASHCACHE 0.000 MB
celstorage02: FC_BY_DIRTY FLASHCACHE 0.000 MB
celstorage03: FC_BY_DIRTY FLASHCACHE 0.000 MB
[celladmin@celstorage01 ~]$

=> Drop flashcache 

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e drop flashcache
celstorage01: Flash cache celstorage01_FLASHCACHE successfully dropped
celstorage02: Flash cache celstorage02_FLASHCACHE successfully dropped
celstorage03: Flash cache celstorage03_FLASHCACHE successfully dropped
[celladmin@celstorage01 ~]$

=> Convert to write through mode

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e alter cell flashCacheMode = WriteThrough
celstorage01: Cell celstorage01 successfully altered
celstorage02: Cell celstorage02 successfully altered
celstorage03: Cell celstorage03 successfully altered
[celladmin@celstorage01 ~]$

=> Create flashcache 

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e create flashcache all
celstorage01: Flash cache celstorage01_FLASHCACHE successfully created
celstorage02: Flash cache celstorage02_FLASHCACHE successfully created
celstorage03: Flash cache celstorage03_FLASHCACHE successfully created
[celladmin@celstorage01 ~]$

=> Verify flashcache mode 

[celladmin@celstorage01 ~]$ dcli -g xxlistcells cellcli -e list cell attributes flashCacheMode detail
celstorage01: flashCacheMode:         WriteThrough
celstorage02: flashCacheMode:         WriteThrough
celstorage03: flashCacheMode:         WriteThrough
[celladmin@celstorage01 ~]$ 

That's it.