Series: Part 1 of 3

Part 1 Exadata Health Check

A Practical DBA Guide

• Part 1: Exadata Health Check
• Part 2: Exadata Performance Investigation
• Part 3: How to Tune Oracle Database to Use Exadata Efficiently

Introduction

When an Oracle Database is running on Exadata, DBAs usually have several important questions:

• Do we need to do anything as DBAs to take full advantage of Exadata?
• If the database becomes slow, how do we determine whether the problem is in the database or in the Exadata infrastructure?
• What should we check before changing parameters, rebuilding indexes, or tuning SQL?
• How can we verify that the Exadata platform itself is healthy?

These questions are the reason for this three-part Exadata Investigation & Performance Tuning series.

In this first article, we concentrate on one question:

Is the Exadata infrastructure healthy?

The series is:

• Part 1: Exadata Health Check
• Part 2: Exadata Performance Investigation
• Part 3: How to Tune Oracle Database to Use Exadata Efficiently

The purpose of Part 1 is not to prove that every component is perfect. The purpose is to collect enough evidence to decide whether the infrastructure is a credible cause of the performance problem.

Understanding the Investigation Layers

A report such as “the database is slow” can originate from many layers.

Use this as an investigation model:

Application → DB Instance → RAC/GI → ASM → RoCE/RDMA → 
Storage Cells → XRMEM/Flash → CellDisk/GridDisk → Physical Storage

This is not a literal statement that every I/O request passes through every component in that order.

Depending on the symptom, the investigation may involve:

SQL/Plan | DB CPU/Memory | Wait Events | RAC/Cache Fusion | 
ASM | Cell CPU/Memory | Flash/XRMEM | Network | Disk | IORM

This is why we should not immediately blame storage, RAC, the database, or Smart Scan.

Follow the evidence, correlate by time, and compare with known-good behavior.

What We Will Check in Part 1

This health check covers:

• Exachk
• Database and instance status
• Database-server OS health
• Grid Infrastructure and RAC
• ASM
• Storage cells
• Exadata alerts
• Physical disks
• CellDisks and GridDisks
• Storage mapping
• CellDisk I/O metrics
• Current versus historical metrics
• Cell CPU and memory
• Smart Flash Cache
• XRMEM / RDMA cache
• Single-block I/O
• Smart Flash Log
• RoCE / RDMA storage network
• RAC interconnect
• IORM
• Cell-to-cell comparison
• Performance baselines

1. Start With Exachk

My troubleshooting starting point depends on the reported problem, for sure Exachk is not a start point.

However, for the overall health of an Exadata environment, Exachk should be part of the regular operational process.

Do not simply open the report and count warnings. Concentrate on:

FAIL | WARNING | Critical Issues | Version Recommendations |
MAA Scorecard | Configuration Differences

A better approach is:

Previous accepted Exachk → Current Exachk → New FAIL/WARNING? →
→ Configuration changed? → New exposure?

A newly appearing warning is generally more interesting than a warning that has existed for months and has already been reviewed.

DBA practice

Keep a known-good Exachk report before major maintenance. Compare the post-maintenance report against it. That turns Exachk into a change-detection tool rather than a static checklist.

2. Check Database and Instance Status

For RAC:

SELECT inst_id,
       instance_name,
       host_name,
       version,
       status,
       database_status,
       startup_time
FROM gv$instance
ORDER BY inst_id;

Confirm that every expected instance is running on the correct node.

Then check the database:

SELECT name,
       db_unique_name,
       open_mode,
       database_role,
       protection_mode
FROM v$database;

For a normal primary database, for example:

DATABASE_ROLE = PRIMARY | OPEN_MODE = READ WRITE

When Data Guard is involved, verify that role and open mode match the design.

3. Check Database-Server OS Health

Before jumping directly to the storage cells, check the database servers.

Useful commands include:

uptime
vmstat 1 10
mpstat -P ALL 1 10
free -g
top

Look for:

• Sustained high CPU
• High run queue
• Memory pressure
• Swapping
• One RAC node much busier than its peers
• Unexpected processes consuming CPU
• Network errors

Example:

DBNode01 CPU 32% | DBNode02 CPU 35% | 
DBNode03 CPU 98% | DBNode04 CPU 31%

The real question is not whether 98% is “high.”

Why is DBNode03 behaving differently from the other nodes?

Do not diagnose from one snapshot. Use:

Sustained problem + Time correlation + Workload correlation

4. Check Grid Infrastructure and RAC Resources

From the GI environment:

crsctl stat res -t

Review:

• Database resources
• RAC instances
• VIPs
• SCAN listeners
• Local listeners
• ASM
• Disk groups
• Services

Pay attention to:

OFFLINE | INTERMEDIATE | UNKNOWN

But do not assume every OFFLINE resource is a problem.

Is the resource running where it is expected to run?

A service configured for two nodes does not need to be online on every node.

5. Check ASM Health

Check disk groups:

SELECT name,
       state,
       type,
       total_mb,
       free_mb,
       ROUND(free_mb / NULLIF(total_mb,0) * 100,2) free_pct
FROM v$asm_diskgroup
ORDER BY name;

Review:

• Disk groups mounted
• Free space
• DATA versus RECO utilization
• Abnormal growth
• Space available for rebalance and maintenance

Also:

asmcmd lsdg

Then inspect disks:

SELECT group_number,
       disk_number,
       name,
       path,
       header_status,
       mode_status,
       state,
       mount_status,
       total_mb,
       free_mb
FROM v$asm_disk
ORDER BY group_number, disk_number;

Values such as OFFLINE, CLOSED, UNKNOWN, or FORMER require interpretation in context. FORMER does not automatically mean failure.

DBA Note — CellCLI, dcli, or exadcli?

For repeated checks across all cells, centralized execution with dcli or exadcli can be very useful.

For this article, commands are intentionally shown one cell at a time:

First let us understand the output. Then automate the repetition.

6. Check Storage Cell Status

Start CellCLI:

cellcli

Then:

CellCLI> LIST CELL DETAIL

In a multi-cell system, compare peers:

Cell01 CPU 25% | Cell02 CPU 28% | Cell03 CPU 97%

The important question is:

Why is Cell03 behaving differently from Cell01 and Cell02 during the same workload?

Peer comparison is one of the strongest Exadata troubleshooting techniques.

7. Check Exadata Alerts

Run:

CellCLI> LIST ALERTHISTORY
CellCLI> LIST ALERTHISTORY DETAIL
CellCLI> LIST ALERTHISTORY WHERE severity = 'critical' DETAIL

Look for:

• Physical-disk failures
• Flash failures
• Predictive failures
• Temperature issues
• Memory problems
• Filesystem issues
• CELLSRV or other service restarts
• Network problems
• Hardware errors

Severity can include:

clear | info | warning | critical

A performance incident that coincides with a hardware alert must be treated differently from one where all hardware is healthy.

8. Check Physical Disks, CellDisks, and GridDisks

Physical disks:

CellCLI> LIST PHYSICALDISK
CellCLI> LIST PHYSICALDISK DETAIL

CellDisks:

CellCLI> LIST CELLDISK
CellCLI> LIST CELLDISK WHERE status != 'normal' ATTRIBUTES name,status

GridDisks:

CellCLI> LIST GRIDDISK ATTRIBUTES name,status
CellCLI> LIST GRIDDISK WHERE status != 'active' ATTRIBUTES name,status

Look for failed, missing, predictive-failure, or abnormal components.

9. Understand the Storage Mapping

A useful simplified relationship is:

Physical Device/LUN → CellDisk → GridDisk →
ASM Disk → ASM Disk Group → Database Files

Use:

CellCLI> LIST DISKMAP

This helps correlate a suspicious ASM disk with the GridDisk, CellDisk, and physical device behind it.

10. Check CellDisk I/O Metrics

Exadata metric names are systematic.

Example:

CD_IO_RQ_R_SM

means:

CD=CellDisk | IO=I/O | RQ=Requests | R=Read | SM=Small

Similarly:

CD_IO_RQ_R_LG

represents large-read requests.

For small reads, correlate:

CD_IO_RQ_R_SM_SEC → Requests/sec  
CD_IO_BY_R_SM_SEC → Throughput 
CD_IO_TM_R_SM_RQ → Avg latency/request

For large reads:

CD_IO_RQ_R_LG_SEC → Requests/sec 
CD_IO_BY_R_LG_SEC → Throughput 
CD_IO_TM_R_LG_RQ → Avg latency/request

Also review:

CD_IO_UTIL | CD_IO_UTIL_SM | CD_IO_UTIL_LG | CD_IO_ERRS

One metric alone rarely tells the story.

Discover metrics

CellCLI> LIST METRICCURRENT WHERE objectType = 'CELLDISK'
CellCLI> LIST METRICDEFINITION ATTRIBUTES name,description WHERE objectType = 'CELLDISK'
CellCLI> LIST METRICDEFINITION CD_IO_RQ_R_LG DETAIL

Cumulative versus rate

CD_IO_RQ_R_LG     → cumulative
CD_IO_RQ_R_LG_SEC → rate

A cumulative count of 50 million does not mean 50 million current IOPS.

11. Current Metrics Versus Historical Metrics

METRICCURRENT answers:

What is happening now?

For example:

CellCLI> LIST METRICCURRENT CL_CPUT DETAIL

If the incident happened earlier, use history:

CellCLI> LIST METRICHISTORY
CellCLI> LIST METRICHISTORY CD_IO_RQ_R_LG DETAIL

The rule is:

Problem now → METRICCURRENT
Problem in past → METRICHISTORY + AWR/ASH + application timeline

Collect the evidence before it disappears.

12. Check Cell CPU and Memory

CPU:

CellCLI> LIST METRICCURRENT CL_CPUT DETAIL

Related metrics can include:

CL_CPUT_CS | CL_CPUT_MS

Ask:

Sustained? | All cells or one? | Same incident window? | 
Smart Scan load? | Maintenance/rebalance?

Memory metrics:

CL_MEMUT | CL_MEMUT_CS | CL_MEMUT_MS
CL_SWAP_IN_BY_SEC | CL_SWAP_OUT_BY_SEC | CL_SWAP_USAGE | CL_RUNQ

Examples:

CellCLI> LIST METRICCURRENT CL_MEMUT DETAIL
CellCLI> LIST METRICCURRENT CL_SWAP_USAGE DETAIL

A high memory percentage alone is not enough. Stronger evidence is:

Memory pressure + Swap activity + Alerts + Performance degradation

13. Check Smart Flash Cache

Configuration:

CellCLI> LIST FLASHCACHE DETAIL

Metrics:

CellCLI> LIST METRICCURRENT WHERE objectType = 'FLASHCACHE'

Focus on concepts:

OLTP reads from Flash | DW reads from Flash | 
Misses | Bypass | Utilization | Writes

Example metrics:

FC_IO_RQ_R
FC_IO_RQ_R_DW
FC_IO_RQ_R_MISS
FC_IO_RQ_R_MISS_DW

Database-side checks:

SELECT name,
       value
FROM v$sysstat
WHERE name LIKE 'cell flash%'
ORDER BY name;

Also review:

physical read requests optimized
physical read total bytes optimized

Do not treat these as a pure Flash hit ratio because optimized reads can include other Exadata optimizations.

14. Check XRMEM / RDMA Reads

On supported systems:

Database Buffer Cache → XRMEM → Smart Flash Cache → Persistent Storage

Configuration:

CellCLI> LIST XRMEMCACHE DETAIL

Database statistics:

SELECT name,
       value
FROM v$sysstat
WHERE name LIKE 'cell RDMA%'
   OR name LIKE 'cell xrmem%'
ORDER BY name;

Possible statistics include:

cell RDMA reads
cell RDMA reads eligible
cell RDMA reads issued
cell RDMA reads rejected - ineligible
cell xrmem cache read hits

Some XRMEM/RDMA reads bypass the normal CELLSRV path, so database statistics and waits are especially important.

15. Check Single-Block I/O Waits

Single-block I/O matters greatly for OLTP and index access.

SELECT event,
       total_waits,
       ROUND(time_waited_micro / 1000000,2) time_s,
       ROUND(time_waited_micro / NULLIF(total_waits,0) / 1000,3) avg_ms
FROM v$system_event
WHERE event LIKE 'cell single block%'
ORDER BY time_waited_micro DESC;

Depending on software versions, waits may identify Flash, partner Flash, XRMEM/RDMA, or another path.

The better question becomes:

Which single-block I/O path is slow?

16. Check Smart Flash Log

At database level:

log file sync          → foreground commit wait
log file parallel write → LGWR redo-write wait

Check LGWR:

SELECT event,
       total_waits,
       ROUND(time_waited_micro / 1000000,2) total_seconds,
       ROUND(time_waited_micro / NULLIF(total_waits,0) / 1000,3) avg_ms
FROM v$system_event
WHERE event = 'log file parallel write';

At cell level:

CellCLI> LIST FLASHLOG DETAIL
CellCLI> LIST METRICCURRENT WHERE objectType = 'FLASHLOG'

Useful metrics can include:

FL_ACTUAL_OUTLIERS | FL_DISK_FIRST | FL_DISK_IO_ERRS

17. Check the RoCE / RDMA Storage Network

A storage problem and a storage-network problem can look identical from the database.

Useful OS checks can include:

ip -s link
rdma link

Look for errors, drops, retransmission, RDMA problems, or one interface behaving differently from peers.

Discover Exadata host-interconnect metrics:

CellCLI> LIST METRICDEFINITION ATTRIBUTES name,description WHERE objectType = 'HOST_INTERCONNECT'

Examples:

N_MB_DROP | N_MB_DROP_SEC | N_MB_RDMA_DROP | N_MB_RDMA_DROP_SEC | N_MB_RESENT | N_RDMA_RETRY_TM

At database level we have below wait event:

cell interconnect retransmit during physical read

One occurrence is not proof. Strong evidence is:

High frequency + Performance impact + Network/RDMA evidence + Same time window

Cable Topology Verification: Run the logical validation script to ensure hosts and switches connect properly:

# /opt/oracle.SupportTools/ibdiagtools/verify_roce_cables.py

Verify the proper oracle-rdma-release software versions are being used on the database servers.

rpm -qa |grep oracle-rdma-release

Check the adapter firmware versions on the database servers

# /opt/oracle.SupportTools/CheckHWnFWProfile -action list

Fabric Discovery & Scanning: Use newer diagnostic options like rocelinkinfo, configure_roce_hostinfo.sh, or health-check utilities to map and inspect the active RoCE links

18. Check RAC Interconnect Health

Do not confuse Exadata storage-network traffic with RAC Cache Fusion traffic.

SELECT inst_id,
       event,
       total_waits,
       ROUND(time_waited_micro / 1000000,2) total_seconds
FROM gv$system_event
WHERE wait_class = 'Cluster'
ORDER BY time_waited_micro DESC;

Typical waits can include:

gc cr request
gc current request
gc buffer busy acquire
gc buffer busy release

If Cluster waits dominate while User I/O is normal, investigate hot blocks, service placement, instance affinity, inter-instance traffic, and RAC interconnect before blaming storage.

19. Check IORM

CellCLI> LIST IORMPLAN DETAIL

The question is:

Is storage slow, or is this database intentionally receiving fewer resources?

A low-priority workload delayed because IORM protects production is not a hardware failure.

20. Compare All Storage Cells

Do not average away a bad cell.

Use a compact peer comparison:

Cell01: CPU 31% | Large Read 1.2 GB/s | Small Read 14K IOPS | Errors 0
Cell02: CPU 29% | Large Read 1.18 GB/s | Small Read 13K IOPS | Errors 0
Cell03: CPU 93% | Large Read 0.41 GB/s | Small Read 5K IOPS | Errors 12

Cell03 deserves investigation.

Possible causes:

Disk | Flash | Cell CPU
Memory | Network | Data distribution
Rebalance | Maintenance | Hardware

Follow the evidence.

21. Build a Known-Good Baseline

The best comparison is usually:

Current Exadata ↔ Known-good behavior of the same Exadata

Capture baselines for:

• Exachk
• DB-server CPU/memory
• Cell CPU/memory
• Flash activity
• CellDisk latency
• Small-read IOPS
• Large-read throughput
• Smart Scan throughput
• RDMA activity
• Host-interconnect retransmissions
• Database AAS
• Top waits
• RAC waits
• Redo latency
• IORM
• Storage utilization

Example:

NORMAL 10:00 → Cell CPU 30% | Smart Scan 15 GB/s | DB AAS 20
PROBLEM 10:00 → Cell CPU 95% | Smart Scan 4 GB/s  | DB AAS 110

Now we have evidence.

22. My Exadata Infrastructure Investigation Workflow

DB slow → Define scope/time → DB-node health → RAC/GI →
ASM → Cells → CPU/Memory/Flash/XRMEM/Disk → 
RoCE/RDMA → IORM → Compare peers → Healthy?

If No, fix or escalate the infrastructure issue.

If Yes, move to:

AWR/ASH → Waits → SQL → Execution Plan → Smart Scan/Offload → Workload design

That is Part 2.

This is a guide, not a rigid script. Use the evidence already available.

Final Exadata Health-Check Checklist

Database Servers

[ ] Expected RAC instances online
[ ] Database role/open mode correct
[ ] DB-server CPU normal
[ ] No unexpected memory pressure/swapping
[ ] Nodes reasonably balanced

Grid / RAC

[ ] GI resources in expected state
[ ] VIP/SCAN/listeners healthy
[ ] Services on expected nodes
[ ] No unexplained Cluster-wait issue

ASM

[ ] Disk groups mounted
[ ] Free space acceptable
[ ] ASM disks in expected state
[ ] No unexplained offline/missing disks

Storage Cells

[ ] Cell services healthy
[ ] No unexplained critical/warning alerts
[ ] CPU reasonable
[ ] Memory/swap healthy
[ ] No unexplained peer imbalance

Storage / Cache / Network

[ ] Physical disks normal
[ ] CellDisks normal
[ ] GridDisks active
[ ] CellDisk I/O latency understood
[ ] Flash/XRMEM healthy
[ ] No abnormal RDMA drops/retransmission
[ ] IORM behavior understood

Operations

[ ] Latest Exachk reviewed
[ ] New FAIL/WARNING items understood
[ ] Current health compared with baseline

Conclusion

A proper Exadata health check is not only:

CellCLI> LIST PHYSICALDISK

It is:

DB Servers → RAC/GI → ASM → Storage Network 
Cells → CPU/Memory → XRMEM/Flash → CellDisk/GridDisk 
Physical Storage → IORM

After these checks we may reach an important conclusion:

The Exadata infrastructure is healthy—but the database is still slow.

In Part 2 — Exadata Performance Investigation, we will correlate:

DB Time | AWR/ASH | Wait Events | Top SQL | exe-Plans 
Smart Scan | Offload | Flash/XRMEM | Cell metrics ..etc

To identify where the time really went.

Oracle References

Cross-check technical details against the Oracle documentation for your exact Database and Exadata System Software release:

• Oracle Exadata Database Machine Maintenance Guide
• Oracle Exadata System Software User’s Guide
• Exadata Metrics and CellDisk I/O Monitoring
• Monitoring Exadata Smart Flash Cache
• Monitoring Exadata RDMA Memory / XRMEM
• Monitoring the Exadata Host Interconnect
• Monitoring Exadata Smart Flash Log
• Administering Exadata I/O Resource Management

Articals in the same Series:

• Part 1: Exadata Health Check
• Part 2: Exadata Performance Investigation
• Part 3: How to Tune Oracle Database to Use Exadata Efficiently

Other Articals

🔖 Next: Part 1/3 – Oracle Exadata Patching (Oracle Homes)

🔖 Next: Part 2/3 – Oracle Exadata Patching (RoCE Switches)

🔖 Next: Part 3/3 – Oracle Exadata Patching (Cell Storage)

🔖 Next: Part 4/5 – Oracle Exadata Patching (KVM Host)

🔖 Next: Part 5/5 – Oracle Exadata Patching (KVM Guest)

Tags: