After reading my earlier blog about log file sync tuning a reader sent an interesting issue, worth blogging about it.
Problem
Excessive wait time for ‘log file sync’ event while wait time for ‘log file parallel write’ is minimal. See statspack top events below
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
-------------------------------------------- ------------ ----------- --------
CPU time 18,435 31.06
PL/SQL lock timer 4,370 12,137 20.45
log file sync 57,939 8,565 14.43 <-----
db file sequential read 1,303,477 7,030 11.85
db file scattered read 505,147 3,992 6.73
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
---------------------------- ------------ ---------- ---------- ------ --------
...
log file sync 57,939 6,571 8,565 148 0.5
...
log file parallel write 215,112 0 1,138 5 1.9
...
Average wait time for ‘log file sync’ event is 148ms and it is 5ms for ‘log file parallel write’ event. There is an oddity here. Wait time for ‘log file sync’ event seems bit excessive compared to ‘log file parallel write’ event. Ratio between number of waits for ‘log file sync’ and ‘log file parallel write’ is approximately 4. In an average case scenario, if each commit resulted in 4 log writes, then I would expect an Average wait time of 20ms for ‘log file sync’ event. Wait time of 148ms? Needs further analysis.
pstack, pmap and more
Database is running in a Solaris 10 server and database version 9.2.0.8. Since this is a Solaris operating environment, we can use proc utilities to debug this further. We used a script to dump pstack and pmap of LGWR in a loop. This wasn’t much help, but we could guess that LGWR was waiting for I/O since there were many lines in the stack with kaio and aio wait calls. Few lines from pstack output printed below.
6924: ora_lgwr_ERPPRD2 ----------------- lwp# 1 / thread# 1 -------------------- ffffffff7ddd4838 semsys (4, 3, ffffffff7fffd5cc, 1, ffffffff7fffd5b8) 00000001005b2694 ksliwat (0, 3c0, 77e8a7790, 1037c0b30, 3, 0) + 274 .. ----------------- lwp# 2 / thread# 2 -------------------- ffffffff7ddd3a34 kaio (6, 0, ffffffff7def7f80, 10, ffffffff7def9c18, ffffffff7f700a00) ----------------- lwp# 3 / thread# 3 -------------------- ffffffff7ddd2f1c lwp_park (0, 0, 0) ffffffff7ddcc748 cond_wait_queue (1038e5a70, 1038e5a80, 0, 0, ffffffff7def9d08, 0) + 28 ffffffff7ddccca8 cond_wait (1038e5a70, 1038e5a80, 0, 0, 1c00, 0) + 10 ffffffff7db03a60 _aio_idle (1038e5a00, 1, 1038e5a80, 0, ffffffff7def9c18, 0) + 28 ffffffff7db034fc _aio_send_sigev (1038e5a00, 0, 104b60, ffffffff7ddd2cc8, ffffffff7db03498, ffffffff7dc08000) + 64
dtruss
I/O statistics did not provide much clarity either. We need to find if LGWR is suffering from a performance issue. To see if LGWR is suffering from any OS related issues, we need to trace system calls from LGWR and Truss utility provides such a facility. Suffering from truss related paranoia, we didn’t want to run truss against LGWR since that can affect performance, more importantly database stability.
Fortunately, dtruss came handy. dtruss is based upon dtrace utility and by design, dtrace is safe in Solaris Sparc platform.
Test setup
As a test setup:
- We connected to the database locally and identified foreground process associated with our session.
- We started dtruss on LGWR and the foreground process concurrently.
- We also turned on sqltrace on this foreground process.
Then, we generated couple of MBs of redo from our session, followed by a commit.
Few lines from dtruss output for foreground process printed below:
... 13934/1: 27767 10 write(0x5, "\n", 0x1) = 1 0 ........(1) 13934/1: 27803 8 semsys(0x0, 0x3, 0x9) = 0 0 .........(2) 13934/1: 27996 376193 semsys(0x4, 0x3, 0xFFFFFFFF7FFFAD0C) = 0 0 .........(3) 13934/1: 28105 55 write(0x5, "WAIT #1: nam='log file sync' ela= 367609 p1=5354 p2=0 p3=0", 0x3A) = 58 0 ...
From the output above:
- Line #1: Foreground process posted LGWR using semsys call.
- Line #2: Then user process went to sleep for 376.1 ms on its own semaphore (waiting for LGWR post aka ‘log file sync’ event).
- Line #3: shows that user process wrote event wait timing in to the sqltrace trace file.
dtruss output of LGWR
Let’s look at dtruss output of LGWR. Few lines from dtruss output for LGWR process printed below:
PID/LWP RELATIVE ELAPSD SYSCALL(args) = return ... 6924/5: 240044 230587 pwrite(0x103, "", 0x100000) = 1048576 0 ..........(1) 6924/5: 240091 39 kaio(0x4, 0x0, 0x1038E9DD0) = 0 0 6924/1: 2447794 232035 kaio(0x2, 0xFFFFFFFF7FFFD400, 0x0) = 1 0 6924/7: 226276 354665 pwrite(0x103, "", 0x6DC00) = 449536 0 6924/7: 226307 26 kaio(0x4, 0x0, 0x1038EB1F0) = 0 0 6924/1: 2447839 123214 kaio(0x2, 0xFFFFFFFF7FFFD400, 0x0) = 1 0 6924/1: 2457909 60 semsys(0x2, 0x3, 0xFFFFFFFF7FFFCEA4) = 0 0 6924/1: 2468590 48 lwp_park(0x1, 0x9, 0xFFFFFFFF7F700200) = 0 0 6924/1: 2468606 12 lwp_park(0x1, 0xB, 0xFFFFFFFF7F700200) = 0 0 6924/1: 2468632 10 kaio(0x2, 0xFFFFFFFFFFFFFFFF, 0x1) = -1 Err#22 6924/9: 249884 226754 pwrite(0x103, "", 0x100000) = 1048576 0 6924/1: 2468669 228315 kaio(0x2, 0xFFFFFFFF7FFFD400, 0x0) = 1 0 6924/9: 249923 25 kaio(0x4, 0x0, 0x1038EB1F0) = 0 0 6924/1: 2468720 124856 kaio(0x2, 0xFFFFFFFF7FFFD400, 0x0) = 1 0 6924/11: 217949 352556 pwrite(0x103, "", 0x7FE00) = 523776 0 6924/11: 217977 23 kaio(0x4, 0x0, 0x1038E9DD0) = 0 0
Above output shows [Line #1] shows that LGWR wrote 1MB of redo using pwrite call and it took approximately 230ms for the write call to complete. Indeed LGWR took 230ms for log file writes. But, ‘log file parallel write’ average is only 5ms. Is average misleading and hiding longer write times?
Searching for ‘log file parallel write’ in google, I hit Jonathan Lewis blog entry . Of course, this is a bug and Jonathan has documented this bug much better. In summary, wait time for ‘log file parallel write’ is not printed correctly.
Back to our issue: Simply put, it is an I/O problem that needs further review. It is unfortunate that ‘log file parallel write’ wait time was confusing the reader. Reader is planning to work with storage layer group to review and improve I/O performance.
Image may be NSFW.
Clik here to view.
Image may be NSFW.Clik here to view.
