[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: softdep performance issue

Softdep makes disk writes more efficient, from this alone you can
presume that less disk writing may occur, e.g. writing the same sector
ten times should only result in one combined disk write.
When you extract a tarball, or worse do a rm -rf, you are in effect
reading and re-writing from a smallish set of disk sectors, especially
in the case of the rm.  Cosequently something that previously had
lots of disk IO to break up the CPU use, now becomes almost entirely
CPU-bound, with the result that other (esp. lower priority) processes
are competing for an exhausted resource, the CPU.

You should find that simply running the processes at a lower priority
(nice rm -rf ...) will make your system usable during these operations.

This is not a bug, because without softdep, your system is much more
likely to be diskIO-bound, with any processes writing to disk taking
*much* longer as they compete for disk writes in the same way that
CPU competes with softdep on.

If you want to prove the above, try:
# mkdir /var/1 ; cd /var/1 ; nice tar xf /tarball.tar &
# mkdir /var/2 ; cd /var/2 ; nice tar xf /tarball.tar &

With softdep on and off and run 'top' in another window.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Dom De Vitto                                       Tel. 07855 805 271
http://www.devitto.com                         mailto:dom_(_at_)_devitto_(_dot_)_com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

-----Original Message-----
From: owner-misc_(_at_)_openbsd_(_dot_)_org [mailto:owner-misc_(_at_)_openbsd_(_dot_)_org] On Behalf Of
Sent: Wednesday, December 24, 2003 12:32 AM
To: misc_(_at_)_openbsd_(_dot_)_org
Subject: softdep performance issue

this is kind of a follow-up to a previous problem that i reported but was
never solved.  posted as 'issues multitasking during disk io', my problem
was (in short):

extracting a source archive would send all other processes on my system to a
grinding halt.. or at least slow their execution to the point of where i
could walk away, get coffee, and come back before things like ps  and df
were done providing output.

in my original mail, i indicated i had tested it with softdep on and off.
this was in error, because i assumed mount -u -o rw was turning off softdep,
when it was actually staying on.  my bad.  this turned out to be the
problem, actually.  so a more accurate description of my problem is:

softdep seems to cause serious multitasking problems on my system.

from what i've read about softdep, i thought it would  merely enhance
performance by updating some things in ram, then writing them in an orderly
fashion to disk.  but something is wrong here.

when i mount my filesystems without softdep, all other processes execute as
normal, and my system behaves very well, despite the fact that rm -rf /(some
large directory with many small files) takes quite a while.  i can even
extract 10 or more such source archives, thoroughly saturating my ide bus,
and all other tasks execute as normal.

however, when softdep is on, extracting even one large source archive or rm
-rf /(some large directory with many small files) is very fast, but all
other processes on my system have trouble executing as normal.  this also
includes running binaries from a ramdisk (which should partially rule out
ide bus 'saturation')

although the affects of this are somewhat less pronounced, i've reproduced a
noticeable performance decrease by enabling softdep on completely different
(and even lesser) systems, including ones with scsi.

as freebsd, which (as far as i know) includes the roughly same softdep code
doesn't exhibit these symptoms, i'm suspicious that there's something
different with the softdep implimentation in openbsd that's causing this.

i'd appreciate some insight into this problem, or mabye a developer to look
into it further


controller$ dmesg
OpenBSD 3.4-current (GENERIC) #60: Sat Nov  1 15:42:29 MST 2003
cpu0: Intel Pentium II ("GenuineIntel" 686-class, 512KB L2 cache) 351 MHz
real mem  = 167358464 (163436K)
avail mem = 148955136 (145464K)
using 2068 buffers containing 8470528 bytes (8272K) of memory mainbus0
(root) bios0 at mainbus0: AT/286+(4b) BIOS, date 05/07/99, BIOS32 rev. 0 @
0xfb0b0 apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
pcibios0 at bios0: rev. 2.1 @ 0xf0000/0xb6e8
pcibios0: PCI IRQ Routing Table rev. 1.0 @ 0xfdf10/128 (6 entries)
pcibios0: PCI Exclusive IRQs: 5 10 11
pcibios0: PCI Interrupt Router at 000:07:0 ("Intel 82371SB PCI-ISA" rev
pcibios0: PCI bus #1 is the last bus
bios0: ROM list: 0xc0000/0x8000
pci0 at mainbus0 bus 0: configuration mode 1 (no bios) pchb0 at pci0 dev 0
function 0 "Intel 82443BX PCI-AGP" rev 0x03 ppb0 at pci0 dev 1 function 0
"Intel 82443BX AGP" rev 0x03
pci1 at ppb0 bus 1
vga1 at pci1 dev 0 function 0 "ATI Mach64 GZ" rev 0x3a wsdisplay0 at vga1:
console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation) pcib0 at pci0 dev 7
function 0 "Intel 82371AB PIIX4 ISA" rev 0x02 pciide0 at pci0 dev 7 function
1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 wired to compatibility,
channel 1 wired to compatibility wd0 at pciide0 channel 0 drive 0: <FUJITSU
MPG3102AT E>
wd0: 16-sector PIO, LBA, 9773MB, 16383 cyl, 16 head, 63 sec, 20015856
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
wd1 at pciide0 channel 1 drive 0: <FUJITSU M1636TAU>
wd1: 16-sector PIO, LBA, 1226MB, 2490 cyl, 16 head, 63 sec, 2511840 sectors
wd1(pciide0:1:0): using PIO mode 4, DMA mode 2 uhci0 at pci0 dev 7 function
2 "Intel 82371AB USB" rev 0x01: irq 5 usb0 at uhci0: USB revision 1.0 uhub0
at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered "Intel 82371AB Power Mgmt" rev
0x02 at pci0 dev 7 function 3 not configured emu0 at pci0 dev 16 function 0
"Creative Labs SoundBlaster Live" rev 0x07:
irq 10
ac97: codec id 0x43525914 (Cirrus Logic CS4297A rev 4)
ac97: codec features headphone, 20 bit DAC, 18 bit ADC, Crystal Semi 3D
audio0 at emu0 "Creative Labs SoundBlaster Live Digital Input" rev 0x07 at
pci0 dev 16 function 1 not configured rl0 at pci0 dev 18 function 0 "Realtek
8139" rev 0x10: irq 5 address
rlphy0 at rl0 phy 0: RTL internal phy
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0 pms0 at pckbc0 (aux
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16 biomask c020 netmask c020
ttymask d022
pctr: 686-class user-level performance counters enabled
mtrr: Pentium Pro MTRR support
dkcsum: wd0 matched BIOS disk 80
dkcsum: wd1 matched BIOS disk 81
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302

# this is the command i can use to most obviously see the performance

# this is with softdep on, on all devices except the / one $ tar zxf src.tmp
& [1] 25909
$ ps       # i can barely type ps over ssh and have it appear...........
# this is where it sticks.  i'm waiting here with a blank line for 30
seconds or more (usually far more)
  PID TT   STAT      TIME COMMAND   # each line here
 5945 p0  Is      0:00.03 -sh (sh)              # takes about 5 seconds
26674 p1  Is      0:00.06 -sh (sh)             # to show up.
21559 p2  Is      0:00.09 -sh (sh)
 5078 p2  D       0:00.40 tar zxf src.tar.gz
 8454 p2  I       0:00.37 gzip -dc

# this is with softdep off completely.
$ tar zxf src.tmp &
[1] 17500
$ ps
# this time, output appears instantly..
 9488 p0  Is      0:00.08 -sh (sh)
27219 p1  Is+     0:10.07 irssi
6045 p2  D       0:00.04 tar zxf src.tar.gz
7055 p2  I       0:00.02 gzip -dc
18940 p2  Is      0:00.06 -sh (sh)

Visit your host, monkey.org