software RAID10 performance

[ link: raid10-performance | tags: raid linux | updated: Thu, 27 Jul 2006 12:50:09 ]

A buddy of mine works at IBM, he has a big ass SCSI raid array (DS300) on which he runs RAID5 (which is bad, BTW). He wanted to know what the overhead of running software RAID10 (aka RAID1+0) on this system would be. His numbers are here.

The setup:

I have 8 SATA drives in two RAID0 arrays built over four RAID1 arrays:

    # cat /proc/mdstat  | grep ^md
    md5 : active raid0 md1[0] md2[1]
    md6 : active raid0 md3[0] md4[1]
    md4 : active raid1 sdh3[1] sdf3[0]
    md3 : active raid1 sde3[0] sdg3[1]
    md2 : active raid1 sdd3[0] sdb3[1]
    md1 : active raid1 sda3[0] sdc3[1]

Disks sda through sdd are on the onboard controller. Disks sde and sdf are on a 32/33 PCI controller, and sdg and sdh are on another (on the same 32/33 PCI bus).

The data:

Here are the raw hdparm throughput benchmarks for a couple of the disks:

    # hdparm -tT /dev/sda ; hdparm -tT /dev/sdd
    /dev/sda:
     Timing cached reads:   3432 MB in  2.00 seconds = 1715.40 MB/sec
     Timing buffered disk reads:  182 MB in  3.01 seconds =  60.47 MB/sec

    /dev/sdd:
     Timing cached reads:   3528 MB in  2.00 seconds = 1762.51 MB/sec
     Timing buffered disk reads:  212 MB in  3.00 seconds =  70.56 MB/sec

Here is dstat running in another terminal:

    ----total-cpu-usage---- -disk/total -net/total- ---paging-- ---system--
    usr sys idl wai hiq siq|_read write|_recv _send|__in_ _out_|_int_ _csw_
      0   2  79  18   0   1|  12M    0 |1382B 1254B|   0     0 |1216   225 
      1  12   0  86   0   1|  61M    0 |2447B 2145B|   0     0 |1623  1022 
      0  11   0  89   0   0|  61M   12k|1930B 1770B|   0     0 |1616  1255 
      1  13  14  70   0   2|  49M 1598k| 910B 1174B|   0     0 |1680  1790 
    <snip>
      0   0  93   7   0   0|5116k    0 | 802B  752B|   0     0 |1154   103 
      0  14   0  85   0   1|  70M    0 |1348B 1168B|   0     0 |1682  1151 
      1  14   0  85   0   1| 137M   52k|1896B 1682B|   0     0 |3267  3307 
      0  50  50   0   0   0|   0    16k| 264B  648B|   0     0 |  77    30 

Now here are the numbers for the RAID1 devices:

    # hdparm -tT /dev/md1 ; hdparm -tT /dev/md3
    /dev/md1:
     Timing cached reads:   3556 MB in  2.00 seconds = 1776.49 MB/sec
     Timing buffered disk reads:  182 MB in  3.02 seconds =  60.25 MB/sec

    /dev/md3:
     Timing cached reads:   3556 MB in  2.00 seconds = 1777.38 MB/sec
     Timing buffered disk reads:  216 MB in  3.00 seconds =  71.96 MB/sec

And here is the dstat output ran in another terminal:

    ----total-cpu-usage---- -disk/total -net/total- ---paging-- ---system--
    usr sys idl wai hiq siq|_read write|_recv _send|__in_ _out_|_int_ _csw_
      0  13   4  82   0   1|  57M    0 |1176B 1056B|   0     0 |1579  1396 
      1  12   0  85   0   2|  61M    0 | 954B  836B|   0     0 |1601  1485 
      0  14   1  83   0   2|  64M  142k| 142k 7146B|   0     0 |1881  1767 
    <snip>
      0  12  19  68   0   1|  57M    0 | 712B  752B|   0     0 |1571  1393 
      1  13   0  83   0   3|  73M    0 | 646B  654B|   0     0 |1695  1773 
      1  10   0  85   0   4|  73M  368k| 706B  696B|   0     0 |1718  1784 
      1   8  76  15   0   0|  13M    0 |3226B 3308B|   0     0 |1250   396 

Here are the metrics for the RAID10 devices:

    # hdparm -tT /dev/md5 ; hdparm -tT /dev/md6
    /dev/md5:
     Timing cached reads:   3536 MB in  2.00 seconds = 1766.50 MB/sec
     Timing buffered disk reads:  164 MB in  3.02 seconds =  54.38 MB/sec

    /dev/md6:
     Timing cached reads:   3544 MB in  2.00 seconds = 1770.50 MB/sec
     Timing buffered disk reads:  230 MB in  3.02 seconds =  76.22 MB/sec

And dstat running in another terminal:

    ----total-cpu-usage---- -disk/total -net/total- ---paging-- ---system--
    usr sys idl wai hiq siq|_read write|_recv _send|__in_ _out_|_int_ _csw_
      0  57   1  22   0  20|  50M    0 |1414B 1298B|   0     0 |1925  3262 
      1  57   0  21   0  21|  54M    0 | 646B  654B|   0     0 |1974  3756 
      1  55   0  21   0  23|  59M   52k| 646B  670B|   0     0 |2068  3615 
    <snip>
      0   9  75   9   0   7|  17M  106k|1748B 1650B|   0     0 |1522   311 
      1  56   0  12   0  31|  86M    0 | 646B  654B|   0     0 |3068  1090 
      1  55   0  14   0  30|  77M  183k| 712B  784B|   0     0 |2845  1860 
      0  53  12  14   0  21|  50M   34k|1182B 1348B|   0     0 |2180  1913 

Conclusions:

If you read the results you will discover that the read throughput actually stays pretty much the same through the test regardless of the level of RAID you throw at it. That's probably good because it shows that Linux MD is not adding additional overhead. However, this is not the full picture. What would be more interesting is to do a write test to determine what impact writing to two disks at the same time does to the performance.

TODO:

When I don't care so much about my data, try an hdparm write test and dbench.

Bart's Blog

/raid10-performance

About

I am an embedded Linux software developer and consultant operating under Jukie Networks Inc in Ottawa, Canada.

RSS Feed - Full Content

RSS Feed - Full Content

ipv6 ready

Valid XHTML 1.0 Transitional

Created with Vim

Created with Perl




Tags

abiword adam android apt asm ata audio bacula bash bios board-game boot bug busybox bzr c code conference debian debug desktop devel disk dns dpkg dream email embedded find firefox font fs fun g1 gdb geode git git scm git-find git-graft go google gpg grub hardware html hw inkscape ion3 ipsec ipv6 ipv6summit.ca irc irssi joke kdb kernel klips kqemu kvm laptop lastfm ldap lighttpd linux lua lvm mail make mercurial meta mindblown mmc mouse-free mp3 mpd music mutt neovim net nexusone nfs nvim oclug ogre ols oom OOo openswan opteron osx ottawa perl photography power pxe qemu raid redhat rogers ruby sata sbc scm screen script scsi security sheeva shell skype spam squid ssd ssh suse svn tags talk thinkpad ubuntu uml unix urxvt usb uzbl vi video vim vimgit virt vpn vserver web weechat wifi wind wmii wmiirc-lua work x x86 x86emu xen xen-box-setup xfs xterm zsh

Posts

[ 20240214132102 ]
new git learnings

[ 20240203164450 ]
new vim config

[ 20181212145129 ]
improving find -exec efficiency

[ 20130409092758 ]
automount mmcblk devices

[ 20130316190507 ]
tunnelbroker vs IRC

[ 20110616180255 ]
HDD -> SDD

[ 20110517204617 ]
how to manually create a 6in4 tunnel

[ 20110501202915 ]
Presentation slides published!

[ 20110320190458 ]
What is IPv6, and why should I care?

[ 20110309230106 ]
Ottawa IPv6 Summit 2011

[ 20110309225149 ]
I am now an IPv6 Sage

[ 20101214145558 ]
It's a Holiday Miracle

[ 20101103095130 ]
ipv6 on your desktop in 2 steps

[ 20101020113026 ]
growing a live LVM volume

[ 20100915175649 ]
distributing DNS list through radvd

[ 20100908173402 ]
Canadian ipv6 drought

[ 20100903190623 ]
ipv6 certification

[ 20100723192259 ]
m4a to mp3

[ 20100723134522 ]
git 1.7.2 is out

[ 20100523092001 ]
console=ttyS0 with grub2

[ 20100426162644 ]
How many times is my function used within an executable?

[ 20100412100135 ]
vmlinux on Ubuntu

[ 20100328152951 ]
where your WIND coverage ends

[ 20100326113354 ]
Nexus One live in Ottawa on WIND Mobile

[ 20100311133411 ]
the WIND excitement

[ 20100302160743 ]
sata hotswap pico-HOWTO

[ 20100227093435 ]
serving http content out of a git repo

[ 20100225104049 ]
pimped out zsh prompt

[ 20100221184126 ]
live termcasting of your terminal over telnet

[ 20100215083725 ]
Debian on UBIFS upgrade on SheevaPlug

[ 20100214205532 ]
nexus one

[ 20100130150527 ]
skype on Debian Linux (64bit)

[ 20100125191021 ]
I am so peeved at Rogers

[ 20091205130455 ]
running really nice

[ 20091120225902 ]
notmuch for vim

[ 20091113102221 ]
squid and apt

[ 20091104195344 ]
using WIP branches to save every edit

[ 20091104194146 ]
using git workflows to avoid loosing intermediate changes

[ 20091019183503 ]
reflections on ACM Reflections

[ 20091013174248 ]
bacula rejected Hello command

[ 20091007235046 ]
pimping out git log

[ 20091002094400 ]
virtualization primer

[ 20090922110756 ]
adding an external encrypted volume under Debian

[ 20090905112529 ]
the dreaded process of rooting Rogers Dream

[ 20090830174551 ]
Cyanogen's recipe for Cupcake/Donut-like pastry

[ 20090827100232 ]
simpler android rooting

[ 20090821095427 ]
androids don't like water

[ 20090820215358 ]
prettier function tracing

[ 20090816151854 ]
Baking a cupcake

[ 20090814131559 ]
Hello Android!

[ 20090813100025 ]
sim unlocking a G1

[ 20090731185522 ]
rooting the droid

[ 20090727210307 ]
getting into android

[ 20090714094340 ]
importing an old project into git

[ 20090702113222 ]
Why pick Git?

[ 20090628131255 ]
select loop for X events

[ 20090625181315 ]
portable printf

[ 20090623225800 ]
switching to uzbl

[ 20090622223147 ]
Linux Symposium

[ 20090622214023 ]
bringing git-format-patch to bzr

[ 20090612215638 ]
nfs local caching with fscache and cachefilesd on Lenny

[ 20090610202041 ]
Scott Chacon smacks git around

[ 20090610150039 ]
how would you read a file into an array of lines

[ 20090609215141 ]
libguestfs

[ 20090609000208 ]
tiding up the PATH

[ 20090608232531 ]
only showing relevant messages in mutt by default

[ 20090608010405 ]
git-vim hacking

[ 20090530223801 ]
mark-yank-urls: fix bug allowing shell to interpret the url

[ 20090509212648 ]
wmiirc-lua v0.2.8 release

[ 20090509113125 ]
wmiirc-lua moving to github

[ 20090504101605 ]
two terminals one PWD

[ 20090501172645 ]
splitting files out of a commit

[ 20090409155905 ]
git workflow: git amend

[ 20090401112030 ]
how old are these files in git?

[ 20090330181138 ]
sles 11 on kvm

[ 20090323194942 ]
android true type font

[ 20090322203939 ]
popen with stdin, stdout, and stderr

[ 20090320214228 ]
shrinking URLs

[ 20090304004744 ]
readlater

[ 20081112150409 ]
splitting patches with git

[ 20081011081638 ]
creating busybox symlinks

[ 20081002215121 ]
wmiirc-lua v0.2.5 release

[ 20080916155113 ]
git-svn strangeness

[ 20080915112959 ]
installing git man pages quickly

[ 20080913112345 ]
wmiirc-lua v0.2.4 release

[ 20080825100454 ]
Kernel Walkthroughs - booting

[ 20080813210205 ]
Linux Kernel Booting

[ 20080719211329 ]
printable OLS/2008 schedule

[ 20080715214447 ]
wmiirc-lua updates

[ 20080713194704 ]
Git Screencast

[ 20080713143429 ]
four steps to reproducible Debian installs

[ 20080705150651 ]
USB2.0 enclosure benchmark

[ 20080703230924 ]
Linux Kernel Walkthroughs posted

[ 20080702113602 ]
Introducing the Ottawa Ruby folks to Git

[ 20080628160732 ]
Authenticating Linux against OSX LDAP directory

[ 20080627142123 ]
Canada Day Events 2008

[ 20080613162541 ]
Linux Kernel Walkthroughs

[ 20080510083828 ]
is my usb device connected to a fast port?

[ 20080509111534 ]
show more git info on zsh prompt

[ 20080430104202 ]
git-vim

[ 20080412100337 ]
color your word

[ 20080404105620 ]
show current git branch on zsh prompt (2)

[ 20080303200359 ]
how to track multiple svn branches in git

[ 20080301134220 ]
fixing X for GeodeLX

[ 20080108002540 ]
kvm nfs hang

[ 20080107160836 ]
screen -c relative path bug

[ 20080105132854 ]
WeeChat spell suggestions

[ 20071219221358 ]
show current git branch in zsh

[ 20071217141037 ]
wmiirc-lua kitchen sink repository

[ 20071212100316 ]
protecting sshd from OOM killer

[ 20071204234232 ]
wmiirc-lua v0.2.1 remembers a bit more

[ 20071013205336 ]
wmiirc-lua v0.2 has suspend and raw modes

[ 20070929112345 ]
wmiirc-lua debianization

[ 20070924104140 ]
zsh tip of the day - global aliases

[ 20070915094213 ]
wmiirc-lua v0.1.1

[ 20070913130838 ]
comparing two directories

[ 20070909204125 ]
git-rebase --interactive

[ 20070908115905 ]
zsh tab completion awesomeness

[ 20070902000736 ]
wmiirc in lua

[ 20070831150306 ]
debugging with -dbg libraries

[ 20070831142646 ]
svn status like output in git

[ 20070829141847 ]
Git Cheat Sheet

[ 20070821142038 ]
switching to abiword

[ 20070811105746 ]
forwarding ssh and X through screen

[ 20070807112531 ]
git-svnup

[ 20070724082355 ]
reducing power consumption

[ 20070722123734 ]
Makefile template

[ 20070722002649 ]
less, colourful

[ 20070719162359 ]
irssi docs

[ 20070716114553 ]
qemu eats up /dev/shm

[ 20070710214512 ]
git-clean in svn land

[ 20070705113139 ]
ipw2200 not working

[ 20070627191916 ]
git slides updated

[ 20070618220649 ]
unpopular debian packages on my system

[ 20070611125852 ]
git-svn with multiple branches

[ 20070517085321 ]
Linus on Git at Google

[ 20070510134551 ]
vim modelines insecure

[ 20070504205042 ]
bios disassembler

[ 20070504124124 ]
dd hex arguments

[ 20070503013555 ]
urxvt mouseless url yanking

[ 20070502211941 ]
gitdiff.vba v2

[ 20070419234350 ]
india

[ 20070418155857 ]
zsh fun

[ 20070418143632 ]
pipe to pastey.net

[ 20070418094151 ]
vimgrep alias

[ 20070416202545 ]
mouse-free

[ 20070406141850 ]
ATA messages via SCSI layer

[ 20070330221019 ]
GITDiff vim plugin

[ 20070329011735 ]
git presentation for OCLUG

[ 20070328123631 ]
fixing vim's [[ and ]] for bad code

[ 20070316092236 ]
pxeboot and nfsroot with debian

[ 20070312134706 ]
etc snapshots with git

[ 20070307004041 ]
remote power switch

[ 20070222215355 ]
klips-less openswan git tree

[ 20070221041316 ]
git caching for v1.5.x

[ 20070218002214 ]
klips loses zlib

[ 20070209172606 ]
vim and linux CodingStyle

[ 20070207205427 ]
my kqemu install

[ 20070204100100 ]
leaner meaner openswan

[ 20070115111917 ]
wmii+ruby xlock action

[ 20070112131252 ]
cloning xterms in wmii+ruby

[ 20070102010551 ]
wmii+ruby talk for OCLUG

[ 20061228220641 ]
dump and restore

[ 20061218100219 ]
C style

[ 20061101002027 ]
fetching all git branches from remote

[ 20061028111607 ]
local caching for git repos

[ 20061020145437 ]
automatic version creation with git

[ 20061018213306 ]
wmii w/ ruby wmiirc

[ 20061018201907 ]
small fonts

[ 20061007151802 ]
google-codesearch from vim

[ 20060928020813 ]
shell commands

[ 20060920093957 ]
letting screen apps use the ssh-agent

[ 20060908223613 ]
mpdscribble stream support

[ 20060907125149 ]
glGo on ubuntu/dapper amd64

[ 20060906163240 ]
lbdb and mutt

[ 20060902145643 ]
vim tutorial

[ 20060902135722 ]
fixing your terminal

[ 20060828124713 ]
apt-get pdiffs

[ 20060824224842 ]
256 colour xterms

[ 20060824152658 ]
dynamic IPcomp

[ 20060824145428 ]
inkscape++

[ 20060818150516 ]
tags/cscope for system headers

[ 20060805131557 ]
opteron 170, part 4

[ 20060805101941 ]
opteron 170, part 3

[ 20060803233234 ]
opteron 170, part 2

[ 20060802210126 ]
opteron 170

[ 20060729144129 ]
OLS keysigning / 2006

[ 20060728105500 ]
git-find findings

[ 20060727162941 ]
starting on git-find

[ 20060727113632 ]
git-graft and git-find brainstorm

[ 20060726224531 ]
pretty function tracing

[ 20060713174723 ]
uml and multiple network segments

[ 20060707182236 ]
lastfm artist and title to clipboard

[ 20060706162256 ]
reverting a git changeset

[ 20060629212003 ]
user #3

[ 20060628083456 ]
firefox crashes with form input

[ 20060621151402 ]
git vs hg

[ 20060612222204 ]
ldap account management

[ 20060612194523 ]
stupid ldap

[ 20060608092157 ]
rpm hell is right

[ 20060605095726 ]
OpenSSH VPNs

[ 20060604114317 ]
Lenovo lost a customer

[ 20060601234010 ]
generating html colourized sourcecode

[ 20060601211716 ]
ion3 greatness and acting on X selections

[ 20060526085644 ]
software RAID10 performance

[ 20060525234148 ]
learning to love git

[ 20060524121638 ]
recent vim7 articles

[ 20060516095748 ]
bootstrapping debian on my sbc

[ 20060428145140 ]
entropy injection

[ 20060423140628 ]
Adam

[ 20060414202507 ]
converting mp3s to CD

[ 20060413232836 ]
secure apt-get

[ 20060412194423 ]
xen domain0 on debian

[ 20060410220525 ]
LVM2 on RAID1 mirror

[ 20060410102824 ]
building a RHEL4 kernel w/ kdb support

[ 20060407230939 ]
xen on debian

[ 20060407230818 ]
Upgraded look

[ 20060330131334 ]
Flattered by a copy

[ 20060328165153 ]
vim7 from source

[ 20060308123539 ]
Perl, Catalyst, CPAN, and Debian

[ 20060308123302 ]
last.fm

[ 20060128124841 ]
carcassonne and zombies

[ 20060120135931 ]
Election Humour

[ 20050925130002 ]
ldap on debian

[ 20050528190034 ]
Error while mapping shared library sections

[ 20050124130158 ]
IRC over email gateway

[ 20050110225522 ]
brute force attacks sshd?

[ 20050108095026 ]
LDAP authentication (part 1)

[ 20041124130146 ]
sawfish workspace themes

[ 20041113082651 ]
Mini-DV to divx using mencoder

[ 20041004084525 ]
notes on vserver

[ 20040922104334 ]
fast kernel logging

[ 20040803104122 ]
Digital Rebel for sale... GONE

[ 20040714202912 ]
OpenOffice resources

[ 20040603175746 ]
photo editing

[ 20040601082817 ]
WRT54G

[ 20040503205227 ]
Digital Rebel

[ 20040420200136 ]
Open Office templates

[ 20040326082602 ]
bash vi editing mode

[ 20040315204142 ]
debian install CDs

[ 20040312155542 ]
change of jobs

[ 20040308091554 ]
spamassasin extras

[ 20040305163216 ]
cool debian tools

[ 20040305155708 ]
first post!





Bart Trojanowski
http://www.jukie.net/~bart
bart@jukie.net