D. John Shakshober
Current ind
u
strv trends ILl\'C mm'Cd from centrali zed computing ofkredlw
uniprocessors and svmmetric m ultiprocessing ( S M P ) S\'S
tems to m ultinode, highh· a\·ailable and sellable s\·stems, called cl ust
ers. The TruCiuster mu lticomputer svstem tor the D igital U N 1 X en\'i
ron ment is the latest cluster p roduct from Digital Eq u i pment Corporation.1
In this paper, \\'e discuss our test and resu lts on at
our-node Al phaServcr8400 5/350 TruCiuster configuration
s u pporting the Oracle Para l lel Server database application. We evalu ate this system u nder the Transaction Processing Pert(m11ance Council's TPC-C be nchmark to prm·ide pcrt
(Jrmance results in the competiti\ e market f(>r database Jpp lications.The TPC:-C
ben
chm�
1rk is a medium-complcxit\', on -line trans:tction Jlro
cessing ( O LTP) ,,·orklo:�d.' ' I t i s based on :111 ord er-enrrv workload, \\'ith diff-eren t transaction ty
pes ranging hum simple transactions to med i u m-complcxitvtra
nsact
ions that have2
to50
times the number of cal ls of a simple transaction! Torun the TPC-C benchmark on a clustered system, the
operating system and the database engine must present
a single
chtabJse
to the benchmark client. Thus the TruClustc:r sy
stem running the Oracle PJra l lcl Server difters greath· �i·om a network-based c luster S\'Stem l)\' two signiliGlnt katures. first, rhe
Digit:tl U N I X distri b uted ra1,. disk ( D R D ) sen·ice en�
1blesthe
distrib u ted Oracle Parallel Senn to access a l l Ll\1' d isk \'O i umcs regardless of their p lwsicll location in the c luster. Second , the Oracle Par,1llel Sen er uses Digita
J'
s d istri b uted lock maiLlger(
D LM ) to S\'llchronize a l l access to sh ared resources (such as i tJ memory cache blocks or disk blocks) across a Tru Clu
ster sysrem .In tuning
the system
u nder test, \\'e used the DRD Jnd the D LMsnv
ices
to balance the database across the TruC i uster m u lticomputer system .The
config
uration inclu
des
a
spe
ci<llized pe ripheral compo nent interconnect ( PC l ) known as the M E M O RY CHANN EL interconnect to greJtlv improve the band\\'id th a nd btenc\' bet\\'een t\\'O or more member nodes.' vVe tuned the S\
'
Stem under test to attain the peak band"·
idr
h of 100 megah\'tcs per second ( M B/s )for hean· internode communication d u ring ch eck poin ting b\· using a dedicated
PC:l
bustor the
ME!'vlORY CHA N N EL i mcrcon ncct. We also tunedthe S\'stcm u nder test to usc
the
1·en· large mcmon· tccllllologv and trade off mcmon· r()rthe
databasecache with mcmon· for DLM locks to impro1-c the
through put. ( For a discussion of this tcc
h
nologv, see the section Pcdormance E1·al uation Mctho
dologv.) vVc measu red the maxim um
throughput,
the 90th percentile response time for each transaction type, and the keying and think times. Finally, we com pared
ourmeasured throughput and pricc/pcd(m11ancc with
competitive vendors like Tandem Computers and
Hewlett-Packard Company.
The rest of the paper
is
organized as f(>ilows. In thenext
section, we provide a svnopsis of the TruCi ustcr technology and i ntroduce the Oracle Parallel Scn·er, an optional Oracle product tlut enables th
e user to useTruCiu
s
tcr tcchnolog1·with the Oracle
relational database management S\'Stcm . Following th;lt, we gin: an m-cn·icll' of th
e TPC-Cb
cnchm
;Jrk. Next, 11-edescribe the system u nder test and our
ped(mnance
ev;lluation methodology. Then we discuss om pcrf<>r mance measurement results and
com
pare
them with competitive vendor results. Finally, 1vc present ourconcluding remJrks and discuss our tlltlll'C work. TruCiuster Clustering Technology
Digital's TruCiuster configuration consists of inter
connected computers ( uniprocessors or SM Ps) and e
xt
ernal d isks connected to one or more shared, smal lcomputer systems interC1ce ( S CS I ) bu
ses
providi ngscn·iccs to clients.'' It presents a single rJII' vo lume
nai1Kspacc to a client with better application av;lilabil
ity than a single system and better scala
bi
lity than an SM !'. A TruCiuster configuration supports highly par a l lcl izcd database managers, such as the Oracle PJralkl,- - - -
SERVER 1 M EMORY CHANNEL ADAPTER
Serl'cr, to pr01·ide i ncrementa l pedormance scaling of at least
80
percent r()r transJction processing applications. The undcrll'i
ng
tcchnologl' to pro1·ide
thisincremental growt
h
i ncludes a PCI- based M EMORY CHAN N E L interconm:ct t()r communicati
onbetii'Cen
cluster
m
emb
ers.'' The M E MO RY CHANI'FLinterconnect provides a 100-MB/s, memory-m<lppcd
con nection between cluster members.' The cluster
members map transfers ti·om the M EM O RY
C H AN N E L interconnect i n to their memory u
si
ng standard mcm
orv :1cccss i nstructions. The use of memory store i nstructions rather than special 1/0 i nstructions prc)l'ides low IJtcncv ( tii'O microseconds) and Jow OITrhcad f()r a transfl:r of am· Jength.-Tbe TruCi ustcr r(>r Digit;ll U N I X product su pports up to eight ( t(w r r(>r con1mcrcial D LM/DRD - based appl ications ) cluster mem bers con nected to a com mon cluster imcrconncn. The computer
svstems
supported within a cluster arc AlphaSe rver systems of varyi ng processor speed Jnd num ber of processors. The member systems run :1pplicarions ( tor example, user applications ), as well JS monitor the state of each member system,
e
ach shored disk, the M EMO RYC HAi'l N E L i nterconnect, and the netlvork. These
cluster mem bers commu nicate o1·er the M El'dO RY
CHAN NEL i n terconnect.'' � A M E MO RY CHANNEL con tigu ration consists of a M ENI O RY CHANN E L adapter i nstalled in J l'C I slot and link cables t o con nect the adapters. I n J connguration \\'ith more than
two members, the M EJVIO RY CHANNEL adapters
are con nected to a 1YI EMORY CHAI': N E L bu b. A
typical TruCi ustcr configuration with a M E MO RY
CHANNEL hu
b is shown in
figure l .Applications can attain high availability by connect i ng two or
m
oremember
svstcms to one or more
SERVER 2 MEMORY CHANNEL ADAPTER SHARED STORAGE LOCAL STORAGE Figure 1
A Truc:l ustcr Contigmation ll'ith M EM O RY C H AN N E L H u b
4X
s h a red S C S I buses
,
thus constructing an A1 ail able S e rver Em·ironmcnt ( A S E ). A shared SCSI bus is requ ir
ed onJvfix
two- mem ber contigurations th�lt donot have a M E M O RY C H A N N E L h u b . A lthough M EMORY CHAN N EL is the onlv supported cl u ster in tcn:onnccr, Ethernet :md
fiber
d istributed d ata imcrraec ( FD D I ) arc supported for connecting clients to cluster members. Dis ks arc connected either lou l k( i .e., nonsh::tred ) to a SCS I bus or to �l sh�1tnl SCS I bus bct\\'een tii'O or mmc member S\'Stcms. A single node i n the c l ust
e
r is used to scn'C the disk to otl1er cluster members. D isks on local buses obviously become unavail<lblc upon f�1 i l u rc ohhc server nod e . The SCSIcontroller s
upported in this contlguration is the l'C : I d i s k adapter, KZPSA.The distinguis
h
ing kature ofthe
TruCiustLTsottware is its support of rhc
MEMORY
C H AN N EL as a cluster i nterconnect, thus pro1·i d i ng indusrn· lcadership pcrt(
mn�mcc ro imraclusrcr com m u tliu tion .'' The Tru C :I ustcr software i ncludes the f(Jl loll'ing components: the D L M , the connection manager, the D RD, and the c luster commu nication service. The D LM facil itates svnchronitation to shared resou rces to�11lmember SI'Stcms i n a cluster b1· means of �1 run-rime lihratY Cooperating processes usc the D LM to S\'n chronize access to a sh�1red resource, a
[) J\1)
dei'ICC, J til e , or a program. The D Uvl sen·ice is primari l1· usedlw the Oracle Parallel Server to coordinate :�cccss to the uche and shared disks that h�ll'e the database insta l led .''
The
connection m�magcr maint<li ns inform:�tion :-rbourthe cluster contlguration �md m<lintains a commutlica tion path betll'een cKb cluster member fix usc by the
D LM . The DL1VIuscs this con figuLltion cbta and other con nection mJnagcr sen·ices to maintain �l disrrib ured lock data
b
ase. The D ll..D allOI\'S t he c:>:pmting ofcl usrcrll·ide ra11· dc,·iccs. This aii()II'S disk- based uscr- ln·cl
:1ppl ications to run ll'ithin the cluster, regardless of ll'herc in the cluster the �JCtu:�l physic:�! storage resides. Therdorc a DRD service al loll's the Oracle Par�1 l lcl
Server paral lel access to stoLJgc media ti·om multiple
cl uster mem hns. The c l u ster comnH1 11icuion sen ic
e
is used to a l loc:1te the lVI EMORY C H .\ N N L I . address space �llld Jll<lp it to the processor main memory. TPC-C BenchmarkThe TPC : - C : bcnchm;lrk depicts the activit\' or· a generic \\·hoks<llr supr1licr compan1·. The h ierarclw
i n the T l'C :-C
busi
ness em·ironmcm is sho11·n i n Figure 2 . The com pam· consists o f a number
of gcographicJI Iv distributed sales d istricts and associ<Jrcd ll'archouscs. fu rther , there a rc 10 districts under each warehouse with c1ch district serving 3 ,000
( 3K)
CLIStomcrs. All the ll'archouses maint<lin �1 stock of 1 0 ,000
i tems sold lw the companv. As the compam· gro\\'s, ne11· ll'�ln:houscs and associated S;Jks districts arc cre ate d . The business actiYitl· consists o t. customcr ca l l s
to l1l acc n c11 orders or request the st;Jtus of n.:isting
orders, pavmcm entries, processing orders r()r dcJi,·etT, and stock-Jc,·cl cx�unination. The orders on an a1·erage arc composed of 1 0 mder li nes ( i . e . , l ine items ). Ninety-nine percen t of all orders C;Jn be met bv a loul 1\'are housc, :llld onlv one percem o f rhun need to be sold Lw a remote 11·arehouse.
The T l1C-C logic:1l d atabase components consist of
n i ne t•Jblcs.' hgurc