Performance Evaluation of Transaction Processing Systems By Walter H. Kohler, Yun-Ping Hsu, Thomas K. Rogers, Wael H. Bahaa-El-Din Abstract Transaction processing Performance and price systems are complex in /performance are important nature and are usually attributes to consider when characterized by a large evaluating a transaction number of interactive processing system. Two terminals and users, major approaches to a large volume of on- performance evaluation line data and storage are measurement and devices, and a high modeling. TPC Benchmark volume of concurrent and A is an industry standard shared database accesses. benchmark for measuring Transaction processing a transaction processing systems require layers of system's performance software components and and price/performance. hardware devices to work Digital has implemented in concert. Performance TPC Benchmark A in a and price/performance are distributed transaction two important attributes processing environment. for customers to consider Benchmark measurements were when selecting transaction performed on the VAX 9000 processing systems. Model 210 and the VAX 4000 Performance is important Model 300 systems. Further, because transaction a comprehensive analytical processing systems are model was developed and frequently used to operate customized to model the the customer's business performance behavior of TPC or handle mission-critical Benchmark A on Digital's tasks. Therefore, a certain transaction processing level of throughput and platforms. This model was response time guarantee are validated using measurement required from the systems results and has proven to during normal operation. be an accurate performance Price/performance is prediction tool. the total system and maintenance cost in Introduction dollars, normalized by the performance metric. Digital Technical Journal Vol. 3 No. 1 Winter 1991 1 Performance Evaluation of Transaction Processing Systems The performance of a Modeling uses simulation transaction processing or analytical modeling system is often measured techniques. Compared to by its throughput in the measurement approach, transactions per second modeling makes it easier (TPS) that satisfies a to produce results and response time constraint. requires less computing For example, 90 percent resources. Performance of the transactions must models are also flexible. have a response time that Models can be used to is less than 2 seconds. answer "what-if" types This throughput, qualified of questions and to provide by the associated response insights into the complex time constraint, is called performance behavior of the maximum qualified transaction processing throughput (MQTh). In a systems, which is difficult transaction processing (if not impossible) to environment, the most observe in the measurement meaningful response time environment. Performance definition is the end- models are widely used in to-end response time, research and engineering i.e., the response time communities to provide observed by a user at a valuable analysis of terminal. The end-to-end design alternatives, response time represents architecture evaluation, the time required by all and capacity planning. components that compose Simplifying assumptions the transaction processing are usually made in system. the modeling approach. The two major approaches Therefore, performance used for evaluating models require validation, transaction processing through detailed simulation system performance are or measurement, before measurement and modeling. predictions from the models The measurement approach is are accepted. the most realistic way of This paper presents evaluating the performance Digital's benchmark of a system. Performance measurement and modeling measurement results from approaches to transaction standard benchmarks have processing system been the most accepted form performance evaluation. The of performance assessment paper includes an overview of transaction processing of the current industry systems. However, due standard transaction to the complexity of processing benchmark, the transaction processing TPC Benchmark A, and a systems, such measurements description of Digital's are usually very expensive, implementation of the very time-consuming, and benchmark, including the difficult to perform. distinguishing features of the implementation and the benchmark methodology. The 2 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Performance Evaluation of Transaction Processing Systems The benchmark can be run in either a local area network (LAN) or a wide area network (WAN) configuration. The related performance measurement throughput metrics are results that were achieved tpsA-Local and tpsA-Wide, by using the TPC Benchmark respectively. The benchmark A are also presented. specification defines Finally, a multilevel the general application analytical model of the requirements, database performance behavior of design and scaling rules, transaction processing testing and pricing systems with response time guidelines, full disclosure constraints is presented report requirements, and and validated against an audit checklist.[1] measurement results. The following sections provide an overview of the TPC Benchmark a-an Overview benchmark. The TPC Benchmark A Application Environment simulates a simple banking The TPC Benchmark A environment and exercises workload is patterned key components of the after a simplified banking system under test (SUT) application. In this model, by using a simple, update- the bank contains one intensive transaction type. or more branches. Each The benchmark is intended branch has 10 tellers and to simulate a class of 100,000 customer accounts. transaction processing A transaction occurs when a application environments, teller enters a deposit not the entire range of or a withdrawal for a transaction processing customer against an account environments. Nevertheless, at a branch location. Each the single transaction teller enters transactions type specified by the at an average rate of one TPC Benchmark A standard every 10 seconds. Figure 1 provides a simple and illustrates this simplified repeatable unit of work. banking environment. Digital Technical Journal Vol. 3 No. 1 Winter 1991 3 Performance Evaluation of Transaction Processing Systems Transaction Logic an account, updates the The transaction logic current cash position of of the TPC Benchmark A the teller and branch, workload can be described and makes an entry of the in terms of the bank transaction in a history environment shown in Figure file. The pseudocode shown 1. A teller deposits in in Figure 2 represents the or withdraws money from transaction. Terminal Communication o The tested system must For each transaction, the preserve the effects of originating terminal is committed transactions required to transmit data and ensure database to, and receive data from, consistency after the system under test. The recovering from data sent to the system - The failure of under test must consist of a single durable at least 100 alphanumeric medium that contains data bytes, organized as database or recovery at least four distinct log data fields: Account_ID, Teller_ - The crash and reboot ID, Branch_ID, and Delta. of the system The Branch_ID identifies - The loss of all or the branch where the teller part of memory is located. The Delta is the amount to be credited o Eighty-five percent of to, or debited from, the the accounts processed specified account. The data by a teller must belong received from the system to the home branch under test consists of (the one to which the at least 200 data bytes, teller belongs). Fifteen organized as the above percent of the accounts four input fields and processed by a teller the Account_Balance that must be owned by a results from the successful remote branch (one to commit operation of the which the teller does transaction. not belong). Accounts Implementation Constraints must be uniformly distributed and randomly The TPC Benchmark A imposes selected. several conditions on the Database Design test environment. o The transaction The database consists processing system must of four individual files support atomicity, /tables: Branch, Teller, consistency, isolation, Account, and History, and durability (ACID) as defined in Table 1. properties during the The overall size of the test. database is determined by the throughput capacity of the system. Ten tellers, each entering transactions at an average rate of 4 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Performance Evaluation of Transaction Processing Systems store the history records one transaction every 10 generated during 90 eight- seconds, generate what is hour days of operation at defined as a one-TPS load. the published system TPS Therefore, each teller capacity. For a system that contributes one-tenth (1 has a processing capacity /10) TPS. The history area of x TPS, the database is must be large enough to sized as shown in Table 2. Table 1 Database Entities ___________________________________________________________________ Fields Record__Bytes_Required_______Description___________________________ Branch 100 Branch_ID Identifies the branch across the range of branches Branch_ Contains the branch's current cash Balance balance Teller 100 Teller_ID Identifies the teller across the range of tellers Branch_ID Identifies the branch where the teller is located Teller_ Contains the teller's current cash Balance balance Account 100 Account_ID Identifies the customer account uniquely for the entire database Branch_ID Identifies the branch where the account is held Account_ Contains the account's current cash Balance balance History 50 Account_ID Identifies the account updated by the transaction Teller_ID Identifies the teller involved in the transaction Branch_ID Identifies the branch associated with the teller Amount Contains the amount of credit or debit (delta) specified by the transaction. Digital Technical Journal Vol. 3 No. 1 Winter 1991 5 Performance Evaluation of Transaction Processing Systems Table 1 (Cont.) Database Entities ___________________________________________________________________ Fields Record__Bytes_Required_______Description___________________________ Time_Stamp Contains the date and time taken between the BEGIN TRANSACTION and _____________________________COMMIT_TRANSACTION_statements_________ Benchmark A uses two basic Table 2 metrics: Database Sizing o Transactions per second ____________________________ (TPS)-throughput in TPS, Record subject to a response Number_of_Records_____Type__ time constraint, i.e., 1 x x Branch the MQTh, is measured records while the system is in a sustainable steady-state 10 xx x Teller condition. records o Price per TPS (K$/TPS)- 100,000 x x Account the purchase price and records five-year maintenance costs associated with 2,592,000 x x History one TPS. records Transactions per Second. _______________________________ To guarantee that the For example, to process tested system provides 20 TPS, a system must use fast response to on-line a database that includes users, the TPC Benchmark A 20 branch records, 200 imposes a specific response teller records, and time constraint on the 2,000,000 account records. benchmark. Ninety percent Because each teller uses of all transactions must a terminal, the price of have a response time of the system must include less than two seconds. The 200 terminals. A test that TPC Benchmark A standard results in a higher TPS defines transaction rate is invalid unless the response time as the size of the database and time interval between the number of terminals are the transmission from the increased proportionately. terminal of the first byte of the input message to the Benchmark Metrics system under test to the arrival at the terminal of the last byte of the output message from the system under test. 6 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Performance Evaluation of Transaction Processing Systems The reported TPS is the o Price of additional total number of committed products required transactions that both for the operation, started and completed administration, or during an interval of maintenance of the steady-state performance, priced systems. divided by the elapsed o Price of products time of the interval. The required for application steady-state measurement development. interval must be at least 15 minutes, and 90 percent All hardware and software of the transactions must used in the tested have a response time of configuration must be less than 2 seconds. announced and generally Price per TPS. The K$ available to customers. /TPS price/performance metric measures the total TPC Benchmark a Implementation system price in thousands Digital's implementation of dollars, normalized of the TPC Benchmark A by the TPS rating of the goes beyond the minimum system. The priced system requirements of the TPC includes all the components Benchmark A standard and that a customer requires uses Digital's distributed to achieve the reported approach to transaction performance level and processing.[2] For example, is defined by the TPC Digital's TPC Benchmark A Benchmark A standard as implementation includes the forms management and o Price of the system transaction processing under test, including monitor software that all hardware, software, are required in most real and maintenance for five transaction processing years. environments but are not o Price of the terminals required by the benchmark. and network components, The following sections and their maintenance provide an overview of for five years. Digital's approach and implementation. o Price of on-line storage Transaction Processing for 90 days of history Software Environment records at the published TPS rate, which amounts The three basic functions to 2,592,000 records per of a general-purpose TPS. A storage medium is transaction processing considered to be on-line system are the user if any record can be interface (forms accessed randomly within processing), applications one second. management, and database management. Digital has developed a distributed transaction architecture (DECdta) to define how Digital Technical Journal Vol. 3 No. 1 Winter 1991 7 Performance Evaluation of Transaction Processing Systems the major functions are forms management to be partitioned and supported performed at a remote by components that fit location, whereas the together to form a complete application is processed transaction processing at a central location. system. Table 3 shows the The Digital transaction software components in a processing software typical Digital transaction components are separable processing environment. because their clearly defined interfaces can Table 3 be layered transparently Transaction Processing onto a network. How Software Components these components may be ____________________________ partitioned in the Digital Component________Example____ distributed transaction Operating VMS processing environment is system illustrated in Figure 3. Communications LAT, DECnet Database VAX Rdb /VMS TP monitor VAX ACMS, DECintact Forms DECforms ___Application______COBOL______ Distributed Transaction Processing Approach Digital transaction processing systems can be distributed by placing one or more of the basic system functions (i.e., user interface, application manager, database manager) on separate computers. In the simplest form of a distributed transaction processing system, the user interface component runs on a front-end processor, and the application and database components run on a back-end processor. The configuration allows terminal and 8 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Performance Evaluation of Transaction Processing Systems TPC Benchmark A Test initiate transactions and Environment communicate with the front- end processors. Front-end The Digital TPC Benchmark processors communicate with A tests are implemented in a back-end processor using a distributed transaction the DECnet protocol. processing environment The test system is the using the transaction configuration of components processing software used in the lab to measure components shown in Figure the performance of the 3. The user interface target system. The test component runs on one or system uses RTEs, rather more front-end processors, than user terminals, to whereas the application generate the workload and and database components measure response time. run on one or more (Note: In previously back-end processors. published reports, based Transactions are entered on Digital's DebitCredit from teller terminals, benchmark, the RTE emulated which communicate with front-end processors. the front-end processors. In the TPC Benchmark A The front-end processors standard, the RTE emulates then communicate with the only the user terminals.) back-end processors to The RTE component invoke the application servers and perform o Emulates the behavior of database operations. terminal users according The communications can to the benchmark take place over either specification (e.g., a local area or a wide think time, transaction area network. However, parameters) to simplify testing, the o Emulates terminal TPC Benchmark A standard devices (e.g., allows sponsors to use conversion and remote terminal emulators multiplexing into the (RTEs) rather than real local area transport terminals. Therefore, [LAT] protocol used by the TPC Benchmark A tests the DECserver terminal base performance and price servers) /performance results on o Records transaction two distinctly configured messages and response systems, the target system times (e.g., the and the test system. starting and ending The target system is the times of individual configuration of hardware transactions from and software components each emulated terminal that customers can use device) to perform transaction processing. With the Digital distributed transaction processing approach, user terminals Digital Technical Journal Vol. 3 No. 1 Winter 1991 9 Performance Evaluation of Transaction Processing Systems Figure 4 depicts the test system configuration in the LAN environment with one back-end processor, multiple front-end processors, and multiple remote terminal emulators. 10 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Performance Evaluation of Transaction Processing Systems TPC Benchmark A Results CommunicationsDECnet-VMS 1 Phase IV We now present the results of two TPC Benchmark A TP monitor VAX ACMS V3.1 1 tests based on audited benchmark experiments Dictionary VAX CDD/Plus 1 performed on the VAX 9000 V4.1 Model 210 and the VAX 4000 Model 300 systems. Application VAX COBOL 1 [3,4] These two systems V4.2 are representative of Digital's large and small Database VAX Rdb/VMS 1 transaction processing system V4.0 platforms. The benchmark was implemented using Forms DECforms V1.2 1 the VAX ACMS transaction ___management__________________ processing monitor, the VAX Rdb/VMS relational database management system, and the Table 5 DECforms forms management system on the VMS operating VAX 4000 Model 300 Back-end system. Tables 4 and 5 System Configuration show the back-end system ____________________________ configurations for the VAX Component______Product______ Quantity 9000 Model 210 and the VAX Processor VAX 4000 1 4000 Model 300 systems, Model 300 respectively. Table 6 shows the system configuration of Memory 64 MB the front-end systems. Tape drive TK70 1 Table 4 VAX 9000 Model 210 Back-end Disk DSSI 3 System Configuration controller ____________________________ Component_____Product_______ QuantiDisks RF31 18 Processor VAX 9000 1 Operating VMS 5.4 1 Model 210 system Memory 256 MBCommunications DECnet-VMS 1 Tape drive TA81 1 Phase IV Disk KDM70 2 TP monitor VAX ACMS 1 controller V3.1 Disks RA92 16 Dictionary VAX CDD/Plus 1 V4.1 Operating VMS 5.4 1 system Digital Technical Journal Vol. 3 No. 1 Winter 1991 11 Performance Evaluation of Transaction Processing Systems Table 5 (Cont.) VAX 4000 Model 300 Back-end System Configuration ____________________________ Component______Product______ Quantity Application VAX COBOL 1 V4.2 Database VAX Rdb/VMS 1 system V4.0 Forms DECforms 1 ___management_____V1.2_________ 12 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Performance Evaluation of Transaction Processing Systems Table 6 Front-end Run-time System Configuration ___________________________________________________________________ Component___________Product____________________Quantity____________ Processor VAXserver 3100 Model 10 10 for VAX 9000 back-end 3 for VAX 4000 back-end Memory 16 MB for VAX 9000 back-end 12 MB for VAX 4000 back-end Disks RZ23 (104 MB) 16 Operating system VMS 5.3 1 for VAX 9000 back-end VMS 5.4 1 for VAX 4000 back-end Communications DECnet-VMS Phase IV 1 TP monitor VAX ACMS V3.1 1 Forms_management____DECforms_V1.2______________1___________________ Table 7 VAX 9000 Model 210 Maximum Qualified Throughput ___________________________________________________________________ ___________Response_Time_(seconds)___________ TPS (tpsA- System________________Local)__________Average_____90_percent__Maximum VAX 9000 Model 210 69.4 1.20 1.74 5.82 VAX_4000_Model_300____21.6____________1.39________1.99________4.81_ Measurement Results Both configurations have The maximum qualified sufficient main memory throughput and response and disk drives such time results for the TPC that the processors are Benchmark A are summarized effectively utilized with in Table 7 for the VAX no other bottleneck. 9000 Model 210 and the VAX Both systems achieved 4000 Model 300 systems. well over 90 percent CPU Digital Technical Journal Vol. 3 No. 1 Winter 1991 13 Performance Evaluation of Transaction Processing Systems utilization at the maximum o Response Time Frequency qualified throughput Distribution. Figure under the response time 6 is a graphical constraint. In addition to representation of the the throughput and response transaction response time, the TPC Benchmark A time distribution. specification requires that The average, 90th several other data points percentile, and maximum and graphs be reported. transaction response We demonstrate these data times are also marked on and graphs by using the the graph. VAX 9000 Model 210 TPC o Transactions per Benchmark A results. Second over Time. o Response Time in The results shown in Relationship to TPS. Figure 7 demonstrate Figure 5 shows the the sustainable maximum 90th percentile and qualified throughput. average response The one-minute running times at 100 percent average transaction and approximately throughputs during 80 percent and 50 the warm-up and data percent of the maximum collection periods qualified throughput. of the experiment are The mean transaction plotted on the graph. response time still This graph shows that grows linearly with the the throughput was transaction rate up to steady during the period the 70 TPS level, but of data collection. the 90th percentile o Average Response response time curve has Time over Time. The started to rise quickly results shown in Figure due to the high CPU 8 demonstrate the utilization and random sustainable average arrival of transactions. response time in the experiment. The one-minute running average transaction response times during the warm-up and data collection periods of the experiment are plotted on the graph. This graph shows that the mean response time was steady during the period of data collection. 14 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Performance Evaluation of Transaction Processing Systems the basic construction Comprehensive Analytical Model of the model and the customization made to Modeling techniques can model the execution of TPC be used as a supplement Benchmark A on Digital's or an alternative to the transaction processing measurement approach. systems. The model can also The performance behavior be used to study different of complex transaction transaction processing processing systems can workloads in addition to be characterized by a set the TPC Benchmark A. of parameters, a set of Response Time Components performance metrics, and the relationships among The main metric used in them. These parameters the model is the maximum can be used to describe qualified throughput the different resources under a response time available in the system, constraint. The response the database operations time constraint is in of transactions, and the form of "x percent of the workload that the transaction response times transaction processing are less than y seconds." system undergoes. To To evaluate throughput completely represent under such response such a system, the size time constraint, of the parameter set the distribution of would be too huge to transaction response manage. An analytical times is determined by model simplifies, through first decomposing the abstraction, the complex transaction response time behavior of a system into nonoverlapping and into a manageable set of independent components. parameters and policies. The distribution of Such a model, after proper each component is then validation, can be a evaluated. Finally, powerful tool for many the overall transaction types of analysis, as well response time distribution as a performance prediction is derived from the tool. Results can be mathematical convolution obtained quickly for any of the component response combination of parameters. time distributions. A comprehensive analytical The logical flow of a model of the performance transaction in a front-end behavior of transaction and back-end distributed processing systems transaction processing with a response time system that is used to constraint was developed implement TPC Benchmark and validated against A is depicted in Figure measurement results. This 9. The response time of model is hierarchical and a transaction consists of flexible for extension. The three basic components: following sections describe front-end processing, Digital Technical Journal Vol. 3 No. 1 Winter 1991 15 Performance Evaluation of Transaction Processing Systems o Back-end processing includes the execution of application, database access, concurrency control, and transaction commit processing. The back-end processing usually involves a high back-end processing, and degree of concurrency communication delays. and many disk I/O activities. o Front-end processing o Communication delays usually includes primarily include the terminal I/O processing, communications between forms/presentation the user terminal and services, and the front-end node, communication with the and the front-end and back-end systems. In the back-end interactions. benchmark experiments, no disk I/O activity (Note: These response time was involved during the components do not overlap front-end processing. with each other.) Within the back-end system, The model is configured in the transaction response a two-level hierarchy, time is further decomposed a high level and a into two additional detailed level. The use components, CPU delays of a hierarchy allows and non-CPU, nonoverlapping a complex and detailed delays. CPU delays include model that considers many both the CPU service and components and involves the CPU waiting times of many parameters to be transactions. Non-CPU, constructed easily. nonoverlapping delays Because of the hierarchical include: approach, the model also o Logging delays, which provides flexibility include the time for for modifications and transaction log writes extensions, and validation and commit protocol of separate submodels. delays The high-level model o Database I/O delays, assumes the decomposition which include both of transaction response waiting and service times, as described in the times for accessing Response Time Components storage devices section, and models the o Other delays, which behavior of the transaction include delays that processing system by an result from concurrency open queuing system, as control (e.g., waiting shown in Figure 10. The for locks) and waiting queuing system consists of for messages servers and delay centers, which are connected in a Two-level Approach 16 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Performance Evaluation of Transaction Processing Systems o No intratransaction queuing network with the parallelism exists following assumptions: within individual transaction execution. o The front-end processing o No mutual dependency does not involve any exists between disk I/O operation, and transaction response the load on the front- time components. end systems is equally o Transaction arrivals to balanced. the processors have a o The back-end is Poisson distribution. a shared-memory multiprocessor system These assumptions with symmetrical loads correspond to Digital's on all processors (or TPC Benchmark A it can be simply a testing methodology and uniprocessor). implementation. The front-end CPU is distribution. The major modeled as an M/M/1 queuing input parameters for this center, and the back-end high-level model are the CPU is modeled as an M/M o Number of front-end /m queuing center. The systems and the front- transactions' CPU times on end CPU service time per the front-end and back-end transaction systems are assumed to be exponentially distributed o Number of CPUs in the (coefficient of variation back-end system and the equal to 1) due to the back-end CPU service single type of transaction time per transaction in the benchmark. (Note: o Sum of the back-end An approximation of M/G/m database I/O response can be used to consider a time, journaling I/O coefficient of variation response time, and other than 1 for the other delay times (i.e., back-end transaction CPU the mean for the LOD service time, especially delay center's 2-Erlang in the multiprocessor distribution) case when the bus is o Response time constraint highly utilized.) (in the form of x Database I/O, logging percentile less than I/O, and other delays are y seconds) modeled as delay centers, with appropriate delay The main result from the distributions. For the high-level model is the model of the TPC Benchmark MQTh. This high-level A workload, the database model presents a global I/O, journaling I/O, and picture of the performance other communication and behavior and manifests the synchronization delays are relationship between the combined into one delay most important parameters center, called the LOD of the transaction delay center, which is processing system and MQTh. represented by a 2-Erlang Digital Technical Journal Vol. 3 No. 1 Winter 1991 17 Performance Evaluation of Transaction Processing Systems Some of the input parameters in the high- level model are dynamic. use spin-locks as the The CPU service time of mechanism for processor- a transaction may vary level synchronization, with the throughput or and the processor spins number of processors, (i.e., busy-waits) in and the database I/O or the case of a conflict. other delays may also In the model, the depend on the throughput. A busy-wait overhead is good example of a dynamic considered to be part model is a tightly coupled of the transaction multiprocessor system, with code path, and such one bus interconnecting the processors and with contention elongates the a shared common memory transaction CPU service (e.g., a VAX 6000 Model time. 440 system). Such a Four detailed-level system would run a single submodels are used to copy of the symmetrical account for the dynamic multiprocessing operating behavior of these system (e.g., the VMS parameters: CPU-cache- system). The average bus-memory, busy-wait, I/O CPU service time of group, and LOD. transactions is affected by both hardware and software The CPU-cache-bus-memory factors, such as submodel consists of many a. Hardware contention that low-level parameters results from conflicting associated with the accesses to the shared workload, processor, cache, bus and main memory and bus, and memory components that causes processor of multiprocessor systems. speed degradation and It models these components longer CPU service time. by using a mixed queuing network model that consists b. Processor of both open and closed synchronization overhead chains, as shown in Figure that results from 11. The most important the serialization of output from this submodel accesses to shared is the average number data structures. Many of CPU clock cycles per operating systems instruction. The busy-wait submodel analysis to derive busy- models the spin-lock wait time. The I/O grouping contention that is submodel models the group associated with the two commit and group write major VMS spin-locks, mechanisms of the VAX Rdb called SCHED and IOLOCK8. /VMS relational database This submodel divides the management system. This state of a processor into submodel affects the path several nonoverlapping length of transaction states and uses probability because of the amortization of disk I/O processing 18 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Performance Evaluation of Transaction Processing Systems path length, the busy- among grouped transactions. wait overhead, and the CPU The LOD submodel considers utilization. the disk I/O times and the By applying the initialized lock contention of certain parameters to the critical resources in the submodels, the values VAX Rdb/VMS system. of these parameters are Integrating the Two Levels refined and input to the of the Model high-level model. The output parameters from the The two levels of the model high-level model are then are integrated by using fed back to the detailed- an iterative procedure level submodels, and this outlined in Figure 12. It iterative process continues starts at the detailed- until the MQTh converges. level submodels, with In most cases, convergence initial values for the is reached within a few MQTh, the transaction iterations. Model Predictions end-to-end model to the The back-end portion of TPC Benchmark A on two VAX the model was validated platforms, the VAX 9000 against measurement Model 210 and the VAX 4000 results from numerous Model 300 systems, and DebitCredit benchmarks then compare the results. (Digital's precursor of The benchmark environment the TPC Benchmark A) on and implementation are many VAX computers with described in the TPC the VMS operating system, Benchmark A Implementation running VAX ACMS and VAX section of this paper. Rdb/VMS software. [5] Because both the VAX With sufficient detailed 9000 Model 210 and parameters available (such the VAX 4000 Model 300 as transaction instruction systems are uniprocessor count, instruction cycle systems, there is no other time, bus/memory access processor contending for time, cache hit ratio), the processor-memory the model correctly interconnect and memory estimated the MQTh and subsystems. Such contention many intermediate results effects can therefore be for several multiprocessor ignored when modeling a VAX systems. The model was uniprocessor system. The then extended to include transaction processing the front-end systems. In performance prediction this section, we discuss for the VAX 9000 Model applying this complete 210 system is a successful Digital Technical Journal Vol. 3 No. 1 Winter 1991 19 Performance Evaluation of Transaction Processing Systems example of the application Additional predictions of our analytical model. were made later, when an We needed an accurate early prototype version estimate of TPC Benchmark of the VAX 9000 Model 210 A performance on the VAX system was available for 9000 Model 210 system testing. A variant of the before a VAX 9000 system DebitCredit benchmark, much was actually available smaller in scale and easier for testing. The high- to run, was performed on level (MQTh) model was the prototype system, with used with estimated values the emphasis on measuring for the input parameters, the CPU performance in a LOD and transaction CPU transaction processing service time. The estimated environment. The result LOD was based on previous was used to extrapolate the measurement observations CPU service time of the TPC from the VAX 6000 systems. Benchmark A transactions The other parameter, back- on the VAX 9000 Model 210 end transaction CPU service system and to refine the time, was derived from the early estimate. The results of these modifications o Timing information of supported the previous the VAX 9000 CPU high-end estimate of o Memory access time and performance of 70 TPS cache miss penalty of and refined the low-end the VAX 9000 CPU performance to be 62 TPS. o Prediction of cache hit The final, audited TPC ratio of the VAX 9000 Benchmark A measurement system under the TPC result of the VAX 9000 Benchmark A workload Model 210 system showed 69.4 TPS, which closely o Transaction path length matches the prediction. of the TPC Benchmark A Table 8 compares the implementation results from benchmark o Instruction profile measurement and the of the TPC Benchmark A analytical model outputs. implementation Table 8 The high-level model Measurement Compared predicted a range of to Model Predictions MQTh, with a high end of ____________________________ 70 TPS and with a strong Measured Modeled probability that the System MQTh MQTh high-end performance was ____________________________ achievable. VAX 9000 Model 210 69.4 70.0 ___VAX_4000_Model_300____21.5__ 20.8 The VAX 4000 Model 300 TPC Benchmark A results were also used as a validation case. VAX 4000 Model 300 20 Digital Technical Journal Vol. 3 No. 1 Winter 1991 Performance Evaluation of Transaction Processing Systems systems use the same CMOS performance by different chip as the VAX 6000 Model vendors. But it is 400 series and the same only one transaction 28-nanosecond (ns) CPU processing benchmark that cycle time. However, in represents a limited class the VAX 4000 series, the of applications. When CPU-memory interconnect evaluating transaction is not the XMI bus but processing systems a direct primary memory performance, a good interconnect. This direct understanding of the memory interconnect results targeted application in fast main memory access. environment and The processor, cache, and requirements is essential main memory subsystems before using any available are otherwise the same benchmark result. as in the VAX 6000 Model Additional benchmarks 400 systems. Therefore, that represent a broader the detailed-level model range of commercial and associated parameters applications are expected for the VAX 6000 Model to be standardized by the 410 system can be used by Transaction Processing ignoring the bus access Performance Council (TPC) time. The TPC Benchmark in the coming years. A measurement results are Performance modeling is within 7 percent of the an attractive alternative model prediction, which to benchmark measurement means that our assumption because it is less on the memory access time expensive to perform and is acceptable. results can be compiled more quickly. Modeling Conclusion provides more insight Performance is one of the into the behavior of most important attributes system components that in evaluating a transaction are treated as black processing system. However, boxes in most measurement because of the complex experiments. Modeling nature of transaction helps system designers processing systems, a to better understand universal assessment of performance issues and transaction processing to discover existing or system performance is potential performance impossible. The performance problems. Modeling also of a transaction provides solutions for processing system is improving performance by workload dependent, modeling different tuning configuration dependent, or design alternatives. The and implementation analytical model presented dependent. A standard in this paper was validated benchmark, like TPC and used extensively Benchmark A, is a step in many engineering toward a fair comparison performance studies. of transaction processing The model also helped Digital Technical Journal Vol. 3 No. 1 Winter 1991 21 Performance Evaluation of Transaction Processing Systems References the benchmark process to 1. Transaction Processing size the hardware during Performance Council, preparation (e.g., the TPC Benchmark A Standard number of RTE and front- Specification (Menlo end systems needed, the Park, CA: Waterside size of the database) Associates, November and to provide an MQTh 1989). goal as a sanity check 2. Transaction Processing and a tuning aid. The Systems Handbook model could be extended (Maynard: Digital to represent additional Equipment Corporation, distributed configurations, Order No. EC-H0650-57, such as shared-disk and 1990). "shared-nothing" back-end 3. TPC Benchmark: A Report transaction processing for the VAX 9000 Model systems, and could be 210 System (Maynard: applied to additional Digital Equipment transaction processing Corporation, Order No. workloads. EC-N0302-57, 1990). Acknowledgments 4. TPC Benchmark: A Report for the VAX 4000 Model The Digital TPC Benchmark 300 System (Maynard: A implementation and Digital Equipment measurements are the result Corporation, Order No. of work by many individuals EC-N0301-57, 1990). within Digital. The authors 5. L. Wright, W. Kohler, would like especially to and W. Zahavi, "The thank Jim McKenzie, Martha Digital DebitCredit Ryan, Hwan Shen, and Bob Benchmark: Methodology Tanski for their work and Results," in the TPC Benchmark A Proceedings of the measurement experiments; International Conference and Per Gyllstrom and on Management and Rabah Mediouni for their Performance Evaluation contributions to the of Computer Systems analytical model and (December 1989): 84-92. validation. 22 Digital Technical Journal Vol. 3 No. 1 Winter 1991 ============================================================================= Copyright 1991 Digital Equipment Corporation. Forwarding and copying of this article is permitted for personal and educational purposes without fee provided that Digital Equipment Corporation's copyright is retained with the article and that the content is not modified. This article is not to be distributed for commercial advantage. Abstracting with credit of Digital Equipment Corporation's authorship is permitted. All rights reserved. =============================================================================