Conference ulysse::rdb_vms_competition

65.0. "Combatting Tandem's SQL Debit Credit" by HPSVAX::KASTNER () Wed Jan 13 1988 20:23

I've spent a lot of time looking at the details of Tandem's NonStop SQL
debit credit benchmark, which they called TopGun, and have come to the
conclusion that:

	o They used every trick they could (but that's benchmarking!)
	o The tricks they used are not necessarily applicable in the 'real
	  world'
	o Tandem's Debit Credit is about four times easier than the Digital
	  Debit Credit guidelines I've seen
	o Digital has a marketing challenge in that our conservative Debit
	  Credit should not be directly compared to Tandem's
	o Using the same tricks Tandem did, I believe we can achieve very
	  competitive costs/tps and COO/tps, and that we should do such
 	  testing in order to allow the sales force to compare apples to
	  apples for the customer (before moving on to more realistic measures
	  of likely customer-environment performance)
	o The Digital spring product offensive will likely make us much
	  more competitive with Tandem
                                                     
Without running debit credit on Tandem (yet), we can't accurately assess
what a Tandem would do running a conservative debit credit.  However, we
can look at several of their 'go fast' tricks and make a reasoned assessment
of what that trick bought them.  

I'd like comments on the following arguments and numbers from those of you
familiar with Tandem, TopGun, or Digital debit credit.  

Because Tandem 'clustered' multiple systems, I'll use a single
8-processor VLX configuration for these arguments because it scales to standard
Digital configurations without getting into lot's of cluster performance
arguments.  Tandem used four identical 8-processor systems to generate the
208 tps numbers quoted in the press.

Baseline:
        8 VLX processors (at 3 mips and 8MB each)
	20 disks at 128MB each (they wanted heads) includes 90 day history
	tps based on mirrored disks; costs based on mostly unmirrored
	1 56KB X.25 line to each processor
	each processor had its own set of branch/teller/account files

	tps (90%-2 seconds) = 58
	5 year COO (discounted maintenance cash flow)	$2,995,000
	COO/tps 	$51.6K

	Source:  Tandem publication #84160, TopGun Benchmark Workbook


Restating the Baseline for Conservative Debit Credit

1.  Performance at 95%-1 second response time (end-to-end, measured at driver
system [which is overly conservative]) drops about 50% based on extrapolation
from the data points provided in Tandem's TopGun report.  Tandem claims
180-200 milliseconds in the response time is due to the network, driver X.25
process, and driver system dispatching overhead.
                                
I feel very comfortable in downgrading by 50% on this issue.

2.  Tandem did NO presentation services, arguing that this was done by an
(uncosted) intelligent controller at the branch.  This argument has some
real world weight, but deviates from the debit credit definition.

Because Tandem's Pathway product uses INTERPRETED COBOL, then ANY presentation
services get expensive quickly.  The cost of mapping 30 fields per transaction
is, I estimate, worth from 20% to 50%.  I know our own testing shows that
presentation services in TDMS contributes substantially to making debit
credit CPU-bound.

3.  Tandem, by using the 'intelligent controller at the branch' argument,
 simulated only 1000 active tellers for 1000 branches, not the 10,000 called
for.  Doing so eliminated the possibility of branch record contention, and
saved memory and virtual circuits.  Worth maybe 5%-10%.

4.  Every processor had its own comm line to local disk account/branch/teller
files (ABT files). 'Not on us' transactions were routed to the proper
processor.  Tandem gained considerably by partitioning the workload VERY
evenly across the available processors, scaling the database at 8
tps/processor.  This resulted in an overscaled database (8 processors x
4 systems x 8 tps/processor = 256 tps...they only got 208 tps).

Digital (and any other vendor) could do similar partitioning, so  I don't
consider this a trick.  However, not all (not many!) customer applications
lend themselves to the symmetric partitioning Tandem did.

If Tandem had to use central, single ABT databases, then they would hit
the wall on disk process throughput or some such software limitation.  They
would need much less hardware, but they'd get much less work done too.

HPS debit credit testing this spring should shed some light on the cost/value
of partitioning.

Let's assume for this memo that partitioning is an accepted debit credit
practice.  Tandem can use it with no penalty.  We can use it too.

5.  Tandem gets the same throughput with SQL as with their file system.
 Sounds competitively bad.  In fact, Tandem used proprietary extensions
to SQL and accessed the teller and branch files by relative record key.
Relative access, of course, is a very fast access method.

Tandem made some claims about doing the account balance update in SQL in
the disk process itself, a nice feat.   However, NonStop SQL can only do
this with single column updates.  This means applications that have to update
multiple rows will have a longer code path.

Lastly, Tandem treated the ABT database as files.  There were no (expensive)
JOINS or other SQL constructs used.  To Tandem's credit, SQL is not required
by debit credit standards.  My point is that just because Tandem used NS SQL,
we should not counter with Rdb when RMS might do the job.  On the other hand,
as I conclude below, we may be in better shape than many people think...which
is the point of this exercise.                                

6.  Tandem used and costed  mirrored disks.  Worth maybe  5% to throughput.
                                                                           

Conclusion

A conservative approach to Debit Credit, such as the approach Digital uses
for Debit Credit, when applied to Tandem will lead to considerably reduced
Tandem throughput.  My estimate is that Tandem would get 10-15 tps on the
baseline described above with a conservative Debit Credit, or roughly a
quarter of what they're quoting in the marketplace.  Such is competitive
benchmarking.

I have no doubt that we can perform today in the 10-15 tps range, if not
with one machine, then with a small cluster.  New products this year will
push us to even higher levels.

Using Tandem's TopGun methodology and definition of debit credit in a Digital
debit credit should result in competitive performance and very competitive
COO/tps.  Partitioning, branch MicroVAXes for presentation services, etc.
work just as well in our favor as they did for Tandem.  In fact, I think
we should put some SMP machines together and generate some 100+ tps numbers
if only to say, yeah, we can do that too.

So, Tandem is not as awesomely good as first appears.  And Digital is not
nearly as bad as corporate mythology would have us believe. 

Peter Kastner
Corporate Systems Group
13 January 1988



            

      While much of this analysis has merit, I would question some of
    the facts.
    
      It is my understanding that Tandem's database is genuinely
    distributed. That would support true vertical partitioning of the
    data. We cannot do that.
    
      I believe they have a true distributed data dictionary that supports
    such partitioning. Again, we do not.
    
      As for the presentation services, don't we do the same type of
    off-loading with ACMS on the Rdb benchmarks?
    
      It is my understanding that Tandem heavily employs parallel
    processing to generate their transaction rates on a series of
    relatively small CPUs. Again, we do not do that.
                      
      I think you would be hard-pressed to achieve the 10-15 TPS milestone
    that you discussed with Rdb. However, you are quite right that given
    a VAXcluster system, the aggregate TPS rate would probably exceed the
    10-15 range.
    
      The market consensus is that Tandem has an 18-24 month technology
    edge on the rest of the industry. That does not diminish the fact
    that Digital's relational database is very easy to use and has a
    far greater range of development software to which it will interface.
    The network on which Rdb runs is also generally accepted as the network
    technology leader. It is on this basis that we can challenge Tandem.
    I would swallow very hard before engaging Tandem in a technology
    fight. In addition, being on the leading edge does not necessarily
    buy anything for our customers. 
    
      Consequently, the crux of the discussion becomes, "What does all
    that shining technology buy you?" In and of itself, technology buys
    the customer nothing. It is in the appropriate use of technology,
    the available products, and the number of useful applications where
    we can bury Tandem, not technology.
    
    ---- Michael Booth 
    
      

    	When confronting Tandem with "real world" transaction processing
    requirements head on, Digital can and does win.
    Two cases come to mind:
    
    	1.Alcoa chose Digital (about a year ago, one dept) when a technical
          team tried application development in the Tandem environmnent
    	  and then the VMS environment. VMS won without question.
    
    	2.About 5 months ago Tandem were forced into a benchmark situation
    	  where they had been claiming an easy 20tps on a VLXI system
          they were proposing against an 8700.
    	  In the actual benchmark Digital achieved 6 tps and Tandem
          managed 7 .Even tho thier system was significantly less expensive
          than the 8700 configuration, they lost the business for obvious
          reasons. 
          This was NAPA WILMA by the way.
    	
    I appologize for lack of software detail here, I was involved only
    in presenting to the customers concerned at various times.All
    benchmarks were I believe carried out in the now defunct MR02 benchmark
    centre. Now defunct because HPS own it for OLTP "work".
    
    

    Be Careful with dated references.
    In the last 6 months, Tandem has radically improved application
    development. The NON-STOP SQL database and 4GL are now excellent.
    Tandem used to be quite beatable in application development. That
    "gap" has narrowed substantially.
    
    Was the benchmark mentioned in example 2 run against the new NON-STOP
    SQL (of Cood & Date fame) or the old one. I'm just interested for
    general information purposes.
    
    I too, think Digital can beat Tandem. But I believe that generally
    we will be more successful selling "big company pluses" (i.e. better
    support, more available applications, superior network integration,
    education, etc.). Trying to fight Tandem at their strength is hardly
    tempting to me.
    
    I'll draw a nice parallel. Fighting Tandem on the technical merits
    of a relational database would be like Tandem attempting to fight
    DEC on the technical merits of networking and communications. How
    many would they win?
    
    ---- Michael Booth

    Just a note to be slightly wary of the benchmark comparisons with
    Tandem.  The original D/C benchmark numbers (ie) pre Topgun 
    were 10 TPS for the VLX.  That's 10tps per processor!  320 tps for
    the TopGun system.  Even if you cut it in half, it skews the $/TPS.
    Tandem has made a corporate commitment to NonStop SQL so they
    are quoting numbers in SQL transactions, but their Enscribe numbers
    are considerable higher.  
    
    If we negate the effects of using SQL, we are in effect placing
    our benchmark against the higher numbers derived with Enscibe.
        
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    We need to use our benchmark comparisons to get past the issue of
    performance, and onto the issue of the best solution for the customer's
    problem.                                                
        
    DEC offers a richer set of products and possiblities to the customer
    than does Tandem.  Dec is easier to program and operate.
    Dec has a long history of successfully solving customer problems.  
    
    We need to stress the importance of these issues and our other strengths.
    
    

    another thought
    Tandem has very few, (read no) application solutions written to
    access NonStop SQL.  They may be selling the heck out of the new
    technology, but the end user is still buying an application written
    in the old Enscribe data file manage.                           
    
    nobody asked...I was justzz thinking