Conference ulysse::rdb_vms_competition

	The result was announced by Database Associates and audited 
	by Codd & Date.

	Ketil

Re:  .0 and .1

As of today (Oct 24, 1991) we (Digital) have not received any auditing 
results for review from TPC (Transaction Processing Performance Council). 
Until we receive the audited results from TPC we won't be able to comment
on the Rdb performance figures.  In the past, Oracle has not always done a
good job of tuning Rdb to maximize its performance. 

In addition, this paper compares Rdb V4.0A (the currently shipping version) 
and Oracle V6.2 (which is not their most current version and does not 
provide cluster support).  I'm not sure that I would agree that it was a 
true apples_to_apples comparison.

We believe that Rdb V4.1 will provide additional substantial performance 
improvements over V4.0 for many applications.

    I too can make any database system look good/bad!
    
    So what else does this say.
    
    -Jay
    

    re .0


    Considering Oracle's claim that they want a good working relationship
    with Digital, and their decision to tone down their anti-Rdb adds, I
    find this report strange.  Is this the work of a renegade Oracle
    employee, or is Oracle proving once again that they are a bad business
    partner?
    
    There various opinions over the relative merits of the different TPC
    benchmarks.  Note that Oracle only releases results for TPC-B.  They do
    not release results for TPC-A.  (FYI, Rdb's TPC-A results compare very
    favorably to other vendors who release results)  Also, note that
    anybody else who runs any Oracle (for benchmarks or production) cannot
    release performance information since this would violate Oracle's
    licensing agreement.  Two questions are:

    - Why doesn't Oracle release TPC-A results.  Oracle people claim that
    TPC-A is a measure of TP performance, not database performance, and is
    irrelevant in reporting Oracle's performance since Oracle is a
    database, not a TP product.  Other people claim that the reason they
    are not released is because the results are probably not favorable to
    Oracle.  (Note that other people who know the results of TPC-A
    benchmarks for Oracle cannot release them.)

    - Why doesn't Oracle allow other other groups (such as Digital) to
    release benchmark results.  I know of no good defense for this position
    (other than it keeps other vendors from doing to Oracle what they do to
    other vendors.)  There are many unfavorable ways to interpret why they
    choose to do this (for example, benchmark results aren't reproducible,
    performance on customer system is not in-line with performance claims).


    Finally, one thing to keep in mind is that no customer runs TPC-B (or
    TPC-A for that matter) to get their business done.  The only results
    that do matter is how the database runs the customers application. 
    Most applications are significantly more complicated than TPC-A or
    TPC-B.  The Transaction Processing Council (owners of the TPC
    benchmarks) is considering a more realistic benchmark called
    order-entry.  I believe that this benchmark will be much more useful
    than TPC-A and TPC-B in comparing performance of different database
    systems.

    Oracle hired Database Associate to tune Rdb. They did the best they
    could. They hired Codd and Date to audit. 
     
    I am in the process of writing up a rebuttel of sorts.
    
    Barry

	Barry, can you mail me what you have until now? I need it for 
	monday.

	I shall have a meeting with this customer who gave me the paper.

	Thanks in advance

	Ketil

	BGO01::KETIL 

    No, I am not going to mail out a 1/2 assed rebuttal but at least
    understand the facts.
    
    Oracle hired a company named Database Associates to configure and run a
    TPC-B test. They were asked to make Rdb look as good as they could.
    David McGoveran, a well respected DB man, designed the Rdb database.
    Oracle hired Codd and Date to audit the benchmark. This was his first
    hands-on experiancee with Rdb though. Oracle intentionally selected the
    configuration that makes Rdb look the worst (above 4 cpus) and Oracle
    look the best. Oracle submitted this benchmark to the TPC since there
    is no rule forbidding it. We, Digital, have no official TPC-B numbers
    at this time since we withdrew our cluster test when the council
    changed the specification after the fact. 

    Needless to say, our own people tell me they could have far higher
    numbers than David did. The problem is, oracle can run around comparing
    two audited TPC benchmarks and we cannot stop them. As you know, their
    license forbids us from benchmarking their products and publicizing the
    results. We are an open business practices company. In this case it
    works to our disadvantage.

    All I can say, is educate your customer on what the TPC-B measures
    (nothing useful) and doesn't measure (a RDBMS system or realistic
    transactions). Remind them that Oracle paid for this.
    
    As for our new TPC-A numbers, until they are registered with the TPC we
    cannot publicly disclose them. All I will say is that Rdb 4.0-->4.1
    show 40-51% throughput improvement on our part of the TPC-A test. 

    
    Barry
    

    The real question remains:
    Why don't we Digital publish our TPC-B numbers for the 6000 series.
    If we do it we can do three things at once:
    1. Show that our TPC-B numbers are much better than Oracle published.     
    2. Show that Oracle twisted RDB to look so bad.
    3. Show that TPC metrics are meaningless
    
    Yochai
    

    re .9
    
    From my understanding, the question is not only what does Oracle do to
    make our numbers so bad, but what so they do to make their numbers so
    good.  If they changed their licensing agreement, it would be
    interesting to see if anyone could reproduce their benchmark results.
    
    Also, it would be very interesting to have an independent analysis of
    the implementation techinques used in all vendors benchmarks with
    implementation techiniques used by real customer applications.  It is
    my understanding that benchmarks tend to use features that are
    difficult or unreasonable to use in customer applications...making
    benchmarks even more meaningless.

    Re -1.
    
    I am very curious to know what change in the TPC-B benchmark made RDB
    perform so bad ...
    
    I saw some figures about 4.1 performance testing which estimate that
    RDB will perform the TPC-B much better than the numbers published by
    Oracle...
    
    Yochai

History:

Both TPC-A and TPC-B are derived from the DebitCredit benchmark as
defined by an article in Datamation about 6 years ago.  That benchmark
defined what might be considered a banking application with branches
where customers typically (85% of the time) did business at their own
branch.  15% of the time they transacted business at another branch.
This 85/15 split was intended to force a degree of sharing typical in
real applications on the measurements.  They fully allowed for
distributed implementations in which the source of the transaction, a
teller at a specific branch, determined who executed the transaction.
For example, if there was a machine local to the branch it would always
execute the transaction, and 15% of the time it would have to reference
another branch's account data.  This also allows a specific transaction
to be routed to a specific server, even if all the servers are on a
single machine.  TPC-A completely preserved these characteristics of the
original benchmark.  We use ACMS for this routing function.

In its original definition, TPC-B also preserved this characteristic.
In TPC-B, Digital implemented the routing via application-level code.
ORACLE proposed that the benchmark definition be changed to prohibit the
use of application-level code for doing the routing.  This not only
imposes the 85/15 split on the application, in practical terms it imposes
a uniform distribution of arrival rates across the entire set of servers
(think of it this way, it assumes that you will walk up to a BoA ATM as
often in San Francisco as in Los Angeles, even though you live in Los
Angeles).  The TPC approved this change to the benchmark specs.  This
means that we must either use a saleable product to do the routing,
giving up some of the CPU to that product and thus yielding unacceptable
results, or eliminate the need to perform the routing.

ORACLE avoids the routing by implementing global buffers.  We have a
global buffer scheme (optionally) in V4.1, but it isn't mature enough for
the TPC-B case.  There are some anomalies that make it unsuitable for
update-intensive cases such as TPC-B.  This was PLANNED behavior.  We
planned to do Global Buffers in two phases, with only the first phase
being part of V4.1.  Had we known that the TPC-B benchmark was going to
be changed, I imagine that the priority for tuning our global buffer
implementation would have gone up and V4.1 would have been planned out
differently.  As it is, we are stuck between a rock and a hard place.

Global Buffers are not a panacea for this application.  They have
inherent limits in scalability of distributed systems, which is why
Tandem uses a partitioning model in their database system.  It just so
happens that ORACLE has a good way to make us look really bad for a
specific case right now.  A pathological case to be sure, but one that
will cause us lots of pain nonetheless.

Hal

    Please be aware that KODA is painfully aware of the "feature set" in
    the Rdb product that gives us less than optimal TPC-B performance.  In
    other words, we know there is a problem and we are working extremely
    diligently in solving the problem expeditiously.
    
    As Hal mentioned, our global buffering strategy is new and needs
    further nurturing and bit-twiddling before it can be called an
    olympic-class buffering implementation.  But that is just one of the
    problems; we are working on the problem as a whole, with many
    engineering groups, to solve the complete problem.
    Rick

    Thank you all for your detailed explanations.
    I guess we are all frustrated by the Oracle aggresive marketing when we
    all know that Rdb is a much better database than Oracle.
    
    As a matter of fact as a technbology consultant in Israel ACT I ran
    many benchmarks against Oracle and RDB ALWAYS performed better,
    so It was hard for me to understand why should RDB TPC-B numbers be
    worse than Oracle's.
    
    Regards,
    Yochai
                                                 

It's a really awful situation.  I feel really bad for the KODA folks
because they did an incredible job on V4.1 and I believe that in nearly
every customer application out there we will absolute blow the
competition out of the water.  Meanwhile, all that people can talk about
is the TPC-B situation....

re -1: Thanks for the pat on the back - it's greatly appreciated!

We believe we can solve the TPC-B problem - it's just a matter of time and
resources... (like I always say, "A little code, a little data, no problem!" :-)

Rick

    
    
    
    I work in the Center for Migration Services in Colorado Springs,
    Colorado.  I am part of the Database Team which is headed by Dave
    Munns.  We are aware of ORACLE customer dissatisfaction which is
    based on the following:
    
                   
               1)  Price performance based on ORACLE's exorbitant
                   licensing fess.
    
               2)  Lack of timely and quality technical support.  We have
                   an ORACLE license in our department, and ORACLE took
                   about two months to help us get it installed correctly.
    
               3)  ORACLE's agressive anti-Digital and anti-Rdb ad campaigns
    
               4)  ORACLE's tactics of refusing to allow the publishing of
                   benchmarks without their permission.
    
               5)  ORACLE's response (lowering their licensing fees) when
                   they find Digital is closing in, persuading a customer to
                   seriously consider an Rdb migration.
    
    
    One of our key migration strategies is to recommend the TRIM/tools, a
    data management tool that allows rapid application development on top
    of Rdb database structures.  In most cases, the translation of ORACLE's
    SQL*FORMS (V. 3.0) structures can be automated about 85 - 90%, and the
    resulting system will perform better because Rdb/VMS is actually superior
    to ORACLE, and it is more closely coupled with VMS than any relational
    database system that runs on the VAX.
    
    We have a "Five Year Cost of Ownership" presentation study that Dave
    put together for the Information Management Partners Conference last
    summer.  We keep this study updated with the most current ORACLE
    prices.  If you want to see some really dramatic figures, write,
    e-mail, or call us and request a copy of this study.   Be sure to
    specify what format you want: SDML (VAX Document) or PS (Postscript).
    When you see the cost difference between a Digital customer choosing
    Rdb/VMS, and choosing ORACLE, you'll see why it makes good sense to
    decide on Rdb/VMS.
    
    We believe strongly in Rdb/VMS and want to help you understand
    migration issues concerning ORACLE when these opportunities arise.
    
    Thanks for your interest,
    
    
    Bill Slater                           E-Mail Address : TYFYS::SLATER
    Software Specialist                   DTN            : 523-2018    
    Center for Migration Services
    Digital Equipment Corporation
    1110 Chapel Hills Dr.
    Colorado Springs, Colorado,  80908                  

    Note 1017 should be nominated for "Note Of The Year". We need to
    fully understand the issues around the claims of our competitors. I
    hope people who read this note, and do not understand the TPC-B
    benchmark will take the time to learn why TPC-B has got nothing to do
    with 99% of the customer's applications!
    
    In the short term Oracle can claim these high numbers, thus causing us
    extra work explaining reality. But if the customer does not understand
    why Oracle does not do TPC-A or what TPC-B is really showing, they
    might still buy from Oracle. Our job is to help educate the buying
    public.
    
    I will look forward to seeing the new Oracle account next
    year when they did not get their expected performance, when they get
    their maintenance bill, when they could not get any support, when the
    application is not portable to all those other platforms, when they get
    their upgrade license, when they need other products to help integrate
    the enterprise...
    
    I agree that engineering has done a good job. We should still be able
    to outsell Oracle, even though our TPC-B numbers on ONE platform are
    not as good!
    
    Ken McMullen
    
    (What are Rin Tin Tin, Lassie and Oracle? Two movie stars and a dog!)
                     

I wouldn't call TPC-B biased towards a multi-server architecture.  TPC-B
is (or one could say, has been) biased to demonstrate the capabilities of
a database system in the absence of any locality of reference to the
database.  ORACLE's implementation will actually not handle TPC-B well as
the number of nodes in a cluster grows beyond some number, for that you
need Tandem-style partitioning.  I know of know move to throw away TPC-B.

TPC-C is a full TP benchmark based on an Order-Entry application.

Hal

Hal,

By 'multi-server' I mean the database system can start up multiple instances
of a server process, for example, to take advantage of SMP. Isn't it true
that a server oriented approach is best for TPC-B? Since a single instance
of a server can be a bottleneck at high tps's then multiple instances, or
a multi-server approach is the best, right? In this regard, ORACLE's
implementation is probably ideal. 

John.

	I just discovered this notesfile, and thought I could use it for some
	information, as well as a plug.

	First, as an introduction, I am in the Software Performance Group of
	TNSG, and am the project leader for the TPC-C benchmark development.
	I also represent Digital at TPC (along with others).

	Second, Hal Berenson etc. have given details about the Oracle vs. Rdb
	episode. I would like to add that TPC at its recent meeting passed a 
	new rule that will restrict the kind of test that Oracle sponsored
	thru' Database Associates. In short, the new policy deals with a test
	sponsor doing a test with another vendor's products (hardware, software,
	etc.). If the latter chooses to question the former on the test not 
	being a good-faith effort, the sponsor will have to demonstrate that
	the test was indeed a good faith effort, AND CONVINCE at least
	2/3 of the TPC that it was (this is considered a substantial hurdle).
	Unfortunately, this new policy will only apply to new tests, and thus
	not to the Oracle/DBA test.

	Here's something that some of you could use when dealing with the 
	TPC-B vs. TPC-A testing. This is due to Charles Levine of Tandem, who
	is my colleague on the TPC-C development subcommittee:

	"TPC-B is like how fast you can idle your car, while TPC-A is like how
	fast you can drive your car."

	Charles also uses this as a justification as to why Tandem does not 
	believe in doing TPC-B tests.

	If I were to extend this analogy, I would say that TPC-C is like how
	best you can manage your overall traffic!

	Which brings me to the plug. At the last TPC meeting, the TPC-C 
	benchmark was approved for a public review. This means that the TPC
	thinks that the benchmark definition is mature enough to become a 
	standard and invites public comments. Prior to this, the benchmark 
	underwent a company review (same as a public review, only limited
	to the TPC member companies). For the public review, I am looking for 
	volunteers to carefully read the benchmark draft and send comments
	(this will be about a 90-100 page document, available mid-January).
	My earlier request for volunteers for the Company Review elicited
	many (15-20) responses to receive the document, but comments from only 
	ONE reviewer! So serious inquiries only; this will require a substantial
	commitment of your time.

	Send your requests to me at santur::shah or shah@tay1.dec.com

	Time permitting I will enter a detailed note about TPC-C.

    The following is the text off a copy of some material I was given
    yesterday. This is intended to be sent to the Digital installed base,
    possibly through ON$DECK magazine (sort of an Australian version of
    Digital Review).
    
    I have to prepare a response to this for ON$DECK publication, for our
    Sales force, and for Australian/NZ management (with whom ORACLE is
    doing a good sucking-to job at the moment - fancy tryin to become a
    friendly CSO at the same time you're going after installed base!).
    
    :-(
    
Dear VAX/VMS User,

The VAX market for RDBMS is certainly very busy.

A recent Australian survey of ON$DECK readers indicated that the most important
issue to consider when making a decision to purchase an RDBMS is performance.

In today's heterogeneous computing environment involving multiple hardware
platforms and numerous end user applications such as payroll, analysis,
reporting and inventory control, most user sites require a relational database
technology that is fast, efficient and very flexible.

Oracle's performance of its RDBMS software has show itself to be a clear leader
in this field.

Independent tests have shown ORACLE's superiority when it comes to true price
performance effectiveness of an RDBMS in a VAX/VMS environment.

To learn the results of these tests, Oracle invites you to read the enclosed
flyer to find out how you can order your copy of the FULL DISCLOSURE.

* ON$DECK magazine published by CMP publications



			    THE VAX/VMS MARKET
		    THE FACTS SPEAK FOR THEMSELVES

Database Associates, an independent database consulting firm, have announced Rdb
TPC-B benchmark results that confirm the dominant position of ORACLE technology
in the VAX/VMS market. This is the first true "Apples to Apples" comparison
between ORACLE and Rdb:

SOFTWARE	HARDWARE	PERFORMANCE
ORACLE V6.2	VAX 6000-560    153.93 tpsB

Rdb  4.0A	VAX 6000-560	60.13 tpsB
Rdb  4.0A	VAX 6000-540	57.04 tpsB

(All numbers audited by Codd & Date, in full compliance of the TPC-B spec.)

FACT:
*  ORACLE:   2.5 TIMES AS FAST AS RDB.
   Rdb delivered only 60.13 tpsB on a six-processor VAX 6000 Model 560, just
   39% of the 153.93 tpsB recorded by ORACLE on the very sample platform.

FACT:
*  ORACLE:   NEARLY TWICE THE PRICE/PERFORMANCE.
   Rdb:      $31,600 per TPC-B transaction.
   ORACLE:   $17,156 per TPC-B transaction.

FACT:
*  ORACLE scales superbly not only on SMPs, but VAXclusters as well.

   WANT TO KNOW ALL THE *FACTS* ABOUT PRICE-PERFORMANCE, TOTAL SYSTEM COSTS AND
   TRUE SCALABILITY IN A RDBMS ENVIRONMENT?

   IF SO, THEN CALL ORACLE TODAY TO RECEIVE A COPY OF THE BENCHMARK FULL
   DISCLOSURE REPORT.

   CALL 008 023 462

                  I N T E R O F F I C E   M E M O R A N D U M

                                        Date:     20-Dec-1991 02:46pm AES
                                        From:     Keith Anderson
                                                  ANDERSON KEITH
                                        Dept:     Marketing
                                        Tel No:   [61] 6 2754832, 2 5617035

TO:  GEOFF WEST                           ( WEST GEOFF@A1@SNOC02 )
TO:  NEVYL LENNOX                         ( LENNOX NEVYL@A1@SNOC02 )
TO:  Rolf Jester                          ( JESTER ROLF@A1@SNOC02 )
TO:  ELVERY ROBIN                         ( ELVERY ROBIN@A1@SNOC02 )


Subject: ORACLE on VAX TPC-B benchmark attack                                  

Geoff

Just a quick note to let you what've I found so far:

-   The TPC-A, TPC-B tests managed by the TP Council have little relationship 
    to real world customer application benchmarking, being based on an 
    unlikely banking scenario. TPC-B is more of a pure database test, TPC-A 
    includes user interfaces, TP monitors and so is more relevant. [Mark, is 
    this right?] TPC-C will be another benchmarking test based on an Order 
    Entry application, which we look forward to as being more useful.

-   We are not permitted to release any ORACLE performance information we may 
    have gathered, as performance disclosure is denied under the ORACLE 
    licensing agreement under which anyone purchases ORACLE. (NOT an Open 
    Business practices company).

-   ORACLE sponsored Database Associates to do the Rdb benchmark - it was not 
    run by Digital. DA had never used Rdb before.

-   We would get better Rdb results on TPC-B than DA did, but due to the 4 and 
    6 CPU systems used, its not clear how close we'd come to ORACLE.

-   I don't yet know what the disk, memory, and I/O configurations were: 
    imagine if ORACLE ran on DSSI disks and DA used HSC-50 : the 
    price/performance of the HSC's is terrible (good performance, very large 
    price). Database systems are normally I/O bound : a different hardware 
    config would make lots of difference.

-   I don't think we've done any TPC-B tests on 6000-560 or 540 config's, as 
    we commonly do TPC-A, which includes the ACMS TP monitor etc.

-   When ORACLE and DA did these sets of tests, the TPC had no rule on a test 
    sponsor submitting test results on another vendors products. Since then 
    the TPC has decided that in future, that if a test sponsor submits test 
    results with another vendor's products (hardware, software, etc.), and if 
    the latter chooses to question the former on the test not being a 
    good-faith effort, the sponsor will have to demonstrate that
    the test was indeed a good faith effort, AND CONVINCE at least
    2/3 of the TPC that it was (this is considered a substantial hurdle).
    Unfortunately, this new policy will only apply to new tests, and thus
    not to the Oracle/DBA test.

-   In real world situations, it is uncommon for Rdb to lose a benchmark 
    against ORACLE on VAX/VMS. If ORACLE can corner the customer into a single 
    VAX with 4 or 6 CPU's then we're at some risk, but due to ORACLE's lack of 
    flexible performance tuning options, we still have a chance. However, we 
    cannot mention specific instances without invalidating the T+Cs of the 
    ORACLE licensing agreement (that forbids disclosure).

-   ORACLE sales on VAX/VMS are a small fraction of what they were two or 
    three years ago. Rdb/VMS has won the wars - ORACLE should expend their 
    marketing efforts on a joint win-win strategy with Digital to promote 
    ORACLE on ULTRIX. There is the odd occasion in which ORACLE on VAX is a 
    better solution to customer needs than Rdb, but thats not common.

-   The Butler-Bloor report "Comparative Database Performance on VAX" clearly 
    positions Rdb as superior in most respects, and esp performance. 

-   A guy in DBS Marketing US is writing a rebuttal, and I'll pass something 
    onto ON$DECK too.

I think there are two issues here: a. is ORACLE going after Rdb/VMS business a 
good thing for Digital? b. how do we compete against a cashed up marketing 
campaign?

I suggest that the first question is answered in the affirmative if you think 
that we make money out of hardware and should let third-parties take the 
software profit dollars. 

I cannot spend like ORACLE on that kind of an advertising campaign to promote 
Rdb. I can compete by written articles, etc and we should not waste effort by 
just competing against TPC-B benchmarking, but more generally as ORACLE is 
not so good in most database areas.

Over 3 years ago, ORACLE was usually a better RDBMS than Rdb/VMS but that was 
ORACLE V5 and Rdb/VMS V2 days. We now have ORACLE V6, and Rdb/VMS V4 has 
outpaced ORACLE in terms of features, benefits, performance, reliability, 
open interfaces, and all that at a much lower cost of ownership.

Regards
Keith

    Do the new TPC rules on releasing the performance numbers for other
    vendors supersede the licensing agreement.  Specifically, can we do a
    benchmark for the latest version of Oracle, and present the results to
    TPC.  Assuming Oracle objects and we are able to prove the correctness
    of the result to two-thirds of the members, can we then release the
    numbers?  Or can Oracle still hide behind their licensing "agreement".


    Maybe we should have an ad where we display a chart of TPC-A numbers
    for Rdb/VMS 4.1 and other database vendors on competing platforms.  The
    chart would include companies that refuse to release TPC-A numbers. 
    The space for the performance of of vendors with missing numbers could
    be filled in with something like "company secret".

    The text of the ad could say something like..."We know what the TPC-A
    performance numbers are for these products, but the licensing
    agreements for these companies won't let us release these numbers. 
    Is the reason their licenses don't let anyone release TPC-A numbers
    because these companies also know the TPC-A performance of their
    products?"

    re: .27
    
    Good questions ... You might want to copy your thoughts to Walt Kohler
    at TPSYS::KOHLER.