Conference 7.286::digital

    This is probably a symptom of half of engineering being directed to
    make the software work on 10 zillion different platforms. Rather than make
    it work on one platform well.
    
    Dave

    re: .0
    
    The best way to solve this class of problem is to have a management
    chain that is concerned about product quality.  While quality must
    always be traded off against time-to-market (else we'd never
    ship anything) there needs to be a strong concern for quality, all
    the way up the management chain, so that a legitimate quality concern,
    such as lack of sufficient testing, is never suppressed by upper
    management in order to ship the product "this quarter".
    
    Sometimes that "best way" isn't achievable.  Not being a manager
    myself, I don't have much leverage in making it happen.  Therefore,
    I have an alternative strategy, which allows me to develop quality
    products in an environment that is less concerned with quality than
    I am.
    
    My approach is to build time enough into my schedule to make sure
    that I am producing a quality product.  There is time for checking
    and thorough testing, and time to keep an ear open to the CSCs for
    feedback from customers.  I think I am the only person in my group
    who regularly reviews the CSC's customer contact reports for my
    product.
    
    I don't hide my strategy from my immediate management---my schedules
    include everything I plan to do, even though that means I don't produce
    code as fast as I could.  My management has learned to tolerate my
    peculiarities.  They even held still for a demand I made recently that
    a particular bug (in my code) was serious enough that customers who
    sent SPRs about it should get a corrected image in return.  That
    required several weeks of perseverence on my part, but I got it done.
    
    I am fortunate to have an understanding management.  Maybe the
    "valueing differences" program has worked on my behalf.
    
    I am not sure whether to recommend that everyone take the position that
    I have.  I've been in the computer programming profession since 1963,
    so I can get away with more than most.  However, I do recommend it
    to anyone who has a serious quality concern and a manager who lets
    people "do their own thing".
    
    Whenever I wonder if I am "doing the right thing" I think about the
    Challenger disaster, in which engineers let their management override
    their quality concerns.  I also think about the Hubble Space Telescope,
    which is operating at half of its expected resolution because an
    important test was omitted, due to its cost.
    
    re: .1---I wouldn't blame the need to support Ultrix.  Additional
    requirements should make projects take longer and cost more, not
    fail to deliver a quality product.
        John Sauter             

    re: .0 + .2

    I have spent most of the last 15 years doing various parts of QA/QC and can
    tell you that I have never seen a perfect piece of software.  That does not
    mean that I have never released any products.  I have let hundreds go that
    have bugs in them.  The reasons are that most of those bugs were either
    defined in the release notes or so complicated to find that the risk to the
    customer was minimal compared to how long it would take to find them.  Then
    there were the ones I just missed :-(.

    There is no way to find every bug.  A customer that expects that will be
    very disappointed in all software they buy whether it is ours, I*M, Apple
    or anyone else's.  What they should expect is that the software has had a
    reasonable amount of testing done before it is dropped on them.  The
    situations in .0 cry out that there was not enough testing done.  John
    Sauter has the correct attitude, plan time for testing.

    The biggest problem with tight schedules is that the testing time is used
    for more development.  It always seems to take more time to develop
    software than is allocated.  So the solution many people (both management
    and programmers) take is to steal the testing time.  This is generally
    robbing Peter to pay Paul.  If the time is not spent in testing then it
    will be spent in tracking SPR/QARs and trying to please irate customers.
    
    The only thing that I can do for a perfect piece of software is to add time
    to the process.  Since I have never seen perfect software, my testing has
    been a benefit to the customer and to the company.  There is a quote that
    always comes to mind when someone wants to take way the time needed to do
    testing, "The bad taste of a failed product will stay with a customer much
    longer than the sweet flavor of an on-time delivery."

    Lee G. with_no_easy_solution_to_the_problem

Personally, I'd blame anyone who calls a "person" a "resource".

No sense of worth => lousy quality.

The beginning to solving this problem:


	Step 1:  Please read .0 one more time.

	Step 2:  Take 1-2 minutes to "feel" how it makes you feel about the
                 product(s) you are responsible for.  How does this make
		 customers feel?  Reluctant to become acclimated to DEC
		 products?  Worse?

	Step 3:  Don't ponder on note .0 anymore.  Do something about it!
                 and DO IT NOW!


These steps are meant for ALL -- management or not.


	+ Dennis

    In the almost three years I have been with dec, and dealing with the
    developers on the sna interconnect products, it is my impression that
    these developers are the best in the business (i have worked for three
    other major vendors). We do not produce perfect software; if we did I
    would not have a job.  Customers do not expect perfect software, they
    do expect their problems resolved in a timely and efficient manner.
    
    The problem;
    
    I have seen one version of a product shipped without being tested due
    to time to market.  It was a management decision and management blew
    it.          NEVER SACRIFICE ACCURACY FOR SPEED.
    
    After introduction of another product and problems started comming in,
    I heard the manager for that product say "oh, we knew about that
    problem, we figured we would fix it when customers started complaining"
    
    or, how about this one;
    
    "we knew about that problem but did not think customers would ever see
    it"  (earth to management, come in management).
    
    But my all time favorite is;
    
    "that problem will be fixed in the next release"
    
               "when will the next release be available?"
    
    "we cannot commit to a next release" (not all the dots on the dice
    here).
    
    
    These types of management, time to market, and a very informal support
    structure causes our customers to have the opinion that our software
    is not reliable.
    
    Our inability to secure and maintain our customers confidence in our
    ability to provide reliable software in a timely manner is costing
    us money and eventually, customers.
    
    It gets back to the same old problem, that being management.     

    David,

    I can not totally agree that it is a "management problem".  If the
    developer(s) allows the software to go out before it is ready, then it is
    their problem too!   Time is needed to test and should not be reduced just
    because a deadline will be missed.  John, in .2, has the right attitude; he
    is responsible for his software, before, during, and after the deliver to
    the customer.

    I don't work with the main Digital engineering groups but I have worked
    with many different programming organizations at different companies.  One
    of the things that I know from this experience is that most developers are
    focused on solving the problems.  Good software needs a second group that
    is focused on the customer.  The second group must think and act for the
    customer so that the problems are found by this group before the customer
    finds them.  That job at Digital is left up to the programming groups to
    do.  Some do it better than others.  Some assign resources (both time and
    equipment) to this function, others do not.  The Phase Review process is
    one way that Digital has come up with so that the software development will
    have all of the necessary steps.  But the effort will fail if the Phases
    are just another to-do item in the development of the software.

    The management problem I see is that not everyone who is developing
    software understands that the real software development requires more than
    just a piece of software.  The solution is training on the real process.
    The solution is NOT more reports, procedures, etc.

    IMHO,
    Lee G.

RE: .0

The short form answer (for those who don't want to read all of what follows)
to .0's question of "what to do" is TELL THE ENGINEERS WHO WORK ON THE PRODUCT.
No engineer that I know of intentionally sets out to produce poor quality
products that won't install properly.  However, we can't possibly test
everything under all possible conditions and configurations.  When we find a
new case where an installation procedure fails, we can add it to the set of
configurations that we do test installations on, so that it doesn't happen
again, but ONLY IF WE HEAR ABOUT IT.

Long form answer follows.


>	On one particular product, we've had customers call in and
>	ask, "I've just received version N.N of product XYZ. Should
>	I bother to install it ?"
>
>	The product in question has failed to even INSTALL correctly
>	for the last 3 releases.

Did you or the customer ever report any of these installation problems back to
the engineering group for the product?  All of our programming tools products
are test-installed both by the development group and SQM before they are sent to
software manufacturing (the SSB).  What most likely has happened here is that
there is some aspect of the customer's hardware/software configuration that is
causing the installation to fail.  SQM and the development groups test install
on several different representative configurations, but it is impossible to test
them all.  The engineering group for the product should be told about this
case so that they can figure out what the causative factor is, fix the problem,
and make sure that the installation procedure is tested against that
configuration in the future.  Engineers are not mind readers.  They cannot fix
problems of this sort unless they hear about them.

>       One product COULD NOT HAVE EVER BEEN
>	INSTALLED CORRECTLY, it was so "messed up" on the
>	distribution tape.

SQM supposedly exists to prevent Engineering from sending kits to Manufacturing
if they have this problem.  The SSB, in turn, sends sample production media
back to SQM and Engineering to verify that the distribution tapes being
manufactured are the same as the kit that was submitted.  I know of one product
(that shall remain nameless, to protect the guilty) that had this problem and
in fact it was this product that resulted in the present procedures being put
in place so that it couldn't happen again.  That incident happened in 1984.
If the case you're citing has happened since then, it means that there's been
a breakdown in the quality control system somewhere.  I suggest starting by
alerting the SSB to the problem.  It may be an isolated case of bad media.
If it is a case of engineering submitting a bad kit, SSB and SQM can see to it
that it doesn't happen again.

>	We have products that use optimization during compilation,
>	and often the answer to the customers' problems is simply
>	"Compile without the optimizer."  

"Compile without the optimizer" is a perfectly valid response, AS A WORKAROUND
TO THE PROBLEM AT HAND.  In SDT, at least, we view this ONLY as a workaround
and NEVER as a permanent solution to the problem.  When this sort of bug is
reported and a way to reproduce it is provided, it is fixed promptly and the
program that reproduces the bug is added to the test system to make sure that
the bug stays fixed.  The key, once again, is to make sure that the development
group finds out about all such cases.

>       One product introduced
>	new in-line optimization in a recent release ( before ironing
>	out the problems in the OLD optimizer ).  This new
>	feature often doesn't follow the rules stated for it,
>	"inlining" functions that it's not supposed to.

So tell the development group.  Whining in NOTES conferences accomplishes
nothing.

--PSW

re: .8  Tell the development group...

This was the wrong day for me to read this.  I've been trying all afternoon
to install a product.  Something between the "Files are being moved to ..."
and the start of the IVP has DCL errors in it.  However, the installation
continues, the IVP generates errors, but neither KITINSTAL, VMSINSTAL, or the
IVP complain.  Of course, the product startup file dies.

I called Colorado and told them what I think the product is.  The release
notes call it one thing, the installation guide calls it another.  I got a
call back from Colorado and the person calling can't help me because he doesn't
know either of the products I think it may be.  I get sent back to the call
screening folks who then send me to Atlanta.  Atlanta calls back and doesn't
know anything about the product, but between us, we think we have found the
product's real name.  I get put into the queue for what we think is the proper
team.  I'm about to go home and call back from there, since it is about 5:30
here in Dallas.

This is NOT a complaint about the support folks.   They have been trying
very hard to help me, but Paul how in the !@#$ am I supposed to tell the
developers when we can't even figure out what the product's name is???!!

Bob

RE: .9

>but Paul how in the !@#$ am I supposed to tell the
>developers when we can't even figure out what the product's name is???!!

Presumably you got the tape because you ordered a part number from the SDC.
The part number is on the tape label itself.  This ought to be enough
information to identify the product in question.

--PSW

RE: .0

Left one out:

>       One product introduced
>	new in-line optimization in a recent release ( before ironing
>	out the problems in the OLD optimizer ).  This new
>	feature often doesn't follow the rules stated for it,
>	"inlining" functions that it's not supposed to.

I know that particular product.  This sort of thing is exactly why that product
is being replaced with a complete rewrite, from the ground up, using more
recent (and we hope more reliable and maintainable) compiler technology.

--PSW

    re: .7 "not just a mgmnt prob, engr too...
    
    ummmm -- good news, bad news... this is happening at SMARTS too. This
    exact same discussion has been going on for the better part of the last
    year, and frankly, Engineering is looking pretty good these days. Try
    making increasing functionality fit into the deliverable set by a date
    set last year which does not change.... (AKA creeping featur-itus).
    
    PS: SMARTS = big project, done on-site, in the field.
    
    Bruce Evans (who sympathizes, but has to go back to work now...)

    Re: the compiler optimizer that doesn't work:
    
    Why spare us?  I, for one, want to know what comiler is emitting bogus
    code so that I know what to avoid (probably not the language itself,
    just the particular construct.)   Please tell us what is doing this.
    It'll save lots of people some grief.

re .10

Tape?  What tape?  I'm a system manager at an ACT.  We pull all of our kits
over the net.  The tapes usually show up a few months after the customers get
theirs.

I guess this is drifting too far from the purpose of this conference, so I'll
stop here.

Bob

    about the Time to Market vs Quality and Testing conflict...
    
    I just recently was exposed to some information that may illuminate
    the problem.
    
    It seems that most of us have studied enough about CPM and PERT
    to use these ideas, at least informally, in planning our projects,
    including software development.  WELL, THESE METHODS WILL ALWAYS
    GIVE AN UNDERESTIMATE OF THE TIME NEEDED TO COMPLETE A PROJECT,
    USUALLY BY 40% TO 70%.
    
    According to the literature of the past 20 odd years, any of the
    popular methods like CPM, PERT, PPERT and so on, are biased on the
    low side when they try to calculate the duration of an entire project.
    Since academicians talk to academicians and engineers talk to
    engineers, I am not surprised that this critical fact has not been
    recognized, but its time to recognize that YOU NEED TO PUT A SIZABLE
    HEDGE in your project plan.
    
    The Project Management Facility has incorporated Monte Carlo simulation
    to provide better estimates.  That is probably a more politically
    correct solution, but make sure you plan enough time up front. 
    I am convinced that we systematically see all kinds of projects
    (not just software development) come in late due to this common
    planning failure.  DON'T BE A VICTIM.
    
    Regards,
    
    Dick

    
    re. .8
    
>The short form answer (for those who don't want to read all of what follows)
>to .0's question of "what to do" is TELL THE ENGINEERS WHO WORK ON THE PRODUCT.
    
    I just can't help responding to this one (and I'll probably hate 
    myself later for it :-}).  I avoid sw engineers at all costs.  I 
    have been called nasty names for trying to talk to an engineer
    about a customer problem.  IMHO, Only those with extremely thick skins 
    should ever *consider* attempting to speak with an engineer about a 'bug' 
    in their product.
    
    jc
                     
    
    
    

	RE: Telling the engineers

	I know engineers who will bit off peoples heads for bothering them.
	I know still more who will help when ever they can. Unfortunatly *they*
	often have managers who will jump on them if they spend too much time
	on the phone. Support is the CSCs job they'll be told. The answer is
	that "telling the engineers" doesn't just mean call them up or send
	mail. Telling the engineers is what SPRs and QARs are there for. IN
	fact this is often the best way to tell them because there is a record
	and people's reviews often depend on resolving SPRs and QARs.

	RE: What's the product in a net kit.

	One assumes one remembers where one copied the kit from. Mail to the
	system manager on that node may provide a pointer to a project leader.

			Alfred

    It's interesting to observe how topics like this cause comments like
    "it's a management problem," and "no, it's an engineering problem."
    As if the two are totally unrelated!  No, I'm not just warming up to
    the "we're on the same team" cliche.

    Why do we view management and engineering/technical work as two
    distinct, non-overlapping functions performed by different people?
    I hope we're not going to take after certain auto manufacturers...

    The problem is a management problem, in the sense that lack of testing
    time is the result of poor management of time, by ALL involved.  The
    "engineers" need to manage this resource as part of their management
    of the compromises which result in a product.  The "managers" need to
    understand the engineering criteria which are leading their individual
    contributors to jump up and down and scream "more time, more time!"

    I take what I was going to say back.  It's not a management problem
    at all... It's a breakdown of communications.  But we ALL own the
    problem.  And we all have to work toward the solution.  "Engineers"
    need to respect the constraints "managers" are working within, and
    give them proper weight in their design (did you ever think of the
    schedule as being designed?).  "Managers" need to respect the expertise
    of their engineers, and believe them when they say they need time for
    feature x or testing of feature x.

    Engineering is a big compromise.  Rigid bodies bend, elastic bodies
    break, compilers wait until code freeze to act weird.  We don't have
    infinite money, ideal materials, or unlimited computing resources.
    And we certainly don't have enough time to make up for the other areas
    we need to compromise on.  As engineers, we should be stepping up to
    the challenge of time management, rather than trying to say that it is
    up to "management" to deal with that one resource.

    Sorry, this turned into a flame and I'm not in an editor...
    Pretend there was a "flame on" at the top.

	re:  Many previous...

	My concern in posing .0 was not to present a point-by-point,
	product-by-product gripe list.

	I am concerned that the total picture seems to indicate
	poor product quality control.  Each of the problems I mentioned
	may have a solution, or a "reason".  Taken individually,
	they may be minor.

	Collectively, they are a nightmare.  Not just for CSCs ( though
	one recent poorly-produced patch is generating 20-30 calls/day
	for our group ).  What happens when a DEC salesman tries
	to go into an existing account and sell new stuff ?  Does
	he do battle against IBM, HP, Wang, Sun AND Digital's
	software QA system ?

	Perhaps we ( Digital ) need to re-work the Software Engineering
	side's funding so that some reasonable percentage of people
	are dedicated purely to maintenance, not new features, etc.

	It might be good to do more "real-world" field testing....with
	some kind of assurance that a given field-test site would
	actually "beat-up" the product....

	But, most important, I think that some level of accountability
	ought to be enforced.  If someone lets a piece of junk ship,
	repeatedly, then he/she ought to be called before 
	management at a high level, and be asked to justify what
	happened.  A nice, long conversation with one of those
	disenchanted customers ( "Should I bother to install it?" )
	might go a long way toward bringing reality into the picture.

	Steve H

    This whole mess is basically a Corporate problem.  I have observed the
    product development process (software and hardware) for over three
    years now directly.  I have NEVER seen a project plan with realistic
    timeframes, budgets or deliverables.  Why??...it's simple, if you tell
    the truth, your project doesn't get funded.  This is what we have to
    change.  Product managers must be held accountable for the outcome of
    their plans, or they should find themselves out in the parking lot
    wondering what happened.  Managers who force plans which are
    deliberately distorted or misleading should join them in the parking
    lot.  I personnally have seen millions wasted on turkey projects.

    	Re: Many replies-
    
    	I'm inclined to agree that it's not any one group's issue, but it
    	is a major issue. Now, since hardware is no longer a major concern,
    	the industry is turning to software and saying, "We want more, we
    	want it now, and we want it to do what we buy it to do.".
    
    	Management may hold a bit of blame, for pushing due dates,
        allocating one code writer when three are neeeded, etc.
    
    	The individual software people may hold some blame, by not
        programming carefully enough to keep out those nasty bugs.
    
    	The customer may hold some blame, for not telling the vendor
    	what the needs are.
    
    	The vendor (DEC, in this case) may hold some blame for not doing 
     	a thorough investigation of what the customer(s) DO (DOES) want.
    	(A note some time ago talked about QFD, Quality Factors Deployment.
    	 Personal opinion: EVERY person in Sales, Support, etc. should
    	 learn QFD as a major part of training.)
    
    	As far as the customer(s) is (are) concerned, receipt of software
    	that has bugs or does not fulfill the needs they have, is
        defective.  I've fought for better than 15 years in manufacturing 
    	to try to improve the hardware quality. I'm willing to bet that 
    	software will be even harder, since ALL programmers I've met don't
    	want someone "snooping" in their software....
    
    	Bill

	We definitely have a problem with software quality at Digital.
	Here are some of the issues I see contributing to it:

	Increased competition increases time-to-market pressure and
	reduces profit, so there's more pressure to produce products
	quickly but less money available to produce them with.
	Design and testing seem to be the first thing to suffer
	when the schedule gets tight.

	Lack of modern software management and development techniques
	within Digital. The tools exist now to make us more productive,
	so we can produce competitive (high-quality) products in a
	competitive timeframe at a competitive cost. Unfortunately,
	they require an upfront investment in time and money, which
	individual groups find difficult to make while trying to make
	their existing schedules, and which is not being pushed from
	the upper levels of the company.

	Someone else mentioned supporting multiple platforms for our
	products (and implied that UNIX is to be "blamed" for this).
	Well, like or not, we do need to do more in the UNIX
	marketplace. And it's not the only platform of importance
	(quick, what operating system is on by far the most
	desktops? it ain't UNIX *or* VMS...). Supporting multiple
	platforms isn't a problem - failing to plan on it early in
	life-cycle and properly account for the complications it
	introduces is.

	Trying to do too much in the initial version of a new
	product. I recently wrote a memo on the dangers of this
	in the context of one project - it seemed to strike a
	chord with some people, so I generalized it - I'll post
	it in the next reply. A long-time problem Digital has had,
	which I think is really hurting us now in the quality and
	time-to-market departments, is not wanting to release the
	first version of a product until it does everything we can
	imagine it doing. We've got to curb our appetites...

	Integration with other components. Few software products are
	completely independent - in today's world integration on
	many levels is becoming more and more important. This adds
	complexity to the development process that is *never* fully
	accounted for in planning. This is particularly painful
	when separate but inter-dependent products are in development
	simultaneously (do the words "incompatible baselevel" mean
	anything to you? yes, I can tell by the sound of your
	screaming they do:-). We need better ways to manage
	integration problems.

	Related to a couple of points above is not involving the
	customer enough (or early enough). A lot of engineering time
	is wasted when we design and implement a product in relative
	isolation, then we spring it on field test customers and find
	out it doesn't meet their needs. They come up with requirements
	they consider critical, but by golly all the engineering time
	has already been allocated to all the nifty little features we
	thought you'd like but you'll actually use once in a blue moon.
	Oh well, we'll add it in V2 (if we ever find the time in between
	fixing all the bugs you found that got by our less-than-
	extensive testing...). This can all be avoided by doing
	a better job of gathering customer requirements in the first
	place (as opposed to sitting in our cubicles trying to think
	of what customers would like), by generating early prototypes
	and showing them to customers, by watching customers use
	field test versions of our products in their daily work
	(contextual inquiry), etc.

	Yes, it doesn't help anything if Digital people know of software
	problems but don't report them. But we in Engineering have to
	be aggressive in gathering problem reports - we can't just
	sit back and wait for customers to come to us. The customers
	seem to have little faith in the SPR mechanism (see the previous
	note on this topic) - Digital needs to do whatever is necessary
	to fix the process so that customers can be assured of having
	responses quickly. That may require changes in the SPR
	adminstration - it also requires that Engineering drop everything
	and deal with problems quickly before they snowball. Better
	yet, of course, is to make the products better quality so
	the customers have less problems to report.

	What is the story on SIX-SIGMA for software, anyway? I thought
	there was supposed to be a big push for quality from the top
	down, but it hasn't filtered down to the trenches as near as
	I can tell. Some of us down here are doing what we can, but
	it's not going to work without the management commitment.

	BTW, I'd recommend the course "Developing Software in a Complex
	Environment" to any engineer or engineering manager - it
	addresses a lot of the problems brought up in this topic.

	I've rambled a bit here and I don't have time to organize
	this into a formal paper (gotta get back to work and make
	sure my new project will be high-quality:-), but I'll point
	out a theme or two I've mentioned. Corporate cost-cutting is
	I think a major thing that's triggered the decrease in software
	quality recently. My personal feeling is that the slippage in
	the market would have been a great opportunity for Digital
	(having an advantage over most of our competitors in terms
	of solid market share and cash reserves) to invest the money
	to improve our productivity and quality, putting us in a
	better position to compete in the sure-to-be-hot market when
	the economic picture brightens. Instead, we've settled for
	doing the things Wall Street thought we should do... Anyway,
	the response of Engineering to decreasing budgets has tended
	to be to reduce design and testing. It's not even an explicit
	"well, let's not bother testing that", it's more a case of
	individual engineers under greater pressure to produce more
	code slighting the testing of their code (I've fallen prey to
	this myself). The proper response of engineering management
	and individual engineers should be to say "let the schedule
	slip, drop some functionality, but the quality work will not
	be compromised".

	Well, enough for now...

	Mike

    I think it's important to agree, before the specification of a new
    product is completed, on the importance of functionality versus quality
    and time-to-market for the initial release of the product. Based on my
    experiences, I believe that it is very important that the initial
    release of any product stick to only the most basic, essential
    features, and concentrate on the quality (reliability, usability,
    performance, portability) of the design and implementation, while
    designing for growth and maintainability. Some of my reasoning follows:

    1. We won't fully understand what the users (end-users and application
    programmers) of the product really need until they start using it in
    their day-to-day work. Past experience shows that many of the features
    which make sense to the developers on paper don't get used in practice,
    while the users will come up with important requirements during field
    test (which tend to lose out to the features appearing in the original
    spec). 

    2. Functionality becomes a trade-off with quality. When resources and
    schedule are constrained, the time to implement additional
    functionality is going to come out of design and testing time. Quality
    will suffer, which means user perceptions suffer, and for subsequent
    releases more time will need to be spent fixing problems leaving less
    for addressing new requirements coming from real usage (i.e., the
    really important requirements).

    3. Functionality becomes a trade-off with time-to-market. More
    functionality adds risk to meeting the schedule. Adding a new feature
    adds more time to the schedule than just the time to design and code
    it. It adds to maintenance, to overall complexity, and generally
    reduces performance.

    4. Simplicity is goodness. For products consisting of libraries
    (widgets, API's, etc...), a major goal is to get other applications to
    buy into using these services, but with limited resources they'll be
    unwilling to invest a great deal of time in them. It is important to
    make it easy for other application developers to simply plug in your
    product. Widget and API documentation detailing dozens of potential
    arguments, most of them for features that 99% of application developers
    would never think of using, will tend to discourage use of your
    services. Likewise, providing users with knobs they never use
    complicates the user interface, and makes the documentation more
    intimidating.

    It *is* important to gather all anticipated requirements initially, and
    plan to explicitly address all requirements (V1 and post-V1) in the
    design, to plan for the future. It is equally important to understand
    which features are truly essential for V1 of the product, and identify
    other important features as "if-we-have-enough-time" (with specific
    metrics for what constitutes "enough-time").

    I feel that by holding to a minimum of functionality in V1 and using
    the time to carefully design for quality and growth, we will have a V2
    product that will better meet users' real needs and be of better
    quality in not much more time than a V1 product that attempts to
    satisfy all anticipated user requirements.

Has anyone else been on a DEC project where the software is not working
at all, each successive layer of management is presenting a
rosier and roser picture of the situation, until finally you reach
a management layer that is so completely removed from reality that
they are trying to solicit customer testimonials on the softare?

    >To my mind, being an individual contributor myself, it is the
    >individual contributor who takes the major portion of the blame.
    >There is no excuse for "giving in" to management pressure to
    >shorten a schedule.  
    
    	Nice words, but it doesn't work.  Management have more
    	clout than engineers, and unless (like you and I) you've
    	been around here for quite a while, I submit it's very difficult 
    	to win a schedule argument with the person who controls
    	your pay packet.
                            
    	That's particularly true in my organization, where we have
    	many small projects, and the project leader is often identical 
    	with the entire project development team (and consequently, is
    	relatively junior). 
    

    re: .27
    
    I don't argue about schedules.  I produce to _my_ schedule, no matter
    what other people would like me to do.  If that isn't good enough,
    they can fire me.  In practice that hasn't been a problem, but if it
    were I'm sure I could find another job.  Particularly if you've had
    experience with DEC products, there are jobs waiting for you among
    customers.
    
    re: .28
    
    Agreeing to develop a prototype is a potential trap.  Making such an
    agreement implies a lot of trust in your management not to misuse the
    prototype.  I've worked for a lot of good people in my years in the
    profession, but I don't think I've worked for anyone I'd trust that
    much.
        John Sauter

	All good points, but I believe that a major problem with software
	quality is that customer's expectations are not set by marketing
	and sales.  I know of several products that were initially 
	midnight hacks that someone said "gee, that's really neat, can we
	sell it?" and out the door it went.  It is just amazing to some
	of the people who developed these things to hear what customers
	are using these 'tools' for.  (Ever wonder about missile tracking
	systems, tank simulations, military design?...)

	What we should do (and is already done to some degree in the U*x 
	world) is have levels of software 'quality'.  To some degree this 
	is already done with the products on ASSETS.  The levels would be 
	something like:

		- full support - things like VMS, compilers.  These are
				 products that have undergone EXTENSIVE
				 testing and have robust support mechanisms.

		- limited support - things like the stuff in ASSETS. It's
				 nice, it works most of the time - if it
				doesn't, we'll take a look at it (for a
				price). Not as much testing - and we	
				ADMIT it.

		- no support -  here goes. Have some source code and fix it,
				modify it, don't call us.  Let's distribute
				shareware - maybe somebody will come back
				for the 'real' stuff.

	Why would anyone want the latter two you ask?  Price for one - they
	should be *much* cheaper.  Possibly you could *upgrade* a product
	in support levels as it matures.  Another reason with number 3 is
	that you could fix it yourself - you are not tied to the 'fixed 
	in future release' syndrome.

	Let's face it = software systems are increasing in complexity and
	we aren't putting the bucks in the quality control loop.  Let's
	be honest with our customers and let THEM decide the level of
	quality that they want.  It is not fair to sell a customer a
	software package that consists of 2 really solid products and
	a couple of 'hacks' thrown in by marketing.  The overall perception
	of the package suffers and the customer is left with possibly
	unusable code and the CSC's with a support nightmare.  In the
	above case, why not just GIVE the customer the code to the hacks
	and say "have fun with this - understand it is unsupported at
	the present time, but we are evaluating getting it to the 
	level of support that you require".

	My 2 cents...

	Rick Beldin
	Alpharetta CSC
	

	

    Perhaps the "freeware" concept could be piloted in the universities
    or with our own employees who like to hack as a means of learning.
    I have never wanted to accept the discipline that professional
    programming for sale implies, that's why I am a consultant.  But
    I learn a lot everytime I try to do some project for personal use.
    Sharing such "NOT FOR SALE" ware can be useful, but lets not confuse
    it with professional programming.  Perhaps the main reason for the
    misuse of prototypes, ASSETS, and so on, is a failure to communicate
    the difference between professional software and hacks to the marketing
    and sales community.  The only solution I can see is to turn all
    programmers into entrepreneurs, contractors who can release or not
    their code without the (well-meaning, but mis-informed) interference
    of anyone else.  That's not a route for everyone, but if you feel
    strongly that you don't have the independence you require
    professionally, that may be your only solution.
    
    Dick Beldin
    Senior Systems Consultant
    Caribbean Operations Manufacturing
    

    
    I cannot believe how readily you all accepted the base premise as
    true!!
    
    Where is your pride? How come none of you developers stood up and told
    the base noter that the quality of your product had NOT deteriorated?
    Is your morale so low that you are ready to roll over and die?!?
    
    We cant possibly WIN if you have no pride.
    
    
    Even if I have made you angry, please read on...
    
    There is always room for improvement, but be careful not to
    take the subjective statements of others at face value.
    
    The base assumption here, that software quality has indeed gotten
    worse, seems to be subjective. SQM for VMS layered products has
    always been tough and quite thorough, and SQM for ULTRIX layered
    products is getting tougher by the month.
    
    Before we assume this is actually true, could someone from
    CSS possibly post some SPR/PRSM/CLD stats here, comparing
    this year to the past three or four years? 
    
    I *DO* know that in our group, we have placed a great deal of emphasis
    on design, specifications, CASE tool use, in house QA, and recently
    have begun a six sigma program. The end result has been a steady
    improvement in quality over the past 2 years. A case in point is
    a real time component we designed and built two years ago. 20,000
    lines of real time C (VAXELNC) code, and we have generated only
    *3* valid QARS since it shipped. (July 89)

    
    Taking pride in one's work will in no way address the specific
    problem that a software product has allegedly been uninstallable for
    three consecutive releases. Neither will examining statistics on SPR
    rates, whether they have been going up or down.
    
    Just as you can listen to lots of negative comments and be
    convinced that Digital sells software sludge, you can listen to lots
    of positive comments and be convinced that we sell the greatest
    products since Tinker Toys. Reality is probably somewhere in the
    middle.
    
    .0 has an excellent opportunity to improve the quality of Digital's
    software. Direct, concrete steps to do so will increase the value of
    this company. Anything else is unproductive rhetoric.
    

    re: .32
    
    I accept the base note as true, even though I have not had personal
    experience with the product, because I've seen similar cases.
    
    I applaud your group's attention to quality.  I wish all software
    development groups in DEC would follow your lead.  I hope the next
    reorganization won't leave you with a management structure that
    dismantles your quality consciousness in favor of "time to market".
        John Sauter

    re:  .32
    
    Pride aside, statistics be-damned, charts and tables yawned-
    
    	PERCEPTION (IN THE CUSTOMER'S MIND) IS REALITY !!
    
    -Barry-
    

    Better quality yields lower costs AND shorter development cycle times.
    If you do it right the first time, you don't have to do it over or fix
    it later.  Please stop asserting the contrary.  

    It is true that it requires making changes in how time is allocated --
    primarily in doing a better job of initial understanding what the needs
    of the REAL customers are (the ones who pay money, not marketing, not
    manufacturing, not sales, not service, not management, not any of the
    so-called "internal customers"). Having the faith that changing the way
    we do work takes some courage, and that is unfortunately not a common
    commodity.  People also have to be allowed to make mistakes -- no one 
    is omnipotent or omniscient.  There is no crime in making an honest
    mistake - the only crime is in not doing something about is so that the
    next person doesn't stumble on the same mistake.

re: .36

  << 					... understanding what the needs
  <<  of the REAL customers are (the ones who pay money, not ...
  <<							...  not any of the
  <<  so-called "internal customers")

Whoa, Walt, I think you've gone a bit too far.

There are parts of Digital that use Digital software to run their business
and just because they pay in transfer cost instead of hard currency doesn't
make them any less a real customer - with real business requirements.  The 
very real needs of internal customers are often far more sophisticated than 
the needs of the rest of the world. And I think that's an important factor 
in the high quality of Digital software.

	/Marvin

    Two of the projects that I manage are shipping source code to other
    vendors including HP and IBM.  I can hardly count the number of
    comments I have received over the past year from managers and engineers
    at HP and IBM regarding the high quality of the code we ship them.
    
    My engineers take quality VERY seriously and based on the quality of
    the code we are seeing from the other vendors we are working with, I
    believe we have little to worry about.
    
    Sorry, but I can't help but brag about these guys!

    From Seybold "Report on Desktop Publishing"
    
    " Apple sponsored a daring and effective demonstration of digital
    technology for the Macintosh. It filled a large room with Macs and
    a variety of multimedia applicatons, and then left them unattended for
    visitors to try themselves. In general , the demonstrations proved that 
    well-written mass-market software does not need much explanation.
    People will accept a digital world if it comes to them on their own
    terms and interacts with them in their own language and according to
    their expectations. When the products behave as we expect, we
    immediately respond,"
    
    Can we honestly say our products have reached this level of quality?
    
    Regards,
    	Phil

Re: .32

>    I *DO* know that in our group, we have placed a great deal of emphasis
>    on design, specifications, CASE tool use, in house QA, and recently
>    have begun a six sigma program. The end result has been a steady
>    improvement in quality over the past 2 years. A case in point is
>    a real time component we designed and built two years ago. 20,000
>    lines of real time C (VAXELNC) code, and we have generated only
>    *3* valid QARS since it shipped. (July 89)

That's nice, Rick, but here's what's been going on upstairs.  In '87 my
group took on the job of writing a terminal emulator for DECwindows, to
ship in a year and a half.  We only got half the people we asked for and
we had no control over the schedule (except potentially we could delay the
entire DECwindows program if we *really* screwed up).

Sure, we wrote and reviewed design specs; we knew exactly what we wanted to
write.  The problem was that there just weren't enough people and there wasn't
enough time to make everything work, even when we started to cut out features.
Still, we stayed on schedule.  We shipped with DECwindows V1 in December '88,
and management said "OK, now you're finished" and broke up the project team,
leaving basically just one developer (me) to write DECterm V2.

Well, I did my best.  There was a very large demand for new features, so
I spent a month or two collecting phase 0 requirements, writing specs and
getting people to review them.  I had to cut out some of these features
when it became clear that I couldn't implement them all in time for the
DECwindows functional code freeze.  I did manage to get the most important
stuff done in time -- at the cost of working nights and weekends for several
months.

Having gotten the new functionality squared away, it was time to work on
Quality.  By my count there were 300 QARs in about a dozen QAR databases
when I started DECterm V2, and 200 when I got pulled off the project.  That's
pretty good when you consider that there were more new QARs filed against
DECterm than against any other DECwindows component.  (The actual number
of bugs was less than the number of QARs because problem reports were often
duplicated in different databases).

Things got worse.  Our quality group was pulled off the project.  I was
press-ganged for a few months to work on another project, and two engineers
from another group worked on DECterm while I was gone (and for a while after I
got back).  Of course they didn't have much time to fix bugs, because they each
had to add a major new feature.  When I got back on the project I had to write
yet another new feature.  There's been hardly any time to tackle that QAR
backlog, which I'm afraid to even count.  (Suffice it to say that DECterm
has, as usual, more open QARs than any other component of DECwindows.)

I laugh at QARs now.  I don't have time to work on them because I'm too
busy working on CLDs.  I try to squeeze in time for things like acknowledging
QARs, attending project meetings, installing the latest baselevel to make
sure the program still more or less works.  I keep getting calls from the
field, and I've started to answer them with "have the customer submit an SPR or
CLD".  In other words, please don't bother me, I'm busy enough as it is.

Well, I've finally had it and I'm leaving VIPS.  I can't complain about my
performance reviews, but it's just too much stress.

So what was our mistake?  Maybe we could have "worked smarter" (but how?).
Maybe we could have refused to add new features (but DECterm is a widely
used program and it seems like just about everyone has ideas for how it
can be improved).

It seems to me that our biggest problems were: THERE WEREN'T ENOUGH PEOPLE.
THERE WASN'T ENOUGH TIME.

If I could have forseen all of this in '87 then *maybe* I could have followed
John Sauter's advice and said "Sorry, but it can't be done".  Of course no
one would have listened to me, especially since there's no way I could have
*proved* that it couldn't be done.

Maybe the skills I need to develop are the ability to forecast how much time
and how many people it takes to develop a given piece of software, and the
ability to push back on management based on this knowledge.  These aren't
easy skills to develop, but I'll give it a shot.

				-- Bob

    	... that I think you have all the credentials to fill:
    
    
    			Ferry's!       :>)  :>)  :>)

RE: .43

>Refuse the project.  Tell the budgetary people it's impossible and they'll
>have to do something about it.  In fact, I would NOT recommend asking
>for more people, but for more time and less functionality.  (Re-read
>Brook's book, "The Mythical Man-Month.")

The problem is that you can always find someone who won't refuse the project.
Given a set of impossible conditions you will always find some manager
who will say "I can do it.", will put together a schloch job, "manage"
the schedule (Does you boss really know the difference between coding
and testing?), become a hero, leave the project before it's over.  Meanwhile 
someone else cleans up the mess.

    
    Re: the many references to quality and Six Sigma
    
    There's been lots said here that I both agree and disagree with,
    but no matter.  There is much we need to do to fix the many problems
    outlined here, but before we do that there needs to be some fundamental
    change in the way we develop ALL things we sell to customers not just
    software.
    
    First, like Motorola and others who have implemented Six Sigma
    like programs, we must understand who defines what is meant by quality.
    The customer defines quality. period.  They let you know what their 
    definition is by voting with their purses.  If your product does what
    they need at a price they can afford they will buy it.  If it doesn't
    they won't.  It really IS that simple.
    
    Second, once we all understand point one then we must manage everything
    we do with the goal of total customer satisfaction.  We should NOT
    manage against schedules, ship dates, revenue, or anything else unless
    those things matter to our customers.
    
    Third, if we do points one and two all the other stuff falls into
    place.  Several notes here have said we don't listen to our customers.
    Those noters are right, we don't and the many complaints here are proof
    of that.  You can say what you want about .0's analysis of what is
    wrong and what should be done, but that is missing the point of .0
    The real point is that we have unhappy customers calling our CSCs and
    calling frequently and in large numbers.  If we don't start listening
    to that and soon, after awhile it won't matter.
    
    There are some companies out there on the leading edge of quality i.e.
    Motorola, winner of the 1988 Malcolm Baldridge Award, and Xerox, winner
    of the 1989 Malcolm Baldridge Award.  In the late 70's, Xerox, for
    example, was almost pushed out of the copier market, that they pretty
    much created remember.  They decided to do something about it and in
    1981 embarked on a program very similar to the one adopted by Motorola.
    I'll share a brief true story which illustrates the new Xerox
    philosophy.  A manager I know in Lou Gaviglia's organization was at
    a function where he happened to be sitting at dinner between the
    Digital account manager (AM) for the Xerox account on his right and the
    Xerox account manager for the Digital account on his left.  Seated to the
    left of the Xerox guy was the Xerox VP in charge of customer
    satisfaction.  This DEX mangager asked the Xerox AM what happened if he
    didn't meet his budget (we all know what happens to DEC AMs who don't
    meet their budgets, don't we.).  The Xerox AM first didn't understand
    why he would be asked such a question, but did answer this way:  In my
    last position here at Xerox, I only made 49% of my budget, but I was
    promoted to this position as DEC account manager (a real plum for him)
    and given a big raise.  When asked why he would get such a reward for
    only making 49% of budget he said that every year Xerox does a customer
    satisfaction survey with every account and that account's rating of
    Xerox on customer satisfaction gave him the highest rating in Xerox on
    customer satisfaction.  At this point the Xerox VP who had been
    listening chimed in with: He did a helluva job.  We are much more
    concerned with our relationship with an account that how much revenue
    it is currently generating.  We know that if the relationship is right
    that when they have money to spend, they'll spend it with us.
    
    At this point the DEC manager turned to his right and asked the DEC
    AM if he heard all that.  His reply: Yes, I heard it and I'd love to
    operate like that, but this month I'm sweating bullets over a purchase
    order I want them to sign!
    
    When we learn to implement our own version of a philosophy that has
    us operating more like Xerox does then we'll be implementing metrics
    and processes that will make problems like those described in .0 go
    away.
    
    Steve
    

	...face to face with the developer.  Speaking of Tomas Lofgren, I
    can no longer resist relating this historical essay which also happens
    to relate to the subject matter at hand.

	It was the summer of 1987 and I was working back in MR01 supporting
    the printing and plotting needs of the High Performance Systems (HPS) group,
    when I ended-up helping out on a few marketing demo projects as a sideline
    to my job.  As one of the developers of the demos, I was fortunate enough,
    to be allowed to attend the DECWORLD '87 event later that summer (to make
    sure the demos didn't crash).  Among these demos, perhaps the more useful
    one was known as the "Monster Monitor", as it allowed various system
    statistics from all eight VAXen in the newly announced 8978 VAXcluster to
    be displayed simultaneusly on a single screen from a remote workstation.
    The Monster Monitor demos (or "MM" for short) were also running directly
    across from the VCS (VAXcluster Console System) demo in the center of the
    DECWORLD display floor and together made-up the System Management demo area.

	The VCS product was managed by none other than our friend Tomas Lofgren.
    I must admit that I was a bit put off by Tomas at first, because he didn't
    think much of our demo software and he felt that VCS was vastly superior
    from a technical standpoint.  I had to agree with many of his points, as
    the demo software was not nearly as robust in certain respects (after all,
    VCS was written by a real product team, with real schedules, and real
    funding, etc., while our demos were whipped-up at nights and on week-ends
    during the two available months prior to the DECWORLD event - and how could
    something built so quickly and cheaply be worth anything to anyone?).

	While at DECWORLD '87, I was able to talk to a good deal of potential
    DEC customers and I managed to gain an unbelieveable wealth of customer
    input (probably 1000 times as much customer interaction as I had in all of
    my previous years at DEC combined).  During the 3 weeks, I began to notice a
    pattern where a customer would walk up and look at our "MM" demos, then turn
    to look briefly at VCS, squinting to read the console event text, then
    turning back to our demos once again.  Sooner or later, the customer would
    begin to ask questions about the MM demo software, and ultimately they
    would go looking for their DEC sales rep for a price quote on the software!
    Unfortunately, the MM demos weren't for sale, but after watching this happen
    enough times, I vowed to eventually develop and deliver these demos as a
    product.  Over time, Tomas Lofgren soon admitted that perhaps the demo
    software did have some value after all (given the customer reactions).
    By the end of DECWORLD, we had accumulated a list of some 50 different
    corporations (as well as governments, etc.) who would be interested in
    purchasing whatever we had in terms of the MM software "as is" (and who
    would definately be interested in the real thing if it were productized).

	Shortly after DECWORLD '87, I began trying to drum up support for a
    project team to productize the "Monster Monitor".  Despite repeated efforts
    on my part, as well as a marketing fellow who I had been working with on
    the demo project (and who subsequently left DEC for a job at Stratus out
    of general disgust), we could not get any real support for such a project
    team.  Ironically, it was Tomas Lofgren who (totally unrelated to my actual
    management chain) ultimately provided me with the moral support and
    encouragement I needed to go it alone and productize the demo software
    anyway.  During that fall and winter of 1987, I spent my spare time during
    the day, along with nights and weekends to totally rewrite the product
    from scratch using newer, better, and much more portable software (using
    the input I had personally collected from customers at DECWORLD).  The
    result of this effort is now available as a Network ASSET package called
    "PULSE" (newer version renamed to "DECPULSE" for legal reasons) and the
    software now does much more in a much better way than it orignally did.

	I must apologize to those of you who had heard this story before,
    but the point is that without that customer feedback, none of us would
    have had any clue that customers would have responded to anything like
    that.  What was even more inspiring from a developer's point of view,
    was to actually be able to rub shoulders with the actual customers
    face to face and find out directly what sorts of things interested them.
    There is an unbelieveable amount of body language and subtle hints of
    what desired features might and could be added to a given piece of
    software that I am convinced can only be obtained by interpersonal
    interaction with the customer.

	There is also a certain sense of responsibility and purpose that
    was instilled in me from that experience which ultimately drove me to
    develop that ASSET package on my own.  It would be nice if there was
    also a way to get a little credit out of the deal (i.e. have the PULSE
    ASSET development effort reflected in a review or something), but that
    is another issue.  The important thing is that the direct developer-to-
    customer link can work, and it can be extremely effective in producing
    software that is percieved as being valuable to the customer.  If only
    there was a way to make what was essentially a fluke in the development
    process (this case) become more of a reality for other products as well.

				    -davo

p.s.	I just shot my 16-line note size limit all to hell...oh well...

    
    re .44
    
    Does anything from the left coast ring a bell ??
    

    This seems like a good point to insert a little pitch for Quality
    Factor Development (QFD) and the "homework" it stimulates its
    participants to do beforehand, particularly Contextual Inquiry.
    
    Contextual:  developers interact *while* users work, *where* they work.
    
    Inquiry:  developers inquire together with users into both the nature
    of the work and the possibilities for software to make contributions.
    
    The Contextual Inquiry course that Karen Holtzblatt, Steve Knox, and
    Sandy Jones designed is scheduled for Sept 10 (follow-up 1/2 day 9/24)
    and for Jan. 14 (follow-up 1/2 day Jan 28.)  I believe you could
    contact any of them to register but I presume that since Sandy is in
    the SUE group in Spit Brook she's "down the hall" for many software
    developers.
    
    You can read more about QFD (and a little bit here and there about
    Contextual Inquiry) in the Notes file at metoo::QFD.  Several of the
    replies under this topic have suggested that if you get the customer's
    viewpoint clear and set your priorities accordingly, some of the tight
    situations of budget and schedule vs. the intention to produce code you
    can attach to your resume might be alleviated.  QFD isn't the universal
    solvent but it's a well proven structure for digesting the implications
    of a profound understanding of the customers.  It's been used many
    times at Digital for planning software (there's a topic at metoo::QFD
    specifically on QFD for software that gives some success stories, and
    one on the history of QFD at Digital that gives plenty of names you can
    talk to.)
    		Russ

    For all those who have contributed to or read this insightful topic,
    please refer to the paper available in topic 36.1 of the METOO::QFD
    NotesFile (which is in .PS form).
    
    This paper offers a large chunk of a solution to improving quality,
    cost metrics and time-to-market for software development projects, and
    the training is available from existing Digital courses.
    
    Regards,
    
    Ray

    re:.41 and my prior reply:
    
    My engineering group and Bob's are part of the same organization. The
    one key factor which differs between our two groups is that we report
    to different cost centers.  Apparently the impact of just one layer
    of management can make all the difference in the world!
    
    
    

Gee, thanks, Rick ;-)

Seriously, the task of improving and maintaining the quality of software is
hard work, both for the engineers involved and for the manager (I speak from
years of experience on both sides in my personal crusade for quality).  
Unfortunately, when crises hit, the tendency of both engineers and managers is 
to fall into reaction mode, which usually short-circuits the ability to 
maintain the plan.  Too often, the first activity to be sacrificed is review - 
design review, code review.  How many of you do this at all?  We do (usually), 
and when we don't, we pay the price.  We've been burned enough times now that 
I've managed to convince my boss (who does not have a software background) to 
believe that my "gloom and doom" estimates (in the words of one of my peers) 
are truly representative of reality, and are actually shorter than how long the 
resuling schedule slips become when we try to take short-cuts.  But this works 
only because we each have enough history in our current positions to actually 
see these results and create some credibility.

But Engineers have responsibilities, too.  You have to be willing to make 
realistic, believable estimates, and admit when you're wrong so that we can
do a better job next time.  You also have to be willing to allow others to
review your work - not because we're on a witch hunt (I've been on the wrong
side of those, too, unfortunately), but because through review, we can catch
several orders of magnitude more potential errors than we can through testing.

And nothing frustrates me more than the still prevalent perception that software
development is "black magic" or an "art".  Sorry, but it's Engineering, and 
Engineering implies professionalism, discipline and openness.  That's where
quality comes from, and it is possible to do this without "stifling creativity".

I'll climb off my soapbox now.

Bruce

    Hi All,

    Well, this looks like a very interesting note. I haven't read it all yet
    (that's this evenings job!) but I thought you folks might be interested
    to know that we have had a similar discussion on this side of the pond.
    There is a note (435.0) in the MARVIN::UK_DIGITAL conference that
    covers quality in general, not just software. However, most of the
    replies seemed to end up talking about software. I wonder why ;-)

    Have Fun,

    Pete Platt - Office Support CSSE.

> However, most of the
>    replies seemed to end up talking about software. I wonder why ;-)
    
    Our hardware Quality problems tend to be "hidden". That is that most
    of them are in two places. 1) Installation and Early Life Failures.
    FS does a great job of correcting these. It increaces our costs a lot
    but at least the pain is short lived for the customer. 2) Manufacturing
    where all the re-work and extra scrap is not generally seen.
    
    S/W quality problems seem to drip out in drabs when you need it to work
    the most. Our H/W end seems to have made improvements over the years
    but I'd guess that we could still save a lot of money if h/w was done
    right the first time every time.
    
    		Alfred

    re: .55
    
    Another reason for more visibility of software problems, I think,
    is our reporting mechanism.  If I find a software problem I can
    use a QAR, SPR or CLD to get it fixed.  If I find a design problem
    in the hardware (one that can't be fixed by replacing the failing
    unit) I don't know where to go.  My only option is to try to work
    around the problem.
        John Sauter

    You'll love this one!
    
    How about if we start classifying bugs in software similarly to how we
    handle ECO's and FCO's.  Only, if its a "retrofit", the programmer gets
    to install the patch personally at the (first) customer's site. 
    
    It occurs to me that this might both elevate the visibility of "bugs"
    beyond the routine and provide the direct customer contact some
    software engineers are asking for.

    Regards,
    
    Dick

Hardware is harder to manufacture, so it tends to go through more extensive
"architecture" and dsign-phase testing. If we could pop out another version
every two hours, you'd start to see the same sort of bit-rot.

If programmers wrote their code with chisels and stone tablets, we'd put
more time into the design before picking up our tools.  (I think there's
some comments along these lines in a Knuth article somewhere.)  Now, the
interesting (and only half-humourous) question is whether the total
amount of bug-free code would diminish....

Martin.

    Hardware engineers would never think of shipping their first attempt at
    a breadboard.  They simulate, breadboard, and prototype all before
    designing the "real" hardware.
    
    Software, we don't do that much.  Taking the time up front to build
    prototypes and generally get to really understand how the thing is
    going to work is not common practice in my experience.  Whenever I am
    running the project we always do explicit throw-away breadboards and
    prototypes so by the time we get around to the real thing it is the
    third attempt and there aren't any surprises.   This does *not* take
    three times as long, as the breadboard can cut LOTS of corners, and
    the prototype can cut quite a few.

    re: .56
    
    Since posting this observation I have learned that there is a problem
    reporting system for hardware design errors.  It starts at Customer
    Services (formerly Field Service) and escalates up, using the CLD
    process.  According to my informant, hardware and software CLDs are
    handled in pretty much the same way once they get to the Region.
    
    I apologize for my inaccurate statement concerning hardware error
    reporting.
    
    re: .59
    
    In some software environments you can't write a prototype or
    breadboard, since if you do you take the risk that it will get shipped
    to a customer.  In DECforms we were lucky---we had a version ready to
    field test when we were required to start over.  The data structure
    that represents a form in memory, which is the heart of the product,
    was at version 3.7 when we shipped DECforms V1.0.
        John Sauter

    I have learned how to sufficiently cripple my breadboards and
    prototypes that they are unshippable.  With breadboards you pick some
    wacko language, or the wrong platform, or write it in DCL or some such
    thing.  It is really a mock-up of the final program, and contains only
    the new functions - nothing old.  This has to be done as quickly as
    possible, so you need an environment that supports very fast
    development.  Everything is traded off in favor of fast development.
    This means almost never using a mainstream DEC platform.
    
    The prototype is harder to cripple, but I take the approach that the
    purpose of the prototype is to learn how to build something I haven't
    done before.  Therefore I don't bother to put into the prototype any
    parts of the product that I *have* done before, beyond the barest
    essentials to support the new stuff.
    
    Max time to develop the breadboard: 1 person for 1 month.
    
    Max time to develop the prototype: 3 people for 4 months.
    
    Max time to develop the shippable: 5 people for 10 months, plus one
    writer.  Any more people or time than that and you are doing too much
    for a v1, and your schedule will slip all over the place.

Hardware prototypes are usually obvious, even to managers (and I are one ;-).

Software prototypes are invisible to everyone except the implementor.

Hardware prototypes are almost never shipped, except in absolute crisis, in
which case they are planned to be replaced with production units.

Software prototypes are shipped when the calendar reaches the magic ship date. 
There is usually no plan for replacement.

(from my experience - your milage may vary)

    Software prototypes are not supposed to be seen only be the
    implementor.  They should be made available like a mini feild test.  No
    point is passing up a good opportunity to get feedback about whether
    you are on the right track, get field training started early, and so
    on.
    
    When I am running the project I set the "magic ship date", and it is
    not set until the prototype is complete.  (How can I possibly estimate
    how long it will take me to do something I've never done before?) If
    someone asks me if I can make it sooner, maybe by putting more people
    on the job, I answer "no".
    
    What I've got to learn to do is what John does, and say "no" even when
    I am not running the show.

   Way back in DIGITAL's history somebody thought that product guys are running
   away with the glory. So let's make process the most important thing.
   We ended up forgetting that process is there only to help the product go
   along a sane path, not to rule the whole affair with an iron-hand, with
   millions of bureaucrats micro-managing a product or a potential product to
   death, creating mounds of paper and nothing else. Smaller competitors
   with nimbler outfits don't have BS to contend with. They just do it and
   finally DIGITAL cancels the effort. Wonder why we are not quick with our
   software products like we used to be?

   Process has to be extremely flexible according to the type of product
   that is being produced. Developing a C compiler whose spec can be
   found at the back of a standard C text book is different from developing
   the latest Distributed Client-Server Gizmo. There has to be a stage of
   Advanced Development where it is made very clear that the output is a
   prototype that's meant more for learning the issues rather than
   a product. The product is to be produced separately in a separate effort.
   Instead our process dictates that we do architecture for zillions of
   years, producing nothing but paperwork and then scramble to put together
   spaghetti code in a short time, only to find that the small competitor
   on the West Coast has been selling a similar product for almost three
   years now.

   Prototyping in software is absolutely essential when it comes to
   products like we are trying to get out the door these days, simply due to
   the nature of the technologies used. The trick is to make the process 
   flexible enough to recognize this fact and not impose stupid requirements 
   where it is certainly out of place. The ultimate objective is to produce a
   product that people need, not to meet deadlines and certainly not to satisfy
   the Process God.

   Nari

    The following Unix shell script was being sent by Unix Support
    Engineering to various support personnel to run on customer systems.
    How did Digital come to this?  What should be done?
    
    
    				-- edp
    
    
#bin
#revision .08
#CLD Information gathering program
#revised 12-12-95 out of development into Beta Testing
#revised 5-21-96 feedback from Joe Watzko, MCS Austria
#'sort -u netstat.out >>checkit.sys' changed to 'sort -u volprint.out >>checkit.#sys'
#please send additions or corrections to cansler@guru.zk3.dec.com
#
# checkit must be run from the root account
#
echo This file must be run from the ROOT account
sleep 2
echo Please wait while I am loading. 
sleep 2
echo                                 .... 
sleep 2                              
echo                                       .....
sleep 2 
echo Thanks for waiting 
sleep 2
echo  
sleep 2 
uname -a > checkit.sys
sleep 2
ls /usr/adm/syslog.dated/* >CHECKIT.OUT
echo ...............CHECKIT.OUT ...............file created
cp CHECKIT.OUT CHECKIT.OUT1
sleep 2 
echo "==========checkit.sys===================================="  >>checkit.sys
sort -u CHECKIT.OUT >>checkit.sys 
sleep 2
strings /vmunix |grep "UNIX V" >vm.out 
echo .............VMUNIX Info...............  File Appended 
echo "++++++++++++VMUNIX INFO +++++++++++++++++++++++++++++++++" >>checkit.sys 
sort -u vm.out >> checkit.sys
sleep 2
strings /vmunix |grep "OSF/1 V" >vm2.out
strings /vmunix |grep "OSF V" >vm3.out
echo .............VMUNIX-OSF1 Info...............  File Appended
echo "++++++++++++VMUNIX-OSF1 Info+++++++++++++++++++++++++++++++++" >>checkit.sys
sort -u vm2.out >> checkit.sys
sort -u vm3.out >> checkit.sys 
sleep 2
grep rz /usr/adm/binary.errlog >RZ.OUT
echo ...............RZ.OUT..... ...............File Appended
sleep 2
echo "==========Rz firmware info============================="  >>checkit.sys 
sort -u RZ.OUT >>checkit.sys 
sleep 2 
grep Firmware /usr/adm/binary.errlog >FIRM.OUT
echo ...............FIRM.OUT...................file Appended
sleep 2
echo "==========================firmware info=============="  >>checkit.sys 
sort -u FIRM.OUT >>checkit.sys 
sleep 2 
grep memory /usr/adm/binary.errlog >MEMORY.OUT
echo ...............MEMORY.OUT.................file Appended
sleep 2
echo "==============memory info================================"  >>checkit.sys 
sort -u MEMORY.OUT >>checkit.sys 
sleep 2 
grep 40  /usr/adm/binary.errlog >KEY.OUT
echo ................KEY.OUT...................File Appended
sleep 2 
echo "==============keyboardiinfo==================="  >>checkit.sys
sort -u  KEY.OUT >>checkit.sys 
sleep 2
ls -Rl /etc/fdmns >fd.out    
echo ................fdmns.OUT...................File Appended
sleep 2
echo "===============domain info==============="  >>checkit.sys
sort -u fd.out >>checkit.sys
sleep 2 
cp /usr/var/adm/smlogs/it.log itlog.out
echo ................itlog.OUT...................File Appended
sleep 2
echo "===============install log info=============="  >>checkit.sys
sort -u itlog.out >>checkit.sys
sleep 2
cp /etc/fstab fstab.out
echo ................fstab.OUT...................File Appended
sleep 2
echo "================fstab info=============================="  >>checkit.sys
sort -u fstab.out >>checkit.sys
sleep 2
presto -pl >presto.out
echo ................presto.info.................File Appended
sleep 2
echo "================presto info=============================="  >>checkit.sys
sort -u presto.out >>checkit.sys
sleep 2
ls -l /var/ase/sbin/ >layered.out
echo ...............misc.info ...............file created
sleep 2
echo "==========misc. info===================================="  >>checkit.sys
sort -u layered.out >>checkit.sys
sleep 2
ls -l /var/dust/ >dust.out
echo ...............dust.info ...............file created
sleep 2
echo "============dust.info=========================" >>checkit.sys
sort -u dust.out >>checkit.sys
sleep 2
/usr/sbin/netstat -i >netstat.out
echo ............netstat info.......................        
sleep 2
echo "============netstat info======================" >>checkit.sys
sleep 2
sort -u netstat.out >>checkit.sys 
sleep 2
cat /etc/resolv.conf >resolv.out
echo ............resolv.conf info.......................        
sleep 2
echo "============resolv info======================" >>checkit.sys
sleep 2
sort -u resolv.out >>checkit.sys 
sleep 2
cat /etc/svc.conf >svc.out
echo ............svc.conf info.......................        
sleep 2
echo "============svc.conf info======================" >>checkit.sys
sleep 2
sort -u svc.out >>checkit.sys 
sleep 2
cat /etc/exports  >exports.out
echo ............exports file info.......................        
sleep 2
echo "============exports file info======================" >>checkit.sys
sleep 2
sort -u exports.out >>checkit.sys 
sleep 2
cat /etc/lvmtab >lvmtab.out
echo ............lvmtab file info.......................        
sleep 2
echo "============lvmtab file info======================" >>checkit.sys
sleep 2
sort -u lvmtab.out >>checkit.sys 
sleep 2
cat /var/adm/messages >msg.out
echo ............message file info.......................        
sleep 2
echo "============message file info======================" >>checkit.sys
sleep 2
sort -u msg.out >>checkit.sys 
sleep 2
cat /etc/sysconfigtab  >sysconfigtab.out
echo ............sysconfigtab file info.......................        
sleep 2
echo "============ sysconfigtab file info======================" >>checkit.sys
sleep 2
sort -u sysconfigtab.out >>checkit.sys 
sleep 2
cat /etc/rc.config >rc_config.out
echo ............rc.config file info.......................        
sleep 2
echo "============rc.config file info======================" >>checkit.sys
sleep 2
sort -u rc_config.out >>checkit.sys 
sleep 2
cat /etc/sia/matrix.conf >matrix.out
echo ............matrix.conf file info.......................        
sleep 2
echo "============matrix.conf file info======================" >>checkit.sys
sleep 2
sort -u matrix.out >>checkit.sys 
sleep 2
voldctl list >voldctl.out
echo ............voldctl info.......................        
sleep 2
echo "============voldctl  info======================" >>checkit.sys
sleep 2
sort -u voldctl.out >>checkit.sys 
sleep 2
voldg list >voldg.out
echo ............voldg info.......................        
sleep 2
echo "============voldg info======================" >>checkit.sys
sleep 2
sort -u voldg.out >>checkit.sys 
sleep 2
volprint -Aht >volpr.out
echo ............volprint -Aht file info.......................        
sleep 2
echo "============volprint -Aht file info======================" >>checkit.sys
sleep 2
sort -u volpr.out >>checkit.sys 
sleep 2
volprint -Am  >volprint.out
echo ............volprint -Am file info.......................        
sleep 2
echo "============volprint -Am file =====================" >>checkit.sys
sleep 2
sort -u volprint.out >>checkit.sys 
sleep 2
voldisk list >voldisk.out
echo ............voldisk list info.......................        
sleep 2
echo "============voldisk list info======================" >>checkit.sys
sleep 2
sort -u voldisk.out >>checkit.sys 
sleep 2
df >df.out
echo ...........df info.......................        
sleep 2
echo "============df info======================" >>checkit.sys
sleep 2
sort -u df.out >>checkit.sys 
sleep 2
echo ............................... 
echo NOW TARing checkit.sys file for shipment to USEG
echo
tar cf checkit.sys.tar checkit.sys 
sleep 5 
echo .. 
echo .. tar creation file completed 
echo ..
compress checkit.sys.tar
echo ...
echo compressions completed
echo ..
echo ..
echo Removing OUT files from your system 
echo ...
echo ....
echo ........
echo .............
rm *.out
rm *.OUT
echo ............FINISHED .....................
echo .. 
echo a checkit.sys.tar file has been created. Please forward this file
echo to USEG Problem management..along with tared vmunix,vmcore and binary_
echo errlog and account information from the questions on the CLD request for 
echo info sheet.

I assume from your words, "The following Unix shell script was being sent..."
                                                           ^^^

that it is no  longer going out?
If this is not true, we need to stop it.
This script indiscriminately deletes all *.out and *.OUT files, whether
they were created by the script or whether they were pre-existing
files on the customer's system!  (not to mention not really making
us look good in general...)

Kevin

..    How did Digital come to this?  What should be done?
    
	I suppose the normal answer should be WAKE UP but I wonder
	if I should not go and sleep :-)

	Joel 

I put a note in the UNIX conference about this a while back. Not much response.

I particularly like the way that it sorts everything: /etc/fstab, 
/etc/sysconfigtab, /etc/rc.config (a shell script, for heaven's sake!). How 
anybody is supposed to make sense of the output I don't know.

Then there's the grepping of various binary files for text information: possibly 
an acceptable UNIX hack, but there are neater ways of getting the information.

I'm scared to to send email to cansler@guru.zk3.dec.com: I might find out what 
this person does.

PJDM

    I think that Bob Cansler left USEG a short while ago.

    That is correct.  Bob did leave.  Specific information on any issue
    that Bob may have been involved with should be addressed to Jim
    RUSURE:: Metsch.

    Eric,

    Your questions are rather open-ended and I am not quite sure to whom your
    question is addressed.  I will assume they are to the general Digital
    community as a whole, simply based on the forum to which the questions
    were posed.
    
    >>>How did Digital come to this?  
    
    My recollection as to why it was determined that USEG should distribute
    the script is that there were too many escalations being received which
    did not contain even the most basic of information.  This script was an
    attempt to glean first-level information.  Too many field persons are
    NOT UNIX literate.  Many of our customers are not UNIX literate.  This
    was simply a starting point.
    
    I wish to temper my remarks about the field and customers.  There are
    many good UNIX literate people out there.  Just not in the same numbers
    as VMS and NT.  The problem managers (Tom et.al.) have to deal with
    this day in and day out. 
    
    >>>What should be done?

    My personal opinion is that if we are to have Digital people out in the
    field that support UNIX, then they should be trained to the level
    necessary to support it.  Of course I would like to see something
    similar to the level of NT training that is occuring in MCS.  But I
    know this is not practical.  Hell, we can't order paper clips, let
    alone spend money on training.
    
    I really believe we should farm most support out to our ISVs.  Similar
    to what IBM does.  That is, for most of the day-to-day minor issues. 
    Our own MCS staff should be trained for the more problematic issues
    that can arise.
    
    It's late an I apologize if my last points were not clear.
    
    Grahame

The trouble is, it seems that the person who wrote the script wasn't UNIX 
literate either.

Sorting fstab or sysconfigtab is ludicrous.

The use of constructs such as

	df > df.out
	sort -u df.out >> checkit.sys

rather than

	df | sort -u >> checkit.sys

shows a basic lack of UNIX skills and/or comprehension.

	echo Please wait while I am loading. 
	sleep 2
	sleep 2
	sleep 2
	echo Thanks for waiting
	sleep 2
	sleep 2

Who's kidding who, here?

For such an important script (first level information for CLDs), it seems to 
have undergone absolutely no peer checking: I can't believe that anybody 
responsible and knowledgeable enough for dealing with UNIX CLDs would let this 
past.

I sent my (polite as possible) opinions of this script, and an alternative 
written by myself that produces an HTML document describing the currently 
running system in a more useful way than this script does, to the person 
mentioned, but it looks like it's too late.

PJDM

>The use of constructs such as
>
>        df > df.out
>        sort -u df.out >> checkit.sys
>
>rather than
>
>        df | sort -u >> checkit.sys
>
>shows a basic lack of UNIX skills and/or comprehension.
>

    What?  We're now grading on style points?  While many of the other
    comments about this script are valid, we surely are not on the quest
    for the Holy Grail of fewest keystrokes...  
    
    				-John

If memory serves me correct (it doesn't always) the sleeps further down in
the script are necessary to insure total write for the concatenation across
the file system (especially Advfs).  There is a little latency there.

    Re .71:
    
    My questions are aimed not at why we need to collect data but how a
    script of the quality of that in .65 could ever be issued by any
    Digital group.
    
    
    Re .74:
    
    > . . . the sleeps . . . are necessary to insure total write . . .
    
    While there may be some latency in physically writing the data to disk,
    I rather doubt that the semantics visible to the user are affected.  If
    it did not appear to applications that data written to a file were in
    that file for subsequent operations, applications would fail left and
    right and the system would be unusable.
    
    
    				-- edp
    
    
Public key fingerprint:  8e ad 63 61 ba 0c 26 86  32 0a 7d 28 db e7 6f 75
To find PGP, read note 2688.4 in Humane::IBMPC_Shareware.

    > While there may be some latency in physically writing the data to disk,
    > I rather doubt that the semantics visible to the user are affected.
    
    That's quite correct.  Put another way, the shell's '>>' operator
    writes and closes the file before returning control to the script, so
    there can't be any overlap.  The sleeps are totally unnecessary.
    
    	Steve
    

Re .73

I wasn't commenting on style, I was commenting on the comprehension of 
UNIX/shell skills.

No semi-skilled UNIX person would use "df > df.out; sort -u df.out >> 
checkit.sys" as opposed to "df | sort -u >>checkit.sys"; not because it has 
fewer keystrokes, but because it's "the UNIX way" (and it doesn't leave a 
temporary file lying around which possibly overwrites an already existing 
file, then has to be deleted).

If the author didn't understand the use of such basic shell concepts as | when 
writing such an important script (even leaving aside sleeps and sorts, 
actually checking if root is running the script, checking that no existing 
files are being overwritten, cleaning up if the script is interrupted, and so 
on), one might be led to wonder about the quality of the people responsible for 
Digital's UNIX engineering. (I'm not denigrating our engineering teams in 
any way, but consider: if one of your customers gave you a script like this, 
what would you think?) This code smacks of a VMS/DCL person who's just 
discovered that > is the same as /OUTPUT.

What does worry me is not the fact that such a script was written, but for such 
an important script, the review process seems to have been either done very 
badly, or bypassed altogether. Doesn't anybody check anybody else's work 
anymore?

PJDM

>Doesn't anybody check anybody else's work 
>anymore?

Get real.  Of course we don't.  We're too busy trying to do more with less, let
alone look over the shoulders of our co-workers.  Welcome to the 00's folks.

    RE: .77
    
    Perhaps the script needed to keep df.out around for something else, but
    later got modified further such that it didn't need to keep it around?
    I can make up all sorts of scenerios that would explain it.  Of course,
    you'll suggest that the author should have used tee and been real
    cleaver by doing both at the same time...
    
    				-John

Not to mention that many of us former VMS types learned UNIX by
getting a workstation and fumbling around and asking questions
and, oh by the way, learning win95 and NT also. We tried to use
moderated code reviews back before we lost 3 engineers we weren't
allowed to replace, even though the work was not reduced.

I'd guess there is a lot of stuff going out the door that never
saw anyone but the original author because no one else had the
time for a code walkthrough. Perhaps we could make use of the
VP's for this???? liesl

    OOh, you *ARE* the wicked one, aintcha!!!
    
    **DIRECT HIT**
    
    **SHIP TAKING ON WATER**
    
    Bravo!