Conference 7.286::digital

    I recently lost 3 hours of revenue time while in the office trying to
    find one printer that would work. All I needed was one simple manual
    (probably worth $5 in paper form) but it took three hours and clearing
    several paper jams, before I could get the info I needed to return to
    my customer site and 'get back on the clock'
    
    Somewhere, somebodys budget looks better because he/she doesnt spend
    money servicing those printers any more, and someone else 'saved' a
    bundle because we dont distrubute paper docs, but I lost real revenue
    from a real customer because of it.
    
    I cite one case, because it was recent enough, and agraviting enough,
    that I recall it well. It has happened many times to me, and I m sure
    to others.

    VTX?  I quit using that tool months ago...
    IF you get in (9 times out of 10 I get the "unavilable" message)
    then the systems you have access to have been cut/limited/are almost
    never available
    
    Not to mention the fact that due to the work-at-home program what
    should have been resolved in about twenty-thirty minutes took over four
    hours when I bothered to log a call on some difficulty!
    
    Not to get anyone in trouble here...
    I don't need managers calling me up and yelling at me (again) about
    what I put in the notes file so they can find out who to  blame...
    
    the point is our internal systems are failing to perform the way they
    used to and it's sad...
    
    but isn't this reminiscent of another note too?
    
    -omwotd-
    
    

    Well, it's now 5:30 PM and the AQS system is still not up. I am beyond
    amazed. This is like trying to run an airline and you can't get your
    reservation system to run. Sigh! Where's Chicken Little when you really
    need him...the sky really is falling.
    
    One of my customer measures IM&T failures like this in the number of cars 
    that don't come off the end of the line. I wonder how we measure the impact
    of this failure...in wasted sales time, in business lost because we
    couldn't quote anything all day or just, oh well, too bad. As I
    understand it, the AQS system was updated over the weekend. I don't
    suppose there was any backup/recovery plan in place. Oh well, too bad
    no skin off anyone's nose (or any other part of their anatomy, I'd bet).
    
    -OFWAMI-
    
    
    
    

Last night (sunday), I tried to get into AQS at 10:30pm to do an important
quote that was due to a customer on monday. AQS was down. I called the CNS 
East Hotline and they didn't really know what AQS was. Then, when they found 
it listed somewhere, it only had 8-hour coverage! Three hours later it was 
still down, so I got into MS Word and did up my own quote.

Earlier this evening I went into the Integrated Repository and tried to find a 
StorageWorks presentation. Use the Keyterm search, it found NO MATCHES for 
StorageWorks and only one match for the word Storage.

The more you try to get more morale up, the more you find out that this 
company is quite a ways from having its act together.

    
    AQS was scheduled to be down till 6am monday.  The US PASE upgrades
    were done this weekend and this affected a lot of applications.  What
    they basically do is all the upgrades roughly once a year on one
    weekend, rather than piecemeal, and this was the one(usually lots of
    complications when you do that many applications at once).  Plus I believe
    this one had been delayed a couple times and so was about 15-18 months
    worth of upgrades for many different applications all done in one
    swoop. 
    
    So for the user sunday night that had been scheduled at least 6 months
    ago.  Aqs should have been back by 6am monday but complications due to
    the upgrade delayed it(mondays also a holiday).  I can say that lots
    of CNS people did work mega hours this weekend on the problems.
    
    I'm not saying the cutbacks haven't affected levels of service, just
    spreading a few facts in this speculating.
    
    Tim
    CNS South/Central Command Center
    

    >    the upgrade delayed it(mondays also a holiday).  
                                ^^^^^^^^^^^^^^^^^^^^^^
    Not according to the official Digital calendar.
    
    
    

re: .6

Some locations had Monday as a site choice holiday for MLK.

Bob

    What's InfoBaun anyway? InfoBahn?

    presume InfoBahn derived from a german infosuperhighway. In germany,
    they have a reasonable speed limit (..er.. none?). Implication may be
    a bahn is better than a freeway.
    
    Isn't a baun an electric shaver ? 

    Braun is an electric shaver trademark ...

    And Baum is a tree... never heard of Baun.

    Should be Bahn, as in autobahn.
    
    Leslie

    I want to thank the last few Noters for the excrutiating attention to
    the detail of what I mistakenly named this particular string. To those
    thoughts, I would add...who really cares. A nit well picked is perhaps
    worth a chuckle, but a nit picked to the Nth degree is just boring.
    Let's get on with some useful discussion here or just let the string
    die and get back to asking each other for the names of long lost
    product managers, account teams or TFSO'ed friends from the past.
    
    And that's all I have to say about that...
    
    -OFWAMI-

    In re "Die Nit" ... are you sure it's not "Das Nit"?

Re:  <<< Note 3638.5 by ALFHUB::WELLS >>>

The message on our AQS system said it was to be down until 10pm sunday night.
I'm not complaining about the upgrade, although I think they could have done a 
better job of communicating the downtime. I'm really concerned about the 
level of support available for AQS.

    InfoBahn, as in AutoBahn    Bahn = Path or Road

Ah, yes, but did they say 10PM in which time zone? 

Here on the left coast we often get told that we're calling after 5, when 
it's really only 3......

They may have meant 10PM in Hawaii...

    
    The CNS person was correct in that AQS only has 8x5 coverage.  This is
    a decision made by the business as to what level of service they want
    to pay for.  You're not the only person I've heard from who thinks
    it should be more, so please make your managers aware of this.
    
    
    Also, I said that these upgrades of over 90 layered products had been put
    off over and over by the businesses(not the people installing them)
    for 15-18 months, but I've found out it's actually been just about 
    2 YEARS since the last upgrade.  And they include not just the
    applications, but also VMS as well as Decnet OSI.  That's a lot
    of change and pressure to put on for one weekend.  
    
    And to further brighten you day, that was only the first wave of upgrades 
    as this coming weekend all the Champ and Smart clusters have to go through
    the same process.  Should be interesting...
    
    Not blaming anyone just putting a little perspective on all this.
    
    Tim
                                                    

...and here I thought we were a global corporation.  Turns out,
we're just a GMA corporation?

If AQS is essential to quoting and closing business, and is used
nationwide if not worldwide (I don't know), just how the heck are you
picking WHICH 8 hours to cover in the 8x5 coverage?

Do you really want folks on the west coast to go home at 2PM?
Do you really think it is valuable to have salesmen have to start their
day at 5AM (when none of their customers are around) just to compensate
for somebody shaving operating costs???

This is really stupid!

    > Do you really want folks on the west coast to go home at 2PM?
    
    please, if you do, come in at 5AM and make sure that your systems are
    operating.  It's sure hard for us GMA-types to get our work done in the
    morning without you.
    
    Mark :-) (kinda)

	I would like to clarify the issue around AQS support.  It is
	not accurate to say that AQS only has 5x8 support.

	In the US, AQS shares clusters with other Customer Order
	Management applications at four regional data centers.  Support
	for these computing environments is provided around the clock
	by CNS.  This includes support for such things as machine
	availability, network connectivity, response time, printer
	queue problems, fax delays, VMS account creation and modification,
	and AQS user registrations.

	The only support that is not available outside of regular business
	hours is that provided by the AQS Development and Business support
	groups.  This type of support includes answering questions on
	how to use AQS, enhancement requests, and errors caused by the
	AQS software.  AQS is a remarkably stable system.  We have never
	heard of a need for these services to be provided after hours, but
	this can certainly be revisited if required.

	The problem reported in the base note seems to me to be a
	training issue.  The person handling the call should have routed
	it to the local data center personnel for resolution.  I am
	bringing it to the attention of our CNS contact.  Hopefully,
	we can prevent this type of response from reoccurring in the
	future.


	Joe Sniderman
	AQS Development

    Well, Mr. Werner.  Was it a training issue?  Did someone forget to tell
    you to read the manual?
    
    Mark

>    Well, Mr. Werner.  Was it a training issue?  Did someone forget to tell
>    you to read the manual?
    
	I guess I wasn't clear.  I was referring to the training 
	of the call handler, not the caller.

    Personally I think it's a bit silly to put off doing upgrades for that
    long, I'm quite sure that a rolling upgrade kinda plan would have
    worked. With all the downsizing going on, I'm sure a spare cluster or
    two is laying about somewhere, maybe take in some trade-in systems and 
    actually use it for "development"?
    
    I'm pretty amazed by the comparisons between our internal systems and
    those of the customers I support. If radar systems and submarines can
    be designed and documented with little/no downtime, what IS our
    problem?
    
    .mike.
    
    (Do we even still support VMS that 2+ years behind?)

    Hi Mike,

    Regarding putting off upgrades for that long: CNS in the South/Central
    Cluster is evaluating a request from a major, Americas-wide customer for 
    on-site expertise to upgrade a cluster from VMS V5.1 (!) and ALL-IN-1 V2.3
    (also !) to OpenVms V6.1 and ALL-IN-1 V3.1, over a week-end ...

    Makes one wonder about things such as:

	- layered product compatibilities from then until now
	- what kind of "system management" has this customer been doing
	- when was the most recent Backup done
	- in what condition are the systems at this customer's _many_ other
	  sites across the Americas
	- and, to quote you "Customers can do it, can we?" - why hasn't the
          customer done this ...

    FWIW,

	Ron

One could also ask, "are we planning to do any more major application
development on VMS, and if not, why bother to upgrade at all?"  In other
words, "what's the *business* justification?"  I thought at this point
that VMS was basically where all our legacy applications resided and
that the replacement applications would utilize OSF/1 or NT servers and
Windows PC clients.  Same situation with DECnet...if TCP/IP is the future,
can we really make a dollars-and-cents case for going Phase IV to OSI?

I guess it depends on how buggy the older versions are...if all the bugs
are pretty much known at this point and all have workarounds (no show-
stoppers), is losing "supported status" all that big a deal?

- paul

>if TCP/IP is the future,
>can we really make a dollars-and-cents case for going Phase IV to OSI?

Because with DECnet/OSI V6.0, you get DECnet over TCP/IP.

But I digress....

--Scott

    	Some customers value stability highly. When this sort of discussion
    came up in another forum I received mail from a support specialist
    supporting a rather old VMS system. The customer regarded the computer,
    its operating system and its peripheral equipment as a single unit that
    would be replaced as such when it reached the end of its economic life.
    
    	The peripheral equipment in this case happened to be a power
    station, with an estimated economic life of 15 to 20 years, and the
    customer saw no reason why he should upgrade from VMS 1.6 until that
    time was up.
    
    	This is an extreme case, of course, but there are many cases where
    a customer doesn't want to keep up with the latest in everything. I
    visited a large GM factory once. The I.S. staff wasn't allowed to alter
    anything on the production systems except during the two weeks in
    August when the whole factory closed. With this limitation they were
    choosing hardware and software upgrades many months in advance, and
    then testing them to death in test environments so they could be sure
    that there would be no glitches found in the two weeks they had to make
    the changeover. By the next summer all the hardware and software
    versions would be *at least* 18 months behind what we were currently
    shipping.

    
    In some businesses, if it ain't broke... don't fix it. 
    
    gb
    

    re .28
    
    And for some companies, two weeks would be considered incredible
    luxury!  There are some (usually chemical/plastic process control)
    applications my group has worked with in the U.S. where there are only 
    2 times during the year that systems can be taken off line - the 4th of
    July weekend, and the Christmas weekend.  And the window isn't the
    whole weekend either - there's usually a "drop dead" point where, if
    the upgrade isn't complete, tested, and stable by then, you have to
    cancel it and have time to restore the old system and make sure IT'S
    complete, tested, and stable for the start of production.  And then you
    wait another 6 months for the next window...
    
    /Butch

    Well, I certainly didn't expect this string to have this long of a
    life. The point that I was trying to make, perhaps too gently, is that
    we sometimes run our business too much as if it were an engineering lab
    environment. By that I mean that too often our IM&T organization seems
    to execute ill-planned upgrades or changes to key, mission-critical
    systems and the folks not appear to be overly concerned with the resulting
    failures and the inconvenience that it causes the users of those
    systems. 
    
    I have carefully chosen to use words like "appears and seems"
    since this is only an impression that one can easily get from talking
    to the apparently untrained (see several notes back on that) folks who 
    have been assigned to respond to the resulting HOTLINE calls. The
    example that started all of this off was the recent AQS system failure
    that resulted from the latest upgrade. AQS was down across a good part
    of the coutry (if not everywhere) all day a that day. I couldn't use it
    for critical quotes and the 1-800-DEC-SALE folks could get to it to
    answer critical technical support questions. We were effectively
    without one of our mission critical systems all day. More recently an
    "MCI problem" (at least that was how it was characterized by the
    HOTLINE folks) effectively paralyzed many of the critical East Coast
    systems, including key VTX nodes. Another day was impacted.
    
    If I was at a customer site and they were telling me these horror
    stories, I would sieze the opportunity to pitch Digital's Business
    Recovery Services. Internally, however, there seems to be an almost
    light-hearted acceptance that this is just the way that Digital runs it
    business. Bull! No company should run like that. No company can run
    like that and not be in trouble. We can't afford a day here and a day
    there of severly crippled administrative system capabilities. As an
    employee, I'm frustrated. As a stockholder, I'm appalled.
    
    -OFWAMI- 

    well, since this note is becoming more about notification
    of services, you might be interested in this (which may
    impact vtx services both TCP/IP and Decnet.)(Internaly,
    vtx systems typically have 3 server points of failure, and
    each can have up to 9 client points of failure, but this
    seems to involve at least the last 2 layers of failover (for vtx) )
    About easynet backbone outage:
    
    
    The date and time for the Corporate Backbone upgrade has changed from
    February 5th to Saturday, February 18th, to accommodate critical business
    processing.  Outsourcing Management Services (OMS) recognizes the need for
    flexibility in arriving at a scheduled date acceptable to all Digital
    businesses.  Critical business applications have requirements that impact
    different time periods in the fiscal calendar.  Through the cooperation
    of OMS and the respective Digital business representatives, we have agreed
    upon the February 18th date.
    
    What and When
    
    On Saturday, February 18th, 7:30AM-12:30PM EST, the Corporate Data Networks
    Backbone will be upgraded to a new routing protocol: Integrated
    Intermediate-System to Intermediate-System (Integrated IS-IS).  The backbone
    routers located at AKO, LKG, MKO,  MRO, and PKO, will be unavailable during
    this timeframe.  This will affect both DECnet and IP network traffic.
    Communications between geography portions of the network including
    communications within the Americas will be impacted.  Communications
    within regional distribution networks will not be affected.
    

Well, it's not a matter of technical feasability to do a rolling upgrade, it's
mainly a matter of cost (and organisational prerequisites, which require effort
to implement). It's *much* cheaper to shut a cluster down, upgrade, and make it
available again when the upgrade is done than to do rolling upgrades.

If anybody could come up with numbers how much it costs the company to have some
business support application not available over a weekend, it would be easy to
compare the amount of lost money with the costs of implementing the structures
for rolling upgrades and of performing them.

Hans.

    Granted, running a nuclear power station or a chemical/plastics factory
    or any time-critical application has it's own requirements for absolute
    and guaranteed uptime and stability, but Digital Equipment Corporation
    does NOT do any of these things internally. (That I know of anyway)
    
    Our internal systems provide information. I see no reason not to
    compare what my (large AeroSpace) customer does with their systems to
    what our internal systems do. As we (Digital) refuse to support things
    that are "x" versions behind, then technology must march forward to
    maintain support. I still say that if the BSY-2 team (2 guys) can keep
    many hundreds of users happy, still serve hundreds of sun users and
    keep VMS and layered products current (Ok VMS 5.5-2 is still pretty
    current, isn't it?) then we can too.
    
    That point made, how many spelling and grammar errors do you see in
    applications? How cool is that? Yeah, nit-picking, but I just hate it
    when an application asks me to "Please enter you selection.".
    
    .mike.

    No, we don't run nuclear plants, but I would imagine that both
    CHAMP/FLD and CHAMP/CSC need to be guarenteed available and stable.
    
    Can't fix customers systems if they can't report problems.  MCS relies
    on those systems being available 24 * 7.

              <<< Note 3638.35 by CSEXP2::ANDREWS "I'm the NRA" >>>

>>    Can't fix customers systems if they can't report problems.  MCS relies
>>    on those systems being available 24 * 7.

But we do use hardcopy when they are not available :-(.

Liz

    re: .34
>    keep VMS and layered products current (Ok VMS 5.5-2 is still pretty
>    current, isn't it?) then we can too.
    
    	I believe VMS 6.0 is now unsupported. The normal rule was that
    support for an old version was dropped 6 months after its replacement
    version shipped. VMS 6.1 will be unsupported before the end of this
    calender year unless there are project slips.
    
    	When I ran a support cluster we had most layered products installed
    (around 80 of them), and most of these layered products had at least
    one release per year, some of them two. If you add to that the fact
    that some specialists wanted to go through the field test cycles for
    their favourite product then simple arithmetic shows an average of 4
    software installations on the cluster in an average week, with peaks
    considerably higher.
    
    	Admitteldy most customers don't have quite such a range of software
    on their sustems, but we can't expect all customers to be running
    "supported" versions.

    VMS support holds on longer than the standard 6 months.  MANY customers
    are still at V5.5-2.
    
    					Steve

Re .38:

>    VMS support holds on longer than the standard 6 months.  MANY customers
>    are still at V5.5-2.

Is Digital contractually committed to support V5.5-2?
				/AHM

    I believe that MCS will accept contracts to support V5.5-2 - I don't
    know what the terms are.  Perhaps someone from MCS can comment.
    
    				Steve

    Mission Critical in MCS will provide support.. they do it for MCI under
    contract.

I wonder if MCS will accept contracts to "support" obsolete versions of *my*
product?
				/AHM

    	When whatever has become MCS was first set up they worked on the
    principle "We'll support anything, but if you're the only customer in
    the world with the thing then the support costs could include training
    an engineer on the product, ...".
    
    	In 1975 in the U.K. the organsiation was proud that they could and
    did still support a PDP-1, and made a point of it with advertising that
    with DEC you would never have unsupported hardware or software. It was
    about that time that they started to support products from other
    manufacturers too.

Alan:

> -< We ought to charge on a sliding scale for obsolete SW support >-

  You mean "We'll charge a lot less to support old, well-known bugs
  (where the answers are already also well-known) than for fresh new
  mysterious bugs?"


  No, I didn't think that's what you meant.  Too bad.

                                   Atlant