Conference azur::mcc

    Doug,	
    	Can I assume this trouble ticket application will be primarly
    driven by data from the Alarms FM?  If so, then we need some form of
    alarm pre-processing prior to generation of a trouble ticket. Some 
    form of a rule based pre-processor will be required to look at the
    alarms which are triggered and form a consolidated problem report.
    At the moment my customer is considering feeding the DECmcc alarms
    into NEXPERT and using NEXPERT to generate the troble tickets. (The
    only reason they havn't this up and running is because I havn't
    figured out the calls required to get the events out of DECmcc!!). 
    
    	We had started writing a simple trouble ticked application
    which would have been driven by the MCC_ALARMS_LOG_ALARM.COM and
    MCC_ALARMS_LOG_EXCEPTION.COM files.  However the number of useless
    exception messages which were generated by DECmcc proved this
    approach was futile. 
    
    	We currently evaluate several hundred rules per hour, on one
    occasion the DNS server died during the night.  The following
    morning there were over 6,000 jobs on the batch queue still
    processing!  If we had been generating trouble tickets for each of
    these we would have had a serious credibility problem with our help
    desk folk. 
    
    	The bottom line is Trouble Ticket generation applications will 
    be purchased but never used unless we allow the customers to dictate
    the rules which determine what generates a trouble ticket.  Now of
    course you could say that  it's the job of the alarms team to
    provide the capabilities to generate intelligent alarms :-)
    
    - Mike     

Doug,

I have had several customers look very closely at DECmcc.  There are actually
several who have purchased the tool.

The discussions around trouble ticketing are quite long, detailed, and often
contain some "religious" issues.

If you are interested, I could try to create a summary and mail it to you.
Perhaps, a better idea would be to have a phone discussion and see if I could
convey the customer's ideas and concerns.

My DTN is 521.4014.  I am usually in my office by 0700 PST.

Regards,

Glen R.

re .2

    Actually, I would prefer to see this discussion carried on in this
    conference rahter than 'off-line'.  This information needs to be
    seen by more people than just me and I think getting visibility
    in this forum would be good for getting people thinking about this.

    Also, I don't feel up the responsibility of being a "gatekeeper" for
    the feedback/requirements...

  >                                                             Now of
  >  course you could say that  it's the job of the alarms team to
  >  provide the capabilities to generate intelligent alarms :-)

    I'm really glad you put a smiley there.   

    I do understand that we have some rather irksome limitations to live
    with with the current Alarms FM.  (Don't think that they haven't heard
    about it, already.)   But in the interest of making the best of what we
    have to sell today, I would be more inclined to say that the network
    manager will need to apply a little more intelligence to the creation
    and application of alarm rules.

    Now I don't know about your particular customer's environment. In
    general, if alarm rules are crafted such that they "fire"
    indiscriminately or gratuitously, then yes, you will need some serious
    processing power to wade through the data and make some sort of
    intelligent assumptions.   Please don't kid yourself into thinking that
    implementing an "Expert System" is going to be a 'turnkey operation'.   
    Their knowledge base must also be crafted (not unlike the way in which
    a network manager needs to carefully consider the parameters for alarm
    rules) and the knowledge base needs to be as extensible as the
    environment (enterprise) to which it is being applied.

    It isn't necesary for ALL rules to create trouble tickets, at least no
    more so than is necessary for all alarm rules to send you irritating
    mail messages when they fire. If you're getting soaked with *thousands*
    of 'automatically generated mail messages' to a  Trouble Ticketing
    system, then you're gonna have problems; some data reduction must be
    done.  However, I would seriously question *why* you're generating that
    many events, especially when we have the capability in DECmcc to only
    subscribe to those alarms/events you are expressly interested in.

Here's the main reason I started this discussion:

    I just spent the morning installing "Target->Hotline" on one of our
    systems here and I was able to use it to receive mail messages
    generated by DECmcc/Alarms FM on a remote system.   Target->Hotline can
    	1) pick out keywords from an incoming Alarm Mail message 
    	2) compare those keywords against a list of personnel identified as 
           'qualified' to work the problems indicated by the 'keyword'
        3) assign an identifier to the Ticket and dispatch it to the 
           appropriate person (from a pool)
    	4) Escalate the problem over time (in various manners)
    	5) (Optionally) send status data back *into* DECmcc for the
           notification  services window to display (e.g. for the purposes
           of notifying escalation, status or clearing of trouble tickets)
           on the Iconic Map...

    This was pretty exciting for to me, since I had just installed the
    Target->Hotline stuff about 45 minutes prior...     None of this, by
    the way, required any coding on my part.  

    Is this sort of thing worthwhile? 

    /.doug

What I understand from the previous , and from feedback that we got from the 
field, is, that there is indeed a need for an 'intelligent' alarms 
preprocessor. I basically would leave the setup and collection of alarms
as it is in DECmcc. eg. DECmcc takes care of the collection of events
(through events or polling). I should than be able to redirect these
alarms into the alarms preprocessor.

I should be able to set up specific rules (by using some syntax), that 
define what has to be done when a certain COMBINATION of alarms occurs.

For example (and these are real requests):

-Notify me when I get more than x circuit up/down events per hour on circuit
 XYZ.

-If alarm X is triggered than: from 9:00 till 17:00 do Y, else do Z

-If I get node down alarms for node A,B,C,D.... and these nodes are on the
 other side of bridge B, only send me an alarm for the bridge.

-Notify me when a circuit/node goes down for longer than x minutes.

and many more can be listed....

re -1: 

What is this tool/utility you installed? What is called? Can you give some
more info?

Thanks

Timo

Are you aware of the TROUBLE TICKETING FUNCTION MODULE that is currently
being developed for the TNMP program in Annecy? There is no need to reinvent
the same wheel again...:-)

Re: TNMP Trouble Ticketing FM

    That's great.  I've read about it in some of the TNMP stuff, but
    the details that I've gotten are very thin. 
	- Where's the functional spec for the application? 
	- When do plan to release it? 
	- Will it be available a la carte from the rest of the TNMP stuff? 
	- How much will it cost (ballpark)? 

Back to the main topic: 

    I'll say it again: The purpose of this topic is generate some concrete
    discussion of specific requirements for Trouble Ticketing applications
    (with respect to DECmcc).    

    Because "Trouble Ticketing" is now a checklist item on virtually all
    RFPs for network management, Digital must be able to:

	1) satisfy that checkbox with something (anything) *now*.  It looks
           like we can do this with near-zero effort with Target Systems'
           "Hotline" product...
        2) give the customer the flexibility and interfaces to
	   implement/integrate other 3rd party trouble ticketing applications
	   that they may already have or be interested in.
	3) be able to articulate a clear vision (including deliverables) of
	   what we _intend_ to deliver as part of well-integrated, EMA/DECmcc
	   application in that area

    /doug

	I am working in the Telecommunications Engineering group in Valbonne,
France.  We are developing a generic Trouble Ticket Functional Module that 
will be compliant with the standards for trouble management currently under 
development or being proposed in the OSI/Network Management Forum (NMF),
T1M1.5 Operations Administration, and Maintenance sub-group, and CCITT SG7 
Question 9.

	The purpose of this Trouble Ticket Functional Module (FM) is to 
provide the services necessary for manipulating and managing Trouble Ticket 
Objects (TTOs).  The following support objects are being developed for use by 
the Trouble Ticket FM to accomplish this goal:

          1.Trouble Ticket Context - represents the storage location,
            defaults, and overall behavior of the TTOs and Trouble Log
            Records.  May also represent a single work area.

          2.Trouble Ticket Object - represents a current outstanding
            problem that has been reported through the reception of an
            alarm report or directly by a user.  It may be related to one
            or more associated Alarm Objects (AAOs) and indicates
            information about the current state of the problem.

          3.Associated Alarm Objects - represent the alarm objects generated 
            by the alarm reports received.

	In addition, PNMP will includes OSI compliant alarm object FM
(persistant alarm objects), an event logging FM, a run time library for OSI
compliant scheduling and discriminator construct functions, and a special 
presentation model for managing these objects together.

	To get further information on this functional module, you can use the
public notes file TAEC::PNMP.  Pointers to all the applicable documents are
contained in that notes file and any/all questions about the TT FM can be
answered there.

	RE: .0,

>    I would like to know if anyone out there has any *specific*
>    requirements regarding trouble ticketing and DECmcc.    Ideally, these
>    would be the requirements of paying (or soon to pay) customers and not
>    just "blue-sky".   I.e. "How does the customer envision an 'integrated'
>    trouble ticketing/tracking facilty would work within the context of
>    DECmcc?"   "What capability(ies) are most critical?" (prioritized list
>    would be nice).

	Could we clarify the use of "trouble ticketing/tracking facilty"?  To
the industry, I think trouble ticketing relates to all the information related
to a problem whereas trouble tracking represents the process of following a
problem through to resolution.  Any feedback on this?

	I would like to suggest the use of "trouble management" instead 
"trouble ticketing/tracking facilty" if these two items will be discussed under
this same note heading.

>    This, by the way, is a serious request for input, so I would urge you
>    to talk to your customers to solicit their input if they have not given
>    it to you already.

	The PNMP product manager (Claude Hary) may also be a good person to
contact for input on this subject.  Also, he would be interested in listening
to any requirements that people may have for this capability.

RE: .1,

>    	Can I assume this trouble ticket application will be primarly
>    driven by data from the Alarms FM?  If so, then we need some form of
>    alarm pre-processing prior to generation of a trouble ticket. Some 
>    form of a rule based pre-processor will be required to look at the
>    alarms which are triggered and form a consolidated problem report.
>    At the moment my customer is considering feeding the DECmcc alarms
>    into NEXPERT and using NEXPERT to generate the troble tickets. (The
>    only reason they havn't this up and running is because I havn't
>    figured out the calls required to get the events out of DECmcc!!). 

	It seems to be generally accepted that alarm correlation (figuring out
which alarm reports belong to which problems) is a seperate issue from trouble
ticketing/trouble tracking.  I would like to suggest that we start a seperate
note for alarm correlation/thresholding since this represents a different set
of processing from the trouble management process.

RE: .2,

>The discussions around trouble ticketing are quite long, detailed, and often
>contain some "religious" issues.

	Glen, could you list out what these issues are?  I would be very
interested to learn more about them.

>If you are interested, I could try to create a summary and mail it to you.

	This would be great if you could post it here or ail it off line.

RE: .3,

>    Also, I don't feel up the responsibility of being a "gatekeeper" for
>    the feedback/requirements...

	Whatever documents/feedback that I get I will post either here or in
the conference TAEC::PNMP.

RE: .4,

	I would like to move this note to a new note called alarms
correlation/thresholding.

	But I think that it is a very valid concern/requirement and I would
like to continue to discuss it.

RE: .5,

>-Notify me when I get more than x circuit up/down events per hour on circuit
> XYZ.

	Will this be hanlded in the next version of MCC?

>-If alarm X is triggered than: from 9:00 till 17:00 do Y, else do Z

	Can this be handled by using a combination of alarm rules?

>-If I get node down alarms for node A,B,C,D.... and these nodes are on the
> other side of bridge B, only send me an alarm for the bridge.

	To me this is an alarm correlation rule that would be handled by an
alrm correlation functional module.

>-Notify me when a circuit/node goes down for longer than x minutes.

		Can this be handled by polling a node?

>What is this tool/utility you installed? What is called? Can you give some
>more info?

	It does not exist yet (except for maybe a midnight hack).  8-)

RE: .6,

>Are you aware of the TROUBLE TICKETING FUNCTION MODULE that is currently
>being developed for the TNMP program in Annecy? There is no need to reinvent
>the same wheel again...:-)

	Being developed by the Telecommunications Engineering Group for the
Telecommunications Business Group.  We are developing a platform (PNMP) and the 
Telecommunications Business Group is developing a program (TNMP).  For more
info, see TAEC::PNMP.

RE: .7,

>    That's great.  I've read about it in some of the TNMP stuff, but
>    the details that I've gotten are very thin. 
>	- Where's the functional spec for the application? 
>	- When do plan to release it? 
>	- Will it be available a la carte from the rest of the TNMP stuff? 
>	- How much will it cost (ballpark)? 

	The TT FM may be purchased seperately from the rest of PNMP.

	Contact the product manager Claude Hary (@VBE or TAEC::) or try the
notes file TAEC::PNMP.  We have exited Phase 1 already and are in Phase 2.

>    Because "Trouble Ticketing" is now a checklist item on virtually all
>    RFPs for network management, Digital must be able to:

	The TT FM being developed as part of PNMP should be generic.  However
if you have specific requirements, it would be good to know what they are as
soon as possible.

	Carl

    
    
    We are adding things in V1.2 that may make you happy.
    
    Re .5
    
>> -Notify me when I get more than x circuit up/down events per hour on circuit
>>   XYZ.
    
    The new OCCURS function in the rule expression can do exactly this:
    The syntax is as follows:
    
    expression = (OCCURS (<entity> <event> "," <count> "," <delta-time>))
    
    expresion = (OCCURS (cicuit foo TransdirChanged, 3, 1:00))
    
    
>> -If alarm X is triggered than: from 9:00 till 17:00 do Y, else do Z
    
    If you do no the name of the rule that you want to watch, you can do
    something like this:
    
    create a rule called watch_9_to_17
    expression = (occurs(domain foo rule X any notification event, -
                        for start = 9:00, until = 17:00))
    
    I'l have to think about the else part...
    
>> -If I get node down alarms for node A,B,C,D.... and these nodes are on the
>>  other side of bridge B, only send me an alarm for the bridge.
    
    The Notification Services group is working on retargetting events.
    You will be able to retarget Alarm events from a particular entity to
    another entity.
    
How does this sound?
    
    aud..

    Doug,
    
       You really need to define what you mean by Trouble Ticketing. 
    There's a big difference between Trouble Ticketing and Trouble
    Tracking Systems.  It seems like you are saying, "I have a Term,
    called ``Trouble Ticketing''.  Okay, everyone, what does it mean?"
    And then its like, "Nope.  Your wrong.  Next?"
    A lot of good ideas have already come up.  But I don't think you can
    just pick the ideas you like best, and say, "Okay everyone, this is
    what Trouble Ticketing means...".
    
    I am not a Network Manager.  But I have read several Marketing reports
    on what Network Managers want and expect for Network Management Tools.
    (Granted this was several years ago.  Things may have -- probably have
    -- changed since then.) The only reply so far that sounds like Trouble
    Ticketing based upon the Marketing research papers I have read is .8 by
    Carl Silva.
    
    Basically, it usually describes "database-like" functionality.
    Typically, the "objects" in the database are equipment that contains
    history records of what broke and when, who worked on the problem, how
    was the problem solved, how long did it take, how much money did it
    cost to fix, etc., etc.  Reports are the Heart of the functionality.
    Very RelationalDatabasey reports (What are all the problems in the last
    two months that happened to node RAMPALs line BNA-0, What are all the
    problems that cost more than $5000.00 to fix, What are all the problems
    that took Fred Flintstone more than 1 hour to fix, etc.) are REQUIRED
    (as opposed to desirable).  Of course, Objects in the MCC sense are
    Entities, not equipment.  I would talk to Carl Silva and the PNMP folks
    before going any further.  If you were thinking something more
    ExpertSystemy then I would use a different term.
    
    Just my 2 cents.
    
    By the way, we are talking big bucks in revenue if and when we
    implement such a thing.  I believe it should to be tied into an
    inventory-like system, but now I'm blue-skying -- shame on me!
    
    -Matt.

>       You really need to define what you mean by Trouble Ticketing. 

    No, I don't.  What is needed is to understand the _customers'_
    requirements for what _they_ are calling 'trouble ticketing'.  I am not
    making this vague on purpose; if it was clear to me already I wouldn't
    have started this whole discussion.

>    There's a big difference between Trouble Ticketing and Trouble
>    Tracking Systems.  It seems like you are saying, "I have a Term,
>    called ``Trouble Ticketing''.  Okay, everyone, what does it mean?"
>    And then its like, "Nope.  Your wrong.  Next?"
>    A lot of good ideas have already come up.  But I don't think you can
>    just pick the ideas you like best, and say, "Okay everyone, this is
>    what Trouble Ticketing means...".

    Nope, you're wrong, Matt.   Thank you for playing; We have some
    marvelous parting gifts for you...      8^)     8^)     8^)    

    Let me apologize loudly if my soapboxing about expert systems, alarms
    et al. clouded the issue at hand: defining the requirements. Also, if I
    offended anyone by specifically _not_ asking for "blue sky", mea culpa. 
    Matt, maybe a little 'blue-sky' wouldn't be so bad if we could keep clear
    what capabilities/primitives exists now and those that we need to add
    or integrate... perhaps shame on _me_.

    My intention is positively NOT to denigrate projects that may be
    getting underway or to discourage any kind of discussion of the way
    things could/should work.  My intent is to help us figure out, based on
    specific customer feedback, what we can deliver NOW.  This discovery
    process shouldn't come at the expense of discussing futures or we'll
    miss some great opportunities...

    I believe I'm repeating myself at this point, but the fact is that we
    (Digital) need to close some business with DECmcc _soon_ and we've got
    to do it based on what we can deliver NOW.   Don't you think it would
    be really neat if Digital could articulate clearly and concisely to a
    customer how we can implement all those "check box" capabilities
    they're asking for without resorting to excessive use of future tense?   

    I am in the process of digesting <burp> Carl's response.   I really
    like what I've scanned so far; thanks for posting it Carl.   I think
    this will be very helpfulf if we can keep the discussion going.   I'm
    also very please to see that the TNMP plans will allow for some
    un-bundling; that's great!  I hope that we can capitalize on this.

    /doug

    Okay.  I'll gracefully bow out of this conversation (exit stage left).
    
    I just saw this note going in a direction that scared me a little.
    I know you are keeping the right priorities/requirements/etc. in mind.
    As Keith Roberts would say, "Just checking."
    
    Good luck, have fun, win lots of prizes.
    
    And I'll take my 2 cents back! :-)
    
    -Matt.

	RE: .11,

>>       You really need to define what you mean by Trouble Ticketing. 
>
>    No, I don't.  What is needed is to understand the _customers'_
>    requirements for what _they_ are calling 'trouble ticketing'.  I am not
>    making this vague on purpose; if it was clear to me already I wouldn't
>    have started this whole discussion.

	I didn't want to cloud the issue either and scare people away from
entering comments against this note.  8-)

	I was just trying to clarify things a bit.

>    I am in the process of digesting <burp> Carl's response.   I really
>    like what I've scanned so far; thanks for posting it Carl.   I think
>    this will be very helpfulf if we can keep the discussion going.   I'm
>    also very please to see that the TNMP plans will allow for some
>    un-bundling; that's great!  I hope that we can capitalize on this.

	Yes, we hope to have a generic TT system that can be used for
creating/maintaining tickets and for some tracking purposes as well (the
differences that we see are spelled out more in the func spec, see TAEC::PNMP).

	Anyway, if anyone has questions on this TT FM, please ask and I'll try
to respond!!

	Also, keep those cards and letters coming!  We need all the feedback we
can get to make sure that it is generic enough for a wide variety of customer
needs.

	Carl

    This morning I went over a 'request for proposal' for a network
    management system, and one of the items was trouble ticketing.  That
    feature scores 50 points out of about 1000 you can get on the proposal,
    but never the less, here's what they wanted.
    
    A trouble ticket would be generated when certain alarm conditions were
    met.  This system expected TONS of alarms to be coming in, yet only
    some would be 'trouble tickets'.  When one was generated, reference
    data from the database would be included in the 'ticket record'.  
    An application would allow the person dealing with the ticket to pull
    up history on that same entity, or similar problems.  History would 
    be kept at least 18 months (I would have thought longer).  From the
    trouble ticket application, the technician would be able to initiate
    tests and look at the results while dealing with the ticket.
    
    One system that must be looked at in this discussion is the Remedy
    system.  They have a workstation based trouble ticket generation and
    tracking system that has some nice features.  For example, not only can
    a management system generate tickets, but they have a client/server
    application allowing users to generate tickets.  Their software
    includes client software for UNIX and DOS.  A user calls up the
    application, fills out the problem, and it's submitted to the ticket
    server.  The server has many applications allowing ownership to be
    transfered, report on average time to repair, and all such things.
    
    Trouble ticketing is something MCC needs for many management bids, and
    often it's just being able to check the box saying we have one.  On the
    other hand, there will be many situations where we're asked to
    interface with a customer's existing trouble system, or customize
    MCC's to put tickets in a particular format or provide special
    features.  Don't wait for the ultimate system - get something out now,
    but don't lock yourself into only that one system.  
    
    Warning: Blue-sky alert
    A demo I'd like to see would make use of DECpresent/DECwrite technology
    with standard forms, filled in by use of live links, and incorporating
    compound documents with graphs and images.  The 'trouble ticket' would
    then be forwarded to a FAX machine (using Message Router of course).
    Anyone want me to reserve them a station at DECworld to show this?
    
    bill
    

	RE: .14,

	Whew!  So we didn't scare away any feedback for this note!  8-)

	Thanks, Bill!

>    This morning I went over a 'request for proposal' for a network
>    management system, and one of the items was trouble ticketing.  That
>    feature scores 50 points out of about 1000 you can get on the proposal,
>    but never the less, here's what they wanted.

	What categories were the other 950 points?
    
>    A trouble ticket would be generated when certain alarm conditions were
>    met.  This system expected TONS of alarms to be coming in, yet only
>    some would be 'trouble tickets'.  

	This means to me that some sort of intelligence should eb added to the
front of the system (such as a mediation device) for intelligent pre-processing
of the alarms (such as pre-correlation, etc.).  Also, what kind of alarms will
they have, when do they occur, etc.?  The distribution of the alarms can say a
lot about their system too.

>When one was generated, reference
>    data from the database would be included in the 'ticket record'.  

	OK.

>    An application would allow the person dealing with the ticket to pull
>    up history on that same entity, or similar problems.  History would 
>    be kept at least 18 months (I would have thought longer).  

	OK.  So this would be the person responsible for making sure that the
problem was resolved?

>From the
>    trouble ticket application, the technician would be able to initiate
>    tests and look at the results while dealing with the ticket.

	This would be the person currently working on the problem/ maintenance
personnel assigned by the responsible person (person above)?

>    One system that must be looked at in this discussion is the Remedy
>    system.  They have a workstation based trouble ticket generation and
>    tracking system that has some nice features.  For example, not only can
>    a management system generate tickets, but they have a client/server
>    application allowing users to generate tickets.  Their software
>    includes client software for UNIX and DOS.  A user calls up the
>    application, fills out the problem, and it's submitted to the ticket
>    server.  The server has many applications allowing ownership to be
>    transfered, report on average time to repair, and all such things.

	This sounds like a good system but I am still looking for detailed info
on it.  Do you know if it is object oriented or MCC compliant?
    
>    Trouble ticketing is something MCC needs for many management bids, and
>    often it's just being able to check the box saying we have one.  On the
>    other hand, there will be many situations where we're asked to
>    interface with a customer's existing trouble system, or customize
>    MCC's to put tickets in a particular format or provide special
>    features.  Don't wait for the ultimate system - get something out now,
>    but don't lock yourself into only that one system.  

	This is being designed into our TT FM (the ability for easy
customization/programmability/external interfaces).
    
>    Warning: Blue-sky alert
>    A demo I'd like to see would make use of DECpresent/DECwrite technology
>    with standard forms, filled in by use of live links, and incorporating
>    compound documents with graphs and images.  The 'trouble ticket' would
>    then be forwarded to a FAX machine (using Message Router of course).
>    Anyone want me to reserve them a station at DECworld to show this?

	We can't reserve a station at DECworld but our TT database will be
based on ULTRIX/SQL.  So if live links work with ULTRIX/SQL, you'll be all set.
Also, we will have the ability to generate an ULTRIX e-mail message which
could be forwarded to a FAX or X.400 mail system.

	Carl

It seems that something called "alarm correlation" is very important
for a practical trouble ticketing application.  My intuitive notion of
this term is that "correlation" means somehow reducing a large
(unwieldy) number of alarms (or more fundamentally, events) to a much
smaller number of alarms that are more meaningful or useful.  Is there
a standard definition of the term, and/or can someone point me to some
relevant literature for a discussion of this issue?

	- Andy

Trouble   MANAGEMENT   System

Wish List:

The system should allow for the following minimum activities: Ticketing,
Updating/Tracking, Escalation, Reporting.

Ticketing:
	
	System should allow for manaul and automated ticket input. Reference
	data and/or profiles should be options to ease ticket creation. The
	message here (from experience) is that the entire system must be
	easy to use with as little effort as possible from its users, or it
	WON'T get used.

	Automated tickets from ALARMS is a big plus but you've probably got
	work to do in the ALARMS module and the generic use of MCC itself.
	Case in point, circuit outages. Circuits bounce all the time but
	that doesn't mean I want a ticket every time. If the alarm is OCCURS
	driven I want to see this on the map, but I don't want a ticket yet.
	The ticket would be driven by (x) OCCURS within an interval OR the
	same OCCURS without its compliment event within a shorter interval
	(ie. it bounced too many times within ten minutes or it went down
	hard for two minutes). This is a complex rule expression indeed. If
	its polling alarms we're talking about then MCC must put the polled
	data up for generic FM use otherwise the system load will be so
	great it will never get off the ground. For instance on every circuit
	you would like to alarm on INB %, OUTB %, Error %, Cong. Loss, etc,
	with todays version it takes seperate polls to do each which puts
	an extreme load on the system. This should all be accomplished with
	a single poll.

	Idea: One thing you might want to try is creating a ticket if there
	is an alarm. If the alarm happens again, search your current tickets
	for the same alarm/entity pair, if found UPDATE the ticket. You could
	also put a timer on the ticket which would automatically close if not
	updated after (x) amount of time. This provides the user historical
	info on all alarms without having to input anything. Maybe this
	auto-close could be user activated so there is control over what can
	and will be closed.

Updating/Tracking:

	Along with this previous paragraph the system needs to be able to
	transfer responsibility, whether this be a real internal organization
	actively using the same tickets or pseudo names (ie. TELCO). Updates
	should be able to be seen real time and the user should NOT have to
	wait for a database batch update/archive.


Escalation:

	Escalation should be driven by severity of some sort. You could 
	probably marry the alarm severity to this function or use an input
	field if manually created. If a transfer of the ticket has occurred
	then BOTH escalation profiles (each group) should be notified to
	keep all informed on the ticket status.

Reporting:

	This is where you win or lose. Brief summary reports are needed for
	all or individual groups. Detailed reports of open or closed tickets.
	Reports should be able to be broken down to dated intervals (day,
	week, month). Statistical reports also make management happy. Reporting
	on different fields is a must. For instance, how many circuit
	outages did I have in October. How about a graph showing me the
	percentages of the types of problems worked (ie. 40% circuit, 38%
	network software, etc.). This is helpful for managers to determine
	staffing requirements.


We have used various platforms over the years: trouble tracking, ITS and
we have also looked recently at TARGET. They all seem to do most of the job
but a real winner is one which is easily set up, automates ticket inputing
as much as possible, and provides easy, powerful, and flexible reports (yeh,
right). One thing is agreed though, MCC needs this NOW (ASAP). Market share
will mean everything 18 months from now and these are the types of applications
to get it! Even if it means putting hooks into an already existing system.

best regards,
brad...

	RE: .16,

>It seems that something called "alarm correlation" is very important
>for a practical trouble ticketing application.  My intuitive notion of
>this term is that "correlation" means somehow reducing a large
>(unwieldy) number of alarms (or more fundamentally, events) to a much
>smaller number of alarms that are more meaningful or useful.  Is there
>a standard definition of the term, and/or can someone point me to some
>relevant literature for a discussion of this issue?

	I have not yet seen a standard definition but I'm sure that one exists
or at least a description exists.  I'll try to find one.

	Basically, I think that it also means organizing alarms that are 
received into various logical groupings so that common problems may be spotted
or common equipment may be located.  Also, this means filtering and 
thresholding to decide when tickets get created based on these groupings.

	Does anyone else want to take a stab at this?

RE: .17,

	Thanks for posting that!
	
>	System should allow for manaul and automated ticket input. 

	Can you further define automated input?  DOes this mean to be able to
read the ticket from an e-mail message or other system?  Or Does this mean
automatic ticket creation?

>	Automated tickets from ALARMS is a big plus but you've probably got
>	work to do in the ALARMS module and the generic use of MCC itself.
>	Case in point, circuit outages. Circuits bounce all the time but
>	that doesn't mean I want a ticket every time. If the alarm is OCCURS
>	driven I want to see this on the map, but I don't want a ticket yet.
>	The ticket would be driven by (x) OCCURS within an interval OR the
>	same OCCURS without its compliment event within a shorter interval
>	(ie. it bounced too many times within ten minutes or it went down
>	hard for two minutes). This is a complex rule expression indeed. If
>	its polling alarms we're talking about then MCC must put the polled
>	data up for generic FM use otherwise the system load will be so
>	great it will never get off the ground. For instance on every circuit
>	you would like to alarm on INB %, OUTB %, Error %, Cong. Loss, etc,
>	with todays version it takes seperate polls to do each which puts
>	an extreme load on the system. This should all be accomplished with
>	a single poll.

	I think this was discussed a little bit before and the next release of
MCC will help.

>	Idea: One thing you might want to try is creating a ticket if there
>	is an alarm. If the alarm happens again, search your current tickets
>	for the same alarm/entity pair, if found UPDATE the ticket. You could
>	also put a timer on the ticket which would automatically close if not
>	updated after (x) amount of time. This provides the user historical
>	info on all alarms without having to input anything. Maybe this
>	auto-close could be user activated so there is control over what can
>	and will be closed.

	To me, this falls into the category of alarm correlation.  Also, if an
auto-close is required then the alarm correlation is not working properly.  An
alarm correlation algorithm should be able to decide whether or not a ticket
should be created through filtering and thresholding.

>	Along with this previous paragraph the system needs to be able to
>	transfer responsibility, whether this be a real internal organization
>	actively using the same tickets or pseudo names (ie. TELCO). 

	Yes, this is a key issue/concern.  The real issue I think is to be able
to represent 3 generic levels of management:

	1. Maintenance Manager - This role would be responsible for the overall
	performance of a given trouble ticketing databse.  A maintenance
	manager may be responsible for verifying that each ticket is
	closed within a certain period of time by reviewing
	statistics of the pervious month's performance off-line. One 
	maintenance manager would be assigned to each trouble ticket database. 
	An example of a maintenance manager would be a supervisor in
	charge of a Network Management Center.

	2. Responsible Person - This role would be responsible for the
	resolution of each problem related to the individual tickets
	that are assigned to it. This role may have limited
	responsibility (defined by a Mainteance Manager) of network and managed
	objects using a set of directives related to managed objects and 
	problem tracking.  This role may also encompass quality of service 
	managers. The same responsible person may be assigned to multiple 
	tickets in multiple trouble ticket databases.  Normally, the 
	responsible person would be an operator working within a Network 
	Management Center reporting to the maintenance manager. This role can 
	be contacted regarding the reported trouble.

	3. Current Responsible Person - This role would be responsible for the
	actually working on each problem related to the individual tickets
	that are assigned to it. This role may have limited responsibility 
	(defined by Responsible Person) of network and managed
	objects using a set of directives related to managed objects and 
	problem tracking.  This role may also encompass field service 
	personnel.  The same current responsible person may be assigned to 
	multiple tickets in multiple trouble ticket databases or the current
	responsible person and the responsible person may be the same.  
	Normally, the responsible person would be field personnel working 
	within a Repair Center.  This role can be contacted regarding the 
	current status of the reported trouble.

	These roles represent generic examples of the logical levels of 
responsibility for problem solving and tracking in a typical 
telecommunications network.  There could be no constraints on the assignment 
of names to each of the previously listed attributes, so that the same name 
may be used for all of the roles or different names may be used.  Access 
control based on these different role types could also be imposed.

>Updates should be able to be seen real time and the user should NOT have to
>	wait for a database batch update/archive.

	Definitely!

>	Escalation should be driven by severity of some sort. You could 
>	probably marry the alarm severity to this function or use an input
>	field if manually created. If a transfer of the ticket has occurred
>	then BOTH escalation profiles (each group) should be notified to
>	keep all informed on the ticket status.

	Can you further define what escalation of a ticket means?  Does it
generate e-mail messages?  Or notifications (such as MCC events)?  Or call
someone up on the phone with DECvoice?  Or all three?  Can you further define
what you mean by an "escalation profile"?

>	This is where you win or lose. Brief summary reports are needed for
>	all or individual groups. Detailed reports of open or closed tickets.
>	Reports should be able to be broken down to dated intervals (day,
>	week, month). Statistical reports also make management happy. Reporting
>	on different fields is a must. For instance, how many circuit
>	outages did I have in October. How about a graph showing me the
>	percentages of the types of problems worked (ie. 40% circuit, 38%
>	network software, etc.). This is helpful for managers to determine
>	staffing requirements.

	Definitely, which is why we have made a mapping between ULTRIX/SQL and
MCC for our ticket storage.

	Thanks for everyone's input!!

	Carl

>	Can you further define automated input?  DOes this mean to be able to
> read the ticket from an e-mail message or other system?  Or Does this mean
> automatic ticket creation?

I was initially meant to be auto-creation from alarms. Or profiles could be
defined for specific fields. For instance, if I enter "PKO" into a field for
a "CIRCUIT OUTAGE" then the profile of names/numbers is entered into
appropriate fields saving time on ticket input.

>	Can you further define what escalation of a ticket means?  Does it
> generate e-mail messages?  Or notifications (such as MCC events)?  Or call
> someone up on the phone with DECvoice?  Or all three?  Can you further define
> what you mean by an "escalation profile"?

Profile examples (escalation intervals user definable)

Severity = 3 (or warning)

If no resolution to the ticket (or the ticket isn't transferred) escalate to
support in 8 hours and management in 24 hours.

Severity = 1 (critical)

If no resolution to the ticket (or the ticket isn't transferred) escalate to
support in 2 hours and management in 4 hours.

The method used could be what you've mentioned above mail, beep, event for
additional alarm, etc. So long as I'm not in the escalation path one or more
of these will do :-). Mail is a mandatory requirement. Although I don't have
any personal experience, you may want to look for hooks into DECalert.

It sounds like you're on the right track but it is a difficult module to build
and have a happy majority.


good luck,
brad...

re .19

    Before I start:    This is _not_ a commercial, for Target->Hotline. 
    It's just that I've been recently tasked to turn this stuff on and see
    what we can do with it from DECmcc...   I _am_ a neophyte (_not_ to be
    confused with neanderthal!)  with the Target->Hotline product, but so
    far it looks like a pretty comprehensive application for opening,
    dispatching and tracking trouble tickets, (is that "Trouble Ticketing"
    ? :^) );   It's also pretty  flexible and amenable to tailoring.

    Actually, it looks to me like all of the features that you described 
    in .19 can be accomplished with Alarms FM-generated mail messages, DCL
    and the Target->Hotline software I've been mucking around with.   
    You'd need to hack up the MCC_ALARMS_MAIL_ALARM.COM procedure a bit to
    enter in the appropriate fields in the message.   Target-Hotline greps
    on key fields and uses that data to forward the ticket (along with text
    of MCC mail message) to the appropriate personnel...   Then they can
    start to work on it.

    Target->Hotline has its own escalation mechanisms, but they can be
    taliored a *lot*.  (Actually, judging from what I've payed with and
    seen in the docs, it looks like you can bend their stuff about any way
    you want...)   They provide a DCL interface to their stuff so you can
    grab trouble ticket data from a DCL command procedure (relevant info
    stored in global symbols) and use that procedure to drive other
    notification mechanisms (e.g. DCL interface to Data Collector AM,
    DECalert, DECtalk, mail, OPCOM, etc...)


    /doug

    P.S.

    Good data coming in.  Please keep up the discussions!

    Doug,
    	Will Trouble-Ticket work with both VMS and Ultrix? 
    
    - Mike

I don't know if they plan to port it.

Is it a requirement that the trouble ticketing application work on an ULTRIX 
platform?   

Can we all have a taste of this magic staff Target->Hotline? Can you please
post a pointer to it so we try it also?

About trouble ticketing: I think that an AUTOMATIC GENERATION of trouble
tickets is not necessary and we should not waste efforts in this direction,
at least until we have a good Alarm Corrolation FM. With the current DECmcc
only an experienced human operator can do the corralation between alarms
and create ONE trouble tickets that will take care of MANY alarms.

I think that we should regard trouble tickets as WORK ORDERs. Issuing a 
trouble ticket is the same as issuing a WORK ORDER to fix some problem. 
A manager cannot issue WORK ORDERS automatically. He/She must first understand
the nature of the problem and only then issue the WORK ORDER. Only a rule
driven AI FM may be smart enough to generate TT automatically. Until then,
we better leave this for the human operator.

Peretz

Perhaps I did not make this clear earlier, but Target->Hotline is a *3rd Party*
product sold by a company called "Target Systems Corporation".  

If you want more specific information on their product (I'm a newbie with it...)
I suggest you contact the Product Manager, Jerry Croteau.   
Target Systems is an ISV, so you can send him VAXmail at Easynet account listed 
below:

/doug

Contact: Jerry Croteau, Product Manager
	 TARGET SYSTEMS CORP
	 33 Boston Post Road West
	 Marlboro, MA  01752  USA
	 508-460-9206  fax 508-481-9187

	 VMSmail:  ISVNET::NM%ISVHUB::CROTEAU

re .22

  It is important for Ultrix to have this functionality. This week we are
going to take a close look at OSI NetExpert. It has both the trouble-ticketing
and alarm filtering/correlation, and it runs on Ultrix. We are looking at this
as a place to feed alarms from multiple management systems, including mcc and
msu.

  Scott

	RE: .9,

	Thanks for clarifying the alarm functionality in 1.2!!!

	RE .10,

>    I am not a Network Manager.  But I have read several Marketing reports
>    on what Network Managers want and expect for Network Management Tools.
>    (Granted this was several years ago.  Things may have -- probably have
>    -- changed since then.) The only reply so far that sounds like Trouble
>    Ticketing based upon the Marketing research papers I have read is .8 by
>    Carl Silva.

	Thanks, I think.  8-)

>I would talk to Carl Silva and the PNMP folks
>    before going any further.

	Or the TAEC::PNMP notes file.
    
>    By the way, we are talking big bucks in revenue if and when we
>    implement such a thing.  I believe it should to be tied into an
>    inventory-like system, but now I'm blue-skying -- shame on me!

	Yes, the biggest problem that is now facing us is a lack of 
inventory/configuration information.  But hopefully th customer has some or all
of this information that could be loaded into an MCC database using a bulk
loader (i.e. downloaded and formatted into MCC format to load via the FCL PM).
Then, normal "churn" could be handled using normal MCC updates via the FCL or
Iconic PM.

	RE: .11,

>>       You really need to define what you mean by Trouble Ticketing. 
>
>    No, I don't.  What is needed is to understand the _customers'_
>    requirements for what _they_ are calling 'trouble ticketing'.  I am not
>    making this vague on purpose; if it was clear to me already I wouldn't
>    have started this whole discussion.

	OK, I would like to call it "Trouble Management" then to indicate both
trouble ticketing and trouble tracking.

>    I am in the process of digesting <burp> Carl's response.   I really
>    like what I've scanned so far; thanks for posting it Carl.   I think
>    this will be very helpfulf if we can keep the discussion going.   I'm
>    also very please to see that the TNMP plans will allow for some
>    un-bundling; that's great!  I hope that we can capitalize on this.

	Let me know if I can further clarify anything, Doug.

	RE: .19,

>I was initially meant to be auto-creation from alarms. 

	OK, so alarm crrelation is needed.

>Or profiles could be
>defined for specific fields. For instance, if I enter "PKO" into a field for
>a "CIRCUIT OUTAGE" then the profile of names/numbers is entered into
>appropriate fields saving time on ticket input.

	OK, so we need to specify categories or templates of probem information
to speed up the ticket creation process.

>Profile examples (escalation intervals user definable)
>
>Severity = 3 (or warning)
>
>If no resolution to the ticket (or the ticket isn't transferred) escalate to
>support in 8 hours and management in 24 hours.
>
>Severity = 1 (critical)
>
>If no resolution to the ticket (or the ticket isn't transferred) escalate to
>support in 2 hours and management in 4 hours.
>
>The method used could be what you've mentioned above mail, beep, event for
>additional alarm, etc. So long as I'm not in the escalation path one or more
>of these will do :-). Mail is a mandatory requirement. Although I don't have
>any personal experience, you may want to look for hooks into DECalert.

	OK, so escalation can be defined as a set of timeout profiles depending
on the porblme (such as above).  And if a ticket is escalated, it will run a
command file to send mail, call me on the phone, etc.

	RE: .20,

>    Before I start:    This is _not_ a commercial, for Target->Hotline. 
>    It's just that I've been recently tasked to turn this stuff on and see
>    what we can do with it from DECmcc...   I _am_ a neophyte (_not_ to be
>    confused with neanderthal!)  with the Target->Hotline product, but so
>    far it looks like a pretty comprehensive application for opening,
>    dispatching and tracking trouble tickets, (is that "Trouble Ticketing"
>    ? :^) );   It's also pretty  flexible and amenable to tailoring.

	Where can I find more information on this product?  Is it integrated
with MCC?

>    Target->Hotline has its own escalation mechanisms, but they can be
>    taliored a *lot*.  (Actually, judging from what I've payed with and
>    seen in the docs, it looks like you can bend their stuff about any way
>    you want...)   

	Is this available on ULTRIX?

	RE: .21,

>    	Will Trouble-Ticket work with both VMS and Ultrix? 

	The PNMP product will be available only on ULTRIX/RISC for the first
release.

	RE: .23,

>About trouble ticketing: I think that an AUTOMATIC GENERATION of trouble
>tickets is not necessary and we should not waste efforts in this direction,
>at least until we have a good Alarm Corrolation FM. With the current DECmcc
>only an experienced human operator can do the corralation between alarms
>and create ONE trouble tickets that will take care of MANY alarms.

	Yes, this is true if we make management/creation of the tickets very
easy (profiles for default info, etc.).

>I think that we should regard trouble tickets as WORK ORDERs. Issuing a 
>trouble ticket is the same as issuing a WORK ORDER to fix some problem. 
>A manager cannot issue WORK ORDERS automatically. He/She must first understand
>the nature of the problem and only then issue the WORK ORDER. Only a rule
>driven AI FM may be smart enough to generate TT automatically. Until then,
>we better leave this for the human operator.

	Interesting point.  But I have heard work orders reference requests for
new services i.e. I call up and ask for a new telephone or a new DECnet
connection and they will come by and hook it up.  Whereas trouble tickets
always refer to problems.

	RE: .24,

>If you want more specific information on their product (I'm a newbie with it...)
>I suggest you contact the Product Manager, Jerry Croteau.   
>Target Systems is an ISV, so you can send him VAXmail at Easynet account listed 
>below:
>
>Contact: Jerry Croteau, Product Manager
>	 TARGET SYSTEMS CORP
>	 33 Boston Post Road West
>	 Marlboro, MA  01752  USA
>	 508-460-9206  fax 508-481-9187
>
>	 VMSmail:  ISVNET::NM%ISVHUB::CROTEAU

	Thanks, Doug!

	Are they part of the MCC SVP program?

	RE: .25,

>  It is important for Ultrix to have this functionality. This week we are
>going to take a close look at OSI NetExpert. It has both the trouble-ticketing
>and alarm filtering/correlation, and it runs on Ultrix. We are looking at this
>as a place to feed alarms from multiple management systems, including mcc and
>msu.

	Is NetExpert integrated with MCC?  Are they also an ISV or SVP?  Could
you post a short description here?

	Carl

    re .26

    Neither Objective Systems Integrators (NetExpert) nor Target Systems
    (Hotline) are currently members of the Strategic Vendor Program.

    /doug

re .26

Someone else has already mentioned this but it is worth another word. I
had recommended 'profiles' to ease ticket input. The profile could easily
be 'reference' data.

brad...

    re .28

    Use of the current DECmcc 'reference' data would probably not be
    enough to satisfy a problem profile.  Most organizations have
    alternate contact lists for 2nd and 3rd shifts.  

    Maybe two or three contacts could be listed separated by commas.

    - Mike

re .26 and .27

  A few months ago OSI NetExpert was listed in the SVP status update as
  "contract in negotiation". Has that fallen through?

 Scott

	RE: .29,

>    Use of the current DECmcc 'reference' data would probably not be
>    enough to satisfy a problem profile.  Most organizations have
>    alternate contact lists for 2nd and 3rd shifts.  

	So we need to have profiles that may be applicable depending on the
time of day?  Good idea.

>    Maybe two or three contacts could be listed separated by commas.

	Well, I would prefer to develop a "user profile" so that each user of
the trobuel ticket FM would be registered with the system.

	Carl

re: <<< Note 1764.30 by NSCRUE::KEANE "U.S. NIS Service Development and Delivery" >>>
                           -< what happened to OSI >-

    Please contact Mike Mitchell (DECmcc SVP Relationship Manager for OSI)
    at DELNI::M_MITCHELL  if you need more details on Objective Systems
    Integrators wrt SVP.

    /doug

    
       There is something that people have asked for that I have not seen
    mentioned here.  Large networks sometimes end up with "large" trouble
    tickets.  A "large" trouble ticket is often dispatched to multiple
    groups/teams/people.  The requirement was for the ability to generate
    Sub-Tickets.  It would allow the problem to be tracked easier (and also
    highlight the magnitude of the problem).
    
    -Matt.

	RE: .33,

>       There is something that people have asked for that I have not seen
>    mentioned here.  Large networks sometimes end up with "large" trouble
>    tickets.  A "large" trouble ticket is often dispatched to multiple
>    groups/teams/people.  The requirement was for the ability to generate
>    Sub-Tickets.  It would allow the problem to be tracked easier (and also
>    highlight the magnitude of the problem).

	Do you think that the levels of management defined in the ticket could
handle this same problem?  Multiple copies of the same ticket may be dispatched
but there is still one person/entity that will own the ticket.

	Carl

    Carl,
    
    No.  I don't think that levels of management meets the requirement.  I
    think the pardigm customers are looking for is that of "divide and
    conquer".  That is to say, the ability to break a large, unmanageable
    problem into many smaller, manageable problems.  Also, there would be
    different groups/teams/people assigned to each Sub-Ticket.  But this is
    all conjecture.  Like I said before, I'm not a Network Manager.  Maybe
    a real live network manager type could respond.
    
    -Matt.

	RE: .35,

>    No.  I don't think that levels of management meets the requirement.  I
>    think the pardigm customers are looking for is that of "divide and
>    conquer".  That is to say, the ability to break a large, unmanageable
>    problem into many smaller, manageable problems.  Also, there would be
>    different groups/teams/people assigned to each Sub-Ticket.  But this is
>    all conjecture.  Like I said before, I'm not a Network Manager.  Maybe
>    a real live network manager type could respond.

	But if the network is being monitored, then a major failure such as a
fiber cable cut may result in many alarms being generated in different network
management centers.  This would probably result in many tickets being generated
which would somehow have to be related together.  Would the ability to relate
tickets created in different management domains together serve the same
purpose?  Or the ability to create tickets in a remote management domain and
ralte it back to my ticket in my domain?  In addition, I may forward copies of
the ticket to other management domains using E-mail but I probably would keep a
central copy since someone would have to coordinate the problem and be
responsible for making sure that it got fixed.

	Carl

    
    I've got a few years of NOC hotline experience under my belt, and have
    worked with a bunch of customers over the past few years on issues
    involving problem management.  I've come to learn one thing about
    trouble ticketing systems (trouble tracking, if you prefer) - no matter
    how exact you get it, the people using the system (our customers) will
    always say "I wish it could do XYZ..." and "This isn't the perfect
    system."  I remember saying those exact words to the HUNDREDs of
    customers who came through Digital's Network Manangement Center @PKO.
    I smile continuously when I speak to network managers and hotline
    support managers and hear them say the same thing.  Trouble tracking
    systems are typically closely tied with internal problem management
    processes.  Thus, customer customization is almost always desired.
    
    In any event, I'd like to go back to a comment by Bill Gassman 6000
    replies ago.  Two things are important here:  First, Digital very much
    needs DECmcc to have "Trouble Ticketing" checked off.  It is an
    important selling feature, and allows us to fight the "IBM has NetMan"
    type arguments.  It also provides customers who have no trouble
    ticketing capabilities today with the ability to track problems.  Even
    if the system does not have every bell and whistle, it beats the paper
    tracking mode.  Note that in many cases our "sophisticated" customer
    base already has some kind of formal hotline environment (I.S.
    hotlines, customer services hotlines, etc), and there is a good chance
    they have a trouble ticketing system.  Transitioning from an existing
    trouble ticketing system to a new one is a BIG deal, kind of like
    international relocation.  Users need to learn a whole new language of
    sorts, and in many cases must retrofit their problem management
    function to fit the new system (not a fun thing!).  So, we should
    determine what DEC's initial goals are for trouble ticketing (who is
    the audience).  
    
    Second, we need to consider what the ideal trouble
    tracking system is for the customers who truly need one.  For this I 
    encourage talking to hotline-type people and network managers.  Get a 
    good understanding of the types of issues that come up in general problem 
    management.  Things like:
    
    		* Accurate problem descriptions
    		* Automated ticket numbering
    		* Automated time of day stamping
    		* Ability to update and refer to defined groups
    		* Ability to generate online (mail-able) and hardcopies of
    		  tickets
    		* Flexible (customizable) field names
    			problem type
    			network domain (backbone, site,...)
    			site location 
    			name of person who called problem in
    			name of person responsible for domain
    			.
    			.
    			.
    		* Standard Reports (summaries, detailed, etc)
    		* Customized reports
    			Sort by field
    			Sort by combinations 
    				Report on all circuit problems between
    					OCT 1989 and JUNE 1990 type stuff
    		* Monthly Reports
    			MTTR
    			# problems called in
    			# problems resolved during phone call
    			# problems referred to vendors (and their MTTR)
    				AT&T
    				MCI
    				Digital
    				IBM
    				PEPSI
    				COKE
    				.
    				.
    				.
    		* Escalation
    			Time based (mail message to defined set of mgt. and
    				    technical people when a problem has
    				    been left unresolved for X time period)
    
    			Severity based (Critical problems automatically
    					generate email to predefined list
    					of people.) 
    
    		* Ability to tie in with other databases (ie, caller gives
    		  badge number and name/telephone number fill in)
    
    		* Ability to automatically log tickets from alarms etc from
    		  MCC (and other systems?)
    
    		* Ability to search databases for keywords/fields matching
    		  a particular ticket so a history of problem type X can be
    		  pulled up.  Junior network management/hotline people need
    		  help in troubleshooting, looking at other sources
    		  and similar tickets is a big help:
    
    			* Search trouble tracking db for similar tickets
    			* Search Network Troubleshooting Guide for 
    			  recommended approach
    			* etc.
    			
    I don't have much time these days so I tried to do a quick head dump. 
    I apologize for the randomness and any repeat information from other
    notes.  Keep in mind the points I brought up earlier:
    
    	* Transitioning to new trouble tracking systems is painful
    	* Sophisticated customers may have one already
    	* You'll never satisfy everyone
    	* Determine who the audience is based on the above
    	* Get the basic problem management functionality in place, and let
    	  the super hi-tech bells and whistles evolve.
    	* Understand that trouble ticketing is not so much a technical
    	  feature but a necessary automation function for problem
    	  management.  Easy of use and simplicity is often very important
    	  because of the volume of problems reported.		
    
    
    
    Thanks for listening.
    
    	Mike

    
    
    Another rather important field in a TT system that I inadvertantly left
    out is "Source of problem report"
    
    		Telephone
    		Email
    		FAX
    		In person (with weapon)
    		Net Mgt station (MCC, OPENview, NetView, etc)
    
    
    Mike

    I now have a few copies of the product description from Remedy - their
    Network Healthware line and the product "Remedy Action Request System".
    It has many of the features that Mike mentioned.  Their approach is to
    get their system running on the "big three" unix based netmgt systems,
    creating a standard of sorts so that users can move to another
    management systems without learning a new trouble system.  They are now 
    on SUN, moving to HP, and have been trying to get Digital to join the 
    party.  If anyone is in a position to work with them, contact me for a 
    copy of their literature.  
    
    bill

    
    I like the idea of working with an existing application.  Trouble
    tracking systems are available today and fairly common in terms of what
    problems they attempt to solve.  While this isn't my call to make, I
    imagine the issue isn't so much get a DEC-developed system in place as
    it is get DECmcc to tie into a reasonable system.  Of course, then
    there's the packaging problem.  I assume a resale agreement would solve
    most of the headaches there.
    
    Mike

	RE: .37,

	Thanks for all the feedback/input!  Its great!!

>    I've got a few years of NOC hotline experience under my belt, and have
>    worked with a bunch of customers over the past few years on issues
>    involving problem management.  I've come to learn one thing about
>    trouble ticketing systems (trouble tracking, if you prefer) - no matter
>    how exact you get it, the people using the system (our customers) will
>    always say "I wish it could do XYZ..." and "This isn't the perfect
>    system."  I remember saying those exact words to the HUNDREDs of
>    customers who came through Digital's Network Manangement Center @PKO.
>    I smile continuously when I speak to network managers and hotline
>    support managers and hear them say the same thing.  Trouble tracking
>    systems are typically closely tied with internal problem management
>    processes.  Thus, customer customization is almost always desired.

	Yes, this is the same reaction that we get.

	Also, when you talk about trouble ticketing, is this a problem that is
going to be inter-jurisidictional or intra-jurisidictional or both?

	The biggest problem with customization is there are fields that need to
perform some algorithm and then there are just fields that ned to be
changed/added/deleted but do not affect the basic algorithm of the trouble
ticket.  For example, escalation is a function that requires the prescence of
certain fields to work (a timeout vaue and/or a priority, etc.).  So it might
be difficult to offer easy customization of this sort of a feature (unless you
sell the source code) but it might be fairly easy to be able to
add/delete/change fields that are "customizable".

>    In any event, I'd like to go back to a comment by Bill Gassman 6000
>    replies ago.  Two things are important here:  First, Digital very much
>    needs DECmcc to have "Trouble Ticketing" checked off.  It is an
>    important selling feature, and allows us to fight the "IBM has NetMan"
>    type arguments.  It also provides customers who have no trouble
>    ticketing capabilities today with the ability to track problems.  Even
>    if the system does not have every bell and whistle, it beats the paper
>    tracking mode.  Note that in many cases our "sophisticated" customer
>    base already has some kind of formal hotline environment (I.S.
>    hotlines, customer services hotlines, etc), and there is a good chance
>    they have a trouble ticketing system.  Transitioning from an existing
>    trouble ticketing system to a new one is a BIG deal, kind of like
>    international relocation.  Users need to learn a whole new language of
>    sorts, and in many cases must retrofit their problem management
>    function to fit the new system (not a fun thing!).  So, we should
>    determine what DEC's initial goals are for trouble ticketing (who is
>    the audience).  

	The way that we are dealing with this question in PNMP is to provide a
sytem that is "as compliant as possible" with existing, developing standards in
the OSI/NMF, T1M1.5, and CCITT.  This and maybe a few other features are
providing our "core" set of problem management functionality and anything
beyond this for the time being is "customization".

>    		* Accurate problem descriptions

	Can you expand on this?  How would it be not accurate?  Is this a
user-edited field or a special, mechanized field?

>    		* Flexible (customizable) field names
>    			problem type
>    			network domain (backbone, site,...)
>    			site location 
>    			name of person who called problem in
>    			name of person responsible for domain

	As I mentioned before, I think we need to divide up what is really
customizable from what is customizable if the customer buys the source code. 
It means two different things and levels of work.

>    		* Standard Reports (summaries, detailed, etc)

	What sort of reports are considered "standard"?

>    		* Customized reports
>    			Sort by field
>    			Sort by combinations 
>    				Report on all circuit problems between
>    					OCT 1989 and JUNE 1990 type stuff
>    		* Monthly Reports
>    			MTTR
>    			# problems called in
>    			# problems resolved during phone call
>    			# problems referred to vendors (and their MTTR)
>    				AT&T
>    				MCI

	To me, as long as a standard database is used any report generation
package that works with that database should be able to generate these reports. 
We have chosen to use ULTRIX/SQL for PNMP's trouble ticketing function.  So
some of these reports could be shipped with the product as "examples" or
"sample reports".

>    		* Ability to tie in with other databases (ie, caller gives
>    		  badge number and name/telephone number fill in)

	By this od you mean the ability to retreive default information from
other databses or to be able to validate fields against other databases or
both?

>    		* Ability to automatically log tickets from alarms etc from
>    		  MCC (and other systems?)

	Both of these features would be good to have in any trouble ticketing
system.

>    		* Ability to search databases for keywords/fields matching
>    		  a particular ticket so a history of problem type X can be
>    		  pulled up.  Junior network management/hotline people need
>    		  help in troubleshooting, looking at other sources
>    		  and similar tickets is a big help:
>    
>    			* Search trouble tracking db for similar tickets
>    			* Search Network Troubleshooting Guide for 
>    			  recommended approach
>    			* etc.

	These are tickets that are still in the database and have not yet been
archived, right?

>    	* Transitioning to new trouble tracking systems is painful
>    	* Sophisticated customers may have one already
>    	* You'll never satisfy everyone
>    	* Determine who the audience is based on the above
>    	* Get the basic problem management functionality in place, and let
>    	  the super hi-tech bells and whistles evolve.
>    	* Understand that trouble ticketing is not so much a technical
>    	  feature but a necessary automation function for problem
>    	  management.  Easy of use and simplicity is often very important
>    	  because of the volume of problems reported.		

	Yes, for this reason we have chosen to start with the current drat
standards that are available in this area, participate in the standards groups
related to this area, and then go from there for the future.

	RE: .39,

>    I now have a few copies of the product description from Remedy - their
>    Network Healthware line and the product "Remedy Action Request System".
>    It has many of the features that Mike mentioned.  Their approach is to
>    get their system running on the "big three" unix based netmgt systems,
>    creating a standard of sorts so that users can move to another
>    management systems without learning a new trouble system.  They are now 
>    on SUN, moving to HP, and have been trying to get Digital to join the 
>    party.  If anyone is in a position to work with them, contact me for a 
>    copy of their literature.  

	They sound like they have a good product.  Do they plan on porting
their application to the MCC environment?  Also, which standards are they
currently complying with?

	Carl

    I recently discovered this discussion around trouble ticketing or more
    generally trouble management. Let me bring my own stone.

    I am responsible for an engineering team within EIC-Valbonne named "Data
    Center Productivity" who develops tools such as DCM (Data Center Monitor)
    and CHOPS (Call Handling for Operations). These 2 tools are sold as ASSETS
    and they are the top-selling ASSETS in Europe (several hundreds of
    customers). They are both part of Polycenter set of tools. They are also
    installed internally on over 1500 systems as they are part of the tools
    set used by IS in Europe to manage DEC datacenters.

    We now have 5 years experience in call handling and we regularly meet
    customers using our products to deliver services/projects around them.

    From a general perspective the various steps that you can have in trouble
    management are the following. These steps are already implemented in CHOPS
    which implements the process of following a problem through to resolution.


    1) A problem arises in the computing environment or a user has a need for a
    specific service: he/she then has the possibility to consult a
    problems/solutions database for known solutions and a notice board where
    recent problems are advertized. This avoids multiple calls to be logged fo
    the same cause. Hopefully he/she will solve his problem without requiring
    additional suuport. If not then see next step.

    2) Acquisition of the problem: Also known as call logging, it allows to
    record the problem description with a unique number together with the
    caller's identification (database of callers) either manually through a phone call to a hotline
    desk or by mail sent to a call handling system (External call logging)

    Once the call is logged, the goal will be to assign it to the right
    specialist.The next steps explain how :

    		problem --> related Product --> available Specialist  

    3) Product determination: This step is key as it will allow to assign the
    call to the right support unit. Each unit/specialist being defined by its
    skills and each skill is in fact described as a "product" inside CHOPS.

    CHOPS will know what is the product (the object of the call) implied in a
    call. This can be performed by a human being (the hotline desk) selecting
    the right item within a list of choices or automatically by CHOPS by
    looking to specific keywords or even better by the Text Screen Adviser
    module which uses AI to deduct the product out of a raw problem statement.

    4) Call assignment: Once the product is known, the call will be assigned
    to a queue which can represent a specialist or a support unit. Assignment
    takes into account skills with primary and secondary support specialist
    and availability. The caller can also be notified automatically.

    5) Call notification: Allows to warn the specialist by mail that a new call
    is waiting for resolution.

    6) Call timers: Calls are time-stamped and According to the priority of
    the call, timers are set and alarms will ring the specialist and its
    manager if the call is not worked and/or solved within a delay. 

    7) Call resolution: Once the call is worked on, a status will show if it
    is Read, Pending, Closed... The specialist is able to enter the solution's
    description and to file it.He can also search for historical data in the
    Solutions database by means of free form or Boolean logic textual
    searches. The caller can also be notified automatically of the solution.
    When the call is closed it can feed automatically a Problems & Solutions
    database. When a call is closed it is documented with elapsed time and
    work time. 

    8) Call escalation: if a call has been wrongly assigned it can be escalated
    into another call handling system managed by another organization. The
    call will then be automatically assigned to a new specialist. Once closed
    the call will also be closed in the originating system and the caller will
    be informed.

    9) Re-assign: several specialsist can work in turn on a call. 

    10) Call reporting: A comprehensive reporting package is included which
    provides standard reports that can be sorted on several criteria. Ad-hoc
    reporting and graphic reports can also be obtained with the help of
    DECdecision. Reports are key as they are the basis for quality improvements
    and level of service provided. 

    Optional links to other applications have been developped such as 

    DCM <--> CHOPS. Used to log specific system alarms into Chops or to
    display a message on DCM related to pending calls.   

    CHOPS --> DECalert (allows to phone an operator, to use a bipper or a
    pager...)


    We provide versions running on both VMS (CHOPS) and Ultrix (CHOX).
    The Ultrix version uses DECwindows and SQL.

    Being part of Polycenter we also committed to be over time EMA compliant.
    We started to evaluate how to integrate our call-handling into DECmcc and
    we are already in touch with the PNMP group on this topic in order to
    benefit of their existing alarm ticketing infrastructure and modules.
    Doing so would allow us to release a first EMA version of CHOPS by Mid-1992.

    Laurent
    

>    I recently discovered this discussion around trouble ticketing or more
>    generally trouble management. Let me bring my own stone.

	One of the difficulties in the telecoms industry is that typically call
handling describes is service/customer oriented problems.  Generally, these
problems will occur in telecoms with a circuit oreintation.  Trouble tickting
describes problems that may or may not be network related and may have a
circuit or network element (equipment) orientation.  Does anyone have any
comments on this?  I would prefer to call call-handling a subset of trouble
ticketing and treat trouble management as a seperate issue.

	Carl

>   I would prefer to call call-handling a subset of trouble
>   ticketing and treat trouble management as a seperate issue.

Apart from the vocabulary issue, I see it the following way:

				event
				  |
			  -----------------
			  |		  |
			  v		  v
			filtering	filtering
			by computer	by human brain
			
			  |		  |	
			  | 		  |
			  v		  v
			alarm		problem
			  | 		  |
			  | 		  |
			  -----------------			
				  |
				  |
			    acquisition
			more or less automated
			 using a set of rules
				  |
			creation of a new ticket/call
			includes user id.
			includes alarm/problem data
				  |
			categorizing problem (problem determination)
			using a set of rules
				  |
				  |
			addition to the ticket of supplementary information
			coming from specific databases
			-technical data such as
				.network topology
			 	.connections
				.systems
			-administrative data such as 
				.user environment:applications used
				.support contracts
				.level of service
				.physical location
				  |
				  |
			assignment & transmission to support team
	

    I've just come from a customer meeting where WAN trouble ticketing
    was the topic.  The help desk staff have a system that
    differentiated between help desk calls and trouble tickets. If 10
    users report a failed link then a 'trouble call' would be logged
    for each of these entries but only one trouble ticket would be
    generated.  As 'trouble tickets' are closed the 'trouble calls' list
    is scanned for users who complained about the problem, the user is 
    updated and the 'trouble call' closed.

    So... some customers uses different queues within their trouble
    management systems to track the 'user complaints' and the actual
    trouble ticket.

    - Mike
    

	RE: .44,

>>   I would prefer to call call-handling a subset of trouble
>>   ticketing and treat trouble management as a seperate issue.
>
>Apart from the vocabulary issue, I see it the following way:

	Thanks for the graph, that was a great way to discuss this issue!!!!!

	However, I would add to it (where device may be network elements or
mediation devices):

                                    event
                                      |
                      +---------------+---------------+
                      |               |               |
                      v               v               v
                   filtering       filtering       filtering
                  by devices      by computer    by human brain
                      |               |               |
                      |               |               |
                      v               v               v
                    alarm        alarm/problem     problem
                      |               |               |
                      |               |               |
                      +---------------+---------------+
                                      |
                                 acquisition
                            more or less automated
                             using a set of rules
                                      |
                        creation of a new ticket/call
                         includes alarm/problem data
                                      |
                 categorizing problem (problem determination)
                             using a set of rules
                                      |
                       assignment of person responsible
                          for resvolving the problem
                                      |
             addition to the ticket of supplementary information
          (may be test info, database info, historical info, etc.): <---+
                        -technical data such as                         |
                                .network topology                       |
                                .connections                            |
                                .systems                                |
                        -administrative data such as                    |
                                .user environment                       |
                                .applications used                      |
                                .support contracts                      |
                                .level of service                       |
                                .physical location                      |
                                      |                                 |
               assignment and transmission to support personnel         |
                            by responsible person                       |
                                      |                                 |
                    receive results from support personnel              |
                                      |                                 |
                       results OK?    |   results BAD?                  |
                              +-------+---------------------------------+
                              |
                              v
                          close-out ticket

	RE: .45,

>                    -< Different queues within the system. >-
>    I've just come from a customer meeting where WAN trouble ticketing
>    was the topic.  The help desk staff have a system that
>    differentiated between help desk calls and trouble tickets. If 10
>    users report a failed link then a 'trouble call' would be logged
>    for each of these entries but only one trouble ticket would be
>    generated.  As 'trouble tickets' are closed the 'trouble calls' list
>    is scanned for users who complained about the problem, the user is 
>    updated and the 'trouble call' closed.
>
>    So... some customers uses different queues within their trouble
>    management systems to track the 'user complaints' and the actual
>    trouble ticket.

	In general, this has been our experience in the telecoms industry and
in the standards bodies that are currently developing this work.

	Keep the feedback coming!!!!!!  Any ideas/input/etc. is greatly
appreciated!

	Carl

I see it this way:

                                    event
                                      |
                      +---------------+---------------+
                      |               |               |
                      v               |               v
                   filtering          |            filtering
           by (MEDIATION?) devices    |           by computer (OS?) 
                      |               |               |
                      |               |               |
                      v               |               v
                    alarm             |             alarm
                      |               |               |
                      |               |               |
                      +---------------+---------------+
                                      |
                                   FILTERING
                               BY HUMAN OPERATOR
                                      |
                                      |
                                      V
                                   PROBLEM
                                      |
                                      |
                                      |
>                        creation of a new ticket/call
>                         includes alarm/problem data
>                                      |
And the rest is the same. I still think that it is a human operator who should
be the one that decide to open a trouble ticket. The operators are already 
there, and the system should assist them, not replace them. Anyway, even if you
want to automate the the decision making process, I believe this is beyond
the scope of the PNMP program and resources.

Peretz Gur-El
>

    Filtering by human / Filtering by operator
    
    You have 2 steps.
    
    I am a user and I notice something wrong in my computer environment. 
    I may decide to search for more information and try to solve it by myself. 
    
    1) If I succeed that's fine, I avoided to log a call and may be saved time
    I can also consult a notice board which informs me of current known
    problems.
    
    2) If I didn't succeed, I can either call the hotline or type a mail
    message that CHOPS will automatically analyse and assign to the right
    specialist. 
    
    This is the way we work at Valbonne. No hotline is required any more.
    
    I also agree with Perez's comment on the need for operators. But I still
    think that in the case of "repeat calls" (multiple users calling for
    the same problem) it should be feasible to automatically detect it

>    I also agree with Perez's comment on the need for operators. But I still
>    think that in the case of "repeat calls" (multiple users calling for
>    the same problem) it should be feasible to automatically detect it

It may be feasible, but all I argue is that it should not be high priority
for implementation. Go for it only AFTER you have a trouble ticketing package
that is execellent in doing the rest of the features mentioned previously.

High priority features in my opinion are (not necessarily in this order):

1. Give the user flexibility to change the format of the forms he/she has
   to fill in order create a ticket. I.E - which fields are mandatory,
   which are optional, size of free text fields, the general appearence
   of the form etc. All these changes should be possible to do by a "Senior" 
   user, not by any user.

2. Strong reporting capabilities, directly from DECmcc. Make it easy for the
   user to extract any type of report, *** AND IT SHOULD BE SIMPLE ***.
   The current way of exporting to RDB, then leaving DECmcc and using some
   other reporting package is NOT SIMPLE and NOT FRIENDLY. The method 
   demonstrated by the PNMP demo in TELECOM 91 looks much better!!

   Example for needed reports:

   * How many <tickets type> happened from <date> to <date>
   * Of the above, how many tickets took over <time period> to fix?
   * What was the average time to fix a problem of a certain type? max time? 
     min time? Who are the quickest "problem solvers"?
   * Sort problems by any field: site, severity, responsible person, type
     etc etc and for any time period.
 
3. Graphs - Give the user the possibility to show it in graphs. They love it
   (especially managers). X-Y, Bars, Pie - all of them are needed.
   Graphs are especially useful to show to high level management the trends
   and improvements (?) in the system, i.e. 

   * Average ticket handling time VS date - should show how we are handling
     tickets faster then before.X-Y or bar graphs is needed here.
   * Distribution of tickets type - Pie chart.

   And of course you should be able to print any report or graph that you see
   on the screen..

4. Retrieving of similar tickets - this is great help for the inexperienced
   operator. He should be able to search for a similar cases and see how
   similar problems were solved in the past. As most people DO NOT associate
   KEYWORDS when they open a trouble ticket it should be possible to search
   EFFICIENTLY for words in the free text fields of the ticket.

And please try also to think of those of us who use ancient and obscure
languages (i.e. HEBREW), and try to make it easier for the language conversion
in the future.

Peretz Gur-El
>

	RE: .49,

>1. Give the user flexibility to change the format of the forms he/she has
>   to fill in order create a ticket. I.E - which fields are mandatory,
>   which are optional, size of free text fields, the general appearence
>   of the form etc. All these changes should be possible to do by a "Senior" 
>   user, not by any user.

	How should we handle the case where certain fields are necessary in a
certain format or size or location in order to be able to offer a feature such
as escalation?

	Carl

>>1. Give the user flexibility to change the format of the forms he/she has
>>   to fill in order create a ticket. I.E - which fields are mandatory,
>>   which are optional, size of free text fields, the general appearence
>>   of the form etc. All these changes should be possible to do by a "Senior" 
>>   user, not by any user.
>
>        How should we handle the case where certain fields are necessary in a
>certain format or size or location in order to be able to offer a feature such
>as escalation?

I did not mean that you should give the user complete freedom. It is OK that
some fields will allways be required by DECmcc and the user will not have any
choice about it. But WHENEVER POSSIBLE allow the user to customize the fields
of the report.

Examples: SEVERITY field may allways be there. but the user can customize 
          the default severity. 

	  TROUBLE DESCRIPTION should be an optional field (It may be that for
          simple systems the NAME of the trouble says it all) and the size of 
          this field should be customizeable by a senior user.

In addition - make it an easy process as possible for a user to fill up a 
trouble ticket form:

   * Let the system fill by itself any field that it can fill (date, time
     creator of the ticket and any defaults the user defines).
   * If there is a small number of choices (i.e SEVERITY) use a similar 
     mechanism used for filling ALARM RULES (POINT & CLICK MB2 to expand field).

Peretz 

	RE: .51,

>I did not mean that you should give the user complete freedom. It is OK that
>some fields will allways be required by DECmcc and the user will not have any
>choice about it. But WHENEVER POSSIBLE allow the user to customize the fields
>of the report.
>
>Examples: SEVERITY field may allways be there. but the user can customize 
>          the default severity. 
>
>	  TROUBLE DESCRIPTION should be an optional field (It may be that for
>          simple systems the NAME of the trouble says it all) and the size of 
>          this field should be customizeable by a senior user.

	OK, this is a good point.

>In addition - make it an easy process as possible for a user to fill up a 
>trouble ticket form:
>
>   * Let the system fill by itself any field that it can fill (date, time
>     creator of the ticket and any defaults the user defines).
>   * If there is a small number of choices (i.e SEVERITY) use a similar 
>     mechanism used for filling ALARM RULES (POINT & CLICK MB2 to expand field).

	OK, this makes sense.

	Thanks for the feedback!

	Carl

    User-defined fields.
    One thing that our customers often told us is that they want to be able
    to change the field name and in some instances the field size. They
    change field names to reflect the vocabulary of their company and make
    the new system more friendly and better accepted.
    
    It should not be too difficult to put field names into a separate and
    user-defined table. It is close to what we do for internationalisation.
    
    Field size, specially coded fields might be a problem when you talk to 
    another system. However these systems are within the customer's
    organization and one might think that consistency becomes his
    responsibility. 
    

	RE: .53,

>    User-defined fields.
>    One thing that our customers often told us is that they want to be able
>    to change the field name and in some instances the field size. They
>    change field names to reflect the vocabulary of their company and make
>    the new system more friendly and better accepted.
>    
>    It should not be too difficult to put field names into a separate and
>    user-defined table. It is close to what we do for internationalisation.
>    
>    Field size, specially coded fields might be a problem when you talk to 
>    another system. However these systems are within the customer's
>    organization and one might think that consistency becomes his
>    responsibility. 

	So we need to have a user-customizable TT system that will have three
kinds of fields:

	User-customizable optional - fields that are not required for
					processing.
	User-customizable mandatory - fields that provide some programmability.

	Mandatory - fields that have required names and/or values for
	processing.

	Any comments?

	Carl