Task:	Faults/support
Group:	cc, johnb, bill, lmb
Stage:	1

Description

A division wide user support and problem-reporting tracking system, covering both procedures and technology. Existing procedures at all four sites will need to be studied. New procedures should be flexible enough to suit our diverse circumstances, and should encourage rather than impede the flow of user support information and experience between sites. The task's name comes from the email addresses used for user support at the KB site.

Report changes

2002-03-14

We've evaluated our candidates for message-tracking software.

2002-02-27

We've added a new requirement to the list.

2002-02-19

The list of requirements for message-tracking software has been substantially expanded.

Several minor additions have been flagged with 2002-02-19

added Report Changes section.

added Current/Completed/Future Actions sections.

Issues

Current User Support Arrangements

BP

All help/fault messages go to help@cogsci, or to 504666, which is forwarded to the Duty Officer's phone. In the past Duty Officer has rotated amongst Computing Officers, but it is currently handled by the CSO, Andrew Woods.

There's a home grown remind system that mails the duty officer about messages he hasn't replied to, although doesn't do anything to check that they have been ultimately solved.

Mail to and from help@cogsci is archived, but nothing snazzy is done with these archives to allow users to use them to solve problems. The systems staff aim to distill useful nuggets out of problem solving to put into user documentation (cheat sheets).

There's no fancy fault tracking system installed other than the reminder system.

AI

User base

There's a diverse user base. A fair proportion of the users may have fairly basic computer expertise, especially people such as joint honours AI/Linguistics or AI/Psychology students, and external MSc Meteorology students doing ISC.

Contact

Use of the support@dai alias is preferred; support staff claim that "it hasn't happened if you haven't mailed support". Despite this the majority of emailed support queries seem to go to the personal addresses of whoever has been answering most support mail in the past few years, and a lot of queries are presented by phone and in person. The support alias is normally manned by the CSO, when there is one, based partly at FH and partly at SB.

Faults

As all machine installations/moves/repairs are carried out by computing staff, this is dealt with by support, and so is handled by whoever is deputising for the currently nonexistent CSO. faults@dai mail is handled by the GNATS tracking system. The faults@dai address is rarely used, except by spammers, who generate a remarkably annoying amount of faults traffic. (This is also true of KB's faults@dcs address.)

Second Line Support

Each CO has areas of responsibility - for example printing is covered by John Berry and mail servers by Craig Strachan. This person is not responsible for doing the work, rather for ensuring that it is done. In practice this is rarely formally applied: if the CSO has a problem he just asks the nearest CO, who may in turn contact someone else.

What Is Supported

There isn't a definitive list. It has recently been decided that a few things (for example users' own web servers) aren't supported. Basically if it's thought to be useful then it'll be supported in some way.

KB

There's a Support Office, room 2421 in JCMB, which is open during weekday working hours. Users can go there with queries or fault reports if they think that they need to present their problem in person. There's also a phone number, 505112, which is answered by the Support office staff. The office is staffed in rotation by the CSOs - Lindsey Brown, Carol Dow and Richard Bell - one at a time, or all three at once at the start of session. Users are discouraged from visiting COs' offices with support queries; these queries should be and generally are dealt with by the Support team. User queries emailed to COs are often simply forwarded to support@dcs.

Users are encouraged to look at the Support web pages for computing support information. In particular there's a Support FAQ there which contains answers to many of the common questions which the team is asked. These pages are maintained by the Support staff.

Most users are assumed to be capable of doing a lot to solve their own computing problems, once they're pointed in the right direction, either by the Support team or by the user community in general on the local newsgroups.

There are two mail addresses which can be used to contact the user support team.

We ask people to report broken things to faults@dcs. This mail goes into a GNATS report-tracking system, in which it can be categorised and assigned to a particular person for investigation. First though the Support Office CSO on duty makes an initial attempt to diagnose and solve the problem, asking for more information if that seems to be necessary. If the CSO needs help, he or she can go to the day's "Duty CO", who will help the CSO to investigate problems, explaining where necessary so that the CSO will be able to handle future instances of similar problems without the need for help. The point of this system is to try to ensure that the CSOs can handle much of the user support themselves, freeing up the COs to spend most of their time doing something other than "fire fighting" or solving immediate day to day problems, hopefully development of services and other work.

That's the theory, anyway. In practice the Duty CO system hasn't always worked quite as well as was initially hoped. Partly this has been due to understaffing. The Duty CO isn't always there when needed, and when present is sometimes reluctant to investigate problems in unfamiliar areas (many of the KB COs are specialists in particular areas and don't know that much about other COs' specialisms, and can be reluctant to interfere for fear of inadvertently breaking something). When the Duty CO does know enough to be able to solve a problem, it's sometimes solved without an adequate explanation being given back to the CSO in the Support Office.

GNATS is also not quite as useful as originally hoped; it gets in the way and obstructs quite as much as it helps. Something better suited to our needs would be welcome (see below).

Any problem not solved by the Support CSO or the Duty CO is referred back to whoever is responsible for the software or hardware in question. This person is identified both by broad category (for example George Ross is in charge of networks, Neil Brown of printing, and so on), and by examining the software: every piece of software is installed using an RPM file, which carries the name of whoever most recently created it, so it's easy to identify the person most recently responsible for any particular bit of software. If he or she can't be contacted, urgent problems are put to other COs more immediately to hand, and non-urgent problems are referred to the responsible person for their later attention.

The other email address provided is support@dcs. This is used for all support queries apart from reports of broken things. Mail to support@dcs is read in a normal mail agent in the "support" user account. No special tracking software is used. Inbound and outbound mail in this account is filed and kept, and can be searched by Support staff, but not by users in general.

Some users are confused about which mail address to use, mailing support@dcs when they should have mailed faults@dcs, and vice versa.

Support queries can also be put to local users using local newsgroups - eduni.dcs.questions is for putting questions to the user community in general, and year-based student groups such as eduni.dcs.cs3 are used by students in a particular year to ask computer usage questions of each other.

User Support in the time of DICE

Assistance for users making transition to dice

Users will need clear explanations of the user-visible differences between DICE and pre-DICE environments. Documents describing these differences and how to cope with them would greatly help a lot of users to make the transition.

2002-02-19 It's not clear who will write these. However we note that the Documentation Framework task says that it may be in a good position to do so.

However we worry that some users won't be able to make the necessary adjustments by themselves. Some don't seem to understand enough about the systems - or perhaps lack a grasp of the terminology? - to be able to cope very well when asked to solve their problems using documentation. These people are going to need some personal help in adapting to DICE. Some of their questions may be answerable by email or over the phone. Other computer queries may be far quicker and easier to answer in person, where a screen can be pointed at, commands typed and icons clicked, to demonstrate problems and solutions.

One support service or separate site-based services?

Arguments for keeping the current separate services:

• There will be enough to do in introducing DICE without facing upheavals in user support as well.
• Continuing pre-DICE systems should carry on with the same user support arrangements as before, and DICE support can simply be integrated into those.
• A personal presence, to provide face to face help, will be ensured, since support services would be at each site. People might dislike having to contact unfamiliar user support people at a remote site.
• Sites are used to catering to users of differing expertise: KB people are all computer experts, whereas people at other sites aren't (necessarily) and shouldn't be expected to be.

Arguments for a unified service (whether based at one site or split between several):

• A big change such as the introduction of DICE is precisely the time when it is easiest to introduce other changes at the same time. People will accept a lot of changes all at once better than they will accept a constant drip-feed of smaller but still significant changes.
• One unified service, using the same working methods and software throughout the division, will make it far easier for support staff to take over each others' user support queries, for example in the case of illness, or where one member of staff has the opportunity to spend time on other duties such as supporting a research project. A variety of support services, each with its own software and working methods, would be far more difficult for staff from elsewhere in the division to cope with.
• Some - though not necessarily all - support queries will undoubtedly be simple, routine matters which can be dealt with from anywhere in the division. These don't necessarily have to be handled at the site of origin. The ability to handle such matters from any site, rather than having to handle them from one particular site, should give us economies of scale in the effort needed to handle such routine queries - giving us extra time to devote to more complex matters, or freeing up time from user support altogether and making us more able to help in other areas such as research support.
• Continuing separate support services would probably have different access methods. This would not be confusing for people moving from one site to another. It would also be unfair for those required to use a less convenient method of contacting user support, when people at other sites were able to use an easier method.
• Having one unified support service would give us the opportunity to have one unified piece of software with which to keep track of all the user support queries. This is especially important for a geographically split service where those responsible for particular components of the service need to be able to find out how their component has been performing at every site in the division.

It's worth examining the arguments for separate services though. We think that any change to user support arrangements will be easier to introduce with the introduction of DICE than at some other time. We think that with staff and student mobility around the division being encouraged and we assume increasing, and with the turnover of people over time, most users will find it easier to deal with the support service in the same way no matter where they are located. DICE systems can not be simply integrated into a number of separate support services, at least not without losing significant communication between sites and between user support teams and DICE maintainers. Contrary to belief at other sites, KB users are not all computer experts; many of the staff are mathematicians or logicians and are not particularly interested in playing with complicated software or powerful hardware, and many of the students also don't have the time or the inclination to develop computer expertise beyond the level that's absolutely required for satisfactory completion of their course work.

However we do think that a personal presence at each site would be welcome, if it can be managed. There are always some support queries which require or greatly benefit from face to face contact; for example, it's sometimes far easier to understand or explain support queries with a personal demonstration than with a long complicated description.

What software handles user support queries?

• If queries and fault reports are handled separately, a message that comes in as a query often turns out to really be a fault report, or vice versa. Messages have to be shuffled back and forth between the two systems. Far easier to simply abolish the distinction and treat all user support messages the same way.
• One of the benefits of tracking software is the ease with which a comprehensive record of support correspondence can be built up and analysed or searched. Someone trying to find the solution to a problem must be able to search to see if the same or a similar problem had been encountered before. This will be especially necessary given the geographical split of the division which will persist for some years to come - a problem reported at site A, solved by someone at site B, may be directly relevant to someone at site C several months later.

2002-02-19 A previous DICE meeting pointed out that any move to make User Support correspondence public would have to be accompanied by a prominent warning at the point of user submission of correspondence (the user support web input form). There was general unease about making the user support records public. Perhaps user-to-user help/support systems (newsgroups, a user-maintained FAQ) would be more appropriate.

In what form(s) should we accept support queries?

Currently support teams are contacted by email, by phone or in person.

Whilst we accept the advantages of a web form, we have a number of questions and observations:

• How strictly would we expect users to use the web form? Would use of the web form be enforced for all user support matters? Would it in other words mean the elimination of putting queries to Support staff in person or by phone, and perhaps the closure of Support offices (since if one isn't allowed to seek help in person from the people there there seems little point in them)? Or was it simply envisaged that a web form would replace the current reporting of queries/problems by email?
• A web form would have to strike a balance between prompting for details, and being quick and easy to fill in. It's possible that some problems which are currently easy to report (a one line email message can be composed and sent very quickly), such as a broken computer in a student lab, or a printer running out of paper, would be distinctly less quick and easy to report using a web form, and would therefore be less likely to be reported. Simplicity would be a definite virtue. (However, a problem which is not important enough for someone to go to the effort of filling in a few fields on a web form is arguably not important enough to be tackled by the support team?)
• Might it be advisable to give users the opportunity to ask quick questions of user support people in person? The user support people would then be able to respond either with a request for the user to use the web form, or (if appropriate, for example if it was a particularly frequently asked question) to give the user a quick pointer to a well documented solution. Forcing both the user and the support person to go through the formality of the web form and the fault-tracking software for every instance of a very frequently asked question might perhaps not be the most efficient use of their time? Unless personal contact with Support staff is to be eliminated, see above.

2002-02-19 We spoke to a CEG member and he confirmed that the more "casual" interpretation was the correct one: it's envisaged that people will still be able to have face to face contact with user support staff, and will still have the opportunity to put questions to them in person and by phone. If the problem can't be dealt with immediately, the support person may then request the user to fill in the web input form so that the Support team has a proper record of the matter as a reminder. The support person may even fill in the form himself/herself. In other words, the web input form will merely replace email as the way to officially register a problem/query.

How can we make systems service announcements to people?

Many (but no all) KB-based users are used to using newsgroups for local mass communication, and read it regularly. However local usenet news seems to have more or less died out at other sites.

Alert messages are a familiar form of communication at some sites, but at other sites they're unknown. Any alert message system would have to cope with the wide variety of login environments (well, X window managers) likely to be in use on DICE machines - past alert message systems may have relied on all users using the same login environment, in which an alert message pops up on startup. Alert messages would also not be seen by people who stayed logged in for long periods of time, unless they made a special effort to check them from time to time.

Use of email for general announcements may be acceptable at some sites but certainly not at all of them - it's greatly resented by some KB users, for instance, who class it as junk mail and actively complain about it.

In our experience web-based bulletin board systems can suffer badly from rot, as their contents grow older and older and are never expired.

We wonder if we can use local newsgroups for announcements, but provide a web interface to them for the people for whom use of a conventional newsreader might be a daunting or tediously involved process?

2002-02-19 This was raised at a previous DICE meeting. It was noted there that Divisional policy is for members to be able to receive announcements in whichever one of several formats they prefer. Will this be implemented? How will we handle announcements concerning DICE machines before we have such a system? Mail all systems announcements widely to mailing lists? Set up a systems announcements web page?

How can we make announcements to just one site?

2002-02-19 Do we have an answer to this question?

Support for "legacy" or pre-DICE systems

The pre-DICE computer systems will still have significant numbers of users for two years after the introduction of DICE. User support services for them will still be needed. How should people get at these services? The same ways that have been used before? Or should we offer just one route into user support for all systems, DICE or pre-DICE?

The latter would perhaps be simpler to operate, as it lessens two of the main worries which occurred to us as we started looking into this area:

1. How will a user, no matter how inexperienced with computers, be able to tell the difference between a DICE computer and a pre-DICE computer?
2. If there are different ways of accessing support services for different sorts of computers, would people remember which was which? Experience suggests that a large minority would be either confused by such a distinction or would deliberately ignore it, leading to a situation where users at one site might start reporting DICE problems in one way while users at another site might report them in a different way. This would not only be unfair but would lead to a serious loss of user support communication between sites.

Message-tracking software

Whatever software is used should have certain characteristics. Some of these reflect our long experience of the GNATS system which has been in use at KB, and to a lesser extent also at SB and FH, for several years.

In order to explain exactly what we need of the software, we need to explain the characteristics of the problems which the user support services will have to deal with. Most problems are either simple or complex:

simple problems

are those which take a trivial effort to resolve fully. They can generally be processed by the support staff as soon as they are encountered. They tend to be about the same subjects week after week - for example simply answered queries, or requests for consumables to be replenished. Once dealt with, they are filed away and forgotten.

complex problems

are more difficult or time-consuming to resolve. If the user support person has difficulty resolving the problem, they may seek help from somebody else. The problem may end up being passed between several people before a solution is found. Email relating to the problem may flow back and forth copiously. Message-tracking software needs to provide a way to classify these problems into areas; to note the people responsible for investigating them; and to be able to present all of the messages relating to the problem together in the one place. It is particularly helpful when trying to solve a complex problem to be able to search through a complete archive of earlier such problems and their solutions.

• allow, and track/file/classify, discussion of the problem by email.
• also file and classify all replies concerning the problem sent by the user support people using the software.
• for follow-up messages, where it is clear exactly which problem a message relates to, the software should automatically file the new message in such a way that it is associated with the series of messages it follows up.
• for follow-up messages where it is not clear exactly which problem a message relates to, the software should give the user support person the opportunity to file the message in the correct place. The software should not assume that the message is a report of a new problem, for instance.
• for simple problems, the software should let the user support people deal with (reply to, classify, file and forget) them very easily and rapidly. Much user support mail consists of simple problems reported with one line messages. The tracking software must be so quick and easy to use that it does not get in the way here - preferably as quick as using a decent mail agent.
• allow, though not require, manual control over its actions. Our KB GNATS installation tends to forge ahead and do things automatically, and often gets it wrong.
• have good facilities for putting things right again once they've gone wrong. A big UNDO button, for instance. KB's Gnats doesn't have this, and too many of its operations are not reversible. Click the wrong button and you're in deep trouble.
• be very configurable. KB's Gnats has a load of facilities we don't use, and as a result the screen gets filled with irrelevant junk, and our method of working has to be bent round the assumptions made by the designers of the software. We want a system which only does what we want it to do, and doesn't impose its own structure based on somebody else's incorrect assumptions about how things should work.
• be capable of sending out reminders to fault-fixers now and again, so that those responsible for fixing problems don't just file and forget them.
• preferably allow users to follow the progress of the resolution of their problems. It would be nice if this could be handled in an automated fashion. (Although there's not much point in this if it doesn't give real, useful information; a bald "received" or "assigned" status message tells the user very little, and can be more frustrating than no information at all.)
• not impose filing systems and categorisations on us to such a degree that we're forced to file/categorise things, then forget where we put them. Again, we want control over it, not it over us.
• refrain from sending too much mail. We need a system capable of handling high volumes of support queries without overwhelming support people or users in unnecessary extra correspondence.
• automatically file all correspondence, and in the correct place, so that related information may all be found together. The KB GNATS system is dependent on everyone remembering to send all email to faults@dcs, and also on all messages having the correct subject header. Both are frequently forgotten, so large parts of conversations can be missed off the record.
• be immune to spam, or protected from it. The GNATS systems at SB and FH and KB all receive lots of junk mail, and automatically reply to it too! It's irritating and wastes our time. We'd like a system which doesn't accept junk mail and never replies to it.
• be interoperable with the bug-tracking system to be established for local DICE software by the software management task, to ease the transfer of problem reports from one system to the other.
• It is essential that full records of all problems be kept in such a way that they build up over time into a comprehensive archive of all problems encountered, their symptoms, their circumstances, their solutions. Furthermore, the software must allow support and systems staff to make sophisticated and accurate searches of this archive. (This is Gnats' main good feature.)
• As previously noted, lots of problems or queries are very simple matters, quickly dealt with and occurring again and again. The software will have to provide us with a way of excluding these simple problems when searching our archive. If we don't have this then any search may bring up so much cruft (to use a DICE buzz word) that it'll be difficult to find the useful items in a sea of small and trivial ones.
• We will be inundated with lots of fairly simple queries that require minimal "paper work" to sign off on a system. With these we really don't want to fill in any kind of complicated form, as it will get in the way and any destinctions we make about jobs will be largely meaningless. We must have a very simple, quick sign-off procedure for simple, quick problems.

Evaluation of software

• GNATS
• Bugzilla
• Request Tracker
• Call Management System

Currently we have noted that the GNATS software (even the newer versions) fails miserably against a large number of these criteria. It seems far more likely that we will go for other software with a stronger match with the metric above.

RT, on the other hand, meets many of the above. Notable exceptions are out of the box interoperability between itself and most bug tracking software (e.g. Bugzilla, which is likely to be the Software Management task's favoured bug-tracking system for DICE). However transfer of a report from one to the other ought in theory to be a simple matter of exporting from RT's SQL database and importing to Bugzilla's SQL database.

RT (Request tracker) itself doesn't have a front end for users to submit queries to (in the strictest sense), but there are a few configurable frame-based front ends out there. And it's all GPL. We have noted that we'll most likely be creating a web front end for submitting mail to anyway.

One recent offer of help has informally come from EUCS, and the maintainer there has offered us the chance to evaluate it for ourselves. More info when we've done that. The software is in-house, and called Call Management Software (CMS). So far our attempts to arrange a visit to see it have not met with success. However one can get a flavour of its facilities at the CMS web site.

Evaluation of message-tracking software

Things we must have

Things we'd very much like to have

And we should also take into account:

Summary

GNATS has too many negatives in our "must have" table to be considered further.

CMS turned out to be a lot more interesting than we had expected. It seems like a good, useable system which does what it was designed to do very well. The CMS team have done a good job, and the EUCS user support teams have a great tool to use. (We'd like to thank the CMS team here for their wonderful helpfulness when showing CMS to us. Thanks!)

However - you could tell there was a "however" coming, couldn't you - the job which CMS was designed for isn't quite the job we want our software to do. CMS is designed around EUCS working methods and patterns of data flow; it would be awkward to bend it into the slightly different shape which we would want. There's also the discouraging need to shell out money for it (however small the cost turned out to be), and there's a slight question mark over adopting software which is so firmly under the control of one department's management. That it happens to be EUCS is irrelevant; the point is that the other packages are all released under public licences and have some degree of global user community backing them. CMS doesn't. Those are both relatively minor points, however; the main reason we haven't chosen it is simply that it promised in several ways to be more awkward to use than its competitors here.

So the choice comes down to Request Tracker ("RT") or Bugzilla. Both seem to perform well in most areas. Both lack a way of dealing very quickly with trivial calls, which is slightly worrying. Our testing will reveal how much this actually matters. Of the two, RT seems to have the smaller user community. But its glorious flexibility, and its ease of configuration, combined with its having been designed from the ground up as a general purpose message tracking system (rather than specifically as a bug-tracking system) gives RT the edge as far as we're concerned.

Actions

Current

Future

web form spec

Specify what will be needed in a web form for the reporting of user support problems. Our initial attempt suggests that it's going to be hard to find a balance between soliciting useful detail and achieving the desired level of simplicity. One thing we're pondering is the possibility of automatically filling in certain fields based on the contents of other fields, for example deducing machine location from machine name.

Choose message-tracking software

implement web form

install and configure message-tracking software

test message-tracking software

test web form

decide on running arrangements for message-tracking software

who operates it, when, how, where, etc.

Completed

2002-02-19

Firm up the list of requirements for message-tracking software

2002-03-15

Survey of available user support message-tracking software

General DICE Comments

Need for Transition Documentation

Publicity

We suggest the need for a strong and loud information campaign. It should be aimed at the whole division, and it should be done sooner rather than later. We should say simply and clearly what DICE consists of and what it does not consist of. We could perhaps include a FAQ which tackles basic questions head on (such as "Is DICE a CS takeover?"; "Will backups be worse under DICE?"; "Shall I have to learn a whole new operating system?")

2002-02-15 we've now started a good information campaign - well done. It seems that we may have judged peoples' attitudes to DICE wrongly: actually, most people seem to be completely uninterested! Either that or the DICE web site is so excellent that it answers all of their questions, meaning that they had no need to turn up to any of the DICE publicity events.

Expertise

2002-02-19 Efforts to give all DICE developers access to Linux machines now seem to be under way.

Visits to Other Sites

Dependencies

 : Deploy 

Please contact us with any comments or corrections. Unless explicitly stated otherwise, all material is copyright The University of Edinburgh