DICE Deployment: Stage 1 Progress

.........................

home

units

doc

.........................

Stage 1 Progress

This page provides an overview of progress for individual tasks for Stage 1

These are separated into two parts:

tasks related to deployment of the DICE infrastructure
tasks related to transitioning from the legacy systems

Some tasks span both areas and but have been allocated to only one area, according to the relative amount of work.

Deployment Tasks

Account management procedures
Account management technology
Applications
Authorisation
Backups
Beowulf
Default and group environments
Documentation
Faults/support
File-system maps
File-system issues
Infrastructure
Kerberos infrastructure
LCFG technology
LDAP infrastructure
Linux platform support
Linux platform (portables)
Mail issues
Mail hub
Network
Printing (both client and server)
Root mail collation
SSH public key distribution
Software management
Technology for editing configuration files
User web space

Transition Tasks

Applications at Buccleuch Place
Applications at Forrest Hill/South Bridge (see Applications)
Common home directories
Legacy system issues: BP
Legacy system issues: FH/SB
Legacy system issues: KB
User account transitional arrangements

Deployment Tasks

Account management procedures

Goals	Status
Produce documented procedures relating to account management	Initial drafts complete

Milestones Due Status

Initial versions of procedures needed 2002-01-18 Complete

Finalise interface to datasources with AMT task 2002-02-08 Complete

Revise procedures based on AMT interface 2002-02-15 On going

Implement any scripts required by procedures 2002-03-31 Done

Get some decisions about disk/web/print quotas 2002-09-01 Not started

Documentation on the procedures 2002-07-01 Started, part of report

Milestones	Due	Status
Initial versions of procedures needed	2002-01-18	Complete
Finalise interface to datasources with AMT task	2002-02-08	Complete
Revise procedures based on AMT interface	2002-02-15	On going
Implement any scripts required by procedures	2002-03-31	Done
Get some decisions about disk/web/print quotas	2002-09-01	Not started
Documentation on the procedures	2002-07-01	Started, part of report

Actions Time
Required Due Status

Initial draft procedures 3 days 2002-01-18 Done

Finalise interface to AMT tools 3 days 2002-02-08 Done

Decide which primary and secondard groups are required. Clash with filesystem issues task? 1 week 2002-04-04 Decision on 2 primary groups, secondary groups ongoing

Update procedures docs based on AMT interface 3 days 2002-07-30 Started

Check how /yesterday, /risky, /public (or whatever they're to be called) will be managed Done

Diskquotas which tasks responsible Done

Find out what the EUCS do with staff who then start a PhD or MSc at the University. Two UUNs? ½ a day Done.

Cross site meeting on quotas Done.

Clarify the role/mechanism for pseudo users 2 days 2002-05-14 Done

Identify the scripts that need written. 1 day 2002-05-20 Done

Script: update_users 1 day 2002-06-12 Done

Script: update_user 1 day 2002-06-12 Done

Script: create_users 1 day 2002-06-12 Done

Script: new_passwd 1 day 2002-06-12 Done

Script: get_users 1 day 2002-06-12 Done

Script: delete_users 1 day 2002-06-12 Done

Actions	Time Required	Due	Status
Initial draft procedures	3 days	2002-01-18	Done
Finalise interface to AMT tools	3 days	2002-02-08	Done
Decide which primary and secondard groups are required. Clash with filesystem issues task?	1 week	2002-04-04	Decision on 2 primary groups, secondary groups ongoing
Update procedures docs based on AMT interface	3 days	2002-07-30	Started
Check how /yesterday, /risky, /public (or whatever they're to be called) will be managed			Done
Diskquotas which tasks responsible			Done
Find out what the EUCS do with staff who then start a PhD or MSc at the University. Two UUNs?	½ a day		Done.
Cross site meeting on quotas			Done.
Clarify the role/mechanism for pseudo users	2 days	2002-05-14	Done
Identify the scripts that need written.	1 day	2002-05-20	Done
Script: update_users	1 day	2002-06-12	Done
Script: update_user	1 day	2002-06-12	Done
Script: create_users	1 day	2002-06-12	Done
Script: new_passwd	1 day	2002-06-12	Done
Script: get_users	1 day	2002-06-12	Done
Script: delete_users	1 day	2002-06-12	Done

Goals	Status
Provide a Perl interface to encapsulate a user account and its management	Done
Synchronising mechanism between updated Database info and LDAP data	Started

Milestones	Due	Status
ACLs for access to KDC	2002-10-01	Deferred - not a show stopper
Kerberise Database access	Stage2	Deferred - not a show stopper
Decide where data should be stored and mastered from	2002-02-01	Done
Create an Account Perl object	2002-04-01	Done
Create an AccountManager Perl object	2002-04-20	Done
LDAP/Kerberos/DB consistency checker/propagation mechanism		Mostly complete
Tools for disk quota manipulation	2002-05-07	Delayed until minimal client is done.
Documentation	2002-07-01	Partial, The perl modules are self documenting

Actions	Time Required	Due	Status
In co-operation witht the AMP and LDAP task, finalise data we need to modify/store in the LDAP/Kerberos/DB		2002-02-05	Done
Give LDAP task our requirements		2002-02-19	Done
Complete Account.pm object	2 days	2002-03-31	Done
Complete datasource: Algorithmic.pm	2 days		Done
Complete datasource: DataSource.pm	2 days		Done
Complete datasource: Infdb.pm	1 day		Done
Complete datasource: Kerberos.pm			Done
Complete datasource: LDAP.pm	1 day		Done
Complete datasource: SMS.pm	2 days		Done
Complete datasource: UidServer.pm	2 days		Done
Complete LocalAcctManager.pm	2 days		Done
Kerberos Management Perl module			Done
Agree with Authorisation task how account disabling will work	1 day	2002-03-15	Done
Documentation, an overview	3 days	2002-07-01
Check how pseudo accounts will work in practice	2 days	2002-05-14	Done
Get required support RPMs for dice-accntmgr installed on the system	1 day	2002-05-14	Done
Private LDAP comments attribute in Person object		2002-06-12	Done
AccountEnable/Disabled LDAP attribute in Person object to be added		2002-06-12	Done
Finalise LDAP/DB synchronisation	2 days	2002-07-30	Done
Get LDAP/DB synchronisation to run automatically	1 day	2002-09-30	Need some Kerb guidance
quota management mechanism		2002-08-30	Started

Milestones	Due	Status
List of applications available under DICE	2002-07-31	Started
List of applications required for Teaching	2002-07-31	Started
Applications required for Teaching installed	2002-08-30

Actions	Time Required	Due	Status
Collate lists of available DICE applications	1 week	2002-07-31	Started
Collate lists of courses and applications required/available (teaching only)	1 week	2002-07-26	Started
Establish priorities for porting		2002-07-26	Started
Documentation for non-KB staff about porting s/w & RPMs		2002-07-26	Started
Get VMware working (carwyn)	1 week	2002-08-15	Started

Goals	Status
User authorisation system (not host)
Access control from all services under DICE
Varying levels of control granularity possible

Milestones	Due	Status
Install on web server	2002-07-15	Started
Install on mail server	2002-07-15	Done

Actions	Time Required	Due	Status
Create amanda DICE RPM with config	2	2002-07-15	Done
Create component	2	2002-07-15	Started
Debug amanda daemon problems	2	2002-10-15	Started

Goals	Status
Provide Basic clusters.
Provide Myrinet networking on 16 node cluster.
Provide full PBS service on both clusters.

Milestones	Due	Status
Source and purchase nodes		DONE
Source and purchase networking kit.		DONE
Source and purchase KVM system		DONE
Source and purchase racking and electrics.		DONE
Configure HP switch.		DONE
Setup first 16 nodes of 64 node cluster.	ASAP	Partially completed.
Setup 16 node cluster.	ASAP	Partially completed.
Setup second 16 nodes of 64 node cluster.	ASAP	Partially completed.
Setup full 64 node cluster.
Setup full 16 node beowulf	23 July 2002	Partially completed
Setup Myrinet on 16 node cluster.
Install PBS on 16 and 64 node clusters.

Goals	Status
Provision of a suitable, configurable shell environment for 'real' users.	DONE
Provision of a simplified, configurable shell environment for pseudo users.
Provision of a suitable basic X setup for 'real' users.	DONE

Milestones	Due	Status
Modify Bash to ignore .bashrc, .bash_login and .bash_profile.		DONE
Package patched Bash in new RPM.		DONE
Finalise role/location information to be held in LDAP database.		DONE
Arrange for the inclusion of this data in the database.		DONE
Develop mechanism for extracting data from LDAP query results.		DONE
Rewrite Main.bashrc, Main.profile and Environment for DICE.		DONE
Indentify RPMs which contribute rc files and rewrite them as necessary.		DONE
Identify 'orphan' rc files (i.e. those not belonging to an RPM) and rewrite as necessary.		DONE
Create new defenv.bash RPM.		DONE
determine contents of KDE desktop and K menu.		DONE
Rewrite system .xinitrc and KDE skeleton files to provide this environment.		DONE
Update KDE RPMs.		DONE
Rewrite pathfix config file for DICE.		DONE
Create new pathfix RPM.		DONE
Package up config		DONE

Actions	Time Required	Due	Status
Check/rewrite existing env.* and profile.* files	three days	2002-05-14	DONE
write new env./profile. files as required by roles	Waiting on final list of roles		DONE
User documentation			IN PROGRESS
Technical documentation			IN PROGRESS

Actions	Time Required	Due	Status
Prepare template for user document	hours	2002-06-02	Done.
Complete script to auto-generate the docs page for the dice-* components and add to crontab	hours	2002-05-28	Waiting for rpmcfg rejig.
Write guidelines to writing documents			Done.
Collate documentation requirements from task leaders		2002-05-28	Done.
Prepare a timetable for production of documents as above.		2002-05-28	Done.
Chase up content for introductory user doc			Done.
Add link to LCFG documentation from www.dice.informatics.ed.ac.uk/doc			Done.
collate doc HOWTOs from postings to cos@inf			Done.
JAVA user environment docs		2002-07-26	Started
Prepare a new informatics/system page		2002-07-26	Done

Actions	Time Required	Due	Status
Trial cable runs on one corner of racking.	4 days	2002-04-01	Partially completed
Install trunking on racking.	1 week	2002-04-08	Trunking sized and sourced.
Install cabling to PC's on shelves 1 and 2.	1 week	2002-04-18	Ongoing
Install OS on 1st 16 nodes and check for heating problems.	4 days	2002-04-28
Install OS on 2nd 16 nodes and check for heating problems	4 days	YYYY-MM-DD
Install OS on 2nd 16 nodes and check for heating problems	4 days	YYYY-MM-DD
Run cabling to 16 node cluster.	4 days	Almost Completed (13/16)
Install OS on 16 node cluster.	1 day	Almost Completed (14/16)
Configure mirror server as interim file server for 16 node cluster.	1 day	Completed.
Install cabling for shelves 3 and 4	4 days	YYYY-MM-DD
Insall OS on remaining 64 node cluster nodes	4 days	YYYY-MM-DD
Setup and Install fileserver.	4 days	YYYY-MM-DD

Goals	Status
Provide a structure for producing and presenting documentation for both users and system staff.	DONE
Automate the method of producing documentation as far as possible.	DONE

Milestones	Due	Status
Agree URLs		Done.
Get DICE Component documentation fully automated.		DONE
Complete a new informatics/system page.		DONE

Goals	Status
Effective procedures and structure for the division's user support services.	DONE
Provide relevant and helpful message-tracking technology for user support.	DONE

Milestones	Due	Status
Agree structure for user support for DICE.		Done.
Working procedures decided.	2002-07-31	In progress.
Message-tracking software working.	2002-07-31	In progress.
Web input form working.	2002-06-07	Done.
Training and familiarity with software and procedures completed.	2002-07-31	DONE
Produce user documentation.	2002-08-15
Produce system documentation.	2002-08-31

Actions	Time Required	Due	Status
Discuss options for structure of user support in next few years	ages		Done.
Find candidate message-tracking packages			Done.
Evaluate message-tracking software candidates			Done.
Get test message-tracker up and running	a day or two	2002-04-23	Done.
implement basic web form	days	2002-04-30	Done.
Get real message-tracker in place	a day or two	2002-05-17	Done.
connect web form to message-tracker via dcs.ed.ac.uk mail server.		2002-05-17	Done.
connect web form to message-tracker via inf.ed.ac.uk mail server. (Waiting for inf.ed.ac.uk mail server.)
RT configuration: Get RT to send out email stamped with correct per-ticket addresses.		2002-07-31	In progress.
RT configuration: Get inf.ed.ac.uk mail server to accept per-ticket addresses and inject them into rt-mailgate.		2002-07-31	In progress.
RT configuration: Get rt-mailgate to accept per-ticket addresses, recognise ticket numbers from addresses, add mail to correct ticket.		2002-07-31	Done.
RT configuration: Stop rt-mailgate from parsing Subject lines.		Not urgent.	Not started.
RT Configuration: add an escalation system (which catches new messages which have been ignored for a certain number of working hours, then nags Support people about them). Not so urgent.			In progress.
RT Configuration: disable auto-replies to new tickets.			Done.
Detailed working procedures		2002-07-31	In progress.
Get Informatics DB people data into RT. (InfDB data goes into LDAP, then we query LDAP from the web form, reject web form input if not a valid DICE user, report details to RT otherwise.)		2002-05-31	Done.
full scale tests and adjustments		2002-07-31	In progress.
Training (to be done when RT config and testing completed.)		2002-07-31
Integration of DICE LCFG inventory with RT. This needs either of the following: an Informatics machine inventory SQL database, fed from LCFG. The dcs.ed.ac.uk SQL inventory web form lookup code can be reused for this. an XML inventory parser perl module, to parse the existing XML-based inf.ed.ac.uk machine inventory file generated from LCFG `inv.*` resources.		2002-07-31	Done.
Integration of DICE LDAP host list with RT, as stop-gap until inventory database link can be set up.		2002-07-31	Done.
Integration of JCMB legacy LCFG inventory with RT.		2002-07-30	Done.
Integration of SB/FH legacy inventory with RT.			Not started.
Integration of BP legacy inventory with RT.			Not started.
Link DICE Bugzilla with RT. (Not necessary to start with. Will revisit once RT up and running.)
Add "send me a copy" tick-box to web form.			Done.
Ruggedise the Apache configuration - i.e. make sure it can easily survive the machine blowing up. Mostly already done since Apache rpm and LCFG component are already in use.			In progress.
Ruggedise the MySQL configuration. MySQL rpm already there and in use.			In progress.
Ruggedise the RT installation and configuration.			In progress.
Documentation: description for sysmans of RT config, focusing on disaster recovery.		2002-08-31	Not yet.
Documentation: for support people on how to use RT, including cross-site working procedures.		2002-08-31	Started
Documentation: for a general audience, describing the computing support service.		2002-08-15	Started

Goals	Status
define set of filesystems/mountpoints for DICE clients/servers	DONE
provide means of creating/generating/storing filesystem maps	Almost done
provide means of distributing maps	Almost done

Milestones	Due	Status
clarify all filesystem needs (input from Common Home Directory and Filesystem Issues)		DONE
specification of filesystem structure/naming		DONE
integration of automounter with map distribution method		DONE
specify sources of map data (requires policy on source management)		DONE
technology available for creating/editing map data		DONE
technology available for building maps		Done
technology available for managing symlinks		Initial version done

Actions	Due	Status
Meet with LDAP group to decide LDAP requirements for filesystem maps	2002-02-19	Done
Test combined home map using dbm	2002-02-19	Done
Install 7.1 machine for gdmr	2002-04-16	Done
Debug ldap/amd	2002-04-30	Done
Map building tools	2002-05-28	Done
Produce proposal for DICE mount points	2002-04-30	DONE

Goals	Status
policy on filesystem usage under DICE (shared writable, groups, nsu/pseudo users, corpora, etc)	DONE
migration path for data on legacy filesystems	N/A

Milestones	Due	Status
clarify user legacy filesytem/group usage		DONE
define usage of secondary groups		DONE
define/publicise filesystem usage options/policy		DONE

Actions	Due	Status

Initial survey of users and filesystems	2002-02-19	Done
Further data collection	2002-04-30	DONE
Proposal for shared filesystem requirements	2002-04-30	DONE
Proposal for group requirements to support shared data	2002-04-30	DONE

Goals	Status
Provide configuration (profile) servers
Provide software distribution servers
Provide installation servers

Milestones	Due	Status
Implement master configuration server	22/3
Implement a slave configuration server	22/3
Implement distribution of data from master config to slave config server	22/3
Restructure header (.h) config files	22/3	PROTOTYPED
Arrange for all existing DICE machines to use new config servers	28/3
Implement new DICE RPM/SRPM master server	28/3
Implement first slave RPM/install server	5/4
Restructure rpmcfg files	5/4
Arrange for all existing DICE machines to use new slave RPM server	17/4
Implement further slave configuration servers	1/6
Implement further slave RPM/install servers	1/6

Milestones	Due	Status
Master KDC server		DONE
Slave KDC servers		DONE
Test/development accounts		DONE
Kerberos client component		DONE
KDC master server data backup/recovery mechanism and procedural documentation		DONE

Goals	Status
An LCFG system which is capable of fully configuring all DICE systems
Conformance to standards allowing future developments of the LCFG infrastructure without substantial modifications to stage1 component code
A documented, exportable system which can be used outside of Informatics to encourage external contributions and collaboration

Milestones	Due	Status
Agreement on new API standards for LCFG components		complete
Documentation and tools for developing DICE/LCFG software according to standards		complete
Documentation and code for a new component framework supporting the standards		complete
A new "om" system using the DICE authorisation service	1/6	started
Minimal conversion of all essential legacy components to new standards	1/5	started
Server Refactoring	1/5	complete
Support for "spanning maps"	1/5	complete
Support for rpmcfg transport in the profile	1/5	complete
Minimal Client Refactoring	1/7	complete
Conversion of all components to use new framework	1/10	started
Support for dynamic reconfiguration in all components	1/10	started

Actions	Due	Status
New remote om (sxw)	1/10	complete
Component conversions	1/10	complete
Automatic maintenance of lcfg.org web site with latest software and documentation (carwyn)	1/8	active

Milestones	Due	Status
Master LDAP Server		DONE
Slave LDAP Servers		DONE
Basic client LDAP replication		DONE
LDAP Schema		DONE
LDAP master server data backup/recovery mechanism and procedural documentation		DONE
LDAP Triggers		STAGE2
Advanced client LDAP replication		STAGE2

Actions	Due	Status
Fix slurpd crashing.		high
Implement ChangeLog client replication.		high
Upgrade LDAP RPM's to newest versions and freeze.		low
LDAP trigger mechanism. [SXW]		low
Replace ldapBuildAmdMaps with trigger mechanism. [SXW]		low
Replace buildcaps with trigger mechanism. [SXW]		low
Replace dummy Hosts data with genuine LDAP feed (spanning map) [SXW].		low
Credential cache code rewrite (remove much code duplication). [SXW?]		STAGE2
Re-installation and recovery procedure testing of stapag.	2002-09-30	DONE
Complete disaster scenario and recovery procedure document.	2002-09-30	DONE
Implement slurpd replication.		DONE
Find causes of and fix LDAP master server crashing.		DONE
Test out and move to BDB backend to resolve locking issues. No longer required.		DONE
Arrange Amanda backups at SB (just rsync it somewhere).		DONE
Make changes to online backup mechanism.		DONE
Upgrade Kerberos RPM's to newest versions and freeze.		DONE
Investigate LDAP replication issues of load/latency.		DONE
Determine frequency/level of online backups.		DONE
Use LDAP rather than NIS at installation.		DONE
Replace dummy Printcap data with genuine LDAP feed [TOBY].		DONE
Replace dummy People data with genuine LDAP feed [NEILB].		DONE
Replace dummy Partitions data with genuine LDAP feed [KTD].		DONE
Replace dummy Group data with genuine LDAP feed [SXW].		DONE
Replace dummy Roles/Capabilities/Netgroup data with genuine LDAP feed [NEILB/KTD].		DONE
Replace dummy amdmap data with genuine LDAP feed [SXW/GDMR].		DONE
Full convert of components to LCFGng.		DONE
Transfer resources onto new INF profile server setup.		DONE
Need to look at ACL update mechanism. Uses capabilities.		DONE
Install slave KDC/LDAP server at FH		DONE
Install slave KDC/LDAP server at SB		DONE
Install slave KDC/LDAP server at BP		DONE
Install slave KDC/LDAP server at JCMB		DONE
Test LDAP slave<-master replication		DONE
Test KDC master->slave propogation		DONE
Test KDC fallover to slaves		DONE
Implement online backup mechanism for Kerberos and LDAP data.		DONE
Rethink current schema/template locations.		DONE
Online backups from crontab entries.		DONE
Feed back local changes on stapag to slapd.conf.tmpl files to the RPM and also the provisional schema extensions.		DONE
AMD+Schema+LDAP (or Autofs+LDAP) investigate. [GDMR]		DONE
Get test local filesystem (home/yesterday) schema into LDAP.		DONE
Get test partition schema into LDAP.		DONE
Modify schemas for schemacheck being enabled and OpenLDAP2.1.		DONE
Upgrade client ldap to v2.0.23 (newest stable release).		DONE
Create a dummy hosts map to test ssh key stuff.		DONE
Redefine resources to use K/L DNS CNAMEs.		DONE
Fix Kerberos component so doesn't ignore krb5to4 resource value.		DONE
Create CO/CSO principals on stapag.		DONE
Stapag upgrade to SMP kernel.		DONE
Upgrade/test from CVS pull of OpenLdap to new Alpha public version.		DONE
Facilitate client rollout.		DONE
Finalize GID for People/Pseudo primary groups.		DONE
Root filesystem backup needed on stapag, do via rsync/rmirror generic.		DONE
Get test printcap schema into LDAP.		DONE
Add capability schema. [SXW]		DONE
Create user account data (correct INF/DICE names/ids) and load as LDAP data.		DONE
Convert old objects to LCFGng components (min. spec)		DONE
Port DICE 6.2 RPMs to the 7.1 repository rebuilding as necessary.		DONE
Produce new RPMCFG dice.h file for 7.1.		DONE
Upgrade stapag to run Kerberos and LDAP		DONE
Upgrade stapag to use Kerberos and LDAP		DONE