The Scottish Government

« Previous | Contents | Next »

Listen

Annex 1 Case Study 1: City of Edinburgh Council

Context

The City of Edinburgh Council tested the applicability of CitiStat in the context of environmental quality of life. The implementation of the Pilot ran from September 2005 to February 2006. It focused on selected operational performance information relating to two Departments, Environmental and Consumer Services (E& CS) and City Development ( CD).

The CitiStat Pilot was conducted in the wider context of impending organisational change. Part of the City Development Department is due to transfer to a new Department of Services for Communities in April 2006. Refuse collection services managed by Environmental and Consumer Services are due to be subjected to a tendering exercise for the first time. It is acknowledged that there is limited joint working between the two Departments, for example, road sweepers employed by E& CS will not routinely report potholes, which are the responsibility of City Development. This lack of joint working is an acknowledged cause of frustration for both elected members and senior managers.

In this context, participation in the CitiStat Pilot appealed to the Council for two principal reasons. Firstly, a desire by senior managers to promote a stronger performance management culture within specific departments. The CitiStat process was seen as a potential way to bring about culture change; to promote better collection and use of data to manage performance and begin to change managerial behaviours. The Council has previous experience of performance management models including the use of the Balanced Scorecard. As a part of its governance processes the Council operates a cabinet system with seven Scrutiny Panels whose duties include reviewing performance on a regular basis. The Council is also subject to a wide range of external audits and inspections and will be undergoing a wide ranging review of its activities as part of the Best Value Audit over the summer of 2006.

It was felt that CitiStat may bring a new degree of rigour and provide an opportunity to drill-down into the data in more detail to allow more thorough discussion of operational service issues. There was an interest in linking cost measures with service indicators to allow more informed discussions about costs and benefits and comparative costs between organisational units. The comparative high costs of refuse collection and rates of sickness absence were also high level concerns.

"There is a huge degree of enthusiasm for CitiStat, in the context of attempts to improve performance management generally. At the end of the day it has to get better at the service end. We're not just doing it for the sake of it - it's got to lead to some improvement at almost street-level, but also hopefully it begins to change the culture of the organisation so that there is a more rapid turnaround, particularly in those things that can be addressed in a fairly fast timescale."

A second factor was the desire to address the 'issues of the day' within the city; to see real improvement at street level through better road maintenance and a cleaner city. These initial concerns are reflected in the choice of success factors for the Pilot which are discussed more fully below.

The CitiStat process

The focus of the Pilot

Appropriate indicators were agreed with service representatives from each Department at the beginning of the CitiStat Pilot.

Table A1.1 below shows the coverage of the 61 indicators selected at the outset for Environmental and Consumer Services. These included issues such as litter, dumping/fly tipping, dog fouling, leaves, abandoned cars, graffiti removal, special uplift of domestic goods, refuse collection complaints, cleanliness scores, environmental warden activity, staff availability - sickness absence and turnover, costs of services, costs of refuse collection, vehicle availability. Furthermore some of these indicators were broken down by Local Development Committee area. Some data was available at the beginning of the Pilot in September 2005 for all these indicators except for ECS - 14 Cost of Service and ECS-15 Cost of Refuse Collection.

Table A1.2 below shows the 35 City Development indicators which include coverage of street light repairs, traffic light repairs, road and footway repairs, utility inspections, customer feedback, staff availability - sickness absence and turnover, squad productivity and public liability claims. In September 2005, data was available for most of these indicators.

Table A1.1: Environmental and Consumer Services - CitiStat Indicators

Table A1.2: City Development - CitiStat Indicators

The structure of the Panel

The Chief Executive chaired the CitiStat Panel sessions, with support from the Director of Finance and Director of Corporate Services. The Council Leader and the Convenor of the Resource Management and Audit Scrutiny Panel also attended as observers. The Podium comprised the Head of Waste Management from E& CS and Head of Transport from City Development; they were supported by other service managers from their Departments at different times.

Figure A1.1 The Structure of the CitiStat Panel - Edinburgh

The CitiStat cycle

The Edinburgh CitiStat Pilot process worked to a tightly defined 4 week timetable, shown in Figure A1.2 below.

Figure A1.2: The Edinburgh CitiStat Cycle

Week 1: Data collection

Performance information on the agreed indicators was collated by the two participating Departments and forwarded in an agreed format to the CitiStat Project Team by the 6 ^th working day of the following month. This was based on an agreed spreadsheet template which presented the performance indicators to be reported and associated targets. The performance against target was shown through a traffic light system, where red indicates a target not being met, amber indicates a target not being met but within an acceptable tolerance and green indicates performance is on target. Trend information and acceptable tolerances were also shown.

Week 2: Data analysis, issue of brief and the CitiStat Session

On the day immediately following submission of CitiStat data, the information was analysed by a dedicated CitiStat Analysis Team made up of representatives from E& CS, City Development, Corporate Services and the Scottish Executive's CitiStat Co-ordinator, with representatives from other relevant business areas invited as and when necessary.

Immediately after this meeting, a CitiStat Brief was produced by the CitiStat Project Team and distributed to the CitiStat Panel. The brief was also issued to the Leader of the Council and to the Convenor of the Resource Management and Audit Scrutiny Panel for information.

This brief was central to the CitiStat process and formed the basis of discussion and action at the CitiStat Session. The brief highlighted and provided background information on key issues, suggested lines of further inquiry, approximate timings for each area of discussion, and prompted action where necessary. At the same time, a separate Departmental brief was issued to the participating Department Heads (and relevant operational staff). This did not contain the level of detail provided to the CitiStat Panel, but provided a list of topics that will be discussed at the next session. This approach was intended to avoid scripted answers to questions raised at the sessions. The involvement of representatives from the service departments in the data analysis process did mean that they were aware of likely areas for discussion prior to the session and would have been able to brief their Service Managers in advance.

Both briefs were circulated with the data collection spreadsheets and Follow-up Action Responses from the previous session. The CitiStat Panel also received a personal briefing from the CitiStat Project Team the day before a CitiStat Session.

From September 2005, the CitiStat Sessions were held on a monthly basis, no later than 10 working days from the beginning of the month. This timetable was to ensure that the information under discussion was as close to being 'current' as possible, the reporting period being that of the previous calendar month. The sessions were intended to be limited to an hour to ensure that specific issues were discussed in a clear and focussed way.

Weeks 3 and 4: Follow-up brief and further actions

A CitiStat Follow-up Brief was produced by the CitiStat Project Team immediately following the CitiStat Session and distributed to all participants. This set out required actions agreed from the CitiStat Session and timescales for completion. Action may have been required to be reported to the following CitiStat Session, or in writing by an agreed date prior to the Session. A template for the provision of follow-up action responses was also provided.

A total of six CitiStat cycles were completed between September 2005 and February 2006.

Evaluation methods

The evaluation of the Edinburgh Pilot proceeded through a series of four evaluation sessions held after the September, October, November and January CitiStat sessions. A total of 19 different Podium members and observers have attended these evaluation sessions. In addition, the Chief Executive and the Directors of Corporate Services and Finance who comprise the Panel have taken part in an evaluative conversation in December 2005. A similar conversation was also held with the Leader of the Council in February 2006. In place of an evaluation session, in December 2005 a questionnaire was circulated to CitiStat non-Panel participants and ten responses received. The evaluator also attended five of the six CitiStat sessions as an observer. A number of participants from Edinburgh also attended the CitiStat Exchange Inquiry Workshop in January 2006.

One of the first evaluation tasks was to generate local success factors for the CitiStat Pilot. These were developed at the first session using postcards, to illuminate the vision of success and challenges of the process. These were later made into a visual display board. This has been used by some managers to promote further conversations about CitiStat with other staff who have not attended the actual Panel sessions. Discussion at evaluation sessions was recorded and presented through a developing learning grid which was circulated to all participants and expanded at each stage, providing a record of discussions, insights and emerging issues as well as continuously assessing progress towards goals. The emerging findings were presented to the January evaluation session and refined in the light of discussion. This was integrated into a fuller storyboard used in the CitiStat Exchange Inquiry event.

Success factors and challenges

Local success factors were developed at the first evaluation session immediately following the CitiStat Panel session. Nine success factors were identified although they are all interrelated and there is some overlap. They reflect the concerns of the Podium members and observers and the challenges they anticipated through the process. The top three priorities were later identified through the responses to the questionnaire. The interview with the CitiStat Panel members identified much common ground in terms of wishing to see improvements in the environment of the city, but placed greater emphasis on cultural change and influencing the behaviour of managers.

Demonstrable action and change to improve performance arising from the CitiStat process

This was the greatest priority of all the success factors. There was a strong desire not to be swamped by data and to ensure that the focus was on action and improvement, whilst still using the process to improve the data available and ensuring that it is useful and actionable.

"There's a desire to make sure we don't get buried in more and more detailed analysis of data. It has to result in somebody doing something - they have to leave that meeting knowing what's expected of them to do something to make improvements. There's a fear we'll just get swamped and nothing happens."

"We never really got onto 'well what can be done?' So for example, sickness absence - in both departments they said 'well if you strip out long term sickness it's a different picture'. So I'd say why don't we get figures that do that? So we can look at short term sickness and look at the impact? We need to look at what really matters here and what can we do something about."

Investigation of the deeper issues - getting to the causes, not symptoms

Investigation of the deeper issues was the second highest priority. The desire to move beyond simply describing the issues presented in the data was evident, as well as a sense that it may be useful to have a more clearly defined focus on a smaller number of key indicators.

"We've got to be on our guard against confusing ourselves with a large amount of information. Not seeing the wood for the trees. It's very easy to collect large amounts of data -it may not always be telling us what we really ought to be doing."

The need for a different style of questioning was also evident, so that initial responses are followed up and patterns and deeper issues may emerge.

"I'm keen that we treat the cause of the problem and not the symptom. It's easy for me to target full litter bins if I deploy my resources fully, so that the next time round we get a positive report. But the next time round…they're really just the symptoms. We need to look at cause….look at what is actually happening."

"Today [at the Panel] the examination tended to be of the individual indicators - there are patterns that warrant more investigation - those examples of littering in the central and North and Leith areas - what is that telling you? Is it about a need for a service change to be educational and awareness raising or is it something else? We need to have a review of that kind of issue if indeed that is an issue worth catching."

"There were a few additional questions that were asked and that's when that kind of discussion came out because it wasn't prepared by the people on the Podium."

Improvement in the environment of the City

The third priority success factor was to see improvement in the environment of the City.

" it [postcard] symbolises the earth and the environment - and that's what this is all about - trying to improve the environment - that's the crux of it."

This is clearly linked to the desire for action and demonstrable change and the view of the Panel members of the importance of seeing service improvement at street level.

A clearer focus of the process to tackle a task of great size and complexity

There was recognition of the size of the task and a desire to make the process more focused and manageable.

"We've got quite a mountain to climb - the mountain represents the Council! It's a massive organisation, multi-functional, multi-talented."

Developing a collegiate approach and greater cross-departmental working

The need for greater cross-departmental working and cooperation presented a significant challenge to the success of the CitiStat process.

"We all think we're quite good and perhaps we are, but we don't want to be pestered or pressed from outside our departmental boxes - the notion is that that's necessary in order to make more of a fist of this project."

Greater comfort with peer review and the process of scrutiny.

There was some welcome for the high level input into the CitiStat process and the process of scrutiny involved.

"We've produced performance information for years now and it's never really been subject to much peer review or scrutiny. I think it's actually quite refreshing that the Chief Executive and the Leader of the Council actually take an interest in that process - albeit that we might feel quite uncomfortable about reporting some of the issues - it's probably long overdue."

There was clearly a sense that the CitiStat process was going to be different from existing performance management models. This dovetails with the Panel's view of the importance of embedding cultural change;

"We're almost trying to work ourselves out of a job - because they get this information themselves and they act on it without having to come to the Panel."

The process was also seen as potentially providing opportunities for reciprocal questioning of the Panel.

"…[we should] recognise the request for action could be two-way. There could be things where you're saying to the Leader or the Chief Executive 'you need to be making some changes' - there are some blockages - it's not just one way."

"Who questions the questioners?"

Communication of the purpose of CitiStat to everyone at all levels

This was related to the need to embed the CitiStat ethos into the wider organisation and to translate the high level data analysis into behavioural change across the organisation.

"…. it's a case of 'is it going to be communicated well enough?' across the organisation so that people actually understand what all this statistical analysis [is about] - is it going to produce anything? And what's in it for me? Is it good, bad or indifferent?"

"I need to get this down to the very bottom level of the services that I deliver [so that] the message is understood by the guy that collects refuse and sweeps the street. So that he's part of the organisation and he understands where we're going and how we're going to achieve that."

"The most important thing is to think about how we communicate this? In the appropriate language - the language of the boardroom is no use to the guy that's sweeping the street and cutting the grass."

Culture change - development of levels of trust that allow honesty and openness

A further challenge related to the broader goals of cultural change was acknowledged as the need to develop trust that would allow people to be honest about performance and open to challenge on it. It was recognised that this would be likely to take time.

" There isn't just suddenly going to be a culture where everyone's going to say 'that's great' - I'll bare my soul - and we'll move on from there. It's going to take trust and it's going to take a bit of building up."

Opportunity to provide fuller information and understanding to elected members

There was a desire to see the best use made of the input from the elected members and to use the process to inform them about operational issues.

" We didn't really have a problem with that. I don't think it made a difference to the answers. I don't think it intimidates the process. In some ways, he probably left more informed than when he came in."

There were different views about whether the presence of elected members as observers would affect the process itself.

"You may get people who are reluctant to say what they actually think because the elected member is sitting there or you may get them desperate to say what they think because the Leader of the Council is sitting there!"

Findings

The content of the CitiStat Panel discussions

CitiStat meetings were held in formal meeting rooms within the Council Chambers and generally the Panel sat facing the Podium members. The content of the CitiStat meetings illustrates the central concerns of the Panel and some of the initial outputs of the CitiStat process in Edinburgh.

From the very start, the process highlighted a number of gaps in the data available and highlighted issues concerned with the definition, basis of the data and ability to disaggregate it in meaningful ways. The questions allowed the Panel to 'drill-down' further into a number of issues by requesting further information, explanations and new indicators or refinement of existing ones. The process was also an opportunity for the Panel to offer the Podium their thanks where there was improvement in performance; for example in relation to improvements in sickness absence amongst manual and craft employees they noted: "it is an exemplar of good practice -it shows what can be done with a bit of an effort. The trick is to sustain it."

Not all the initial agreed indicators in Tables A1.1 and A1.2 were discussed during the process and the focus of the Panel was clearly on a number of issues which are of direct concern to the public and which affect the reputation of the Council. Direct customer feedback indicators were not discussed at the Panel sessions although both Departments were frequently asked about the views of the public on reputational issues and service standards. Full cost data for the agreed indicators was not provided on a consistent basis. The time available for the sessions limited the ability to discuss all the indicators provided or to follow up all the issues identified in the previous meetings.

The responses from the Podium often were explanatory, giving reasons for performance and making reference to constraints such as resources, seasonality, established working practices or wider corporate policies which impact on their operations. In relation to cross-departmental responsibilities, the process highlighted the issues of the respective understandings of the division of corporate and departmental responsibilities for the management of sickness absence and the degree of flexibility of implementation of corporate policy in key areas such as retirement age.

The quality of the data provided and precisely what is being measured were important issues. The theme of how to best measure efficiency, effectiveness, productivity and performance was discussed on several occasions. The adequacy of targets and ability to compare performance with other local authorities was also a concern. Much of the process was concerned with the improvement and refinement of the key data set to enhance its usefulness for performance management.

The achievement of success factors and response to the challenges presented

This section discusses the participant's view of CitiStat and the extent to which there is evidence of progress towards the success factors identified at the beginning of the process.

a) Demonstrable action and change to improve performance arising from the CitiStat process

Evidence from the evaluation activities shows there are some encouraging signs of improvement in key indicators and better awareness of the issues. After only six cycles of the CitiStat process, participants feel it is still too early to expect to see demonstrable impact on performance. However, some assessment of progress towards the success factors is possible.

The early evidence from the evaluation process showed that participants wanted to see greater clarity of the specific actions required by the Panel and there were concerns about the nature of the 'actions' requested as many were felt to be means to ends, rather than ends in themselves.

"I'd like to see more actions coming out of the meetings - from the first meeting - it was quite difficult to pull actions from it. From this meeting, it's easier and I think in future meetings it will become even easier. Otherwise it just becomes another Performance Measurement exercise."

"There's a danger as well, if you focus on actions, some of the actions might actually be 'create a new indicators set' or 'write a report' - these aren't actions per se - these are other bureaucratic means - which might then result in an action - but they could just become more workload which isn't really leading to change."

There was seen to be a risk that the list of actions would become a burden and may impact on other areas of work. Some did clearly present substantial workload issues. There was also a concern that actions should be focused, relevant and achievable.

"The amount of work that we had to do to bring that down to [respond to the request] - it was completely needless….it's not something that is relevant …I tried to say that, but we were then tasked with producing it which actually took the best part of 3 days work".

There was recognition that the CitiStat process was clearly different; that it was not to be simply about monitoring, but ensuring the implementation of actions. The speed of response required was also quite different to the usual ways of working.

"For me it's along the lines of...you have identified x, now what are you doing about it? It's the speed that you're required to implement a change to the service or a change to the way that you report or record and...that is causing some difficulties back at the ranch where I work, just because the Council moves so slowly."

"It's different to the way that we've been working because we've always had a safety blanket of 'we're waiting on a committee decision...or we'll wait in the report coming, or we'll wait until it goes to scrutiny.' This time we're identifying things and making changes extremely quickly."

The CitiStat process clearly gives more immediate feedback to managers. It was seen here as a more proactive and evolving process than other performance management processes and allowed for quicker actions to be taken to address performance. However, there were concerns about how to sustain the momentum of this process and also how not to lose track of 'longer term' actions that wouldn't necessarily get picked up at each cycle.

The evaluation process raised questions about how soon it could be expected to see demonstrable change in the indicators that would be sustained over time. The six cycles showed considerable fluctuations in a number of indicators and it appeared that different indicators may respond differently to intervention. Initial efforts to refine key data on sickness absence appears to have allowed managers to more accurately pinpoint where the issues lie and take appropriate actions to influence performance. However, initial improvements in sickness absence in E& CS, whilst welcomed, do not appear to have been sustained in the short term and a longer period of monitoring would be required before drawing robust conclusions about these outcomes.

Although participants felt that the internal management of the process had improved over the period of the Pilot, it was felt that some kind of mechanism would be required to ensure that actions could be tracked, service changes revisited and long term progress monitored.

b) Investigation of the deeper issues - getting to the causes, not symptoms

The CitiStat process highlighted a number of significant gaps in information available to departmental managers.

"At the first meeting, I asked 'what kind of data have you got?' - 'not a lot' was the answer! 'Well how do you run a business?' - 'well I can't'. Well where the hell do you start from? I can't solve that in a four week cycle of CitiStat! It's much more endemic - it's that kind of deep seated problem that we've got."

It is clear that, even after only six cycles, the CitiStat process has helped to get better data.

"The provision of data for some Departments on a monthly basis is an achievement in itself."

However, some of the further data that is asked for is not easily available and requires considerable staff resources to provide it. The data which is available does not always lend itself to supporting effective questioning. To some extent the process has shown that the Council collects some data which is not particularly useful; available data which may be used for reporting on Performance Indicators, but which is not necessarily actionable data in terms of performance management.

There were also some constraints due to the initial limited agreed scope of the indicators for the Pilot process, although a couple of new indicators were introduced and some others refined.

"Right from the beginning we said that we weren't going to look for new performance data…and we're beginning to see that there are areas that if we really wanted to drill into those we'd need to either find out if they have data and they're not telling us about it (which I'm sure they're not), or that we need to put some resource into trying to collect a bit better data in some of these areas. That's a reflection of the Pilot nature of the process; we didn't want to over-burden people at the beginning."

Given the greater speed of the process, it does raise questions about the nature of the indicators used and a clearer focus on those that can be changed quite quickly. It may be more useful in the longer term to look at core issues and focus on a smaller number of key indicators. The Panel view is that 'liveability services' are ideal for this kind of focus, as well as being the key issues for the public.

There is a clear sense from the Podium members and observers that the process is starting to "drill down" into the data and beginning to go beyond more superficial answers.

"The process is beginning to get the heart of the matter. The Panel seems to be getting to grips with some of the more intricate areas of the day to day management of services and as a result are asking more pertinent questions."

However, the Panel members still think there is some way to go on this issue.

"What we've been getting is superficial material back which has exposed to me a weakness in terms of superficiality - in terms of their understanding of their business - and that's really quite revealing. I think it's been quite useful as well, because it's meant that I've had to think about how to get behind these superficial 'hand-offs' that you get and drill down and ask rather more complicated and rather more difficult questions. I don't think we're there yet….but I think that is where we need to be getting to. We need to be slicker at side-stepping the one off response and asking the supplementary and the[further] supplementary question that goes down and gets beneath the surface of the issues."

It is also evident that as the process has developed there has been less of a focus on the current reported data and more of an emphasis on the follow-up actions.

c) Improvement in the environment of the City

Cleanliness and road maintenance are two elements of environmental quality that are acute political and service issues within the City and which are a high priority for the Panel and the Leader of the Council. There are some limited indicators such as CIMS scores that do measure environmental outcomes, although these only report quarterly and there are questions about the validity of the methodology. There is agreement that it is too early to expect to see demonstrable impacts on the environment in the City, although this goal is a high priority for all participants.

"It is too early to say whether the CitiStat process has had an impact on the environment of the city and the quality of life of Edinburgh residents. If the improvements in performance [we have seen] start to have an impact on the services provided then we should begin to see impacts on the environment and quality of life."

There is general agreement that there are early signs of improvement in a number of areas which may demonstrate some progress towards long term environmental outcomes. These include sickness absence in E& CS, public liability claims processing times, complaints for the non-collection of wheeled bins, the failure rate of core samples and failure rates of utility reinstatement inspections. At this stage, the early impacts appear to have been in terms of internal processes and operational issues.

d) A clearer focus of the process to tackle a task of great size and complexity

There were initial concerns about the need to find a manageable focus for the process given the scale and complexity of the issues under discussion. There is a clear sense that the process has evolved and developed a clearer focus, that people have got used to how it works and have been able to adapt it to suit the circumstances.

"I think the standard of questioning from the Panel has improved. I think they have got more focused in the way that they are running the discussions and the questions they're asking and perhaps being rather more probing than they were."

Changes to the way the briefing went out early on did provide a clearer focus for CitiStat although on occasions there were departures from the brief by the Panel. This can be challenging for the Podium.

"It does seem at times that it's a bit like open season. It's literally like questions can be from any direction…on any of the data - and you have to be semi-prepared for that!"

It was identified that there was a need for clear, precise minute of exactly what's required from one meeting to another and that getting the follow-up briefing right has been part of the learning process.

There are issues for the CitiStat Analysis Team about the technical basis of some of the data.

"None of us who are writing the brief are actually experts in your area. It's actually quite difficult sometimes to work out what is the really punchy question to ask about your area…. and clearly the only way you can actually do that is if you have someone in the back office team that's supporting the CitiStat process that actually understands roads and filling in potholes and street lighting or refuse collection and things like that."

e) Developing a collegiate approach and greater cross-departmental working

The process has shown the need for greater clarity of responsibility for actions; where Departments are responsible and where issues require a corporate response. The management of sickness absence is an example of where the process made the respective responsibilities very clear and this may have contributed to performance improvement.

The need for cross-departmental working is linked to the complexity of the issues that Departmental managers face. The example of the changes to the retirement at 75 years policy illustrates that whilst this may help with retention of older staff in some areas, there are still broader issues such as general recruitment, the age profile of the staff and redeployment, which require a corporate response.

There is some evidence of the development of better working relationships between Departments.

"We've also got out of this in many ways a better working relationship between Corporate Services and the two service areas that we've been working with in terms of being more open about sharing data and trusting each other more than we probably ever had in the past. And that's been incredibly useful."

"I have certainly not experienced any barriers whatsoever to getting further information from the Departments when I needed to do further analysis - in the past we have had that problem."

This is attributed to the involvement of the Chief Executive and the high level priority given to the process. Some participants report that they have a better understanding of services provided by other Departments.

However, this process has not gone as far as necessary to ensure delivery of data on some issues, especially cost data. One Podium member commented;

"I think we've still got to define the cross-departmental working. I think there was a sticky moment …. [we] have been doing whatever we can to try to get the data - we're not the authors of the data and I think we're just going to have to put our hands up and say 'look we've asked for this dozens of times and we haven't got the information. If we had it - it would be there'. There's no other way to say it - somebody else within the chain has just not delivered."

f) Greater comfort with peer review and the process of scrutiny.

It is acknowledged that CitiStat has been useful in raising performance management issues within some Departments and highlighting weaknesses in their knowledge about their activities and areas of weak performance. In terms of the experience of the process itself, although for some there is a feeling of "being put in the spotlight", the process of scrutiny and challenge has been welcomed by other managers.

"The process is adversarial and at times uncomfortable - it is challenging and testing. It is however welcome that difficult issues are being targeted and examined."

Not all participants see the process as having been particularly adversarial. One observer commented;

"In my opinion it is not adversarial enough. The 3 ^rd session did have more of an adversarial feel to it, and the follow-up actions agreed appeared to get more at the heart of the matter and as a result the response to follow up actions were very good. However, the fourth session had the opposite feel and the Panel backed down on a number of issues that could have been probed further."

Most participants have been comfortable with the approach although acknowledge it is challenging, and are pleased that it has opened up debate on improvement actions. Both the Panel and the Podium members are developing more detailed understanding of service issues.

The Panel clearly see a role for the adversarial approach and some of the 'theatre' of the CitiStat process and think that they may need to be more adversarial in the short term. They want the process to be testing without 'crucifying' staff.

"I think we need to be more adversarial - in the short term. I recognise we've a balance to strike - if you're overly adversarial then you're going to create a very negative circumstance …. I think we've got to that point where people really recognise that we are meaning business. Unless you're really quite adversarial, then you're not going to get them to recognise that we really do mean business here. I don't think we're quite at that point yet. We need to get to what I would describe as the tipping point - get to a point where we'll have some really hairy meetings then that would clear the air and we'd make a shift."

The Panel also acknowledge that there is some artificiality about the CitiStat process, in terms of an element of role playing and some repetition of parallel scrutiny processes. They see that the process has provoked some defensiveness rather than a focus on the learning points than can be developed. This has meant that from the point of view of the Panel members the process has lacked some bite.

g) Communication of the purpose of CitiStat to everyone at all levels

Some participants felt that it was important to the success of the CitiStat Pilot that the ethos of scrutinising data for performance management purposes should be embedded into the wider organisation so that high level data analysis is translated into behavioural change across the organisation.

"I sit here on a monthly basis and get challenged by the Chief Executive of this authority then I need to let all of my people know that that's happening...because I'm being challenged on what's happening at the front line - I need to get that out to the four corners of the services that I have responsibility for."

At the outset, there was a difference of views as to how essential or desirable in terms of use of time this was.

"There's no need to involve them at this stage…their time is better served out in the field - physically doing the job."

Observers from the participating Departments did attend some of the sessions and also took part in the evaluation process. However, the CitiStat Panel process has been entirely office based. One observer commented that "it's remarkable that we've not been out of the office!"

"There's definitely mileage in inviting other people that are involved also as observers - all the way down….it's an open process that way. We want to encourage that so that people can sit in a session…and understand it a bit more."

The display board of success factors was borrowed by two Podium members and taken back to their offices. This proved to be one way of communicating messages about CitiStat.

"It acted as a conversational point - 'what the hell is that'? It then allowed a conversation to take place about CitiStat, about what was happening, and it allowed me to give an explanation - almost a Q & A session in itself from that. It helped me because I needed to get understanding about CitiStat and it's driven by what we're doing out on the streets. We're spending public money, we're accountable for that. I'm only as a good as my last complaint really! So it assisted me in getting that agenda across".

The Panel acknowledge that one of the unexpected outcomes of the process has been to expose some attitudinal differences amongst managers as they have seen contrasts in how some individuals have responded to the process.

"There was a stark contrast. The other benefit of this - it does expose attitudinal differences and there are very distinct attitudinal differences and these reflect back into management style and into the competences that people have - it flags up to me that we've got issues around competences that we need to address….it's the indirect consequence of the process."

h) Culture change - development of levels of trust that allow honesty and openness

CitiStat has been part of wider organisational changes which impact on performance management. The involvement of the Chief Executive and Directors has been important in giving a sense of clear and high priorities. The pressure this brings can be useful to Departments seeking to implement change. Some individuals have been seen by the Panel to relish the process.

"Those managers who have got a bit of wisdom about them do appreciate that this is an agenda that they had better get on with. There are tentative signs of some kind of culture change taking place in certain quarters."

Early signs of change are evident to most participants and strategies to achieve change are felt to be in place even though it is generally felt to be too early to expect to see much return on the investment of time and resources into the process to date. CitiStat has promoted wider understanding of the service issues and barriers to performance amongst senior managers.

The CitiStat model provides some important challenges to established cultures and ways of working.

"…we see something wrong, we write a report about it, it's debated and goes backwards and forwards and gets diluted and six months down the line we might get the authority to implement a decision - [with CitiStat] I feel that we're being asked to implement solutions to some quite tricky problems."

Compared to other performance management approaches, CitiStat has provided more immediate feedback. It is a more proactive and evolving process than other performance management processes and allows quicker actions to be taken to address performance.

"The timeliness of data used in the CitiStat process is more useful than in other performance reporting mechanisms in use throughout the authority. For example, with some reports to Scrutiny Panels, the data is often many months out of date and the position may have changed by the time the information is considered".

Given the limited involvement to date in the CitiStat processes, the distinctiveness of the process and demands it makes on people may not be thorough and widely understood.

"….quite a small number of people are involved in CitiStat and there seems to be a lack of understanding about the speed of the process, the way the process works down the line."

The frequency of the CitiStat meetings provides a regular focus on service performance. Uniquely it also brings together all the decision makers in one room and allows decisions to be taken at the time.

i) Opportunity to provide fuller information and understanding to elected members

Two elected members have been part of the formal CitiStat process. The Leader of the Council and the Convener of Resource Management and Audit Scrutiny Panel have attended some Panel meetings as observers. The Panel have taken the view that CitiStat is dealing with management functions and that whilst there is a role for elected members, it is their role to manage the organisation. This view is reflected in the initial decision not to have elected members on the Panel itself, but there remains some ambiguity about the roles of elected members.

"[elected members]…they're to come to observe. That's what they're going to be told - but whether they choose to do that - we can't always control that."

"[Scrutiny Panel chairs]… are they going to be involved? … are they going to chip in? - are they going to be observers?"

In the event, none of the invited three Conveners of the other Scrutiny Panels attended the December session.

There are mixed views amongst participants about value of elected member involvement. Some see value in them having a role and there is some disappointment that they have not had a larger role.

"I was disappointed that they haven't really been here. There needs to be buy-in from the elected members because unless you've got them here they're not actually hearing the explanations given by the Podium members as to the different problems they're having - in achieving or maintaining performance levels. It's actually very important that they hear that I think."

"I am disappointed that there hasn't been much interest - despite the fact that we've invited them on a number of occasions…particularly because one of the complaints at Scrutiny is that they don't know how to deal with performance information and they don't know what sort of questions to ask …in fact, they could have picked up a lot of quite useful things from the approach that we've been adopting here."

"Some people in the organisation only pay attention when there's an elected member raising the issue."

Others think the involvement of elected members is a distraction.

"The presence of elected Members does not assist the process and indeed it acts as a distraction. In general it is more effective for the business to be conducted by officers as a management process. The presence of Members changes both the focus of the meeting and the approach of the officers and the meeting becomes more like the Performance Review Group."

Some participants felt that the process would be more adversarial and beneficial if the Council Leader was a member of the Panel.

The Leader himself has found the CitiStat process fairly helpful, but has found his role as an observer to be 'very frustrating as the political head of an executive process'. As an observer, he was not meant to speak although did so on a number of occasions.

In some ways, the process has missed an opportunity to educate elected members. Many comments were made about the comparison with Scrutiny Panels in which elected members assess performance.

"I have the experience of appearing once a year before one of the Scrutiny Panels which looks at a lot of similar types of indicators. The number of questions and the depth that they can actually go into - looking behind the figures - is restricted - simply by time and the enormous agenda. For elected members [CitiStat] is a far better mechanism - even just to sit in and listen to what is being said and to learn more about the individual services and the difficulties - the issues that are affecting performance and those kinds of things."

"At Scrutiny Panels they're not given the level of detailed analysis that they're given at this session…it's basically just a covering reporting with the data sheets so that they don't know what sort of questions to ask to get at the issues - the difference between Scrutiny and this is that we support the process to come up with relevant questions and provide the analysis."

"I think this is a good model for improving the scrutiny process….the scrutiny process is very hit and miss - sometimes there are quite pertinent questions - other times they're all over the place and there's issues that really they ought to be getting their teeth into and they're just missing them completely...the rigour that this puts into it would actually help that scrutiny process"

"In the Baltimore model - the Mayor - who's the equivalent of our Council Leader - was part of the questioning Panel - I don't know that we should move to that - but we should use it for elected members as much more of an information forum - for them to do more listening and taking in issues."

"You're 12 days into January and you're already discussing December data - that never happens at Scrutiny Panels - you're talking 4-5 months out of data - it's crazy - you've got no opportunity to correct any problems because you're looking at data that's so old...it doesn't matter - it'll have changed since then. At least with this you're looking at data that's only 12 days old."

"On balance members should be involved more, probably over time in the Panel. What goes on at CitiStat should be happening at Scrutiny."

These views have implications for any future development of CitiStat in Edinburgh.

Possibilities and prospects

In considering the future possibilities and prospects for the use of the CitiStat model in Edinburgh there are a number of issues presented by the experience of the Pilot. These include the resources required for the process, the sustainability of the level of input required and the relative costs and benefits of the process. There are also questions about the relationship with parallel performance management processes within the Council, including Scrutiny Panels. Other issues include the potential for changes to the role of elected members, the nature of the process itself and the scope for involvement of other departments, services and agencies.

Resources

All the participating Departments have been and continue to be under considerable other pressures. CitiStat has been a very small, but significant extra part of their workload. The resources required to run CitiStat have been significant. This may have detracted from other areas of work and may have knock-on effects that have not been monitored.

The resource issues for the Panel are also significant to prepare for the session and deal with follow up reports.

"At the minute, we're not getting 'out' so it's an investment. If I didn't think there was a possibility of return then I'd be saying we've had enough [after 4 sessions]. I think it does have the capacity to give us a return and the investment's worthwhile. It's how we can manage to hold onto that over a longer period and time will tell."

"This is a relatively small Pilot, with a relatively small indicator set. The resources that this takes are quite intensive, both in the departments and centrally. If you were to roll this out on a bigger scale, to departments or other areas of the Council, you're not just talking about possibly quite a big resource requirement, but also possibly a change to working practices within those departments. It's something you can't take lightly. It's completely different to the way that normal working practices go on in reporting to scrutiny panels. You've got to weigh up that resource requirement and the way it has a knock-on effect to other areas of the business with what you're actually achieving through the process."

However, this effort has been acknowledged and it has also generally been felt to have been worth it.

"We've acknowledged the amount of effort that you [service managers] have put into get data ready in timescales that in some cases - have hardly ever been achieved before in this Council! Some parts of the Council have never been that good at turning stuff round as quickly as you have. You need to be applauded for the amount of effort you've put in."

"The efforts that E& CS have gone into on sickness absence…there is no way they would have done that if they had not had to come here on a monthly basis and be scrutinised on it. They always recognised that their sickness levels were high. The whole process has been validated because of some of the improvement that has been made. It's been good."

The involvement of the Scottish Executive CitiStat team was acknowledged to have been essential to guiding and making the process work.

The relationship with other performance management processes

The CitiStat agenda does overlap with other scrutiny processes used within the Council, although it is seen as more 'interrogational', tougher and more focused. It also requires follow up information and actions, which are not usually required. CitiStat may be a good model to align these processes more closely and be more strategic about performance management reporting.

"I think if we were more practical, more strategic we'd look at our challenging across Scrutiny, CitiStat and performance management arrangements and reporting in a more strategic way - so that we're kind of aligned what we're trying to do out of all these different activities. That's getting to a level of sophistication that we're a bit away from, but there are ways we could begin to do that and align these different strands."

Potential changes to the CitiStat processes

There is clearly interest in taking forward the learning that has been achieved to date through CitiStat. This could include changes to the composition of the Panel; the process has had limited impact at Director level and there may be scope for restructuring the Panel and Podium to bring them into the process as well as potential for greater involvement of elected members.

There may also be scope for a tighter focus on a smaller number of core indicators. There may also be value in using this kind of approach in cross-agency areas of work such as Community Planning.

"The approach could be utilised for considering performance on commitments agreed by partners and set out in the Community Plan Action Plan. CitiStat could provide a more robust and structured model for scrutinising actions to improve the services provided by all Community Planning Partners."

Conclusions and implications

The CitiStat Pilot has been a valuable tool in Edinburgh. It has assisted general efforts to improve performance across the Council. CitiStat has shown that once service activities are measured more appropriately, managers can be more effective and improve performance.

The CitiStat Pilot has been an opportunity for the Council to review the definition, collection and use of data which is then actionable, rather than historically, available. In particular, what has been distinctive about the process has been the immediacy of identifying and responding appropriately to variations in performance. This has presented a major challenge to established ways of working within local government. There are some useful lessons about the detail of the CitiStat process particularly in relation to the need for clarity of the required actions and follow up required and tracking of actions from month to month. In hindsight, the process may have been more effective if there had been a tighter focus on fewer indicators from the outset.

The CitiStat process has raised the profile of performance management within two key Departments and the management of sickness absence is seen as a specific success. It has helped to convey high level views about respective corporate and departmental responsibilities, although there remain issues about the provision of data that cuts across Departmental responsibilities. Some positive attitudinal change is noted to be observable in the participating officers and over time, there has been a shift towards bringing more strategic solutions forward as the process has developed. The process itself, and crucially the involvement of the Chief Executive, has given managers greater confidence and authority to tackle performance issues.

It is acknowledged that it is rather early to expect to see substantial and sustained improvements in performance outcomes. This remains as a continuing challenge to translate the analytical tools and knowledge that CitiStat has brought into the Council into real improvements in performance. However, it must be acknowledged that CitiStat has been a demanding process for all involved and it would be difficult to envisage sustaining the level of effort required within existing systems and resources. Most interest seems to lie with the idea of adapting the CitiStat approach for use with performance management by Scrutiny Panels, although with adequate support there may be scope for the replication or reinvention of this approach across whole Departments or in partnership contexts.

The Pilot is being assessed internally and the resource implications of wider implementation of Citistat are being considered along with the best means of integrating a CitiStat approach with other performance management and governance arrangements in the Council. Consideration is also being given to its applicability in the areas of cross agency working. A report setting out recommendations on the future use of CitiStat in the Council will be considered during the early summer 2006.

« Previous | Contents | Next »