Oral evidence: The Independent Commission for Aid Impact’s Annual Report 2013-14, HC 523Tuesday 8 July 2014

Oral evidence, 8 July 2014.

Witnesses

Richard Calvert, DG Finance and Corporate Performance, DFID, Nick Dyer, DG Policy and Global Programmes, DFID, and Pauline Hayes, Acting Director, Western Asia Division, DFID, gave evidence.

Chair

Thank you very much for coming in. I know you only came in for the tail end of the last evidence session, but you obviously know what is in the ICAI report. Again, we know who you are, but if you could just introduce yourselves for the record.

Richard Calvert

Of course, and apologies on behalf of Mark Lowcock, who would normally have been here today but I think you know is not available. I am Richard Calvert, the DG for Finance and Corporate Performance. Nick Dyer is DG for Policy and Global Programmes, and Pauline Hayes is the Acting Director for Western Asia.

Chair

Thank you very much. Perhaps starting at the absolute nitty-gritty, you got two green ratings from ICAI during the course of the year, but you also got your first-ever red, which was obviously the TradeMark Southern Africa issue. You also got four amber/reds, which had some pretty substantial criticisms of your work. Do you think ICAI is getting more critical or more confident, and how does it affect your relationships? It is independent, so it will say things you do not like. Does it have an effect on the relationship?

Richard Calvert

I do not think so. Obviously, we have welcomed the scrutiny that ICAI gives us. It was an important part of our process of driving up value for money and challenge and scrutiny over the last three or four years. Over that period, ICAI has learnt much more about the Department. It has continued to develop its methodology; it is looking at more areas. Although it is true that this year we have had, perhaps, a wider spread of traffic light scores, if you unpick that, then they fall into some slightly different categories. Clearly TMSA was a specific project where things had gone wrong. You, as a Committee, have had a detailed hearing on that. We have tried to use that in a very positive way to learn lessons for the organisation as a whole. We have done a lot of work to strengthen our programme management capability and processes in the Department. TMSA was quite a specific case. The thematic reports that ICAI has developed this year, for example those on how DFID learns and on the private sector, show what, for us, is a helpful development of the methodology of ICAI. It looks more broadly across the Department. In both of those areas, ICAI was looking at some really challenging sets of issues for DFID. I remember talking to the commissioners in advance. Particularly in a Department as complex as DFID, dealing with issues and countries with the complexity that we do, there are not many easy answers on organisational learning. We took the report on learning, even with an amber/red rating, as a really helpful input into our own development in this area, and similarly on the private sector. We do not see the pattern of traffic lights this year as concerning in itself. We welcome the way that the ICAI methodology and scrutiny is developing.

Chair

Do you accept that both the private-sector and the learning reports grew out of the fact that, having looked across their reports, ICAI concluded that you had a problem in those areas? Effectively, whilst lots of people knew lots of things, your institutional learning system was not effective. On the private sector, people have often said that the DNA of DFID is not naturally affiliated to that. Whilst they were able to identify one or two specific things that you have done quite well, they implied that, culturally, you were not yet comfortable. Do you accept that? Does that have a real impact on what you have to do?

Richard Calvert

I will take the private-sector one first, and let Nick say something on learning. On the private sector, clearly we have been challenged, not least by our Secretary of State over the last 18 months or so, to really step up our work with the private sector.

Chair

We have challenged her as well.

Richard Calvert

It is a very fair analysis of DFID to say that, if you look back over the last five to 10 years, we have not have a strong private sector DNA in the organisation. We are working very hard to build up our work in this area. We have brought in a lot of new skills. We are developing what we feel is much more coherent programming in this area. We have set out a high-level vision. While the ICAI study clearly identified a number of areas where ICAI felt we were not as strong as we should be, a lot of which we would acknowledge, we would also say that we have been moving quite fast over the last six to 12 months, and we feel we are in a stronger position now than when the work was undertaken. We will be very interested to see the follow-up work that ICAI is doing on the private sector. Obviously, the first study focussed on three specific countries in our portfolio. There will be in three specific countries follow-up work as part of the Year 4 programme. Certainly, we hope that ICAI will see some of the development the Department has made in the meantime.

Chair

Are you able to say what particular lessons you feel you may have learnt during the past year, as a direct result of ICAI’s report, leaving aside TradeMark Southern Africa?

Richard Calvert

There was a challenge around coherence, and how coherent our portfolio was in this area. If you look back at the development of DFID’s private-sector work probably over the last three or four years, it developed in what was probably a rather piecemeal way. In some countries, we had quite strong portfolios developing. In India, for example, we had quite a strong private-sector portfolio. We had activity through a number of wider instruments, such as CDC, which is an important part of our work, and the Private Infrastructure Development Group, which is important. However, we had not got a sufficiently coherent view across the Department as a whole about our activity. That is one of the things we have certainly taken from the ICAI report. We are just in the process of completing our annual resource allocation process for 2015, 2016 and beyond. That has brought together, for the first time, a really coherent view of the overall economic development and private-sector portfolio. ICAI challenged us very reasonably on our capability and the guidance available for staff in this area. Again, that is something we have made good progress on. A lot of the things that ICAI picked out are ones we would have recognised we were making some progress on, but part of ICAI’s role is to really keep us up to the mark in making progress in these areas. Perhaps Nick could say something on learning.

Nick Dyer

The learning report came out at the same time as a review that was done by the Cabinet Office on what they call What Works, or how we use evidence within DFID. We actually had two similar reviews at the same time. The review told me that we have come quite a long way. If you look at the use of evidence in DFID, in terms of the reports that come out of our Quality Assurance Unit, and if you look at the evidence survey we did of staff and their perceptions about taking on evidence more seriously, we have come quite a long way. In fact, the Cabinet Office says that we are a leader across Whitehall in the use of evidence. Now, that may say more about the rest of Whitehall than it does about DFID, but they were very complimentary about it. At the same time, the message is coming out that it is not as consistent across the project cycle or across the office. These are messages that we would recognise. We have, over the last couple of years, tried to improve the generation of our evidence to transmit it better. We have made various system changes to try to get the evidence working and moving more quickly and for people to adopt the evidence. Clearly, we have got more to do. We are actively following up the review by doing two things: very clearly building in messages around using evidence into our new Smart Rules that we are developing for the project cycle, and setting up a task group who are going to take the What Works review and the ICAI review, and identify practical actions in terms of moving forward.

Sir Peter Luff

Do you see ICAI more as an honest friend than a threatening foe?

Richard Calvert

Yes. I do not think we see it as threatening. Any organisation doing the kinds of things we are doing, which are difficult things in difficult places, should not feel that it has not got things to learn from scrutiny and challenge. Clearly, it is important for us that the relationship with ICAI is based on openness and honesty both ways, and that the processes around ICAI reports are reasonably proportionate and efficient. With those qualifications, we feel that ICAI is there to help and support us deliver better value.

Sir Peter Luff

On the question of processes around reports, you produce a management response to each of their reports.

Richard Calvert

Yes.

Sir Peter Luff

ICAI wants them to be more consistent. They say that when you accept improvements are necessary and state that you are doing some of the necessary work already, you should be more fulsome in accepting the recommendations—so accept the whole recommendation—and list the full management actions and fill in in detail how you will respond to their recommendations. What do you say to that?

Richard Calvert

Overall, our response to ICAI recommendations is very positive. We have accepted the vast majority of recommendations. We try to set out in our management responses a reasonably clear and succinct statement of what we are going to do to follow those up. We are always open to further discussion with ICAI about the detail of particular reports or particular follow-up action, if anyone feels that is missing. ICAI obviously take seriously its follow-up work on previous reports, which gives a chance to check follow-up action. Overall, we feel that the processes at the moment are generally proportionate and clear.

Sir Peter Luff

Do you think you could be more consistent?

Richard Calvert

There has clearly been a challenge, particularly around whether we have been consistent in saying whether we accept or partially accept recommendations. In fact, only yesterday we talked to the ICAI commissioners about that. Firstly, we look at each report on its merits, and we look at each set of recommendations. Ministers will look at those as well, and decide exactly how they want to respond. It is fair to say that, over the last year, there have been a number of areas where ICAI had made recommendations on issues that we felt we had already made quite good progress on. This is reflected to some extent by what we were talking about earlier with, for example, the private-sector work, or some of the other areas where things are moving quite quickly in the Department. In those cases, we have sometimes said, “We partially accept your report, because actually we feel we have done quite a lot of it,” rather than simply accepting it. We accept that we could perhaps be a little bit more consistent in some of those areas, but we do not want to put consistency above the appropriateness and relevance of response, case by case. This does not feel to us like a major issue, providing the content of the management responses is serious.

Sir Peter Luff

In a sense, it is not really the detail of what is in the page and what you say in response the report; it is the spirit of the exchange between your organisation and theirs. It is a sense of mutual shared understanding and objectives. So, what else do you do, in your work with the rest of the Department, to reflect ICAI’s recommendations to the broader organisation?

Richard Calvert

It depends very much on the kind of report. So, some ICAI reports are quite specific and quite focussed on particular countries and programmes. Some of them are much broader. I will take some examples of areas where we tried to pull out the wider learning. We are doing a lot of work at the moment on strengthening our programme management procedures. There are lots of things in different parts of the ICAI body of work that tell us things about ways we can strengthen our programme management. Nick has mentioned some of the examples; we have tried to reflect those in the new set of rules and guidance that we have put out in the organisation on programme management. In other areas, it is more about taking themes in a particular area like the private sector, and making sure that we are picking up key themes in the guidance we are putting out of the Department. In some areas, ICAI’s recommendations specifically will have much more prominence. For example, there are not many people in DFID who do not know the TMSA report and the detail of that. That is fine; we are completely comfortable with that.

Sir Peter Luff

Is that informal channels?

Richard Calvert

That is in formal channels as well. We have done a lot of formal communication on TMSA and the lessons that we needed to learn from that. There, we gave that report, as an ICAI report, a lot of prominence. In other areas, ICAI will perhaps sit behind some of the things that we are doing in the organisation. Although it will be communicating with the organisation on learning, private sector and other things, it may be less framed around the ICAI report itself.

Sir Peter Luff

One of the hopes we all have for ICAI is that it will legitimise aid spending and make the investment more acceptable politically. You got a pretty good billing from ICAI commissioners in the last session we have just had. He said there are a lot of lights hiding under very large bushels at DFID. You should be more confident and promote yourselves more assertively. He said you are doing a really good job. Do you sense that empowerment from ICAI’s work?

Richard Calvert

As a Department, it is incredibly important to us that we can tell a positive story about how we are spending money and what results we are delivering. There are a number of sources of challenge and assurance that we use. ICAI is a really important one. The NAO scrutinises us. We have got our own departmental results framework and reporting. Although we are not in any way complacent about what we do, we do feel that there is a lot of positive endorsement of the Department’s work, in all those sources of assurance and scrutiny. ICAI is one of those.

Sir Peter Luff

So, you feel confident and empowered.

Richard Calvert

Yes, but not to the point of saying that we have not got things to learn. Clearly, TMSA landed on us and was a major issue that we needed to deal with. Given what we do and the places we work, we have no room for complacency.

Nick Dyer

ICAI is a good thing. The organisation and the people who come across ICAI regard it as a good thing, and we have responded well to its recommendations. In terms of using the feeling of empowerment in terms of how well we are doing, part of that is how we communicate our successes. Cutting through, as we all know, is quite tricky. I suspect the press are more ready to pick up on the negative stories than the positive stories. We just have to work with that. We are working hard as an organisation to try to tell the positive stories, in terms of how aid works and what value it brings to the UK.

Pauline Latham

ICAI said your corporate results agenda has brought greater rigour and discipline, but it can also distort programming choices. For example, it says that pressure for early results has led to a focus on “quick wins” and easily scalable outputs. Do you agree or not?

Richard Calvert

We do not agree on that. When we first put in place the departmental results framework in early 2011, we were very conscious of the fact that much of what we do is not susceptible to a simple quantification of results. The framework that we set up, and the resourcing plan that we put around that, very consciously did not attempt to turn every programme we do into a set of published results. On the other hand, there are lots of things we do where we feel it is absolutely right that the Department can tell a clear story about what it is delivering. We have been very conscious over the last three or four years of keeping this balance right. We should be clear about results that we intend to deliver. Where they are clearly measurable, we should be accountable for those and we should publish information. Equally, there are lots of difficult things that we do, and will need to continue to do, that simply do not fall into that category. Within the Department, we feel that the guidance we have had from Ministers and the decisions that have been taken on continuing to do difficult things in difficult places enable us to keep that balance about right. Pauline or Nick may want to add to that.

Nick Dyer

The other part of the criticism is that we are choosing the easier results to measure, rather than the more difficult results, in terms of impact. That is a wider challenge, in terms of whether we are going to measure our contribution or our attribution to results. There is a question as to whether it is realistic as an organisation to be measuring real world outcomes that are a function of lots of other things happening, or if we are going to focus on the outputs or even the activities. If you look at our results framework, we have got a range of different results across the different levels. We probably need to be a bit more consistent on that. Focusing on what is within our control is probably the fairer way of measuring if we are having success.

Pauline Latham

Look at something liked gender-based violence. The Secretary of State has put women and girls right at the forefront, and there has been quite a lot of work done on FGM. We witnessed that in Ethiopia. Sensitive, evidence-based approaches can take a very long time for behaviour change when you are talking about things happening village to village. A country like Ethiopia is huge, and you have got to take the population, the Imams and the faith leaders with you. Because it is still a very male-dominated society, you have got to take the heads of the villages with you to change that behaviour. In areas such as this, how can you balance that with having to demonstrate the results that you need to deliver?

Nick Dyer

That is a really good example of an issue that you really have to stick with. It is going to take a generation; this does not happen overnight.

Pauline Latham

Yes, probably more than that.

Nick Dyer

Perhaps more than that. We have got a 10-year programme in Sudan. It has committed itself, albeit over two phases, because it recognises it is going to take a 10-year frame. It is necessary to be realistic about measuring the interventions that we are doing and tracking whether they work or not, because this is a contested area. You can fall back a little bit on indicators of social change and attitude change, which you can measure as you go along. It would be unrealistic to expect the impacts, in terms of actually changing the practices, to happen in year one. You have to get measures about whether your interventions are happening, whether they are working and whether there are changes to cultural perceptions—and you can measure that. I agree with you that, in terms of actual impact, it would be unrealistic to expect that to happen from year one.

Pauline Latham

It is very difficult to prove that you have changed anything, because the concrete examples are not necessarily there yet.

Nick Dyer

They are not. One of the challenges that the Cabinet Office, in their What Works study, put to us was that we should do more quantitative trialling within DFID. You could take an example of an intervention in female genital cutting, and have one intervention in one area and a different intervention in a second area, and compare and contrast as to whether they are having a different impact. There are technical ways you can to do that to bring to life the “how”, in terms of what is working, rather than the “what”. We have taken that message away and have started to think about how we could do that. I agree. In areas where, frankly, we do not know what works, you have to test it as you go along and take some punts.

Fabian Hamilton

We have already discussed TradeMark Southern Africa and that now-famous ICAI red-rated report from December of last year. Can I come back to that? In your response to that ICAI report, the Secretary of State immediately closed TMSA down, and announced that programme controls would be strengthened throughout DFID and that improvements would be made to DFID’s internal audit. I wonder if you could update us on what action has taken place since then.

Richard Calvert

The TMSA report coincided with some work that we were already starting to do in the Department on reviewing our programme management procedures with a view to strengthening the delivery as well as the design components and, in some cases, streamlining the way we worked. What has happened specifically since then is that we immediately had an exercise across the Department to ensure that we were more consistent in completing, and ensuring that sufficient attention was given to, annual reviews. One of the lessons from TMSA was that the annual review process was not working as well as it should have done. We had an immediate exercise in December and January to make sure that we were up to date across the Department with all our annual reviews. Secondly, we have been working on a new set of programme management procedures and control points throughout the programme cycle. This has been brought together in something we have called the new Smart Rules for the organisation, which you may have heard about in other sessions. Smart Rules is, essentially, the new programme management tool kit for the organisation. That was launched on 30 June. A key element of that, which also picks up issues that were raised in TMSA, is the establishment of a senior responsible owner, for each project intervention in the organisation. One of the issues that I personally took from TMSA was lack of clarity about who was really in charge of managing the programme. So, all programmes will have a senior responsible owner. We have an initial cohort of 50 SROs who have been identified and who have had an initial training session. We will be using that cohort, which represents all the business units across DFID, to roll out this SRO model across their teams and across all interventions. By the end of 2014, we will have SROs in place for all our activities. We have implemented a programme of programme management accreditation across the organisation. Again, one of the things we had recognised in the organisation was that the programme management profession was perhaps under-recognised, set against some of the other professional groups in the organisation. So, we have created a framework for programme management professionalisation. We have had an initial group of people accredited as programme managers. We have put in place a learning and development programme, for both programme management staff and those aspiring to programme management roles to undertake accredited learning. Within the organisation, we see this set of issues as probably our biggest single internal change exercise over the 2014 to 2015 period.

Fabian Hamilton

Was most of that a reaction to that ICAI report?

Richard Calvert

We kicked off what we call the end-to-end review of programme management in early 2013. So, that work was kicked off around February and March. What TMSA did was give us a bit of a wake-up call on that. If you like, it added some extra edge and challenge to that process. We would have been on that track, but TMSA gave us some—

Fabian Hamilton

It was like a catalyst, to speed the whole thing up.

Richard Calvert

Yes.

Fabian Hamilton

I wondered whether, with your existing DFID instruments for, for example, supporting the private sector, there were ways to boost investment and infrastructure, such as the Private Infrastructure Development Group, and whether you are confident that there has not been damage to all those other bits, for example through the Private Infrastructure Development Group, that have worked successfully? Infrastructure is so important in development. Also, we are concerned on this Committee that, following what happened with TMSA, you might have become excessively risk averse. Can you reassure this Committee that you are still prepared to take risks where the rewards are great, but that you are monitoring them adequately and properly so that we do not have another TMSA?

Richard Calvert

We very much believe that is where we are. Even if you take the challenges that TMSA were seeking to address, to promote trade and development across Southern and Eastern Africa, that is not something that we are going to give up on. We are looking at the moment at how best we can support that going forward. There are clearly important lessons to take from TMSA about how we do that. Equally, across our other instruments supporting investment or economic development, we see that portfolio continuing to grow. Also, we feel there are some real successes in that portfolio. I do not at all think that the issues with TMSA have left us feeling that we cannot go into that area of trade and development and private-sector development work.

Fabian Hamilton

So, you are not more risk averse than you were.

Richard Calvert

No.

Nick Dyer

Part of the change in the programme management rules is about shifting the culture of the organisation in two ways. One is to focus less on design and more on implementation, which is a consistent challenge from ICAI and others. The other is to empower people to take accountability for their decisions, so that rather than it all coming back to the centre, people can take decisions themselves. Part of that needs to go alongside a culture of avoiding a risk of people being afraid to fail. That is an issue, particularly in the world of scrutiny and criticism. If you are in a space where people feel they are just going to get criticised and punished, then that is not going to work. How then do we do that? It is twofold. One, it is giving the signal—and as a management team, we are very much clearly giving the signal—that well-managed failure is fine. That is a slightly odd phrase. What we mean by that is so long as you are following the systems and the rules, and you are doing your projects properly, things happen. Things do not work. If they do not work, that is fine; if they fail, that is fine. So long as you manage your processes well, that is an okay outcome. Badly managed failure is not fine and should not be tolerated. That requires a shift in quite a lot of the culture in the organisation and for people to feel that they can raise issues and problems early. We are certainly giving those signals, and certainly I hope that, as an organisation, we are starting to surface more things that have gone wrong and we are identifying them earlier.

Jeremy Lefroy

Given that the organisations with which DFID works would be expected to have proper financial controls in place, would it not be possible just to ask them for a very simple figure at the end of every year of funds they have received that have not yet been applied to the programmes that they are contracted to deliver?

Richard Calvert

That is something that we would expect a good programme manager to be doing.

Jeremy Lefroy

Could that not be compulsory across all programmes? Then, something like TradeMark Southern Africa would just be flagged up immediately.

Richard Calvert

The clear expectation—and it is reflected in the guidance we have—is people should know what is happening to the money they are responsible for. So, if that has been lodged with an intermediary for good reasons, they should be completely on top of that. Clearly, in TMSA, we were not. The problems with TMSA were not so much that the processes said it was okay but that we were not doing what we should have been doing.

Jeremy Lefroy

If I could just come back to that question, is it not possible just to have that figure, which enables you to say, “Yes, we have got a problem here, because of the £20 million that we gave them, £18 million is still there”?

Nick Dyer

Clearly, we should not be paying in advance of need.

Jeremy Lefroy

But we do.

Nick Dyer

There have been instances in TMSA where that was one of the challenges. There were two reports we should expect every single project to be doing. One is its annual review, which is looking at the development impact, and the other is its financial report, which may go alongside an audit report as well. The challenge for us as an organisation is that we actually take those reports, look at them and respond to them. If you look at how many annual reports have been late and delayed in the past, we have now corrected that. Clearly we were not doing that well enough. That is an expectation of good programme management. That is something that will be part of the training process. A senior responsible officer, who is responsible for a project, would be expected to know that.

Richard Calvert

We also have clear rules in the organisation that payment in advance of need is not allowed. Our auditors—the NAO, as well as our own internal auditors—look very closely at payment in advance of need every year.

Jeremy Lefroy

At the end of 2013, as far as I remember, we made some very substantial commitments to both the Global Fund and IDA, as part of our 2013 aid spending. Were those given as cash or were they simply as commitments?

Nick Dyer

They were given as commitments. We may well have lodged a promissory note—which is basically a Bank of England promise-to-pay—but the cash is only drawn down and released when it is needed. In many of our multilaterals, that is how we operate. They need the commitment, certainly in the Global Fund’s case, to be able to do the programming and make commitments themselves to countries. However, they do not actually ask for money, and we do not give them money. The Bank of England does not release the money until it is actually required. That release schedule is based on their liquidity policy and an understanding about the cash flows of other donors, and then you make a decision about when to lodge the promissory note and when to draw it down.

Pauline Hayes

At the field end, staff go into a lot of detail before they agree to release funds. They check in advance of disbursement very carefully. They know that they are not supposed to advance in that way. Occasionally, something will happen, because we obviously ask our partners to give us that confidence and assurance. Sometimes, in some environments, stuff can happen and the money disburses more slowly, for example in Afghanistan or Pakistan. Generally speaking, however, that rule is very clear to staff.

Jeremy Lefroy

Nick, you explained very clearly that it is quite possible to have well-organised and well-managed failure, which I do understand. Can you give me examples of programmes where that has happened? How do you report it? The annual report tends to be full of stuff that is working, which is great and very positive, but are you open about this publicly, as well? Do you say, “Yes, we have got programmes where everything was well-arranged and well-managed but, due to circumstances which are completely beyond our control, we have not been able to deliver those results?”

Nick Dyer

I will give two examples. One is an example of what does not work. We recently funded a trial for children who have severe malaria. Past practice has been to rehydrate them—give them liquids. We now know that is actually dangerous, and we should not be doing that. We have stopped doing that; we have stopped doing that in the UK as well, on the back of this trial. That is an example about understanding what works and implementing that message. On projects that are not working as well as we would expect, we do now, as a management team, get a regular report of all the projects that are scoring Bs and Cs, in terms of our reporting. We interrogate those at the board, because they come through. We are now flagging up and identifying early projects that have got issues and problems. Some we have closed down, as well.

Jeremy Lefroy

How long has that been going on for?

Richard Calvert

We put in place the new scoring process about three years ago. The internal reporting has stepped up, as part of this work over the last year on programme management. The consistency of our scrutiny at board level has changed over the last year.

Chair

ICAI’s report questioned the sustainability of the programmes in Afghanistan. The Committee has also highlighted that issue. They recommended that you consult more widely with the Afghan people. Is that happening? How is it happening?

Pauline Hayes

We took on board that recommendation, and we are endeavouring to consult more widely with the Afghan people. We are designing projects and programmes as part of the normal design process. You will appreciate that our ability to consult widely is quite constrained, in that environment. To the extent we can, we are trying to react in a positive way to that recommendation.

Chair

Presumably, you can find some reliable Afghan partners?

Pauline Hayes

Yes. We can do it through third partners and stuff like that.

Chair

How has the election changed things? Do we know what the result is yet?

Pauline Hayes

The initial results were announced yesterday, but they are very preliminary results. There is some way to go yet. The situation is quite fluid, shall we say, at the moment in Kabul, because the losing candidate is obviously not accepting those preliminary results at this stage.

Chair

Is Ashraf Ghani still supposedly the winning candidate?

Pauline Hayes

Yes, from the preliminary results. There needs to be more auditing of results done. The two main electoral commissions need to work together on this, to provide assurances of the results. In terms of how the elections went, the first round went extremely well. It went very smoothly overall. Obviously, we are now at the critical stage. It is going to be a little bit difficult over next few days.

Chair

So, at the moment, you cannot judge your approach until you know the outcome. Is that the situation?

Pauline Hayes

I do not think, depending on who wins, our programme and the shifts that we planned will change fundamentally. We are aware of what both candidates have said about their vision for Afghanistan and their priorities, etc. We are confident that we can work with either of them in the areas that we care about the most.

Chair

Is there likely to be a deal between the candidates, or is it going to be one winner and one disgruntled loser?

Pauline Hayes

I think it is too soon to say. The situation is very much in flux, with the announcement yesterday.

Chair

Okay. I will not pressure you, in that case. It is their politics, not ours.

Pauline Hayes

Exactly.

Chair

Obviously, the Committee felt very strongly about the need to prioritise gender issues. You did adapt your programmes; indeed, you did change the emphasis as a result of that. What progress do you feel you are making? We made that the litmus test. We said that the progress with women that is sustained over the next two or three years will be the test as to what extent our activities have made a real difference.

Pauline Hayes

The big shift since you were out there is the support we are giving to women and girls, both from an economic development perspective but also a political perspective, in terms of their political empowerment. We have been doing a lot of work to support female candidates in the run-up to these elections and the provincial elections. That work will continue beyond this period, in terms of supporting them as they become elected representatives. It is a capacity-building exercise. We have also upped our game in terms of access to justice for women, partnering up with Australia. We are doing a lot more economic development work; we are actively engaging with women. We have specific projects that benefit women. Our Comprehensive Agriculture and Rural Development programme is designed to improve agricultural production. We also have some private-sector work. If you were to go back now, you would be very pleased in terms of what you are seeing with those changes. The other thing you talked about was the importance of the agricultural sector. We have just agreed a second phase of our Comprehensive Agriculture and Rural Development Facility, CARD-F. We are giving great emphasis to that sector, and will continue to do so over the next five years. Agriculture is obviously the bedrock of the economy there.

Chair

Again, somewhat echoing the Committee situation, ICAI found a risk that our aid to Pakistan would continue to increase, but against a background of a lack of concrete progress on areas like anti-corruption and tax reform. As you know, the Committee took a very strong view that, unless the elite are making a proper contribution, it was wrong that the programme should escalate from ordinary taxpayers in the UK, effectively. Is progress being achieved? I know there has been a unit set up to work on the tax issue. Are people actually beginning to pay tax, or even say that they are going to pay tax, given that they were not before?

Pauline Hayes

The programme has not scaled up as rapidly as we anticipated two or three years ago. There are a number of reasons for that. Last year Pakistan had elections, and that slowed down our work generally there, for some time. It is also a recognition of the realities of working in Pakistan, and designing and developing programmes. You will see that the trajectory has not been as rapid as we anticipated three years ago, but it is still going up slightly this year. In terms of tax, the new Pakistan Government, when it came into power last May, very quickly set to work on agreeing a new IMF programme. That has a very strong emphasis on tax reform, as you are probably aware. There are quarterly reviews of that IMF programme, the last one being last month. So far, the Government has delivered on the targets, including on tax work. They are shaping up well. They produced a budget in June, which set out a lot of commitments that would assume a rise in tax. The IMF has been looking at that quite closely, and will continue to look at it closely. The acid test is now that they have set out their stall in this budget, can they deliver it over the next 12 months?

Chair

I do not know whether other members of the Committee did as well, but I got emails from concerned middle-income taxpayers in Pakistan saying, “We are worried that your report means that what will happen is they will just come after us. They will want to get more out of us, and the people above us will continue not to pay.” That seemed to be a perfectly legitimate complaint. Are you satisfied that the people at the top, whether they are politicians or business leaders, will in fact make a contribution, given that a lot of them are paying nothing at all?

Pauline Hayes

I have heard a similar complaint when I have visited. The phrase used was “milking the same cow”. They definitely do need to get to the people at the top. A tax directory has issued MPs and others have to register their interests, and so on. We have just got to keep the Government’s feet to the fire on this agenda. We do raise it all the time at the highest levels. So far, they have met their commitments and the actions in the IMF programme. We just have to keep at it, because it is critical to get the tax take up.

Chair

You have the Committee’s encouragement and support.

Pauline Hayes

I know.

Nick Dyer

One of the programmes that we have created is a capacity-building support from HMRC. We are making that available to other countries on a demand basis. One of the countries that HMRC is now working with is Pakistan. There was an inward visit from the head of the Pakistan Revenue Authority.

Pauline Hayes

It has gone both ways. We had the Pakistanis here, and HMRC have gone to Islamabad. They have actually signed an MOU now, setting out the level of co-operation.

Fabian Hamilton

In its follow-up of its Year 2 report, ICAI was concerned at the lack of progress in the conflict pool and the significant risk issues that it identified in its report will continue to be magnified with the new CSSF. I wonder if you could reassure us that that is not going to happen.

Pauline Hayes

Yes, certainly. CSSF is a different beast. It is definitely a different beast. For one thing, it is not just three Government departments; all Government departments are taking part in it. Also, every country has to develop a strategy that the NSC will scrutinise. From that strategy, they will then produce a bid, which the CSSF will fund. We are taking a more strategic approach to this agenda. I recognise that ICAI was concerned that the monitoring and evaluation frameworks would be weak and so on, but we are setting up the mechanisms for the CSSF now. A lot of work has been done in the last few months at the centre of Government on this. Countries are producing their strategies. The CSSF bids will be submitted over the next two to three months. On the monitoring and evaluation issue, there still is a need to do more work there. I would agree with ICAI on that. We have an opportunity at DFID, because we are well used to developing these kinds of frameworks. We can share what we have learnt with other Government departments, in terms of getting this right. There is a new CSSF secretariat at the centre, and we have got a few DFID staff in there now. Again, they can help shape the governance arrangements and so on for CSSF. We see this as a different beast. It is not just going to be more of the same.

Fabian Hamilton

Are you confident the Conflict, Stability and Security Fund, CSSF, which replaces the conflict pool, is not going to be involved with shoring up militarily some of the organisations and countries that apply for funding? In other words, is there any danger at all that CSSF will be used for arms sales, or for anything that promotes violence?

Pauline Hayes

Well, of course not.

Fabian Hamilton

Forgive me, Pauline; it sounds like a strange question. The reason I raise that is because I have had constituents who have expressed concern to me that this change from the conflict pool to the CSSF will actually allow arm sales to take place, in order to shore up stability. In an effort to promote stability, it will involve armaments. That is obviously a concern they have; that is why I am raising it.

Pauline Hayes

I do not think that is going to happen, and we will certainly do our best to make sure of that. I would say a large proportion of CSSF—I need to check the details—is classed as ODA. That obviously ring-fences what can be done with it.

Fiona Bruce

ICAI remains concerned that DFID has yet to develop appropriate strategic guidance to cover the appointment of contractors and perhaps the learning process from them. Could you tell us how you are going to address this?

Richard Calvert

We have got a big programme of work on strengthening our approach to commercial capability and the commercial performance of the organisation. It is true that ICAI thought we should have some kind of strategic guidance on the types of contractors that should be used in different situations. We have not taken the work forward quite in that way but, as the follow-up report recognised, we have done a lot to strengthen our capability and expertise across the organisation. We have also put in place much stronger practical guidance for people who are taking decisions on programme design. We have taken it down a level in the organisation. Clearly, the choice of individual contractors and the commercial approach needs to reflect the circumstances of each intervention. From my perspective, we have followed the spirit of ICAI’s recommendation, in terms of saying, “You need more guidance and capability within the organisation,” but we have taken it forward in a slightly different way from their original recommendation.

Fiona Bruce

That is interesting, because they say that when the choice is left to programme teams on a case-by-case basis, there can be a serious gap in the strategic approach to using contractors. Are you addressing that?

Richard Calvert

I would have accepted that 18 months or two years ago. I would not accept that now. The things that we have done to strengthen our performance are: firstly, all our Senior Civil Service staff are going through a three-day commercial learning programme. We have put in place a network of specialist commercial advisers across the organisation. We have put in place a key supplier management process with our top dozen or so suppliers. We have put in place much more lesson-learning and sharing, both within the organisation and between DFID and its suppliers. Although I think some of those criticisms would have been fair a couple of years ago, now we feel in a much stronger position on that.

Fiona Bruce

Can you say why the amount of aid delivered by DFID through contractors has risen? It has risen from about £0.9 million in the previous year to £1.4 million in 2013-14.

Richard Calvert

A lot of this reflects the growth in the programme over the last three to four years. We have seen the DFID portfolio, as a whole, rise from a little over £7 billion in 2010-11 to in excess of £10 billion. We have seen some shifts in the balance of work within the portfolio, but the growth in the contractor component is just reflecting that overall programme growth.

Fiona Bruce

Were you in the earlier session? Did you hear the evidence on the relationship with UNICEF?

Richard Calvert

No, I did not, but we know the case.

Fiona Bruce

There was quite concerning evidence there. We would very much want to know how you have developed your strategic oversight of multinationals and NGOs, to avoid that kind of issue reoccurring?

Nick Dyer

Our relationship with multilaterals is twofold: we give core support, through our replenishments, and we use multilaterals as a delivery mechanism from country offices. They put money through multilaterals, in terms of delivering, as in the case of Nigeria and UNICEF that you were mentioning. The expectation is that the country offices themselves are responsible for the delivery of the projects that they are agreeing with the individual multilateral. In the core support, we have a range of mechanisms by which we are engaging with the multilaterals to try to improve their performance. I know one of the challenges has been that we are not doing enough to bring our in-country experience to bear in our engagement with multilaterals. It is getting better. I suspect one of the challenges is that, even if we were to do that, across all our programmes we have still only got 28 country programmes, and many of these organisations are dealing in 70 or 100 different countries. There, it is a question of what knowledge we have, alongside building and drawing on the knowledge of other donors and the multilateral systems themselves. The example of UNICEF is quite interesting, because, in response to the ICAI report on UNICEF, we started doing these portfolio development reviews. This was to learn the generic lessons from our work in-country, and take that back in terms of trying to shape our engagement in the core support that we give to UNICEF in New York.

Fiona Bruce

Are you satisfied that that kind of fairly major concern that was raised by ICAI will not be raised in the future?

Nick Dyer

In terms of the specific challenges in Nigeria, northern Nigeria—as you have seen from the press—desperately needs support, particularly girls in terms of education. It is an incredibly difficult place to work. The process since the ICAI report has been that we agreed to do a review after six months, through the annual review. The annual review was done. The project was then put on a performance improvement plan. We have now got the second annual review just going through our internal quality assurance. Once that internal quality assurance has been completed, we will make a decision about what is next in terms of the programme. It is a question about whether these are specific issues for UNICEF in the very difficult circumstances of working in Nigeria, or they are more generic for the organisation. We can take the generic lessons back to New York, but the specific challenges of working in Nigeria are for the Nigeria country team.

Fiona Bruce

What interested us was that there was another NGO working in Nigeria on a very similar project, which seemed to be more effective. Is there not a concern that you have these very large multinationals that have been established a long time, and that there is not sufficient scrutiny by DFID, on a case-by-case basis, of what they are doing?

Nick Dyer

That raises two questions; one is the scrutiny. I would argue that the combination of our in-country, project-by-project scrutiny plus what we are doing at board level actually does amount to a sufficient amount of scrutiny. The second issue that is implied by your question is whether we are giving enough competition to the choices we make on whether we put our money through UNICEF rather than through an NGO.

Fiona Bruce

Absolutely. That is another question, indeed.

Nick Dyer

I would say three things to that. In many of the places we work, there is not a huge amount of competition because these are very difficult places. Where the competition might exist, it cannot work at the scale we want it to, but that is a question of us engaging with that. The third element points back to what Richard said on encouraging our staff to take a more commercial, market-shaping approach to our work. The work that we have done on commercial advisers and commercial skills will make and is making our staff more cognisant of the fact that, actually, part of our responsibility is to shape the market as well as to direct resources through one organisation. I would hope that we will start getting better at that, because you are absolutely right; we should not just accept that UNICEF are the only option. We should be looking at the wider market and making choices that will build that market as well as deliver the project.

Fiona Bruce

You should not accept, because UNICEF has a certain reputation, that the scrutiny or the oversight does not need to be as strong as with any other project or any other organisation.

Nick Dyer

Organisations evolve, and we have to keep our scrutiny up.

Chair

ICAI were also concerned that you seemed to be taking too long. They understood that you would have finished this process by the end of last month, and you are now telling them it will be a few more months.

Nick Dyer

I checked this morning, after we had spoken to the commissioners yesterday. The annual review has been completed, but because of the seriousness with which we are taking this, we are putting it through a peer review, internally. We are putting it through our quality assurance unit just to make sure that the methodologies, conclusions and analysis were up to scratch. That process will be finished some time this month.

Fabian Hamilton

You have recently decentralised your evaluation function. Are you comfortable that there are clearly defined boundaries between your self-evaluation function, the work of ICAI and the NAO? I wondered what you feel is ICAI’s value added on evaluation, given that their budget is so much smaller than your own self-evaluation function. I know that you do not keep a total record of what you spend on self-evaluation, but ICAI estimates it is over £200 million. That is huge compared with the budget of ICAI, which is about £3 million. In its recent report on how DFID learns, ICAI said that DFID staff failed to apply lessons from your self-evaluation to programmes. How would you respond to that? Do you agree that information from evaluations should be made into a usable form?

Nick Dyer

In terms of the relative function of ICAI, in respect of all the other scrutiny, internal and external, the triennial review looked at that. The triennial review concluded that, particularly with the change in ICAI’s function to do more thematic studies in more depth but fewer of them, it was a good added value to the process of the scrutiny. That is providing an independent view on some themes. The learning study and the private sector study show the value of doing that. Yes, there is still a good role for ICAI in relation to that. In terms of the decentralised evaluations, in 2011 we were doing 12 decentralised evaluations a year. We are now doing 40. So, we have scaled up the number of evaluations. In terms of how much we spend on evaluation, we are doing two types of expenditure on evaluation. We are spending on our own programmes, and calculating the cost of running our own evaluation. The evaluation framework agreement, which you discussed last year, to date has now agreed 44 contracts. There are another 12 in process. The total spend will be about £48 million. I suspect the figure looks bigger because we also fund evaluations of other donors. We fund some World Bank evaluations, and we fund other organisations who do evaluations to allow them to do evaluations. We are doing an awful lot of evaluation. The challenge for all this evaluation work is whether we are doing the right evaluations and evaluating the right things. We did a review this year of our evaluation strategy. We have come to the conclusion that we are not necessarily evaluating the right things. You want to evaluate where there are gaps; you want to evaluate where we do not have sufficient evidence to give us good material; you want to evaluate programmes of a particular size. We want to become smarter at deciding which projects to evaluate. In terms of lessons, last year we did an annual review of our evaluations. We tried to pull all the lessons into one place. We are going to do another annual review this year. By September, we will have tried to pull together all the lessons from the evaluations that we have done internally.

Fabian Hamilton

Nonetheless, do you think you can justify more than £200 million being spent? That is a phenomenal amount of money.

Nick Dyer

Go back to: do we know what works in development? The answer is that, actually, in some cases, yes. In a lot of cases, no. There are huge evidence gaps in evaluation. This is not just about DFID; this is about development. Do we want to fill those gaps through our research, our evaluations and our own internal operations? Yes, we do. The return from knowing what works and stopping what does not work well justifies the investments we are making in evaluation and evidence generation more broadly.

Fabian Hamilton

Are you confident that the public, who fund this, will agree with you?

Nick Dyer

If we are demonstrating what we are stopping doing instead. The example I gave earlier, on the FEAST trial, is incredibly good value for money. We are stopping children dying, because of a bit of research and a trial we did. We can demonstrate a line of sight between the research we funded and changes in practice that will save lives.

Fabian Hamilton

Can I finally pick up on a question I asked the commissioners earlier? I do not know if you were in the room for this. I asked them about the transition to ICAI’s next phase, in May 2015, and whether it would be smooth and well-managed. The response was, “Not as smooth as we would like, given the number of changes that are taking place,” because all of the changes are at the same time. I wondered whether you were concerned about the lack of continuity regarding commissioners and the staff. What would you like to see done differently in ICAI’s next phase, after May 2015?

Richard Calvert

We are happy with the overall plans for transition. We recognise that there is a lot changing in spring 2015, but we recognise that some time ago we put in place plans, in relation to commissioners, contractors and the secretariat function, that we think will enable us to have a sufficiently smooth transition. We are building in, for both the contractors and the commissioners, a degree of overlap. At the moment, as you know, we are in the process of seeking to recruit a new chief commissioner designate, and also a further commissioner who will be appointed over the coming few months. There will then be an opportunity for them to shadow the current commissioner team through the first half of 2015. We have got a process already under way to appoint a new contractor, and we would aim to have the new contractor appointed by around the turn of the year, 2014 into January 2015. Again, that should give us time for overlap. We have also agreed short extensions to the contracts for the current commissioners. Also, we are prepared to look at the wind-down of the current contractor programme of work. We are also talking with the commissioners and the secretariat about potentially shifting some functions out of the contractor responsibility into the secretariat team, to give a little bit stronger permanent capacity in the secretariat. This is a process we need to manage, and we need to make sure we stick to the deadlines. We are confident that we have got a process that can avoid a sense of everything stopping and then having to be restarted again in the spring. In terms of where ICAI goes in its next term, to some extent that is clearly down to the new commissioner team: to shape the programme and to decide where they want to focus. If you go back to the triennial review, that gives the main steer on where we see ICAI going in the future. Clearly, that pointed towards, perhaps, fewer and more in-depth studies. We are already starting to see some of that reflected in the way the ICAI work programme is going. The core function, however, as Nick said in his earlier answer, has been really reinforced.

Chair

Do you have any problems with the possibility of ICAI relocating into DFID’s offices?

Richard Calvert

There is a very good value-for-money argument for doing so, in that if we can accommodate ICAI without cost in the 22 Whitehall building, rather than paying a rent somewhere else in central London, that is a very good reason. We have talked to the commissioners about this, and we are clear that if we do go down that road, which is a very serious possibility, then we will look to find them a sufficiently discrete corner of the building so that they can feel they have got their own space and independence.

Fabian Hamilton

You could build a Chinese wall.

Richard Calvert

Maybe even more than a Chinese wall—maybe a large, 18th Century wall.

Chair

Thank you very much indeed. The relationship will continue, obviously, and I guess some of these issues will come up when we do have the Permanent Secretary in front of us for the annual report. They certainly informed that relationship. Thank you very much for today’s evidence. Oral evidence: ICAI Annual Report, 2013-14, HC 523 21