- International Monetary Fund. Independent Evaluation Office
- Published Date:
- October 2005
Summary of Major Findings and Recommendations
This evaluation examines the technical assistance (TA) provided by the International Monetary Fund (IMF) to its member countries. The evaluation is based on desk reviews of a broad sample of countries, analyses of cross-country data on TA, six in-depth country case studies (including field visits and interviews with counterparts, authorities, and donors), reviews of past evaluations, and interviews with IMF staff and other stakeholders.1
The objective of IMF TA was defined in the most recent policy statement on the subject: “The objective of the IMF TA is to contribute to the development of the productive resources of member countries by enhancing the effectiveness of economic policy and financial management. The IMF TA works to achieve this objective in two ways. First, much of the IMF’s TA seeks to support the efforts of members to strengthen their capacity—in both human and institutional resources—to formulate and implement sustainable, growth-oriented and poverty-reducing macroeconomic, financial, and structural policies. Second, the IMF assists countries in the design of appropriate macroeconomic and structural policy reforms, taking account of the lessons learned by other countries in addressing similar economic policy concerns.”2
The above objective encompasses a very diverse and broad range of TA activities. It includes not only support for long-term institution building in the countries themselves, but also fact-finding and analytical activities that allow the IMF to advise countries on specific policies or in the design of IMF-supported programs. This very wide definition presents special analytical challenges, both in identifying TA priorities as well as in evaluating the effectiveness of TA.
With this in mind, and in an effort to derive operational lessons to enhance the effectiveness of IMF TA, this evaluation unbundles the TA process into three stages—prioritization, delivery, and monitoring and evaluation of impact—and asks three sets of questions:
- Prioritization and resource allocation. How are countries’ TA needs identified? Are they derived from a process able to compare short-and medium-term needs or priorities across sectors and reallocate resources effectively, taking account of evidence on outcomes? What can be done to make the process more strategic so as to increase the relevance of the TA program?
- The delivery process. How effective is the delivery of TA? What factors influence the effectiveness of the various modalities of delivery, including the country-specificity of the diagnosis and proposed solutions, as well as domestic ownership of the recommendations?
- Monitoring progress and evaluating impact. How is progress being tracked and what factors contribute to the impact of TA? What factors are under the control of the IMF or outside it? What can be learned to improve the IMF’s own monitoring and evaluation activities so that lessons from past TA can be a guide for future allocations and design of TA?
In trying to assess the effectiveness of technical assistance, we distinguish between the impact at different stages of the results chain—the immediate improvements in the technical capabilities of agencies receiving TA; the ability of agencies to then apply and enforce that increased capability; and whenever possible, ultimate outcomes on the ground. Attribution of results to the effects of TA clearly becomes more difficult the further out the results chain we go. While such assessments inevitably rely heavily upon qualitative judgments, we have drawn upon various performance indicators and benchmarks wherever possible. (These issues are discussed in greater detail in Chapter 6.)
The first section of this summary presents the main findings. The second section summarizes the major recommendations.
Resources devoted to TA (Chapter 3)
- The IMF provides annually about 300 person-years of direct TA services, amounting to $80 million and equivalent to about 10 percent of the gross administrative budget of the institution.4About 30 percent of it is financed externally. Taking account of indirect support activities and other allocated fixed costs, total expenditure on TA and related activities amounts to about $190 million, or about 23 percent of the gross administrative budget.5 Whatever definition is used, the volume of IMF TA is quite small relative to efforts by bilateral donors.6
- About 70 percent of IMF TA is directed to countries with per capita income below $1,000. TA seems to be well targeted toward the low-income countries. In fact, we found that the less the per capita income of a country the more the TA provided by the IMF (corrected by population and other factors). The volume of IMF TA provided to countries also seems to be positively associated with the country having a program supported by the PRGF or the Extended Fund Facility (EFF), the amount of external financing available, and whether the country is a post-conflict economy.
How TA priorities are being set (Chapter 4)
- At present, TA activities seem to be driven, in large part, by the specific needs of IMF-supported programs and by the new Fund-wide initiatives, such as Reports on the Observance of Standards and Codes (ROSCs), Financial Sector Assessment Programs (FSAPs), and anti-money laundering and combating the financing of terrorism (AML/CFT) assessments. These initiatives have become more important drivers of TA activities than the country-specific policy directions and priorities that emanate from Article IV consultations or the Poverty Reduction Strategy (PRS) process.
- We found a weak link between TA priorities and Poverty Reduction Strategy Papers (PRSPs) or with key policy issues identified in Article IV consultations. In most cases, the PRS process has still not been able to clearly identify major capacity-building needs that could then be taken up by TA. This is a major shortcoming because the PRSP was expected to become the vehicle to provide guidance on broad priorities for the IMF in low-income countries.
- TA activities do not appear to be guided by a medium-term country-based policy framework that would set TA priorities in the IMF areas of expertise across sectors, program needs, and institutional initiatives, and that would be able to balance TA demands stemming from short-term policy needs with medium-term capacity-building needs. This also reduces the possibility of a demand-driven reallocation of TA resources across areas of expertise within the institution.
- That said, individual functional departments have taken a number of important initiatives to be more strategic in setting intra-sector TA priorities and looking at track records in implementation. For example, the preparation of country strategy briefs, departmental assessment missions, and a more country-centered system for evaluating TA requests are facilitating a more coherent process to identify priorities within functional departments. However, they are not a substitute for a medium-term policy framework that can compare a country’s TA needs across departmental lines or institutional initiatives.
- Several initiatives were introduced in 2001 in an effort to provide more strategic coherence to setting TA priorities. These included a system of prioritization filters for TA requests and a pilot initiative to incorporate technical assistance consultations (TCs) into Article IV consultations. The latter were discontinued because staff concluded they were not helpful in identifying TA priorities, while they unduly overloaded Article IV consultations. This evaluation believes the TC initiative was appropriate in spirit but flawed in implementation. There were few incentives for staff and the authorities to take the exercise seriously, since there was no clear link to actual allocation decisions. It was probably also excessive to attempt such an exercise every year. In any event, most of the TC exercises became pro forma, with no serious assessments of how well previous TA had worked or attempts to identify priorities for future TA linked to the country’s overall strategy.
- The TA filters, as currently formulated, are a poor vehicle to provide a meaningful and strategic basis for TA prioritization within and among countries. Some are too broad, hence they do not provide sufficient selectivity (for example, no theme is excluded from the “key policy initiative” filter); some inadvertently exclude areas where there is currently a significant amount of TA being provided (for example, the “main program areas” filter excludes middle-income countries with IMF-supported programs); while some can even give inappropriate signals if followed too mechanically (such as, “a priori” bias in favor of short-term TA missions independently of country context). Taken together, the system of filters appears to be “over determined.”
- What is needed is a prioritization of TA based on a shared vision with the authorities on a medium-term (multiyear) policy framework. This would allow the IMF and the authorities to compare TA priorities across demands emerging from programs, institutional and departmental initiatives, policy directions emanating from Article IV consultations, and (when applicable) the PRS process. It would also put into perspective short-term TA needs (associated with policy advice) with longer-term institutional building, and prevent short-term demands from dominating the TA program.
- Deriving TA priorities in the context of a medium-term framework would also help early on the discussion with other TA providers on an appropriate division of labor. This is a critical first step to improve the coordination of TA with other providers—an area where both IMF staff and donors felt improvements were needed.
- For this process to be successful, area departments must play a leading role. While initiatives taken by the functional departments permit a better prioritization within a particular sector, only the area departments are in a position to set priorities across department lines and between short-term and medium-term needs. Resident representatives should play a key role in this area given their in-depth country knowledge and links with the authorities.
- This evaluation finds that the short-term focus of the current budgeting and TA Resource Allocation Plan (RAP) processes within the IMF hampers medium-term planning for TA. Moreover, without a mechanism to enhance predictability in the provision of IMF TA over the medium term, it will be more difficult for the IMF to coordinate and leverage its TA efforts with those of other TA providers.
The process of TA delivery (Chapter 5)
- Counterparts in member countries have generally been satisfied with the resident experts provided by the IMF, particularly their hands-on role in training and coaching, accessibility, and emphasis on team work. These attributes were highly valued by counterparts.
- The evaluation does not support the findings of the 1999 internal IMF evaluation that TA provided by short-term missions is a priori more cost effective than that provided by resident experts. The relative merits of missions versus resident experts depend heavily on country-specific circumstances, particularly the degree of institutional development.
- The evaluation finds the involvement of the authorities in the preparation of terms of reference (TOR), particularly for long-term experts, to be generally passive. In fact, these are usually prepared by staff for approval by the authorities, frequently without significant engagement by the relevant local officials. This tends to reduce the ownership of the activity and often masks important differences in expectations about final objectives between the authorities and staff, as well as specific policy commitments that are necessary to assure the success of the TA activity. This is a particular problem when the officials in the agencies directly receiving TA, and in charge of implementation, are not sufficiently involved in the formulation process. The case studies suggest that greater up-front involvement by local officials was generally associated with better results.
- Often, TA effectiveness has been undermined by a lack of awareness of institutional, organizational, or managerial features of the recipient country. An unclear distribution of decisionmaking authority and accountability may constrain implementation. More fundamentally, a dysfunctional civil service is a major obstacle to sustaining capacity-building efforts. Progress in civil service reform is, therefore, essential to enhance the effectiveness of TA. While such progress takes time and is generally beyond the IMF’s control, a good understanding of the institutional context and absorptive capacity can be especially important in cases where there is a sudden and sharp increase in TA activity in new environments with which IMF staff is less familiar.
- Country officials suggested that more informal and iterative discussions on a broader set of options before the wrap-up meetings at the end of a TA mission would contribute to enhancing ownership of recommendations. A more interactive approach during the drafting of recommendations would help to ensure that constraints on the ground are fully taken into account. Again, this becomes more serious in short-term missions where less time is available for such interaction prior to the final drafting of recommendations.
- Inadequate dissemination of TA reports within and across agencies in the country is a major problem in environments where bureaucratic and institutional practices limit the free flow of information. This reduces the externalities of the TA activity—which is an especially serious shortcoming in environments with a significant rotation of personnel and hence weak institutional memory. The problem of dissemination is compounded by the fact that many TA reports are classified as confidential by the authorities.
- This evaluation found many instances of weak coordination between the IMF and donors working in similar areas. Coordination failures occurred even under the umbrella of the Technical Cooperation Action Plans (TCAPs). Communication with the World Bank is much better than with other donors, but coordination can still be hampered by differences in objectives, and organizational approach, which can result in various principle-agent problems in the absence of an overall coordinating framework. While coordination with donors should ultimately be the authorities’ responsibility, this is often not the case owing to weak institutional capacity and the fact that the PRSP is not yet sufficiently operational to play such a role in most low-income countries. As a result, the burden of coordinating donors’ efforts often falls to a major single donor or multilateral institution. When the overall involvement of donors is strong in a particular country, and the IMF provides a relatively small fraction of TA, it is not always possible or even appropriate for the IMF to provide leadership in coordinating overall TA efforts. However, it should still seek to coordinate better with donors working in similar areas and, in low-income countries, should help governments make the PRS an effective coordinating vehicle on which it can align its own efforts.
Monitoring the impact of TA and evaluating factors influencing it (Chapter 6)
- The case studies show that, as far as institution building is concerned, progress has generally been achieved in enhancing the technical capabilities of the agencies that the IMF typically supports. Significant variability was found, however, on whether agencies have been able to make full use of those increased capabilities in order to have an impact on the ground or on the ultimate objectives of TA. It is critical that the IMF should understand fully what prevents agencies from doing so.
- Part of the problem is that the present IMF documentation and reporting does not clearly unbundle and track the different stages of TA progress toward its final objectives. Specifically, this documentation is weak in:
- — defining at the outset what are the indicators (benchmarks) that will be monitored to judge whether or not progress is occurring, and how explicitly these indicators have been discussed with the authorities, for example, are these mutually agreed indicators;
- — differentiating between the outcomes of pointed policy advice type of TA (for example, how to restructure or close a specific state bank) and outcomes linked to longer-term capacity building in the same area (such as the ability of agencies to supervise the banking sector and improve their asset position); and
- — unbundling between short- and medium-term indicators that capture different stages of the results chain, for example (1) indicators that track the improved technical abilities of agencies receiving TA; (2) indicators that show whether these agencies are actually enforcing that increased know-how, for example, whether they are performing their final responsibilities; and (3) indicators that track the economic outcomes of that enforcement.
- The absence of a clear unbundling of these stages and the factors influencing the lack of progress limit the ability to use past track record in implementing TA in making decisions about future TA. This is critical because there may be good reasons why TA recommendations have not been implemented. An understanding of these constraints and what can or should be done to overcome them is crucial to setting future priorities.
- Frequently, political interference or lack of support by the authorities prevents agencies from using effectively the new knowledge transmitted by TA. Indeed, the case studies suggest that such resistance by vested interests may mount as these agencies increase their know-how and improve their potential to act. The evaluation found that in these cases the reporting from the field on constraints to progress has often not been candid enough, so that the ways to address such obstacles were generally not discussed frankly with the authorities.
- An understanding of such obstacles and what the authorities can do about them in the future is critical when discussing future TA activities. Up-front commitments to facilitate the effective use of the new knowledge by agencies provide a critical signal of the authorities’ commitment and ownership.
- Tensions among government agencies, a high rotation of officials, and a weak judicial framework are major constraints to the effectiveness of TA. The weak judicial system often limits the ability of agencies to enforce the new knowledge created by TA activities. This constraint is particularly serious in the areas of banking supervision, tax administration, and customs.
- Establishing cash charges for the provision of IMF TA has been suggested as a rationing device that would simultaneously identify high-priority TA needs, reduce any presumed excess demand for TA, and help the IMF finance the cost of TA. The evidence collected does not allow us to reach definite conclusions on these issues—because we do not have a sufficient basis to compare the present situation with an alternative counterfactual.
- However, we are skeptical that cash charges would be a sufficient screening device to help signal ownership and commitment and identify high-priority TA needs. Furthermore, a cash charge policy needs to take into account that two-thirds of IMF-financed TA is directed to countries below $1,000 per capita income. Thus, significant cost recovery will involve charging countries in this income range.
- This evaluation believes that stronger involvement by the authorities in designing TA activities and early commitments to better empower the agencies receiving TA (to implement their new knowledge) are alternative screening vehicles to signal ownership and commitment, with the greatest potential for enhancing the probability of success of the TA activity.
We make six broad recommendations designed to improve priority setting, delivery, tracking, and evaluation. They assume—in line with the current policy statement on TA—that “capacity building” in the IMF’s primary areas of responsibility continues to be a major objective of the IMF’s technical cooperation activities. This requires more effective mechanisms than exist at present for meshing the IMF’s overall strategic objectives with a system for allocating TA resources within a country-driven framework.
- 1. The IMF should develop a medium-term country policy framework for setting TA priorities, incorporating country-specific strategic directions and linked to more systematic assessments of factors underlying past performance.
A number of managerial approaches to establishing such a framework are possible, but the following suggestions could be considered:
- In low-income member countries, the PRS process provides the natural vehicle to identify TA capacity-building priorities in the core areas of expertise of the IMF, although it has typically not yet been used effectively for this purpose. The PRSP should form the basis for a dialogue between the IMF and the authorities on TA priorities for the medium term, with a clear indication of what is expected from each side in order to improve accountability. It should provide the vehicle for collaboration and division of labor with other donors and hence help the overall coordination of TA. The IMF needs to engage countries to help them articulate their medium-term capacity-building needs in its areas of responsibility, in accordance with the PRSP.
- For other member countries, the institutional framework for deriving medium-term strategic priorities is less clear-cut, and a variety of approaches may work best, depending on country circumstances and the intensity of the members’ demand for IMF TA. In cases where there is a relatively high provision of IMF TA, the framework may require periodic in-depth TA consultations with the authorities (possibly every three years) comprising a retrospective analysis of past progress and a forward-looking exercise to identify priorities. To avoid the lack of incentives of the past TC initiative, the resulting TA priorities need to feed more directly into the IMF’s own mechanisms for committing its TA resources. For other countries, a less comprehensive approach, possibly built around Article IV consultations, may suffice.
- Resident representatives could play a greater role in developing these frameworks and this role should be explicitly acknowledged in the TOR for these positions.
- The RAP process should continue as the main annual interaction between the area and functional departments for determining TA activities. In addition to incorporating TA activities stemming from the medium-term framework, it will also incorporate new TA activities that may emerge from the needs of programs or from unexpected new demands from the authorities. This will prevent TA activities from being preempted by short-term needs while at the same time recognizing the need of flexibility to accommodate these types of demands.
- As the IMF moves into a multiyear budget framework, the RAP could also evolve toward a multiyear RAP helping reallocate TA resources within the institution over a longer horizon. For this process to work, each area department would need to spell out their proposed TA country priorities (based on each country policy framework and with the authorities, as discussed earlier), including ranking of priorities across countries in the departments, the list being more notional for the outer years. This approach would allow a comparison between demands from the area departments in the outer years and the existing supply and composition of skills in the functional departments; it would help identify pressure points in the future that would call for a gradual reallocation of resources within and across functional departments. This multi-year budget process and the associated multi-year RAP would make the allocation of TA resources more responsive to demand as time passes.
- This strengthened role of area departments in preparing a multiyear plan of TA priorities (including prioritization across countries) will require both incentives and resources. Senior management in the area departments should ensure that staff considers TA as an integral part of the assistance strategy to countries, with due recognition in staff performance assessments.
2. The IMF should develop more systematic approaches to track progress on major TA activities and to identify reasons behind major shortfalls.
- At the outset of major TA activities, the IMF staff and the authorities should agree on how progress and success of the TA activity will be measured. Suitable indicators or markers of progress over time should be identified. This will avoid the emergence of divergent expectations and provide incentives for more timely reassessments if progress is seriously off track.
- To assist in this process, the IMF staff should unbundle much more clearly the different stages (of the results chain) through which TA may have a final impact, and then explicitly monitor these stages (examples of this approach are provided in Chapter 6). Specifically, it should differentiate between:
- — progress in improving the technical capacities of agencies receiving TA;
- — whether agencies are making effective use of that increased technical capacity, as well as reasons on why this is not happening; and
- — the impact on the ground in terms of relevant economic outcomes.
- Resident experts and headquarters staff in charge of backstopping activities should be candid in reporting obstacles to progress, including political interference or lack of support from the authorities that prevent agencies from making effective use of their improved technical capacity.
- A clearer articulation of benchmarks for measuring progress and the factors behind shortfalls is critical if past track records in implementing TA are to be a useful guide for future TA allocations. To have reallocations of TA across countries based on track records, these monitoring practices will have to be comparable and transparent across countries. The new Technical Assistance Information and Management System (TAIMS) could be the vehicle through which enhanced monitoring practices become unified and more transparent across the institution.
3. Greater involvement by the authorities and counterparts in the design of TA activities and arrangements for follow-up should be emphasized as a signal of ownership and commitment.
- We recommend that IMF staff request the authorities and specialized counterparts to fully participate in the preparation of the TORs and devote sufficient time to help design the activity. Willingness to do so should be one of the factors taken into account in decisions on TA resource allocation.
- For more complex multiyear TA activities, a letter of agreement between the authorities and the IMF could specify commitment and resources including (1) mutually agreed milestones of progress; (2) resource commitments by both the IMF and the authorities—the authorities’ commitment is important to assure sustainability beyond the life of the IMF TA activity; and (3) the critical policy steps that are required from the authorities to ensure necessary institutional changes, such as decrees or the preparation of legislation that complement the TA activity. Early on, staff should help authorities identify the critical legislation or judicial reforms necessary for agencies receiving TA to make the most effective use of it.
4. Stronger efforts should be made by TA experts to identify options and discuss alternatives with local officials prior to drafting TA recommendations.
- The receptivity of TA recommendations seems to be enhanced greatly when IMF experts engage counterparts early on in the design of the activity, explain its motivation, and try to assess the institutional subtleties of the specific environment. A major feedback received in the field concerned the need to allow enough time for informal discussions prior to issuing recommendations and the wider consideration of options. TA missions should allow enough time to incorporate these factors even if the result is somewhat longer missions and correspondingly fewer TA activities.
5. The program of ex post evaluations of TA should be widened and more systematic procedures for disseminating lessons put in place.
Recent initiatives have already moved in the direction of more systematic monitoring and assessment, and should be continued and strengthened.
- The Office of Technical Assistance Management (OTM), in collaboration with other departments, should continue to prepare and update a program of ex post evaluations of TA, including broad TA topics, subregions, or countries that have been heavy users of TA.
- While the TA-providing departments themselves should undertake self-assessments, there would be merit in continuing to undertake selected ex post evaluations by units not directly involved in providing TA, in order to enhance accountability and give a fresh perspective.
- The findings of these evaluations should be made more widely available within the IMF to maximize their potential benefits and avoid the loss of institutional memory. A stock-taking exercise of past and ongoing evaluations throughout the IMF should take place periodically and be made available to management and the Executive Board.
- OTM should also prepare regular reviews assessing shifts in the demand for TA across subject areas that may entail the need to reallocate resources across the functional departments that provide TA.
6. The prioritization filters should be discontinued or replaced by ones that would more effectively guide TA allocation. Either course of action involves strategic decisions on trade-offs that need to be taken explicitly.
- An effective priority-setting process needs two key components: (1) strategic direction by the Executive Board and management on those areas where the IMF will seek to maintain or develop “core competencies” in its TA activities; and (2) an internal resource allocation system that allocates effectively between competing demands, guided by these overall strategic objectives.
- Different approaches to balancing these two components are possible—essentially involving a decision on how decentralized the process should be. Whatever the choice made on this balance, within the IMF’s core competencies, priority setting should be derived from each country’s own strategy with a heavy involvement of area departments (Recommendation 1), according to modalities that select activities with stronger ownership and country commitment (Recommendations 3 and 4), and greater selectivity linked to a more systematic assessment of where TA is being used effectively (Recommendations 2 and 5). Typical dilemmas, such as the balance between short-term/policy advice TA and institution building, between “upstream” and “downstream” approaches to TA, and between TA associated with the IMF global initiatives and country-driven needs, would be resolved on a country-by-country basis.
- The advantage of a more decentralized approach is that TA can be closely aligned with specific country needs and circumstances—which is important given the large variation in country circumstances within the IMF membership. However, given that the TA resources of the IMF are small relative to global efforts, such approach has the risk of spreading the expertise of the IMF too thinly—even within the areas of mandate of the institution. Institution-building activities with long-term experts and strong external financing may lock resources and direct technical staff to backstopping activities—reducing the ability to respond to crisis. If Board and management give significant weight to these dangers, more specific guidelines may need to be issued circumscribing the coverage of the IMF TA program. For example, the IMF may like to think about how much it should get involved in institution building, in downstream approaches to TA, and how much importance should be given to the new institutional initiatives as prime drivers of TA. Providing guidance in these directions may help build a critical mass of expertise, but it may do so at the expense of adaptability to country circumstances. These are the key trade-offs that should be decided as part of the overall TA strategy.
Many of the previous recommendations will entail more staff-intensive approaches to providing TA—from prioritization, to delivery, to monitoring and evaluation. Moreover, if the choice made on strategic priorities is that the IMF should continue to be engaged in medium-term capacity-building efforts, this will call for an even closer collaboration with donors, including in the context of the PRS framework for low-income countries.
On the assumption that the present resource envelope for TA would not be significantly expanded, the above recommendations would probably result in fewer and more selective TA activities to be provided by the IMF. Doing fewer activities, but improving their relevance and probability of success, would, in our view, be an acceptable trade-off that can be made within the current resource envelope. Moreover, requiring more substantive and active involvement by the authorities in articulating priorities, formulating TA requests, drafting TOR, and providing adequate support throughout (and perhaps beyond) the life of the TA activity will also help self-regulate demand and result in fewer but more clearly owned TA requests.