IMF Staff Response to the Evaluation by the Independent Evaluation Office on Structural Conditionality in IMF-Supported Programs
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- April 2008
Executive Board Meeting December 12, 2007
Summary of Comments
Key findings of the report echo those of the 2005 Conditionality Review. In this regard, some of the conclusions reached at the time are worth restating (PIN No. 05/52).
– Directors agreed that: (i) “numbers of conditions are at best a crude metric”, (ii) “it is appropriate for conditionality to cover all critical measures”, and (iii) “even more important than the numbers ofz conditions are the quality and mutual consistency.”
– Directors welcomed “the streamlining of the breadth of coverage of conditionality, in line with the requirement that such conditions be deemed ‘critical’ to the goals of Fund-supported programs” and “that conditionality has shifted away from non-core areas.”
– They were pleased “that the sectoral coverage of the Fund’s conditionality now appears largely unaffected by the presence or absence of a Bank-supported program.”
Against this background, the differences between the IEO and staff on how to interpret the IEO findings stem from a different view about the purpose of conditionality. In staff’s view
– Streamlining was intended to bring about a change in the Fund’s mindset about program design—from comprehensiveness to parsimony, thus serving to focus programs in key areas.
– Criticality cannot be defined mechanistically and must be judged case-by-case. For example, it is not uncommon for a single substantive reform to involve numerous steps and, as a result, it is more likely to include a number of conditions.
While effectiveness of conditionality is difficult to measure, staff has doubts that the metric should relate to furthering reforms beyond the scope of the Fund-supported program except of course for ensuring that program goals can be sustained over the medium-term. A more useful metric is whether program goals—narrowly defined—were achieved and sustained.
Staff do not agree with the report’s implication that the World Bank should necessarily be involved in sectors that the Fund is not covering, nor the related recommendation that the Fund should play a “subsidiary role” in non-core areas of reform.
Staff agree that the link between program goals and strategies—and the explanation of program conditionality—could be better documented in staff reports.
There is also a need to undertake outreach that would expand the understanding of civil society and other stakeholders regarding the purpose and confines of Fund conditionality.
1. The IEO’s report contains a wealth of information and is a valuable complement to the staff’s own conditionality review (CR, 2005). Many of the quantitative findings in the IEO report echo those highlighted in the CR. Given the controversial nature of conditionality, the IEO can bring the credibility of its independence to bear on this subject, which is key for improving program design. As with any evaluation, it is important to identify what can be learned for the future, while acknowledging the limitations of the analysis and the need to provide an assessment that is informed by the governing Board-established conditionality framework.
Comments on the Evaluation
2. While staff concur with many of the IEO’s findings, differences on how to interpret the evidence stem from a different view about the purpose of conditionality. In staff’s view, as embodied in the Conditionality Guidelines (CG), conditionality is intended to assist members to solve their balance of payments problems by focusing on measures that are critical to achieving the goals of the member’s program and monitoring their implementation, interrupting Fund support when program goals are unlikely to be achieved. In effect, conditionality provides the member with predictability regarding the availability of Fund lending while safeguarding the revolving nature of the Fund’s financial resources. The Conditionality Guidelines also call for conditionality to be established where necessary for the implementation of specific provisions of the Articles of Agreement or the polices adopted under them. In contrast, the IEO takes the view that conditionality should foster further reforms in individual areas beyond the program’s goals and beyond the program period—this, in staff’s view, would seem to imply considerable intrusiveness on national policy making and be contrary to the spirit of the CG and the streamlining initiative.
3. Moreover, the streamlining initiative was not—and should not be—a “numbers game.” First, as implicitly recognized in the IEO report, the number of conditions says little about the intensity or ambitiousness of reforms, and the number of individual steps (and therefore conditions) that correspond to one substantive economic action is likely to differ across programs to take account of the nature of this reform, including the authorities’ preferences and technical capacity. Second, as the IEO report indicates, the content of structural measures differs widely, as does their likely difficulty in implementation, so the aggregate number of conditions is not a good measure of burden. Third, while there may have been some expectation that the average number of conditions would decline, the main concern was that a few programs had a very large number of conditions that overwhelmed implementation capacity and that programs covered too many different reform areas. As one example, the 5 largest programs prior to the streamlining initiative had 10 more conditions per program year than is the case for the 2001–05 period.1 Finally, since the aim is to tailor conditionality to the program objectives being supported, it must be recognized that it is not uncommon for a single substantive reform to involve numerous steps. In such cases this reform is likely to include a large number of structural conditions.
4. In sum, streamlining was intended to bring about a change in the Fund’s mindset about program design.2 Prior to the CG, there was a tendency to incorporate measures that were considered economically beneficial to the country—even if not necessary for achieving the goals of the program. In this regard, streamlining was intended to shift program design from comprehensiveness to parsimony in the application of conditionality. More precisely, Fund conditionality would be limited to measures considered critical to achieving and monitoring the program’s goals while ensuring that medium-term external viability is sustained. National authorities may, of course, implement other reforms but—regardless of how beneficial they may be—these should not be subject to Fund conditionality.3
5. Evidence that such a shift has occurred is provided in the IEO report(BP, ¶29-¶35). For instance, there is a downward shift in the relationship between the number of trade-related conditions and trade restrictiveness between programs approved in 1995–2000 and programs approved in 2001–04 (BP, Figure 2.6). As the IEO report notes “the fall in the number of trade-related conditions was not driven by a reduction in trade restrictiveness, but by the Fund’s own initiative to streamline its conditionality.” This is precisely the purpose of the streamlining initiative: to the extent that trade reforms are not critical to achieving the program goals, they should not be subject to Fund conditionality—regardless of their own merits and regardless of the degree of structural distortions involved. The same appears true in other areas, such as privatization-related conditionality (BP, Figure 2.7). In sum, whereas such conditions were often included in the Fund arrangements, the IEO report suggests that this practice has diminished.
6. In addition, criticality cannot be defined mechanistically and must be judged on a case-by-case basis, as highlighted in the country-specific comments (Appendix 1). The IEO report (¶45) contends that “arrangements continued to include conditions that do not appear to have been critical to the program objectives” while “some critical issues, particularly in non-core areas, may have been ignored.”
On the former, the report does not provide concrete evidence beyond the structural depth indicator. But the IEO’s concept of structural depth (SD) does not mesh with the purpose of conditionality. Structural benchmarks, for example, will often be “limited or no” SD measures under the IEO’s definition, as they are intended to signify critical mile posts in the process of program implementation (in the words of the CG, “to serve as clear markers in the assessment of progress in the implementation of critical structural reforms in the context of a program review” (see CG ¶11(d)(ii)). Also, the structural depth indicator does not correctly appreciate how programs are tailored to country-specific circumstances—in this regard, flexibility on how measures are defined is key to program success (e.g., the use of small steps has been found to work best in some countries).4 Thus, the Board-established framework governing program design expressly contemplates that conditions may appropriately have what the report considers to be of “limited or no” SD.5
Moreover, the shift towards core areas suggests that measures subject to conditionality are indeed more likely to be critical than was previously the case. This is reinforced by the IEO’s analysis of permanent program interruptions (which have declined, especially for GRA-supported programs) and the staff’s analysis of waivers, which shows that the “lapsed rate”6 (i.e., the proportion of all SPC that were waived and the program allowed to continue without the measure or an equivalent measure being implemented; see CR) has declined—as would be expected if measures subject to conditionality are indeed critical.7
On critical issues being ignored, the report does not distinguish between measures that may be important (or even critical) for the authorities’ broader reform agenda and those that are critical for achieving the goals of a program. As noted above, only the latter should be subject to Fund conditionality. Accordingly, a Fund arrangement would typically be viewed as supporting only a subset of the authorities’ broader reform agenda.
7. Since one of the purposes of conditionality is to interrupt IMF support when goals are unlikely to be achieved, the IEO’s measure of effectiveness appears to be too narrow.8 While reforms should not be reversed, the findings in the report suggest this occurs only in a small minority of cases (8 percent for core areas and 13 percent for non-core)—and in some 50–60 percent of cases there were further reforms. Staff does not consider that conditionality should be judged ineffective if the country does not undertake further reforms in the area since:
Under the CG, conditionality is supposed to underpin measures critical to the program’s success—not to initiate reforms beyond the program.9
Program measures may have addressed the most important structural reforms in that sector (e.g., if prices are liberalized, there should be no need to do so again).
Even though a measure may have “low structural depth” (as defined by the IEO report) this does not imply that it is not critical to program success. The example provided in the IEO report (¶16, footnote 5) is a case in point: passage of the annual budget is very likely to be critical in a program that aims at economic stability and external adjustment.
8. While effectiveness is difficult to measure, a more useful metric is whether conditionality helped achieve a program’s macroeconomic goals. Staff’s own analysis suggests that structural measures have helped underpin these goals. For example, fiscal structural measures (revenue-related measures) are statistically significantly related to fiscal adjustment, and economic efficiency-enhancing measures are statistically significantly related to growth performance.10 It would also have been useful to highlight the link of timely, targeted Fund technical assistance in the design and implementation of conditionality, given the report’s finding of enhanced compliance with conditionality in areas where substantial TA resources are devoted.
9. Staff agree that the link between program goals and strategies, and the explanation of program conditionality, could be described better in Staff Reports supporting requests for the use of Fund resources. The IEO report notes that “the streamlining initiative did not lead to better explanations of why conditions, either core or non-core, were critical, nor of how these conditions would contribute to program objectives” (¶43) and “sometimes the rush may have only affected the quality of the documentation, but in other instances it may have also hindered the program design and its effectiveness” (¶49). While the CR was somewhat more sanguine, it too noted that “some room for improvement exists at least in the presentation, and possibly also in the formulation, of strategies.”
10. The IEO’s different view about the purpose of conditionality carries over to its views regarding coverage of conditionality between Bretton Woods institutions. In particular, there is no reason why the World Bank should necessarily include (or increase) its conditionality in reform areas no longer covered by Fund conditionality (¶44). The CG require that the Fund include conditionality on measures critical to achieving the goals of the Fund-supported program regardless of whether these lie within or outside the Fund’s core areas. In non-core areas, the CG envisage the Fund drawing on the expertise of other institutions (e.g., the Bank) in the design of its conditionality and its monitoring—hence the “lead agency” concept, whereby one of the two institutions would take the lead in providing substantive policy advice. However, the IEO report does not distinguish between the provision of policy advice—which should be responsibility of the institution with greatest expertise in that area—with conditionality, where each institution is responsible for its own lending decisions and must therefore include conditionality if the measure is critical to the program it is supporting.11,12
Comments on IEO Recommendations
11. While many of the IEO’s findings are consistent with strong implementation of the CG, some do point to areas in need of improvement. Nonetheless, some of the IEO report recommendations do not adequately take into account of the purpose of conditionality and the difference between program design and the provision of policy advice more generally.
12. In this context, staff agree that program documents could do a better job; in particular laying out program goals, how proposed strategies would contribute to achieving these goals, and the conditionality that may be used to monitor that the proposed strategies are in place. Further clarifications in guidance notes on the application of the CG might be useful, including a reaffirmation of the need to use conditionality sparingly and only when critical to achieving program objectives (and program monitoring), or where necessary for the implementation of specific provisions of the Articles of Agreement or policies defined under them.
13. The proposed notional cap has risks and might not accomplish greater clarity. Such cap could prove rigid and constrain program design, as well as open the Fund to a “one-size-fits-all” criticism. Moreover, the IEO report provides no evidence that a larger number of structural conditions was associated with weaker implementation; if anything, the evidence suggests that it had a positive effect in reducing program delays (BP, Figure 2.11) and did not result in lower implementation rates (¶21). Notwithstanding the above, a notional cap or guideline—if applied flexibly and with due regard to special circumstances—could on occasion help increase program focus. As to new non-lending instruments, staff’s view is that policy support instruments (PSI) provide the needed monitoring tool where donor-led programs have a central role and believes that bilateral surveillance remains the best mechanism for the provision of policy advice outside a Fund-supported program.
14. Staff do not think that eliminating structural benchmarks would be appropriate; nor, for that matter, eliminating measures with low structural depth. Benchmarks play many roles (from mile posts in broader reform areas, to the direction of reform in critical areas where it is not possible ex ante to specify the required measures with the precision) required for a performance criteria. In particular, they help define the contours of program reviews and thereby provide the link between progress with the reform agenda and assurances about the availability of Fund financing. Their elimination would weaken the assurances aspect of conditionality and revive the criticism of moving reform goalposts. That said, it could be useful to reiterate in guidance notes that benchmarks should be limited to areas of reform that are critical to achieving program goals. As noted earlier, what might be viewed as low structural depth is, on occasion, key step toward the success of the Fund-supported program.
15. Staff do not agree with the report’s implication that the World Bank should be involved in sectors that the Fund is not covering, nor its recommendation that the Fund should play a subsidiary role in non-core areas of reform. The CG require the IMF to have conditionality on those measures critical to the program it is supporting; and, not surprisingly, the World Bank has similar fiduciary responsibilities. Again, the report seems to confuse provision of policy advice with the need for conditionality. Of course staff is expected to draw on the expertise of other institutions (notably, the World Bank or other MDBs) in non-core areas for policy advice and the design of conditionality. Moreover, initiatives (e.g., JMAP) underway lay out processes for initiating and coordinating work where the Bank has clear lead responsibility and greater expertise and for better coordination in joint areas. Still, the Fund’s own fiduciary responsibilities—confirmed by the Board in many occasions and clearly specified in the CG—would require setting conditionality in all areas critical for program success.
16. The report calls for “explicit Board guidance” for cases where reforms in non-core areas are critical to program goals but the Bank is unable to deliver(¶57). However, this seems a non sequitur since the report separately recommends that all conditions in Fund arrangements “should pertain to the core areas of IMF responsibility” (¶56). The latter implies that conditions should never be set in non-core areas. Thus, by this logic, in cases requiring critical reforms in a non-core area where the Bank is unavailable the only option would be to refuse IMF financing.
17. Staff agree that a framework to monitor whether program documentation lays out goals, strategies, and justification for proposed conditionality would be helpful. This could be done at a higher frequency than the current 5-year cycle of conditionality reviews and could prove useful in informing program design. Staff does not consider that conditionality is primarily intended to foster further reforms than what is needed to achieve the program goals; thus, a monitoring framework linking conditions to specific outcomes would not be appropriate. Instead, conditionality reviews should focus on whether programs achieved their macroeconomic goals.
18. As to the availability of MONA, much of it is already made available to researchers and civil society upon request. For instance, for previous programs and original program requests for current arrangements, the dataset is made available when the information is in the public domain (i.e., the staff report is published, which is the presumption for requests for use of Fund resources). To provide information on real-time basis would be very resource-intensive, but what is currently provided to researchers upon request could be routinely posted on the web.
19. Staff agree that the Fund needs to improve its outreach activities. In fact, the report provides evidence that the vast majority of structural conditions in Fund-supported programs should not be very burdensome to implement, “yet the large number of conditions was widely perceived as intruding in the policymaking process and deterring from society’s sense of ownership.” The resistance and stigma are partly linked to the public’s lack of knowledge about the sources and types of conditions. Likewise, the literature review (BP, Chapter I) suggests a lack of understanding about the purpose of IMF conditionality and how it is intended to work.
It is not clear if the programs covered by the “desk studies” (the 43 programs approved in 1999–2003) are all under the streamlining initiative. It would be useful to include a list of programs covered under each of the analytical methods used, including the programs reviewed for 2004–05.
The claim that AFR did not request follow-up TA from FAD (Table 4.13.3) is inaccurate, as FAD delivered intensive TA since 2004 at AFR’s request including a PFM TA mission (July 2005); tax and custom administration TA mission (March 2006); peripatetic visits in tax administration (January, May and September 2007) and in customs administration (January, May and September 2007); and tax policy TA missions (February, July and September 2007).
In staff’s view the conclusions drawn from the case studies suffer from an overly mechanistic approach to structural conditionality and could have been better balanced if they had considered the broader picture of the environment in which country programs were designed. The comments below highlight the importance of factors other than program design to the ultimate “success” of a program; for instance, in Madagascar, where civil strife impeded progress under the program; and Ghana, where ownership of the new program by the authorities was a key success factor.
Page 54, Table 4.2: Many of the reforms mentioned have laid the ground for subsequent substantial improvements (in particular in PFM), even though they are judged to have only limited short-term effects. Moreover, some of the short-term achievements of the program are not fully reflected in the analysis in Table 4.2. For example, the absence of a crisis in the financial system should be viewed as a success taking into account the circumstances at the time. Specifically, the issue of saisie-attribution was a problem threatening the viability of banks and the program took actions to stop this practice.
Page 87, Table 4.9.4: (i) The report suggests that earlier emphasis on tax administration would have helped, however efforts before the 2000 PRGF had focused on tax policy (i.e., the VAT). The assessment on customs is overly critical—through conditionality and critical TA from the World Bank, the 2000 PRGF managed to kick start the previously stalled reform momentum and deliver critical measures (such as the securitization of PAGODE, the customs IT system). (ii) The assessment should note that a lot of donors are involved in PEM reforms with mixed results. For example, significant delays materialized in the reform of the budget classification, because recommendations by an EU consultant ran contrary to best practices and FAD recommendations. (iii) Considerable progress has been achieved on public procurement since the end of the program, although further improvements are still required. (iv) On governance, the audit office has become operational but has been slow in auditing government accounts. (v) On HIPC, the low use of HIPC resources stemmed from the establishment of a donor committee, an idea that was imposed on the program by donors.
Page 17, ¶29: The report claims that the PRGF of 1999–02 achieved “little or no progress”. While not contesting this conclusion, staff would point out that lack of progress was mainly due to weak institutional capacity/political commitment in the context of election-related fiscal profligacy. A striking phenomenon in Ghana’s case is that with a change in administration, performance under the subsequent PRGF program ending in 2006 was successful. While Fund SC was unchanged under the second PRGF, what did change was the authorities’ commitment to reforms. This highlights that in the end, Fund programs will be successful only if the authorities have full ownership.
Page 17, ¶29: The judgment that the Kenyan program achieved little or no progress should be placed in context. The reported lack of progress is consistent with the fact that no reviews were completed under the PRGF arrangement that was approved in 2000. For example, in Table 4.11.3 (page 97), the observation that there was “no substantial progress in PEM despite the authorities’ high compliance rate” is misleading because the measures implemented were only initial steps in a long reform process. Had the program remained on track, one would have expected to see further milestones met. In a similar vein, in Table 4.11.4, the prospects for successful financial sector reform would have been improved had the program stayed on track with reviews completed.
Pages 97–98, Tables 4.11.3 and 4.11.4: There is an unresolved tension between the discussion of financial sector reforms (output assessment) and of anti-corruption legislation and institutions (usefulness). In the former, the report argues that lack of progress stemmed from requiring only that legislation be submitted to rather than approved by parliament. Conversely, in the latter area, the report argues that conditionality requiring submission of legislation “provides an example of how structural conditionality can actually stigmatize desirable measures.”
Tables 4.12.1, 4.12.3, and 4.12.4. It should be noted that the program was launched when Lao P.D.R. was coming out of a prolonged period of weak policies and macroeconomic instability. In this context, the authorities and the staff agreed that the initial stages of the program had to be focused primarily on securing macro stability. Conditionality aimed to complement the stabilization efforts and build a foundation for a gradual transition to a market-based system, while allowing the authorities time to build consensus on politically more difficult reforms. In this regard, the assessment on the implementation of structural reform seems unduly negative and excludes the authorities’ recent reform efforts.
Page 20, ¶34. Staff strongly disagree that “collaboration [with the World Bank] was weaker in … Lao P.D.R,” which staff believe is not a general consensus among the current and past Fund and Bank teams on Lao P.D.R.
Page 17,¶29: The qualification of Madagascar’s program as having achieved little or no progress seems overly severe in view of the special circumstances under which the program was implemented (civil strife in 2002 causing output contraction by 13 percent).
Page 103, Table 14.13.1: In staff’s view the introduction of a continuous forex market was critical, even if bankers were against it because of the costs involved.
Footnote 11: The report states that: “Fund arrangements provided a monitoring framework for the EU accession process, explicitly (e.g., Romania) or implicitly (e.g., Croatia).” There was, however, no explicit framework in Romania’s accession process.
As noted in CR (2005), Executive Directors stressed during the 2000–02 conditionality review that the number of conditions is only one, imperfect, indicator of the extent of progress with streamlining.
See Summing up of the 2001 Executive Board discussion of Conditionality in Fund-supported Programs, which states “While these principles would need to be interpreted on case-by-case basis, they would shift the presumption of coverage from one of comprehensiveness to one of parsimony.”
Moreover, both the Interim Guidance Note and the Conditionality Guidelines emphasize that in such cases there should be clear delineation between the authorities’ broader reform agenda and measures that are part of the Fund-supported program and subject to Fund conditionality.
For example, the performance criterion on the issuance of a tender for the privatization of SONACOS in Senegal was a key step toward divestiture and, while the process was not without delays, a privatization with key macro implications was ultimately achieved. In other countries a similar measure might be set on actual divestiture. While in both cases the measure is in the end met, only the latter would be assigned a high SD rating in the IEO’s view.
Inasmuch as high SD measures are likely to be more onerous to implement (the IEO finds that their implementation rate has indeed be lower), this suggests that program design has sought parsimony. This is consistent with the CG, which require that conditions be set so that program goals will be achieved while tailoring these to the member’s own political considerations and technical capacity.
See CR (2005) ¶50, Figure 11.
The IEO methodology of examining ex post correlations—no significant correlation is found between compliance and effectiveness and between compliance and depth—might not adequately control for country-specific circumstances. For example, stronger conditionality may have been imposed on harder cases, in which case the lack of correlation could be construed as the Fund, ex ante, imposing the right balance of conditionality at the margin.
In addition to practical and budgetary considerations, this is one of the reasons why the MONA database does not track conditionality once a program goes off-track and the arrangement expires (¶8).
One reason that the report finds that measures in core areas have been more effective is that the subsequent Article IV report may not have tracked further progress in non-core areas.
See The Design of IMF-supported Programs (OP 241, pp. 146–147).
As CR (2005, ¶68) notes, “it is not obvious that gaps in Bank-Fund conditionality need to be viewed with concern. Much more worrisome would be gaps in analytical work, policy advice or technical assistance”—Bank-Fund collaboration on these areas was considered in the Malan Report and is taken up in this report’s implementation plan.
An example could be the privatization of a telecom company. Such privatization (within fixed timetables and short horizons) may be critical to the Fund-supported program because of (i) the budget needs the privatization receipts or (ii) the enterprise is incurring large budgetary losses. In contrast, for a Bank operation—which may be concerned with an overall restructuring the telecommunications sector—the privatization by a fixed timetable is less likely to be critical. Notwithstanding this distinction, Fund staff would want to draw on the Bank’s expertise—for in-stance, on whether the sale makes sense from the overall strategy for the sector and what would be a feasible timetable.