versión On-line ISSN 0718-2724
Journal of Technology Management & Innovation v.5 n.4 Santiago dic. 2010
Journal of Technology Management & Innovation
J. Technol. Manag. Innov. 2010, Volume 5, Issue 4 p.38-56
An Environmental Impact Assessment System for Agricultural Research and Development II: Institutional Learning Experience at Embrapa
Geraldo Stachetti Rodrigues1, Cláudio César de Almeida Buschinelli2, Antônio Flavio Dias Avila3
1 Embrapa Environment, Environmental Management Laboratory. Rodovia SP340, km 127.5, Jaguariuna (SP), CEP 13820-000, Brazil. Telephone +55 (019) 331 12675, Fax +55 (19) 33 I 12640. E-mail: (firstname.lastname@example.org (Corresponding author)
2 Embrapa Environment, Environmental Management Laboratory. Rodovia SP340, km 127.5, Jaguariuna (SP), CEP 13820-000, Brazil. Telephone +55 (019) 331 12636, Fax +55 (19) 33 I 12640. E-mail: email@example.com
3 Secretariat of Management and Strategy, Embrapa Headquarters. Parque Estacao Biologica, Via W3 Norte, Brasilia (DF), CEP 70770-901 - Brazil. Telephone +55 (61) 344843 15, Fax +55 (61) 34484884. E-mail: Flavio.Avila@embrapa.br
"An environmental impact assessment system for agricultural R&D" has been implemented by the Brazilian Agricultural Research Agency (Embrapa) aiming at fulfilling the institutional mission statement of 'introducing sustainable development objectives in all steps of agricultural research'. The impact assessment platform emphasizes close interaction between R&D teams and technology-adopting producers, under actual field contexts, in order to improve both the technology development and the demand probing processes. The proposed integrated environmental indicators system (Ambitec-Agro) has been routinely applied in technology appraisals by all of Embrapa's Research Units, as one of the criteria of its institutional evaluation system, and toward the formulation of the Social Balance Reports, annually published since 1997. The present paper describes the social and environmental dimensions of this integrated impact assessment platform, its roles in the institutional learning process for technology research management, and an impact analysis of proposed agricultural innovations.
Keywords: multi-dimensional evaluation; environmental impact assessment; social impact assessment; Ambitec-Agro; institutional learning
As continuation to a proposed "environmental impact assessment system for agricultural R&D" (Rodrigues et al., 2003a), this paper aims to describe the general procedures for technology appraisal developed at the Brazilian Agricultural Research Agency (Embrapa). Based on a review of the current context of 'agricultural R&D for sustainable development' (EIARD, 2003), the text (i) presents the premises upon which the institutional impact assessment platform has been constructed; (ii) describes the background and methodological bases, following with a (iii) technology innovation typology for allowing the (iv) analysis of impact assessment results obtained to date, a (v) conclusion regarding the institutional learning process for technology research management, and a set of (vi) recommendations for strengthening the impact assessment network.
All these subjects stem from a current institutional orientation in the agricultural R&D sector toward the organization of innovation systems that may, on the one hand, target technology advancements for productive intensification, and on the other hand include a wider agenda for poverty alleviation, environmental management, and social inclusion (Horton and Mackay, 2003). In order to instruct these objectives, agricultural research organizations seek strategies to plan their resources allocation, the management of their capabilities, and the transfer of their results. Such strategies require viewing innovations in systemic terms, with flows of knowledge between research teams and external parties - technology adopting farmers foremost - as key for organizational performance (Hall et al., 2000; 2003).
The proposed wider agenda for agricultural R&D implies viewing innovations as contributions to sustainable livelihoods, recognizing that prosperity (the goal of technology adoption) depends on the satisfaction of diverse and multiple interests of farmers, their families and communities (Adato and Meinzen-Dick, 2002). This is not a novel standpoint, having been brought up decades ago with demands for change in R&D evaluation methods, in order to account for institutional development, sustainability, and environmental impact (Conroy and Litvinoff, 1988).
The focus on sustainable development, environmental management and social inclusion poses a challenge to technology innovation decision making, be this related to what technology to develop, for application where, and for adoption by whom, entailing questions on how to make the innovations available and involve all people concerned in these decisions (Haque, 1991). Actor-oriented, integrative, and participatory approaches are increasingly employed to address the multiple sustainability objectives of different groups of interest. Also, farmer involvement in agricultural technology R&D contributes to identification of appropriate applications, adoption mechanisms, and transfer to larger numbers of users, providing valuable information feedback to research teams (van de Fliert and Braun, 2002).
In order to pursue sustainable livelihoods as the goal for technology innovation, and to exercise integrative, participatory approaches for research, adoption and transfer, agricultural R&D organizations must promote a veritable institutional learning transformation (Watts et al., 2003). This learning and change process has been associated with evaluation procedures (Horton and Mackay, 2003), which purpose it is to help managers and scientists to better understand how their research activities generate effectively impacting results, and to draw lessons about how to improve future research programs (Raina, 2003).
Institutions undertake evaluations in order to organize their (i) needs assessment, (ii) priority setting, (iii) research proposals review, (iv) ongoing research monitoring, (v) completed research appraisal, (vi) research output verification, (vii) impact assessment, (viii) programs and management review, and (ix) staff performance assessment (Horton, 1998). To meet these intents, evaluations must be organized in credible procedures, with little room for speculative convergence of cases (Nelson and Meredia, 2007), or mere plausible links between research and impacts (EIARD, 2003).
Impact assessment methodologies comprise the toolkit for consolidation of innovation systems, to orient innovation demand probing, research priorities setting, research formulation and agricultural practices recommendations (Mackay and Horton, 2003). Given such broad sense benefits and wide expectations for management, institutions must gauge a compromise between how much impact assessment they should do, and how sophisticated the assessments should be (Collison and Tollens, 1994).
Attentive to these institutional tendencies and in concerted action with PROCISUR4 (Puignau, 1998) and other institutional partners, Embrapa has been committed to the systematic assessment of impacts on agricultural technological innovations, as attested by scientific meetings held (Puignau, 1998; Embrapa, 2008) and cooperative research developed on the subject (Rodrigues et al., 1998; Bin et al., 2003; Bonacelli et al., 2003). The accorded approach has been to foster Environmental (and Social) Impact Assessments (EIA) with the declared objective of "promoting solutions for the sustainable development of rural spaces by generating, adapting and transferring knowledge and technology for the benefit of society" (Embrapa, 2004).
Impact assessments were implemented at Embrapa in the early eighties, after an initial organization period (1973-79) when the institution obtained generous public funding, supported by international loans aimed at creating its infrastructure and training its human capital (Neinke et al., 2001). Following this initial phase, a series of financial crises affected most public institutions of the country in the mid 1980s and 1990s, when Embrapa underwent severe budget cuts (Avila and Souza, 2002). The new scenario motivated the initiative of the Administration Board to encourage impact assessment studies, as to improve the visibility of agricultural technologies' role in rural development and better justify public investments.
The mono-disciplinary (economic) view was then complemented by an impact evaluation platform covering economic, social and environment aspects (Avila et al., 2008), systematized as an integrated impact assessment system (Bosshard, 2000). Indicators were selected according to sustainability objectives (Lewandowski et al., 1999), for estimation by scores in a multi-criteria procedure formulated as to avoid the ambiguous step of monetization of environmental goods and social effects (van Pelt, 1994). The proposed impact indicators (Esterhuizen and Liebenberg, 2001) were integrated in scaling checklists, in which original quantitative data obtained in field surveys (measures of area, quantities, concentrations, proportions, numbers) are normalized into a non-dimensional unit for allowing cross-scale and multi-variable aggregation (Bisset, 1987; Rodrigues, 1998).
Aggregation is a much debated step in impact assessment practice. On the one hand it consents integration and accounting for heterogeneous sets of characteristics and effects, while on the other hand it may cause a loss of meaning for some of the criteria in the dataset, if expression of integrated results is not appropriately organized (Andreoli and Tellarini, 2000). The necessity of normalization and aggregation is easier to justify in multidimensional assessment systems that include both environmental (impact and performance) and social (health, food security, welfare) considerations (Pinho and Pires, 1991). Upstream, this means accounting for resources required for technology development (e.g., raw materials, habitats affected). Downstream, it means consideration of the residuals and effects of technology application on environmental quality and social welfare (Porter, 1995).
Compounding on these methodological premises, a system for environmental impact assessment of agricultural technology innovations has been formulated, based on objective indicators, constructed on a flexible platform, acceptable for application on the large diversity of rural activities, environmental situations and their combinations typically included in Embrapa's agricultural R&D program.
A System for Environmental Impact Assessment of Agricultural Technology Innovations - Ambitec-Agro 5
An integrated impact assessment procedure has been developed for motivating farmers to promote agricultural technology conversion and adoption, facilitating technology development project appraisal and socio-environmental impact assessment at the institutional R&D level (Rodrigues and Rodrigues, 2007). The so-called Ambitec-Agro system (Rodrigues et al., 2002; 2003a) has been installed in the impact assessment platform employed yearly by all Embrapa Research Centers to (i) evaluate their technological contributions (Avila et al., 2005), (ii) assist in research project appraisals (Rodrigues et al., 2000) and (iii) assess social and environmental impacts of technology innovations made available through R&D (e.g., Lanna et al., 2004; Rodigheri et al., 2006; Tosto et al., 2006a; Rodrigues et al., 2006; Carvalho et al., 2007; Duarte et al., 2007; Mori et al., 2007; Almeida et al., 2007; Sa et al., 2008; Silva et al., 2008; Almeida et al., 2009).
The Ambitec-Agro structure relies on a series of Principles of technology and rural activity performance, composed by social and environmental compliance Criteria, integrated by series of sustainability Indicators selected from prior experience and field trials (Irias et al., 2004a; Magalhaes et al., 2006). The indicators are scored in field surveys / interviews with farmers / administrators, to obtain change coefficients according to technology or rural activity effects observed in the studied contexts. The change coefficients are weighted by factors related to each indicator's relevance toward effecting socio-environmental impacts and its scale of occurrence (Rodrigues et al., 2003b; Monteiro and Rodrigues, 2006). Finally, impact indices are calculated for each indicator and criterion, and aggregated as a technology innovation socio-environmental impact index.
The Ambitec-Agro system comprises four modules, focused on the productive sectors of Agriculture, Animal husbandry, and Agro-industry environmental impact assessment (Irias et al., 2004b) and a specific module for social impact assessment (Rodrigues et al., 2005a), encompassing 24 criteria and 125 indicators (Table I).
The impact assessment of a given technology innovation with the Ambitec-Agro system is carried-out in three steps:
1. definition of technology innovation use magnitude, geographical area delimitation and users;
2. field survey / interview at the rural establishment scale, applied with innovation-adopting farmers and data filling out in the scaling checklists; and
3. analysis, interpretation, and reporting of impact indices (in formatted templates), with proposition of alternative management practices and technology adaptation, focused on minimizing negative impacts and promoting positive ones.
A stimulated practice is to return these technology assessment reports to farmers and rural establishment administrators, favoring technology use improvement and contributing to local sustainable development.(See Table I at the end of the article)
Ambitec-Agro scaling checklists
The Ambitec-Agro system consists of integrated indicator scaling checklists, in which change coefficients checked in field surveys / interviews are related to quantitative measures of area, quantities, proportions, etc., then standardized as varying from -3 (meaning a major decrease in the indicator) to +3 (meaning a major increase in the indicator), reflecting the effects of the studied technology or rural activity, contingent to each particular assessment context.
The indicators are then weighted according to their defined relevance to conform the assessment criterion and their scale of occurrence. The weighing factors related to the relevance of each indicator are defined on an ad hoc basis according to user criteria in order to better reflect specific situations and add up to ±1 (according to the indicator impact direction, either positive or negative). Hence, the relevance weighting factors consist of a normalization step to equalize the different number of indicators that make up each assessment criterion.
A second series of weighting factors are related to the geographic scale in which the indicator change coefficient occurs in any studied case, as follows:
i. near environment - when the innovation / rural activity effect on the indicator is restricted to the crop area, productive field or facility where the studied innovation is being adopted / activity is being conducted;
ii. proximate environment - when the innovation / rural activity effect on the indicator extends beyond the productive unit, but within the limits of the rural establishment;
iii. surrounding environment - when the innovation / rural activity affects the indicator in an area or environment beyond the limits of the rural establishment.
Once the change coefficients resulting from the field survey / interview are introduced in the scaling checklists, the impact index for each indicator is calculated, according to the given relevance values and scale of occurrence, and then combined to express the impact index for the criterion (resulting range +15, Figure I).
Figure I. Typical scaling checklist
of the Ambitec-Agro system.
The given example for the water quality criterion represents a field observation of moderate reduction in BOD at the proximate environment scale, a major decrease in turbidity also at the proximate environment, a major reduction in the presence of floating materials / oil /
Table of change coefficients for variable
Once all indicator change coefficients are inserted into the scaling checklists, a Technological Innovation Impact Index is calculated for the specific conditions studied, by averaging all the normalized impact indices for the criteria considered. Similarly to the weighting factors included in each indicator scaling checklist, this normalization step allows a new adjustment of relevance values, this time for the different criteria considered in the impact assessment system. With this definition of relevance weights for indicators and criteria (Figure 2), assessments may be better adapted to specific evaluation contexts, by emphasizing relevant local aspects or evaluation objectives, or even by excluding certain aspects that may not appropriately represent meaningful consideration for particular cases (no-effect). (See Figure 2 at the end of the article)
Figure 2. Final environmental impact assessment
display of the Ambitec-Agro system.
Ambitec-Agro insertion in the institutional impact assessment platform
The aim of Ambitec-Agro is to provide a practical, expeditious, low cost, and reproducible socioscum in the surrounding environment; and no-effect in siltation (note that the weighting factor for siltation is zeroed, with corresponding weighting factor being transferred, in the given example, to BOD). Twenty four such scaling checklists, with a total of 125 indicators, comprise the Ambitec-Agro system.
environmental impact assessment procedure for the wide range of agricultural technologies and rural activities concerned in Embrapa's research program. These particular technology innovations, made available through the numerous R&D projects in the decentralized Research Centers, comprise the basic units of Embrapa's impact assessment platform. The institutional coordinating body (the Secretariat for Management and Strategy, SGE) recommends for each of the 37 Research Centers to proceed evaluations on three selected innovations identified as in initial adoption phase, by three consecutive years.
Evaluations are carried out by appointed teams in the Research Centers, normally comprised by dedicated socio-environmental researchers and the members of the group responsible for the research project in which the selected technology innovation development has taken place. A budgetary allowance is made available through SGE for this task, estimated as a minimal amount sufficient to fund interviews / field surveys in ten technology-adopting rural establishments.
Results of these field studies are entered in formatted datasheets and report templates, available through intranet access. These templates facilitate calculations of economic impacts and internal rates of return, as well as discussion regarding social and environmental impact indices (Avila et al., 2008). Lastly, the final integrated impact assessment report composed for each technology innovation is verified for consistency and formulation quality at SGE, which issues a statement for each Research Center regarding completion of the evaluation process.
The technology innovation impact assessment reports issued by the Centers are a pillar of Embrapa's Evaluation and Award System (SAPRE), including the System for Units Evaluation (SAU), with implications in resources distribution, pecuniary awards, research priority setting, and staff performance evaluations. More influential still, this impact assessment platform is one of the main sources of information for composing the yearly Social Balance Reports6, a principal institutional communication mechanism distributed both publicly as accountability for governmental investments and returns, and specifically addressed to the Ministry of Agriculture, Livestock and Food Supply (MAPA), the National Congress, and other administrative bodies.
Integrative levels and potential impacts of agricultural technology innovations
The wide range of agricultural innovations brought out by Embrapa's R&D program span a large spectrum of scientific disciplines, and the most varied applications both regarding productive sectors and socio-environmental utilization contexts. These so-called 'integrative levels of technology development' imply different degrees of complexity, both in terms of research disciplinarity and in terms of the extension of changes brought about with technology adoption. For instance, input use technologies are mostly mono-disciplinary studies, usually aimed at improving efficiency or adapting alternative compounds or materials in order to obtain incremental gains on specific agricultural activities.
Genetic resources, plant and livestock breeding technologies, on the other hand, are typically carried out within the domains of genetics and genetic resources (collections, in situ and ex-situ conservation). Developments in these studies have the potential to cause important breakthroughs both in efficiency, with more productive or pest-resistant varieties; and area expansion, with drought or cold tolerant varieties, for example. Hence, secondary impacts (both beneficial and detrimental) of these technologies can be due to agricultural (and animal husbandry) intensification or due to occupation of new, sometimes marginal areas, but are still related with specific crops and livestock.
Technologies addressing agro-industrial processes and post-harvest practices may be simpler mono-disciplinary, or extremely complex multidisciplinary, inter-institutional endeavors; aimed either at quite specific or very broad applications, to simpler incremental gains in input and energy use efficiencies or quite innovative products and processes. Typically these technologies are related with semi-industrial, work-intensive processing phases of production, so affecting social indicators more importantly.
Innovations classified as improved agricultural practices are in general efficiency related, normally aimed at adapting or integrating proven practices to new specific uses. In this sense these innovations may involve either a few or many branches of knowledge, basic or advanced engineering solutions, and simple or complex integration of resources and production means. In all cases, however, they are aimed at obtaining incremental gains for specific applications, products or production practices.
Finally, integrated management technologies are characteristically more elaborate, invariably ensuing from multi-disciplinary research, applied to entire sets of technological and natural resources comprised by entire production systems, in agro-ecological contexts that include consideration of the interests of people involved.
These 'integrative levels of technology development' reflect on the scale and magnitude of technology implications and impacts, both environmental and social. In order to facilitate the organization of this complexity for impact assessment, the broad spectrum of innovations brought out by Embrapa's research program may be classified, e.g., according to the described 'integrative levels', as follows (the innovations mentioned below are actual examples appearing within Embrapa's portfolio, in one of the cited Social Balance Reports):
Input use technologies, for instance 'use of gypsum in soy (also in coffee, corn) in Cerrado soils', 'Dosage of the growth regulator mepiquat chloride in cotton', 'Bioinsecticide against A. aegypti larvae', 'Use of sewage sludge in agriculture', and the like;
Genetic resources, plant and livestock breeding technologies, for instance 'Black bean cv. BRS Valente', 'Mombaca grass (also Marandu, Tanzania)', 'Carrot cv. Brasilia', 'Banana varieties resistant to Black Sigatoka', 'Embrapa MS58 - Hybrid swine male', 'Laying chicken Embrapa 051 Colonial', and the like;
Agroindustry / Post harvest technologies, for instance 'Standard cuts for ovine carcasses', 'System for small-scale vegetable drying', 'Multiple modules for processing cashew nuts', 'Fabrication of yogurt from soy extract', and the like;
Improved agricultural practices, for instance 'Integrated control of pests in apple', 'Use of gliricidia as live posts for fencing', 'Enriching the herb stratum in Caatinga for ovine raising', 'Nitrogen biological fixation in soybean', 'Integrated management of the wood-wasp (Sirex noctilio) in pine stands', and the like;
Integrated management technologies, for instance 'Integrated cotton production system for the Cerrados', 'Management of native acai stands in floodplains and estuaries of the Amazon', 'Sisplan - computational system for forest management', 'No-till management for soybean', 'Crop rotation system with winter pastures', and the like.
Based on these integrative levels for agricultural technology innovations, an analysis of the socio-environmental impact assessments carried out to date at Embrapa has been performed, allowing a better understanding of the contributions of research to agricultural development, as well as an analysis of the institutional learning process regarding the impacts of innovations made available by research.
The usage of Ambitec-Agro in the impact assessment of technology innovations throughout Embrapa's Research Centers, favored by the institutional platform of the System for Units Evaluation (SAU), is generating a valuable database of technology appraisals, not only as internal documents and project mid-term reports, but also as specialized and independent publications (e.g., Ferreira et al., 2005; Carvalho et al., 2006; Tosto et al., 2006b; Tupy et al., 2006a, b,c,d,e,f,g; Vinholis et al., 2006; Canto et al., 2007; Galharte, 2007; Holanda Filho, 2007; Jacometi et al., 2008; Miele et al., 2008; Sabbag, 2008).
To date, as included in the Social Balance Reports 2005-2009 (published the following years), 123 innovations selected by Embrapa's Research Centers have been evaluated, corresponding to 180 matched social and environmental assessments 7. The aggregated environmental and social impact indices and their frequency distribution, according to the proposed technological integrative levels, can be seen in Figure 3. The insert graph shows that the frequency distribution of the untransformed impact indices tend to normality, despite the one-tailed distribution observed for social impact indices, given that no negative observations occurred for this impact dimension in the whole sample.
Figure 3. Distribution of environmental and social impact indices
obtained in 180 evaluations, relative to 123 technological
innovations classified as pertaining to five different
integrative levels, and frequency distribution of indices'
rankings, obtained with the Ambitec-Agro system.
Embrapa Research Centers, 2005-2007.
No definite relationship has been observed in the dataset between environmental and social impacts (r2=0.052), that is, one can not say whether a more positive impact on the environment will result in increased social benefits. In effect, there is no reason to expect this association to show any predetermined pattern, given the trade-offs between short-term income objectives and long-term environmental ones (van Pelt, 1994), and the very diverse possible interactions and adoption contexts included in the large set of technology innovations studied.
For example, highly positive environmental performance improvements, such as savings in energy expenditure, habitat restoration, or even water quality or biodiversity conservation may well be achieved without immediately impacting peoples' quality of life. Conversely, intensification in natural resources or input uses, which frequently entail negative environmental impacts, usually result in important gains in production scale, worker engagement and work formality, income generation, and social wellbeing improvements.
Several instances of negative environmental impact indices were observed in the dataset (Figure 3), related with agricultural intensification on the one hand, and expansion of agricultural areas onto new territories, resulting in natural habitats occupation, on the other hand. Crop and livestock breeding technologies have been associated with both increased demands on inputs in many cases, and expansion of crops and animal husbandry to new areas, thus frequently resulting in negative environmental impact indices (37.5%). Agroindustry / post harvest technologies (33.3% negative indices) have been linked with increased demand on energy and inputs, which also tend to impose negative environmental impacts due to resources demands and residue emissions.
Input use technologies followed in terms of negative environmental impact indices (30.8%) due to intensification in demand, while integrated management technologies (20% negative indices) were most frequently at the origin of negative environmental impacts due to agricultural expansion onto new, sometimes marginal areas, imposing pressures on natural habitats. Due to their incremental and adaptive nature, improved agricultural practices showed just 7.1% of cases with negative environmental impact indices.
By contrast with environmental impacts, social impact indices obtained in the field studies were without exception positive (Figure 3), allowing to infer from the large dataset that, in general, agricultural technology innovations forwarded by the Research Centers contribute favorably toward sustainable livelihoods and rural development.
This prevalence of highly positive social impact indices has been shown to be especially true for those innovations classified as agro-industry / post harvest (mean impact indices for all years = 3.96) and integrated management technologies (mean impact indices for all years = 3.39; Table 2). These two sets of innovations showed significantly higher social impact indices than the other three types, for all years in which assessments were carried out. While no statistical differences were observed in the ANOVA among years (environmental indices' p-value = 0.339; social indices' p-value = 0.104), specific contrasts showed that social impact indices were always significantly higher than environmental impact indices (See Table 2 at the end of the article).
On the other hand, besides showing fewer negative results, environmental impact indices were almost invariably higher for innovations classified as improved agricultural practices (except for year 2006, when agro-industry / post harvest technologies faired better) and integrated management technologies (Table 2). These two sets of innovations were shown to be significantly more effective for saving resources, managing emissions, and conserving / restoring ecosystems' functions, all explicitly considered criteria in Ambitec-Agro (see Table I).
This kind of analysis and interpretation of results, focusing on aggregate impact indices and their tendencies (as positive or negative effects) represents the first step for technology research prioritization and transfer strategy. These are based on the norm enunciated for the assessment system, which states "recommendation of agricultural technology is conditioned to improvement of the environmental and social performances of the activity to which technology is applied, as measured by designated indicators" (Rodrigues et al., 2003a). This norm does not mean that negative impact indices will immediately impose that an innovation should be discontinued, but rather indicates major issues for research continuation, emphasizing trade-offs pointed out in the assessments.
Much improved detailing of these trade-offs is possible in the impact assessment procedure, with analysis of the impact indices at the disaggregated criteria and indicator levels. At these levels, aspects pertaining to use of inputs and resources, environmental quality (atmosphere, water, soils and biodiversity conservation), natural habitat restoration, as well as product quality and food security, employment generation and quality, income, health, among many other indicators can be pondered upon, with reference elements obtained in real field contexts, together with technology adopters.
It is actually at this level of detail that reports are presented in the templates provided in the impact assessment platform. By delving into these reports, technology research teams are better equipped to define new steps for technology development and transfer, while research administrators can better decide on priorities for new R&D calls.
Discussion - Institutional Learned Lessons
Hundreds of evaluations, of a myriad of technological innovations have been carried out and gathered since the inception of systematic impact assessment practice at Embrapa. The strategic objective of extending impact evaluations beyond the economic internal rate of return or benefit / cost analyses, introducing social and environmental evaluations, has been crucial for the critique of the institutional role in agricultural development. This process, however, has not been always smooth, and the provision of a basic methodological approach may have facilitated the internalization of the impact assessment practice (Andrade and Moreira, 2009).
In what concerns the proposed methodological approach, a recent independent study on the practice of impact assessment at Embrapa has been carried out. This study pointed out both "ameliorating and funneling tendencies, by which creativity and normativeness coexist... in the institutional learning experience and the quotidian of technical innovation" (Andrade, 2008). In addition to this institutional-level critique, the applicability of Ambitec-Agro has been included in a review of several impact assessment methods (Payraudeau et al., 2005). The study emphasized the need for methods to be transparent in order to facilitate farmer participation, simple to allow uncomplicated field application, and sufficient in number and scope of indicators to avoid gaps in the assessments. The results obtained to date at Embrapa, reviewed in the present study, seem to imply that these features have been satisfactorily met.
The early decision of involving actual field surveys and farmer participation in the assessments has also served valuable objectives. First, impact causality attribution to specific innovations can be more immediately identified, excluded or harmonized, based on clear definition of technology application context, case-by-case, with active contribution of the actual users, according to their effective practice in the field (Gottret and White, 2001).
Second, this context definition step of the evaluation corresponds to the 'pressure premise' in EIA science: be adaptable to local socio-economic contexts, physicochemical environments and production scales (OECD, 1993). Third, the next phase in the assessments, the field survey / interview, scoring of technology effects, and impact indices calculations corresponds to the 'state premise' of EIA science: express the effects of changes on the quality of the environment and natural resources, including social, economical and ecological concerns. And fourth, the reporting step (with the provided templates and forms) corresponds to the 'response premise' of EIA science: offer the basis for issuing recommendations for decision making (including for farmers), contributing toward local sustainable development (OECD, 1993).
The experience on Impact Assessment at Embrapa and the recognition received from the Brazilian and international R&D organizations have shown that the adoption of a multi-dimensional approach contributes to a better understanding of the impacts of agricultural research as compared to more conventional, disciplinary evaluations. Once an integrated multi-dimension approach can be exercised, a main lesson is that impact assessment in agricultural research has a stronger influence when it is also integrated through feedbacks to institutional management, favoring research priority setting, planning, award / promotion assignment and formal institutional accountability, as consolidated in the Social Balance Reports.
Some brief recommendations can be proposed to extend the scope and strengthen the institutional impact assessment platform presented here:
Assessment methodology should evolve at a pace compatible with institutional learning, demanding continued training;
A formal, permanent discussion forum must be assigned the task of promoting methodology innovation and field application critique;
A designated critical analysis conference should meet yearly with external advisors, as a feed-back to evaluators and research teams;
The learning process constructed with repeated evaluation cycles should be accompanied by incentives for publication in influential periodicals;
Having the technology innovation impact assessment platform established, stimulate organization of a new, complementary approach, dedicated to impact assessment at the macroregion / country scales, with national statistics / databases, to allow decision making on national agricultural policies. This is an audacious scientific challenge, demanding a vigorous institutional effort.
The research presented in this text has been made possible through continued support from Embrapa, the 'Programa Cooperativo para el Desarrollo Tecnologico Agroalimentario y Agroindustrial del Cono Sur' (PROCISUR), the 'Fundacao de Amparo a Pesquisa do Estado de Sao Paulo' (FAPESP), and the 'Conselho Nacional de Desenvolvimento Cientifico e Tecnologico' (CNPq). We thank our colleague Dr. Inacio de Barros (INRA-France) for the many suggestions on an earlier version of the manuscript. Special gratitude is addressed to our colleagues who carry out Impact Assessment studies for their many contributions and relentless critique, and to farmers and rural administrators, for lending their time and knowledge in favor of the development of the general approach and its applications. We also thank to the enthusiastic national and international partners that contributed to the improvement of the IA developments at Embrapa, especially during the last decade.
4 The Cooperative Program for the Technological Development of the Agro-food and Agro-industry in the Southern Cone, a consortium of the National Institutes of Agricultural Research of South America.
6 Embrapas yearly Social Balance Reports are available at:
7 The complete lists of technological innovations and their respective impact assessment indices, applied in the following analysis, can be obtained in the annual Social Balance Reports, available in the main web-page of Embrapa, at: http://bs.sede.embrapa.br/2009/impacto.html
ADATO, M., Meinzen-Dick R. (2002). Assessing the impact of agricultural research on poverty using the sustainable livelihoods framework. Washington, D.C.: International Food Policy Research Institute. FCND Discussion Paper 128; EPTD Discussion Paper 89. [ Links ]
ALMEIDA, CO. de, Matos, A.P. de, Cardoso, C.E.L, Sanchez, N.F., Teixeira, F.A., Elias Junior, J. (2007). Avaliacao de impactos da producao integrada de abacaxi no estado do Tocantins: um estudo de caso de um sistema em transicao. Embrapa Mandioca e Fruticultura Tropical, Cruz das Almas. Documentos 167. [ Links ]
ALMEIDA, CO. de, Matos, A.P. de, Cardoso, C.E.L., Sanchez, N.F., Teixeira, F.A., Elias Junior, J. (2009). Evaluation of social impact of integrated pineapple production in the State of Tocantins - Brazil. Acta Horticulturae, 822, 169-176. [ Links ]
ANDRADE, T. (2008). Inovacao tecnologica e impactos ambientais no setor agropecuario. Proceedings of the IV Encontro da Associacao Nacional de Pesquisa e Pos-Graduacao em Ambiente e Sociedade, Brasilia, 4-6 June, 2008. [ Links ]
ANDRADE, T. N. de, Moreira Junior, A. (2009). Aperfeicoamento gerencial e inovacao tecnologica. Sociologies, 11 (22), 198-230. [ Links ]
ANDREOLI, M., Tellarini, V. (2000). Farm sustainability evaluation: methodology and practice. Agriculture, Ecosystems and Environment. 77, 43-52. [ Links ]
AVILA, A.F.D., Magalhaes, M.C., Vedovoto, G.L, Irias, L.J.M., Rodrigues, G.S. (2005). Impactos economicos, sociais e ambientais dos investimentos na Embrapa. Revista de Politico Agricola. 14(4), 86-101. [ Links ]
AVILA, A.F.D., Rodrigues, G. S., Vedovoto, G.L. (2008). Avaliacao dos impactos de tecnologias geradas pela Embrapa: metodologia de referenda. Brasilia (DF): Embrapa Informacao Tecnologica, 189 p. [ Links ]
AVILA, A. F.D., Souza, G. S. (2002). The importance of impact assessment studies for the Brazilian Agricultural Research System in Brazil. International Conference on Impacts of Agricultural Research and Development: why has impact assessment research not made more a difference? San Jose (Costa Rica), February 4-7. [ Links ]
BIN, A., Salles-Filho, S.L.M., Paulino, S.R., Rodrigues, G.S. (2003). Impactos ambientais na agricultura: um metodo de avaliacao de programas tecnologicos. Revista do Centro de Ciencias da Administracao, 9(1), 76-83. [ Links ]
BISSET, R. (1987). Methods for environmental impact assessment: a selective survey with case studies. In: Biswas, A.K., Geping, Q. (Eds). Environmental Impact Assessment for Developing Countries. Tycooly International, London, pp 3-64. [ Links ]
BONACELLI, M.B.M., Zackiewicz, M., Bin, A. (2003). Avaliacao de impactos sociais de programas tecnologicos na agricultura do Estado de Sao Paulo. Espacios, 24(2), 5-24. [ Links ]
BOSSHARD, A. (2000). A methodology and terminology of sustainability assessment and its perspectives for rural planning. Agriculture, Ecosystems and Environment. 77, 29-41. [ Links ]
CANTO, A.M.M.E., Sena, M. das G.C. de, Melo, E.S., Lima, H.G. (2007). Avaliacao participativa da sustentabilidade das inovacoes adotadas pela agricultura familiar do semiarido baiano: uma experiencia em construcao. Revista Brasileira de Agroecologia,2(1), 136-139. [ Links ]
CARVALHO, G.R., Fernandes, E.N., Muller, M.D., Oliveira, A.F. de. (2007). Avaliacao dos impactos ambientais e sociais da tecnologia "capim elefante pioneiro no sistema de pastejo rotativo". Proceedings of the XLV Congresso da Sociedade Brasileira de Economia, Administracao e Sociologia Rural. Londrina, 22-25 July. [ Links ]
CARVALHO, G.R., Ometto, A.R., Furtado, A.L.S., Mangabeira, J.A. de C, Guimaraes, M., Valladares, G.S., Batistella, M. (2006). Impacto de algumas tecnologias geradas pela Embrapa Monitoramento por Satelite. Embrapa Monitoramento por Satelite, Campinas, Documentos 50. [ Links ]
COLLINSON, M.P., Tollens E. (1994). The impact of the international agricultural centers - measurement, quantification and interpretation. Experimental Agriculture, 30(4), 395-419. [ Links ]
CONROY, C, Litvinoff, M. (1988). The Greening of Aid: Sustainable Livelihoods in Practice. Earthscan Publications Ltd, London. [ Links ]
DUARTEJ. de O., Garcia, J. C, Matoso, M.J., Santana, D.P. (2007). Avaliacao dos impactos economicos, sociais e ambientais da cultivar de sorgo granifero BR 304 na safra 2005/2006. Embrapa Milho e Sorgo, Sete Lagoas. Documentos 67. [ Links ]
EIARD (European Initiative for Agricultural Research for Development, Task Force on Impact Assessment and Evaluation). (2003). Impact assessment and evaluation in agricultural research for development. Agricultural Systems, 78(2), 329-336. [ Links ]
EMBRAPA. IV Piano Diretor da Embrapa: 2004-2007. (2004). Secretaria de Gestao e Estrategia. Embrapa, Brasilia. [ Links ]
EMBRAPA. International Workshop on Methodological Innovation in Impact Evaluation of Agricultural Research. (2008). Secretariat for Management and Strategy (SGE-Embrapa), Standing Panel for Impact Assessment (SPIA) SC-CGIAR, and Study Group on Organization of Research and Innovation (GEOPI), Unicamp. Brasilia 12-14 November. [ Links ]
ESTERHUIZEN, J.M.C., Liebenberg, G.F. (2001). The use of indicators within a comprehensive impact assessment approach in three South African research programmes. Agriculture, Ecosystems and Environment, 87(2), 233-244. [ Links ]
FERREIRA, CM., Silva, S.C. da, Lanna, A.C., Barrigossi, J.A.F., Wander, A.E. (2005). Climatic zoning for upland rice in Brazil: economic, social and environmental impacts. Proceedings of the Conference on International Agricultural Research for Development, Stuttgart-Hohenheim, 11-13 October. [ Links ]
GALHARTE, C.A. (2007). Avaliacao de impactos ambientais da integracao lavoura-pecuaria: estudo de caso da inovacao tecnologica da Embrapa. Masters Thesis, Escola de Engenharia de Sao Carlos (EESC), Ciencias da Engenharia Ambiental. Sao Carlos. [ Links ]
GOTTRET, M.A.V.N., White, D. (2001). Assessing the impact of integrated natural resource management: challenges and experiences. Conservation Ecology, 5(2), 17. [ Links ]
HALL, A., Clark, N., Rasheed, S.V., Sivamohan, M.V.K., Yoganand, B. (2000). New agendas for agricultural research in developing countries: policy analysis and institutional implications. Knowledge, Technology and Policy, 13(1), 70-91. [ Links ]
HALL, A., Sulaiman, V.R., Clark, N., Yoganand B. (2003). From measuring impact to learning institutional lessons: an innovation systems perspective on improving the management of international agricultural research. Agricultural Systems, 78(2), 21 3-241. [ Links ]
HAQUE, M.M. (1991). Sustainable development and environment: a challenge to technology choice decision-making. Project Appraisal, 6(3), 149-157. [ Links ]
HOLANDA FILHO, Z.F. (2007). Avaliacao de impactos socioambientais da tecnologia de cultivares de banana resistentes a sigatoka-negra em area de assentamento de reforma agraria no Estado de Rondonia. Masters Thesis, Fundacao Universidade Federal de Rondonia - Nucleo de Ciencias e Tecnologia, Porto Velho. [ Links ]
HORTON, D. (1998). Disciplinary roots and branches of evaluation: some lessons from agricultural research. Knowledge and Policy, 10(4), 31 -66. [ Links ]
HORTON, D., Mackay, R. (2003). Using evaluation to enhance institutional learning and change: recent experiences with agricultural research and development. Agricultural Systems, 78 (2), 127-142. [ Links ]
IRIAS, L.J.M., Gebler, L, Palhares, J.C.P., Rosa, M.F. de, Rodrigues, G.S. (2004a). Avaliacao de impacto ambiental de inovacoes tecnologicas agropecuarias - aplicacao do Sistema Ambitec. Agricultura em Sao Paulo. 51(1), 23-40. [ Links ]
IRIAS, L.J.M., Rodrigues, G.S., Campanhola, C, Kitamura, P.C., Rodrigues, I., Buschinelli, CCA. (2004b). Sistema de avaliacao de impacto ambiental de inovacoes tecnologicas nos segmentos agropecuario, producao animal e agroindustria (Sistema Ambitec). Embrapa Meio Ambiente. Jaguariuna. Circular Tecnica 5. [ Links ]
JACOMETI, W.A., Paulino, S.R., Queda, O. (2008). Avaliacao de impactos ambientais e sociais da certificacao Eurep-Gap em propriedades de limao Tahiti. Informacoes Economicas, 38(8), 62-78. [ Links ]
LANNA, A.C., Ferreira, CM., Barrigossi, J.A.F. (2004). Analise do impacto ambiental da cultivar de feijao BRS Perola. Embrapa Arroz e Feijao, Goiania. Comunicado Tecnico 80. [ Links ]
LEWANDOWSKI, I., Hardtlein, M., Kaltschmitt, M. (1999). Sustainable crop production: definition and methodological approach for assessing and implementing sustainability. Crop Sciences, 39, 184-193. [ Links ]
MACKAY, R., Horton D. (2003). Expanding the use of impact assessment and evaluation in agricultural research and development. Agricultural Systems, 78(2), 143-165. [ Links ]
MAGALHAES, M.C., Vedovoto, G.L, Mas., L.J.M., Vieira, R. de C.M.T., Avila, A.F.D. (2006). Avaliacao dos impactos da pesquisa da Embrapa: uma amostra de 12 tecnologias. Embrapa, Secretaria de Gestao e Estrategia, Brasilia. Documentos I 3. [ Links ]
MIELE, M., Girotto, A.F., Mores, M., Amaral, A.L. do, Zanella, J.R.C. (2008). Avaliacao dos impactos economicos e sociais do programa de erradicacao da doenca de Aujeszky no Estado de Santa Catarina. Proceedings of the XLVI Congresso da Sociedade Brasileira de Economia, Administracao e Sociologia Rural. Rio Branco, 20-23 July. [ Links ]
MONTEIRO, R.C., Rodrigues, G.S. (2006). A system of integrated indicators for socio-environmental assessment and eco-certification in agriculture - Ambitec-Agro. Journo/ of Technology Management and Innovation. 1 (3), 47-59. [ Links ]
MORI, C. de, Scheeren, P.L, Minella, E., Fontaneli, R.S., Santos, H.P. dos, Lorini, I. (2007). Avaliacao de impactos economicos sociais e ambientais de algumas tecnologias geradas pela Embrapa Trigo. Embrapa Trigo, Passo Fundo. Documentos 90. [ Links ]
NEINKE, B., Avila, A.F.D., Pardey, P. (2001). Agricultural R&D in Brazil: Policy, Investments and Institutional Profile. IFPRI, Embrapa/Fontagro. Washington, DC. 109p. [ Links ]
NELSON, M., Maredia, M. K. (2007). International agricultural research as a source of environmental impacts: challenges and possibilities. Journal of Environmental Assessment Policy and Management. 9(1), 103-1 19. [ Links ]
OECD (1993). Core set of indicators for environmental performance reviews. Environmental Monographs 83. Organization for Economic Cooperation and Development. Paris. [ Links ]
PAYRAUDEAU, S., Hayo, M.G., Van der Werf, H. (2005). Environmental impact assessment for a farming region: a review of methods. Agriculture Ecosystem and Environment. 107, 1-19. [ Links ]
PINHO, P., Pires, A.R. (1991). Social impact analysis in environmental impact assessment: a Portuguese agricultural case study. Project Appraisal. 6(1), 2-6. [ Links ]
PORTER, A.L. (1995). Technology assessment. Impact Assessment, 13, 135-151. [ Links ]
PUIGNAU, J.P. (1998). Valoracion Economica en el Uso de los Recursos Naturales y del Medio Ambiente. Dialogo 51. IICA/PROCISUR, Montevideo. [ Links ]
RAINA, R.S. (2003). Disciplines, institutions and organizations: impact assessments in context. Agricultural Systems, 78(2), 185-211. [ Links ]
RODIGHERI, H.R., lede, E.T., Penteado, S. do R.C., Reis Filho, W. (2006). Avaliacao dos impactos do programa de manejo integrado de pragas para o controle da vespa-da-madeira em plantios de pinus no sul do Brasil. Embrapa Florestas, Colombo. Comunicado Tecnico 158. [ Links ]
RODRIGUES, G.S. (1998). Avaliacao de impactos ambientais em projetos de pesquisas - fundamentos, principios e introducao a metodologia. Embrapa Meio Ambiente, Jaguariuna. Documentos 14. [ Links ]
RODRIGUES, G.S., Brown, M.T., Miller, W., Ruz, E., Riquelme, H. (1998). Natural resource valuation, environmental impact assessment and sustainability: the role of the NIARs in the Southern Cone. In: Puignau, J.P. (Ed). Valoracion Economica en el Uso de los Recursos Naturales y el Medio Ambiente, IICA/PROCISUR, Dialogo 51. Montevideo, pp. 113-116. [ Links ]
RODRIGUES, G.S., Buschinelli, C.C. de A., Irias, L.J.M., Ligo, M.A.V. (2000). Avaliacao de impactos ambientais em projetos de pesquisa II: avaliacao da formulacao de projetos. Embrapa Meio Ambiente, Jaguariuna. Boletim de Pesquisa 10. [ Links ]
RODRIGUES, G.S., Campanhola, C, Kitamura, P.C. (2002). Avaliacao de impacto ambiental da inovacao tecnologica agropecuaria: um sistema de avaliacao para o contexto institucional de P&D. Cadernos de Ciencia e Tecnologia. 19(3), 349-375. [ Links ]
RODRIGUES, G.S., Campanhola, C, Kitamura, P.C. (2003a). An environmental impact assessment system for agricultural R&D. Environmental Impact Assessment Review. 23(2), 219-244. [ Links ]
RODRIGUES, G.S., Campanhola, C., Kitamura, P.C. (2003b). Avaliacao de impacto ambiental da inovacao tecnologica agropecuaria: AMBITEC-AGRO. Embrapa Meio Ambiente, Jaguariuna. Documentos 34. [ Links ]
RODRIGUES, G.S., Campanhola, C, Kitamura, P.C, Irias, L.J.M., Rodrigues, I.A. (2005a). Sistema de avaliacao de impacto social da inovacao tecnologica agropecuaria (Ambitec-Social). Embrapa Meio Ambiente, Jaguariuna. Boletim de Pesquisa e Desenvolvimento 35. [ Links ]
RODRIGUES, G.S., Rodrigues, I.A., Tupy, O., Camargo, A.C. de, Novo, A.L.M., Bonadio, L.F., Tokuda, F.S., Andrade, E.F., Shiota, CM., Silva, R.A. da. (2006). Avaliacao socio-ambiental da integracao tecnologica Embrapa Pecuaria Sudeste para producao leiteira na agricultura familiar. Agricultura em Sao Paulo. 53(2), 35-48. [ Links ]
RODRIGUES, G.S., Rodrigues, I.A. (2007). Avaliacao de impactos ambientais na agropecuaria. In: Gebler, L and Palhares, J.C.P. (Eds.) Gestao Ambiental na Agropecuaria. Embrapa Informacao Tecnologica, Brasilia, pp. 285-310. [ Links ]
SA, C.P. de, Bayma, M.M.A., Carneiro Jr, J.M. (2008). Aspectos economicos e ambientais da utilizacao do amendoim forrageiro para a recria-engorda de bovinos de corte no Acre. Proceedings of the XLVI Congresso da Sociedade Brasileira de Economia, Administracao e Sociologia Rural. Rio Branco, 20-23 July. [ Links ]
SABBAG, O.J. (2008). Avaliacao de impactos ambientais pos-certificacao EurepGap na cultura do abacaxi em Guaracai (SP). Pesquisa Agropecuaria Tropical, 38(4), 284-289. [ Links ]
SILVA, O.F. da, Lanna, A.C, Wander, A.E., Barrigossi, J.A.F., Santos, A.B. dos. (2008). Impacto socioeconomico e ambiental da soca de arroz produzida na microrregiao do Rio Formoso, Estado do Tocantins. Revista Redes, 13(1), 28-48. [ Links ]
TOSTO, S.G., Brandao, E.S., Macedo, J.R. de, Capeche, C.L. (2006a). Avaliacao de impacto ambiental - producao de tomate de mesa ecologicamente cultivado no municipio de Sao Jose de Uba, RJ - uma aplicacao do metodo Ambitec-Agro. Embrapa Solos, Rio de Janeiro. Boletim de Pesquisa e Desenvolvimento 105. [ Links ]
TOSTO, S.G., Capeche, C.L, Ferraz, R.P.D., Andrade, A.G. de, Brandao, E.S., Costa, J.R.P.F. da. (2006b). Avaliacao de impacto ambiental - o caso do projeto "Aeroporto Internacional do Rio de Janeiro - Tom Jobim" - uma aplicacao do metodo Ambitec-Agro. Embrapa Solos, Rio de Janeiro. Boletim de Pesquisa e Desenvolvimento 93. [ Links ]
TUPY, O., Primavesi, O. (2006a). Avaliacao dos impactos economicos, sociais e ambientais da Embrapa Pecuaria Sudeste: Analise estrategica de custos de producao de leite. Embrapa Pecuaria Sudeste, Sao Carlos. [ Links ]
TUPY, O., Vinholis, M. de M.B., Primavesi, O., Bernardi, A.C de C. (2006b). Avaliacao dos impactos economicos, ambientais e sociais de tecnologia da Embrapa Pecuaria Sudeste: casinha tropical-abrigo individual para bezerros. Embrapa Pecuaria Sudeste, Sao Carlos. [ Links ]
TUPY, O., Primavesi, O., Rodrigues, A. de A. (2006c). Avaliacao dos impactos economicos, sociais e ambientais de tecnologia da Embrapa Pecuaria Sudeste: cultivar de cana-de-acucar IAC86-2480, desenvolvida pelo IAC, e testada e recomendada pela Embrapa Pecuaria Sudeste para alimentacao de bovinos. Embrapa Pecuaria Sudeste, Sao Carlos. [ Links ]
TUPY, O., Vinholis, M. de M.B., Souza, G.B. de, Nogueira, A.R. de A., Primavesi, O. (2006d). Avaliacao dos impactos economicos, sociais e ambientais de tecnologia da Embrapa Pecuaria Sudeste: metodo alternativo para determinacao de fibra em detergente neutro e detergente acido. Embrapa Pecuaria Sudeste, Sao Carlos. [ Links ]
TUPY, O., Anchao, P. P., Vinholis, M. de M.B., Primavesi, O., Bernardi, A.C. de C. (2006e). Avaliacao de impactos economicos, sociais e ambientais de tecnologia da Embrapa Pecuaria Sudeste: sobressemeadura de aveia forrageira em pastagens tropicais irrigadas no periodo seco. Embrapa Pecuaria Sudeste, Sao Carlos. [ Links ]
TUPY, O., Primavesi, O., Camargo, A.C. de. (2006f). Avaliacao dos impactos economicos, sociais e ambientais de tecnologia da Embrapa Pecuaria Sudeste: tecnicas de producao intensiva aplicadas a propriedades familiares produtoras de leite. Embrapa Pecuaria Sudeste Sao Carlos. [ Links ]
TUPY, O., Primavesi, O., Barbosa, P.F. (2006g). Avaliacao dos impactos economicos, sociais e ambientais de tecnologia da Embrapa Pecuaria Sudeste: utilizacao de touros da raca canchim em cruzamento terminal com femeas da raca nelore. Embrapa Pecuaria Sudeste Sao Carlos. [ Links ]
VAN DE FLIERT, E., Braun A.R. (2002). Conceptualizing integrative, farmer participatory research for sustainable agriculture: from opportunities to impact. Agriculture and Human Values, 19(1), 25-38. [ Links ]
VAN PELT, M. J. F. (1994). Sustainability-oriented appraisal of agricultural projects. Journal of International Development, 6(1), 57-78. [ Links ]
VINHOLIS, M. de M.B., Pedroso, A.F., Primavesi, O., Tupy, O., Bernardi, A.C.C. (2006). Impactos economico,social e ambiental de um abrigo individual movel para bezerros. Interface Tecnologica. 3, 73-82. [ Links ]
WATTS, J., Mackay, R., Horton, D., Hall, A., Douthwaite, B., Chambers, R., Acosta, A. (2003). Institutional learning and change: an introduction. International Service for National Agricultural Research, ISNAR Discussion Paper No.03-10, 19 p. [ Links ]
Accepted November 29, 2010