Hostname: page-component-68c7f8b79f-m4fzj Total loading time: 0 Render date: 2026-01-02T02:57:59.642Z Has data issue: false hasContentIssue false
  • English
  • Français

Evaluation Logics in the Third Sector

Published online by Cambridge University Press:  01 January 2026

Matthew Hall
Matthew Hall*
Affiliation:
London School of Economics and Political Science, Houghton St, London WC2A 2AE, UK
*
e-mail: m.r.hall@lse.ac.uk
Get access

Abstract

In this paper I provide a preliminary sketch of the types of logics of evaluation in the third sector. I begin by tracing the ideals that are evident in three well-articulated yet quite different third sector evaluation practices: the logical framework, most significant change stories, and social return on investment. Drawing on this analysis, I then tentatively outline three logics of evaluation: a scientific evaluation logic (systematic observation, observable and measurable evidence, objective and robust experimental procedures), a bureaucratic evaluation logic (complex, step-by-step procedures, analysis of intended objectives), and a learning evaluation logic (openness to change, wide range of perspectives, lay rather than professional expertise). These logics draw attention to differing conceptions of knowledge and expertise and their resource implications, and have important consequences for the professional status of the practitioners, consultants, and policy makers that contribute to and/or are involved in evaluations in third sector organizations.

Résumé

Résumé

Dans cet article j’offre une esquisse préliminaire des divers types de logique d’évaluation du tiers-secteur. Je commence par décrire les idéaux que trois pratiques d’évaluation du tiers-secteur bien définies et distinctes font clairement apparaître. A partir de cette analyse, je trace ensuite les contours de trois logiques d’évaluation : une logique d’évaluation scientifique (observation scientifique, preuves observables et mesurables, procédures d’expérimentation objectives et solides), une logique d’évaluation bureaucratique (procédures pas-à-pas complexes, analyse des objectifs choisis) et une logique d’évaluation didactique (ouverture au changement, grande variété de perspectives, préférence pour l’expertise non-professionnelle). Ces logiques attirent notre attention sur des conceptions divergentes du savoir et de l’expertise et leurs implications en termes de ressources. Elles ont aussi des conséquences importantes pour le statut professionnel des praticiens, des consultants et des décideurs, qui contribuent à et/ou sont partie intégrante des évaluations des organisations du tiers-secteur.

Zusammenfassung

Zusammenfassung

In diesem Beitrag präsentiere ich einen vorläufigen Entwurf über Formen der Bewertungslogiken im Dritten Sektor. Zunächst verfolge ich die Leitbilder, die in drei gut verständlichen, jedoch sehr unterschiedlichen Bewertungspraktiken des Dritten Sektors ersichtlich sind: das logische Rahmenwerk, die wichtigsten Hintergründe für Veränderungen und die Sozialrendite. Unter Bezugnahme auf diese Analyse präsentiere ich sodann eine vorläufige Darstellung dreier Bewertungslogiken: eine wissenschaftliche Bewertungslogik (systematische Beobachtung, beobachtbare und messbare Nachweise, objektive und robuste experimentelle Verfahren), eine bürokratische Bewertungslogik (umfangreiche, schrittweise Verfahren, Analyse beabsichtigter Ziele) und eine lernende Bewertungslogik (Offenheit für Veränderungen, viele unterschiedliche Perspektiven, Laienexpertise statt professioneller Expertise). Diese Logiken lenken die Aufmerksamkeit auf unterschiedliche Auffassungen von Wissen und Expertise und deren Auswirkungen auf die Ressourcen. Sie haben weitreichende Konsequenzen für den professionellen Status der Praktizierenden, Berater und Entscheidungsträger, die in Organisationen des Dritten Sektors zu Bewertungen beitragen oder sich mit diesen befassen.

Resumen

Resumen

En el presente documento, proporciono un esbozo preliminar de los tipos de lógica de evaluación en el sector terciario. Comienzo por seguir el rastro a los ideales que son evidentes en las tres prácticas, bien articuladas pero muy diferentes, de evaluación del sector terciario: el marco lógico, las historias de cambio más significativas y la rentabilidad social de la inversión. Echando mano de este análisis, esbozo después provisionalmente tres lógicas de evaluación: una lógica de evaluación científica (observación sistemática, evidencia observable y mensurable, procedimientos experimentales objetivos y robustos), una lógica de evaluación burocrática (procedimientos complejos, paso a paso, análisis de los objetivos perseguidos) y una lógica de evaluación de aprendizaje (apertura al cambio, amplia gama de perspectivas, experiencia técnica secular en vez de profesional). Estas lógicas llaman la atención sobre diferentes concepciones del conocimiento y de la experiencia técnica y sobre las implicaciones de sus recursos, y tienen consecuencias importantes para el estatus profesional de los profesionales, asesores y aquellos que toman las decisiones que contribuyen a, y/o están implicados en, evaluaciones en organizaciones del sector terciario.

摘要

摘要

本文对评估第三产业的逻辑类型进行了初步探讨。首先,我针对三种论述缜密却又彼此迥异的第三产业评估方法——逻辑框架、最显著的改变案例和社会投资回报,分别梳理了每种方法的思路。在此分析的基础上,我试着描摹出三种评估逻辑的大致梗概:一种是科学型的评估逻辑(系统观察、可观察和度量的证据、客观严密的实验步骤),一种是官僚型的评估逻辑(复杂、按部就班、分析预定目标),还有就是学习型的评估逻辑(接受改变,广阔的视角,重视外行而非专业人士的意见)。在如何看待知识和专业技能及资源的意义这个问题上,奉行不同逻辑的人会有不同的理解。同时,对那些负责和/或参与第三产业评估的从业者、顾问和政策制定者来说,上述逻辑对其专业身份亦有重要影响。.

要約

要約

本論文では第3セクターの評価における理論のタイプの略図を提示する。まず、論理的な枠組み、著しい変化、社会的な投資利益率という3つの連結されてはいるが、全く異なる第3セクターの評価実践を辿る。次に、この分析を利用する評価においての3つの理論、すなわち科学的評価論理(組織的観察、観察可能で測定できる証拠、客観的で強健な実験手順)、官僚の評価論理(複雑、段階的な手順、意図された目的の分析)、学習評価論理(変更に対する寛容さ、より専門的な知識)について暫定的に概説する。これらの論理から知識と専門的技術の異なる概念と資金の含意における異なる概念が注目されるが、開業医、コンサルタント、第3セクターの組織評価にかかわる政策立案者の専門的地位に重要な結果を与えるといえる。.

ملخص

ملخص

أقدم في هذا البحث وصف تمهيدي لأنواع تقييم المنطق في القطاع الثالث. أبدأ من خلال تتبع المثل العليا التي هي واضحة في ثلاث معالم واضحة مع ذلك مختلفة تماما˝ ممارسات تقييم القطاع الثالث: الإطار المنطقي، قصص أهم تغيير، والعائد الاجتماعي على الإستثمار. بالإعتماد على هذا التحليل، أنا بعد ذلك حددت مبدئيا˝ ثلاثة منطق للتقييم:منطق التقييم العلمي،(أدلة ملحوظة وقابلة للقياس ، إجراءات موضوعية وتجريبية قوية)، منطق التقييم البيروقراطي (إجراءات معقدة، خطوة بخطوة ، تحليل أهداف مقصود) وتعلم منطق التقييم (الإنفتاح للتغيير، مجموعة واسعة من وجهات النظر، وضع بدلا من الخبرة المهنية). هذا المنطق لفت الإنتباه إلى إختلاف مفاهيم المعرفة والخبرة و آثار الموارد الخاصة بها ، ولها آثار هامة على الوضع المهني للممارسين، والإستشاريين، وصانعي السياسات التي تسهم في و/أو تشارك في عمليات التقييم في منظمات القطاع الثالث.

Keywords

EvaluationLogicsPerformance measurementAccountabilityExpertise

Information

Type
Original Paper
Information
Voluntas: International Journal of Voluntary and Nonprofit Organizations , Volume 25 , Issue 2 , April 2014 , pp. 307 - 336
Copyright
Copyright © International Society for Third-Sector Research and The Johns Hopkins University 2012

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Abma, T. A. (1997). Playing with/in plurality: revitalizing realities and relationships in Rotterdam. Evaluation, 3, 25–48. doi: 10.1177/135638909700300103.CrossRefGoogle Scholar
Acumen Fund (2007). The Best Available Charitable Option. Retrieved September 19, 2011 from http://www.acumenfund.org/uploads/assets/documents/BACO%20Concept%20Paper%20final_B1cNOVEM.pdf.Google Scholar
Bagnoli, L., & Megali, C. (2011). Measuring performance in social enterprises. Nonprofit and Voluntary Sector Quarterly, 40, 149–165. doi: 10.1177/0899764009351111.CrossRefGoogle Scholar
Barman, E. (2007). What is the bottom line for third sector organizations? A history of measurement in the British voluntary sector. VOLUNTAS International Journal of Voluntary and Nonprofit Organizations, 18, 101–115. doi: 10.1007/s11266-007-9039-3.CrossRefGoogle Scholar
Benjamin, L. M. (2008). Account space: How accountability requirements shape nonprofit practice. Nonprofit and Voluntary Sector Quarterly, 37, 201–223. doi: 10.1177/0899764007301288.CrossRefGoogle Scholar
Blalock, A. N. (1999). Evaluation research and the performance management movement: from estrangement to useful integration? Evaluation, 5, 117–149. doi: 10.1177/13563899922208887.CrossRefGoogle Scholar
BOND. (2003). Logical framework analysis. Guidance notes No.4.Google Scholar
Bouchard, J. M. (2009a). The worth of the social economy. In Bouchard, J. M. (Ed.), The worth of the social economy: An international perspective (pp. 11–18). Brussels: P.I.E. Peter Lang.Google Scholar
Bouchard, J. M. (2009b). The evaluation of the social economy in Quebec, with regards to stakeholders, mission and organizational identity. In Bouchard, J. M. (Ed.), The worth of the social economy: An international perspective (111–132). Brussels: P.I.E. Peter Lang.Google Scholar
Campos, J. G., Andion, C., Serva, M., Rossetto, A., & Assumpe, J. (2011). Performance evaluation in non-governmental organizations (NGOs): An analysis of evaluation models and their applications in Brazil. VOLUNTAS International Journal of Voluntary and Nonprofit Organizations, 22, 238–258. doi: 10.1007/s11266-010-9145-5.CrossRefGoogle Scholar
Carman, J. G. (2007). Evaluation practice among community-based organizations: research into the reality. American Journal of Evaluation, 28, 60–75. doi: 10.1177/1098214006296245.CrossRefGoogle Scholar
Charities Evaluation Service. (2008). Accountability and learning: Developing monitoring and evaluation in the third sector. London: Charities Evaluation Service.Google Scholar
Clear Horizons. (2009). Quick start guide: MSC design. Retrieved November 1, 2012 from http://www.clearhorizon.com.au/wp-content/uploads/2009/02/quick-start_feb09.pdf.Google Scholar
Conlin, S., & Stirrat, R. L. (2008). Current challenges in development evaluation. Evaluation, 14, 193–208. doi: 10.1177/1356389007087539.CrossRefGoogle Scholar
Crewe, E., & Harrison, E. (1998). Whose development? An ethnography of aid. London: Zed Books.Google Scholar
Dart, J., & Davies, R. (2003). A dialogical, story-based evaluation tool: The most significant change technique. American Journal of Evaluation, 24, 137–155. doi: 10.1177/109821400302400202.CrossRefGoogle Scholar
Dart, J., & Davies, R. (2003b). Quick-start guide: A self-help guide for implementing the most significant change technique (MSC). Retrieved November 1, 2012 from http://www.clearhorizon.com.au/wp-content/uploads/2008/12/dd-2003-msc_quickstart.pdf.Google Scholar
Davies, R. (1998). An evolutionary approach to organisational learning: an experiment by an NGO in Bangladesh. In Mosse, D., Farrington, J. & Few, A. (Eds.), Development as process: Concepts and methods for working with complexity (pp. 68–83). London: Routledge.Google Scholar
Davies, R., & Dart, J. (2005). The ‘most significant change’ (MSC) technique: A guide to its use. Retrieved November 1, 2012 from http://www.mande.co.uk/docs/MSCGuide.pdf.Google Scholar
Department for International Development (DFID). (2009). Guidance on using the revised logical framework. DFID Practice Paper.Google Scholar
Durie, S., Hutton, E., & Robbie, K. (2007). Investing in Impact: Developing Social Return on Investment. Retrieved September 14, 2012 from http://www.socialimpactscotland.org.uk/media/1200/SROI-%20investing%20in%20impact.pdf.Google Scholar
Easterling, D. (2000). Using outcome evaluation to guide grantmaking: Theory, reality, and possibilities. Nonprofit and Voluntary Sector Quarterly, 29, 482–486. doi: 10.1177/0899764000293009.CrossRefGoogle Scholar
Ebrahim, A. (2002). Information struggles: The role of information in the reproduction of NGO-funder relationships. Nonprofit and Voluntary Sector Quarterly, 31, 84–114. doi: 10.1177/0899764002311004.CrossRefGoogle Scholar
Ebrahim, A. (2005). Accountability myopia: Losing sight of organizational learning. Nonprofit and Voluntary Sector Quarterly, 34, 56–87. doi: 10.1177/0899764004269430.CrossRefGoogle Scholar
Ebrahim, A., & Rangan, V. K. (2011). Performance measurement in the social sector: a contingency framework. Social Enterprise Initiative, Harvard Business School, working paper.Google Scholar
Eckerd, A., & Moulton, S. (2011). Heterogeneous roles and heterogeneous practices: Understanding the adoption and uses of nonprofit performance evaluations. American Journal of Evaluation, 32, 98–117. doi: 10.1177/1098214010381780.CrossRefGoogle Scholar
Eme, B. (2009). Miseries and worth of the evaluation of the social and solidarity-based economy: For a paradigm of communicational evaluation. In Bouchard, J. M. (Ed.), The worth of the social economy: An international perspective (pp. 63–86). Brussels: P.I.E. Peter Lang.Google Scholar
Enjolras, B. (2009). The public policy paradox. Normative foundations of social economy and public policies: Which consequences for evaluation strategies? In Bouchard, J. M. (Ed.), The worth of the social economy: An international perspective (pp. 43–62) Brussels: P.I.E. Peter Lang.Google Scholar
Fine, A. H., Thayer, C. E., & Coghlan, A. T. (2000). Program evaluation practice in the nonprofit sector. Nonprofit Management and Leadership, 10(3), 331–339. doi: 10.1002/nml.10309.CrossRefGoogle Scholar
Fowler, A. (2002). Assessing NGO performance: Difficulties, dilemmas and a way ahead. In Edwards, M. & Fowler, A. (Eds.), The earthscan reader on NGO management (pp. 293–307). London: Earthscan.Google Scholar
Friedland, R., and Alford, R. R. (1991). Bringing society back in: Symbols, practices, and institutional contradictions. In Powell, W. W. & DiMaggio, P. J. (Eds.), The new institutionalism in organizational analysis (pp. 232–266). Chicago: University of Chicago Press.Google Scholar
Gasper, D. (2000). Evaluating the ‘logical framework approach’: Towards learning-oriented development evaluation. Public Administration and Development, 20, 17–28. doi: 10.1002/1099-162X(200002)20:1<17::AID-PAD89>3.0.CO;2-5.3.0.CO;2-5>CrossRefGoogle Scholar
Greene, J. C. (1999). The inequality of performance measurements. Evaluation, 5, 160–172. doi: 10.1177/13563899922208904.CrossRefGoogle Scholar
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park: Sage Publications.Google Scholar
Hoefer, R. (2000). Accountability in action? Program evaluation in nonprofit human service agencies. Nonprofit Management and Leadership, 11(2), 167–177. doi: 10.1002/nml.11203.CrossRefGoogle Scholar
Howes, M. (1992). Linking paradigms and practice: Key issues in the appraisal, monitoring and evaluation of British NGO projects. Journal of International Development, 4, 375–396. doi: 10.1002/jid.3380040404.CrossRefGoogle Scholar
Keystone (undated). Impact Planning, Assessment and Learning. Retrieved September 19, 2011 from http://www.keystoneaccountability.org/sites/default/files/1%20IPAL%20overview%20and%20service%20offering_0.pdf.Google Scholar
Jacobs, A., Barnett, C., & Ponsford, R. (2010). Three approaches to monitoring: Feedback systems, participatory monitoring and evaluation and logical frameworks. IDS Bulletin, 41, 36–44. doi: 10.1111/j.1759-5436.2010.00180.x.CrossRefGoogle Scholar
Kaplan, R. S. (2001). Strategic performance measurement and management in third sector organizations. Nonprofit Management and Leadership, 11(3), 353–371. doi: 10.1002/nml.11308.CrossRefGoogle Scholar
LeRoux, K., & Wright, N. S. (2011). Does performance measurement improve strategic decision making? Findings from a national survey of nonprofit social service agencies. Nonprofit and Voluntary Sector Quarterly (in press).Google Scholar
Lingane, A., & Olsen, S. (2004). Guidelines for social return on investment. California Management Review, 46, 116–135. doi: 10.2307/41166224.CrossRefGoogle Scholar
Lounsbury, M. (2008). Institutional rationality and practice variation: new directions in the institutional analysis of practice. Accounting, Organizations and Society, 33, 349–361. doi: 10.1016/j.aos.2007.04.001.CrossRefGoogle Scholar
Marquis, C., & Lounsbury, M. (2007). Vive la resistance: Competing logics and the consolidation of U.S. community banking. Academy of Management Journal, 50, 799–820. doi: 10.5465/AMJ.2007.26279172.CrossRefGoogle Scholar
Marsden, D., & Oakley, P. (1991). Future issues and perspectives in the evaluation of social development. Community Development Journal, 26, 315–328. doi: 10.1093/cdj/26.4.315.CrossRefGoogle Scholar
McCarthy, J. (2007). The ingredients of financial transparency. Nonprofit and Voluntary Sector Quarterly, 36, 156–164. doi: 10.1177/0899764006296847.CrossRefGoogle Scholar
New Economics Foundation (NEF) (2007). Measuring real value: A DIY guide to social return on investment. London: New Economics Foundation.Google Scholar
New Economics Foundation (NEF) (2008). Measuring value: A guide to social return on investment (SROI) (2nd ed.). London: New Economics Foundation.Google Scholar
New Philanthropy Capital (NPC). (2010). Social return on investment. Position paper. Retrieved November 1, 2012 from http://www.thinknpc.org/publications/social-return-on-investment-position-paper/.Google Scholar
Nicholls, A. (2009). We do good things, don’t we? ‘Blend value accounting’ in social entrepreneurship. Accounting, Organizations and Society, 34, 755–769. doi: 10.1016/j.aos.2009.04.008.CrossRefGoogle Scholar
Office of the Third Sector (UK Cabinet Office). (2009). A guide to social return on investment. London: Office of the Third Sector.Google Scholar
Olsen, S., & Nicholls, J. (2005). A framework for approaches to SROI analysis. Retrieved November 1, 2012 from http://ccednet-rcdec.ca/sites/ccednet-rcdec.ca/files/ccednet/pdfs/2005-050624_SROI_Framework.pdf.Google Scholar
Porter, M. E., & Kramer, M. R. (1999). Philanthropy’s new agenda: creating value. Harvard Business Review, November/December, 121–130.Google Scholar
Reed, E., & Morariu, J. (2010). State of evaluation 2010: Evaluation practice and capacity in the nonprofit sector. Innovation Network. Retrieved November 1, 2012 from http://www.innonet.org/client_docs/innonet-state-of-evaluation-2010.pdf.Google Scholar
Roberts Enterprise Development Fund (REDF) (2001). Social return on investment methodology: Analyzing the value of social purpose enterprise within a social return on investment framework. Retrieved November 1, 2012 from http://www.redf.org/learn-from-redf/publications/119.Google Scholar
Roche, C. (1999). Impact assessment for development agencies: learning to value change. Oxford: Oxfam GB doi: 10.3362/9780855987701.CrossRefGoogle Scholar
Rosenberg, L. J., & Posner, L. D. (1979). The logical framework: a manager’s guide to a scientific approach to design and evaluation. Washington DC: Practical Concepts.Google Scholar
Scott, J. C. (1998). Seeing like a state: How certain schemes to improve the human condition have failed. New Haven: Yale University Press.Google Scholar
SIDA. (2004). The logical framework approach: A summary of the theory behind the LFA method. Stockholm: SIDA.Google Scholar
SIDA. (2006). Logical framework approach—with an appreciative approach. Stockholm: SIDA.Google Scholar
Stark, D. (2009). The sense of dissonance: Accounts of worth in economic life. Princeton: Princeton University Press.10.1515/9781400831005CrossRefGoogle Scholar
Urban Institute. (2006). Building a Common Outcome Framework to Measure Nonprofit Performance. Retrieved September 19, 2011 from http://www.urban.org/UploadedPDF/411404_Nonprofit_Performance.pdf.Google Scholar
W. K. Kellogg Foundation (2004). Logic model development guide: Using logic models to bring together planning, evaluation and action. Battle Creek: W. K. Kellogg Foundation.Google Scholar
Wallace, T., Bornstein, L., & Chapman, J. (2007). The aid chain: Coercion and commitment in development NGOs. Rugby, UK: Practical Action Publishing.10.3362/9781780440019CrossRefGoogle Scholar
Waysman, M., & Savaya, R. (1997). Mixed method evaluation: A case study. American Journal of Evaluation, 18, 227–237. doi: 10.1177/109821409701800123.Google Scholar
Weber, M. (1904/1949). The methodology of the social sciences (E. A. Shils & H. A. Finch (Eds. and Trans.). Free Press: New York.Google Scholar
World Bank (2006). Conducting quality impact evaluations under budget, time and data constraints. Washington, DC: Independent Evaluation Group.Google Scholar
World Bank. (undated). The LogFrame handbook: A logical framework approach to project cycle management. Retrieved November 1, 2012 from http://www.wau.boku.ac.at/fileadmin/_/H81/H811/Skripten/811332/811332_G3_log-framehandbook.pdf.Google Scholar