Research and Tools
“Making evidence-based decisions in schools seems impossible when there are hundreds of activities to evaluate, little to no research evidence about most of them, and immovable budget deadlines to meet. Everyone wants a program that ‘works,’ but working can mean different things to different stakeholders.”
— Fiona Hollands, Founder & Managing Director
Overview
EdResearcher collaborates with education agencies, other researchers, instructional designers, and software engineers to conduct studies of educational programs, policies, and practices; and to provide guidance, tools, technical assistance, and training in economic evaluation and evidence-based decision-making.
Key Goals
Key goals are to inform and facilitate decision-making that
engages stakeholders
encourages consideration of both tangible and intangible costs and returns to investing in education
attends to how resources are distributed to meet varying needs.
Economic evaluation
-
Hollands, F. (2016, January 14). How one researcher uses cost-benefit analysis to rule out the worst tools. EdSurge.
-
Hollands, F. M., & Levin, H. M. (2017). The critical importance of costs for education decisions (REL 2017–274). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development.
Hollands, F. M., Pratt-Williams, J., & Shand, R. (2021). Cost Analysis standards & guidelines 1.1. Cost Analysis in Practice (CAP) Project. https://capproject.org/resources
Cost Analysis Standards Project. (2021). Standards for the economic evaluation of educational and social programs. American Institutes for Research.
-
Cost Analysis in Practice Project (CAP Project)
A variety of tools and templates to facilitate the execution of cost analysis and cost-effectiveness analysis of educational and other social programs.CAP Project Online Modules
A set of 5 free, asynchronous, online modules on designing a cost analysis, collecting and recording cost data, analyzing cost data to produce cost estimates, and reporting cost and cost-effectiveness metrics.E$timator
Free, online software with an embedded database of educational resource prices to facilitate cost analysis and cost-effectiveness analysis in education. -
Belfield, C., Hollands, F., & Levin, H. (2011). Providing comprehensive educational opportunity to low income students: What are the social and economic returns? New York, NY: Campaign for Educational Equity, Teachers College, Columbia University.
Hollands, F. M., Bowden, A. B., Levin, H. M., Belfield, C. R., Cheng, H., Shand, R., Pan, Y., & Hanisch-Cerda, B. (2014). Cost-effectiveness analysis in practice: interventions to improve high school completion. Educational Evaluation and Policy Analysis, 36(3), 307-326.
Hollands, F. M., Kieffer, M. J., Shand, R., Pan, Y., Cheng, H., & Levin, H.M. (2016). Cost-effectiveness analysis of early reading programs: a demonstration with recommendations for future research. Journal of Research on Educational Effectiveness, 9(1), 30-53.
Hollands, F. M., Leach, S. M., Stone, E., Head, L., Wang, Y., Shand, R. L., Yan, B., Dossett, D.H., Chang, F., Chang, Y., & Pan, Y. (2021). Costs and Effects of School Nursing on Student Attendance and Absenteeism. Poster presented at the Annual Meeting of the AERA, Online.
In addition to understanding whether educational programs and practices are effective at improving student and teacher outcomes, it is critical for education decision-makers and policymakers to understand the resources--personnel, materials, equipment, facilities, and other inputs--required to implement them with fidelity. It’s also important to assess benefits relative to costs to determine whether investing in specific activities provides a reasonable return on investment. EdResearcher conducts economic evaluations of educational programs either as standalone studies or as part of a team executing an efficacy trial. We also provide technical assistance, tools, and training in this area.
Decision-making about educational programs and investing education funding
-
Yan, B. & Hollands, F. (2018). To fund or to defund: Making the hard decisions. School Business Affairs, 84(8), 11-13.
Hollands, F., & Kushner, A. (2021, August 23). How to choose tools and programs that are worth the investment. Kappan.
Hollands, F. & Holmes, V. (2022, March 9). In the age of evidence-based decision-making, where can education decision makers really turn for evidence? Evidence & Policy Blog.
Hollands, Fiona, & Dossett, Dena. (2021, March 9). Use of evidence to inform school and district budget decisions. Research 4 Schools.
-
Hollands, F., Pan, Y. & Escueta, M. (2019). What is the potential for applying cost-utility analysis to facilitate evidence-based decision-making in schools? Educational Researcher, 48(5), 287-295.
Hollands, F. M., Pan, Y., Kieffer, M. J., Holmes, V. R., Wang, Y., Escueta, M., Head, L., & Muroga, A. (2021). Comparing evidence on the effectiveness of reading resources from expert ratings, practitioner judgements, and research repositories. Evidence & Policy. https://doi.org/10.1332/174426421X16366418828079
Chang, Y., Hollands, F. M., Holmes, V. R., Shand, R., Evans, P., Blodgett, R., Wang, Y. & Head, L. (2022). Does ESSA assure the use of evidence-based educational practices? Educational Policy. https://journals.sagepub.com/doi/10.1177/08959048221127989
Hollands, F. M., Shand, R., Yan, B., Leach, S. M., Dossett, D., Chang, F., & Pan, Y. (2022). A comparison of three methods for providing local evidence to inform school and district budget decisions. Leadership and Policy in Schools. https://doi.org/10.1080/15700763.2022.2131581
-
Hollands, F., Yan, B., Leach, S., Shand, R., Dossett, D., Chang F., Pan, Y., Wang, Y. & Head, L. (2020). LEA use of evidence in budget decisions. Poster session at the Annual Meeting of the SREE, Online.
-
DecisionMaker®
Software to facilitate school, district, or state education agency decisions that involve a choice between alternative educational activities, strategies, and interventions.Evidence-based Investment Calculator
A semi-automated Excel template to help education administrators determine what percentage of their school/district funding is spent on educational practices supported by research evidence that is correlational or more rigorous in design, and would therefore meet ESSA Tier III or higher evidence requirements.Master Evidence Repository
A database summarizing the research evidence available for a number of widely available, school-based programs that are designed to address student academic and other needs.Make Better Decisions Together: A Ten-Step Framework for Education Leaders
An online, 5-week, non-credit course for education leaders at the school, school board, district, and state level on applying a cost-utility framework to make evidence-based decisions that engage stakeholders and build buy-in. Accompanying one-on-one or one-on-team professional coaching available for a fee.
Decisions about implementing, tweaking, expanding, or eliminating educational programs and practices are influenced by many factors beyond costs and effectiveness at improving student or teacher outcomes. Other considerations may include: feasibility of implementation; burden on school staff; fit with the education agency’s mission; alignment with local values; and the interests of school board members, parents, and other community members. EdResearcher studies the realities of decision-making in K-12 schools, school districts, state education agencies, and institutions of higher education. We also develop tools, resources, and training to help decision-makers juggle multiple considerations.
Evaluating District and School Programs
-
Hollands, F., Chang, Y., & Holmes, V. (2021, June 9). What does it mean to call a program ‘evidence-based’ anyway? Education Week.
Hollands, F. (2016, May 9). Is competency-based education worth the investment? EdSurge.
-
Hollands, F. M. & Pan, Y. (2018). Evaluating digital math tools in the field. Middle Grades Review 4(1), Article 8.
Shand, R., Leach, S., Hollands, F., Chang, F., Pan, Y., Yan, B., Dossett, D., Nayyer-Qureshi, S., Wang, Y., & Head, L. (2022). Program value-added: A feasible method for providing evidence on the effectiveness of multiple programs implemented simultaneously in schools. American Journal of Evaluation. https://doi.org/10.1177/10982140211071017
Leach, S. M., Hollands, F. M., Stone, E., Shand, R., Head, L., Wang, Y., Yan, B., Dossett, D., Chang, F., Chang, Y., & Pan, Y. (2022). Costs and effects of school-based licensed practical nurses on elementary student attendance and chronic absenteeism. Prevention Science. https://doi.org/10.1007/s11121-022-01459-0
Hollands, F. M., Leach, S. M., Shand, R., Head, L., Wang, Y., Dossett, D., Chang, F., Yan, B., Martin, M., Pan, Y., Hensel, S. (2022). Restorative Practices: Using local evidence on costs and student outcomes to inform school district decisions about behavioral interventions. Journal of School Psychology 92,188-208. https://doi.org/10.1016/j.jsp.2022.03.007
-
Relevance and Credibility Indices
Rubrics to help decision-makers evaluate the relevance and credibility of existing studies of educational programs, strategies or interventions and determine how the findings might apply in their own context. Tutorials available at DecisionMaker®. (Register free in DecisionMaker® and click on Tutorials tab).Evaluation Criteria, Measures and Data to Collect
A list of criteria that can be used to evaluate educational programs and practices with suggested evaluation methods and data to collect. -
Developing a Digital Toolkit for Math.
Example of a decision-making process in a school seeking to identify online math tools to include in a digital toolkit to support math instruction.Selecting an SEL Curriculum.
Example of a decision-making process in a school that used the cost-utility decision-making framework to inform a decision about selecting a social-emotional learning curriculum.Selecting an Implementation Model for A Statewide Student Information System
Example of a state education agency applying the cost-utility framework to evaluate different ways of implementing a statewide student information system (SIS) to determine what would be affordable and also acceptable to districts:
Part I). Pros and cons of various SIS implementation models
Part II). Cost-utility Analysis of SIS Implementation Models
Part III). SIS Models Cost-utility Analysis
Evaluating educational programs and practices that are being implemented alongside many other school activities is challenging because they may complement, supplement, or even work against each other. Furthermore, even programs that are known to be effective must be implemented with fidelity to assure results. Strong theories of change and continuous improvement efforts are needed. EdResearcher collaborates with education agencies to evaluate and improve educational programs in context.
Educational technology and online learning
-
Hollands, F. & Holmes, V. (2023, June 27). How AI tutoring can reshape teachers’ days. Education Week.
Hollands, F. (2017, August 3). What role does research play in EdTech decision-making? WCET Frontiers.
Hollands, F., Griffin, A., & Saxberg, B. (2017, July 10). Ed Tech decision makers are under pressure in higher education. The 74.
Hollands, F. (2017, June 11). Educational technology vendors in higher ed: Friends or foes? EdSurge.
-
Hollands, F.M. (2011). Is online and blended learning cost-effective? 27th Annual Conference on Distance Teaching and Learning, Madison, WI.
-
Hollands, F. M. (2003). The impact of computer use on the individualization of students' learning experiences in public middle school science classrooms. (3071378) [Doctoral dissertation, Columbia University]. ProQuest
Hollands, F. M. (2012). Using cost-effectiveness analysis to evaluate School of One. Paper presented at Annual Meeting of the AERA, Vancouver, Canada.
Hollands, F. M. & Bakir, I. (2016). Automated detectors of learner engagement and affect: progress towards personalized learning. Proceedings of MIT Learning International Networks Consortium, 2016. Cambridge, MA, 122-131.
Hollands, F. M., & Escueta, M. (2017). EdTech decision-making in higher education. Teachers College, Columbia University. (ERIC Report ED 602951).
Hollands, F., & Escueta, M. (2019). How research informs educational technology decision-making in higher education: the role of external research versus internal research. Educational Technology Research and Development, 68(1), 163-180.
Hollands, F. (2023). Evaluation of Day of AI 2023: AI Literacy for K-12. EdResearcher. https://www.edresearcher.net/eval-day-of-ai
Twenty-five years ago, the President’s Committee of Advisors on Science and Technology (PCAST) stressed the need for research to identify ways in which the use of technology could strengthen education in the United States. Since then, EdResearcher has been studying the use of technology and its growing application to online learning in K-12 and in higher education. We help education agencies identify and evaluate digital tools and strategies to enhance or reimagine existing instructional approaches.
Alternative Credentials
-
March 2014: The Future of the Massive Open Online Course. TEDx Teachers College, New York, NY.
May 2018: Keynote speech at Open edX conference, Montreal, Canada: “MOOCs 2025.”
-
Hollands, F. (2017, December 20). The metamorphosis of MOOCs. EdSurge.
-
Hollands, F. & Tirthali, D. (2014). MOOCs: expectations and reality. Full report. Teachers College, Columbia University. ERIC report ED547237
Hollands, F. & Tirthali, D. (2014). Why do institutions offer MOOCs? Online Learning, 18(3), 1-19.
Hollands, F. & Tirthali, D. (2014). Resource requirements and costs of developing and delivering MOOCs. The International Review of Research in Open and Distance Learning, 15(5), 113-132.
Hollands, F. M. & Tirthali, D. (2015). MOOCs in Higher Education, Institutional Goals and Paths Forward. Palgrave Macmillan, NY. Selected as a CHOICE ‘Outstanding Academic Title’ for 2016.
Hollands, F. M. & Kazi, A. (2018). Benefits and costs of MOOC-based alternative credentials: 2017 - 2018 baseline survey results. Teachers College, Columbia University.
Hollands, F. M. & Kazi, A. (2019). Benefits and costs of MOOC-based alternative credentials: 2018-2019 results from end-of-program surveys. Teachers College, Columbia University.
Hollands, F., & Kazi, A. (2019). MOOC-based alternative credentials: What’s the value for the learner? EDUCAUSE Review.
Hollands, F. M., Kazi, A., Javier, K., & Ginsberg, Y. C. (2023). Benefits and costs of participation in MOOC-based alternative credentials: MicroMasters and Specializations. EdResearcher #2023-1. https://www.edresearcher.net/2023-1
Hollands, F. M., Ginsberg, Y. C., Kazi, A., & Javier, K. (2023). Employer support for employee participation in alternative credentials. EdResearcher #2023-2. https://www.edresearcher.net/2023-2
Kazi, A., Hollands, F.M., Javier, K., & Ginsberg, Y.C. (2023). Benefits of earning alternative credentials: one-year follow-up. EdResearcher #2023-3. https://www.edresearcher.net/2023-3
The arrival of Massive Open Online Courses (MOOCs) 10-15 years ago was accompanied by an explosion of non-degree credentials being offered by universities, companies, and even non-governmental organizations. While many observers pronounced the end of higher education as we know it, these credentials have largely supplemented rather than replaced traditional higher education programs. EdResearcher was among the first to systematically investigate the MOOC phenomenon and predict its trajectory.