The Evaluation and Continuous Improvement (ECI) function is an important cornerstone of the Administrative Core. ECI supports evaluation of research and educational activities within the Irving Institute as a whole, and specifically for each of the Resources that comprise the Institute. From inception, our evaluation approach has focused on key framing questions, which have evolved over time to help guide our strategy and activities:
- Develop: Have we put in place the structures and mechanisms for meeting CTSA goals in an effective and timely way? Have we engaged the right mix of stakeholders in building these structures?
- Demonstrate: What services, strategies and tools are being provided by the CTSA and utilized by translational researchers? How effective are they in meeting the needs of our target audiences?
- Disseminate: What are the longer-term impacts of CTSA services and programs? To what extent are the innovations developed at the Irving Institute being disseminated to and utilized by other CTSA Hubs?
Broadly, our activities focus on:
- Needs assessment
- Service improvement
- Impact assessment
- Informative case studies
- Grant reporting
- NCATS Common Metrics Initiative and reporting
To implement evaluation across the Institute, ECI uses an adapted community-based participatory research model by engaging the administrative leadership across all Irving Institute Resources. Thus, our “module leadership-based participatory evaluation” approach involves formulating and implementing meaningful evaluation plans relevant to the aims and needs of the leaders directing each Resource. This approach of partnering with Resource leadership has direct impact on the decision-making process across all Irving Institute Resources and increases the likelihood that findings from evaluation efforts will be implemented by Irving Institute leaders to improve services to the clinical-translational research community. Tracking occurs at the Resource level, with assistance from the Biomedical Informatics Resource (BMIR) and the Irving Institute administration.
Our future plans include:
- 1. Implementation of Common Metrics Initiative: In partnership with Irving Institute Resources and more broadly at CUMC, ECI is taking the lead in implementing the NCATS Common Metrics Initiative, which uses a set of common metrics across all CTSA hubs. According to NCATS, “The Common Metrics Initiative will use a set of developed common metrics that will help to focus all activities, as a network and at the hubs, on making significant, measurable improvements in research translation and workforce development.” http://www.tuftsctsi.org/research-services/research-process-improvement/common-metrics-initiative/
- 2. Expansion of Existing Databases and Current Tracking Efforts: We will expand upon our CTSA Hub-specific metrics, indicators, and data collection and tracking methods by working with our Executive Leadership team and senior Irving Institute staff to refine specific goals, impacts, long-term and short-term outcomes, and metrics appropriate for assessing quality and continuous improvement. Data will be both qualitative and quantitative.
- 3. Assistance with Experimental Studies: We also intend to assist Resources in conducting experimental studies to test interventions that are hypothesized to improve service efficiency and effectiveness. Our experimentation in the past has been both formative and qualitative through the testing of alternative approaches in pilot studies and fellowship training programs.
- 4. Collection and Analysis of Qualitative Data: In addition to tracking systems and surveys to examine short-term quantitative data, we will continue to conduct qualitative in-depth interviews and analysis to gauge trainee and investigator perceptions of strengths and weaknesses of programs, services, activities, and associated outcomes, including the collection of “success case histories”.
- 5. Monitoring and Decision-Making: To achieve quality and continuous improvement, metrics will be collected within each Irving Institute Resource, with assistance from ECI staff as needed. Results will be provided to the Executive Leadership team on an ongoing basis and also be made available to stakeholder groups for their input. Once a year data will be provided to the External Advisory Board (EAB) for review and will be the basis for recommendations for improvement and dissemination.
The ECI team consists of:
|Harold A. Pincus, MD, Director, ECI
Co-Director, Irving Institute
Vice Chair for Strategic Initiatives, Department of Psychiatry,
Director of Quality and Outcomes Research,
New York-Presbyterian Hospital
|Boyd F. Richards, PhD, Co-Director, ECI
Director, Center for Education Research and Evaluation
Professor of Medical Education (in Pediatrics)
|Lynn Elinson, PhD
Consultant, Irving Institute
|Zainab Abedin, MPH
Senior Evaluation Specialist, Irving Institute
|Michelle McClave, MS
Associate Director for Programs and Communication, Irving Institute