The following evaluation plan was written for a federal Department of Ed physical fitness grant…
The Promoting Wellness initiative will be evaluated quarterly to document its success in meeting its objectives and milestones and to assess its impact on the youth, families, and schools it serves. As discussed below, the evaluation is designed to ensure that: a) implementation will be monitored systematically and on an on-going basis; b) specific progress measures will be used to assess the quality and completeness of project activities; and c) specific progress measures will be aligned with the goals, targets and expected outcomes set forth in this application narrative so that progress towards achieving them can be accurately assessed.
SWA will contract with an external evaluator to ensure that an extensive evaluation is conducted on the Promoting Wellness project. The qualified third party evaluator will provide objective summative and formative evaluation services for the grant program. Services will likely include: monitoring, fidelity checks and audits, development of summative evaluation reports, development of the semi-annual progress report, and establishment of the formative evaluation system. The External Evaluator will collect and analyze all project data on a quarterly basis, reporting findings within one month of the close of each quarter. In this way, the Advisory Team and all project stakeholders (parents, community members, teachers, administration, partners) will have four opportunities annually to assess project success and make recommendations on any modifications that may be necessary. The Advisory Team will finalize all modification actions.
The evaluation will include both formative (process) and summative (impact) performance measures. Also, the evaluators will help project partners collect required data regarding GPRA measures. Formative evaluation methods will relate to the effectiveness of the project’s procedures, practices and activities in implementing the project and in meeting project milestones in conformance with the proposed timelines. A key purpose of the formative evaluation will be to collect, analyze, and disseminate data over the course of the project to help the project partners and staff to stay “on track” in implementing project activities and to promote ongoing project improvement. To this end, formative evaluation data will be shared with project partners and staff on an ongoing basis throughout the project. The key formative evaluation questions are: a) Are project partners carrying out project activities with fidelity to the proposed design and management plan—i.e. Has the project met its targets for implementing project activities related to each goal and objective? Has the project met its annual targets for staffing, training, coordinating delivery of counseling services, and engaging in parent outreach and related services? b) Has project staff used information effectively throughout the project—including formative evaluation data—for self-assessment and program improvement?; c) Has the project met its targets regarding implementation of specific initiatives and public dissemination of project results? d) Are the project’s procedures, policies, and management effectively supporting the project in accomplishing its proposed activities and meeting its milestones? e) Are project activities focused on achieving the objectives and conforming to timelines set forth in the project proposal? f) Do the policies and feedback mechanisms support ongoing review and program improvement? Formative evaluation methods include frequent and ongoing interviews of counselors, administrators, and other project stakeholders; structured observations of project activities and participation in counselor meetings; frequent reviews of project training schedules and records; frequent reviews of documents and procedures used to advertise the project, recruit staff, reach out to parents/community and communicate with students, teachers, staff, partners, parents and others; questionnaires and survey instruments administered several times during the course of each year; and reviews of interim and year-end reports prepared by project staff. Formative evaluation results will be included in Evaluator reports at least semi-annually. Formative evaluation results will also be shared with project partners and staff, in writing and/or verbally, on a frequent basis. Formative evaluation data will be included in evaluation and year-end reports to U.S. Department of Education. Summative evaluation methods will address project implementation and consequent changes in youth outcomes regarding student behavior and academic achievement objectives. Key summative evaluation questions are: a) Is the project achieving its objectives and performance targets? and b) What is the project’s impact on student behavior, attitudes and academic achievement, as well as on the behavior and attitudes in relevant areas of other stakeholders? Summative evaluation will examine the impact of the project on:
- Academic achievement, including analysis of grades and performance on state and local assessments. Data will be collected, disaggregated (i.e., students who have engaged in counseling and related services and students who have not), and assessed.
- Behavioral factors, including analysis of attendance, disciplinary referrals, and attrition. Data will be collected, disaggregated (i.e. students who have engaged in counseling and related services and students who have not), and assessed.
- Aspirational and future success-related factors, including analysis of vocational exploration assessments (i.e., comparison 7th grade pre-counseling responses to post-counseling high school responses to assess impact of counseling on students’ college and career plans).
- Summative evaluation reports will be submitted to the project partners, and summative evaluation data and results will be included in year-end reports to the U.S. DOE.
For each stated Project Wellness objective, process measures, data collection methods, tools, and timelines are detailed on the chart below:
Finally, through evaluator and Advisory Team efforts, the Promoting Wellness initiative will evaluate progress through mandated GPRA measurements, including:
GPRA 1: The percentage of grantees closing the gap between their student/mental health professional ratios and the student/mental health professional ratios recommended by the statute.
|Baseline 2008-2009 Data|
|Schools targeted in the grant||Students enrolled||# ofChild-adolescent psychiatrists||# ofSchool Psychologists||# ofSchool Counselor||#of School Social Workers||# of Other Qualified Mental Health Professionals|
|Southern Wilson Academy||100||0 FTE||0 FTE||0 FTE||0 FTE||0 FTE|
GPRA 2: The average number of referrals per grant site for disciplinary reasons in schools participating in the program.
|Baseline 2008-2009 Data|
|Schools participating||Number of students Enrolled||Number of referrals for disciplinary reasons|
|Southern Wilson Academy||100||270|