Lessons Learned Key Success Factors for Results Monitoring
Consider using the following client outcome measures and benchmarks, taken from ONESTEP’s Delivering value: Employment Indicators in Ontario’s Community Based Training Sector. These benchmarks were arrived at in 2004 using an Ontario-wide sample of 1,100 clients from ten CBET agencies…
Consider using the following client outcome measures and benchmarks, taken from ONESTEP’s Delivering value: Employment Indicators in Ontario’s Community Based Training Sector. These benchmarks were arrived at in 2004 using an Ontario-wide sample of 1,100 clients from ten CBET agencies.
Gather program monitoring results in relation to program targets and integrate this evaluation into a variety of levels including staff meetings, management meetings and Board meetings. Everyone should have a clear understanding of core measures on a monthly basis and staff should know these results, as appropriate. Utilize staff as active part in developing strategies to respond to issues identified.
Make results public and accessible.
Form a Quality Assurance committee or working group made up of program managers and staff. Taking this step helps reinforce the view that QA is an ongoing collaborative investment that is dedicated to continuous improvement. It is also a practical step because QA data gathering needs to be co-ordinated and findings need to be summarized. It might also be added that another body (which involves Board Members) has to decide what QA findings are communicated, how they are communicated, and to whom. The function of most QA committees or working groups is to guide implementation rather than strategy.
Advocate for funding of QA activities, and assign dedicated resources to accomplish QA goals.
Gather QA data for a period of at least 3 months to obtain baseline measures for your agency. Performance on each of the recommended standards can easily be determined using simple counts and percentages. No complex analyses need to be applied, and data can easily be entered on a spreadsheet for quick, ongoing tabulation.
Present your baseline findings and compare them to the recommended benchmarks. It is, of course, possible that one or more targets are not reached, in which case, it is important to find out why and develop a plan of action in response.
If you are truly not satisfied or unhappy with a particular result, don’t look for excuses, look for explanations. Examine your results holistically. Put the results in context – remembering local labour market conditions and the client base itself. If, on the other hand, the results point to program areas that admittedly need improvement, then that too is a positive outcome. Consider the following actions:
Integrate program evaluation into strategic planning. Program evaluation can be integrated into strategic planning in a number of ways. For example, program evaluation can shed light on how well an organization responds to challenge; how quickly it can solve a problem; how well its various program components work, and how well it team works as a whole. All of this information helps an agency assess how quickly and effectively it meets its community’s needs.
Look for different ways to evaluate beyond core results. Important to measure customer satisfaction that provides excellent feedback from clients. Feedback can then be incorporated into modifications to services.
Evaluate specific aspects of your program, in addition to its core outcomes.
o Meet with staff to brainstorm solutions, perhaps visit other agencies to learn how they deliver similar programs.
o Undertake a case file review. As clearly demonstrated in Delivering Value, many CBT clients are multi-barriered, so it may make sense to ascertain whether or not benchmark shortfalls are occurring primarily among the most highly barriered clients. Some clients are seriously challenged by their barriers and need your agency’s help in overcoming them – both in terms of concrete assistance as well as emotional support and advice. It is these clients, especially, that need the counselling in employment counselling.
o Reflect on organizational capacity: Does your agency have enough resources to meet this target? What training supports need to be put in place?
o Consider program re-design: are there shortcomings in program content or delivery?
o If the shortfall is in one or more of the employer-related standards, then it is critical to being contacting those employers who rated the program critically to find out why, and what you can do about it. This is essentially a job development function.
o To make program improvements, set goals, set time lines, and in doing so, set the pace and your expectations. To help set the pace (and set expectations), it is important for you to describe clearly what you would like to see the organization achieve with respect to improvement in a given time frame - the time frame chosen will be expressed in months – not weeks. If the goals are realistic, and if the time frame is right, then the process of working toward those goals will likely maintain its direction and momentum.
