Expertise and innovation

One of the attributes that best exemplifies our dedication to providing the highest level of evaluation services is the expertise of our team members. Anyone can undertake an evaluation, but doing an evaluation that makes sense, is comprehensive, and is usable (i.e., provides a clear path forward) requires knowledge, experience, and skill. In many cases, particularly with community-based projects which involve many different dynamics (culture, language, history, identity), it is unrealistic for an individual or even a pre-made team to possess the required understanding of the situation, to get the most out of the evaluation. This is where our project-tailored approach to building a team comes into play.

At Walling Consulting, our project teams are built specifically around the needs of each client and their projects. Our teams are comprised of a “core team” and a “expert support team”. The core teams are made up of our regular team members, all of whom are highly specialized in the various areas of evaluations and who provide the highest level of confidence and trust in the methods and results of the process. The expert support teams are built for each project with experts who have significant hands-on experience and who are still working in their fields (i.e., not full-time consultants). Their involvement is aimed at ensuring that the evaluation methodology and process is pertinent, comprehensive, and sound, from the start. The quality of an evaluation is directly dependent on the process and methodologies, and having a comprehensive understanding of the situation from the get-go can avoid many major issues (e.g., missing important evaluation questions, indicators, presence of biases, understanding of pertinent outputs).

Connectivity

We are a generational family-run organization that works in community, health, and social services evaluation and development, while being ourselves part of a cultural and linguistic minority community in Quebec. Connection to the communities we serve, fostered through listening, understanding, co-creating, and communicating, is a central aspect to how we offer our services. Given the nature of many of our projects, working with underrepresented and vulnerable groups often struggling with major personal and societal challenges, we owe it to our clients and their stakeholders to develop a strong bond and we encourage our consultants to draw from their lived experiences to do so.

Connectivity is not just developed through effective communication protocols, but is also fostered through openness, listening, and two-way dialogue. We do not want the consultation process, particularly when it comes to interviews, focus groups or any communication between our team and community-members, to be cold and impersonal. We do not want interviewees to simply view us as another individual they will meet once in their life, sat at the other end of the table or screen and prodding them for information on some of their most difficult struggles in life; or, worse yet, for them to feel like research subjects in a study where all they are is a datapoint.

At Walling Consulting, our team members are our greatest asset, and a large part of that is their history and background. A history and lived experience which they are free to draw from to connect with and comfort the participants of the evaluation process. This background and experience are something we take into consideration when crafting our teams, to ensure that our process is culturally and socially appropriate.

Adaptability

Evaluations are complex processes and new information and opportunities, errors, and changes in circumstances can always arise, making adaptability a necessary trait to ensure a smooth and valid process. As much as we prioritize proactivity in evaluation planning, design, and implementation, given the importance of having a strong foundation, reactivity is equally important. We reflect this in many of our evaluation methodologies, by recommending our clients go with both inductive and deductive evaluation methods. This approach allows us to measure what the clients initially want, but also opens the evaluation process to learn from itself and adapt itself to new information that arises at every stage.

Efficiency

The business structure of Walling Consulting is different than most other consulting firms, with our core team of evaluation experts (full-time) supplemented by a much wider team of associate experts in various fields (usually employed in their specific fields and called upon when a project requires it). Given that our evaluation experts are so knowledgeable and experienced in planning and undertaking evaluations, while being responsible for most of the work, we save a significant amount of (billable) time for our clients. This also applies for our expert support team, which are called on punctually and when they will have the most impact on the evaluation process, to maximize efficiency and minimize the costs. We are very cost and resource conscious and actively work to save as much time and money for our clients. Our team is at the optimal size to provide these benefits. The above benefits, in addition to clear and efficient communication (always the responsibility of the project lead), and lack of any administrative bloat or hassle, drastically reduces costs for our clients (and us) and increases the speed at which we can deliver projects. Based on our prior public contracting experiences, we estimate that generally, compared to similar sized and larger consulting firms, we offer our services for 20% to 30% less than our competition.

Trust and safety

Trust in our team and process, alongside safety of the work we do, are paramount. For an evaluation to be valid, people (clients, stakeholders, community) need to trust in our expertise and our process. An evaluation, regardless of its quality, falls apart and becomes useless the moment people start distrusting the team, the methodology or the results. How was the data collected? Why wasn’t this looked at? Why wasn’t I contacted? The results don’t show X? These are all common questions that, if mishandled, can lead people, organizations, and communities to very quickly dismiss the work that was done. It is also a very understandable reaction. We have all seen lackluster evaluations with questionable methodologies; what is important is for us as expert consultants is to ensure that we have the satisfactory answers to all these questions, that we don’t do meaningless evaluations or use meaningless metrics, and that we involve stakeholders and the community throughout the evaluation process.

Indeed, communication and education are amongst our most important tools. We do not expect clients and their stakeholders to be experts in evaluations (otherwise why hire us?), but it is our job to build their trust in and educate them on us, our methodology, and the results. This is prioritized through every step of the project, from the beginning to the end, and is done through various methods, depending on the nature of the project.

Generally, when planning an evaluation, we provide our clients with a clear explanation on the various options presented to them, the benefits and drawbacks, and what can be expected from each alternative. We take great care in following and documenting why each decision was made. At the end of the projects, when providing the final report, we include all this information in appendices, allowing interested parties to go through the entire evaluation process and validate for themselves. We also recommend forming stakeholder groups or steering committees to act as a source of feedback throughout the project, while being ourselves very present in the communities (organizing kick-off events, community consultations, etc.), if the project can benefit from it. We also generally hold open consultation periods for stakeholders and community members, usually about one month long, following the publication of the final documentation, during which people or organizations can contact us and ask any questions they have.

When working with indigenous and vulnerable communities, we strongly promote knowledge transfer throughout the consultation process. We have strict protocols in place to transfer all data, information, analyses, and results to these clients and purge them from our records, ensuring that the knowledge gained remains in the hands and under the control of these communities.

In the same vein, especially when working with sensitive data and human participants, safety is a necessary and primary consideration, and demonstrating how important it is to us also contributes to the feeling of trust. As such, our evaluation processes are always culturally appropriate and conscious, ethical, and transparent. Our team members are all trained and skilled in working with underrepresented and vulnerable groups, our evaluation methods are always fully transparent (What type of questions will we be asking? Why are we asking these questions? What will the data be used for? How can this impact me?), our data collection is strictly confidential (we will not share identifying information with the client or anyone else), we always seek informed consent, and we have zero tolerance for harassment or abuse from our team or participants in any of our processes.

Client ownership

Our final pillar, client ownership, is central to our evaluation strategy and ties together many of the above-mentioned principles. An often-overlooked aspect of evaluations is the inestimable value of fostering a sense of ownership over the process, not just for the client organization, but for their staff and community as well. Indeed, far too often, evaluations are undertaken by a consultant or teams set out to measure indicators (data, surveys, interviews), get surface level analysis, write a report, and then leave it in the hands of the client. Through our work, approximately 80% of which has been oriented on evaluation of community-based services over the past 15 years, we are more often than not informed during discovery and project planning about prior evaluations, strategic plans, logic models, etc. that had been developed and never used since. Indeed, most evaluations, plans, and models are completed and forgotten. Of those that are used, many only scratch the surface of the potential benefits that can be drawn from them. One of the primary reasons for this is lack of client and stakeholder ownership.

Indeed, when clients and stakeholders don’t feel as though they are a part of the development and undertaking of the evaluation process, it creates a distance between them, the process, and the results. How can we expect someone to be able to communicate and promote the work that is/has been done if they themselves don’t fully understand the process or are not engaged in it?

Generally, we have a few methods we prioritize to foster this sense of ownership. The first is the use of co-creative methods when designing and planning the evaluation. We want our clients, their stakeholders, and their communities to participate in, influence, and understand the evaluation process. This is done through collaborative methods and activities in which project scope and objectives can be defined, various evaluation methods are proposed, presented, tested, and our data analysis methods are presented in varying complexity to allow customers to understand what is being done. For report writing, our open consultation for draft final reports is a way to allow the voice of the community to be heard and integrated into the work. The co-creation steps are also generally a fantastic source of alternate data that can be collected.

Following co-creation, and linking back with connectivity and trust, having a strong communication with the various involved parties is extremely important. How can a customer be expected to have a sense of ownership over a process if we send out a survey, wait a month, write up a report, and then provide them with a technical report two months after the last discussion on the subject? Having a very active and feedback driven communication process, with frequent updates and summary reports being provided generally on a weekly basis, is something we recommend in every project. As mentioned earlier, we also take the needed time to educate our clients about the specifics of the evaluation process, working with them to grow their pride and confidence in the methods and the work being done. We also provide clients with the opportunity to undertake frequent evaluations of the work being done, allowing us to figure out what has worked and what could be improved, while also giving us the information necessary to be proactive instead of reactive.

For longer term and large-scale projects, we also highly recommend identifying client and stakeholder champions (within the client organization and stakeholder communities) that can be the voice of the project when we are gone. These champions, depending on their number, can also form a steering committee or focus group to ensure the posterity of the process.

Page updated

Google Sites

Report abuse