HTSG did not have existing impact evaluation processes in mid 2014, at the start of the research. They had no funder or expert staff to act as a guide for performing evaluations, and no expertise, norms, technologies or techniques to migrate into evaluation work. However, they did express a need-to-know impact, in diverse ways. These needs, motivations or will to know impact preceded evaluation data capture and even projects themselves.
In May 2014, the author emailed a range of NGOs to enquire about the possibility of establishing collaborative research into how NGOs performed impact evaluations. HTSG emailed back in July expressing an interest. Khun, HTSG’s director, asked Susan his administrator, to respond to the offer of collaboration.
“I have been asked to clarify that, as of yet, HTSG does not have a full-fledged system for evaluating project impact. However, we are very interested in learning more about how impact evaluations are made at other NGOs, particularly ones whose goals are similar to those of HTSG; we also hope to eventually develop and implement our own impact evaluation strategies.” (Susan, HTSG administrator/intern)
Further email exchanges lead to a first field trip to HTSG in August 2014. During this visit the author and Khun shared early morning car rides together from the hotel to HTSG’s offices. After a few minutes of small talk on the first morning, Khun raised a pressing concern: the competitive funding environment HTSG were in.
“We are facing a challenging time for our future. Some of our programs may need to close, they are not funded for the long-term, but often for 1 or 2 years only. The funding environment is really difficult …” (Khun, HTSG Director)
During a meeting and coffee at HTSG’s office, Khun added more. There were growing numbers of Thai, regional and international philanthropic organisations open to funding bids, and he mentioned a foundation run by a prominent Thai retailer that HTSG could bid to for funds.
Author: “Surely, you could make funding bids to them?”
Khun: “Perhaps. … Yes, we can approach and submit applications, but they often have their own focus, or differing aims, and maybe we can’t always match them. We must think carefully before bidding … choose which ones we can have most success with. If we are rejected, it can mean we waste a lot of time in applying” (Khun, August 2014)
HTSG had multiple funders, including five that Susan identified: a US pharmaceutical multi- national with a charitable foundation in Thailand; three large INGO’s based in three countries;
and the UN. HTSG also had a local donation system with branded boxes placed around their home city, but this generated little income.
Bidding for funds implied nuanced strategies and considerations, gauging HTSG and funder agendas, and assigning staff time to writing bids. Applying for grants and designing future planned impacts and evaluation processes meant HTSG had to respond flexibly but strategically to diverse funder interests and standards.
Knowing impact involved writing narratives of planned evaluation processes and target impacts into bids and reports. Susan was given the task of writing the English language versions of HTSG’s funding applications and project reports. She was an intern from a prominent US Ivy League university, volunteering in Asia at HTSG for two years. HTSG had been taking advantage of this internship program since 2011 to boost their human resources and English communications capacity. Report-writing tasks required knowledge of various funder forms, templates and processes, financial reporting standards, collecting NGO certifications, appealing to funding agency interests, responding to specific funding streams, and matching funder priorities. Susan described this work as “kinda confusing!”.
“There are different application forms, different project templates, different funders, different evaluation forms, and reports, so it’s kinda confusing! There are clusters of reporting periods too, like December, July, half-year reports, annual reports y’know.” (Susan)
Khun, a development sector manager with over 20 years of experience, added that it was easier when bids and reports were carefully planned and staggered at different times of the year. Susan nodded her head, but stressed that it was still “really tough” to prepare reports, evaluations and bids to meet requirements. Tough or not, HTSG’s need-to-know impacts, and the communication of their aims, goals, outcomes and potential impacts was shaped by diverse and differing funder niche interests, development approaches, grant standards, templates, forms and digital application systems.
In August 2014, HTSG had no guide or overall strategy for knowing impacts, no systems or processes set up. The email excerpt below points to HTSG’s need to learn professional evaluation, particularly the need for quantitative measuring of outputs or outcomes. Their goal of a holistic system was self-lead, without a guide or expert staff.
“We do try to give estimates of the numbers of beneficiaries benefiting from each project, but as we mentioned to you when you visited, we still have not developed a system of measuring impact. We are very keen to come up with an impact evaluation system that will allow us to tell donors, ‘so and so many children have improved in this particular area, and we know this because...’. Hopefully we will be able to work on this with your kind help!” (Susan)
Without a dominant authority or guide, HTSG were learning but struggling to define their needs and learning requirements by themselves.
Even with funder blessings and positive previous evaluations, it was questionable whether knowing impact was sufficient for HTSG to successfully accept and run projects. This was illustrated when Khun described a case where an external evaluator had recommended a project extension and the funder had guaranteed finances. However, HTSG couldn’t accept the extension.
“Despite them [INGO donor] wanting to continue, it was difficult work, very taxing getting through the mountains, and we didn’t have the staff capacity … to do the participation, reporting and evaluating.” (Khun)
HTSG in this case knew their planned impacts and had both evaluator and funder agreements, but they needed to avoid risks and costs they couldn’t afford. As we shall see in the next section, knowing impact is not enough; there are also many organisational challenges to overcome. This section has illustrated the early-cycle need-to-know impact at HTSG, even before projects begin, and the diverse kinds of impact that HTSG needed to know. The diversity and uncertainty contrasts with more confident evaluators or NGOs harbouring more stable funding streams. Arguably, HTSG’s uncertain funding climate, and flexible but risky “needs-to-know” are more prevalent in the NGO sector than stable funder/NGO relations. HTSG had no dominant guides, limited resources, and didn’t know how to create the “full-fledged system” they desired. HTSG’s need-to-know impact was linked to their competitive funding environment, self-regulated, reactionary, and unstable.
Activities involved in HTSG’s multiple needs-to-know impact included:
2. communicating impact plans in funding applications;
3. matching specific bids to specific funder standards, niche interests and online systems; 4. strategically managing funding opportunities, project needs and NGO sustainability;
and
5. defining, imagining and building a “full-fledged” evaluation system to know impacts.