Collecting primary data for impact evaluations can be both costly and time-intensive, especially in terms of planning, designing survey instruments, setting up field protocols, training enumerators, data collection, supervision, quality assurance, and handling field-level challenges. Evaluators and researchers often partner with survey firms to implement and manage field operations for data collection.
Depending upon the survey partner’s capacity, the experience of primary data collection can either be enriching or stressful for the research team. More importantly, the capacity of the survey partner has a direct bearing on the quality of data being collected. Certain measures can be adopted to facilitate smooth coordination and communication with the survey partner to ensure quality data.Based on our experience of implementing surveys in rural and urban parts of India, in this blog we share lessons and good practices that research teams can adopt when working with survey partners.
1. Detailed Terms of Reference (ToR)
While searching for a survey partner, having a detailed and well-defined ToR, with the call for applications, is essential as it allows the survey firms and research team to assess the eligibility and scope of work. The ToR should clearly define the context (including study area and sample, population), details of the intervention, scope of work (such as the training requirements, length of the survey, number of respondents to be interviewed, supervision procedures, software and hardware requirements, qualifications of key personnel, etc), key timelines, and deliverables. The ToR should also describe the research team’s expectations regarding quality assurance (such as audio audits, and back-checks) and reporting mechanisms to be adopted by the survey firm.
2. Conducting training of trainers and development of training manual
It is important for members of the research team to be closely engaged with the survey firm to develop training modules (especially enumerator instructions) to standardize instructions being given to the enumerators during training. Conducting a training of trainers, prior to the enumerator training, is useful in getting insights from senior field team members on various counts, including issues around translation, additional response options, interpretation of questions, etc. Wherever possible, having the research team participate in enumerator training is also recommended as it helps to get enumerator feedback, incorporate changes, and clarify their doubts as well. The enumerators need to understand the purpose of the research, and the meaning and objectives of each question. Also, those involved in designing the evaluation and developing the measurement will be best placed to transfer this knowledge.
3. Conducting in-house data quality checks
The research team should vet the survey firm’s quality assurance plans prior to data collection. The quality assurance plans should include in-field monitoring measures (spot checks, partial and full accompaniments), checks on data (mechanisms to conduct high-frequency checks), audio audits (if possible, to collect this data using the CAPI software) and back-checks. Apart from this, the team should plan to conduct an independent audit of the incoming data at regular intervals to corroborate the information being provided by the survey firm. This also helps to keep a close eye on the outcomes of interest. Resources available from J-PAL and DIME can be helpful for monitoring data quality.
4. Dedicated resources for in-field monitoring
While an audit of the incoming data by the research team is helpful in getting a broad picture of the inconsistencies in the responses, it is also important for the research team to dedicate some time to undertake in-field monitoring. This allows the team to gain a better understanding of enumerator performance and participants’ understanding of the survey. A field auditor from the research team is responsible for observing surveys in the field on a regular basis, checking if field protocols are being followed (for example, those pertaining to sampling and respondent selection), providing feedback to supervisors on the enumerator’s performance and to report deviations. It is essential to have a good understanding of the sample area and be proficient in the local language to conduct these checks.
5. Setting up regular check-ins
Regular check-ins between the research team and survey partner serve as useful platform to discuss field challenges and other issues that could affect timelines or data quality. These check-ins are also useful to discuss and track important parameters such as the enumerator’s productivity, the number of enumerators in the field, and their attendance. These parameters are essential in assessing whether the survey will be completed in time and, if required, identifying measures to increase productivity. The research team can conduct independent audits of the incoming data and field activities, and provide feedback to the survey firm during the check-ins. Similarly, the survey firm could also flag errors that come up through their quality assurance systems and propose resolutions for the same during these discussions.
6. Attaching deliverables to tranche payments
The contract with the survey firm should establish a clear set of deliverables, timelines and associated tranche payments. For example, in case a baseline study is being conducted, the second tranche payment can be disbursed once the training material is submitted and approved, and the CAPI form is developed. The contracts should also clearly lay out the course of action in case the survey firm is not able to submit a deliverable on time.
7. Audit of budgets and financial reporting
The quality of data will be affected if the survey is not adequately funded, or in-field operations are under-budgeted. As a first step, and ideally before the contract is finalized, the research team with support from their finance team can do a detailed review of the proposed budget by the survey partner to ensure that various activities are adequately budgeted and there are sufficient funds earmarked for in-field operations.
For more information, email at cjain@3ieimpact.org.
Very interesting.