|
?:abstract
|
-
Data collection monitoring plays a key role in containing the total survey error and ensuring the data quality. Monitoring procedures based on a non-statistical approach include direct observation of the data collection process (especially in telephone surveys) and re-contacting respondents. However, the increasing use of computer-assisted survey instruments offers the opportunity to automatically record paradata and to apply monitoring procedures based on a statistical approach that allows for near real-time controls. To this end, a set of performance indicators can be defined and implemented to assess the adequacy and observance of the survey protocols and to uncover any problematic situations that need to be addressed quickly. A basic set of performance indicators generally includes outcome rates – calculated on the basis of survey dispositions (e.g., completed interviews, refusals, breakoffs, non-contacted units, ineligible units) – at regular time intervals during the field period. In addition to the basic set, ad-hoc indicators can be defined using other recorded paradata or response data to support the survey-specific monitoring goals and then assist in finding inefficiencies in data collection. Once indicators are defined, control charts can be used to display them. Control charts help balance cost and thoroughness of monitoring activities by using statistical principles to differentiate potentially problematic cases from those that vary naturally around a process average. In this way, survey managers are guided in making targeted interventions, without spending time exploring false alarms. The decision rules for when to intervene are based on empirical control limits that are derived from data on the process itself, rather than arbitrary specifications. The operational definition of each performance indicator and its level of measurement (e.g., categorical, continuous, or count) define the specific type of chart to be used. The strategy outlined above has several strengths, as it can be combined with non-statistical procedures to better optimize the selection of cases for direct observation or re-contact, making the monitoring more costefficient. Besides, this approach can uncover various forms of departure from the survey protocols (e.g., a glitch in the interviewing software, an intentional error in coding the answer to a question to avoid follow-up questions or other hard-to-identify falsifications), and thus compensate for the weaknesses of non-statistical monitoring procedures. This work focuses on the system of performance indicators and control charts developed for the EU-SILC (European Union Statistics on Income and Living Conditions) survey carried out in Italy. Some illustrative examples are given using data from the 2019 survey edition. The data were collected by an external company on behalf of the Italian National Statistical Institute in CAPI or CATI mode. The system of control charts for the EU-SILC survey is mainly aimed at understanding whether interviewers are working in compliance with the interviewing protocol or, if not, which actions have to be taken to improve their work. In defining the set of performance indicators, two kinds of aspects are considered: on one hand, the constraints dictated by both the collected information and the interview protocol, which were agreed upon with the external company; on the other hand, the effects of undesired sources of variability in the process data. The monitoring procedure is designed to be carried out at regular time intervals during the field period. Two sequential types of control charts allow the consistency of each performance indicator to be assessed: 1. among all interviewers; 2. at the interviewer level, for any interviewer with at least one point outside the control band in the previous step. Besides, particular emphasis is placed on monitoring the coding of textual variables, such as occupation, by developing an ad-hoc procedure to assess for each interviewer: - the presence of possible concentrations of classification codes; - the tendency to confirm previously assigned codes, with reference to the respondents involved in past editions of the survey. Finally, in addition to the charts, the monitoring procedure produces a tabular report listing the interviewers with at least an out-of-control event, along with the threshold values at which the outof-control event occurs. For the flagged interviewers, other information and statistics are also reported to help decide on the type of intervention to be implemented before the end of the field period (e.g., supplemental training for interviewers who systematically assign inappropriate codes).
(xsd:string)
|
![[This page as RDF]](https://data.gesis.org/gesiskg/static/rdf-icon.gif){kind=icon}