Experiences from conducting semi-structured interviews

Notes: Experiences from conducting semi-structured interviews in empirical software engineering research

Introduction

• Interviewees feel comfortable
• Four areas deserve special attention when planning and coducting interviews:
  • Issues related to the effort necessary to accomplish the interview
  • The skills of the interviewer
  • Interaction between interviewer and interviewees
  • Tools and project artifacts

Background

• Individual interview and Group interview (Focus group)
• Structured interview (Specific objectives and questions) and Unstructured interviews (Few specific questions)
• Semi-structured interviews(Combination of both, focused interviews)
• Factors related to planning and conducting interviews:
  • collecting historical data
  • Eliciting opinions and impressions
  • Identifying terminology
  • Clarifying things that happened during the study in cases where interviews are used in combination with observations
• Different ways to record:
  • Audiotape (Tape recorder)
  • Scribe
• Interviewer expert of the interview topic
• Interviewer skills:
  • Nonjudgemental and sesitive
  • Letting people talk
  • Paying attention and express questions clearly
• Approach sensitive questions:
  • Create an atmosphere of trust are suggested
  • Ensuring confidentiality
  • Phrasing questions in a non-threatening manner
  • Not express dismay or openly disagree with what the respondent says
• Different types of interview questions:
  • Experience questions
  • Openion questions
  • Feeling questions
  • Avoid "why"-questions
  • Avoid "yes" or "no" answer questions
• Different elicitation techniques:
  • Visual elicitation (pictures and drawings etc.)

Experience collection

• Table 1. Overview of studies

Interviewing experiences

• Necessary Effort
  • Understanding amount of effort an interview study requires in planning phase for allocating resources and schedualing
  • In what follows, the different activities related to research interviews are described
  • Whether it is better to have one or two interviewers is also discussed
• Activities
  • Time estimates are given on two of the activities:
    • Summary writing and transcriptions
  • Impossible to give time estimates for all the activities because of differences between studies.
  • Realistic plans and estimates can be made
  • Activities that are required:
    • Schedualing (make appointments)
    • Collecting of background information (read project documentation, collect information from personal CVs, etc.)
    • Preparing interview guides (time required for preparing interview guides varied substantially from study to study, serveral researchers making the guide together or needing to adapt guides to each subject)
    • Discussions/meetings (several researchers involved, necessary to spend time on meetings and discussions before, after or between the interview)
    • Summary writing (takes three to four hours to summarize one of audio taped interview)
    • Transcribing (takes eight hours to transcribe one hour of audio taped material)
  • Efforts in relation to analysis must also be taken in to account
    • Analysis is time consuming in nature
    • Will exceed the time spent on the acitivites listed above by several times
• One or two interviewers
  • Some situations two interviewers beneficial
    • Interviewees talk more with two interviers
    • More follow-up questions
    • More information will be collected
    • Dividing responsibilities
    • Challenging to listen closely to what is said and at the same time plan the next question
    • Opportunity to discuss and verify their interpretation of the interview
    • Understand the subject correctly
  • Disadvantages of two interviewers:
    • Workload doubled with some activities (planning)
• Qualifications
  • Comprehensive knowledge of the software engineering area uder investigation
  • Good interviewing skills
    • Encouraging the interviewees to talk freely
    • Asking relevant and insightful questions
    • Following up and exploring interesting topics
  • Interviewer must master human interaction and have excellent conversatinal skills.
  • Open the interview by re-explaining confidentiality and anonymity
  • Explain the purpose of the interview and intentions
  • Ask for permission in case the interview being audio recorded
  • Create comfortable atmosphere in which useful information can be eclicited
  • Should be nonjudgemental and sensitive
  • Let people talk
  • pay attention 
  • Express him-or herself clearly
  •  
  • Be gentle, open and sesitive towards the interviewees but at the same time not take everything said at face value and bear in mind what a subject has said earlier in the interview
  • Clarify issues during interview
  • Interviewer should enjoy the interview, or at least give the appearance of doing so
  • These skills are developed mainly through practice
  • Pilot interviews and role plays should be conducted to train the interviewer
  • Use a tidy and well-organised process (schedule appointments early and be flexible to changes
  • Gain the trust of the subject (ensure confidentiality, explain research motives, describe how data will be applied and in which paper going to be used)
  • Be courteous at all times (thank the subjects for their contribution)
  • Be will prepared (bring slides to the interview with the interviewee's name on, your research goals, a short presentation of the research institution you represent etc.)
  • Take care with your appearance and make a good first impression (two strategies)
    • Dressing up to make a formal and seious impression
    • Dressing down to appear less threatening
  • Dress at approximately the same level of smartness and formality as the interviews
  • If the interview is coducted in the subjects' work place, it is beneficial to have an idea of the dress code in the company and adapt to it
  • Allow the interviewees to view the questions in advance, so they can prepare for the interview
  • Talk informally with the interviewees before or after the interviews to facilitate a friendly and relaxed atmosphere
  • Use humour (contribute positively to the interview, laughing together can create a more relaxed and open climate, humour and bonhomie must not be used excessively or in inappropreate situations, because it is important to project an image of gravitas and seriousness
  • Be active and hsow interest by nodding, paying attention and asking follow-up questions
  • Be careful not to argue or question the answers you get, interview may become defensive and lose respect for you
• Qualitfications in software engineering
  • Extensive knowledge of the theme of the interview to know which topics are important to follow up
  • Different view regarding the knowledge of the interviewer, claimed that it is easier for researchers to study areas in which they have little or no expreience, because familiarity with a topic migh cause them to overlook details about which they assume they are already infromed, researchers familiar with the topics of study may give the informants the impression that they already know the answers and consequently, the informants may feel that the researchers are testing them
  • Extensive knowledge of the interviewer about theme of the interview will be able to understand:
    
    • Information that the interviewees give
    • Ask the relevant follow up questions
    • Be able to clarify ambiguities
    • Control the interview
  • Difficult for an interviewer with limited knowledge to understand:
    • What is important and to follow up interesting and relevant topics further
    • quality of the interview depends to an overwhelming extent on the interviewee
• Interaction issues
  • How to handle a subject that either says the barest minimum or strays from the question are outlined
  • Silent interviewees (barely talk, answers are short, unwilling to elaborate)
    • Fear the consequences of talking too much
    • See no potential benefit in participating in the interview and therefore no point in cooperation
    • Can be very difficult to obtain information from them
  • Strategies to cope with subjects that say the barest minimum:
    • Asking questions that cannot be answered only with "yes" or "no"
    • Feign ignorance and ask for details that already well known  to the interviewer
    • Important not to be perceived as being too much of an expert
    • Ensure that interviewees understand that there no "right" answers, because software developers tend to believe that anyone coming to interview them is really there to evaluate them
    • Two different types of silent interviewee:
      • Uninterested ones (mangers told them to participate and consequently they are not motivated to proffer information, do not believe that the interview is relevant to what they are doign, 
      • Shy ones (technically orientated)
    • Regarding unresponsive interviewees:
      • Motivate them in advance
      • Contact them directly and tell them about your goals and intentions
      • Have two interviewers 
      • Understand why interviewees feel this way and try to adapt the questions so that they become more relevant to them.
    • With shy interviewees:
      • Different approach interviewees feel comfortable
      • Warm them up by encouraging them to talk about issues that they were really interested in, such as technical issues, it will make them talk about relevant topics later in the interview
      • Consciously use general interviewing skills, talking informally, not appearing threatening, create a comfortable climate for the shy interviewees
  • Interviewees that talk too much (talk too much about irrelevant topics)
    • Interview time is valuable should not be wasted
    • Emphasized stopping highly verbal respondant who goes off track
    • Discourage talking (stopping nodding the head, interjecting a new question as soon as the respondent pauses for breath and stopping taking notes)
    • Interrupt (although feel awkward and impolite)
    • Letting them finish and then change the topic
    • Difficult to know whether or not the information will be found to be useful during the analysis
    • Cutting the subject off too soon might result in the loss of relevant information
  • Interview questions (Six different types of questions)
    
    • Behaviour/experience questions (elicit descriptions of experiences, behaviour and actions)
    • Opinion/value (investigate what people think about certian issues)
    • Feeling questions (aim to understand emotional responses of people to their experiences and thoughts
    • Knowledge quesitons (identify what factual information the respondent has)
    • Sensory quesitons (capture the experiences of the senses)
    • Background/demographic quesitons (identify the characteristics of the person being interviewed
  • All of these questions can be asked in the present, past or future tense
  • Ask "what" and "how" questions
  • Avoid "why" and questions with "yes" and "no" answers
  • Different questions:
    • Describe how they work: 
      • Can you describe what you have been working with?
      • Can you explain how you tested the solution?
    • Refelexive:
      • What could have been done differently in the project?
      • What was most challenging for you as developer/manager in this project?
    • Very detailed:
      • How many lines of code have you written in your career?
    • Series of questions(risky)
  • Question techniques: (Probes, prompts)
    • Anything more?
    • Ask informal questions at the beginning (create relaxed atmosphere)
    • Ask very specific questions at the beginning (Can you  describe how you worked with this task?) follow-up questions, (is this how you normally work?)
    • Start with general questions and folow up with more specific, to ensure that important issues are covered
  • Sensitive questions
    • Issues require special attention
    • Might be uncomfortable with certain questions, hence unwilling to discuss certain topics, hold back information, not be completely honest
    • Avoid situations which subjects feel intrusive, uncomfortable, feel offended and refuse to participate, privacy invade
    • Handle such issues with care, create an atmosphere of trust and ensure confidentiality, should be asked late in the interview, when subject is sympathetic to the interview
    • Consider phrasing of question, choose words carefully, phrase in general manner to avoid feeling personal exposure, use "we", "us", "they", and "them" can ensure that the researcher is on the same side as the respondent
    • Normalizing perceived deviance (not express dismay or openly disagree with whtat the respondent says, the interviewees do not feel judged and an open atmosphere is created
    • Potentially sensitive in software engineering studies:
      • Issues related to the economy
      • Opinions about colleagues and customers
      • Explanations of why things went wrong
      • Questions related to the interviewee's own competence and mistakes
    • Late interview, ensure confidentiality, phrasing questions in polite and respectful manner, avoid most sensitive areas and refraining from expressing dismay at what the respondent says.
    • Questions in software engineering are more impersonal and less emotional than in other disciplines (psychology, socialogy)
  • Tools and artifacts (audio taping, illustrates)
    • Tape recorder (video taping, audio taping and note taking)
    • Focus on conversation, note taking, eye contact, avoid distractions, skills, recording equipements smooth the proces of transcribing or making summaries, 
    • Visual artifacts, different elicitation techniques to encourage the informants to reveal what they know, feel, think, or believe, photographs, drawings, artifacts or items can be used, projective aids or devices, project documents, UML diagrams, code excerpts, or screenshots
      • easier for interviewees to remember what they had been working with
      • easier for interviewers to ask good follow-up questions related to the artifacts
      • interviewees talked a great deal and provided rich and informative information

Reporting from interviews

• Quality of the data obtained from semi-structured interviews depends on the planning and conducting of the interviews
• Minimum information should be described in studies reporting research interviews:
  • The interviewees should be described in terms of number of interviewees, how they were selected (e.g. their roles in the project or company under study) and how were recruited
  • The interviews should be described in terms of number of interviews (possibly number of interviews with each interviewee) as well as duration and location of the interviews
  • The number of interviewers and their roles if several
  • The interview guide(s) that were used during the interviews, as well as other tools or project artifacts

Conclusions and future work

• Semi-structured interviews frequently used as data collection technique in software engineering field
• Involve high costs, and quality related to how coducted
• Challenging to ensure that the interviewees experience the interview in a positive way
• Advice on planning and coducting interviews are useful
• Collect, systematize and share experiences with interviews within the field of software engineering in order to increase  the probability of collecting measures of high quality
• Four main areas are central when planning and coducting from 280 interviews:
  • estimating the effort
  • ensuring that interviewer has the required skills
  • ensuring good interaction between interviewer and interviewees
  • using the appropriate tools and project artifacts
• Devloping interview guidelines, scheduling of interviews, trascribing interviews
• Advantages and disadvantages of having one or two interviewers
• Advice how to improve interview skills is provided, and argued that it is very important that the interviewer is knowledgable in the field
• Experiences of dealing with difficult interviewees are also provided
• Experiences with different types of questions are described
• Be cautious with sensitive questions 
• Audiotape is preferable to avoid loss of information
• Project artifacts (UML diagrams, code and other visual items)
• Information about the interviews that should be included when reporting studies
• Investigate how the conduct of interviews affects the quality of the collected data within the field of software engineering
• How to best combine interview data with other quantitative and qualitiative data sources

 Reference:

https://ieeexplore.ieee.org/document/1509301