Interview Protocol

Interview Protocol

Consent: Hello! Thank you for agreeing to be interviewed today. Today I will be asking you about some common physics problems introductory students are asked. Before I explain the procedure of the interview, do you consent to being video recorded for the duration of this interview?

Introduction: Thank you! We will be video recording though out the interview. I’m also going to ask you to use this pen during the interview. It has a microphone in it to better pick up your voice and it will record everything you write down. As I said, I will be asking you about some common introductory physics problems. I’m particularly interested in how your initial “gut” response compares to your final answer. Because of this, there will be a timed portion and untimed portion of the interview. During the time portion, I will give you 4 problems to answer. You will have 30 seconds to answer each problem. I will also ask you to rate how certain you are of your answer on a scale of 1 to 7: 1 being “guessing” and 7 being “absolutely certain”. After you have completed all 4 problems, I will then ask you about each problem individually. During this portion of the interview, you will have as much time as you like to work through the problems. I ask that you talk through your thought process, like a “think-a-loud”, during this portion of the interview. Do you have any questions before we begin? [If they do have questions, just keep following up with “Anything else?” until they say no.]

Timed Portion: Great! Let’s begin. As I said, I will be giving you 30 seconds to answer each problem and rate how certain you are of that answer on the scale below the problem. Once all 4 questions are done, we will continue to the untimed portion of the interview. I do not need you to think-aloud for this portion. Are you ready to begin? [Give the questions individually so the interviewee cannot move on to without completing the previous question. Plan for the interviewee to use all 30 seconds but if they finish early it’s okay to continue. Give both screening questions then both target questions.]

Questions (in order given)

Untimed Portion

[Before the untimed portion look at the confidence levels of each questions. If they are all within +-1 of each other, ask the questions in the order the interviewee answered them. If not, ask about questions in the order of rising confidence.]

First questions for all problems: Can you tell me what you think the question is asking in your own words?

Second question for all problems: Now that you have more time to think about the question. Can you spend some more time thinking about the question? Please think aloud. It’s okay to change your answer if you want just to talk through your thought process. [All other questions are specific to the given problem.]

Magnet Screening-Target

Screening:

  1. Can you explain your reasoning for your (initial) answer?
    1. [If the interviewee changed their answer:] Can you explain your reasoning for your final answer?
  2. [If they answer but don’t mention Newton’s 2nd law. If they do, skip to 2a-c.] What physics principles did you use to come to your answer?
    • [If they don’t mention what forces they consider for Newton’s 2nd law.] Can you talk about how you used Newton’s 2nd law? What forces were you thinking about?
    • [If they don’t mention how forces impact motion but mention them summing to zero.] You mentioned that “__________” [use interviewee words about forces summing to zero]. Why is that important? What would happen if it weren’t true?
    • [If they don’t mention how forces impact motion and don’t mention them summing to zero]. So, you mentioned all these forces. Can you talk about how they work?
  3. [If they say not enough information]. What information would you need to solve this problem?
    • [If they mention normal force] Well, the question says the table pushes up with 30 N. With that, could you solve the problem?
    • [If they mention weight] Well, the question says the block weight 50 N. With that, could you solve the problem?

Target:

  1. Can you explain your reasoning for your (initial) answer?
    1. [If the interviewee changed their answer:] Can you explain your reasoning for your final answer?
  2. [If they answer and don’t mention Newton’s 2nd law. If they do, skip to 2a-c.] What physics principles did you use to come to your answer?
    • [If they don’t mention what forces they consider for Newton’s 2nd law.] Can you talk about how you used Newton’s 2nd law? What forces were you thinking about?
    • [If they don’t mention how forces impact motion but mention them summing to zero.] You mentioned that “__________” [use interviewee words about forces summing to zero]. Why is that important? What would happen if it weren’t true?
    • [If they don’t mention how forces impact motion and don’t mention them summing to zero]. So, you mentioned all these forces. Can you talk about how they work?
  3. [After they’ve mentioned the friction force.] So, you’ve mentioned “________” [use the interviewee’s words about friction]. Would you mind explaining what you know about friction? How does it work?
    • [If they say friction points up.] Can you explain why you think friction points upward?
    • [If they say friction points down.] Can you explain why you think friction points downward? (Slap 1)
      • [If they don’t catch that friction should be pointing up.] Can you do Newton’s 2nd law explicitly for me? (Slap 2)
      • [If they don’t catch that friction should be pointing up.] If the magnet is at rest, what should the sum of the forces be? What would that mean for the friction? (Slap 3)
      • [If they don’t catch that friction should be pointing up.] Because the magnet is at rest, the sum of the forces should 0. What would this mean for the friction? (Slap 4)
      • [If they don’t catch that friction should be pointing up.] Is there any situation where the friction could be pointing up? (Slap 5)
        • [If they say yes.] Is it possible that the friction is pointing up in this situation?
        • [If they say no.] What if I told you the friction is pointing upward? How would you reconcile that with Newton’s 2nd law? (Slap 6)
  4. [If they say not enough information] What information would you need to solve this problem?
    • [If they mention weight]. Well the problem says the magnet weight 10 N. With that, could you answer the question?
    • [If they mention force of the hand] Well the problem says the hand pushes with 6 N. With that, could you answer the question?
    • [If they mention coefficient of friction] Is there any way to find the force of friction without the coefficient of friction?
      • [If they say no] If the magnet is going to stay still, what does that mean about the forces? (Slap 1)
      • [If they don’t get to Newton’s 2nd law] Could you use Newton’s 2nd law to solve this problem? (Slap 2) (Go to Newton’s law protocol 2b)

Incline Screening-Target

Screening:

  1. [Ask this for both the normal and friction force. Ask 1(a), then 2, then 1(a), then 3.] Can you explain your reasoning for your (initial) (first/second) answer?
    1. [If the interviewee changed their answer:] Can you explain your reasoning for your final (first/second) answer?
  2. [For the normal force question: If they don’t mention Newton’s 2nd law. If they do, skip to 2a-c.] What physics principles did you use to come to your answer?
    1. [If they don’t mention what forces they consider for Newton’s 2nd law.] Can you talk about how you used Newton’s 2nd law? What forces were you thinking about?
    2. [If they don’t mention how forces impact motion but mention them summing to zero in the y-direction.] You mentioned that “__________” [use interviewee words about forces summing to zero]. Why is that important? What would happen if it weren’t true?
    3. [If they don’t mention how forces impact motion and don’t mention them summing to zero in the y-direction]. So, you mentioned “_________” [use interviewee’s words about forces in y-direction]. Can you talk about how they work?
  3. [For the friction force questions: If they don’t mention the friction equation. If they do, skip to 3a-b.] What physics principles did you use to come to your answer?
    1. [If they say f1 > f2] Can you explain why the friction in Case 1 is greater than the friction in Case 2?
    2. [IF they say f1 = f2] Can you explain why the friction is the same in both cases? (Slap 1)
      1. [If they don’t catch that f1 > f2.] Can you compare the frictions explicitly for me? (Slap 2)
      2. [If they don’t catch that f1 > f2.] According to the equation you wrote, friction depends on the normal force. How does the normal force change from Case 1 to Case 2? How does this impact friction? (Slap 3)
      3. [If they don’t catch that f1 > f2.] The normal force decreases from Case 1 to Case 2. What does that mean for the friction forces? (Slap 4)
      4. [If they don’t catch that f1 > f2.] Is there any way for the friction forces to not be equal? (Slap 5)
        1. [If they say yes but focus on the surface.] Is there a way for the forces to be different without changing the box and surface?
        2. [If they say yes and focus on the normal force]. Is it possible that the friction forces are not equal in this situation?
        3. [If they say no.] What if I told you f1 > f2? How would you reconcile that with the friction equation? (Slap 6)
  4. [If they say not enough information] What would you need to answer this question?
    1. [If they mention normal force] Well, even without the numerical values is there a way you can compare the normal forces?
    2. [If they mention coefficient of friction] Would the coefficient of friction matter here?
      1. [If they say yes] So, it’s the same box on the same surface. Would the coefficient of friction change?

Target:

  1. Can you explain your reasoning for your (initial) answer?
    1. [If the interviewee changed their answer:] Can you explain your reasoning for your final answer?
  2. [If they don’t mention the friction equation. If they do, skip to 2a-c.] What physics principles did you use to come to your answer?
    1. [If they say decreases]. Can you explain why the friction decreases as the angle increases?
    2. [If they say remains the same]. Can you explain why the friction remains the same as the angle increases? (Slap 1)
      1. [If they don’t catch that friction should decrease.] Can you compare the friction explicitly for me? (Slap 2)
      2. [If they don’t catch that friction should decrease.] According to the equation you wrote friction depends on the normal force. How does the normal force change as the angle increases? (Slap 3)
      3. [If they don’t catch that friction should decrease.] The normal force is decreasing as the angle increases. What does that mean for the friction force? (Slap 4)
      4. [If they don’t catch that friction should decrease.] Is there a way for the friction to not remain the same?
        1. [If they say yes but focus on the surface.] Is there a way for the forces to be different without changing the block and surface?
        2. [If they say yes and focus on the normal force]. Is it possible that the friction force does not remain the same as the angle increases then?
        3. [If they say no.] What if I told you the friction decrease as the angle increases? How would you reconcile that with the friction equation? (Slap 6)
    3. [If they say increases.] Can you explain why the friction increases as the angle increases? (Slap 1)
      1. [If they don’t catch that friction should decrease.] Can you compare the friction explicitly for me? (Slap 2)
      2. [If they don’t catch that friction should decrease.] According to the equation you wrote friction depends on the normal force. How does the normal force change as the angle increases? (Slap 3)
      3. [If they don’t catch that friction should decrease.] The normal force is decreasing as the angle increases. What does that mean for the friction force? (Slap 4)
      4. [If they don’t catch that friction should decrease.] Is there a way for the friction to not increase as the angle increases?
        1. [If they say yes but focus on the surface.] Is there a way for the forces to be different without changing the block and surface?
        2. [If they say yes and focus on the normal force]. Is it possible that the friction force does not remain the same as the angle increases then?
        3. [If they say no.] What if I told you the friction decrease as the angle increases? How would you reconcile that with the friction equation? (Slap 6)
  3. [If they say not enough information] What would you need to answer this question?
    1. [If they mention normal force] Well, even without the numerical values is there a way you can compare the normal forces?
    2. [If they mention coefficient of friction] Would the coefficient of friction matter here?
      1. [If they say yes] So, it’s the same box on the same surface. Would the coefficient of friction change?
Advertisements

Interviews in DBER

I chose to read the two physics papers and the chemistry paper that used interviews. The two physics papers were about student self-efficacy, student views about the nature of science, and the change in these qualities after undergraduate research experience (Physics 1) and how students reason about physics problems and whether “expert” problem solving strategy actually exemplifies conceptual understanding (Physics 2), The chemistry paper was about student conceptual understanding of equilibrium and fundamental thermodynamics in terms of chemical reactions and the alternative conceptions students hold (Chemistry 1).

I think it was very serendipitous that these were the papers I chose because the way the interview data was analyzed and presented were different in each paper giving a nice range of examples. In Physics 1, the interviewed were semi-structured and did not seem to be originally intended for research purposes. I get this impression because the authors only interviewed one of two cohorts mentioned and do not explicitly probe for self-efficacy or student views about the nature of science. The authors decided to present their data in a “case study” style with a total of three students representing the total population. They used an emergent coding scheme and provided no statistics through out the paper; only quotes from students.

In contrast, Physics 2 appeared to be more structured than Physics 1 (though they still fall into the semi-structured category) as they developed their questions to specifically probe student reasoning of quantitative problems. They also went with a “case study-esce” style where they described two students in detail but then took it a step further and used the two student profiles to analyze the other students. The broke down two of the problems on the protocol and labeled each student as “Pat-like” or “Alex-like” (the pseudonyms for the two students) in their thinking. A chart describing the break down was the only numerical representation of the data. I believe they used a semi-emergent coding scheme (from video data, not transcripts) as the researchers were looking for specific things but did not predict what the reasoning patterns would look like. Physics 2 was also the only paper to mention a framework though it was used as an explanation of the students’ reasoning patterns not a predictor. They discussed symbolic forms which is when people combine a symbolic template ( =  + ) with a conceptual schema (an informal idea or meaning that can be represented in a mathematical equation or expression).

(As an aside, I love how Physics 2 provided a link to the interview protocol and the full transcripts of Alex and Pat.)

Chemistry 1 was the most structured of the interviews (they were structured interviews) and the most diverse in terms of their interviewees. Physics 1 and 2 interviewed undergraduate students from a single university while Chemistry 1 interviewed undergraduates and graduates from a total of four universities. This paper also had the most “numbers” as it provided a breakdown of the interviewed population as well as percentages of how many students showed the alternative conceptions they were searching for. These authors used a non-emergent coding as they started with a list of 25 alternative conceptions they were searching for (though they did expand this list to 30 after finding 5 more alternative conceptions).

Overall, I feel the claims the papers were trying to make were supported by the evidence the authors provided. Physics 1 made an argument that self-efficacy and views on the nature of science are related and presented three “case studies” where this seemed to be the case. Physics 2 wanted to document different reasoning patterns in students, used two students to develop their “reasoning schema”, and then applied it to the rest of the students. Chemistry 1 wanted to measure how common certain alternative conceptions were in physical chemistry students and did this by analyzing student responses during interviews with these alternative conceptions in mind.

Multiple Choice Survey

During this unit we discussed several different Multiple Choice surveys used in our various fields. I was most surprised by the lack of validation and purposeful design with the surveys in my field. Perhaps I should have expected it as the FMCE was created in the 1990’s but having no validation and even admitting to lack of face validity was surprising to me. I felt the other fields did a much better job of validating their instruments and reporting on that process.

I’m hoping when Alex comes in the next few weeks that he shows us how to use the jMetrik software or something else to do some of the tests everyone else’s papers discussed. I feel that software is much easier to use than coding in Excel.

Force and Motion Conceptual Evaluation (FMCE)

I chose the Force and Motion Conceptual Evaluation (FMCE) for the Physics Multiple Choice Survey. The FMCE is a 43-question survey designed by Ron Thornton and David Sokoloff to assess calculus and non-calculus introductory physics students’ thinking about motion and forces. Reading the original paper, I was surprised to see that the development of the survey was almost accidental.

Thornton and Sokoloff were interested in how to improve student Newtonian thinking through lab activities and lecture demonstrations. While developing different labs with new technology (as it was the 90’s) and new demonstrations, they also developed questions they believed targeted different aspects of motion such as force, it’s relationship to velocity and acceleration, Newton’s Laws, and motion graphs. These questions eventually became the FMCE. The first paper cited in the FMCE paper is one of the author’s original explorations of adding a new technology to a lab where Questions 14-21 and 40-43 make their first appearance as an assessment of the lab.

Questions 14 – 21 on the FMCE

When discussing validity in the paper, Thornton and Sokoloff do not reference any statistical tests and only make claims about “correlations”, statistical measures that indicates the extent to which two or more variables fluctuate together, between student responses on questions designed to target the same concepts (but do they target the same concepts?). The one connection to the CLASS paper I saw was that the authors of the FMCE implied they had interviews with physics experts and students about the interpretation of and answers to the questions on the survey. But these interviews showed that a few of the questions, like Question 6, were answered incorrectly by physics experts on their first attempts.

Questions 1 – 7 on the FMCE

I know there are more robust analysis of the FMCE (as I’ve read abstracts for presentations about them while attending conferences) so I think my next paper will have to be one of them. I can only imagine how many issues of bias or lack of validity and reliability there are with a survey that was developed ad hoc style.

Theoretical Frameworks

This week’s reading was about theoretical frameworks in DBER. Over the few years I have been in the PER community, I have at least heard of these frameworks (resources, transfer, misconception, p-prims, constructivism, etc.) and thought I had used some of them (transfer and grounded theory) in my research. However, over the past year I have started to question if I’ve ever used a theoretical framework while doing research. Specifically, the first two projects I worked on claimed to use a grounded theory framework where we let the data lead us to our conclusions. I’ve since learned that grounded theory is letting the data analysis build a theory which we did not do. The only person I currently know who is using grounded theory, by the second definition, would be Anne Alesandrini at the University of Washington. Mila and I thought a comparison of Anne’s approach to student reasoning and our approach could reveal interesting patterns that neither of our frameworks could reveal alone. (As I said in class, this proved to be a dead end.)            

I use Dual Process Theory (DPT) as the theoretical framework for my current research. This theory argues that there are two thought processes in human reasoning: the fast, intuitive process and the slow, analytical process.

DPT argues that when a person is presented with a situation the fast process immediately proposes a model to base a response to the situation on. At this point the slow process may or may not intervene to further analysis this model. If it does not intervene, the person’s response will be based on this first proposed model. If it does intervene, the fast process will either agree with the model, which means the response will be based on the model, or disagree with the model, which means the fast process will be asked to propose a new model. I’m currently interested in how the slow process decides to intervene and I believe it might have something to do with home the fast process communicates with the slow process (which is currently unknown).

            I also wanted to quickly touch on a point made in the “Mixed Methods for Education Research” section of the Bordner et al reading. There was once a view that qualitative and quantitative methodologies could not be used together. I really liked the quote that “qualitative and quantitative methods constitute alternative, but not mutually exclusive, strategies for research” (pg. 8). My current research plan employs both qualitative and quantitative methods with student response data and statistics (we’re currently talking with Alex from Chemistry to figure out which tests will be best for our survey data). Even when I first learned the difference between these methods, I never thought of them as exclusive, just different. I was surprised there was such a debate over which to use.

Jan 15 Reading

While this week’s reading covered topics ranging from forming research questions to major ethical dilemmas, most of them fall into the category of things I think I both think about too much and not enough. I can easily fall down the rabbit hole with these topics so let’s get into it and hopefully I won’t get lost in Tulgey Wood along the way.

            Beginning with the Craft of Research (CoR), we explored the importance of writing and the process of developing research questions. I was surprised by the authors’ “me vs. the community” theme while discussing the purpose of writing, e.g. “I risk losing my identity” (pg. 12) and “We wish we could tell you how to balance your belief in the worth of your project with the need to accommodate the demands of teachers and colleagues” (pg. 15). I’ve never viewed learning proper reporting techniques as a threat to my identity. Has anyone in the class felt this way at all? Would you mind sharing it with us?

            I enjoyed the formulaic process given in developing research questions part of the reading. Some of the steps seem obvious and second nature to me such as determining the larger context of your questions (pg. 38-40), turning positive questions into negative ones (pg. 40), and asking the biggest question “So what?” (pg. 43). For my own research, I am currently struggling with the idea of the “feeling of rightness (FOR)” and how it impacts my students’ reasoning patterns. My research uses the theoretical framework of Dual Process Theory (DPT) which argues that there are two thought processes involved with human reasoning. I am dissatisfied with this framework’s hand waviness around how and when Process 1 and Process 2 communicate with each other and how that effects student reasoning in physics. This is my larger context. My current theory is that a person’s FOR is how Process 1 and 2 communicate. But how is FOR different from “feeling of wrongness (FOW)”, certainty, or confidence? Are they functionally the same for the work I want to do? These are my negative questions. For the big “So what?”, these questions focus on a current hole in theoretical understanding of student reasoning. Understanding this gap can help instructors guide students into using different reasoning paths and processes. Did anyone else see themselves in the decision processes described in Chapter 3 or was it completely new to you?

            For Responsible Conduct of Research (RCoR), I was mostly interested in the conflict of interest and ethics discussions. On pg. 86, the authors discuss some of the realities of research funding: “A private sponsor might withdraw support from a project if it does not produce the ‘right’ results. Success in the stiff competition for research grants can rest on having the ‘right’ preliminary results…The pressure simply to survive, much less profit personally, can and does create conflicts of interest.” These conflicts of interests have caused confusion over public health issues with products like leaded gas and cigarettes, the resurgence of preventable diseases by creating distrust in vaccines, and created fake science “facts” like needing to drink 8 glasses of water a day. The medical ethics section is the part that makes me glad I chose physics over become a doctor who needs to make all these decisions. Does anyone else think about these conflicts of interests and ethics cases much?

The last paper by Schwartz I have read before and I tend to agree with him. I chose to go into Physics because it was the first course in my academic career that challenged me and “made me feel stupid”. I do not think I am the best physics graduate student by a long shot but I really like the challenge. Does anyone else feel that way about their discipline?