Inter-observer Agreement

After 2 weeks of training the temporary RA, the two RAs were still not able to hit the minimal gold standard of 80% inter-observer agreement (IOA). IOA is when two observers observe one or more students simultaneously, and at the end of the session, their coding of the student’s behaviors are compared. If the IOA is above 80%, then this is an indicator that the data were collected reliably.

Since official data collection has started, I had no choice but to step foot into the classrooms and be the IOA data collector. That means my coding of the students’ behaviors must match that of my RA for at least 80% of the 10-second intervals over 30 minutes, across three classes. Eighty percent of the 540 intervals!!

Prior to this, I’ve worked with my RA over the past 2 weeks on clearly defining active engagement, passive engagement, and non-engagement for 3rd to 5th grade students. My RA has thrown at me all sorts of random description of classroom behavior, and we have tried very hard to set our guidelines on how to code those behaviors. However, it’s one thing to have on paper what are active engagement behaviors, but to see the active engagement in action is a different ball game.

Knowing what I was getting myself into, I didn’t sleep too well last night…

It’s one of those funny-when-you-look-back-at-it moments, the teacher announced, “I will give you 20-minutes to work on a concept map in your small groups,” and BOOM! the class explodes into a series of “undefined” behaviors. Eyes were roaming everywhere, not quite at the teacher nor the assigned materials. Quiet raised hand is like a waving hand while answering aloud. Thankfully we survived this classroom, barely scraping through with an inter-observer agreement of 80.9%.

Journal articles make research studies look so neat and tidy. Research in real life is messy.

Anyway, I think I can sleep well tonight.