"The difference between science and the fuzzy subjects is that science requires reasoning, while those other subjects merely require scholarship."
~ Lazarus Long
~ Lazarus Long
Prepared for EDTEC 700, Using Observations to Measure Instructional Value, Spring 2007. 
This artifact includes the observational protocol, raw data, and post-observational conclusions.
Value the use of data as the starting point for professional work.
Here's Your Sign: A field study at the airport.
EDTEC 700 is a series of one-unit courses which collectively offer an exposure to the latest trends in the field of Educational Technology. This section concentrated on the use of non-invasive observations as a way to improve either instructional interventions or human performance. The first class session concentrated on observations taken in the real world, and the second class focused on the virtual world.
For the first assignment, I decided to investigate human performance at the local airport. My subject was the flight status board located in Terminal One, which serves both arriving and departing travelers.
This artifact provides a good look at what data can bring to performance improvement. Until you gather information about the real world, you can never know if your products will stand up to the rigors of your users. Limit conditions, null cases, and strange concatenations will normally not manifest themselves in a development environment, but only "in the wild". The other benefit of collecting data is to compare performance over time. Reactions to the airport sign depend upon the ebb and flow of passenger traffic, the time of day, even the type of lighting available. Without checking to see how a certain intervention is being put to use in the performance context, a designer can only guess at what techniques or interventions will be more effective. With data, the designer will know.
For this study, we were required to develop a collection plan completely from scratch. We therefore had to divine the possible data we might encounter and prepare for it accordingly. Still, several subjects manifested behaviors that were entirely unexpected. For example, while the collection protocol targeted airplane passengers, limousine drivers also used the sign, as did people there to collect family members. Some people viewed the sign multiple times. What were these people doing? Why did they keep coming back? What aspects of the sign were inadequate? Clearly, observations must be augmented by surveys, but observational data provides the foundation upon which to frame the survey questions.
I encountered several small challenges while building my observation protocol and data collection plan. Our instructions were to observe and “let the data talk to us” during post analysis. I realized this would require a wide-cast net to capture adequate dimensions, and planned accordingly. Some dimensions I selected were highly subjective, such as a subject’s demeanor. What constitutes a demeanor of “bored”, “agitated” or “energized”? I therefore modified my data collection plan to capture demeanor (for simplicity) based on behaviors (for objectivity). In this way, another evaluator could review the behaviors I’d recorded, and decide for themselves whether my interpretation of demeanor was justified.
I was able to stretch my technological capabilities with this exercise. Even with no real background in database programming, I was able to create a custom tool with about three hours of work. My collection instrument was a pocket PC. Standing around the airport working on an electronic device helped me blend better than using a clipboard. The device was also equipped with a camera, which helped capture context data.
The tool incorporated a robust collection scheme tied to an exportable database for later automated analysis and graphing in a business spreadsheet application. To proof the tool, I engaged in a short usability test. This is something I have been guilty of skipping in the past, but have learned to value through my experience with the EDTEC program. After the test, I revised my collection tool so that certain fields were filled in automatically. I also reordered basic demographic information to the top, which allowed me to effectively make two observations on a subject in the same amount of time.
This work taught me several things about my future role within the field of educational technology. I have always been excited by the possibilities afforded by handheld devices, but this was the first time I was able to reap the benefits of such a scheme in a practical setting. The experience was an unqualified success. As handheld devices become at once more powerful, programmable and ubiquitous, I’m convinced they’ll have a profound effect upon the landscape of technology-driven learning interventions. We have already seen how easily audio can be deployed across myriad contexts through podcasting. I expect we’ll see interactive handhelds used to collect assessment and evaluation data while they facilitate delivery of directed content. Data collection using handheld devices will be a prominent tool in my future work within the Coast Guard.
