[Air-l] Opinions about Web Survey Software

[Ulf-Dietrich Reips]

Hi Charlie Balch, all,
sorry for the delayed reply. Was away for a few days...
Congratulations to keeping your software product open and in development!
Below some thoughts on a few issues.

At 9:44 Uhr -0500 5.7.2006, Charlie Balch wrote:
>
>Thanks for giving me some additional direction for my research. My reading
>of the web survey literature suggested that order effects were best
>controled by leaving the order fixed for all participants and accepting that
>there is an order bias that can be controlled for.

It can only be controlled for by varying order in 
"orderly ways" (how else would you know if and 
how much your results would be biased by order?). 
Essentially, there are two ways of doing this: 
balancing and randomization. For example, two 
items can be ordered in two ways, A first or B 
first. By balancing these orders, i.e. making 
sure that half of the participants get order AB 
and the other half BA, we effectively cancel out 
any order effects. (Of course, this method has 
limits: some contents can only be displayed in a 
particular order or there may be so-called 
asymmetrical transfer.) Randomizing the order of 
items spares us from creating all possible orders 
and has about the same effect (with a certain 
remaining low probability for influences of 
order).

>I'm going to look some more into order bias and methodologies to control for
>it.  One of the reasons that I wrote my own survey system was that I wanted
>to be able to respond to best practices and explore "new" possibilities such
>as my image item types (I know they have been around on paper but I haven't
>seen them on web surveys).  It would be trivial to add the code to randomize
>order of items and/or item groups. I could record both the response and the
>order of the items responded to but it seems easier to me to just accept the
>bias -- my head hurts when I think about the size of output for those
>exponential possibilities. Very few would have the skill to properly analyze
>it. This brings up another concern of mine.

In both cases, balancing and randomization, you 
wouldn't need to analyze it, because you 
eliminated order effects by design. Recording the 
actual orders would make sense, though, in case 
there are unexpected patterns in the results.

>How much should the software protect the user from stupid mistakes? We agree
>that there are at least some occasions where order does matter. Every little
>bit of software feature creap creates a more cumbersome interface and the
>possibility that the option will be used incorrectly.

I completely agree and see this as a general problem with many softwares.

>I suspect that you
>would agree that reseachers are reluctant to learn complex interfaces. I
>shudder to think of giving lots of options that might be randomlly selected
>without thought of the consequences.

Personally, I believe these decisions should be 
based on knowledge about the intended audience, 
usability considerations, and in a good 
educational sense. Check out WEXTOR, our (Web) 
experiment generator, at 
http://psych-wextor.unizh.ch/wextor/en/ , where 
we have tried to achieve an optimal balance for a 
dual audience: researchers and students. WEXTOR 
automatically avoids many pitfalls naïve 
researchers step into on the Internet, and it 
implements techniques that have shown to be 
useful in Internet-based research.

>Once I get the dissertation behind me, I intend to rewrite BIRAT into a more
>portable environment such as PHP and MYSQL. Perhaps it should be another
>discussion but I was interested in your thoughts about free platforms.

Imho this is the way to go, as long as they ar 
useable. Proprietary software is always a risk.

>I
>also look forward to the day when all software is free. Even so, portability
>versus usability are a concern. My experience is that ASP and the associated
>Access DB are very common in academic servers and a "no brainer" to install.

Access has been notoriously vulnerable.

>PHP/MySQL applications often make up for their lack of cost and improved
>efficiency by their difficulty to install.  What good is a publically
>available application if most folks can't figure out how install it?

A good reason to run it on a Mac, where PHP comes 
pre-installed and MySQL is easily added (e.g. via 
MAMP).

>
>Of course BIRAT also uses other languages like HTML, CSS, Javascript and
>such as well.  We live in interesting programming times.
>
>Just in case the above has left you in less than serene, you might want to
>look at another of my free applications: http://serenesound.com. I've been
>amazed at how it has been adopted by the medical community.
>
>Finally, any chance you can send me the content of the papers/book you
>mention below?  I'd love to read them but am traveling and thus away from my
>school library.

Some of it is on 
http://www.psychologie.unizh.ch/sowi/team/reips/publikationsliste.html 
Let me know if you need any of the other chapters.
Best --u