[Air-l] Opinions about Web Survey Software

Wed Jul 5 07:44:09 PDT 2006

Fascinating information Dr. Reips!

http://birat.net is a proof of concept project associated with my
dissertation on best practices in Internet research. While the system is
stable, effective, and in world-wide use, I do not consider it complete.
Given the changing nature of the net, BIRAT probably never will be complete.
I intend for BIRAT to evolve as possibilities evolve and persons like you
identify best practices.

Thanks for giving me some additional direction for my research. My reading
of the web survey literature suggested that order effects were best
controled by leaving the order fixed for all participants and accepting that
there is an order bias that can be controlled for. 

I'm going to look some more into order bias and methodologies to control for
it.  One of the reasons that I wrote my own survey system was that I wanted
to be able to respond to best practices and explore "new" possibilities such
as my image item types (I know they have been around on paper but I haven't
seen them on web surveys).  It would be trivial to add the code to randomize
order of items and/or item groups. I could record both the response and the
order of the items responded to but it seems easier to me to just accept the
bias -- my head hurts when I think about the size of output for those
exponential possibilities. Very few would have the skill to properly analyze
it. This brings up another concern of mine.

How much should the software protect the user from stupid mistakes? We agree
that there are at least some occasions where order does matter. Every little
bit of software feature creap creates a more cumbersome interface and the
possibility that the option will be used incorrectly. I suspect that you
would agree that reseachers are reluctant to learn complex interfaces. I
shudder to think of giving lots of options that might be randomlly selected
without thought of the consequences. 

Once I get the dissertation behind me, I intend to rewrite BIRAT into a more
portable environment such as PHP and MYSQL. Perhaps it should be another
discussion but I was interested in your thoughts about free platforms. I
also look forward to the day when all software is free. Even so, portability
versus usability are a concern. My experience is that ASP and the associated
Access DB are very common in academic servers and a "no brainer" to install.
PHP/MySQL applications often make up for their lack of cost and improved
efficiency by their difficulty to install.  What good is a publically
available application if most folks can't figure out how install it? 

Of course BIRAT also uses other languages like HTML, CSS, Javascript and
such as well.  We live in interesting programming times.

Just in case the above has left you in less than serene, you might want to
look at another of my free applications: http://serenesound.com. I've been
amazed at how it has been adopted by the medical community.

Finally, any chance you can send me the content of the papers/book you
mention below?  I'd love to read them but am traveling and thus away from my
school library.

Charlie Balch
LSU Doctoral Candidate
http://charlie.balch.org

-----Original Message-----
From: air-l-bounces at listserv.aoir.org
[mailto:air-l-bounces at listserv.aoir.org] On Behalf Of Ulf-Dietrich Reips
Sent: Wednesday, July 05, 2006 8:26 AM
To: air-l at listserv.aoir.org
Subject: Re: [Air-l] Any opinions about Opinio web survey software

Hi Charlie, all,

At 6:21 Uhr -0500 5.7.2006, Charlie Balch wrote:
>I would be very interested in seeing your results.

No problem, please just send me a reminder if you don't see our posting here
before the end of the year. Btw., the first presentation of the results will
be at the Web data collection workshop in Dubrovnik this fall: 
http://pdw2006.internet-research.info/

>I'm also interested in why you think item randomization is important.  
>I'm aware that there is some bias towards answering areas in web 
>surveys.  I'm also aware of the argument that any changes to a survey 
>at the participant level create different environments and thus make the
data questionable.

Admittedly, for some surveys and applications item randomization is not
important or even harmful (e.g. with validated measures in personality
research). However, there is a vast literature about order effects and
context effects that cleary indicates vulnerability of survey results to
fixed orders. The best solution to get rid of these problems is item
randomization.
As an illustration I would like to point you to a study two of my students
and I reported in 2001 in Dimensions of Internet Science
(http://www.psychologie.unizh.ch/sowi/reips/dis/). 
Changing the order of just 2 items made a difference of about 100 minutes
(!) in reported television consumption per week (an effect of context and
social desirability). Also, the order of groups of items influenced dropout
behavior and data quality in the Web experiment.

>By the way, http://birat.net is free including the source in ASP, but 
>only runs on Windows Servers, and does not provide item randomization.

Thank you for the pointer (I also saw your earlier post and took a look at
the system). A great initiative, but you may want to reconsider the platform
restriction and set of features. In particular, I am afraid (or rather I am
happy) the Internet will render most platform-dependent systems obsolete
within the foreseeable future for a number of reasons you'll find below in
an excerpt from a recent article.
So better switch strategies ;-)

Cheers, --u

P.S. I liked "Dissertation Hell" as the building specification in your sig
*grin*

Excerpt from
Reips, U.-D., & Lengler, R. (2005). The Web Experiment List: A Web service
for the recruitment of participants and archiving of Internet-based
experiments. Behavior Research Methods, 37, 287-292. 
http://homepage.mac.com/maculfy/filechute/BSC515.pdf

"A number of tools have been developed for Internetbased experimenting that
form a general framework of reference for the methodology. These tools can
be grouped into two general classes of "software": programs and Web
services. Programs follow the traditional format.
They need to be installed on a computer and run locally.
The working of the program depends on the computer's configuration, which
may vary considerably over time (as other software is installed) and from
user to user. Different types of operating systems may not allow a user to
install the software at all. Upgrades and updates may be necessary. However,
the user is in control of the service and independent of a connection to the
Internet. An example of a tool for Internet-based experimenting (in this
case for Web-based decision-making experiments) of the program type is
WebDIP (Schulte-Mecklenbeck & Neun, 2005).
Web services, on the other hand, run on a server that is connected to the
Internet. Users access it via a Web browser and can use it only while they
are connected to the Internet. Because the functionality of Web browsers is
less dependent on the operating system (sometimes they are even referred to
as being platform independent), all who access a Web service are likely to
see and experience almost the same interface (but see, e.g., Dillman &
Bowker, 2001, for browser-related problems in Internet-based research). Web
services spare the user from upgrading and updating, since this is done by
the Web service administrators at the server. Nothing is installed on the
user's computer, saving space and time." (p. 287)
--
PD Dr. Ulf-Dietrich Reips

	    President, Society for Computers in Psychology (http://scip.ws)
	    Editor, International Journal of
Internet Science (http://www.ijis.net)
                     Universität Zürich
	    Psychologisches Institut		 
                     Rämistr. 62
	    8001 Zürich, Switzerland

iScience portal (http://psych-iscience.unizh.ch/)
_______________________________________________
The air-l at listserv.aoir.org mailing list is provided by the Association of
Internet Researchers http://aoir.org Subscribe, change options or
unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org

Join the Association of Internet Researchers: 
http://www.aoir.org/