[Air-l] Opinions about Web Survey Software (order effects/WEXTOR/Access)

Tue Jul 25 06:38:29 PDT 2006

U.-D.
Sorry for my delayed reply - these are fascinating topics and I feel very
fortunate to have the conversation with one of the leading researchers in
this field. I have read all your papers that I could find. They were
brilliant! I'm pleased to note that I'd already cited some of them in my
dissertation and have added more as a result of my recent reading.

Order
I think you would agree that order can only be changed when the item options
are nominal. Even with that restriction, I'm still not agreeing that
randomizing the options is useful. As I understand it, there is no strong
pattern to pattern responses as some participants will tend to answer at the
top, others at the middle, and some at the bottom of the options when they
are motivated to complete the survey at all. I can see some benefit of
testing for such patterns and discarding responses that appear to be
following a pattern. I don't see that randomizing order helps to control for
such pattern responses. 

I also appreciate that participants will often respond to what they perceive
as the first "correct" answer. Randomizing the item order would certainly
change the results when a significant number of participants answer in this
way. ***Does a change in results indicate more accurate results?*** It seems
to me that an instrument that is susceptible to this sort of error is
inheritably neither reliable or valid. How can you trust your data if the
participants are not considering the items?

What do you think about randomizing the items? I don't like the idea as I
think commonality of experience is essential.

WEXTOR
My respect for the WEXTOR website! I created an account on WEXTOR and ran
through a sample experiment. WEXTOR is a wonderful tool for prototyping
experiments that I wish I'd know about when I took a few of my graduate
statistical courses. Perhaps BIRAT will be a companion software product one
day. Another interesting companion product would be branching system that
helped researchers select appropriate statistics. A few questions about the
target population and the type of data collected would make this relatively
easy. 
http://psych-wextor.unizh.ch/wextor/en/

Access
I'm not sure that I would call Access vulnerable any more than other
databases are to most common attacks such as SQL injection. I certainly
agree that Access is not as robust and bad code would allow Denial Of
Service attacks.  Fortunately the Access SQL logic and tables are easily
ported to other databases. I'm sure that it has a lot to do with my first
programming experiences but I find the ASP/Access combination easier for
rapid prototyping. I also find the resulting product capable of handling
reasonably robust demands. As noted before, I do intend to rewrite in
PHP/MySQL.

I'm just going to touch on the Macintosh/PC discussion. Macs are great and
overpriced. If you really want to use open source at the right price point,
run a 'nix flavor on PC hardware.  

Charlie Balch
Professor CIS Arizona Western College (New hire)
Doctoral Candidate LSU (Dissertation hell)
Nice Guy (Just ask my wife)
http://charlie.balch.org

-----Original Message-----
From: air-l-bounces at listserv.aoir.org
[mailto:air-l-bounces at listserv.aoir.org] On Behalf Of Ulf-Dietrich Reips
Sent: Tuesday, July 11, 2006 8:51 AM
To: air-l at listserv.aoir.org
Subject: Re: [Air-l] Opinions about Web Survey Software

Hi Charlie Balch, all,
sorry for the delayed reply. Was away for a few days...
Congratulations to keeping your software product open and in development!
Below some thoughts on a few issues.

At 9:44 Uhr -0500 5.7.2006, Charlie Balch wrote:
>
>Thanks for giving me some additional direction for my research. My 
>reading of the web survey literature suggested that order effects were 
>best controled by leaving the order fixed for all participants and 
>accepting that there is an order bias that can be controlled for.

It can only be controlled for by varying order in "orderly ways" (how else
would you know if and how much your results would be biased by order?). 
Essentially, there are two ways of doing this: 
balancing and randomization. For example, two items can be ordered in two
ways, A first or B first. By balancing these orders, i.e. making sure that
half of the participants get order AB and the other half BA, we effectively
cancel out any order effects. (Of course, this method has
limits: some contents can only be displayed in a particular order or there
may be so-called asymmetrical transfer.) Randomizing the order of items
spares us from creating all possible orders and has about the same effect
(with a certain remaining low probability for influences of order).

>I'm going to look some more into order bias and methodologies to 
>control for it.  One of the reasons that I wrote my own survey system 
>was that I wanted to be able to respond to best practices and explore 
>"new" possibilities such as my image item types (I know they have been 
>around on paper but I haven't seen them on web surveys).  It would be 
>trivial to add the code to randomize order of items and/or item groups. 
>I could record both the response and the order of the items responded 
>to but it seems easier to me to just accept the bias -- my head hurts 
>when I think about the size of output for those exponential 
>possibilities. Very few would have the skill to properly analyze it. This
brings up another concern of mine.

In both cases, balancing and randomization, you wouldn't need to analyze it,
because you eliminated order effects by design. Recording the actual orders
would make sense, though, in case there are unexpected patterns in the
results.

>How much should the software protect the user from stupid mistakes? We 
>agree that there are at least some occasions where order does matter. 
>Every little bit of software feature creap creates a more cumbersome 
>interface and the possibility that the option will be used incorrectly.

I completely agree and see this as a general problem with many softwares.

>I suspect that you
>would agree that reseachers are reluctant to learn complex interfaces. 
>I shudder to think of giving lots of options that might be randomlly 
>selected without thought of the consequences.

Personally, I believe these decisions should be based on knowledge about the
intended audience, usability considerations, and in a good educational
sense. Check out WEXTOR, our (Web) experiment generator, at
http://psych-wextor.unizh.ch/wextor/en/ , where we have tried to achieve an
optimal balance for a dual audience: researchers and students. WEXTOR
automatically avoids many pitfalls naïve researchers step into on the
Internet, and it implements techniques that have shown to be useful in
Internet-based research.

>Once I get the dissertation behind me, I intend to rewrite BIRAT into a 
>more portable environment such as PHP and MYSQL. Perhaps it should be 
>another discussion but I was interested in your thoughts about free
platforms.

Imho this is the way to go, as long as they ar useable. Proprietary software
is always a risk.

>I
>also look forward to the day when all software is free. Even so, 
>portability versus usability are a concern. My experience is that ASP 
>and the associated Access DB are very common in academic servers and a "no
brainer" to install.

Access has been notoriously vulnerable.

>PHP/MySQL applications often make up for their lack of cost and 
>improved efficiency by their difficulty to install.  What good is a 
>publically available application if most folks can't figure out how install
it?

A good reason to run it on a Mac, where PHP comes pre-installed and MySQL is
easily added (e.g. via MAMP).

>
>Of course BIRAT also uses other languages like HTML, CSS, Javascript and
>such as well.  We live in interesting programming times.
>
>Just in case the above has left you in less than serene, you might want to
>look at another of my free applications: http://serenesound.com. I've been
>amazed at how it has been adopted by the medical community.
>
>Finally, any chance you can send me the content of the papers/book you
>mention below?  I'd love to read them but am traveling and thus away from
my
>school library.

Some of it is on 
http://www.psychologie.unizh.ch/sowi/team/reips/publikationsliste.html 
Let me know if you need any of the other chapters.
Best --u
_______________________________________________
The air-l at listserv.aoir.org mailing list
is provided by the Association of Internet Researchers http://aoir.org
Subscribe, change options or unsubscribe at:
http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org

Join the Association of Internet Researchers: 
http://www.aoir.org/