[Air-L] Trust Games - Annotation Tasks

Fri Jul 26 06:06:51 PDT 2024

Thank you for your email. I am currently out of the office on vacation and will have limited access to email. I will return on August 15th and will respond to your message as soon as possible after that date.

Best regards

Vielen Dank für Ihre E-Mail. Ich bin derzeit im Urlaub und habe nur eingeschränkten Zugang zu meinen E-Mails. Ich kehre am 15. August zurück und werde Ihre Nachricht so bald wie möglich nach diesem Datum beantworten.

Mit freundlichen Grüßen,

Paloma

On 17 Jul 2024, at 14:18, Shulman, Stu via Air-L <air-l at listserv.aoir.org> wrote:

> Follow Up
> 
> I am grateful to the list for generating excellent people willing to work
> on projects. Already we have a nice group forming with novel exchanges of
> ideas in every meeting. Collaboration like this is fun and hopefully the
> game, Trust Defender, will be as well.
> 
> I do need more students, including some willing to label Twitter data as
> soon as today. I am testing a new approach to remunerating "micro" tasks
> that are limited to 5 minutes. I need to know the average speed, as well as
> the approximate best task size, and how fast the best human annotators can
> record reliable observations. I just did a small test and my rate was 1.45
> seconds per item. For this initial "live-deleted-suspended" task, the key
> is a combined speed and accuracy score. If you make a keystroke error, you
> need to go back and fix it, which slows you down. If you let mistakes go
> unfixed, we will find them using measurement against a known gold standard
> during the initial rounds of training on this task.
> 
> The base remuneration is $20 for completion of tasks that have a total of
> 40 minutes allotted (8 tasks) during a roughly 24 hour period. It is
> definitely possible to complete all of these current tasks with close to
> 100% accuracy in less than 40 minutes. I am planning to start with a
> requirement for labeling 60 items in five minutes (5 seconds per item) in
> the early rounds. Then we will test the effect of increasing the number of
> items to 75 and 100 in later rounds. I'm trying to work out a prize system
> for the fastest and most accurate annotators over time. We have recently
> enhanced our IRB compliance architecture for this work. AS previously,
> deleted and suspended Tweets are not visible and now all Tweet metadata are
> also hidden from annotators. Our datasets have, in some cases, more than
> 70% of the items suspended. In these cases, it can be fascinating (also
> shocking) and definitely educational to see what is still live on Twitter.
> 
> One goal is to better understand how DiscoverText can be further modified
> to have default, ready-to-play games, as well as the traditional framework
> where it is simply a flexible tool, like a spreadsheet. Leaders of the
> games will create new tasks, codes, rules, parameters, code books,
> assignments, peer groups, and focus the students. Much of the content of
> the games can be created by the professors or teams of students. Our role
> now is creating archetypal game formats  such as the deductive "live,
> deleted, suspended" or a key new hybrid deductive/inductive "trust, don't
> trust, need more info" game that has nothing to do with speed or accuracy.
> More fully inductive educational games such as "bot, troll, or citizen" are
> possible. Small adaptations to the existing crowdsource software will make
> the "labeling" task more game-like. Students will train or play on default
> games out-of-the-box that are easy to launch, but professors will also
> adapt the framework with very minor training to define parameters for their
> own games. I will be working with professors to design rubrics so that
> students can play very short games in the fall then use the experience as
> fodder for discussion.
> 
> As a reminder: DiscoverText is 100% free for academic research and
> teaching. Anyone using DiscoverText to test or implement these annotation
> games can also pursue their own independent research agenda with
> other academics. We are creating a process that will streamline the
> onboarding of entire classes to enable "quick launch" games. There will be
> leaderboards and anonymization options.
> 
> https://discovertext.com/mentions/
> https://calendly.com/discovertext
> 
> Stu
> 
> 
> 
> 
> On Thu, Jul 11, 2024 at 8:38 AM Shulman, Stu <stu at texifter.com> wrote:
> 
> Trust Games
> 
> I am looking for collaborators to help prepare a free educational online
> game suitable for secondary and collegiate classrooms focused on whether
> content is trustworthy or not. Please contact me if you want to be a part
> of this effort. It might operate something like the "Which Face is Real"
> application, but might be used instead for identifying and discussing
> untrustworthy accounts on Twitter as a gamified learning module for classes
> this fall. I have most of the pieces ready, but I am not an expert in
> games. I'd like to form an ad hoc team and have this operational for
> September and October of 2024. My goal is to offer an IRB-compliant game
> platform that generates usable research results and better informed student
> discussions in advance of the U.S. election in November.
> 
> Annotation Tasks
> 
> I have a new set of annotation tasks related to planning for the game
> development. I need motivated undergraduates willing to label batches of
> Tweets under conditions that test core features of gamification in
> labeling, starting with speed and accuracy. In addition to getting paid
> more for being the fastest/most accurate labelers, students will see some
> remarkable datasets that have historical significance. If you know Jr. or
> Sr. undergraduates in the US or Canada with a >3.9 GPA, tell them to send
> me a resume.
> 
> Thanks AoIR!
> 
> --
> Dr. Stuart W. Shulman
> Founder and CEO, Texifter
> Editor Emeritus, *Journal of Information Technology & Politics*
> 
> 
> 
> 
> -- 
> Dr. Stuart W. Shulman
> Founder and CEO, Texifter
> Editor Emeritus, *Journal of Information Technology & Politics*
> ResearchGate Profile <https://www.researchgate.net/profile/Stuart-Shulman>
> _______________________________________________
> The Air-L at listserv.aoir.org mailing list
> is provided by the Association of Internet Researchers http://aoir.org
> Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org
> 
> Join the Association of Internet Researchers:
> http://www.aoir.org/