[Air-L] Extracting Twitter Data

Tue Sep 22 17:12:51 PDT 2020

Hi all,

Just briefly, I think it is still possible to do targeted searches via advanced search, it just seems the actual advanced search popup interface is broken (??). Although I am not sure what Guido is referring to.

It seems as though you will need to type in the search string into the search field. Here is a topical example for Australia today (with policy backflip to fund a full fibre broadband network, aka NBN), looking at "NBN" mentions during the election period in 2013: https://twitter.com/search?q=%22NBN%22%20since%3A2013-07-13%20until%3A2013-09-07&src=typed_query&f=live

I've had a quick look in Outwit Hub and it is still pulling the individual tweets for the above search string. (My old scrapers are not working, which is not surprising as they would only last until the next update anyway, and I haven't used since around 2018ish. Outwit Hub is pulling the individual tweets in the 'guess' field.)

To get more useful information and take into account the ~4500 tweet limit will need a refined search string strategy.

Cheers,
Glen

From: Air-L <air-l-bounces at listserv.aoir.org> On Behalf Of Mosallaeipour,Fatemeh
Sent: Tuesday, 22 September 2020 10:42 PM
To: Guido Anselmi <guido.anselmi at unimib.it>
Cc: air-l at listserv.aoir.org
Subject: Re: [Air-L] Extracting Twitter Data

Thank you Fabio, Marc, and Guido,

I'll try your suggestions and will let you know about the results. However, I think what Guido says is correct! But will try and keep you updated.

Cheers,

Afrooz

________________________________
From: Guido Anselmi <guido.anselmi at unimib.it<mailto:guido.anselmi at unimib.it>>
Sent: Tuesday, September 22, 2020 3:48:46 AM
To: Mosallaeipour,Fatemeh <fm455 at drexel.edu<mailto:fm455 at drexel.edu>>
Cc: air-l at listserv.aoir.org<mailto:air-l at listserv.aoir.org> <air-l at listserv.aoir.org<mailto:air-l at listserv.aoir.org>>
Subject: Re: [Air-L] Extracting Twitter Data

External.

Hi

it used to be possible to gather older tweets for free through ad hoc scrapers that leveraged a ' bug' in the advanced search, as far as i know that particular issue has been patched very recently, so at the present moment ( the last 2 days afaik) there is no way of getting what you want for free. i have no doubt that a way of restoring previous access will be found quickly but at the present moment the only way of getting what you want is through screen scraping, probably using some kind of browser emulation like Selenium which is kinda unwieldy.

there are multiple project of twitter scrapers on github, in the coming weeks probably one of the will find a way to acquire data your best chance is to bookmark them and wait

best
Guido

Il giorno mar 22 set 2020 alle ore 05:00 Mosallaeipour,Fatemeh <fm455 at drexel.edu<mailto:fm455 at drexel.edu<mailto:fm455 at drexel.edu%3cmailto:fm455 at drexel.edu>>> ha scritto:
Dear colleagues,

I hope this email finds you all well and safe.
Help needed! I am trying to download the videos and tweets shared in a specific period. I only need the information on public accounts that is accessible for any Twitter user: the name of the user who has shared it, the date, the video, and the tweet itself. I have been using the advanced search tool on Twitter, entering the date, and desired hashtags and accounts. It is a huge amount of data and I have to redo the process once more to get some more information and complete what I currently have. I have already looked for codes; however, a friend informed me that such information is only available for free, even with coding, for the last 30 days. I am looking to retrieve data from last November though.

Has anyone gone through the same obstacle? Is that true that the only way for me is to do it manually? If not, is there any code for R?

I appreciate your help!

Best,

Afrooz Mosallaeipour
Drexel University
Ph.D. Student in Communication, Culture, & Media

_______________________________________________
The Air-L at listserv.aoir.org<mailto:Air-L at listserv.aoir.org<mailto:Air-L at listserv.aoir.org%3cmailto:Air-L at listserv.aoir.org>> mailing list
is provided by the Association of Internet Researchers http://aoir.org<http://aoir.org><https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Faoir.org%2F&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=jWP8KTQFOmAjyVvSg%2BOyfkto%2FpfNXluiXQ%2BvQ%2BYC9Ys%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Faoir.org%2F&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=jWP8KTQFOmAjyVvSg%2BOyfkto%2FpfNXluiXQ%2BvQ%2BYC9Ys%3D&reserved=0>>
Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org<http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org><https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flistserv.aoir.org%2Flistinfo.cgi%2Fair-l-aoir.org&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=VB6Rp0B5E783ILAyJxqDw1%2BDVzrUY5ayWPOV%2BYI3B5E%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flistserv.aoir.org%2Flistinfo.cgi%2Fair-l-aoir.org&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=VB6Rp0B5E783ILAyJxqDw1%2BDVzrUY5ayWPOV%2BYI3B5E%3D&reserved=0>>

Join the Association of Internet Researchers:
http://www.aoir.org/<http://www.aoir.org><https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.aoir.org%2F&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=rsjHPgka6gbLbZCeV9MaydW0o%2BznVrgz%2BXTp%2BPNHbdE%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.aoir.org%2F&data=02%7C01%7Cfm455%40drexel.edu%7C31f81963f6864308366308d85ecbf76e%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637363577426896014&sdata=rsjHPgka6gbLbZCeV9MaydW0o%2BznVrgz%2BXTp%2BPNHbdE%3D&reserved=0>>
_______________________________________________
The Air-L at listserv.aoir.org<mailto:Air-L at listserv.aoir.org> mailing list
is provided by the Association of Internet Researchers http://aoir.org<http://aoir.org>
Subscribe, change options or unsubscribe at: http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org<http://listserv.aoir.org/listinfo.cgi/air-l-aoir.org>

Join the Association of Internet Researchers:
http://www.aoir.org/<http://www.aoir.org/>