Scraping Twitter without an API or user account with Tweeds

Tweeds is a very powerful Twitter scraping utility that enables OSINT professionals to retrieve tweets, media, and perform Twitter geocode-based scraping.

December 24, 2022

Twitter-focused OSINT

In recent months, Twitter has garnered a huge amount of controversy. Regardless, the platform remains a crucial source of data – especially when we consider the fact that on-the-ground developments concerning the ongoing conflict in Ukraine are being Tweeted in near real-time. Whilst we at OS2INT have focused a significant portion of our time focusing on OSINT techniques for Telegram, we have admittedly neglected the value of Twitter. In this last OS2INT OSINT Tool Review article for 2022, we are going to turn our attention to a fantastic Twitter scraping utility called Tweeds that packs a huge punch — and we will say it already, it is a simply fantastic utility to use!

What’s is Tweeds?

In recent months, Twitter has seen an intense amount of scrutiny and controversy. How the platform develops over time, and how this affects OSINT opportunities remain to be seen. But, for now, at least, we should continue to look at OSINT opportunities available on the platform – especially concerning near real-time data of events taking place in Ukraine. It is most certainly possible to use several OSINT utilities for Twitter.

Scraping Twitter content without an API or user account

However, these utilities mostly rely on the Twitter API; and this presents a huge problem for OSINT professionals as it can take days – if not weeks – to obtain an API from Twitter. At the same time, usage of the Twitter API also means that activity is limited both in terms of the number of tweets that can be retrieved as well as rate-limiting. But, there is now a solution and it is called Tweeds. What you need to know about Tweeds is that it can scrape Twitter user content without the need for an API or a user account!

Installation and deployment

This is one of the reasons why we love Tweed so much, it is incredibly simple to install and get going on your command-line interface (CLI). All that is required is to invoke pip install tweeds and that is it, you are ready to go!

Another reason why we feel wholeheartedly obliged to sing the praises of the developer behind Tweeds is because of the easy-to-understand details about the usage of the tool within the CLI. The developer has gone to great lengths to outline all of the tool’s scraping capabilities in addition to providing many different usage examples.

So, what does it do?

As indicated in Tweeds’ Github repo, the tool has many different capabilities including the ability to scrape hashtags, threats, images, videos, statistics, and Twitter history. Even better, Tweeds enables users to output scraped data in CSV and JSON, which is especially important for data analysis at a later stage. Even better still, additional scraping options include:

  • Date / time-based scraping that filters Tweets either sent after or before a specified data
  • Filter Tweets before a specified year
  • Scraping limit that enables users to specify the number of Tweets to scrape
  • Identify and scrape Tweets near a particular location and also scrape geocoded Tweets
  • Scrape Tweets only from verified users
  • Exclude Tweets containing one or more links
  • Extract Tweets that only contain images, videos, or both
  • Scrape Tweets with a minimum or maximum number of likes

Why you should use Tweed

It is quite obvious, there are many use cases for Tweeds in addition to the one we have consistently pointed out throughout this article. But, let’s look at other use cases. One clear use case for using Tweeds is for disinformation-related investigations or retrieving all Tweets made concerning a hashtag related to a high-profile event such as a terrorist-related incident. Additionally, the geocode-based scraping capability also enables users to pull all Tweets within the radius of a specific location – again, this may be of particular use for OSINT’ers monitoring a high-profile attack within a specific area.

Geolocating scraped Tweets using OSINT

Needless to say, there are plenty more use cases for Tweeds, but it is very clear that this is very likely to become the go-to tool for Twitter-focused OSINT activities.

Putting the ‘INT’ into OSINT

Although Tweeds is a fantastic utility for scraping content from Twitter, how that data can be effectively analysed is key to churning the same data into intelligence. So, during our tests of Tweeds, we scraped from several known Twitter accounts involved in the dissemination of pro-Russian disinformation and processed the output comma-separated value (CSV) files through Paliscope YOSE.

Visualising scraped Twitter data using Paliscope YOSE

The scraped data enabled us to visualise connections that existed between the accounts we scraped using Tweeds – in this case, Tweets that were effectively re-tweeted by several other disinformation actors. This was enabled very quickly and efficiently by using YOSE’s latest feature which enables users to identify connections between Twitter handles. This was further visualised through the multi-trace functionality within YOSE that quickly identified links between Twitter handles with up to two degrees of separation. Lastly, we looked even more closely at the mutual links between suspected disinformation actors. All-in-all, our ability to visualise Tweeds’ scraped data in a very slick link analysis interface enabled us to develop intelligence from raw scraped data.

Our final reflections as the year draws to an end

Tweeds is without a doubt the next go-to Twitter OSINT utility. It is lightweight, packed with plenty of very useful scraping features, and very efficient in terms of speed and processing. It was such a pleasure to test this utility to the fullest extent, and we have most certainly taken plenty of time to test all of the Tweeds’ features. There is no better way to round off an already fantastic year in OSINT than testing out this great utility and writing this article.

So, here is to 2023, and we look forward to producing lots of new OSINT content for our followers!

Merry Christmas to you all, and a very happy New Year ahead!

Start learning today

Select on the relevent tags to filter our posts

Let's talk today Are you ready to begin discussing our range of Intelligence Analysis training and capability development solutions?