OSINT Toolbox Talk: Geo-Monitoring Telegram users, scraping Twitter data, and investigating Yandex accounts

OSINT Workflow

Scraping and visualising Facebook friend lists


One question I have repeatedly been asked during various training projects is “how can I extract the friends list of a target on Facebook and visualise it as a link analysis chart?”. It is undoubtedly a huge challenge for many Digital Investigators and OSINT Analysts; and it seems that people are instinctively drawn to various Github repositories in order to find the answer. In fact, to address this challenge with expert precision, there is no need to use any Command Line Interfaces. Instead, we are going to show you how you can achieve this task through the use of two Google Chrome extensions, one Sock Puppet account, Microsoft Excel and a data visualisation tool.

Before we jump right in, I am obliged to point out that accounts used for scraping are not allowed without Facebook’s permission, and can therefore be locked by Facebook anti-scraping mechanisms. Also note, the use of Sock Puppets on social media platforms and within official settings (Law Enforcement / Government / Military) is often strictly regulated by legislation. Investigators should direct their attention to their national legislation before undertaking any activities on social media. The information in this article is for information / educational / research purposes only and should not be interpreted in any other way.


One important point that I should indicate in this OSINT Workflow article that this technique may only be applied to Facebook friend lists that are publicly visible. However, if you have a Sock Puppet that is connected to the target Facebook user, then the process that we are about to show you will also apply. We should stress that there is no way to bypass Facebook security and privacy mechanisms to uncover hidden friend lists.

TOOLS REQUIRED

  1. Google Chrome: For the purpose of this OSINT Workflow article, we shall use Google Chrome as it is the most compatible browser for the extensions indicated below.
  2. Webpage Auto Scrolldown: A Google Chrome extension that can be used to auto-scroll down a target Facebook profile’s friend list. It can be activated by clicking the icon on the top right of the Google Chrome browser and clicking once more in order to deactivate it. The extension can be downloaded and installed via this link.
  3. Instant Data Scraper: Another Google Chrome extension that is a very handy tool that is used to extract URLs and allow you to export them within a comma-separated value (CSV) file. The extension can be downloaded and installed via this link.
  4. Microsoft Excel: We will use Excel in order to build our Edge and Node lists by applying some functions.
  5. Gephi: Gephi is a data visualization and exploration software for all kinds of graphs and networks. Gephi is a free / open-source tool that is compatible with Windows, MacOS and Linux operating systems. The software can be downloaded and installed via this link.

PREPARING THE OSINT ENVIRONMENT

It goes without saying, all SOCMINT activities should be done through the use of a Sock Puppet for a variety of reasons. If you have not created a Sock Puppet account for Facebook, please read through our comprehensive guide.

With your tools installed and your Sock Puppet configured, it is now time to access Facebook and view the friend list of the target profile. The URL should be: https://www.facebook.com/YOURTARGET/friends

SCRAPING THE FRIEND LIST

We are now on the target profile friend list. If your target only has up to 200 friends, then there is no real need for you to use the Webpage Auto Scrolldown. Otherwise, you should go ahead and click on the Webpage Auto Scrolldown icon that can be found in the top right of your Google Chrome browser. However, you may need to click on the browser ‘jigsaw’ icon and pin the extension to the top bar.

When you click on the Webpage Auto Scrolldown icon, the extension will work its way down to the bottom of the Facebook page. You will also notice that the target profile’s likes will also be expended – this is very normal, just allow the tool to finish what it needs to do.

Once you are at the bottom of the Facebook page, click on the Webpage Auto Scrolldown icon once more – this will prevent the scroller from forcing its way back down to the bottom of the page and allow you to regain control of your mouse scroll-wheel.

Navigate yourself back up to the friend list using the browser scrollbar or the mouse scroll wheel, then click anywhere on the whitespace within the friend list. Make sure not to accidentally click on another Facebook user profile otherwise, you will need to repeat the above process. Once you have clicked on an area of whitespace (perhaps in-between two of the target’s friends) you can now click on the Instant Data Scraper icon which should be located on the top right area of the browser window.

Once clicked, a pop-up window should open. The pop-up should contain a list of Facebook profile URLs and names. These URLs are the target’s friends. Inside the pop-up, you should click on the green button named ‘CSV‘ – once clicked, the target’s friend list will download in a Comma-Separated Value (CSV) file.

IMPORTING THE CSV TO MS EXCEL

Now we have our friend list downloaded as a CSV file, you should switch to MS Excel.

scraping user data from Facebook

To import the CSV file into MS Excel, you should open a blank worksheet and then navigate to the data tab located on the top of the MS Excel window. You should see on the left-hand side of the ribbon a button named ‘From Text/CSV‘, click on this. This will now open a new window where you can navigate to the Facebook target’s friend list CSV file. You should double-click on the file or select it and then click ‘Open’.

A pop-up window should appear in MS Excel providing you with some options with regards to importing the CSV file to MS Excel. Click on the dropdown arrow under ‘File Origin’ and then select UTF-8. Next, you should clickTransform Data‘ located in the bottom right of the pop-up window. By clicking on the button, you are now finalising the import of the CSV file into MS Excel.

web scraping on Facebook

The Power Query Editor should open in a new window on your screen. Within Power Query Editor, you should remove the first row by clicking on theRemove Rows‘ button and then selectingRemove Top Rows‘. A separate window will now open asking you how many rows you wish to delete. As we only need to remove the first row, we will input1‘ and clickOk‘.

Next, we click onClose & Load To…‘ and selectExisting Worksheet‘, then clickOk‘.

CREATING THE NODE LIST IN MS EXCEL

You should see the friend list of the Facebook target in MS Excel.

extracting friend lists on Facebook

Our first task is to rename ‘column1′, column2’, ‘column3’ and ‘column4’ with ‘url‘, ‘img‘, ‘fullname‘ and ‘description‘. After this, delete the column namedcolumn5‘ as it will add no overall value to the link analysis chart that we will produce at the end.

Facebook OSINT

Return to the first row of columns and add three new columns in front of the column named ‘url’ and name themid‘, ‘label‘ and ‘role‘.

Advanced SOCMINT techniques

Next, you should input a formula that will extract the profile username from each of the individuals within the friend list. The formula will extract the username from the ‘url’ column by counting the number of characters within the URL string and extract the username. The formula to do this is  =IF(ISNUMBER(FIND("profile.php",D2)),MID(D2,41,LEN(D2)+1-41),MID(D2,26,LEN(D2)-25)). You should paste the formula into the first entry under the column named ‘label’.

Conducting OSINT activities on Facebook

Next, within the first entry for the column named ‘id’ input the formula =B2, this formula will copy the value from the ‘label’ column. In the first entry for the ‘role’ column, go ahead and type infriend‘. To apply the aforementioned formulas to the rest of the fields, simply click on the entry you wish to copy (for example, B2); in the bottom-right corner of the cell, you should see a square; using your mouse, right-click, hold and drag down across all of the relevant cells within the same column (as shown in the animated image above). This process can be repeated for the rest of the fields.

The Node List is now complete, I would recommend you save it as a standard MS Excel file with a name such as ‘[TARGET NAME]-nodelist’ for example. Don’t close the spreadsheet just yet as we will need it very shortly!

CREATING THE EDGE LIST IN MS EXCEL

To create the Edge List, you should start by opening a blank worksheet in MS Excel.

Facebook intelligence gathering

You should begin by naming the first four columnssource‘, ‘target‘, ‘type‘ and ‘weight‘. Next, return to the Node List and select all of the entries from the ‘id’ or ‘label’ column and copy them. Returning to your Edge List, right-click on the first blank entry within the ‘target’ column and paste the items as values.

In the ‘source’ column, this is where the target’s user name should be placed. You can start by inputting the target name in the first blank cell and then copying and pasting the target name into the rest of the blank cells in the ‘source’ column.

In the column named ‘type’, the following rule applies. If the individuals are friends of the target, you should input ‘Undirected’. However, if the individuals are followers of the target, you should input ‘Directed’.

The last column ‘weight’ is nice and simple, just input ‘1’.

Using the same processes indicated in the preceding section, you can now populate the remainder of the fields by using the copy and paste function.

Now it is time to save this spreadsheet. Again, save it as a regular MS Excel file with a name such as ‘[TARGET NAME]-edgelist’ for example. Don’t close the spreadsheet just yet as we will need it very shortly!

MULTIPLE FACEBOOK TARGETS

OSINT tips and tricks for law enforcement

In the likely event that you have multiple target profiles to analyse on Facebook, you can apply all of the processes indicated above and populate all of your data within the same Node and Edge Lists. The only thing you will have to make sure of is that you account for the different target username (as indicated in the animated image above).

EXPORTING THE NODE AND EDGE LISTS

MS Excel for Intelligence Analysts

You should export both your Node and Edge Lists as Comma-Separated Value (CSV) spreadsheet. As indicated in the animated image above, click on ‘File‘ and then selectExport‘ from the left sidebar. Here, you should selectChange File Type‘, then selectCSV (Comma delimited)‘, then ‘Save As‘, and then save the file in the same folder as your original spreadsheets. You may close your MS Excel windows.

CREATING THE LINK ANALYSIS CHART WITH GEPHI

With the Node and Edge Lists created in CSV format, you can now open Gephi and create a ‘New Project‘.

Intelligence charts with Gephi

With our project created, we can select the ‘Data Laboratory‘ button and use this area to import your Node and Edge Lists.

Facebook user OSINT tools

Starting with the Node List, ensure that the ‘Nodetab is selected in the top-left of the Data Laboratory window and then click on ‘Import Spreadsheet‘. Next, navigate to and select the Node List CSV file that you want to import into your Gephi Project. A pop-up window should now appear. In the ‘General CSV Options (1 of 2)’, you can go straight ahead and selectNext‘, and do the same in the next section titled ‘Import Settings (2 of 2). In the following pop-up window named ‘Import Report’, make sure to selectAppend to existing worksheet‘, then click Ok‘. You will now see the Node List imported into the Gephi Data Laboratory.

SOCMINT tips and tricks

Next, we want to import the Edge List. Select the tab namedEdge‘ located in the top-left section of the Data Laboratory window and then click on ‘Import Spreadsheet‘. You can start by navigating to and selecting the Edge List CSV file that you want to import. Follow the exact same process as you have done with the Node List and make sure that you selectAppend to existing worksheet‘ when the ‘Import Report’ popup appears. After this, you will see that the Edge List has now been imported into the Gephi Data Laboratory.

Effective SOCMINT and OSINT tools

It is now time to view and configure the Link Analysis chart. Click on the ‘Overview‘ tab located at the top of the Gephi window. If you do not see a Link Analysis chart in this section, navigate to the top bar and selectWindow‘ and then ‘Graph‘. After following this instruction, you should now see a link chart appear. At this point, it is time for you to configure the layout of the Link Analysis chart. You can do this by selectingForceAtlas 2‘ located in the ‘Layout’ tab to the left of the screen. You can now configure your desired visualisation settings (as indicated in the animated image shown above) and then clicking ‘Run’. Lastly, within the ‘Graph’ window, you can configure text options by selectingT‘ to show the label of each node and adjust the size of the text as you see fit.

And that is it! We have now scraped a target friend list from Facebook and visualised the data within a Link Analysis chart. However, there is much more that we can do inside Gephi in order to exploit the data that we have collected and visualise it across different views. In our next OSINT Workflow article, we will take a closer look at Gephi and the data visualisation power that it holds!


Let's talk today Are you ready to begin discussing our range of training and capability development solutions?