Table of Contents
February 13, 2024
|12m
How to Scan Bulk IOCs with VirusTotal?
If you have ever been part of security teams like the Security Operation Center (SOC) or CIERT teams, you might have dealt with many threat analyses as part of your incident response and malware analysis job. Capturing IOAs and IOCs and analyzing captured IOAs and IOCs are some of the crustal parts of the incident response and investigation procedure. And, of course, if you get lengthy lists of URLs, domains, IPs, and files, then it’s not just crustal, but also it’s a laborious task.
I still remember the early days of my professional job, where my team was used to get hundreds of URLs, domains, IPs, and file hashes from our security advisory partners to analyze and block them on all our security devices. We used to spend several hours just analyzing the captured IOCs every day. It was frustrating! The worst part is that the security services were not as mature as they are today. We must thank VirusTotal and other security services that have made security professionals’ lives a lot easier now. Today, we can automate such things with the help of services like VirusTotal combined with scripting languages like Python.
We created this post to help security analysts who need to validate or identify IOCs like URLs, domains, IPs, and file hash in bulk every day. We will show you a complete process to submit a list of IOCs to VirusTotal to the VirusTotal’s API using Didier Stevens’s Python Script. Let’s see what VirusTotal is and its API offerings. About Didier Stevens and his Python tools, automate the IOC scanning process by learning how to scan bulk IOCs with VirusTotal in this blog post.
Note: If you want to submit multiple files to VirusTotal to scan. See here.
A Short Note About VirusTotal and Its API Service
VirusTotal is a free online service that analyzes files and URLs, enabling the identification of viruses, worms, trojans, and other kinds of malicious content using antivirus engines and website scanners. It also enables the generation and sharing of threat intelligence with its huge URL and file analysis database.
VirusTotal offers both public and private APIs that allow users to programmatically interact with their services. The public API has some limitations, like only allowing 500 requests per day and 4 requests per minute. The private API, on the other hand, provides more flexibility and advanced capabilities like allowing users to choose their own request rate and quota, download submitted samples, get more detailed analysis reports, etc.
If you are an individual security researcher who wants to scan multiple URLs, domains, IPs, and file hash with VirusTotal less frequently, then public API services are enough to have. If you are working for a corporate company that needs to do a lot more than just scan URLs, domains, IPs, and file hash, then you should get private API services.
Public vs Private API
Here is a quick comparison of VirusTotal’s public and private APIs:
| Feature | Public API | Private API |
|---|---|---|
| Request rate limit | 500 requests/day, 4 requests/min | Flexible based on service tier |
| File download | No | Yes |
| Additional metadata | No | Yes e.g. first submission date, prevalence etc. |
| File behaviors | No | Yes |
| Advanced hunting APIs | No | Yes e.g. YARA based hunting |
| SLA | No | Yes |
Didier Stevens and his Contributions to the Cybersecurity World
If you have been in the security landscape for some time, then you definitely know that Didier Stevens, is a renowned security researcher who has made numerous contributions to the cybersecurity community. He is the author of many popular open-source security tools written in Python, including:
oledump: to analyze OLE files like MS Office documents
pdf-parser: to analyze PDF documents
oletools: tools to analyze MS OLE2 files (Structured Storage)
peepdf: PDF analysis tool
VirusTotal: to search VirusTotal for hashes, URLs, IPs and domains
These are only a few tools that he created. He has tons of tools in his ToolSuite. However, these are quite popular tools that are very useful for malware analysts, forensics investigators, and security researchers to analyze and gather threat intelligence.
Anyways, these tools are not the point of discussion in this blog post. We have another tool or a Python script vtsearch to cover in this blog post. Didier Stevens’s TirusTotal tool allows interfacing with VirusTotal APIs, making it easy to gather reputation data and scan bulk IOCs.
Prerequisites to Validate Bulk IOCs with VirusTotal
To start scanning bulk IOCs with VirusTotal, you need a few things in place:
VirusTotal API Key: The first requirement is to sign up on VirusTotal and get an API key. This key will be used to authenticate API requests. With the free public API key, you can make only 500 requests per day with a rate limit of 4 requests/min. For bulk scanning, it is recommended to get the private API key, which has higher rate limits.
Python Interpreter: You need Python 3.x installed to run the vtsearch tool. Python can be downloaded for all major operating systems from python.org. Make sure to add it to your PATH.
Python IDE (Optional): Using an IDE like PyCharm or Visual Studio Code can improve the development experience. But any text editor would also work.
vtsearch Tool: This is a modified Python tool which is originally created by Didier Stevens and is used to interface with the VirusTotal API. You can download it from Didier’s blog or GitHub repo. This script was originally created to run on Python 2. We modified it to work on Python 3. Now this tool has been tested on Python 3.9.
That’s pretty much all you need to start submitting bulk IOCs to VirusTotal for analysis. Make sure Python and the vtsearch tool are configured correctly and you have the API key. Now we are ready to automate the scanning of bulk URLs, domains, IPs, and file hashes by programmatically interfacing with VirusTotal.
What vtSearch.py can Capable to Do?
vtsearch.py is a Python program that is designed to search VirusTotal for hashes, IPs, domains, and URLs using free public APIs. And, returns the result to a CSV file and CLI. Some of its capabilities:
Customize delay time in sending search queries.
Force all queries to be sent to VirusTotal, even if found in a local database.
Calculate the md5 of the file and search it
File to keep track and skip not-found searches.
wait 1 hour when VirusTotal limitations exceeded
Why You Need to Submit IOCs to VirusTotal in Bulk?
The main intention of this program is to automate the IOC verification process in malware analysis, forensic investigation, or threat detection procedures.
Security Advisory or Threat Intelligence teams often share a list of IOCs with the Organization’s Security Team in order to block the IOCs on their Proxy, Endpoint, Firewalls, SIEM, and other security solutions. You can use this script not only to identify the infected assets but also to curb the spread of malware infection on the corporate network.
It’s a very well-known fact that not all security products are perfect and provide 100% protection against thriving threats. This script also helps Security teams to determine the list of IOCs that were flagged as malicious by the Vendors of their products and block them on their security products if not blacklisted and submit the unidentified IOCs to their vendors in order to block them globally.
How to Scan Bulk IOCs with VirusTotal?
The procedure is simple and straightforward. You just need to set up your Python environment with the VirusTotal API key. Just follow these steps to leverage VirusTotal APIs for scanning multiple IOCs in an automated manner:
Step 1: Signup on VirusTotal and Acquire API Key
Go to VirusTotal.com and create a free account.
Navigate to your Profile and note down the API key provided. This will be used for authentication.
Step 2: Download the vtSearch script
Download the Python script from here and place it on your machine. We copied it to the c:\\TheSecMaster Directory on our machine.
Download the Script from Git.
Step 3: Install Python interpreter
Download the latest Python 3.x from python.org and install it.
Add Python to a PATH environment variable.
How to Install Python:
Install Python on Windows: http://thesecmaster.com/step-by-step-procedure-to-install-python-on-windows/
Install Python on Linux: http://thesecmaster.com/3-ways-to-install-pycharm-on-linux-mint-and-ubuntu/
To ensure Python interpreter is working on your machine, run this command: python -V
Step 4: Install PyCharm or Conda (Optional Step)
This step is totally optional. You can use any code editor of your choice. We recommend using either Pycharm or Visual Studio Code as it has excellent Python support through extensions.
For this demo, we will use PyCharm as our chosen IDE for Python development. You can check out how to install PyCharm on Windows here. If in case you are not a fan of any IDE, you can directly download the Python interpreter and use it on your CLI.
Download and install PyCharm Community Edition from jetbrains.com/pycharm. Make sure to customize the installer to add Anaconda Python environment support.
Once setup is complete, open PyCharm. Open the directory in the vtSearch script is kept. Go to File > Open > Browse the directory. Click the gear icon and select “Add.” Locate the Python executable in the <conda env> folder in the Anaconda installation.
Don’t forget to add the VirusTotal API key to the script. As you can see, we have added our API in Line #14.
IMP Note: Replace this key with your own key. We deleted this API key before we published this post. You can’t use this API key shown in the screenshot.
As soon as you open the directory, PyCharm will create an isolated Virtual Environment for this project. Make sure the Python interpreter is configured for the project. To set up the Python Interpreter in your PyCharm, Click on the gear icon in the top right corner, then click on Setting. If you don’t have the Python interpreter, click on ‘Add Interpreter’, browse, and select the Python.exe file.
Great! Now your PyCharm is configured to run the vtSearch.py script.
Let us know if you need any help setting up the IDE.
Step 5: Run the Python script to Submit bulk IOCs to VirusTotal
Before running the vtSearch.py let’s learn about a couple of arguments to pass. This script accepts arguments like -f, -t, -e, and -d.
-f: It is a flag used to indicate the file has a list of md5 hash values.
-t: It is a flag used to indicate the file has a list of IPs, URLs, or Domains.
-e: It is a flag used to indicate the file has a list of sha256 hash values.
Some examples to Run the script:
python vtSearch.py -f hashes.txt
python vtSearch.py -t url urls.txt
python vtSearch.py -e sha256 hashes.txt
python vtSearch.py -f –d 30 hashes.txt
We have a file Sample.txt in that we saved 10 phishing domains. Let’s run the script to scan the domains with VirusTotal.
python vtSearch.py -t url Sample.txt
The script creates a csv file to save the results. Here is the content of the csv file.
Step 6: Run the Python script using PyCharm (Optional Step)
Once The vEnv and Interpreter are ready, then set up the parameters to run the script. To set up the running parameters, Click on the 3 vertical dots. Select the parameters.
We set up -t url Sample.txt as the parameter and click on the Run button.
Another CSV file will be created, which looks like this.
Manually analyzing and validating lengthy lists of IOCs is a repetitive, monotonous, and cumbersome task. But we can easily automate scanning bulk IOCs by utilizing VirusTotal’s API and open-source tools like vtsearch.
Key Advantages:
Faster than manual analysis – Automated scanning of bulk IOCs is significantly faster than manual lookup.
Error-free – No chances of human errors when using automation.
No human intervention needed – Once scripted, you just need to run it for hands-free IOC scanning.
Fully automated – Scheduling recurrent scans is easy to validate new IOCs automatically.
By leveraging VirusTotal’s API capabilities and Didier Stevens’s handy vtsearch tool, security teams can supercharge their threat hunting and intelligence workflows.
Automating the scanning and validation of bulk IOCs like URLs, domains, IPs, and file hashes can save hundreds of hours of repetitive manual work. This allows highly skilled resources to focus on other high-priority tasks like threat analysis, hunting, response, and improving security posture.
In summary, if you frequently deal with large IOC lists as part of your job, do check out the vtsearch Python tool to easily submit bulk IOCs to VirusTotal.
We hope this article helped understand how to submit bulk IOCs to VirusTotal. Thanks for reading this post. Please share this post and help secure the digital world. Visit our website, thesecmaster.com, and our social media page on Facebook, LinkedIn, Twitter, Telegram, Tumblr, Medium, and Instagram and subscribe to receive updates like this.
You may also like these articles:
Arun KL
Arun KL is a cybersecurity professional with 15+ years of experience in IT infrastructure, cloud security, vulnerability management, Penetration Testing, security operations, and incident response. He is adept at designing and implementing robust security solutions to safeguard systems and data. Arun holds multiple industry certifications including CCNA, CCNA Security, RHCE, CEH, and AWS Security.
- November 20, 2024
Learn Something New with Free Email subscription
Learn Something New with Free Email subscription
Subscribe
Subscribe
Subscribe
Subscribe
- November 20, 2024
