14 Best OSINT Tools We Use in Our SOC

In this digital age, pretty much every individual and organization leaves behind a trail of information about themselves on the internet. Their digital footprints become potential sources of intelligence that can be gathered without consent.

You may be aware that such information is out there, but collecting it is not as simple as just extracting whatever you want from the public internet. Gathering intelligence is a science – you need to understand how and where to look to find the relevant bits. This is where OSINT (Open Source Intelligence) tools come into play.

Open Source Intelligent tools help locate and assemble the required intelligence about a target from the complex web of interconnected networks. Since these tools are open source, they can be utilized by anyone – but they are most heavily used by hackers and security professionals who rely on such information daily.

These OSINT tools can be deployed for both offensive and defensive objectives depending on the user’s intent. As an information security practitioner, I aim to educate peers on the OSINT capabilities used in our daily operations. In this post, we will cover what comprises OSINT, why we need it, what intelligence we typically seek, who else leverages these techniques, and a list of go-to OSINT resources for security investigations.

What is OSINT?

OSINT stands for Open-Source Intelligence. It refers to the practice of gathering intelligence from publicly available sources and data. Unlike classified information, which requires special access, OSINT utilizes open source data that can be accessed legally without any restrictions. This includes information found on the internet, public records, news articles, social media platforms, commercial data sources, and more.

In the context of cybersecurity, OSINT is used by information security teams to gather intelligence about external threats targeting an organization. It helps map an organization’s digital footprint and attack surface by consolidating relevant publicly available data. This allows security analysts to identify potential vulnerabilities in the organization’s online presence which could be exploited by attackers. Common use cases for OSINT in cybersecurity include external threat intelligence, attack surface mapping, infrastructure mapping, identifying network vulnerabilities, and more.

Why Do We Use OSINT in Our SOC?

As I said earlier, security professionals love Open Source Intelligent tools to make tedious tasks simpler. We use a handful of OSNT tools and techniques in our Security Operations Center (SOC) to strengthen our security posture:

In essence, integrating OSINT gives our SOC greater context, visibility and insights to prepare, detect, respond and recover from security threats targeting the organization.

What Information We Try to Gather Using OSINT Tools?

In SOC, we spend most of our time in monitoring and investigating suspected events and incidents. We always in search of threat intelligence and information available on new malware, vulnerability, attack campaigns, security updates, and there are many more things to list. Bear in mind, security teams use Open Source Intelligent tools to gather data which are publicly available. OSINT can’t be used to steal the data kept under hood.

The goal is to collate the above external intelligence through legal and ethical means. Analyzing this data allows us to continuously assess risks, revise controls, and improve overall security posture against a dynamic threat landscape.

14 OSINT Tools We Use in Our SOC

In our SOC, we employ a mix of paid and free OSINT tools as part of our workflows to gather, analyze and visualize security intelligence. While individual tools have particular strengths and limitations, together, they enable continuous monitoring, holistic visibility, and informed decision making against a rapidly evolving threat landscape. Key tools in our Open Source Intelligent toolkit include:

Recon-ng

Recon-ng is an open-source web reconnaissance framework written in Python that is highly extensible. We leverage Recon-ng for gathering intelligence on domains, companies, individuals etc. by tapping into dozens of APIs and public data sources.

Pros:

Cons:

Recon-ng allows even junior analysts to automate the process of collecting relevant data from various public sources and APIs on the internet. It has an interactive interpreter to easily configure modules and execute commands. The framework comes packed with dozens of builtins suited for common recon activities like resolving domains, finding subdomains, fetching WHOIS records etc.

But what makes Recon-ng extremely versatile is its support for community-developed custom modules. Analysts can create their own modules tailored to specific data gathering needs and integrate proprietary data feeds. This enables leveraging Recon-ng for focused objectives like gathering intel on threat actors, compromised credentials, vulnerable systems, etc. It outputs results to a database, which can then be conveniently filtered, analyzed and correlated.

Given its flexibility, extensibility and automation capabilities, Recon-ng forms an integral part of our day-to-day security reconnaissance needs to uncover hidden threats and inform better decision-making.

Maltego

Maltego is a proprietary investigative tool specializing in graphical link analysis for gathering and connecting publicly available information across the internet. We leverage Maltego’s visualization capabilities for ad-hoc research and threat investigations.

Pros:

Cons:

Maltego takes in seed data points like phrases, names, websites, domains etc. and transforms them into a graphical network map representing connections and relationships between related entities. This makes understanding complex real-world interconnections effortless compared to staring at tables of data.

Analysts can click through the visualization to rapidly gather intelligence on threats. For instance, starting from a suspicious IP address, an analyst can pivot through owners, connected hosts, used technologies, geographic associations, and related leaks to uncover hidden threats. Case management features allow collaboratively investigating incidents.

Maltego integrates well with other security tools for seamless workflows. The desktop client works well for small localized teams while the cloud-based offering enables globally distributed teams.

While pricing can make scalability cost-prohibitive for smaller teams, Maltego’s unique graphical link analysis and ease of use makes it a very handy addition to a SOC’s OSINT toolkit – both for proactive threat hunting as well as speeding up incident investigations.

URL Scan

URL Scan is an invaluable free online service we utilize in our SOC for analyzing and scanning websites for potential threats. It safely renders submitted pages and extracts useful security insights.

Pros:

Cons:

URL Scan accepts a website URL and renders the page in a contained sandbox environment to study its behavior. This allows our analysts to see what content loads without directly exposing our assets to potential threats.

The service extracts useful metadata like cookies set, resources loaded, redirects followed, scripts executed, etc., providing visibility into activities websites attempt when visited. URL Scan also detects some known threats vulnerabilities based on service fingerprints and common patterns.

We find URL Scan extremely convenient for quick initial reconnaissance of suspicious websites, domains and pages encountered during threat hunting. Analysts can share reports with additional context and forensic snapshots for collaborating with other teams.

While automation, customization and data retention is limited compared to other offerings, URL Scan’s ease of use, freemium access, and useful website security insights make it an invaluable addition to our web-focused Open Source Intelligent capabilities.

SpiderFoot

SpiderFoot is an Open Source Intelligent automation tool that integrates over 200 modules to gather intelligence from various public data sources. We leverage SpiderFoot for recon on domains, netblocks, emails, names, etc.

Pros:

Cons:

SpiderFoot takes seed inputs like IP addresses, domains, emails etc. and automatically queries over a hundred public data sources like search engines, Pastebin, WHOIS records, satellite maps and more to map associated entities. This helps reveal related infrastructure, technologies, documents, leaks etc. with little manual effort.

Analysts can choose from pre-built modules and feeds covering threats, networks, locations etc. Results get stored locally in a database for convenient filtering and analysis. The web UI allows managing scans and reviewing results. For larger teams, SpiderFoot offers a cloud hosted option with shared access.

While the UI is dated and workflows complex for beginners, SpiderFoot’s breadth of data sources, automation capabilities, and tactical integrations makes it a versatile addition to a modern SOC’s external intelligence gathering toolkit – both for threat hunting as well as speeding up incident response.

FOCA

FOCA (Fingerprinting Organizations) is an open source OSINT tool that helps reveal an organization’s digital footprint by extracting metadata and hidden information from public documents and files. We utilize FOCA for gathering intelligence from documents like PDFs and DOCX during investigations.

Pros:

Cons:

FOCA allows uploading potentially sensitive documents like financial reports, presentations, and spreadsheets gathered during the reconnaissance of an organization. It then systematically extracts all metadata, authorship information, hashes, URLs, emails, etc. embedded within files.

This enables analysts to conveniently extract intelligene from documents without tedious manual review – identification details, associated entities, usage trails, etc. FOCA can recursively scan documents and websites to build an ‘organization pyramid’ visualizing structures.

Collected information gets presented in different web interface tabs without much stitching requiring manual analysis. While the CLI and UI leave much to be desired, FOCA’s automated document data extraction provides invaluable help to a SOC’s OSINT toolkit – both for insider threat hunting as well as gathering supplementary incident data.

Shodan

Shodan is an Open Source Intelligent search engine that aggregates intelligence on Internet-connected devices and systems. We frequently leverage Shodan for gathering insights on IPs, domains, technologies, vulnerabilities etc.

Pros:

Cons:

Shodan offers a bird’s eye view of global Internet-connected assets by continuously scanning and recording information from public-facing systems. This includes details on open ports, banners, services, technologies, physical devices, etc.

Analysts can search Shodan using targeted queries to find information like vulnerable software versions, unpatched systems, misconfigured services, exposed privileged accounts etc. It also provides useful summaries and metadata around owners, locations, vulnerabilities etc.

We find Shodan extremely helpful for the initial reconnaissance about suspicious IPs, domains and systems during threat hunting and investigations. It helps analysts quickly gauge associated weaknesses and pivot to other intelligence sources.

While Shodan’s invaluable visibility comes at a steep pricing, thoughtful usage by skilled analysts makes it an integral component of a modern SOC’s external intelligence capabilities – both for proactive threat analysis as well as incident response.

theHarvester

theHarvester is a handy Open Source Intelligent tool for gathering emails, names, subdomains, IPs, URLs etc. From hundreds of public sources. We utilize theHarvester for automating initial external reconnaissance.

Pros:

Cons:

TheHarvester streamlines early recon by allowing analysts to specify domains, companies or keywords to automatically scan search engines, DNS records, WHOIS databases, PGP repositories, job boards and more to retrieve associated email addresses, hosts, employee names and other preliminary intelligence.

This provides a quick high-level overview of an entity’s online footprint before turning to more specialized tools. All information retrieved gets aggregated in a local HTML report for additional filtering and pivoting.

While theHarvester requires CLI comfort initially and lacks native visualizations, its simplicity, reliability and breadth of public sources tapped makes it an essential early addition to many SOC OSINT gathering workflows. This allows manually focusing higher value customized tools on intelligence leads validated by theHarvester’s automated initial sweeps.

Whois

Whois is a commonly used OSINT lookup that retrieves registration details associated with domains, IPs and other internet infrastructure assets. Our SOC frequently employs Whois for gathering initial intelligence on targets of investigations.

Pros:

Cons:

Analysts use Whois during threat investigation by inputting suspicious domains, IPs, ASNs and email addresses to uncover registration details, including entities behind them, associated locations, relationships, timelines, etc.

While an individual Whois lookup offers quick snapshot of a single target’s attributes, integrating Whois queries into automated reconnaissance workflows allows large-scale collection of metadata associated with infrastructure assets linked to threats.

This registration intelligence offers useful supplementary evidence for pivoting investigations, validating attacker associations, reporting incidents etc. However, reliability of data can vary across registrars and regions.

Given its ubiquity as a basic internet administration lookup protocol, easy availability and ability to reveal hidden cyber threat intelligence breadcrumbs, Whois capabilities are almost universally utilized by SOCs as a low-effort standard component early in most investigative OSINT chains.

Google Dorks

While Google search is ubiquitous, Google Dorks empower mining hidden insights. We construct specialized Google searches using advanced operators to uncover concealed organization intelligence.

Pros:

Cons:

Google Dorks allows creatively exploiting Google’s vast indexed data for security reconnaissance through clever search syntax, operators, and settings. Analysts can thoroughly scan surface web and uncover exposed documents, credentials, sensitive processes, etc. by artfully chaining together Dorks.

For instance, specialized searches can reveal vulnerable systems through unintentional public exposure of login portals, backup files, server manuals etc. Dorked Google searches can also highlight insider threats via exposed employee credentials and data in caches, code repositories, clumsy uploads etc.

While seemingly innocuous in isolation, stitching together insights from thoughtful Google Dorking across people, domains, locations and technologies can unmask hidden relationships and threat intelligence. However, this requires nuanced chaining of search queries while accounting for usage limits.

Through its unparalleled access to global indexed data, ingenious usage of Google Dorks serves as a free public baseline for many SOC OSINT explorations – both for proactively hunting hidden threats as well as speeding up incident investigation through exposure research.

Creepy

Creepy is a niche Open Source Intelligent T tool that we leverage for gathering target location intelligence from images and social media profiles. Creepy helps analysts visually map subject movements and activities.

Pros:

Cons:

Creepy allows analysts to input a social media profile or feed URL to automatically scrape, extract, and map all embedded geotagged images to a visual timeline map. This reveals movements and events associated with subjects over time.

Analysts can analyze patterns for personal or work locations, establish lifestyle routines, track relationships, identify gaps and anomalies warranting closer investigation, etc. Geofencing can alert on movements into areas of interest.

However, legal considerations necessitate caution – while ostensibly public posts get analyzed, informed consent is still requisite especially under certain privacy regimes. Outdated libraries also constrain current reliability.

Still, with due discretion, Creepy’s unique ability to transform scattered public location embedding into consolidated individual movement intelligence makes it a useful addition to an SOC OSINT toolkit – both for insider threat monitoring as well as contextualizing external incidents.

OSINT Framework

OSINT Framework serves as an invaluable public resource providing a central directory of OSINT tools and sources conveniently categorized by data type. Our SOC contributors continuously explore and evaluate additions to enhance our capabilities.

Pros:

Cons:

OSINT Framework organizes over 450 OSINT tools at time of writing spanning categories like networks, email, usernames, documents, imagery, locations etc. This drastically accelerates discovering capabilities aligned to analytical needs.

Framework curation helps benchmark tool coverage, capabilities, and gaps – both suggesting substitutions as well as areas warranting custom inhouse solutions. Competitive analysis ensures availability of best tools for evolving needs in a fragmented vendor space.

The reference architecture provides SOC teams a template to assemble tailored OSINT pipelines by picking and choosing components. However, tool capabilities still require hands-on examination before integration given the open nature of additions.

By democratizing access through structured knowledge sharing on public platforms, resources like the OSINT Framework underpin advancement of open-source intelligence tradecraft.

Subnet Calculator

While simple, Subnet Calculator provides an invaluable utility for expanding target netblocks during threat investigations. We heavily utilize it for pivoting to additional IPs within identified suspicious subnets.

Pros:

Cons:

During incident response, analysts frequently need to quickly analyze all addresses within identified malicious IP subnets for associated hosts, relationships, vulnerabilities etc. Manually assessing IP ranges is tedious.

Subnet Calculator allows conveniently specifying a subnet like 185.19.0.0/22 and instantly expanding it to reveal all usable IP addresses from 185.19.0.1 to 185.19.3.254. This expanded set gets exported for threat intelligence analysis.

The calculator also allows assessing subnet capacity, splitting subnets, finding address locations etc. helping analysts better understand the structure of identified suspicious infrastructures.

While serving a narrowly focused capability, Subnet Calculator’s utility in speeding up tedious yet ubiquitous threat infrastructure enumeration makes it an indispensable staple that enhances other SOC OSINT tools.

Trape

Trape is an open-source people tracking tool that covertly follows targets as they browse the web and aggregates trailing information left behind. We tactically use Trape for gathering intelligence on high priority threats like active attackers.

Pros:

Cons:

Trape allows analysts to stealthily track web movements of specified users by leveraging website session identifiers and out-of-band browser fingerprints. As targets browse the internet unaware, Trape silently observes page visits, device types, locations, etc. building detailed user behavior dossiers.

While ostensibly analyzing public activities, informed consent considerations for such adversarial tracking necessitate caution and discretion especially under expansive privacy regimes. Data sets also require securing given proliferation risks.

Still, through covert yet legal inspection of web movements, Trape earns its place in an SOC OSINT toolkit against sophisticated threats where traditional visibility falls short and ethics permit its circumscribed application.

TweetDeck

TweetDeck serves as a customizable social media dashboard for curating real-time streams from Twitter, Facebook, Instagram etc. Our SOC leverages TweetDeck to monitor relevant threats, campaigns and events.

Pros:

Cons:

TweetDeck allows creation of customizable feeds and alerts tuned to organization interests across social networks. Analysts can dial in on threat actors, exploit chatter, vulnerability discussions, brand threats, leaks, etc. in a single screen.

Keyword warnings, geo-tags and user filters help filter signal from noise. Collected intelligence gets shared with relevant incident response teams for mitigation like account takeover alerts, early exploitation threats etc.

However, analysts need to cautiously validate credibility of user posts given possibility of leaks, misinformation and impersonators. Also discretion is important as seemingly public posts likely still expect privacy.

Still, TweetDeck’s specialty in consolidating real-time global cross-platform social media discourse makes it a quick and convenient finger on the pulse of latest external threats – serving both proactive threat visibility for SOCs as well as speeding up contextual response.

Who Else Can Use These OSINT Tools?

While we have focused on OSINT tools in the context of our security operations center (SOC), these techniques and tools have diverse applications across multiple functions. For example, penetration testers and bug bounty hunters can gather public intelligence on an organization to help prioritize testing based on exposed technologies and vulnerabilities. Cyber threat analysts can research details on the latest hacking techniques, campaigns, and vulnerable software versions to improve threat detection. Incident responders can quickly pivot on indicators like IP addresses or file hashes to establish the scope of security compromises. Security researchers can monitor hacking forums and code dumps to understand adversary tactics. Investigators and journalists can ethically piece together public information to uncover non-obvious details about inquiry subjects. Business analysts can monitor competitors for technology shifts and product roadmaps to advise internal stakeholders. Additionally, geopolitical analysts can leverage foreign language sources to better grasp localized perspectives impacting global interests and events.

Well, you no need to be in security landscape to use these tools. Since, these are open source tools anybody can use these Open Source Intelligent resources. OSINT (Open Source Intelligence) tools are versatile and can be utilized by a diverse range of individuals and organizations for various purposes. Cybersecurity professionals commonly use these tools to identify potential threats and vulnerabilities in networks and systems. Law enforcement agencies leverage them for criminal investigations and monitoring activities of interest. Journalists and researchers find OSINT tools invaluable for uncovering stories, fact-checking, and gathering detailed information for their work.

Intelligence agencies use these tools for national security purposes, including monitoring international developments and potential threats. In the corporate sector, intelligence and risk analysts employ OSINT for competitive intelligence, market analysis, and due diligence investigations. Human rights organizations and NGOs utilize these resources to document violations, monitor crises, and support their campaigns and operational planning.

Private investigators rely on OSINT for conducting background checks and gathering evidence. Activists use these tools to gather and disseminate information on various issues, while IT and network administrators use them to stay informed about the latest cybersecurity threats.

Each of these groups utilizes OSINT tools for different purposes, ranging from security assessments and criminal investigations to research, journalism, and educational activities. It’s important for all users to consider ethical and legal guidelines when using these tools.

Wrap Up

Open Source Intelligence or OSINT refers to the techniques leveraged to legally and ethically gather and analyze publicly available information from open sources. The data compiled can include details on threats, adversaries, vulnerabilities, technologies, movements, motivations and more.

OSINT is increasingly being utilized by security teams to generate external context and intelligence not available from within organizational security tools and data sets alone. We outlined why thoughtfully integrating OSINT strengthens SOCs with greater visibility for anticipating, detecting, responding and recovering from modern security threats spanning the cyber kill chain.

We covered the types of technical intelligence SOCs typically seek via OSINT like details on latest adversary campaigns, vulnerable infrastructure, compromised credentials and breach indicators. We also enumerated some key considerations for collecting this data ethically and ensuring protection once aggregated.

In the dynamic web-driven world we operate in, no organization has the luxury of relying solely on internal threat visibility. Public data holds immense clues; and as malicious actors increasingly leverage OSINT to target victims, defender security teams must out-innovate through better open data capabilities.

We listed some popular OSINT tools employed in our SOC – both free as well as paid – along with their key capabilities and limitations. While individual tools have particular strengths, selectively combining OSINT techniques provides invaluable visibility informing risk-prioritized response. Care needs to be taken to ensure legal and ethical usage.

With surface web, deep web and dark web forming a virtual intelligence mosaic, blindspots can prove catastrophic. A vigilant 360 degree OSINT-enhanced cyber view offers few places for threats to hide. Controlled usage also allows gathering invaluable public data intelligence to convince traditionally data-averse leadership on approaching security investments.

What other readers have found as useful additions to their security OSINT toolkits? We invite you to share your suggestions and comments below to help strengthen community knowledge. Thanks for reading this post. Please share this post and help secure the digital world. Visit our website, thesecmaster.com, and our social media page on Facebook, LinkedIn, Twitter, Telegram, Tumblr, Medium, and Instagram and subscribe to receive updates like this.  

Read More: