DNS enumeration is considered one of the most important information-gathering techniques. Penetration testers spent a lot of time to find all available subdomains of a target, as they unlock multiple new attack opportunities.
There are plenty of tools out there that can make your life easier. However, as time passes, the tool number is increased so much that it triggers a headache in a junior pen-tester.
The purpose of this article is to present the most common sub-domain discovery methods and then evaluate as many tools as possible in real-life scenarios.
How subdomain finder works
Each tool uses different methods to enumerate subdomains. The article will not cover in-depth each method. Methods that depend on external input will be used in a fairly way. For example, all tools will be tested with the same wordlist for brute-forcing.
The most common methods for subdomain enumeration are :
Search Engines
One of the most common subdomain enumeration techniques is via using search engines (Google, Bing, etc.). All search engines use thousands of spider bots to crawl the internet constantly and keep track of billion of subdomains.
Google is powerful when enumerating subdomains like this : site:example_domain.com
Many times the results are filled with the same subdomain.
In case you want to exclude similar subdomain results use the minus flag : -site:subdomain.example.com .
Search engine enumeration is passive and not a single packet is sent to the target!
DNS Historical Data
Historical data can be a valuable source of old DNS records. Securitytrails.com holds over a 3.4 trillion DNS records and it also has a separate subdomain enumeration function.
DNS Records
DNS servers create a DNS record to provide important information about a domain or hostname, particularly its current IP address. The most common DNS record types are:
- Address Mapping record (A Record)—also known as a DNS host record, stores a hostname and its corresponding IPv4 address.
- IP Version 6 Address record (AAAA Record)—stores a hostname and its corresponding IPv6 address.
- Canonical Name record (CNAME Record)—can be used to alias a hostname to another hostname. When a DNS client requests a record that contains a CNAME, which points to another hostname, the DNS resolution process is repeated with the new hostname.
- Mail exchanger record (MX Record)—specifies an SMTP email server for the domain, used to route outgoing emails to an email server.
- Name Server records (NS Record)—specifies that a DNS Zone, such as “example.com” is delegated to a specific Authoritative Name Server, and provides the address of the name server.
- Reverse-lookup Pointer records (PTR Record)—allows a DNS resolver to provide an IP address and receive a hostname (reverse DNS lookup).
- Certificate record (CERT Record)—stores encryption certificates—PKIX, SPKI, PGP, and so on.
- Service Location (SRV Record)—a service location record, like MX but for other communication protocols.
- Text Record (TXT Record)—typically carries machine-readable data such as opportunistic encryption, sender policy framework, DKIM, DMARC, etc.
- Start of Authority (SOA Record)—this record appears at the beginning of a DNS zone file, and indicates the Authoritative Name Server for the current DNS zone, contact details for the domain administrator, domain serial number, and information on how frequently DNS information for this zone should be refreshed.
All these records can easily be retrieved with an online service or via the Linux command-line tool dig as shown below :
Dig can easily find all youtube’s nameserver by using the NS parameter, short flag adjusts the answer size :
# dig youtube.com NS +short ns1.google.com. ns2.google.com. ns3.google.com. ns4.google.com.
Now we can query youtube’s nameserver for all DNS records with ANY parameter
# dig @ns3.google.com youtube.com ANY ; <<>> DiG 9.11.5-P1-1-Debian <<>> @ns3.google.com youtube.com ANY ; (2 servers found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61955 ;; flags: qr aa rd; QUERY: 1, ANSWER: 16, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;youtube.com. IN ANY ;; ANSWER SECTION: youtube.com. 300 IN AAAA 2a00:1450:4017:800::200e youtube.com. 300 IN A 172.217.17.174 youtube.com. 600 IN MX 10 aspmx.l.google.com. youtube.com. 600 IN MX 30 alt2.aspmx.l.google.com. youtube.com. 3600 IN TXT "facebook-domain-verification=64jdes7le4h7e7lfpi22rijygx58j1" youtube.com. 345600 IN NS ns3.google.com. youtube.com. 600 IN MX 50 alt4.aspmx.l.google.com. youtube.com. 600 IN MX 40 alt3.aspmx.l.google.com. youtube.com. 3600 IN TXT "google-site-verification=OQz60vR-YapmaVrafWCALpPyA8eKJKssRhfIrzM-DJI" youtube.com. 345600 IN NS ns1.google.com. youtube.com. 345600 IN NS ns4.google.com. youtube.com. 600 IN MX 20 alt1.aspmx.l.google.com. youtube.com. 60 IN SOA ns1.google.com. dns-admin.google.com. 271534513 900 900 1800 60 youtube.com. 3600 IN TXT "v=spf1 include:google.com mx -all" youtube.com. 86400 IN CAA 0 issue "pki.goog" youtube.com. 345600 IN NS ns2.google.com. ;; Query time: 78 msec ;; SERVER: 216.239.36.10#53(216.239.36.10) ;; WHEN: Sat Sep 28 13:19:16 DST 2019 ;; MSG SIZE rcvd: 543
DNS Zone Transfers
DNS service is critical for the whole internet and plenty of DNS records are replicated in multiple servers for redundancy purposes. The easiest way to copy records between DNS servers is via zone transfers.
Zone transfers should only be allowed between trusted hosts. Do not forget to set the appropriate IPs in the access lists at # /etc/named.conf of your DNS server.
If a DNS server is misconfigured then the attacker may get confidential information of your internal or external hosts. The following one-liner can check if the first DNS server of the target domain is vulnerable.
#target=zonetransfer.me;ns=$(dig @1.1.1.1 $target NS +short|head -1);dig @$ns $target AXFR
; <<>> DiG 9.11.5-P1-1-Debian <<>> @nsztm2.digi.ninja. zonetransfer.me AXFR
; (1 server found)
;; global options: +cmd
zonetransfer.me. 7200 IN SOA nsztm1.digi.ninja. robin.digi.ninja. 2019031901 172800 900 1209600 3600
zonetransfer.me. 300 IN HINFO "Casio fx-700G" "Windows XP"
zonetransfer.me. 301 IN TXT "google-site-verification=tyP28J7JAUHA9fw2sHXMgcCC0I6XBmmoVi04VlMewxA"
zonetransfer.me. 7200 IN MX 0 ASPMX.L.GOOGLE.COM.
zonetransfer.me. 7200 IN MX 10 ALT1.ASPMX.L.GOOGLE.COM.
zonetransfer.me. 7200 IN MX 10 ALT2.ASPMX.L.GOOGLE.COM.
zonetransfer.me. 7200 IN MX 20 ASPMX2.GOOGLEMAIL.COM.
zonetransfer.me. 7200 IN MX 20 ASPMX3.GOOGLEMAIL.COM.
zonetransfer.me. 7200 IN MX 20 ASPMX4.GOOGLEMAIL.COM.
zonetransfer.me. 7200 IN MX 20 ASPMX5.GOOGLEMAIL.COM.
zonetransfer.me. 7200 IN A 5.196.105.14
zonetransfer.me. 7200 IN NS nsztm1.digi.ninja.
zonetransfer.me. 7200 IN NS nsztm2.digi.ninja.
_sip._tcp.zonetransfer.me. 14000 IN SRV 0 0 5060 www.zonetransfer.me.
14.105.196.5.IN-ADDR.ARPA.zonetransfer.me. 7200 IN PTR www.zonetransfer.me.
asfdbauthdns.zonetransfer.me. 7900 IN AFSDB 1 asfdbbox.zonetransfer.me.
asfdbbox.zonetransfer.me. 7200 IN A 127.0.0.1
asfdbvolume.zonetransfer.me. 7800 IN AFSDB 1 asfdbbox.zonetransfer.me.
canberra-office.zonetransfer.me. 7200 IN A 202.14.81.230
cmdexec.zonetransfer.me. 300 IN TXT "; ls"
contact.zonetransfer.me. 2592000 IN TXT "Remember to call or email Pippa on +44 123 4567890 or pippa@zonetransfer.me when making DNS changes"
dc-office.zonetransfer.me. 7200 IN A 143.228.181.132
deadbeef.zonetransfer.me. 7201 IN AAAA dead:beaf::
dr.zonetransfer.me. 300 IN LOC 53 20 56.558 N 1 38 33.526 W 0.00m 1m 10000m 10m
DZC.zonetransfer.me. 7200 IN TXT "AbCdEfG"
email.zonetransfer.me. 2222 IN NAPTR 1 1 "P" "E2U+email" "" email.zonetransfer.me.zonetransfer.me.
email.zonetransfer.me. 7200 IN A 74.125.206.26
home.zonetransfer.me. 7200 IN A 127.0.0.1
Info.zonetransfer.me. 7200 IN TXT "ZoneTransfer.me service provided by Robin Wood - robin@digi.ninja. See http://digi.ninja/projects/zonetransferme.php for more information."
internal.zonetransfer.me. 300 IN NS intns1.zonetransfer.me.
internal.zonetransfer.me. 300 IN NS intns2.zonetransfer.me.
intns1.zonetransfer.me. 300 IN A 81.4.108.41
intns2.zonetransfer.me. 300 IN A 52.91.28.78
office.zonetransfer.me. 7200 IN A 4.23.39.254
ipv6actnow.org.zonetransfer.me. 7200 IN AAAA 2001:67c:2e8:11::c100:1332
owa.zonetransfer.me. 7200 IN A 207.46.197.32
robinwood.zonetransfer.me. 302 IN TXT "Robin Wood"
rp.zonetransfer.me. 321 IN RP robin.zonetransfer.me. robinwood.zonetransfer.me.
sip.zonetransfer.me. 3333 IN NAPTR 2 3 "P" "E2U+sip" "!^.*$!sip:customer-service@zonetransfer.me!" .
sqli.zonetransfer.me. 300 IN TXT "' or 1=1 --"
sshock.zonetransfer.me. 7200 IN TXT "() { :]}; echo ShellShocked"
staging.zonetransfer.me. 7200 IN CNAME www.sydneyoperahouse.com.
alltcpportsopen.firewall.test.zonetransfer.me. 301 IN A 127.0.0.1
testing.zonetransfer.me. 301 IN CNAME www.zonetransfer.me.
vpn.zonetransfer.me. 4000 IN A 174.36.59.154
www.zonetransfer.me. 7200 IN A 5.196.105.14
xss.zonetransfer.me. 300 IN TXT
zonetransfer.me. 7200 IN SOA nsztm1.digi.ninja. robin.digi.ninja. 2019031901 172800 900 1209600 3600
;; Query time: 190 msec
;; SERVER: 34.225.33.2#53(34.225.33.2)
;; WHEN: Sat Sep 28 14:09:04 DST 2019
;; XFR size: 48 records (messages 1, bytes 1903)
Do not forget to test all your DNS servers!
The expected output of a correct-secure configuration is the following :
target=youtube.com;ns=$(dig @1.1.1.1 $target NS +short|head -1);dig @$ns $target AXFR +short ; Transfer failed.
The best way to tackle zone transfers is via enabling transactions signatures (TSIG).
Brute Force Subdomains
Many tools use brute force to enumerate subdomains. Grab your wordlist, and sequentially try to resolve all combinations. This method can be used recursively and on top of all other methods to detect subdomains of already found subdomains.
Some administrators may think that if they create a wildcard domain ( an A Record of *.domain.com) that they will be protected, as all requests will be successfully answered during the attack. This is misjudgment as the attacker will filter the IP of a 100% never exist domain from the results.
Brute Force tools are as powerful as the used wordlist. All must-have DNS subdomain discovery wordlists can be found on Daniel Miessler’s Seclists.
Reverse DNS Lookup IPs
Reverse DNS lookup is the reverse of a forward DNS lookup. A reverse DNS lookup returns the hostname when you provide an IP.
Forwards DNS: dns.google.com resolved to 8.8.8.8
Reverse DNS: 8.8.8.8 resolved to dns.google.com
Pointer (PTR) records provide what is known as “reverse DNS”. PTR records assign IP addresses to a hostname instead of mapping a hostname to an IP address.
Many times the PTR records will not be as useful as you think. Once again dig with -x flag is our tool, we will do fDNS to youtube.com and then try to rDNS the resolved IP.
# dig youtube.com +short 172.217.20.110 #dig -x 172.217.20.110 +short ams17s01-in-f14.1e100.net. fra02s28-in-f14.1e100.net.
It is common for companies to buy ranges of IPs. Attackers may reverse DNS subnets of IPS near all the known public IPs. For example, if YouTube is resolved into 172.217.20.110 then an attacker may try rDNS the 172.217.20.110/24 subnet to enumerate further subdomains.
Many times PTR records are used to increase the trust of a webmail server. System administrators should use the right PTR records according to their needs and keep their number to the minimum value.
Analyze Secure Certificates
The Subject Alternate Name (SAN) of SSL/TLS certificates can be used to extract domains and subdomain names.
The following script can be used to extract such information via command line :
Installation #git clone https://github.com/appsecco/the-art-of-subdomain-enumeration.git Example SSL certification subdomain enumaration # python san_subdomain_enum.py www.cartier.com www.userguide.tw.cartier.com www.userguide.ru.cartier.com www.userguide.cartier.com www.cartier.com www.au.cartier.com ...
Tools
AMASS
The OWASP Amass Project has developed a tool to help information security professionals perform network mapping of attack surfaces and perform external asset discovery using open source information gathering and active reconnaissance techniques.
Installation
$docker build -t amass https://github.com/OWASP/Amass.git $docker run -v ~/amass:/amass/ amass enum --list $docker run -v ~/amass:/amass/ amass enum -brute -w /wordlists/all.txt -d example.com
DNSDUMPSTER
DNSdumpster.com is an online free domain research tool that can discover hosts related to a domain. Finding visible hosts from the attackers’ perspective is an important part of the security assessment process.
FINDSUBDOMAIN
An online tool that allows security engineers to monitor vulnerabilities, identify gaps and weaknesses to prevent attacks. With FindSubdomains.com , pentesters can check the endpoints for vulnerability, including subdomains in the development environment, technical domains open to the public, and much more. The free version shows up to 50 subdomains.
PENTEST-TOOLS
Pentest-tools.com is a site that includes multiple penetration testing tools. One of them called “Find Subdomains” which has two flavors: a) free and b) paid service.
ASSETFINDER
Assetfinder is a new tool created by Tom Hudson or Tomnomnom in Go. It finds domains and subdomains potentially related to a given domain by checking several resources online ( facebook,virustotal etc).
Installation
$ go get -u github.com/tomnomnom/assetfinder $ go install github.com/tomnomnom/assetfinder
Usage
$assetfinder [--subs-only] <domain> $assetfinder --subs-only example.com www.example.com example.com m.example.com dev.example.com products.example.com support.example.com privatee191.example.com fireeye-testevent.example.com gerendes.cherochk100.example.com ...
SUBLIST3R
Sublist3r is a python tool designed to enumerate subdomains of websites using OSINT. It helps penetration testers and bug hunters collect and gather subdomains for the domain they are targeting. Sublist3r enumerates subdomains using many search engines such as Google, Yahoo, Bing, Baidu and Ask. Sublist3r also enumerates subdomains using Netcraft, Virustotal, ThreatCrowd, DNSdumpster and ReverseDNS.
Usage
root@kali:~# sublist3r -h
usage: sublist3r [-h] -d DOMAIN [-b [BRUTEFORCE]] [-p PORTS] [-v [VERBOSE]]
[-t THREADS] [-e ENGINES] [-o OUTPUT]
OPTIONS:
-h, --help show this help message and exit
-d DOMAIN, --domain DOMAIN
Domain name to enumerate it s subdomains
-b [BRUTEFORCE], --bruteforce [BRUTEFORCE]
Enable the subbrute bruteforce module
-p PORTS, --ports PORTS
Scan the found subdomains against specified tcp ports
-v [VERBOSE], --verbose [VERBOSE]
Enable Verbosity and display results in realtime
-t THREADS, --threads THREADS
Number of threads to use for subbrute bruteforce
-e ENGINES, --engines ENGINES
Specify a comma-separated list of search engines
-o OUTPUT, --output OUTPUT
Save the results to text file
Example: python /usr/share/sublist3r/sublist3r -d google.com
SUBFINDER
SubFinder is a subdomain discovery tool that discovers valid subdomains for websites by using passive online sources. It has a simple modular architecture and has been aimed as a successor to sublist3r project. SubFinder uses Passive Sources, Search Engines, Pastebins, Internet Archives, etc to find subdomains and then it uses a permutation module inspired by altdns to generate permutations and resolve them quickly using a powerful bruteforcing engine. It can also perform plain bruteforce if needed. The tool is highly customizable, and the code is built with a modular approach in mind making it easy to add functionalities and remove errors.
Features Comparison
The following table summarises all the features of the above command line subdomain scanners. The online subdomain tools do not provide the methods used to collect the subdomain, as a result, they are excluded from the feature comparison.
The features comparison table is based on the available documentation of each tool on 24/10/2019.
| FEATURES | AMASS | SUBLIST3R | ASSETFINDER | SUBFINDER |
| Brute Force | ✓ | ✓ | ✓ | |
| Reverse DNS | ✓ | ✓ | ✓ | |
| Zone Transfers | ✓ | |||
| Certificates | ✓ | ✓ | ||
| Ask | ✓ | ✓ | ✓ | |
| Baidu | ✓ | ✓ | ✓ | |
| bing | ✓ | ✓ | ✓ | |
| DNSDumpster | ✓ | ✓ | ✓ | |
| DNSTable | ✓ | ✓ | ||
| Dogpile | ✓ | ✓ | ||
| Exalead | ✓ | ✓ | ||
| ✓ | ✓ | |||
| HackerOne | ✓ | |||
| Netcraft | ✓ | ✓ | ✓ | |
| IPV4info | ✓ | ✓ | ||
| PTRArchive | ✓ | ✓ | ||
| Riddler | ✓ | ✓ | ||
| SiteDossier | ✓ | ✓ | ||
| ViewDNS | ✓ | |||
| Yahoo | ✓ | ✓ | ✓ | |
| AlienVault | ✓ | |||
| BinaryEdge | ✓ | |||
| BufferOver | ✓ | |||
| CIRCL | ✓ | |||
| CommonCrawl | ✓ | ✓ | ||
| DNSDB | ✓ | ✓ | ||
| HackerTarget | ✓ | ✓ | ✓ | |
| Mnemonic | ✓ | |||
| NetworksDB | ✓ | |||
| PassiveTotal | ✓ | ✓ | ||
| Pastebin | ✓ | |||
| RADb | ✓ | |||
| Robtex | ✓ | |||
| SecurityTrails | ✓ | ✓ | ||
| ShadowServer | ✓ | |||
| Shodan | ✓ | ✓ | ||
| Spyse | ✓ | |||
| Sublist3rAPI | ✓ | |||
| TeamCymru | ✓ | |||
| ThreatCrowd | ✓ | ✓ | ||
| ✓ | ||||
| Umbrella | ✓ | |||
| URLScan | ✓ | |||
| VirusTotal | ✓ | ✓ | ✓ | ✓ |
| WhoisXML | ✓ | |||
| ArchiveIt | ✓ | ✓ | ||
| ArchiveToday | ✓ | |||
| Arquivo | ✓ | |||
| LoCArchive | ✓ | |||
| OpenUKArchive | ✓ | |||
| UKGovArchive | ✓ | |||
| Wayback | ✓ | ✓ | ✓ | |
| CRT.SH | ✓ | ✓ | ||
| Certspotter | ✓ | ✓ | ||
| Threatcrowd | ✓ | |||
| Bufferover.run | ✓ | |||
| ✓ | ||||
| CertDB | ✓ | ✓ | ||
| FindSubdomains | ✓ | ✓ | ✓ | |
| Censys | ✓ | |||
| Entrust CT-Search | ✓ | |||
| GoogleTER | ✓ | |||
| ThreatMiner | ✓ |
I blog quite often and I really appreciate your information. The article has really peaked my interest.
I am going to bookmark your site and keep checking for new information about once a
week. I opted in for your Feed too.
Pretty nice post. I just stumbled upon your blog and wished
to say that I’ve truly loved browsing your blog posts. After all I’ll be
subscribing to your feed and I am hoping you write again very
soon!
OWASP Amass also employs CRT.SH, CertSpotter, ThreatCrowd, BufferOver.run, Censys, Entrust CT and Google CT.