> For the complete documentation index, see [llms.txt](https://davidtancredi.gitbook.io/pentesting-notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery.md).

# Content Discovery

- [Katana](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/katana.md): project discovery crawler
- [Cariddi](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/cariddi.md): Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
- [gobuster](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/gobuster.md): sudo apt install gobuster (if not installed)
- [dirb](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/dirb.md): Assorted commands
- [Gospider](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/gospider.md): Crawler in go
- [Hakrawler](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/hakrawler.md)
- [webpalm](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/webpalm.md): tool written in go to visualize tree structure of websites
- [OpenDoor](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/opendoor.md): OWASP WEB Directory Scanner
- [Feroxbuster](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/feroxbuster.md): Crawler in Rust
- [httpx](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/httpx.md): httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.
- [JS File Analysis and Scraping](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/js-file-analysis-and-scraping.md): Tips and tricks to extrapolate juicy info from JS files
- [Waymore](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/waymore.md): Grab Urls from wayback machine
- [Waybackurls](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/waybackurls.md): Grab waybackmachine urls , install from source if go install fails
- [Urlfinder](https://davidtancredi.gitbook.io/pentesting-notes/r3dcl1ff/webapp-pentest/content-discovery/urlfinder.md): Quick tool from Project Discovery