You’re staring at a website with hundreds of pages and you need every single URL — for a site audit, a redirect map, a competitor analysis, maybe just your own sanity check. Copy-pasting them one by one is out of the question. The good news? There are several fast ways to pull every URL into a clean list, and most of them are free.
What You’ll Need
The right method depends on whether you own the site or not, how large it is, and how comfortable you are with the command line. I’ll walk through all five so you can pick the one that fits your situation.
METHOD 1
Grab URLs From the XML Sitemap
This is the fastest option — if the site has a sitemap. It requires zero tools and works in about 60 seconds. I always try this first.
Step 1. Navigate to https://example.com/sitemap.xml in your browser. Many sites also use /sitemap_index.xml as the main entry point.
Step 2. If the sitemap loads, press Ctrl+U (or Cmd+U on Mac) to view the raw XML source. Every <loc> tag contains one URL.
Step 3. Copy the page source, paste it into a text editor, and use Find & Replace to strip the XML tags. Or run this one-liner in your browser console to extract every URL instantly:
// Paste into browser console on the sitemap page
const urls = [...document.querySelectorAll('loc')]
.map(el => el.textContent);
console.log(urls.join('\n'));METHOD 2
Crawl With Screaming Frog SEO Spider
Screaming Frog is the gold standard for site crawling. It’s what I reach for on any site over a few dozen pages. The free version handles up to 500 URLs, which covers most small and mid-size sites.
Step 1. Download and open Screaming Frog. In the top bar, enter the site’s root URL (e.g., https://example.com) and click Start.
Step 2. Let the crawl finish. You’ll see the URL count ticking up in the bottom-left corner. For large sites this can take several minutes — don’t close it.
Step 3. Once done, go to the Internal tab at the top of the results table. This shows every internal URL discovered.
Step 4. Click Export (top right) → Export All → save as CSV. Open the CSV in Excel or Google Sheets — the URL column is column A.
METHOD 3
Use wget on the Command Line
If you’re comfortable with a terminal, wget is a powerful one-liner that crawls a site and logs every URL it visits. No installs beyond wget itself, which comes pre-installed on Linux and macOS (Windows users can install it via GNU wget for Windows).
Run this command in your terminal:
wget --spider -r -nd -nv --no-cache \
--reject-regex "(feed|category|tag|author|\?|#)" \
https://example.com 2>&1 | grep '^--' | \
awk '{ print $3 }' | sort -u > urls.txtThis tells wget to crawl recursively (-r) without downloading files (--spider), then pipes the log through awk to extract the URLs and sort -u to deduplicate them. The result lands in urls.txt in your current directory.
METHOD 4
A Simple Python Script
When I need a reusable solution or want full control over filtering, I write a small Python crawler. This step trips up a lot of people who think it requires advanced coding — it really doesn’t. Here’s a script that does the job cleanly.
Step 1. Install the dependencies if you haven’t already:
pip install requests beautifulsoup4Step 2. Save this as crawl.py and run it with python crawl.py:
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
START_URL = "https://example.com"
visited = set()
to_visit = [START_URL]
domain = urlparse(START_URL).netloc
while to_visit:
url = to_visit.pop(0)
if url in visited:
continue
try:
res = requests.get(url, timeout=8)
visited.add(url)
soup = BeautifulSoup(res.text, "html.parser")
for tag in soup.find_all("a", href=True):
link = urljoin(url, tag["href"])
if urlparse(link).netloc == domain and link not in visited:
to_visit.append(link)
except Exception:
pass
with open("urls.txt", "w") as f:
for u in sorted(visited):
f.write(u + "\n")
print(f"Done. {len(visited)} URLs saved to urls.txt")METHOD 5
Export From Google Search Console
This one only works for sites you own and have verified in Google Search Console. But when it applies, it’s incredibly useful — you’re getting the exact list of URLs Google has indexed, which is often what you actually care about.
Step 1. Log into Google Search Console and select your property.
Step 2. In the left sidebar, click Pages (under the Indexing section). This shows all URLs Google has discovered and indexed.
Step 3. Click the Export button in the top-right corner of the table, then choose Google Sheets, Excel, or CSV. The export includes the full URL list along with indexing status.
Quick Method Comparison
| Method | Needs site access? | Best for | Max URLs (free) |
|---|---|---|---|
| XML Sitemap | No | Quick check, any site | Unlimited |
| Screaming Frog | No | SEO audits, GUIs | 500 URLs |
| wget | No | Command line users | Unlimited |
| Python script | No | Custom filtering | Unlimited |
| Search Console | Yes (owner only) | Indexed URL list + status | Unlimited |
Pro Tips & Common Mistakes to Avoid
📌 What You Can Do Now
You now have five solid methods for extracting every URL from a website — from a 60-second sitemap grab to a fully custom Python crawler. In my experience, combining the sitemap method with a Screaming Frog crawl catches about 99% of URLs on any well-structured site. Once you have your list, the natural next step is filtering it: remove non-canonical URLs, strip query parameters, and sort by page type. That clean list becomes the foundation for a redirect map, a content audit, or an SEO gap analysis.