There are numerous factors you could have to have to locate each of the URLs on an internet site, but your actual target will ascertain what you’re looking for. For example, you might want to:
Establish every single indexed URL to research issues like cannibalization or index bloat
Accumulate existing and historic URLs Google has seen, specifically for website migrations
Locate all 404 URLs to Recuperate from put up-migration problems
In Just about every situation, a single tool gained’t Supply you with all the things you require. However, Google Look for Console isn’t exhaustive, plus a “site:instance.com” lookup is restricted and challenging to extract data from.
Within this submit, I’ll walk you thru some instruments to develop your URL checklist and just before deduplicating the info using a spreadsheet or Jupyter Notebook, based on your internet site’s dimension.
Outdated sitemaps and crawl exports
For those who’re searching for URLs that disappeared with the live web site a short while ago, there’s an opportunity someone on your team may have saved a sitemap file or a crawl export ahead of the improvements had been produced. Should you haven’t presently, look for these information; they are able to normally supply what you would like. But, in the event you’re looking at this, you almost certainly didn't get so Fortunate.
Archive.org
Archive.org
Archive.org is an invaluable Instrument for Website positioning jobs, funded by donations. When you hunt for a domain and choose the “URLs” choice, you can accessibility nearly 10,000 detailed URLs.
However, There are several constraints:
URL Restrict: You could only retrieve around web designer kuala lumpur 10,000 URLs, that's insufficient for larger websites.
High quality: Many URLs may be malformed or reference resource documents (e.g., photos or scripts).
No export option: There isn’t a designed-in approach to export the record.
To bypass the lack of an export button, use a browser scraping plugin like Dataminer.io. Having said that, these limitations suggest Archive.org might not provide a complete solution for bigger web sites. Also, Archive.org doesn’t show irrespective of whether Google indexed a URL—but when Archive.org identified it, there’s a great chance Google did, far too.
Moz Professional
Even though you could possibly usually use a backlink index to discover external websites linking for you, these instruments also find URLs on your internet site in the process.
The best way to use it:
Export your inbound one-way links in Moz Pro to secure a swift and straightforward list of goal URLs from the internet site. Should you’re dealing with a huge Web-site, think about using the Moz API to export knowledge outside of what’s manageable in Excel or Google Sheets.
It’s crucial that you Be aware that Moz Professional doesn’t verify if URLs are indexed or discovered by Google. Having said that, considering that most websites implement precisely the same robots.txt principles to Moz’s bots because they do to Google’s, this process typically performs well as being a proxy for Googlebot’s discoverability.
Google Lookup Console
Google Search Console provides quite a few valuable sources for making your list of URLs.
Links stories:
Similar to Moz Professional, the Links section delivers exportable lists of target URLs. Sad to say, these exports are capped at 1,000 URLs Every single. You'll be able to utilize filters for precise internet pages, but given that filters don’t implement to your export, you might need to count on browser scraping resources—limited to five hundred filtered URLs at a time. Not great.
General performance → Search Results:
This export offers you a list of internet pages obtaining lookup impressions. When the export is restricted, you can use Google Research Console API for greater datasets. There's also totally free Google Sheets plugins that simplify pulling additional considerable information.
Indexing → Internet pages report:
This segment presents exports filtered by situation form, while they are also constrained in scope.
Google Analytics
Google Analytics
The Engagement → Web pages and Screens default report in GA4 is a wonderful resource for amassing URLs, with a generous limit of 100,000 URLs.
Better still, it is possible to implement filters to build different URL lists, effectively surpassing the 100k Restrict. Such as, in order to export only weblog URLs, adhere to these actions:
Move 1: Increase a section for the report
Move 2: Simply click “Develop a new segment.”
Phase 3: Outline the phase that has a narrower URL pattern, for instance URLs that contains /weblog/
Take note: URLs found in Google Analytics might not be discoverable by Googlebot or indexed by Google, but they provide valuable insights.
Server log information
Server or CDN log information are Most likely the ultimate Instrument at your disposal. These logs capture an exhaustive checklist of every URL path queried by people, Googlebot, or other bots during the recorded period.
Concerns:
Info dimension: Log information could be enormous, countless internet sites only retain the last two months of information.
Complexity: Analyzing log files is usually complicated, but many instruments can be found to simplify the process.
Blend, and excellent luck
When you’ve collected URLs from all these resources, it’s time to combine them. If your web site is small enough, use Excel or, for more substantial datasets, tools like Google Sheets or Jupyter Notebook. Guarantee all URLs are consistently formatted, then deduplicate the checklist.
And voilà—you now have a comprehensive listing of present-day, old, and archived URLs. Great luck!