Truncate page URLs
-
We have some pages (for example a contact us form) for which the URL is modified by the CMS depending on the referring page (this helps to put the form submission in context for the sales reps who get the contact submission).
The SEOmoz crawler considers each URL a new page -- and so numbers like in diagnostics are all inflated as the same page is listed multiple times (e.g. for too many links)
Is there a setting to change what the crawler considers to be the same page?
Here are two URLs for the same page that the reports treat as separate pages:
http://www.spirent.com/About-Us/Contact_us.aspx?referurl=0F528F4D703D8BB3523738D6373AA8AD
http://www.spirent.com/About-Us/Contact_us.aspx?referurl=10ACDA6055244E369395223437FDCF30
The page is actually: http://www.spirent.com/About-Us/Contact_us.aspx
Thanks
Ken
-
As you can see here, this is an issue as Google are indexing many variations of the same page although this means that somewhere is linking to them unless your site is set up so that even a crawler passing through links to your contact page is creating the query parameter in the URL's.
To resolve this, you need to add the following to your robots.txt file:-
Disallow: ?referurl=
This will prevent any URL's passing that query parameter from getting crawled and indexed ensuring that only the originals of the pages will appear in search engines and not flag as duplicate content.
Hopefully, someone from SEOmoz can add as to whether there is an option for obeying robots.txt directives within their crawler so that these URL's are not listed as I'm not sure.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Solving URL Too Long Issues
Moz.com is reporting that many URL's are to long, these particularly affect Product URL's where the URL is typically https://www.domainname.com/collections/category-name/products/product-name, (You guessed it we're using Shopify). However, we use Canonicals that ignore all most of the URL and are just structured https://www.domainname.com/product-name, so Google should be reading the Canonical and not the long-winded version. However, Moz cannot seem to spot this... does anyone else have this problem and how to solve so that we can satisfy the Moz.com crawl engine?
Moz Pro | | Tildenet0 -
Moz page optimizer - avoid keyword stuffing
Hi Moz community, A quesiton about our dutch travel insurance website. Two weeks ago we had a ranking drop from 17 to position 35. The project is still fresh (we started in may this year). I quess penguin 4.0 didn't like our rapid linkbuilding. But we also receive a message in page optimization tool for keyword stuffing. When we analyze the homepage www.reisverzekering.net for reisverzekering (meaning travel insurance) Moz suggests to avoid keywords stuffing. But I dont understand, we don't use the keyword that much and Yoast doesnt give this message. How much key
Moz Pro | | remkoallertz0 -
Page with "Missing Title Tag" isn't a page
Hello, I am going through the various errors that the Moz Pro Crawl report and some non-existent pages keep coming up in the report. For example, one error category is "Missing Title Tag" with one page identified. But this page http://www.immigroup.com/news/“http%3A/crs.yorku.ca”?page=2 isn't real. It would have been a 404 were there not a redirect for everything that is /news/gobbledygook to /news. So my question is: when moz (or GA for that matter) identifies these pages as "real" and having errors, do I need to take this seriously? And what do I do about it? Thanks! George
Moz Pro | | canadageorge0 -
Backlink of competitor inner page
How to understand from where the link to a competitor's internal page http://www.exampledomain.com/contest/book are coming? I mean, we want promote the same "event" but we want understanding where the competitor is already present. Open Site Explorer doesn't have any data for the website listed but I'm sure that the page has a lot of backlinks (I've verified the first 50 manually in Google.it) Many Thanks for all your advices.
Moz Pro | | YESdesign0 -
Concerned About Individual Pages
Okay. I've setup a campaign for www.site.com and given a list of keywords. So after the initial crawl we'll have some results. What I'm looking for tho is how do individual pages on my site rank for the list of keywords given. And then be able to go to a screen in seomoz with data for that particular page with recommendations and stuff like that. Is this what's going to happen or do I need to create a campaign for each url i want to track? If all will work as I'd like in the example above, should I then add the second list of keywords that some other pages should rank for? Will it get to be a big mess or can I relate the keywords to pages in some way? It seems like what I'm looking for is what this program should be... Thanks!
Moz Pro | | martJ0 -
Only one page has been crawled
I am running a campaing for three weeks now and first two crawls was ok but the last one is showing only one page crawled. the subdomain I am tracking is: www.cubaenmiami.com I have everything correct in my site. Regards Alex
Moz Pro | | esencia0 -
My campaigns are not analyzing all my pages.
Hi I created a campaign against http://www.universalpr.com, and this campaign reports that only one page has been crawled. This site uses a jsvascript redirect to the real page which can be found through the following: www.universalpr.com/wps/portal/universal/univhome/!ut/p/c5/04_SB8K8xLLM9MSSzPy8xBz9CP0os_hQdwtfCydDRwN_Jw9LA0-LAOPQYCdDI_9QY_1wkA6zeAMcwNFA388jPzdVPzi1WL8gO68cANNcdLU!/dl3/d3/L2dBISEvZ0FBIS9nQSEh/ Now I also attempted to create a campaign against this page in case that the javascript redirect was breaking things, but that campaign also reported 1 page crawled. Can anyone instruct me as to what I'm doing wrong? Thank you
Moz Pro | | jcmoreno0 -
In page analysis
Hello, Seomoz did a crawl of my whole page, but under in page analysis grade rankings there have been no crawls? how can I initiate one?
Moz Pro | | AdamGymyGym0