Crawl Diagnostics returning duplicate content based on session id
-
I'm just starting to dig into crawl diagnostics and it is returning quite a few errors. Primarily, the crawl is indicating duplicate content (page titles, meta tags, etc), because of a session id in the URL.
I have set-up a URL parameter in Google Webmaster Tools to help Google recognize the existence of this session id. Is there any way to tell the SEOMoz spider the same thing? I'd like to get rid of these errors since I've already handled them for the most part.
-
You the man! Thanks!
-
Hi Cody,
The best way is to block Rogerbot within your Robots.txt from crawling specific pages of your site. In your case protecting Rogerbot from seeing the pages with a session ID.
More information could be found here on Rogerbot.Be cautious and test it out, but the lines you would have to add to your Robots.txt are probably:
User-agent: rogerbot
Disallow: /*sessionidHope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Drupal 8 tags and categories cause duplicate content shown in MOZ
Hi all, There is something difficult to trace that is causing duplicate content that is related to categories and tags i.e. https://example.com/contact Associated Pages https://example.com/tags/business https://example.com/taxonomy/term/41 example 2 https://example.com/category/example-category-1 Associated Pages https://example.com/category/occupiers-liability example 3 https://example.com/tags/test https://example.com/tags/test-2 Above two pages display same content (maybe due to similar posts feature) My question here is: Is this caused by Drupal website misconfiguration (or one of its modules) since website uses similar posts feature or it's something else. Duplicate content for example.com/index.php issue has been solved by redirects. Should something similar be done in case of tags / categories? Any discussion / suggestions on that matter are greatly appreciated. Thank you.
Moz Pro | | Optimal_Strategies0 -
Duplicated content generated by keywords
Hello! I am kind of new to SEO and MOZ, so I really need your help to understand why some of my keywords generate duplicated content. Meaning, in my blog posts I use various SEO keywords. It shows up that in my MOZ crawl analysis, I have these keywords listed as duplicates: so two/three different keywords are pointing to the same articles and are considered duplicates? I really don't understand how it is possible. Did it also happen to you? I highly appreciate it. Thank you
Moz Pro | | DianaC0 -
Is it normal for Moz to report on nofollow pages in crawl diagnostics?
I have a dev version of my website, for example, devwww.website.com. The htaccess page has a noindex and nofollow request, but I got crawl issues reported from these pages in my Moz report. Does this mean that I don't have the development site hidden from search like I thought I did?
Moz Pro | | houstonbrooke0 -
How do a run a MOZ crawl of my site before waiting for the scheduled weekly crawl?
Greetings: I have just updated my site and would like to run a crawl immediately. How can I do so before waiting for the next MOZ crawl? Thanks,
Moz Pro | | Kingalan1
Alan0 -
Crawl Diagnostics shows two title and meta tag errors but they are false positives.
I got one hit each on "Missing Meta Description Tag" and "Title Missing or Empty" but in the source of my page they are clearly there: <title>Protein Powder | Compare and Get the Best Prices</title> <meta name="keywords" content="protein powder, whey protein, protein supplement, whey protein isolate, hydrolyzed whey" /> I understand there are conventions which may or may not be followed by Drupal (I read an earlier question where ordering and W3C conventions were suggested) but i'm not sure how to fix them given Drupal will just overwrite any hand editing the next time something is built and importantly, I can't get the crawl to work on cue - it works on the automatic once a week crawl in the main campaign summary but every time I've specifically used the Crawl Test tool it gives me a "There was an error submitting your request to the crawler. Please try again later" so I can't really test any changes. Given Google seems to be recognising the title tag - ie showing it in the results - Do I put this down as seomoz just not working? Kind Regards, Brian
Moz Pro | | btrr690 -
SEOmoz crawl error questions
I just got my first seomoz crawl report and was shocked at all the errors it generated. I looked into it and saw 7200 crawl errors. Most of them are duplicate page titles and duplicate page content. I clicked into the report and found that 97% of the errors were going off of one page It has ttp://legendzelda.net/forums/index.php/members/page__sort_key__joined__sort_order__asc__max_results__20 http://legendzelda.net/forums/index.php/members/page__sort_key__joined__sort_order__asc__max_results__20__quickjump__A__name_box__begins__name__A__quickjump__E etc Has 20 pages of slight variations of this link. It is all my members list or a search of my members list so it is not really duplicate content or anything. How can I get these errors to go away and make search my site is not taking a hit? The forum software I use is IPB.
Moz Pro | | NoahGlaser780 -
Crawl Diagnostics and missing meta tags on noindex blog pages
Hi Guys/Gals We do love the Crawl Diagnostics, but do find the missing meta tags ("Missing Meta Description" Tag in this case) somewhat spammy. We use the "All in One SEO Pack" for our blog and it does stick in noindex,follow (as it should) on the pages that is of no use to us. "2008/04/page/2/" and the likes. Maybe I'm wrong but should the Diagnostics tool not respect the noindex tag and just ignore any warnings, since it should really mean that these pages are NOT included in the search index. Meaning that the other meta tags are really useless. Any thoughts?
Moz Pro | | sfseo0 -
Crawl completed but no report available for download?
I put 2 crawls in the same day. One came back and delivered a report that I could download. The other one is completed (says so on the page) but there's no way for me to download the report. How do I get a hold of it? Thanks!
Moz Pro | | LiliArancibia0