The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Canonical question - Would this create some sort of crawler redirection loop?
What happens if a canonical link, links to the url with / but the main url does not have the / For example: rel="canonical" href="https://www.exampleURL.co.uk/"> Main URL - https://www.exampleURL.co.uk (without the /) 301 Redirect https://www.exampleURL.co.uk/ to https://www.exampleURL.co.uk Would this create some sort of crawler redirection loop?
On-Page Optimization | | Evosite10 -
Not sure if I need to be concerned with duplicate content plus too many links
Someone else supports this site in terms of making changes so I want to make sure that I know what I am talking about before I speak to them about changes. We seem to have a lot of duplicate content and duplicate titles. This is an example http://www.commonwealthcontractors.com/tag/big-data-scientists/ of a duplicate. Do I need to get things changed? The other problem that crops up on reports is too many on page links. I am going to get shot of the block of tags but need to keep the news. Is there much else I can do? Many thanks.
On-Page Optimization | | Niamh20 -
Our urls for adwords are slightly different from current urls presented on site (weused htaccess to help create shorter urls). How important is it that the adwords url match the sitemap url for keywords on those pages?
Hello, We have dynamic urls that we have made into short urls through htaccess and code manipulation. Some of our adwords urls are different from our page urls - for example a) Latest version of page www.abc.com/x-y-z.html b) Previous version of url www.abc.com/x+y+z.html c) raw original version www.abc.com/yyy/zzz?category=X&Product-code=Y etc etc. Would my ranking for keywords on the page improve if I diligently made all of them the same? They all go to the same page even now, and no 404 errors or anything. Thanks Sam
On-Page Optimization | | samgold0 -
Avoid Keyword Self-Cannibalization. Please Help
This has been addressed plenty of times but I cannot find an example that addresses my issue so that is why I am posting this. I am getting the following Self-Cannibalization error for my Homepage and I am trying to fix it but I just don't see it or maybe I do not understand this correctly. http://fake-diploma.com Keyword Fake Diploma Cannibalizing link"How to make a fake diploma", "How to get a fake diploma", "Making a Fake High School Diploma", "Fake Diploma Template", and "Framing your fake diploma" My understanding is that for Self-Cannibalization to occur I would have to have a link on this page pointing to another page using "Fake Diploma" as my anchor text since I want this page to rank for Fake Diploma. I do have the left sidebar which contains my most recent post and my of my titles do include Fake Diploma but I thought that since they were not an Exact Match and are actually longtail keywords that they did not matter or cause Self-Cannibalization. Am I wrong? How do I fix this.
On-Page Optimization | | diplomajim0 -
Help with the indexation of my page
Hi all, I have a problem with my website. When writing site:www.pinesapiensa.com there're no pages indexed although the webmaster tools tells me that the sitemap file has been processed in 13 May and the number of indexed paged are 21. ¿What could be happening? I have to mention that there are two domains "www.piensapiensa.es" and "www.piensapiensa.com" addressing the same website and there's a redirection from piensapiensa.com to piensapiensa.com but it doesn't work properly. Thanks
On-Page Optimization | | juanmiguelcr0 -
Duplicate content because of content scrapping - please help
We manage brands websites in a very competitive industry that have thousands of affiliate links We see that more and more websites (mainly affiliates websites) are scrapping our brand websites content and it generate many duplicate content (but most of them link to us back with an affiliate link). Our brand websites still rank for any sentence in brackets you search in Google, Will this duplicate content hurt our brand websites ? If yes, should we take some preventive actions ? We are not able to add ongoing UGC or additional text to all our duplicate content and trying to stop those websites of stealing our content is like playing cat and mouse... Thanks for your advices
On-Page Optimization | | Tit0 -
Telephone numbers on page getting classed as 404s by SEOMoz
Hi there, I have a number of clients who have their telephone numbers on their sites (understandably of course!) and SEOmoz is classing them as links and therefore a 404 in the crawl software. The protocol is added in the code so if viewing the page on a mobile you can call the client. Should I be doing anything else? Webmaster does not pick these up as 404s so I am wondering if this is an SEOmoz bug or that I should be adding a no-follow? Thanks jT
On-Page Optimization | | Switch_Digital0 -
Wordpress categories tags and robots.txt
I am relatively new at this and see a variety of people that seem to disagree on if you should block google from indexing category and tag pages through robot.txt or no-follow because of google viewing it as duplicate content. I tryst this communities answers over the web at large obviosly, so what do you all think? Thanks, Steven
On-Page Optimization | | sfmatthews0