The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do I need to worry about these 404s after changing permalinks structure?
Not long ago I wanted to change the permalink structure on a blog where the full date was included in the post urls to one where it was just /category/blog-post-title. The site is on a managed host and they setup the redirects for me and then I changed the permalink structure in the WP settings. Everything worked fine after and all old posts now redirect to the new structure. However in Google search console I can see that my 404s report has jumped from a few hundred to several thousands but it seems to be counting the urls paths to elements such as images within the old posts permalink structure eg /2012/6/some-post/some-image-in-post all the old posts themselves and their content redirects fine though. Do I need to worry about these 404s? It annoys me to see so many in my reports like that even if they aren't hurting my site 😞
On-Page Optimization | | linklander0 -
Need Professional Help with Site Structure, Page Authority, and Internal Linking
We are a 10 year + small (1000 pages) niche ecommerce site (Magento) that has recently lost rankings to a competitor. We out perform them in every metric so I do not understand them leap frogging us to the top spot. This has forced me to look at my site structure, page authority (rank), and internal linking. After reviewing a moz crawl issues report, here are some of my observations: Root domain has a PA of 40 Top 3 $ Category pages have a PA of 22, 18, 18 Multiple meaningless blog posts and other category/product pages have PA’s of 30+ Here is a screenshot of the crawl report with internal links, links, etc showing. I need some help - thoughts, suggestions, next steps in analysis?
On-Page Optimization | | SammyT0 -
When, if ever, would you need to use: example.com/en
Is there any reason to have /en on your website if your website is only in English? Or is it worth having in case you are preparing to translate into other languages? And is there any advantage to being: en.example.com over example.com/en Thanks
On-Page Optimization | | CosiCrawley0 -
I need some help...
I am completely perplexed here guys. I have accomplished all of the the things that the On- Page Analysis tool says that we need to perform as far as( Keyword laden page titles and webpages) yet the report comes back and gives the webpage a C and says that we still need to correct these issues. Can anyone explain this? The keywords are: " real estate augusta ga" " property management augusta ga" the address is: www.aubenrealty.com Thanks in advance, C
On-Page Optimization | | AubbiefromAubenRealty0 -
Too Many on page links! Will "NoFollow" for navigation help?
I am getting to many on page links ( for all my pages). Here is my website: http://www.websterpowerproducts.co.uk I think it is to do with the the navigation bar down the right hand side. I don't really want to get ride of this as it offers users a way of getting where they want without lots of clicking. I was wondering if adding a "NoFollow" tag to each of they links would stop the link juice getting diluted by the navigation bar. Many Thanks
On-Page Optimization | | WebsterPowerTools0 -
Broken links detected by SEOMOZ
hi there, the SEOMOZ crawl has detected two "broken links" : http://www.tarifakitesurfcamp.com/feed/
On-Page Optimization | | juanmiguelcr
http://www.tarifakitesurfcamp.com/comments/feed/ how can I fix the problem? I know that these two links are related to the user's subscription (hypothetical) to my website updates. Thanks in advanced.0 -
Google Results Title Tag HELP
Can anybody tell us why Google changes your title tag in the SERP? If you check out the below link or type in 'days inn', you will see the 2nd result for www.daysinnrc.co.uk just says 'Days Inn' but on the actual site the title tag for this page is 'Days Inn UK | Days Inn | Daysinnrc.co.uk' http://www.google.co.uk/#hl=en&sclient=psy-ab&q=days+inn&oq=days+inn&gs_l=hp.3...4110.4110.4.4297.1.1.0.0.0.0.0.0..0.0...0.0...1c.1.kWVC24EnCHE&pbx=1&bav=on.2,or.r_gc.r_pw.r_qf.&fp=7680231318a44bb0&bpcl=35466521&biw=1920&bih=934 This has happened with another site too, does anybody know why? Thanks
On-Page Optimization | | SEOwins0 -
SEOmoz crawl error
Hi, I'm getting a crawl error and it complains about there being missing meta description... But, the errors are all for non existent index files in directories that only contain pdf files and some thumbs of the front page... Just started trying to learn this stuff...! Cheers Rod
On-Page Optimization | | DrWho0
