Blocked URL's by robots.txt
-
In Google Webmaster Tools shows me 10,936 Blocked URL's by robots.txt and it is very strange when you go to the "Index Status" section where shows that since April 2012 robots.txt blocked many URL's.
You can see more precise on the image attached (chart WMT)
I can not explain why I have blocked URL's ? because I have nothing in robots.txt.
My robots.txt is like this:User-agent: *
I thought I was penalized by Penguin in April 2012 because constantly i'am losing visitors now reaching over 40%. It may be a different penalty?
Any help is welcome because i'm already so saturated.
Mera
-
I added the 2 lines several hours because i saw that Google had crawl some zend routes.
I changed the entire content of the website 2 weeks ago and during this process i notice the problem.
Many thanks Mark for your help.
-
I just checked your robots file, and I see this:
User-agent: *
Disallow: /cars/index
Disallow: /cars/details
-
I don't have any meta tags to block URL's
-
I attached the image
-
Check in webmaster tools what Google is seeing for your robots.txt - this will confirm what you see is what Google is seeing
Can you paste the full text here?
-
You can also exclude URLs by meta tag. Do you have any meta tags like those on this page?
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Question on Google's Site: Search
A client currently has two domains with the same content on each. When I pull up a Cached version of the site, I noticed that it has a Cache of the correct page on it. However, when I do a site: in Google, I am seeing the domain that we don't want Google indexing. Is this a problem? There is no canonical tag and I'm not sure how Google knows to cache the correct website but it does. I'm assuming they have this set in webmaster tools? Any help is much appreciated! Thanks!
Technical SEO | | jeff_46mile0 -
On March 10 a client's newsroom disappeared out of the SERPS. Any idea why?
For years the newsroom, which is on the subdomain news.davidlerner.com - has ranked #2 for their brand name search. On march 10 it fell out of the SERPs - it is completely gone. What happened? How can I fix this?
Technical SEO | | MeritusMedia0 -
Will an XML sitemap override a robots.txt
I have a client that has a robots.txt file that is blocking an entire subdomain, entirely by accident. Their original solution, not realizing the robots.txt error, was to submit an xml sitemap to get their pages indexed. I did not think this tactic would work, as the robots.txt would take precedent over the xmls sitemap. But it worked... I have no explanation as to how or why. Does anyone have an answer to this? or any experience with a website that has had a clear Disallow: / for months , that somehow has pages in the index?
Technical SEO | | KCBackofen0 -
Why am I not showing up in the SERP's or Google Local?
I have been trying to optimise the following site for both Google SERP's and Google Local - Pixel Primate The URL has been around for around 3 years now but they just updated the website and launched it in December 2012. I did the on-page optimisation early in January 2013 and Google seems to have indexed the changes, for the home page at least. One major keyword I am targeting for the home page is 'Web Design Leicester'. I understand that the DA is fairly low (24) so this is something I need to improve. However, I've experienced positive results fairly quickly from just on-page optimisation for other sites I have worked on. The site just doesn't seem to be ranking at all for any keywords. Maybe the industry type is just extremely competitve but I find it very strange to not be visible anywhere in the SERPs. The site does not seem to have any penalties as it ranks for 'Pixel Primate' and all pages appear when doing a site: search. Also what's strange is that I set up the Google Local listing years ago but it doesn't appear anywhere in the local listing, not even when I search for it manually. Any suggestions would be appreciated.
Technical SEO | | CWseo0 -
What's the issue?
Hi, We have a client who dropped in the rankings (initially from bottom of the first page to page to page 3, and now page 5) for a single keyword (their most important one - targeted on their homepage) back in the middle of March. So far, we've found that the issue isn't the following: Keyword stuffing on the page External anchor text pointing to the page Internal anchor text pointing to the page In addition to the above, the drop didn't coincide with panda or penguin. Any other ideas as to what could cause such a drop for a single keyword (other related rankings haven't moved). We're starting to think that this may just have been another small change in the algorithm but it seems like too big of a drop in a short space of time for that to be the case. Any thoughts would be much appreciated! Thanks.
Technical SEO | | jasarrow0 -
Warnings for blocked by blocked by meta-robots/meta robots Nofollow...how to resolve?
Hello, I see hundreds of notices for blocked by meta-robots/meta robots nofollow and it appears it is linked to the comments on my site which I assume I would not want to be crawled. Is this the case and these notices are actually a positive thing? Please advise how to clear them up if these notices can be potentially harmful for my SEO. Thanks, Talia
Technical SEO | | M80Marketing0 -
How do i properly combine these two schema's from schema.org
So we're redoing our reviews/testimonials page on our website right now and moving over to the schema.org format as described here: http://schema.org/Review But we would like to combine each of our reviews with a location for which it was reviewed using this: http://schema.org/LocalBusiness What i can't wrap my head around would be the correct syntax? is it just the first block and then the next block? or is there a way of putting the actual physical address within the review page itself? So is this the correct way to do a page full of reviews that are reviewing various physical locations? * <div< span="">itemprop="reviews" itemscope itemtype="http://schema.org/Review"></div<>* <span< span="">itemprop="name">Value purchase</span<> -* by <span< span="">itemprop="author">Lucas</span<>,* <meta< span="">itemprop="datePublished" content="2011-03-25">March 25, 2011</meta<>* <div< span="">itemprop="reviewRating" itemscope itemtype="http://schema.org/Rating"></div<>* <meta< span="">itemprop="worstRating" content = "1"/></meta<>* <span< span="">itemprop="ratingValue">4</span<>/* <span< span="">itemprop="bestRating">5</span<>stars* <span< span="">itemprop="description">Great microwave for the price. It is small and</span<>* fits in my apartment. 1. <div< span="">itemscope itemtype="http://schema.org/LocalBusiness"></div<> 2. # <span< span="">itemprop="name">Beachwalk Beachwear & Giftware</span<> 3. <span< span="">itemprop="description"> A superb collection of fine gifts and clothing</span<> 4. to accent your stay in Mexico Beach. 5. <div< span="">itemprop="address" itemscope itemtype="http://schema.org/PostalAddress"></div<> 6. <span< span="">itemprop="streetAddress">3102 Highway 98</span<> 7. <span< span="">itemprop="addressLocality">Mexico Beach</span<>, 8. <span< span="">itemprop="addressRegion">FL</span<> 10. Phone: <span< span="">itemprop="telephone">850-648-4200</span<> <div< span="">itemprop="reviews" itemscope itemtype="http://schema.org/Review"></div<>* <span< span="">itemprop="name">Value purchase</span<> -* by <span< span="">itemprop="author">Lucas</span<>,* <meta< span="">itemprop="datePublished" content="2011-03-25">March 25, 2011</meta<>* <div< span="">itemprop="reviewRating" itemscope itemtype="http://schema.org/Rating"></div<>* <meta< span="">itemprop="worstRating" content = "1"/></meta<>* <span< span="">itemprop="ratingValue">4</span<>/* <span< span="">itemprop="bestRating">5</span<>stars* <span< span="">itemprop="description">Great microwave for the price. It is small and</span<>* fits in my apartment. <div< span="">itemscope itemtype="http://schema.org/LocalBusiness"></div<> <span< span="">itemprop="name">Beachwalk Beachwear & Giftware</span<> <span< span="">itemprop="description"> A superb collection of fine gifts and clothing</span<> to accent your stay in Mexico Beach. <div< span="">itemprop="address" itemscope itemtype="http://schema.org/PostalAddress"></div<> <span< span="">itemprop="streetAddress">3102 Highway 98</span<> <span< span="">itemprop="addressLocality">Mexico Beach</span<>, <span< span="">itemprop="addressRegion">FL</span<> Phone: <span< span="">itemprop="telephone">850-648-4200</span<>
Technical SEO | | adriandg0
