Why are these results being showed as blocked by robots.txt?
-
If you perform this search, you'll see all m. results are blocked by robots.txt: http://goo.gl/PRrlI, but when I reviewed the robots.txt file: http://goo.gl/Hly28, I didn't see anything specifying to block crawlers from these pages.
Any ideas why these are showing as blocked?
-
Hi,
Your robots.txt file is very .. steroid healthy. It has his own universe
Are you 100% sure all of the entries are legit and clean ?
First thing I would do is to check Web M;aster Tools for the mobile subdomain. If you don't have it yet, that will be a good place to start - to verify the m subdomain.
Once in WeB Master Tools - you can debug this in no time.
Cheers.
-
but, even when i search from my mobile device, I get the same results (that m. is blocked)
-
I can't submit because I haven't claimed m. in GWT
-
If you haven't already done so, I recommend testing your robots.txt file against one of your mobile pages (such as m.healthline.com/treatments) in Google Webmaster Tools. You can do this by logging into GWT, then click Health, then Blocked URLs.
If you have already tested it in GWT, can you let us know what the results said?
-
Another good article from the community
-
So after a little it or research as I never ever came past this before as all the site we do are responsive, I found this
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=72462
It seems Google wont index a website that they think is a mobile website within the main serp, and vice verse ...
Hope that helps, cause it had me puzzled
Regards
John
-
Which directory are you storing your mobile website files within ...
-
Oh, sorry, on further investigation I see its just your mobile site that are being blocked ...
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
A grade optimised posts not showing in SERPs
Hi all, I've been using Moz to research, optimise and grade a broad range of copy and blog posts over the years. After the optimisation process I've always seen a relatively quick improvement of pages/posts in SERPs. I am currently working on a new website launched earlier in the year on a subdomain. There's a sitemap, fresh content added every month and the site has an verified Google Analytics and Search Console account. The content is quite niche with low traffic data for related terms, however, I am finding that after three or four weeks the optimised posts aren't displaying in the top 50 results in Google. These are the posts: https://sykeshome.europe.sykes.com/cut-the-cost-of-customer-support-use-a-work-at-home-model/ - optimised for "Cut the cost of customer support" (and also "Cut the cost of customer support: use a work-at-home model") https://sykeshome.europe.sykes.com/quality-and-compliance-in-a-work-at-home-environment/ - optimised for "Quality and compliance" (and also "Quality and compliance in a work-at-home environment") As a new website launched on a subdomain there aren't currently any inbound links, but I wanted to know if I am simply being impatient in expecting the above posts to rank higher (if only slightly), or if there could be a reason optimised content with a Moz A grade isn't showing in the first 50 results. Any advice or pointers would be much appreciated. Jonathan
Intermediate & Advanced SEO | | JCN-SBWD0 -
Google Is Indexing My Internal Search Results - What should i do?
Hello, We are using a CMS/E-Commerce platform which isn't really built with SEO in mind, this has led us to the following problem.... a large number of internal (product search) search result pages, which aren't "search engine friendly" or "user friendly", are being indexed by google and are driving traffic to the site, generating our client revenue. We want to remove these pages and stop them from being indexed, replacing them with static category pages - essentially moving the traffic from the search results to static pages. We feel this is necessary as our current situation is a short-term (accidental) win and later down the line as more pages become indexed we don't want to incur a penalty . We're hesitant to do a blanket de-indexation of all ?search results pages because we would lose revenue and traffic in the short term, while trying to improve the rankings of our optimised static pages. The idea is to really move up our static pages in Google's index, and when their performance is strong enough, to de-index all of the internal search results pages. Our main focus is to improve user experience and not have customers enter the site through unexpected pages. All thoughts or recommendations are welcome. Thanks
Intermediate & Advanced SEO | | iThinkMedia0 -
Why would a domain show up in Webmaster Tools but not latest links?
I am going through links and trying to figure out what to disavow. I found a domain under "Who links the Most" I wanted to see what the exact link was and I can't find it when I download all links. Why would that be?
Intermediate & Advanced SEO | | EcommerceSite0 -
Sitelinks in non-brand based organic search results
Hi all, I have a question for everyone. Sitelinks have been around for a while now & I've always seen them when the search is for a brand's name. However, today, when looking at the rankings for one of the campaigns we manage, we noticed there were sitelinks in the number #1 & #2 positions in Google (Australia) for the search term "Dance Costumes". Whilst both the companies have Dance Costumes in their title, so do all the other results & so I don't see why it warrants the sites to be relevant via their brand name.
Intermediate & Advanced SEO | | KBB_Digital
Note: The results are organic results, not paid results (where you can add sitelinks). Firstly, has anyone seen this before (screenshot attached)?
And secondly, is there markup/schema that allows you to do this (none that I know of)? danceCostumes-sitelinks.png0 -
Robots.txt issue for international websites
In Google.co.uk, our US based (abcd.com) is showing: A description for this result is not available because of this site's robots.txt – learn more But UK website (uk.abcd.com) is working properly. We would like to disappear .com result totally, if possible. How to fix it? Thanks in advance.
Intermediate & Advanced SEO | | JinnatUlHasan0 -
Why does Google show Titles different than the coded titles?
Hi, I've noticed that on many pages Google shows on the SERPS titles that he chose for me and not necessarily the ones coded in the Title tag (usually small difference like adding suffix etc.). Why is that? Thanks
Intermediate & Advanced SEO | | BeytzNet0 -
What is best practice to eliminate my IP addr content from showing in SERPs?
Our eCommerce platform provider has our site load balanced in a few data centers. Our site has two of our own exclusive IP addresses associated with it (one in each data center). Problem is Google is showing our IP addresses in the SERPs with what I would assume is bad duplicate content (our own at that). I brought this to the attention of our provider and they say they must keep the IP addresses open to allow their site monitoring software to work. Their solution was to add robots.txt files for both IP addresses with site wide/root disallows. As a side note, we just added canonical tags so the pages indexed within the IP addresses ultimately show the correct URL (non IP address) via the canonical. So here are my questions. Is there a better way? If not, is there anything else we need to do get Google to drop the several hundred thousand indexed pages at the IP address level? Or do we sit back and wait now?
Intermediate & Advanced SEO | | ovenbird0 -
Do search results differ greatly when you search on mobile?
If you have a site with responsive design, is Google likely to look upon you more favourably and dramatically change rankings?
Intermediate & Advanced SEO | | BlueLinkERP0
