SEOMOZ crawler is still crawling a subdomain despite disallow
-
This is for our client with a subdomain. We only want to analyze their main website as this is the one we want to SEO. The subdomain is not optimized so we know it's bound to have lots of errors.
We added the disallow code when we started and it was working fine. We only saw the errors for the main domain and we were able to fix them. However, just a month ago, the errors and warnings spiked up and the errors we saw were for the subdomain.
As far as our web guys are concerned. the disallow code is still there and was not touched. User-agent: rogerbot Disallow: /
We would like to know if there's anything we might have unintentionally changed or something we need to do so that the SEOMOZ crawler will stop going through the subdomain.
Any help is greatly appreciated!
-
Thanks Peter for your assistance.
Hope to hear from the SEOMOZ team soon with regards to this issue.
-
John,
Thanks for writing in! I would like to take a look at which project you guys were working with that this is happening. I will go ahead and start a ticket so we can better answer your questions You should hear from me soon!
Best,
Peter
Moz Help Team. -
I have heard of this before recently, I think possibly the moz crawler all or sometimes now just ignores the disallow because it is not a usual S.E crawler.
Hopefully one of the staff can provide some insight in this for you.
All the best.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Still Need to Write Title & Description Tag?
My SEO has advised me that Google has stopped using title and description tags for search results and as such, it is no longer necessary to write specific title and description tags. I see that Yoast seems to pull text to create these tags and sometimes it looks like it reflects the best elements of the content, sometimes it does not. Should I be asking our SEO team to write dedicated title and description tags or is it best practices to leave it to the Yoast plugin? My SEO is of the opinion that writing these tags is not a productive use of time as Google will serve results based on the user inquiry rather than the content on his tags. It sounds logical but it would be reassuring to receive further confirmation of this. Thoughts?"
Intermediate & Advanced SEO | | Kingalan1
Thanks, Alan0 -
Should I run my Shopify store on a subdomain or buy a new domain for it?
I'm planning to set up a subdomain for my Shopify store but I'm not sure if this is the right approach. Should I purchase a separate domain for it? I'm running Wordpress on my website and want to keep it that way. I want to use Shopify for the ecommerce side. I want to link the store from the top nav and of course I'll use CTA's in a variety of ways to point to merchandise and other things on the store side. Thanks for any help you can offer.
Intermediate & Advanced SEO | | ims20160 -
Google Adsbot crawling order confirmation pages?
Hi, We have had roughly 1000+ requests per 24 hours from Google-adsbot to our confirmation pages. This generates an error as the confirmation page cannot be viewed after closing or by anyone who didn't complete the order. How is google-adsbot finding pages to crawl that are not linked to anywhere on the site, in the sitemap or linked to anywhere else? Is there any harm in a google crawler receiving a higher percentage of errors - even though the pages are not supposed to be requested. Is there anything we can do to prevent the errors for the benefit of our network team and what are the possible risks of any measures we can take? This bot seems to be for evaluating the quality of landing pages used in for Adwords so why is it trying to access confirmation pages when they have not been set for any of our adverts? We included "Disallow: /confirmation" in the robots.txt but it has continued to request these pages, generating a 403 page and an error in the log files so it seems Adsbot doesn't follow robots.txt. Thanks in advance for any help, Sam
Intermediate & Advanced SEO | | seoeuroflorist0 -
Is there a problems with putting encoding into the subdomain of a URL?
We are looking at changing our URL structure for tracking various affiliates from: https://sub.domain.com/quote/?affiliate_id=xxx to https://aff_xxx_affname.domain.com/quote/ Both would allow us to track affiliates, but the second would allow us to use cookies to track. Does anyone know if this could possibly cause SEO concerns? Also, For the site we want to rank for, we will use a reverse proxy to change the URL from https://aff_xxx.maindomain.com/quote/ to https://www.maindomain.com/quote/ would that cause any SEO issues. Thank you.
Intermediate & Advanced SEO | | RoxBrock0 -
Does subdomain hurt SEO on main site
This client sells event management software and puts all their clients on different subdomains of their main domain. Looking in SEO tools like OSE, when I run a backlink analysis, it pulls up all the backlinks to the subdomains as well as those for the main domain. In webmaster tools when I look at queries, impressions and clicks, they get at least 30 times more traffic and impressions on keywords found in their subdomains and very few on their own. In other words, all these tools are providing a collective analysis of main domain and all subdomains. All the backlinks and keywords recorded for those subdomains are not at all relevent to the keywords they want to rank for. For example, their software supports Boy Scouts, so keywords they rank for according to WT include merit badge, scout camp, etc., but of course, that's on the subdomain. As a result, if you were to take a snapshot of their online presence as these tools do, you would think they were a boy scout website and not a software developer if you include the subdomain, along with its PR, backlinks, keywords, etc. So the question I have is, does Google connect all these subdomains with the main domain and then water down the main site with irrelevant keywords, content and backlinks? Or does Google see all those subdomains as completely separate and we don't need to worry or move their clients off their subdomain? I'm worried about Google assigning a "boy scout" relevancy to them. Am I wrong? What would you do?
Intermediate & Advanced SEO | | katandmouse0 -
Client site is lacking content. Can we still optimize without it?
We just signed a new client whose site is really lacking in terms of content. Our plan is to add content to the site in order to achieve some solid on-page optimization. Unfortunately the site design makes adding content very difficult! Does anyone see where we may be going wrong? Is added content really the only way to go? http://empathicrecovery.com/
Intermediate & Advanced SEO | | RickyShockley0 -
Follow or nofollow to subdomain
Hi, I run a hotel booking site and the booking engine is setup on a subdomain.
Intermediate & Advanced SEO | | vmotuz
The subdomain is disabled from being indexed in robots.txt Should the links from the main domain have a nofollow to the subdomain? What are you thoughts? Thanks!0 -
Franchise sites on subdomains
I've been asked by a client to optimise a a webpage for a location i.e. London. Turns out that the location is actually a franchise of the main company. When the company launch a new franchise, so far they have simply added a new page to the main site, for example: mysite.co.uk/sub-folder/london They have so far done this for 10 or so franchises and task someone with optimising that page for their main keyword + location. I think I know the answer to this, but would like to get a back up / additional info on it in terms of ranking / seo benefits. I am going to suggest the idea of using a subdomain for each location, example: london.mysite.co.uk Would this be the correct approach. If you think yes, why? Many thanks,
Intermediate & Advanced SEO | | Webrevolve0