Site with 2 domains - 1 domain SEO opimised & 1 is not. How best to handle crawlers?
-
Situation:
I have a dual domain site:
Domain 1 - www.domain.com is SEO optimised with product pages and should of course be indexed.
Domain 2 - secure.domain.com is not SEO optimised and simply has checkout and payment gateway pages.I've discovered that Moz automatically crawls Domain 2 - the secure.domain.com site and consequently picks up hundreds of errors.
I have put an end to this by adding a robots.txt to stop rogerbot and dotbot (mozs crawlers) from crawling domain 2. This fixes my errors in Moz reports however after doing more research into 'Crawler Control' I figure this might be the best option.
My Question:
Instead of using robots.txt to stop moz from crawing all of Domain 2 should I use on each page of domain 2?
I believe this would then allow moz and google to crawl Domain 2 but also tell them both not to index it.
My understanding is that this would be best, and might even help my overall SEO by telling google not to give any SEO value to the Domain 2 pages? -
Hello!
I can answer this from a Google / SEO perspective (a non-moz tool perspective).
First you want to be sure the secure subdomain content is not indexed.
-
If the secure subdomain is NOT indexed, leave the robotos.txt crawl blocking in place. You don't want and don't need Google crawling secure pages and payment pages. Just be sure they truly all are private pages. If they are NOT indxed, the crawl block is best - this will prevent google from crawling, and if they can't crawl they can't index.
-
If the secure pages ARE indexed
-
remove the robots.txt crawl block.
-
Add meta noindex on all the pages
-
Wait for them to be noindexed (removed from google)
-
Then, block them from being crawled with robots.txt - which will prevent them from being crawled, and thus prevent them from being indexed as well.
-
-
Hey, Dave here from the Help Team!
Jumping in to answer the technical question, you can definitely use the meta robots tag instead of a disallow directive in your robots.txt file. I would like to point out that Meta Noindex is something we report in Site Crawl so you would see an influx in that issue category but you can mark them as "ignored" as you see fit.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is a Good Domain Authority Score?
I am wondering what is a Good Domain Authority Score? This is my website https://gopaintsprayer.com currently DA and PA are quite high, but the traffic is decreasing, I don't understand the reason, you can help. Thank you !
Getting Started | | gogoanimetp0 -
Moz can't crawl my site.
Moz cannot carry out the site crawl on my online shop. Not really sure what the issue is, it has no problem getting onto my site when you use www. before the address, but it needs to be able to access bluerinsevintage.co.uk Stuck as what to do, we are a shopify store. Anyone else had this problem, or know what i need to change so they can crawl the site? thjis is the page they are getting when trying to get on bluerinsevintage.co.uk but if they use www.bluerinsevintage.co.uk the site comes up.
Getting Started | | bluerinsevintageAdam
0 -
How to have MOZ site crawl pre-launch
Hi, Our new website is about to launch. We would love to have moz.com SITE CRAWL our site before launch. For issues like "missing meta description" and everything else that moz.com checks. We would love to do it before we launch. The new site is currently on a different domain than our live site. example.com <-- this is our live site. new-site.com <-- this is our "staging" server with the new site. We have a long running campaign for example.com Do we need to create a new campain for new-site.com ? Or is there some other simpler way? When we launch we will switch the site from new-site.com to example.com .. example.com will be the address for the new site.. Any ideas or suggestions? best practices? edit Forgot to say thank you for your help and input 🙂
Getting Started | | tandvarden0 -
Moz Pro for simple local seo status reporting
I have a new client on a small budget with a local yoga studio. There's been no SEO work on this site so far, so, after the basics, I plan to work through the items on this list: https://mza.bundledseo.com/blog/top-20-local-search-ranking-factors-an-illustrated-guide I'm a new Moz Pro user and am looking for advice on how to use Moz Pro tools with my client to review progress over time...a simple checklist of reports/metrics to share and track, and to use in weekly status reports. Or, in other words, how can I configure and use Moz Pro tools for basic progress tracking? This is not a sophisticated, SEO-savvy client, so I want to keep things very simple. Thanks for any ideas or pointers. Dave
Getting Started | | dgbruns0 -
My website does not allow all crawler to crawl, Now my question is that whether i need to give permission to moz crawler if yes then whaat is moz bot name?
My website does not permit all crawler to crawl website. Whether ii need to give permission to moz bot to crawl website or not? If yes what is the moz bot name?
Getting Started | | irteam0 -
Why is "preview this query" button greyed out in Add & Manage Queries?
I am trying to set up an industry query in Add & Manage Queries but the "preview this query" button greyed out. I filled in the Query Name, Add Label: Industry and matches phrase exactly under Search Query. Thank you for your help.
Getting Started | | jojobo0 -
Site traffic vs other sites
I have a quick question. Does Moz have a tool I can use to see how much traffic my site gets a month versus some competition? I know Alexa has this but you have to create an account. I am just wondering if Moz has one and I am missing it.
Getting Started | | trumpfinc0 -
'Domain Does Not Respond To Web Requests'
Hi everyone, This seems to be a fairly common query on the Q&A section, but I haven't been able to find a solution by reading through previous threads. When I try to set up an SEOmoz campaign for spryz.co.nz, I get that ol' favourite error message: 'We have detected that the domain spryz.co.nz does not respond to web requests. Using this domain, we will be unable to crawl your site or present accurate SERP information.' The problem isn't being caused by a Robots.txt file, and the site has experienced 99% uptime since it was launched.Traffic stats show that visits are coming through to the site via the search engines, suggesting that it may not be all crawlers that fail to access the site. I've tried to set up this campaign several times throughout the day, since I've read that sometimes Roger goes on the blink, but I've still not been successful. Any suggestions as to why Roger might be unable to crawl my website would be great.
Getting Started | | e8creative0