Bloking pages in roborts.txt that are under a redirected subdomain
-
Hi Everyone,
I have a lot of Marketo landing pages that I don't want to show in SERP. Adding the noindex meta tag for each page will be too much, I have thousands of pages.
Blocking it in roborts.txt could have been an option, BUT, the subdomain homepage is redirected to my main domain (with a 302) so I may confuse search engines ( should they follow the redirect or should they block)
marketo.mydomain.com is redirected to www.mydomain.com
disallow: / (I think this will be confusing with the redirect)
I don't have folders, all pages are under the subdomain, so I can't block folders in Robots.txt also
Would anyone had this scenario or any suggestions?
I appreciate your thoughts here.
Thank you
Rachel
-
Thank you so much for you answer!
the home page in the subdomain is redirected but none of the actual pages in the subdomain are, and because there are so many of them, it would be easier to block them in robots.txt, even if there is small change that Google will still index them. But because the home page is redirected, I don't want to confuse Google with a Disallow: /
Could I do Disallow: / and then Allow: /homepage.html
-
Under usual circumstances, Google won't index redirecting addresses which it considers to be a shallow form of 'doorway' page (Google doesn't like to rank those). If I am reading your post right, no pages on the redirected sub-domain can be visited as they all now redirect. Google should start dropping those old URLs from its index automatically. It's important to note that you should be using 301s to eventually de-index the old URLs. If you're using 302s then you're telling Google that the old pages are only being redirected temporarily and they will return (which could mess with indexation)
If I am reading your post wrong and some pages are still live on the old subdomain and are not redirecting, and you want Google to redirect most addresses but some are staying (for whatever reason) and you still want to de-index those specific ones, Meta no-index (sorry) really is your best bet! Robots.txt tells Google not to crawl a page but it doesn't tell Google not to index a URL if external metrics (inbound links) are strong enough. Both are 'directives' and Google is forced to obey neither
I think from what you are saying, I'd just leave the redirects and let Google do its work. Make sure they're 301s, though. You're right that Robots.txt might end up confusing things. Robots.txt can also sometimes 'kill' the SEO authority of a page. If you did that for all the redirecting pages, no equity would flow through your 301s (hazardous)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Redirection chain and Javascript Redirect
Hi, A redirection chain is usually defined as a page redirecting to another page which itself is another redirection. URL1 ---(301/302)---> URL2 ---(301/302)---> URL3 But what about Javascript redirect? They seem to be a different beast: URL1 ---(301/302)---> URL2 ---(200 then Javascript redirect)---> URL3 From what I know if the javascript redirect is instant Google counts it as a 301 permanent redirection, but I'm still not sure about if this counts as a redirection chain. Most of the tools (such as moz) only see the first redirection. So is that scenario a redirection chain or no?
Technical SEO | | LouisPortier0 -
Should the canonical tag for the redirected pages be changed
Hi! Does anyone know if the canonical tag of the old redirected page should be changed, and include the URL of the new destination? Thanks in advance.
Technical SEO | | AnahitG0 -
How to resolve warning of pages with redirect chain when its your http:// to https://www.
how do I write a 301 redirect in the htaccess file so that http:// goes straight to https://www. Moz replyEli profileHey there!Thanks for reaching out to us!
Technical SEO | | VelocityWebsites0 -
Robots txt. in page with 301 redirect
We currently have a a series of help pages that we would like to disallow from our robots txt. The thing is that these help pages are located in our old website, which now has a 301 redirect to current site. Which is the proper way to go around? 1- Add the pages we want to disallow to the robots.txt of the new website? 2- Break the redirect momentarily and add the pages to the robots.txt of the old one? Thanks
Technical SEO | | Kilgray0 -
No index on subdomains
Hi, We have a subdomain that is appearing in the search results - I want to hide this as it looks really bad. If I were to add the no index tag to the sub domain would URL would this affect the whole domain or just that sub domain? The main domain is vitally important - it is just that sub domain I need to hide. Many thanks
Technical SEO | | Creditsafe0 -
Should I do a 301 redirect
Hi Everyone, Hope you can help me out here. I have .co.uk & .ie website with similar content. On a particular section of the .co.uk website it is updated daily (Q&As, Blog posts etc) .ie does have this section but to a lesser degree, no daily updates etc, I was wondering if we should simply do a 301 redirect when someone is on the .ie website to .co.uk, it means the user is getting a much better experience however not entirely the consequences from search engines on this? Thanks
Technical SEO | | Paul781 -
Rebranding / Redirecting
Hi I have a client who wants to re-brand their shopify clothing store under new domain name. Whilst still a clothing store its going to have different department structure and product pages and will hence be a different store/site. Is there anyway to pass any of the history/authority of existing site to the new one such as 301 redirecting the top level pages of existing site to nearest equivalent pages of new site etc or best to just redirect the old site domain to the new ? Cheers Dan
Technical SEO | | Dan-Lawrence0 -
Page not Accesible for crawler in on-page report
Hi All, We started using SEOMoz this week and ran into an issue regarding the crawler access in the on-page report module. The attached screen shot shows that the HTTP status is 200 but SEOMoz still says that the page is not accessible for crawlers. What could this be? Page in question
Technical SEO | | TiasNimbas
http://www.tiasnimbas.edu/Executive_MBA/pgeId=307 Regards, Coen SEOMoz.png0