Can we listed URL on Website sitemap page which are blocked by Robots.txt
-
Hi,
I need your help here.
I have a website, and few pages are created for country specific. (www.example.com/uk).
I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page)
I really appreciate your help.
Thanks,
Nilay
-
if the content is of benefit to the user then include them in your navigation. Why are you blocking them in the first place, duplicate content?
-
Hi Zora,
But the pages which i have blocked are only visible in specific country. Addition, i have blocked theme also, so you think it's good to put those url on website or link them in website?
-
Hi Zora,
Thanks for your time.
-
Hi Jarno,
Thanks for your time.
Should i put URLs anywhere in my websites which are being excluded by Robot.txt?
Thanks,
Nilay
-
Hi Nilay,
I actually did this yesterday by accident.
I recommend you remove the blocked pages from your XML sitemap, otherwise Google will display a "warning" after you submit it.As far as your HTML sitemap, it does not really matter.
I think you are okay to keep the links there. -
Nilay,
if you have blocked them from your robots.txt but you do enclude them in your sitemap xml or html then they will be indexed unless you enclode a meta robots in it with a noindex tag. If that tag is not in the pages and you enclude that page in your sitemap Google will feel it as important content in list it in the SERPs.
Hope this helps
Regards
Jarno
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Which Is More Important? Building a web page for customer reviews or a careers page?
Hello, I am wondering which would be more important to have on a website a customer review page or a careers page? And as far an SEO advantage which is more important and why?
On-Page Optimization | | Nicks1230 -
URL, page title, item name - which is most important for google ranking
We are a bridal store and are able to use different information in the URL, Page title and item name. In item name we give the product a name for us to identify ie. Alex Lace Dress in Black/Nude, Ivory/Nude, Red/Red In Page Title we use the suppliers name and product code as well as the item name ie. Jadore j8075 Alex Lace Dress Online Australia URL = alex-lace-dress/ Are we using the correct format ? What could we do to improve them?
On-Page Optimization | | CostumeD0 -
Impact of multiple links on the same page to the same url (different anchor text) ?
Hi, On our category pages, for every product we have several links pointing to the product : on the image, on the product name, on the short description, on "read more", and a javascript onclick on the entire div. Could this have a negative impact for link juice distribution, or is it counted as only 1 link with the first anchor text found on the page ? Thanks,
On-Page Optimization | | Strelok0 -
Page Titles For Local - Help on URL Structure
Trying to figure out the best way to construct localized urls for the dental website. For example, If I have the URL:
On-Page Optimization | | Czubmeister
http://www.kooskidental.com/services/cosmetic-dentistry/
and If I want to make it local to the city I would use: http://www.kooskidental.com/services/richardson-tx-cosmetic-dentistry/ But what happens is that I have other options off the menu like: http://www.koooskidental.com/services/richardson-tx-cosmetic-dentistry/teeth-whitening/ But if I am trying to rank for richardson tx teeth whitening, I would have to do http://www.koooskidental.com/services/richardson-tx-cosmetic-dentistry/richardson-tx-teeth-whitening/ But that's pretty long and ugly and I don't think I need richardson-tx in their twice. If I am trying to rank for richardson tx cosmetic dentistry and richardson tx teeth whitening, what would be the best structure for the url's?0 -
Can I use Same Keyword for Multi pages Title Tags?
Hello All, I am working on client website and currently they are targeting One Keywords for multi pages. As I have search with Allintitle: Search query and Google display around 37 pages of website which carry same keyword in "Title Tags". I have told to client to change the "Title Tags" but they want that keyword for all relevant pages. So I want to know is that harm in Search Engine Ranking? Note: They have not done the link building activities for multi pages with same Keyword, they are using only in "Title Tags" only
On-Page Optimization | | jemindesai0 -
If I want to rank well on one keyword would it be better to optimize multiple pages on the website for the keyword or should I only optimize one page for that keyword?
If I want to rank well on one keyword would it be better to optimize multiple pages on the website for the keyword or should I only optimize one page for that keyword?
On-Page Optimization | | CustomOnlineMarketing0 -
Why would my homepage be ranked lower (Page Rank 2) than my other pages on the site (PR3) ?
Why would my homepage be ranked lower (Page Rank 2) than my other pages on the site (PR3) ?
On-Page Optimization | | dmurtagh0 -
How do you block development servers with robots.txt?
When we create client websites the urls are client.oursite.com. Google is indexing theses sites and attaching to our domain. How can we stop it with robots.txt? I've heard you need to have the robots file on both the main site and the dev sites... A code sample would be groovy. Thanks, TR
On-Page Optimization | | DisMedia0