Summarize your question.Sitemap blocking or not blocking that is the question?
-
Hi from wet & overcast wetherby UK
Ones question is this...
" Is the sitemap plus boxes blocking bots ie they cant pass on this page http://www.langleys.com/Site-Map.aspx "
Its just the + boxes that concern me, i remeber reading somewherte javascript nav can be toxic.
Is there a way to test javascript nav set ups and see if they block bots or not?
Thanks in advance
-
I use Screaming Frog SEO Spider (free version) to check the internal link structure of a website. If a page is blocking ALL spiders it will pick it up.
Another thing I would say would be to check in Google Webmaster Tools to see if there are any crawl errors.
And the last thing I would add is to make sure that you have a non-JavaScript way to find all the pages on your website - through strong internal linking or a manual sitemap page that isn't generated through JS.
Hope this helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Search console says 'sitemap is blocked by robots?
Google Search console is telling me "Sitemap contains URLs which are blocked by robots.txt." I don't understand why my sitemap is being blocked? My robots.txt look like this: User-Agent: *
Technical SEO | | Extima-Christian
Disallow: Sitemap: http://www.website.com/sitemap_index.xml It's a WordPress site, with Yoast SEO installed. Is anyone else having this issue with Google Search console? Does anyone know how I can fix this issue?1 -
.htaccess Question
Hi,I have a website www.contractor-accounts.co.uk that has an .htaccess file that strips .php and forces a closing brace /. The site is now over 6 months old and still has a very low ranking with MOZ also rating the site as DA/PA = 1 which seems to indicate some sort of issue with the website. Can anyone offer any suggestions as to why this site is ranking poorly as much of the onpage SEO has been completed to a level of 90%+ for specific keyterms so I'm probably either looking at routing of the framework of so other technical SEO issues possibly? Any help much apreciated... <ifmodule mod_rewrite.c=""><ifmodule mod_negotiation.c="">Options -MultiViews</ifmodule> RewriteEngine On # Redirect Trailing Slashes...
Technical SEO | | ecrmeuro
# RewriteRule ^(.)/$ /$1 [L,R=301]
RewriteCond %{REQUEST_URI} /+[^.]+$
RewriteRule ^(.+[^/])$ %{REQUEST_URI}/ [R=301,L]
# Redirect non-WWW to WWW...
RewriteCond %{HTTP_HOST} ^contractor-accounts.co.uk [NC]
RewriteRule ^(.)$ http://www.contractor-accounts.co.uk/$1 [L,R=301] # Handle Front Controller...
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^ index.php [L]</ifmodule>0 -
Google My Business Service Area Question
Hello Moz Friends I just wanted to make sure I'm doing things correctly. On google my business your given the option to list your service area. I serve the entire state of Colorado with my internet marketing services. So I listed Colorado as my service area. but Moz Friends, is this the wrong idea? Like should I list the major cities and call it good? So instead of service area Colorado, I should put Denver, Colorado Springs, Pueblo etc Thank you for your friendly help Chris
Technical SEO | | asbchris0 -
What are the steps to submitting a sitemap for a blog?
We are in the process of a website migration and need to submit a site map for our website and blog. What are the steps to follow for submitting a site map for the blog? Can we submit with just the /blog URL or do we need to include each category?
Technical SEO | | Sable_Group0 -
Generating a xml sitemap?
Hi What is everyone's preferred method of generating an XML sitemap? Just wondering if one piece of software is better than others?
Technical SEO | | TheZenAgency1 -
Questions about Redirects
Hi, I am trying to make sure that I can determine if a site has a 301 redirect set up to redirect the site from domain.com to www.domain.com and am hoping that you can confirm the following for me, or let me know if I am off track: is http://www.internetofficer.com/seo-tool/redirect-check/ a reliable way to check if a 301 redirect is set up? is Screaming Frog SEO Spider a good tool to use to see if a redirect is in place? if I search for site:www.domain.com and site:domain.com, I should only get results for the site being indexed, not for the site that has the 301 redirect set up, right? For example, if www.domain.com is set up to redirect to domain.com, then I should get no search results for site:www.domain.com and only show indexed pages for domain.com. If I search for site:www.domain.com and site:domain.com and get results for both, then does this mean that the redirect is not set up? if a redirect is set up from www.domain.com to domain.com, should the crawl report should only show one page crawled on www.domain.com? if a crawl report shows same number of pages for www.domain.com as for domain.com, does that mean that redirect is not set up properly? Thanks in advance for your help! Carolina
Technical SEO | | csmm0 -
Pagination question
I have a website http://www.example.com with pagination series starting with page1.html upto page10.html. With backlinks to some of the pages ( page1.html, page2.html----page7.html). If i include rel="next" and rel="prev" on page1.html to page10.html pages. Will value of those links will be transfered to http://www.example.com This is what i interpret from http://bit.ly/mUOrn2 Am i right ?
Technical SEO | | seoug_20050 -
Robots.txt question
I want to block spiders from specific specific part of website (say abc folder). In robots.txt, i have to write - User-agent: * Disallow: /abc/ Shall i have to insert the last slash. or will this do User-agent: * Disallow: /abc
Technical SEO | | seoug_20050