20 x '400' errors in site but URLs work fine in browser...
-
Hi, I have a new client set-up in SEOmoz and the crawl completed this morning... I am picking up 20 x '400' errors, but the pages listed in the crawl report load fine... any ideas?
example -
-
Most major robots obey crawl delays. You could check your errors in Google Webmaster Tools to see if your site is serving a lot of error pages when Google crawls.
I suspect Google is pretty smart about slowing down its crawl rate when it encounters too many errors, so it's probably safe to not include a crawl delay for Google.
-
Sorry, one last question.
Do I need to add a similar delay for Google Bots, or is this issue specifically a Roger Bot problem?
Thanks
-
Fantastic, thanks, Cyrus and Tampa, prevented many more hours of scratching head!!!
-
Hi Justin,
Sometimes when rogerbot crawls a site, the servers and/or the content management system can get overwhelmed if roger is going to fast, and this causes your site to deliver error pages as roger crawls.
If the problem persists, you might consider installing a crawl delay for roger in your robots.txt file. It would look something like this:
User-agent: rogerbot
Crawl-delay: 5This would cause the SEOmoz crawlers to wait 5 seconds before fetching each page. Then, if the problem still persists, feel free to contact the help team at [email protected]
Hope this helps! Best of luck with your SEO!
-
Thanks Tampa SEO, good advice.
Interestingly, the URL listed in SEOmoz is as follows:
www.morethansport.co.uk/brand/adidas?sortDirection=ascending&sortField=Price&category=sport and leisure
But when I look at the link in the referring page it is as follows:
/brand/adidas?sortDirection=ascending&sortField=Price&category=sport%20and%20leisure
notice the "%" symbol instead of the spaces.
The actual URL is the one listed in SEOmoz but even if I copy and paste the % version, the browser removed the '%' and the page loads fine.
I still can't get the site to throw-up a 400.
-
Just ran the example link that you provided through two independent HTTP response code checkers, and both are giving me a 200 response, i.e. the site is OK.
This question has been asked before on here; you're definitely not the first person to run into the issue.
One way to diagnose what's going on is to dig a little deeper into the crawling report that SEOmoz generated. Download the CSV file and look at the referring link, i.e. on which page Roger found the link. Then go to that page and look if your CMS is doing anything weird with the way it outputs the links that you create. I recall someone back in December having the same issue and eventually resolved it by noticing that his CMS put all sort of weird slashes (i.e. /.../...) into the link.
Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best Site Architecture tool for analysis my & competitors site?
Hello All, I am really confused with my current architecture for Ecommerce site, can you please suggest any tool or software where I can analysis mine and competitors site architecture ? Thanks!
On-Page Optimization | | pragnesh96390 -
HTML Site SEO (NO CMS)
I have got a client site, which is dated (2007) and has not been shifted to any recognised CMS yet. It is HTML based. Is it possible to SEO on such a site? Is it even worth it? If it is possible to do SEO on this, any suggestions will be highly appreciated. Thank you.
On-Page Optimization | | ArthurRadtke3 -
Acquired Old, Bad Content Site That Ranks Great. Redirect to Content on My Site?
Hello. my company acquired another website. This website is very old, the content within is decent at best, but still manages to rank very well for valuable phrases. Currently, we're leaving the entire site active on its own for its brand, but i'd like to at least redirect some of the content back to our main website. I can't justify spending the time to create improved content on that site and not our main site though. What would be the best practice here? 1. Cross-domain canonical - and build the new content on our main website? 2. 301 Redirect Old Article to New Location containing better article 3. Leave the content where it is - you won't be able to transfer the ranking across domain. Thanks for your input.
On-Page Optimization | | Blenny0 -
Flickr v. On-Site Images
My apologies in that I have searched for this, have seen discussions on it and haven't seen a definitive answer on the question of hosting & displaying on-domain images v. using a source like flickr to host all of your images. I have a client that is mostly a local search play in a very tourism heavy area. I'm investigating this option for a few reasons. 1. Ease of use. The new flickr app is brilliant. So if he's out giving a tour, takes a picture, it can be seamlessly integrated to his account & then shot off to all of his portals (website, facebook, twitter, etc.). It's a small client & he's not tech savvy, so this option suits him very well. 2. SEO. With all of the tagging, geo components, and it playing nice with Google Images search, I thought this was a viable option in hosting the majority of his on-site images. I've seen opinions on this before. But I was wondering if there any further opinions on the subject. Not sure if there's anything 'definitive', but any help or insight would be appreciated.
On-Page Optimization | | BrianWhitaker0 -
Linking within Secondary Site
So we've got a secondary site that has quite a bit of authority & links that used to have all types of info on parasailing. All those pages are gone and homepage is now a salespage (management decision, not mine) Our main site sells a wide range of tours and activities and does have a page for parasailing. The secondary site uses the same template/navigation as our main site (again, not my decision). Do you think that's an effective way to send link juice to our main site? The secondary site has some pretty awesome high authority sites linking to it. I've considered 301'ing the whole site to our main site but it's got a really solid domain name and I'd like to take up 2 SERP listings (main and secondary site) Is there a better way to have double listings but still send a good amount of link juice?
On-Page Optimization | | SoulSurfer80 -
No Data Available for this URL
Hi,
On-Page Optimization | | ostiguyj
I really don't understand why I have this message "No data available for this URL"
in my SEOMOZ campain. (www.bienchezsoi.ca) When I look at my page rank, I get a score of 0 I have no idea of to fix it. Please help. Thanks0 -
The URL Inside
Howdy SEO'ers, I have a quick question for the SEO gurus out there. When constructing "better" search friendly URLs would one of these be better than the other? Example 1: http://Domain.com/Category/Sub-Category/Product-name Example 2: http://Category.Domain.com/Sub-Category/Product-name In this example the category could be phones and the sub-category brands of phones. Is either one of these URLs "better" than the other in terms of ranking? Thanks! I'll hang-up and listen to your answer. 🙂 Jonathan
On-Page Optimization | | creativedepartment0 -
Should we create separate product descriptions for our customers' web sites?
Still got my SEO learner plates on, but I'm trying to help a small e-commerce site which makes and sells baby products . They have upwards of 150 independent retailers also selling their products. Mindful of the fact that many of these retailers are copying the same product descriptions to use on their own web sites, I wondered if there was any value in creating separate sets of product descriptions, one for our web site and one for all their trade customers, in order to minimise the amount of duplicated content devaluing our site. In theory Google ought to know that ours is the original source of the content, but some testing has shown customers ranking higher for the same product descriptions. We have a separate area on the site for trade, which contains lots of media information they can use, and we could include a set of product descriptions in this area for trade customers to download, keeping a unique set of product descriptions for ourselves. We won't stop duplicate content entirely, I realise - but do you think it's worth the effort of trying to implement? Our web developer thinks it's a total waste of time and not worth bothering with for the miniscule benefit he thinks we'll gain. Grateful for any pointers.
On-Page Optimization | | Mandy_Cochrane0