20 x '400' errors in site but URLs work fine in browser...
-
Hi, I have a new client set-up in SEOmoz and the crawl completed this morning... I am picking up 20 x '400' errors, but the pages listed in the crawl report load fine... any ideas?
example -
-
Most major robots obey crawl delays. You could check your errors in Google Webmaster Tools to see if your site is serving a lot of error pages when Google crawls.
I suspect Google is pretty smart about slowing down its crawl rate when it encounters too many errors, so it's probably safe to not include a crawl delay for Google.
-
Sorry, one last question.
Do I need to add a similar delay for Google Bots, or is this issue specifically a Roger Bot problem?
Thanks
-
Fantastic, thanks, Cyrus and Tampa, prevented many more hours of scratching head!!!
-
Hi Justin,
Sometimes when rogerbot crawls a site, the servers and/or the content management system can get overwhelmed if roger is going to fast, and this causes your site to deliver error pages as roger crawls.
If the problem persists, you might consider installing a crawl delay for roger in your robots.txt file. It would look something like this:
User-agent: rogerbot
Crawl-delay: 5This would cause the SEOmoz crawlers to wait 5 seconds before fetching each page. Then, if the problem still persists, feel free to contact the help team at [email protected]
Hope this helps! Best of luck with your SEO!
-
Thanks Tampa SEO, good advice.
Interestingly, the URL listed in SEOmoz is as follows:
www.morethansport.co.uk/brand/adidas?sortDirection=ascending&sortField=Price&category=sport and leisure
But when I look at the link in the referring page it is as follows:
/brand/adidas?sortDirection=ascending&sortField=Price&category=sport%20and%20leisure
notice the "%" symbol instead of the spaces.
The actual URL is the one listed in SEOmoz but even if I copy and paste the % version, the browser removed the '%' and the page loads fine.
I still can't get the site to throw-up a 400.
-
Just ran the example link that you provided through two independent HTTP response code checkers, and both are giving me a 200 response, i.e. the site is OK.
This question has been asked before on here; you're definitely not the first person to run into the issue.
One way to diagnose what's going on is to dig a little deeper into the crawling report that SEOmoz generated. Download the CSV file and look at the referring link, i.e. on which page Roger found the link. Then go to that page and look if your CMS is doing anything weird with the way it outputs the links that you create. I recall someone back in December having the same issue and eventually resolved it by noticing that his CMS put all sort of weird slashes (i.e. /.../...) into the link.
Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsure about improvement of inner sites
Hi ! I want to improve the ranking of Home on my site. Now I did some on-page SEO on Home itself for the keyword "Webdesign Freiburg". How much does it make sense to put this keyword also into inner sites - although I primarily want to rank Home for this keyword? Does Home profit from putting the keyword into subpages? Cheers and thanks Marc
On-Page Optimization | | RWW0 -
Where would I start with optimizing my site
I am new here and am would love some constructive community feedback about a website I am working on:
On-Page Optimization | | paddyaran
Things like tags, serps, content, loading time, design, functionality, and anything that will improve.... The site is www.aranislands.ie Thanks!!0 -
What are the Best On-Site SEO Practices before an E-commerce Site Goes Live?
Hello, I’m working on a client’s E-commerce website. This website is not live yet. Before the site goes live, I am curious to know what the best practices of On-site SEO are. Please let me know from which factor should I start analyze? Thanks.
On-Page Optimization | | TopLeagueTechnologies0 -
The value of changing URL structure
Dear Moz members, There have been many questions on this forum on this topic but I cannot find one that completely answers my question. We launched our new website about 7 months ago and the website contains around 3.000 product pages. The average page authority of the product pages is quite still quite low (12). The URL structure is built like this: www.website.com/catalog/ID/productname/ (with right keywords in the product name). So e.g the current URL is Our competitors rank higher on certain keywords while page authority (and DA) are significantly lower. Their URL structure is set up like this: www.website.com/productname/. Our most import keyword is "grafmonumenten" and the link we would like to rank on is: https://www.denhollandsche.nl/grafmonumenten/ My question now is:
On-Page Optimization | | stepsstones
1: how important is the length/depth of the URL structure?
2: is it beneficial for us to change the structure (www.website.com/productname/) use permanent redirects? My expierence is that changing the 'page url' can cause a short term drop in the serps, but can have positive effects on the longer term. Thanks for helping me out!0 -
Hierarchy and consistency in ecommerce URLs
One of the first things I remember reading about SEO and URLs, a long time ago, is that keywords are important, and hierarchy is important, for search engines and for users. Hierarchy in URLs would give the search engines an idea of the structure of the site, and users would be able to edit the URLs to continue navigating. I'm wondering about URLs, hierarchy and usability lately, since I've seen that ASOS uses a new URL structure on their site. At first glance, I thought it was brilliant, so I would like to get all of your opinions as well. For those of you that haven't seen the URLs: for categories, ASOS uses a structure as you would expect it, but for products they don't insert the category in the URL. Instead they insert the brand name as the first part of the URL, followed by the product title. Some examples: Category:
On-Page Optimization | | DocdataCommerce
www.asos.com/women/dresses/... Product:
www.asos.com/french-connection/french-connection-tie-waist-pocket-stripe-dress/... I can see the importance of brand name for a site like ASOS, and like how they stressed this by inserting not the category but the brand for products. I don't know how much ASOS still relies on organic non-ASOS related keyword traffic, but still. Now, for hierarchy, I guess a good internal linking structure will tell the search engines about the hierarchy of a site as well, right? So perhaps hierarchy in the URL isn't that important? Perhaps something like this would be just as good as anything, given a good internal link structure? www.onlinestore.com/category/
www.onlinestore.com/subcategory/
www.onlinestore.com/brand/product-title/ Now, I understand that if you use this structure, you wouldn't be able to have men/shirts and women/shirts, but let's say that you don't have subcategories that use the same names. In this case, how important is hierarchy? And, what do you think about this URL structure for an ecommerce site for which brands are important?0 -
SEOmoz crawl error
Hi, I'm getting a crawl error and it complains about there being missing meta description... But, the errors are all for non existent index files in directories that only contain pdf files and some thumbs of the front page... Just started trying to learn this stuff...! Cheers Rod
On-Page Optimization | | DrWho0 -
Number of Pages a Site Contains
Hi, How does the number of pages indexed from a single website affect that websites ability to compete in the SERPS? Cheers
On-Page Optimization | | Jurnii0 -
URL question
Hi guys, the pro campaign thing you got going is wicked, love it. I'm recieving good results with my keywords and have noticed that categories that go beyond sub/sub/sub don't do to well. So I wanna move those that do one step up which makes it go from: http://spytunes.com/practice-guitar/advanced-routine/scales/aeolian to here http://spytunes.com/practice-guitar/advanced-routine/aeolian The existing menu system that follow all these categories across the site will soon go so it won't be a user friendly problem, I will have other type of menus. But, and here is the question: Would I greatly benefit from taking the non existent menu away and just go for: http://spytunes.com/practice-guitar/aeolian while i'm at it? Or do I stick with my current structure? I guess my real question is; how much is there to flat URLs? Cheers -dan lundholm spytunes.com
On-Page Optimization | | spytunes0