20 x '400' errors in site but URLs work fine in browser...
-
Hi, I have a new client set-up in SEOmoz and the crawl completed this morning... I am picking up 20 x '400' errors, but the pages listed in the crawl report load fine... any ideas?
example -
-
Most major robots obey crawl delays. You could check your errors in Google Webmaster Tools to see if your site is serving a lot of error pages when Google crawls.
I suspect Google is pretty smart about slowing down its crawl rate when it encounters too many errors, so it's probably safe to not include a crawl delay for Google.
-
Sorry, one last question.
Do I need to add a similar delay for Google Bots, or is this issue specifically a Roger Bot problem?
Thanks
-
Fantastic, thanks, Cyrus and Tampa, prevented many more hours of scratching head!!!
-
Hi Justin,
Sometimes when rogerbot crawls a site, the servers and/or the content management system can get overwhelmed if roger is going to fast, and this causes your site to deliver error pages as roger crawls.
If the problem persists, you might consider installing a crawl delay for roger in your robots.txt file. It would look something like this:
User-agent: rogerbot
Crawl-delay: 5This would cause the SEOmoz crawlers to wait 5 seconds before fetching each page. Then, if the problem still persists, feel free to contact the help team at [email protected]
Hope this helps! Best of luck with your SEO!
-
Thanks Tampa SEO, good advice.
Interestingly, the URL listed in SEOmoz is as follows:
www.morethansport.co.uk/brand/adidas?sortDirection=ascending&sortField=Price&category=sport and leisure
But when I look at the link in the referring page it is as follows:
/brand/adidas?sortDirection=ascending&sortField=Price&category=sport%20and%20leisure
notice the "%" symbol instead of the spaces.
The actual URL is the one listed in SEOmoz but even if I copy and paste the % version, the browser removed the '%' and the page loads fine.
I still can't get the site to throw-up a 400.
-
Just ran the example link that you provided through two independent HTTP response code checkers, and both are giving me a 200 response, i.e. the site is OK.
This question has been asked before on here; you're definitely not the first person to run into the issue.
One way to diagnose what's going on is to dig a little deeper into the crawling report that SEOmoz generated. Download the CSV file and look at the referring link, i.e. on which page Roger found the link. Then go to that page and look if your CMS is doing anything weird with the way it outputs the links that you create. I recall someone back in December having the same issue and eventually resolved it by noticing that his CMS put all sort of weird slashes (i.e. /.../...) into the link.
Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 and the base URL
Hi, please bear with me as I'm pretty new to all this! I've my base URL but obviously want to add keywords to it for seo purposes. Should I redirect from the base URL to the URL with the keywords appended? So my landing page goes from say www.moz.com to www.moz.com/keywords-here. If I do that, should I replicate all the meta data (descriptions etc) on the original landing page? Or does it not matter? Thanks, Nick
On-Page Optimization | | nickwoodward0 -
From an SEO perspective, which is preferable in the URL for a non-English site: local language or English?
For example, the article is titled ' गर्मी में त्वचा की देखभाल कुछ यूँ करें' (English Meaning: Skin Care Tips for Summer), which one of these three URLs is best for SEO purpose? 1. example.com/ गर्मी-में-त्वचा-की-देखभाल/ (URL in local language, Hindi in this case) 2. example.com/summer-skin-care-tips/ (URL uses English words with English translation) 3. example.com/garmi-mein-twacha-ki-dekhbhaal/ (Hindi words, but written in Roman English) A response will be much appreciated, Amit
On-Page Optimization | | DusBus0 -
Toxic URL???
Hi I have a URL that produced page 1, number 1 to 3 for most of our industries top phrases. Then we received a google penalty, (as did several of our competitors on the same day). We were effectively wiped from google. After much disavowing we were allowed back into the search results, this took about 3 months. I have employed the services of a top London SEO company for over a year now and have seen no significant improvement. I believe they are doing there best, however there results are VERY poor. According to the various tools, (searchmetrics, woorank, semrush) to name but a few, our site scores very well, yet we are not getting the results. Page one seems to be full of totally new websites, most of which I have never heard of, and have appeared from nowhere. Should I scrap our URL and put up a completely new one, and put a redirect from the original one? This would be a biggy since our url has been around for 20 years. Thanks for reading. Andy
On-Page Optimization | | First-VehicleLeasing0 -
URL, Breadcrumb/Site Hierarchy Display, User (and Bot) Expectations
TL;DR: Do parts of URLs that are used throughout the web quite consistently have any influence on robots (or users)? Are there any studies? What would you use for pages that are something between a tag-page and a wiki-like article? Long version: On a site with a lot of content, I decided to go for tags to present articles on that topic together. My first thought was to simply list those under the URL /tag/{Tag_Name}. Short. Simple. Grabs the core meaning - on this page you'll find stuff about the tag. But: those tag-pages will be more than just lists of the tagged pages (let's say they are articles on various topics and products with certain attributes and the same tag can apply to a product and an article). The tag pages themselves will often talk a lot about the use of said tag - extensively, without blabbering. It is aimed at being a landing page and hub for the tag/keyword. Having this in mind, I pondered using /wiki/. It does fit in some respects, but it really is not a wiki. /info/, /lexicon/, /knowledge/ and other ideas came to mind but the more I thought the weirder I did find most ideas. What I am now wondering: Do these parts of URLs (/tag/, or /product/, or /wiki/) that are not really keywords in most cases have any influence on search engines? They are used quite consistently across the web and therefore could be used as signals. I suspect, though, that they might have more influence on shaping user expectation. (If I see /wiki/ in an URL or site hierarchy display (breadcrumb), I expect ... well, a wiki-style page; if I see /tag/ I expect a collection of stuff with that tag.) What would you chose if it is not quite a tag, nor quite a wiki but something in-between? Or do you think it does not matter at all? (Breadcrumbs will be used and google has used them for display in just about all SERPs.) Are there perchance any studies concerning these parts of URLS? Regards Nico
On-Page Optimization | | netzkern_AG0 -
Optimize URL
Hello, My website have been running over five years. I have just reviewed and seen some URLs had not good. It is http://www.vietnamvisacorp.com/faqs/who-need-visa-to-vietnam---1.html, containing characters "---1". Should I remove unnecessary characters "---"?. Thanks for any advice!
On-Page Optimization | | JohnHuynh0 -
Can't see why been marked 'Avoid Keyword Stuffing'
Hi SEOmoz! I'm a newbie, first post, here goes... Working my way through On-Page Report Cards. Noticed this page http://www.vintageheirloom.com/vintage-chanel/vintage-chanel-bags flagged with 'Avoid Keyword Stuffing in Document'. Keyword is 'Vintage Chanel bags' and there is just one instance of it on this particular category page?? Any ideas? Any general pointers for me on www.vintageheirloom.com would also be much appreciated. Thanks SEOmozzers...
On-Page Optimization | | well-its-1-louder0 -
Hi . I have a question regarding a surge in 302 temporary direct errors following reactivation of my site reviews.
Hi. i submitted the following question to the support team at volusion regarding a large number of 302 temporary redirects on the SEOMOZ crawl of following the reactivation of product reviews on my website. Here is the question and their response. Any opinions. Thanks TRANSCRIPT: Hi Howard, Looking at the link you have referenced, this page is unable to be accessed unless you are logged into your site, so SEO isn't applicable for the page. We would not be able to change the 302 redirect for this page. Please don't hesitate to contact us if you have any further questions. Thanks,
On-Page Optimization | | mrkingsley
Meredith G.
Here for you 24x7x365 Volusion Support Check out our online support: http://support.volusion.com | 4 days ago (102hrs ago) [8/19/2012 10:06:00 AM] by Customer | | Hi . I had a report of 302 temporary redirects when SEO moz did a simulated site crawl. This is an example of the address of one of the many 302 temporary redirect errors which I think consume link juice or have some type of negative effect on SEO. http://www.mrgrabbar.com/reviewhelpful.asp?ProductCode=GT100&ID=562&yes=yes Since it is a live link I cannot seem to redirect it with a 301 redirect in the redirect manager. Is there anything I can do to handle these temporary redirects. They only are appearing now since I reactivated my reviews and I am concerned about the SEO effect that this may soon have.0 -
How is my on-site SEO looking like?
I know this is a broad question. My site's content has been written more than one year ago and haven't been changed so far. Our main goal is to make the application hosted in the site work better every day, so we don't worry much about writing content. The URL is http://www.onlinelogomaker.com
On-Page Optimization | | rpedri0