Rogerbot crawls my site and causes error as it uses urls that don't exist
-
Whenever the rogerbot comes back to my site for a crawl it seems to want to crawl urls that dont exist and thus causes errors to be reported...
Example:- The correct url is as follows:
/vw-baywindow/cab_door_slide_door_tailgate_engine_lid_parts/cab_door_seals/genuine_vw_brazil_cab_door_rubber_68-79_10330/
But it seems to want to crawl the following:
/vw-baywindow/cab_door_slide_door_tailgate_engine_lid_parts/cab_door_seals/genuine_vw_brazil_cab_door_rubber_68-79_10330/?id=10330
This format doesn't exist anywhere and never has so I have no idea where its getting this url format from
The user agent details I get are as follows:
IP ADDRESS: 107.22.107.114
USER AGENT: rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected]) -
The first thing I would do is download the crawl report as an excel sheet. You can do this from your crawl report page.
From there, sort by the 404 error column, bringing "True" to the top. The top of the list is now the broken URL's. One of the very last columns on the right is the "referrer" column. This will show you the page where Roger is getting the bad link from.
Make Sense?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Using Weglot on wordpress (errors)
Good day to you all, Does anyone have experience of the errors being pulled up by Moz about the utility of the weglot plugin on Wordpress? Moz is pulling up URLs such as: https://www.ibizacc.com/es/chapparal-2/?wg-choose-original=false These are classified under "redirect issues" and 99% of the pages are with the ?wg-choose parameter in the URL. Is this having an actual negative impact on my search or is it something more Moz related being highlighted. Any advice be appreciated and a resolution .. Im thinking I could exclude this parameter.
Moz Pro | | alwaysbeseen0 -
How to fix the Crawl Diagnostics error and warnings
hi im new to the seo world and i dont know a lot about it , so after my site get crawled i found 1 error and 151 warning and 96 notices , it that bad ?? and plz cam someone explain to me how to fix thos problem , a will be very thankful
Moz Pro | | medlife0 -
What's the best way for SEO newbie to analyze & fix a site after being hit by Panda?
Hi, I have a prospective client who was in the top 3 on Google for two of their primary keywords. They fell way back in the rankings immediately after Panda was rolled out on September 27, 2012. Two weeks ago, they were at position #118 for one keyword. After looking for them in Rank Checker today, they cannot be found in Google at all. Here's my question. Because of the "bad links" (some pointing to Porno sites)... what's the possibility that this situation cannot be fixed? I don't know... maybe I'm asking an irrelevant question. I'm attempting to assess the situation so I can go back and present my findings to the prospective client. I'm committed to understanding what's going on with their website so I can assess the situation properly. Fixing their problem starts with a correct assessment. They have a ranking problem, and I know I can fix that... IF all their site needs is white hat <acronym title="Search Engine Optimization">SEO</acronym>. What I DON'T want is... to sell them <acronym title="Search Engine Optimization">SEO</acronym> services, only to find out in 3-6 months... I made an incorrect diagnosis of the problem, and therefore sold them the wrong solution. I know I can close the sale if I can show them with reasonable substantiation that the damage is not beyond repair. I'm familiar with the basics of <acronym title="Search Engine Optimization">SEO</acronym>, but I'm unfamiliar with how "bad linking" might effect the long-term commitment to optimizing. They're wondering if they should start over on another website. I was attempting to do an assessment to better understand if my typical approach on this site would be sufficient. Also... I wanted to get an assessment/report to show them something to substantiate my conclusion(s) about their website. If Open Site Explorer is sufficient to do the link analysis... great. At least I know I'm working with the right tool. All I have to do is learn how to use the tool quickly. At this point... I'm not sure which tool would be helpful. So... can you speak to the following 2 questions: 1.) How do you know when ranking problem is beyond fixing? 2.) What software/tool is ideal for doing some link analysis in order to assess the problem, and prescribe a solution? Thanks so much! Ramon
Moz Pro | | 48dayscoach0 -
"Loading" error in Open Site Explorer
The "top pages" keeps hanging while trying to retrieve social metrics "FB Shares" "Tweets" and "Google +1". I've tried in Chrome and Firefox. Can I just turn these off? ose-hanging.jpg
Moz Pro | | cpaddock0 -
Crawl Diagnostics - Crawling way more pages than my site has?
Hello all, I'm fairly new here, more of a paid search guy dabbling in SEO on the side. I have a client that I have in SEOMoz and the Crawl Diagnostics report is showing 10,000+ pages crawled and I think the site has at most 800 pages (e-commerce site using freewebstore.org as the platform). Any reasons this would be happening?
Moz Pro | | LodestoneGen0 -
SEOMOZ Crawling Our Site
Hi there, We get a report from SEOMOZ every week which shows our performance within search. I noticed for our website www.unifor.com.au that it looks through over 10,000 pages, however our website sells less than 500 products so not sure why or how so many pages are trawled? If someone could let me know that would be great. It uses up a lot of bandwidth doing each of these searches so if the amount of pages being trawled reduced it would definitely assist. Thanks, Geoff
Moz Pro | | BeerCartel750 -
How do I retrieve crawl and ranking data about a site from the past?
Hey. One of my main clients has asked to see the crawl data and rankings data for the past eight months. He wants to have tangible evidence of the effects of Penguin. I would like that info too. Is it possible to retrieve that information on a weekly crawl and ranking basis through SEO Moz and if so, how do you do it? I simply want to show a graph, timeline and brief explanation across several main keywords... Help me as you guys always do - You rock Best Ben
Moz Pro | | creativeguy0 -
Crawl test. Bot crawled only 200 or so links when it should have crawled thousands
Hi everyone, I just recieved my crawl test report and its only given me 200 or so URL's when my site has thousands, any thoughts?
Moz Pro | | Ev840