Thinking aloud - what if WE could run rogerbot from our desktops?
-
Total, total noob question, I know - but is rogerbot performance bound because of bandwidth and processing capacity? I understand if it is, but I am wondering for those of us with very large sites if we would be able to offload the burden on SEOmoz resources by running our own local licensed version of rogerbot, crawl the sites we want and the upload the data to SEOmoz for analysis.
If this was possible would we be getting more immediate results?
-
On the topic of a private crawl (or distributed crawl), these are cool ideas, but not something we currently have in our plans. Having the crawl centralized allows us to store historic data and ensure polite crawling. This may take a little extra time (we are indeed doing a lot of crawls, as well as processing them and retrieving link data for each of them), but we are actively working on on our infrastructure to reduce our crawling and processing time.
While the first crawl does take a number of days, subsequent crawls are started on the same day each week, and should take roughly the same amount of time to complete, controlling for external factors. So in general you should have fresh crawl data right around weekly, give or take a day or two.
As for your specific crawls, I'd be happy to look into them for you. I'll send you a separate email to discuss.
-
Still waiting for my custom crawl launched on April 28th (it's May 2nd) to complete... seriously? If SEOmoz is overwhelmed, first of all, congratualations on being so popular, but I am not getting any timely data at all.
-
Or, instead of running Rogerbot locally, we could run some distributed processing (ie. BOINC) to help offload some of the pressure from your cloud?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can I make sure a desktoppage is shown in the (desktop) search results instead of the mobile page?
When I search for my brandname, the mobile version of the customer support page is shown in the (desktop) results. We use a m.example.nl mobile webpage. To try to solve the problem, we’ve adjusted the following: Made sure the homepage is marked according to schema.org Homepage expanded with textual content and headings containing our brandname Removed all the textual content from the mobile customer support page Added the mobile customer support page to the mobile sitemap What can we change more in settings/marking/sitemap, to make sure our desktop homepage is shown in the brandname results?
Technical SEO | | WillieBV0 -
Do you think have to re-submit my site to search engines after I made improvements?
Some time ago I started to do SEO for a one-page website and didn't get any positive result: no traffic, no filled in online booking form (yet another, multiple page website offering the same service yielded in multiple filled-in "schedule an appointment" forms). I found out my one-page website was considered to be "keyword-spamming" and converted it to a multiple page one. Its domain authority went up, but it doesn't still bring any traffic. I am thinking maybe I have to let the search engines know that it has been updated so they stop penalizing it? Do you think it might help and if yes, what exactly I should do? Will be thankful very much for any suggestion!
Technical SEO | | kirupa0 -
Fetch as Google Desktop Render Width?
What is Google's minimum desktop responsive webpage width? Fetch as Google for desktop is showing a skinnier version of our responsive page.
Technical SEO | | Desiree-CP0 -
Do you think my client is being hit for duplicate content?
Wordpress website. The client's website is http://www.denenapoints.com/ The URL that we purchase so that we could setup the hosting account is http://houston-injury-lawyers.com, which shows 1 page indexed in Google when I search for site:http://houston-injury-lawyers.com On http://www.denenapoints.com/ there is <link rel="<a class="attribute-value">canonical</a>" href="http://houston-injury-lawyers.com/"> But on http://houston-injury-lawyers.com it says the same thing, <link rel="<a class="attribute-value">canonical</a>" href="http://houston-injury-lawyers.com/" /> Is this how it should be setup, assuming that we want everything to point to http://denenapoints.com/? Maybe we should do a 301 redirect to be 100% Sure? Hopefully I explained this well enough. Please let me know if anyone has any thoughts, thanks!
Technical SEO | | georgetsn0 -
Mobile site ranking instead of/as well as desktop site in desktop SERPS
I have just noticed that the mobile version of my site is sometimes ranking in the desktop serps either instead of as well as the desktop site. It is not something that I have noticed in the past as it doesn't happen with the keywords that I track, which are highly competitive. It is happening for results that include our brand name, e.g '[brand name][search term]'. The mobile site is served with mobile optimised content from another URL. e.g wwww.domain.com/productpage redirects to m.domain.com/productpage for mobile. Sometimes I am only seen the mobile URL in the desktop SERPS, other times I am seeing both the desktop and mobile URL for the same product. My understanding is that the mobile URL should not be ranking at all in desktop SERPS, could we be being penalised for either bad redirects or duplicate content? Any ideas as to how I could further diagnose and solve the problem if you do believe that it could be harming rankings?
Technical SEO | | pugh0 -
I think google thinks i have two sites when i only have one
Hi, i am a bit puzzled, i have just used http://www.opensiteexplorer.org/anchors?site=in2town.co.uk to check my anchor text and forgot to put in the www. and the information came up totally different from when i put the www. in it shows a few links for the in2town.co.uk but then when i put in www.in2town.co.uk it gives me different information, is this a problem and if so how do i solve this | | | | | | | | |
Technical SEO | | ClaireH-184886
| | | | | | | | |0 -
I think my ranking report is not accurate.
I think my ranking are not accurate... It says this but it think it is fault. Under here i put what it says. Is it possible that i get the wrong results? The site is www.mando-control.es And an other questions is : when people in spain search do they automaticly search on google.es or google.com ? | Mando a distancia
Technical SEO | | seoroyal
| Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| Mando distancia | pending | Moved unknown 0 spaces | 2 | Moved unknown 0 spaces |
| Mandos a distancia | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| mandos distancia | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| mandos distancias Keyword History Ranking Analysis | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces | | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| Mando distancia | pending | Moved unknown 0 spaces | 2 | Moved unknown 0 spaces |
| Mandos a distancia | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| mandos distancia | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| mandos distancias Keyword History Ranking Analysis | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |0 -
Mobile site ranking in desktop searches
A robots.txt file is properly implemented on the mobile site to block Googlebot from crawling, yet, when running a site: query for the site in google, it still returns over 104,000 pages from the mobile site in the index. Why could this be happening?
Technical SEO | | craigsmith3330