Thinking aloud - what if WE could run rogerbot from our desktops?
-
Total, total noob question, I know - but is rogerbot performance bound because of bandwidth and processing capacity? I understand if it is, but I am wondering for those of us with very large sites if we would be able to offload the burden on SEOmoz resources by running our own local licensed version of rogerbot, crawl the sites we want and the upload the data to SEOmoz for analysis.
If this was possible would we be getting more immediate results?
-
On the topic of a private crawl (or distributed crawl), these are cool ideas, but not something we currently have in our plans. Having the crawl centralized allows us to store historic data and ensure polite crawling. This may take a little extra time (we are indeed doing a lot of crawls, as well as processing them and retrieving link data for each of them), but we are actively working on on our infrastructure to reduce our crawling and processing time.
While the first crawl does take a number of days, subsequent crawls are started on the same day each week, and should take roughly the same amount of time to complete, controlling for external factors. So in general you should have fresh crawl data right around weekly, give or take a day or two.
As for your specific crawls, I'd be happy to look into them for you. I'll send you a separate email to discuss.
-
Still waiting for my custom crawl launched on April 28th (it's May 2nd) to complete... seriously? If SEOmoz is overwhelmed, first of all, congratualations on being so popular, but I am not getting any timely data at all.
-
Or, instead of running Rogerbot locally, we could run some distributed processing (ie. BOINC) to help offload some of the pressure from your cloud?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Running 2 different domains
Hi guys, Something i'd like appreciate any opinion on... We own a local domain www.goodycard.co.nz but we're expanding overseas. I've purchased www.goodyhq.com as we'd like to drop the 'card' from URL. We can't get www.goodycard.com. The local site has a decent ranking, but obviously, the newly created HQ site doesn't. What's the best way to tie these in (or even ditch the .co.nz) but retain rank? And what's the best way for users to discover the HQ site? Any insight would help, as both sites are kind of different. One has a local member base and the other just sells software so it really depends on the region of search. Cheers!
Technical SEO | | r.moss0 -
GWT Fetch & Render displays desktop version of site as mobile
Hi team, I noticed that when I request a desktop rendering in GWT using fetch and render, pages render as the mobile version. Screenshot attached. It's related to the VHS units in our CSS (as far as I'm aware). Does anyone know what the implications of this may be? Does it mean googlebot can only see the mobile version of our website? Any help is appreciated. Jake jgScJ
Technical SEO | | Jacobsheehan0 -
Should I redirect desktop users visiting mobile page to desktop version?
So I am redirecting mobile users to mobile version of the page and also have alternate attrubute set up for that: What about the opposite case? When user from desktop computer visits mobile version of the page. Should I redirect him back to desktop version?
Technical SEO | | poiseo1 -
Mobile & desktop pages
I have a mobile site (m.example.com) and a desktop site (example.com). I want search engines to know that for every desktop page there is a mobile equivalent. To do this I insert a rel=alternate on the desktop pages to the mobile equivalent. On the mobile pages I insert a rel=canonical to it's equivalent desktop page. So far so good BUT: Almost every desktop page has 4 or 5 copies (duplicate content). I get rid of this issue by using the rel=canonical to the source page. Still no problem here. But what happens if I insert a rel=alternate to the mobile equivalent on every copy of the source page? I know it sounds stupid but the system doesn't allow me to insert a rel=alternate on just one page. It's all or nothing! My question: Does Google ignore the rel=alternate on the duplicate pages but keeps understanding the link between the desktop source page & mobile page ? Or should I avoid this scenario? Many Thanks Pieter
Technical SEO | | Humix0 -
Mobile URLs in the desktop SERPs
Our real estate website URL is listed on desktop search as well as the mobile URL. I've read several blogposts on this subject but I still don't understand the fix for this. I've read to use rel=canonical tags. But does that stop Google from listing it in the desktop SERP? Is there a way to stop this without blocking the mobile site which is what our programmer wants to do? Or is this something we have to live with until Google fixes this issue?
Technical SEO | | MassMedia0 -
Mobile URL parameter (Redirection to desktop)
Hello, We have a parallel mobile website and recently we implemented a link pointing to the desktop website. This redirect is happening via a javascript code and results in a url followed by this paramenter: ?m=off Example:
Technical SEO | | echo1
http://www.m.website.com redirects to:
http://www.website.com/?m=off Questions: Will the "http://www.website.com/?m=off" be considered duplicate content with "http://www.website.com" since they both return the same content? Is there any possibility that Google will take into consideration the url ending in "/?m=off"? How should we treat this new url? The webmaster tools URL parameter configuration at the moment isn't experiencing problems but should we submit the parameter anyway in order not to be indexed or should we wait first and see the error response? In case we should submit this for removal... what's the best way to do it? Like this? Parameter: ?m=off Does this parameter change page content seen by the user? - doesn't affect page content Any help is much appreciated.
Thank you!0 -
I think my ranking report is not accurate.
I think my ranking are not accurate... It says this but it think it is fault. Under here i put what it says. Is it possible that i get the wrong results? The site is www.mando-control.es And an other questions is : when people in spain search do they automaticly search on google.es or google.com ? | Mando a distancia
Technical SEO | | seoroyal
| Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| Mando distancia | pending | Moved unknown 0 spaces | 2 | Moved unknown 0 spaces |
| Mandos a distancia | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| mandos distancia | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| mandos distancias Keyword History Ranking Analysis | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces | | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| Mando distancia | pending | Moved unknown 0 spaces | 2 | Moved unknown 0 spaces |
| Mandos a distancia | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| mandos distancia | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |
| mandos distancias Keyword History Ranking Analysis | pending | Moved unknown 0 spaces | 1 | Moved unknown 0 spaces |0