Ajax4SEO and rogerbot crawling
-
Has anyone had any experience with seo4ajax.com and moz?
The idea is that it points a bot to a html version of an ajax page (sounds good) without the need for ugly urls. However, I don't know how this will work with rogerbot and whether moz can crawl this. There's a section to add in specific user agents and I've added "rogerbot".
Does anyone know if this will work or not? Otherwise, it's going to create some complications. I can't currently check as the site is in development and the dev version is noindexed currently.
Thanks!
-
Hi Philip!
This question is a bit intricate.
With AJAX content like this, I know Google's full specifications
https://developers.google.com/webmasters/ajax-crawling/docs/specification
indicate that the #! and ?escaped_fragment= technique works for their crawlers. However, Roger is a bit picky and isn't robust enough yet to use only the sitemap as the reference in this case. Luckily, one of our wonderful users came up with a solution using pushState() method. Click here:
http://www.moz.com/blog/create-crawlable-link-friendly-ajax-websites-using-pushstate
to find out how to create crawl-able content using pushState. The only other thing I can think of is to run a crawl test if the site is live. You'll have to remove the noindex tag, but updating the robots.txt to allow rogerbot but have a wildcard disallow for the other crawlers should still keep the site from being indexed.
Hopefully this will help!
Best,
Sam
Moz Helpster
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl results on keyword rankings differ from actual search rankings
For the second week in a row, the weekly keyword rankings in my MOZ account show "not in top 50" for most keywords. However, when I check some of my rankings I still broadly have the rankings that I used to have. It seems that the MOZ crawl is experiencing some difficulties. Does anyone have suggestions how to find out what's going on? If I can't rely on the crawl outcomes, then the tool is not very useful anymore!
Moz Pro | | matteis0 -
OK Crawl Test Link Question Again!
I've downloaded a crawl test and column G Link Count reads 62 and yep there are a total of 62 links on the page in question. Column AM Internal Links reads 303 and yep there are somewhere in the order of 303 pages pointing at this one. Root Domains is surprisingly low at 6, so maybe there are only 6 domains linking to this page. BUT... External Links read 51. There are not 51 links pointing away from this domain on this page, no way hozay, so can anybody tell me what is meant by 'External Links? A humble thank you in anticipation of an education. Jem
Moz Pro | | JemRobinson0 -
Its been over a month, rogerbot hasn't crawled the entire website yet. Any ideas?
Rogerbot has stopped crawling the website at 308 pages past week and has not crawled the website with over 1000+ pages. Any ideas on what I can do to get this fixed & crawling?
Moz Pro | | TejaswiNaidu0 -
Data Update for RogerBot
Hi, I noticed that rogerbot still give me 404 for http://www.salustore.com/capelli/nanogen-acquamatch.html refferal form http://www.salustore.com/protocollo-nanogen even I made changes since a couple of week. Same error with one "Title Element Too Short" on our site. Any suggestion on how to refresh it? Best Regards n.
Moz Pro | | nicolobottazzi0 -
How to force SeoMoz to re-crawl my website?
Hi, I have done a lot of changes on my website to comply with SeoMoz advices, now I would like to see if I have better feedback from the tool, how can I force it to re-crawl a specific campaign? (waiting another week is too long :-))
Moz Pro | | oumma0 -
SEomoz slow to crawl?
Hello - I am just trying out the trial and it said the next crawl was nov 1st but I see no change in any of the errors since the initial crawl... so just waiting to find out if what I changed was fixed or not. Is this normal ?
Moz Pro | | Bethany_BabyBrowns0 -
Crawling a website with redirects
Hi, I started a campaign for a website which uses multiple redirects before showing the real content. in the crawling report only one page is crawled. Is there a way to let the crawler pass the redirects to get usefull reports? The website is www.cegeka.be Thank you
Moz Pro | | Cegeka0 -
Crawl Diagnostics and missing meta tags on noindex blog pages
Hi Guys/Gals We do love the Crawl Diagnostics, but do find the missing meta tags ("Missing Meta Description" Tag in this case) somewhat spammy. We use the "All in One SEO Pack" for our blog and it does stick in noindex,follow (as it should) on the pages that is of no use to us. "2008/04/page/2/" and the likes. Maybe I'm wrong but should the Diagnostics tool not respect the noindex tag and just ignore any warnings, since it should really mean that these pages are NOT included in the search index. Meaning that the other meta tags are really useless. Any thoughts?
Moz Pro | | sfseo0