Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Question about robots file on mobile devices

Intermediate & Advanced SEO

462

Locked

Andy-Halliday last edited by

Hi

We have a robots.txt file, but do I need to create a separate file for the m.site or can I just add the line into my normal robots file.

Ive just read the Google Guidelines (what a great read it was) and couldn't find my answer.

Thanks in Advance

Andy
1 Reply Last reply
Reply Quote 0
Andy-Halliday last edited by

Hi Patrick

I thought that but just wanted to triple check before writing a nice list for my developers tomorrow.

Thanks

Andy
1 Reply Last reply
Reply Quote 0
PatrickDelehanty last edited by

Hi there

Yes, if you have a mobile site, you should create a separate robots.txt and sitemap, among other things.

I would also create a Google and Bing Webmaster Tools profiles for your mobile site so you can watch it with more granularity.

Hope this helps! Good luck!
1 Reply Last reply
Reply Quote 2

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Unanswered questions in forums

What should be done with forum questions that go unanswered for a long time (i.e. year or longer)? Are these types of questions valuable content? Should we opt out of having these types of pages indexed? Since these pages are just one sentences doesn't seem like it is adding value to the site.
Intermediate & Advanced SEO | | nandaMesa

0
Silo Architecture and Mobile First

This goes to the age-old SEO argument - how many links in the navigation. We are a well-known brick and mortar brand We have 20,000 SKUs and over 500 categories and sub-catetgories. 95%+ of our backlinks go to the home page. We don't have a blog, but it's in the works. Our site is not responsive. It serves up different versions based on device type, but is not an "M Dot". Our rankings are pretty strong in spite of a large number of technical SEO issues (different discussion). Currently, our e-commerce desktop site is "Siloed" (I'm new to the company - I didn't do it). The home page links via the top nav to categories. The category pages link to subcategories via sidebar navigation, or via images on the category pages (instead of product images). It's pretty close to textbook silos, and it's very near how I would have designed it. This silo architecture passes the most link juice to our categories which target our highest search volume (head) terms. The categories pass link juice (albeit significantly less) to our subcats which target secondary terms. In terms of search volume and commercial value, our tiers line up very neatly. On average, the targeted subcat terms get about 1/6 of the volume of our head terms. The Silo concept has been around forever, and is evangelized by Bruce Clay and other respected SEOs. Every time I've siloed an ecommerce site, the rankings improve dramatically, so who am I to argue? So, what's the problem? Read on... Our mobile navigation, on the other hand, links to every category and subcategory via flyout navigation (I didn't do this, either). In theory, this distributes an equal amount of link juice to all categories and subcategories. It robs link juice from our categories and passes it to subcategories. Right now, this isn't a problem. Rankings are based on the desktop site, and minor adjustments are made for mobile rankings. When Mobile First rolls out, our mobile nav will be the default navigation for Google, and in theory, link juice distribution across the site will change radically, and potentially harm our rankings for our head terms. I always study site architecture for a number of respected ecommerce sites. Target and Walmart, for example, link to every category and subcategory through their mobile and desktop navigation. Wayfair takes a silo approach on mobile and desktop, linking in tiers. I would argue that Walmart and Target have so much DA/TF/CF that they don't give a damn about targeted link juice distribution - it's all about UX. Wayfair's backlink profile is strong, but it's not Walmart or Target, so they need to be concerned about link juice distribution - hence the silo approach. Have the Google spokespeople said anything about this? I see this as a potential landmine across the industry. Is this something I should be concerned about? Has anyone had any experience with de-siloing a website? Am I making a big deal out of a non-issue? Please - no arguments about usability. UX is absolutely part of the equation. Usability is a ranking factor, but if our rankings and traffic take a nose dive, UX isn't going to matter. This is a theoretical discussion discussion on link juice distribution, and I know that compromises need to be made between SEO and UX.
Intermediate & Advanced SEO | | Satans_Apprentice

0
Schema question

Hi all, We have two Trustpilot schemas (Local Business) on our web pages ( One on desktop / one on mobile) but we are finding that it is not updating the number of reviews in the search results. When using the tool : https://developers.google.com/structured-data/testing-tool/ , the test results are coming back ok. I have two ideas as to why it may not be working; 1) The duplication of the schema code is causing issues 2) We had to change the html code for all of our 50+ backend pages using a search&replace WordPress plugin to save a vast amount of time. Maybe this is plugin related? The fact that the google testing tool gives back positive results adds to the confusion. I test both of the theorised issues to see if it provides a fixes. Can anyone shed some further light on this issue? Is there something obvious I am missing? All responses are greatly appreciated! Thanks, Tom p.s. Example Page: https://www.allcleartravel.co.uk/asthma-travel-insurance/
Intermediate & Advanced SEO | | AllClearMarketing

0
Robots.txt, does it need preceding directory structure?

Do you need the entire preceding path in robots.txt for it to match? e.g: I know if i add Disallow: /fish to robots.txt it will block /fish
/fish.html
/fish/salmon.html
/fishheads
/fishheads/yummy.html
/fish.php?id=anything But would it block?: en/fish
en/fish.html
en/fish/salmon.html
en/fishheads
en/fishheads/yummy.html
**en/fish.php?id=anything (taken from Robots.txt Specifications)** I'm hoping it actually wont match, that way writing this particular robots.txt will be much easier! As basically I'm wanting to block many URL that have BTS- in such as: http://www.example.com/BTS-something
http://www.example.com/BTS-somethingelse
http://www.example.com/BTS-thingybob But have other pages that I do not want blocked, in subfolders that also have BTS- in, such as: http://www.example.com/somesubfolder/BTS-thingy
http://www.example.com/anothersubfolder/BTS-otherthingy Thanks for listening
Intermediate & Advanced SEO | | Milian

0
Google: How to See URLs Blocked by Robots?

Google Webmaster Tools says we have 17K out of 34K URLs that are blocked by our Robots.txt file. How can I see the URLs that are being blocked? Here's our Robots.txt file. User-agent: * Disallow: /swish.cgi Disallow: /demo Disallow: /reviews/review.php/new/ Disallow: /cgi-audiobooksonline/sb/order.cgi Disallow: /cgi-audiobooksonline/sb/productsearch.cgi Disallow: /cgi-audiobooksonline/sb/billing.cgi Disallow: /cgi-audiobooksonline/sb/inv.cgi Disallow: /cgi-audiobooksonline/sb/new_options.cgi Disallow: /cgi-audiobooksonline/sb/registration.cgi Disallow: /cgi-audiobooksonline/sb/tellfriend.cgi Disallow: /*?gdftrk Sitemap: http://www.audiobooksonline.com/google-sitemap.xml
Intermediate & Advanced SEO | | lbohen

0
Embedded mobile page?

I have a client who wants a mobile version of their homepage. Normally, I use responsive design to accomplish this for the SEO benefit, but in this case the client wants very different information on the mobile home page than their regular home page. I don't want to go to a dedicated mobile version of the page because they get a fair amount of mobile traffic and so it would probably have a significant negative impact on their SEO to do so. So I was thinking I would add a hidden div to the home page which includes everything they want on the mobile home page and then use CSS to hide the regular content and show the hidden content if someone reaches the page from a smart phone. What do you think about this idea? Would I run afoul of Google's anti-cloaking "rules"? Has anyone done something like this before? Thanks!
Intermediate & Advanced SEO | | farlandlee

0
Another E-commerce Canonical Question

Hi guys, Quick question: one of our clients has an e-commerce site with a very poor canonical tag setup and thousands of pages of duplicate content. Let's use this as an example: BRAND > Category > Type > Color
Four separate pages/URLs. The BRAND page lists all products.
The Category page lists all BRAND products for that category.
The Type page lists all BRAND products of a specific type in that category.
The Color page lists all BRAND products of a specific type in that category of a specific color. Anyway, these generate four separate URLs: /BRAND
/BRAND/Category
/BRAND/Category-Type
/BRAND/Category-Type-Color Avoiding duplicate content and product listings, I would appreciate your proposed canonicalization strategy/feedback.
Intermediate & Advanced SEO | | elcrazyhorse

0
Archive or no archive?... That is the question!

When running a classified site, what is best practice for what to do with expired ads? Should they stay on the site with a sold stamp perhaps? Or should they be moved to an archive subdomain, with the original URL 301 redirecting to the new archive ad? I'm kinda thinking the second option but I suppose the only issue with this is you would have to have a consistent flow of new ads on the site to prevent categories from getting too thin. Thoughts on this and any other/better solutions would be much appreciated. Thanks.
Intermediate & Advanced SEO | | Sayers

0