Robots.txt file issues on Shopify server
-
We have repeated issues with one of our ecommerce sites not being crawled. We receive the following message:
Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.
Are you aware of an issue with robots.txt on the Shopify servers? It is happening at least twice a month so it is quite an issue.
-
I leave you 2 url, but it seems that the robots do not let you modify it.
They propose you to put pages as non-index....
https://community.shopify.com/c/Shopify-Discussion/How-can-I-edit-robots-txt/td-p/432485/page/4
https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines
greetings
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ranking issues with my local business website.
Hi, I have local business website "asapgaragedoorrepair.com" which im seeing weird ranking issues during last few months . i would like to share it with you guys and get help from you. first days i have indexed the website ,it was ranking good with google and then after 2-3 weeks i have noticed that none of the pages ranking anymore. i have looked for any issues in the optimization and i fixed them and indexed again. right now i have weird issues. 1- the website was ranking for another page in my website other that the one that i wanted for keyword" Garage Door Repair Pasadena " .(it was ranking with garage door installation page). i changed the content ,optimization on the other page and indexed. still the same issue. then i had no ways other that 301 to the right page . 2- when i check the rankings with "ahrefs" website or "Moz" or other tools that i have, it shows that im ranking for 16th position for " Garage door repair Pasadena ". but when i search it by myself im getting no results. (this is bothering) maybe i dont know something . pls help me 3- in Moz or ahref website shows the ranking like the image i have attached. 2 different results for each keywords. why is that? and why i can not see them when i search them by my self? htPQR
Moz Pro | | Mishel2980 -
Domain authority get down significantly. Internal MOZ Issue? Google Algoritm Update?
I've noticed that several websites have lost DA since middle of october. I attached a picture. Could be this a MOZ internal issue like happened a time ago? Not only the DA get down, I have lost C-blocks links, too
Moz Pro | | NachoRetta
I check in http://health.moz.com/ and found this Crawl Services Outage: http://health.moz.com/incidents/fbn00m7xxzx2
But not appear to have direct relation with my question.
Or may be a google algoritm update? 3anVE5J.png2 -
The pages that add robots as noindex will Crawl and marked as duplicate page content on seo moz ?
When we marked a page as noindex with robots like {<meta name="<a class="attribute-value">robots</a>" content="<a class="attribute-value">noindex</a>" />} will crawl and marked as duplicate page content(Its already a duplicate page content within the site. ie, Two links pointing to the same page).So we are mentioning both the links no need to index on SE.But after we made this and crawl reports have no change like it tooks the duplicate with noindex marked pages too. Please help to solve this problem.
Moz Pro | | trixmediainc0 -
Issue with Data Updates on Campaign
I'm reviewing a campaign I am running for a client, the Keywords data has not updated yet even though it's supposed to update every monday. The competitive analysis shows a date of August 14, which is odd becuase I'm on a trial account?
Moz Pro | | amerihope0 -
Why can't I add my facebook page to SEOMOZ? Also having other facebook issues.
Hi, I have no trouble adding my twitter page in SEOMOZ, but its giving me an error when I try to load my facebook page http://www.facebook.com/pages/Eugene-Computer-Geeks/226660334011653 . I also tried adding my personal facebook page which is tied to the Eugene Computer Geeks facebook page, but SEOMOZ wont accept that either. My business facebook page is tied to my personal account, and its also not showing up on the facebook search. Any idea how I can make my business show up? I wish I could just start over fresh and have my buinsess setup with it's own facebook account. Thanks.
Moz Pro | | eugenecomputergeeks1 -
Why does SEOMoz crawler ignore robots.txt?
The SEOMoz crawler ignores robots.txt It also "indexes" pages marked as noindex. That means it is filling up the reports with things that don't matter. Is there any way to stop it doing that?
Moz Pro | | loopyal0 -
Anyone having issues with OSE?
Everytime i try to access the tool i get the following error: "Error 310 (net::ERR_TOO_MANY_REDIRECTS): There were too many redirects."
Moz Pro | | Anest0 -
Meta description tag in rss xml file?
The SEOmoz crawl diagnostic tool is complaining that I'm missing a meta description tag from a file that is an RSS xml file. In my <channel>section I do have a <description>tag. Is this a bug in the SEOmoz tool or do I need to add another tag to satisify the warning?</description></channel>
Moz Pro | | scanlin0