Sitemap: https://www.nichejewellery.co.uk/sitemap.xml User-agent: * Disallow: /index.php* Disallow: /catalog/product_compare* Disallow: /catalog/category/view* Disallow: /catalog/product/view* Disallow: /wishlist* Disallow: /admin* Disallow: /nichej_magento* Disallow: /catalogsearch* Disallow: /checkout* Disallow: /onestepcheckout* Disallow: /customer* Disallow: /review/product* Disallow: /sendfriend* Disallow: /enable-cookies* Disallow: /LICENSE* Disallow: /LICENSE* Disallow: /skin* Disallow: /js* Disallow: /directory* Disallow: /news/tag* Disallow: /themesettings* #Lets consider each groups of commands separately. #Stop crawling user account and checkout pages by search engine robot: Disallow: /checkout* Disallow: /onestepcheckout* Disallow: /customer* Disallow: /customer/account* Disallow: /customer/account/login* #Blocking native catalog and search pages: Disallow: /catalogsearch* Disallow: /catalog/product_compare* Disallow: /catalog/category/view* Disallow: /catalog/product/view* #Sometimes Webmasters block pages with filters.. Disallow: /*?dir* Disallow: /*?dir=desc Disallow: /*?dir=asc Disallow: /*?limit=all Disallow: /*?mode* #More reasonable to use canonical tag on these pages. #Blocking CMS directories. Disallow: /app/ Disallow: /bin/ Disallow: /dev/ Disallow: /lib/ Disallow: /phpserver/ #Disallow: /pub/