Ответ сервера
HTTP/1.1 200 OK Server: nginx/1.2.0 Date: Mon, 26 Nov 2012 10:23:58 GMT Content-Type: text/html; charset=WINDOWS-1251 Transfer-Encoding: chunked Connection: keep-alive X-Powered-By: PHP/5.2.10 Set-Cookie: f6603a382416f3359f2ad788b76134e8=-; path=/ Set-Cookie: virtuemart=289614895732083c547b810946ebe3b9; path=/ Expires: Mon, 26 Jul 1997 05:00:00 GMT Cache-Control: no-store, no-cache, must-revalidate Pragma: no-cache Set-Cookie: VMCHECK=OK Last-Modified: Mon, 26 Nov 2012 10:23:58 GMT Cache-Control: post-check=0, pre-check=0
Файл robots.txt, размер: 1.67 Kb , проверить на
User-agent: * Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /editor/ Disallow: /help/ Disallow: /images/ Disallow: /includes/ Disallow: /language/ Disallow: /mambots/ Disallow: /media/ Disallow: /modules/ Disallow: /templates/ Disallow: /installation/ Disallow: /category_id,25/option,com_virtuemart/Itemid,34/ Disallow: /category_id,26/option,com_virtuemart/Itemid,34/ Disallow: /category_id,27/option,com_virtuemart/Itemid,34/ Disallow: /category_id,28/option,com_virtuemart/Itemid,34/ Disallow: /category_id,29/option,com_virtuemart/Itemid,34/ Disallow: /category_id,30/option,com_virtuemart/Itemid,34/ Disallow: /category_id,31/option,com_virtuemart/Itemid,34/ Disallow: /category_id,32/option,com_virtuemart/Itemid,34/ Disallow: /component/page,shop.browse/category_id,15/option,com_virtuemart/Itemid,34/index.php?&page=shop.browse&category_id=15option=com_virtuemart&Itemid=34&vmcchk=1&Itemid=34 Allow: /component/page,shop.browse/category_id,25/option,com_virtuemart/Itemid,34/ Allow: /component/page,shop.browse/category_id,26/option,com_virtuemart/Itemid,34/ Allow: /component/page,shop.browse/category_id,27/option,com_virtuemart/Itemid,34/ Allow: /component/page,shop.browse/category_id,28/option,com_virtuemart/Itemid,34/ Allow: /component/page,shop.browse/category_id,29/option,com_virtuemart/Itemid,34/ Allow: /component/page,shop.browse/category_id,30/option,com_virtuemart/Itemid,34/ Allow: /component/page,shop.browse/category_id,31/option,com_virtuemart/Itemid,34/ Allow: /component/page,shop.browse/category_id,32/option,com_virtuemart/Itemid,34/ Allow: /component/page,shop.browse/category_id,15/option,com_virtuemart/Itemid,34/