robots.txt

https://www.howtogeek.com/113439/how-to-change-your-browsers-user-agent-without-installing-any-extensions/ If acces to robots.txt is denied, try changing the user agent 1)Open Firefox ,inside the URL tab search for “about:config” 2)Then search for preference Name: useragent 3)Make a right click then > New > String 4)A dialog box will pop up; edit preference name “general.useragent.override” . 5)Enter the string value for the useragent as “GoogleBot”