https://www.howtogeek.com/113439/how-to-change-your-browsers-user-agent-without-installing-any-extensions/
If acces to robots.txt is denied, try changing the user agent
1)Open Firefox ,inside the URL tab search for “about:config”
2)Then search for preference Name: useragent
3)Make a right click then > New > String
4)A dialog box will pop up; edit preference name “general.useragent.override” .
5)Enter the string value for the useragent as “GoogleBot”