robots.txt
In short
robots.txt is a text file on your website that tells search engine robots which parts of the site they are allowed to visit.
The file always lives in the same place: straight under the domain, as in yourdomain.no/robots.txt. It consists of simple lines saying which robots a rule applies to, and which folders they should stay out of. Google states that the file is mainly used to avoid overloading your site with requests.
It does not hide pages
A block in robots.txt looks like it keeps a page out of Google. It does not. According to Google, a blocked page can still appear in search results if other websites link to it. If you want to keep a page out, a noindex tag or a password is the right tool.
The most common mistake is the opposite one. A line blocking the whole site is left behind after development, and the site drops out of the index.
Check yours in one minute
Open yourdomain.no/robots.txt in your browser. If it says "Disallow: /" right under "User-agent: *", the whole site is closed to search engines. Tell whoever runs your website straight away. If the file does not exist, that is not a problem. Google treats it as if there are no restrictions. If you see a line starting with "Sitemap:", it shows where your sitemap lives.
Need a new website?
Get a free design draft of your website's front page, tailored to your business and your goals. You see what your website could look like before you decide, and you commit to nothing.