bb4e8899c4
Google insists on indexing pages so it can figure out if they can be removed from the index. see: https://support.google.com/webmasters/answer/6332384?hl=en This change ensures the we have special behavior for Googlebot where we allow indexing, but block the actual indexing via X-Robots-Tag |
||
|---|---|---|
| .. | ||
| assets | ||
| controllers | ||
| helpers | ||
| jobs | ||
| mailers | ||
| models | ||
| serializers | ||
| services | ||
| views | ||