- This topic has 2 replies, 1 voice, and was last updated 7 years, 7 months ago by
.
Viewing 3 posts - 1 through 3 (of 3 total)
-
Posts
-
When you disallow a directory on my WordPress adult site, does that include all sub directories or just the main one?
Hey, I am having a blond moment, okay?
Every URL that starts with that text – so subdirectories are included.
Realize however that robots.txt is case sensitive so if you exclude /scripts/ but you have links that are to /Scripts/ the spider can crawl URLs in /Scripts/…
Yeah I kind of figured that, just for some reason, it wouldn’t click in the old noggin.
Viewing 3 posts - 1 through 3 (of 3 total)