Updating a robots.txt file with WordPress.

After trying to implement my own tip on this site, I learned something new about WordPress.

I could see that a /robots.txt response was coming through, but from where? This threw me off plenty, as I could not find the file anywhere at all on the file system.

What finally helped me move in the right direction was the constant wondering — even while trying to sleep at night — about how a plugin could modify files not in its scope!

The answer was it couldn’t. So, WordPress must be doing the heavy lifting. And it is!

The robots.txt file is generated by WordPress using hooks. Specifically, the robots_txt hook.

So you’d hook into it like so:

/**
 * Disallow Google Images from indexing my pictures.
 * Normally you'd use $public to decide whether
 * to hide something at all. If search engine
 * indexing is disabled, we're really not doing
 * anything by updating the robots.txt file.
 */
function protect_privacy_my_photos( $output, $public ) {
	$output .= "\nUser-agent: Googlebot-Image\nDisallow: path/to/image.jpg";
	return $output;
}

/**
 * Hook name: robots_txt
 * Function to call: protect_privacy_my_photos
 * Priority: 20. Core WordPress is generally 10.
 * Arguments to function called by WP: 2.
 */
add_filter('robots_txt', 'protect_privacy_my_photos', 20, 2);

Join the discussion on Mastodon, Twitter, or write me an email.