Welcome to our lesson on Crawler Errors and
Rejected Ad Requests. Occasionally, technical issues can get in the way of impressions showing
on your site. They can also lead to less relevant ads. Both scenarios can have a negative impact
on your user experience as well as your revenue. The most common issues you may run into are
crawler errors and rejected ad requests.
In this lesson, we will explain what these are,
what causes them, and how you can fix them to make sure you're not missing out on any
revenue. If you have a substantial number of crawler errors or rejected ad requests,
a red bar will appear at the top of your account home page. You can also check the status of
these issues by clicking the gear symbol on the top right of the screen and selecting
"Status." Let's start with crawler errors. First – what is a crawler? A crawler is the
software used to process and categorize the content of web pages. Specifically, the AdSense
crawler visits your site and scans the content in order to provide relevant ads.
error occurs when our crawler can't access your site's pages. So why is this a problem?
If our crawler can't scan your site, we may not be able to target ads to your content
or even serve ads at all. Fixing this issue may result in increased ad revenue, as relevant
ads are more appealing to users. There are three common causes of crawler errors. The
first is "Robot denied." This error means that our crawler tried to access your page,
but was denied by your robots.txt file. A robots.txt file controls which search engine
robots can crawl your site. The file is located at the domain level for your website, for
example: yoursite.com/robots.txt. To fix this, you can grant our crawler access.
For more information on how to do this, please visit our Additional Resources section.
The next common cause of crawler errors is: "Content behind a login." This error signals
that our crawler cannot access your content because it is behind a login, much like a
"robot denied" error.
For instructions on how to set up a login specific to our crawler,
see the Additional Resources section. The third kind of crawler error is "Page not found."
This means our crawler is receiving a 404 error because the page does not exist. To
fix this, make sure your URL is serving correctly. For example, if you recently moved or deleted
the page, try searching your site for any remaining links in the content or navigation
menus that are pointing to the deleted page.
If you see a "Page Not Found" crawler error
for a page that does exist on your site, check with your webhost or webmaster to see if there
was an outage which affected your website. Alternatively, our crawlers might have discovered
a temporary URL that is no longer used for one of your pages. Now let's talk about rejected
ad requests. If you notice an alert in your account letting you know that your site is
experiencing rejected ad requests that may affect your revenue, click the link in the
alert to see how many ad requests were rejected. If the total number is low then your revenue
is likely unaffected. However, if you're seeing a large volume of rejected ad requests or
if you can't see ads on your site because of rejected ad requests, we'd recommend working
to find a solution. When you see rejected ad request errors, it means our crawler is
targeting the wrong URL. This occurs when our URL detection sees that your ad unit is
within an iframe which has no content.
When we are not able to find the correct site information
for an ad request, we serve "blank ads" that blend into the background of your pages. This
can result in lower revenue. Here are two possible reasons you're seeing these errors:
Your ad code is nested within multiple iframes: If your ad code is nested within multiple
iframes, we are not able to determine the correct site information for an ad request
because the crawler gets stuck in the iFrame. You're using a Supply-Side Platform (SSP):
If you make use of a Supply-Side Platform like an ad server or a yield manager and you're
receiving rejected ad request errors with their URLs, please contact the Supply-Side
Platform to determine the best way to ensure that the correct site information is passed
in your ad requests. The method for fixing targeting depends on the domain that triggers
the ad request error and the type of ad code you're using.
For the specifics of how to
fix ad request errors based on on different types of ad code, see the "Additional Resources"