My website Not crawling by Google when I posted on my website. I can check my newly posted URL on Google search it will not index until 24-48 Hours. but my website is Not new it’s has 2 year 8 Months Old. when the same website on blogger platform it will index within the minute. same checked with URLs indexed in Google.
but my website Now on WordPress platform using genesis framework 2.0 is it responsive sometimes it loads slowly.
it’s on CDN network
when I try to change crawl rate from Google webmaster tools there is a message from Google
Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate.
why i am getting this message because of I am using CDN network from CloudFlare . It’s OK but blogger blogs also getting same message probably i think it’s also shared hosting from Google blog spot platform, but I am using the custom domain.
According to Google Crawl rate
Google’s spiders regularly crawl the web to rebuild our index. Crawls are based on many factors such as PageRank, links to a page, and crawling constraints such as the number of parameters in a URL. Any number of factors can affect the crawl frequency of individual sites.
Our crawl process is algorithmic; computer programs determine which sites to crawl, how often, and how many pages to fetch from each site. We don’t accept payment to crawl a site more frequently. For tips on maintaining a crawler-friendly website, please visit our Webmaster Guidelines.
Crawl rate refers to the speed of Googlebot’s requests during the crawl process. It doesn’t have any effect on how often we crawl or how deeply we crawl your URL structure. If you want Google to crawl new or updated content on your site, use Fetch as Google instead.
So How to make website Google bot Crawl friendly?
- Update content more frequently
- socialise your content with twitter Facebook Google+ LinkedIn personally I use twitter feed for it but Google+ there is no free plugin Available but blogger it automatically asks.
- Internal links to know the content of your website for spiders.
- Maintain page speed my Google page speed insights score 83 and also some suggestion about JavaScript css in above fold. but matt cult’s website scored 93 on Google insights using same genesis theme but no extra script like Google Ad sense sharing icons some other plugins.
- Page rank, links from other websites, Helps to crawl frequently also i not understand Parameters in url
- in WordPress also we need to update ping services Under the “Settings” section on the left-hand side click on the “Writing” tab. Scroll to the bottom of the page and find a box with the title “Update Services”.
- here is the list of ping services for WordPress. install cbnet ping optimizer and check the ‘Enable Pinging’ and ‘Limit excessive pinging’ in short time’ boxes – I set the values to 20 pings in 15 minutes. Any new post/page will now automatically be pinged at the set intervals, you don’t need to do any manual pinging.
- Major thing is check your website XML sitemap rebuilding are not when your posted on New content on your WordPress blog i Encountered this 2 times if you any problems with Google XML sitemap plugin use debug function to see PHP errors this problem faced with the combination of custom permalink structure plugin.
- check Robots.txt rules it’s blocking Google or not fetching from Google webmaster tools.
- Check your server up down alerts use Pingdom free website monitoring feature
- Adjust Google if Crawl rate if you can.
- Use fetch as Google from webmaster if above tips not worked for you. other Non-WordPress users need to ping their websites.
Conclusion
fresh content only helps to increase crawl rate my website airtet.in have 168 back links 2 years old some of newly created websites without any backlinks indexing quickly they are updating their content daily with 3+ posts. but I spending time on installation errors on WordPress net work sites. fighting with spammer on my forums and CPU load on my hosting server.
how to change crawl rate from Google webmaster tools?
Change the crawl rate:
- On the Webmaster Tools Home page, click the site you want.
- Click the gear icon
, then click Site Settings.
- In the Crawl rate section, select the option you want.
The new crawl rate will be valid for 90 days. high level means google bot hits your page 2 times per second.
some web hosting companies like GoDaddy blocks google bot and tells please visit later.
other source google webmaster central blog thread on webmaster forum Google.