• Home
  • WordPress
  • web Hosting
    • Free hosting
    • Cheap Hosting
    • comparison cloud , vps, shared, wordpress
    • managed wordpress hosting
    • managed cloud hosting
  • page Speed
  • Deals
  • Services
  • About

RAaz Kumar .com

wordpress tutorials seo hosting etc


Perfect robots.txt file for WordPress, blogger [Rules explained]

I have often been modifying the rules of robots.txt to prevent duplicate search queries on Google.

You can monitor by site:yourwebsite.com compare between  posts and search results, after necessary changes submit sitemap to Google.
You can also use google parameter tool best way to find the duplicate issues.

how to create a robots.txt file

  1. Create text document using notepad rename to robots.txt. The txt extension may hide on mostly computers.
  2. Upload robots.txt to your website root directory .
  3. make sure you can access by yourwebsite.com/robots.txt
  4. generally robots.txt comes with default installation scripts like wordpress, zoomla, blogger also.

USe Robots txt with caution if you are misused it prevents google to crawl you may lose traffic. alway test with google search console before applying any rule.

Before blocking: check actual valid pages vs indexed pages (to avoid low quality pages, auto generated pages index in google may cause thin content penalty due to overall site quality)

what is the use of robots.txt file

  1. The main advantage of robots.txt  to block search engines either entire website or some pages.
  2. To block dynamic urls that are cause duplicate and low quality content issues (blocking login page from google).
  3. In wordpress replytocom and /s?= search strings causes duplicate content on search results page and posts.
robots.txt file example
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Allow: /
  • If user agent * means applicable for all search engines like Google,bing,yandex and other.
  • Allow means Allowing crawl, disallow tells search engines not to crawl
  • Here user agent Google media partners like adsense disallowing from crawling content.
If you want to block a directory using robots txt then you have mention
disallow: /website directory name/
blocks the directory that starting from  /website
blocking whole website
disallow: /
it applicable for child directories/ urls
Simple words : Disallow means not crawl, allow means crawling permission to any bot.
advanced
Disallow: /*? means url  that contains ? mark anywhere in url blocks by robots.txt

Most customized robots.txt for WordPress

User-agent: Mediapartners-Google 
Disallow:

User-agent: *
Allow: /

User-agent: *
Disallow: /cgi-bin/
Disallow: /page/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /recommended/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: *?wptheme
Disallow: ?comments=*
Disallow: *?replytocom
Disallow: /wp-content/plugins/
Disallow: /20
Disallow: *feed
Disallow: *?no_redirect=true
Disallow: /?
Disallow: /search/
Disallow: /?s=
Disallow: ?wptouch_switch

User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /

Below are the optional strings depending upon plugin, mobile urls ? if you are getting traffic from twitter, facebook, feedburner and rss that redirected urls also caches by Google bot to prevent that we have to add ?utm source and medium tags,
wptouch_switch=desktop&redirect
?no_redirect=true
?utm_source
utm_medium
?Action
?utm_campaign=
/?from_index
?wptouch_switch
?redirect
?post_id
?no_redirect
?replytocom

Best robots.txt for blogger

User-agent: *
Disallow: /search
Disallow: /*?
Allow: /
Sitemap: http://www.raazkumar.com/sitemap.xml
In blogger Disallow search helps preventing labels and search max results page crawl by google.
we can block Monthly archive pages by meta tags
Disallow: /*?  helps for blocking mobile redirect urls in blogger like yourwebsite.com/?m=1,
may lose some important mobile urls that already indexed and ranking, but rel canonical already set to desktop version but still google indexing country level domains also.
Also look at. htaccess tutorials (we can do much pretty with .htaccess)

Related topics:

  1. Facebook Free basics/ Internet (why we don’t need)
  2. Embed rss feed / label / frame sitemap in blogger post page
  3. htaccess tutorial tips for wordpress #redirect #leverage browser cache #Gzip #KeepAlive
  4. Today’s Gold Rate Silver Price in India Live Gold, silver Price online daily
  5. sbi online internet banking login Registration with ATM card tutorial

Digital Marketing Tutorials (overview)

  • how to make money online worst best methods
  • understanding google adsense report cpc ctr cpa in Adsense
  • youtube seo tutorial to get 4000 hours watch & 1000 subscribers
  • how to make 100$ day on youtube without making videos?
  • Basic SEO Tutorials for beginers
  • My career In blogging & online Internet marketing
  • Facebook Free basics/ Internet (why we don't need)
  • Email Marketing services for your website
  • facebook page like button on website Popup & sidebar
  • Google page layout Algorithm
  • copyscape plagiarism content protection plugins for wordpress
  • website competitive analysis SEO
  • SEO services
  • Freelancer Review
  • how to make Google crawl more frequently on your website
  • How to setup /delete google analytics [working with]
  • website audit for SEO Analysis Tips
  • make money website ideas Simple Guide
  • social media optimization tutorial tips how to effectively use social media for marketing
  • google ranking factors
  • How to verify facebook, twitter, google plus, youtube accounts
  • google adsense payments wire transfer india EFT bank deductions
  • 3+ things that decrease Adsense earnings
  • Diagnosing Website Traffic Drop (AMP https panda, penguin manual penalties)
  • Google SEO Guidelines (Case Study)
  • Adsense Optimization Improve adsense Earnings
  • Google Algorithm updates
  • website ideas for online business with example (successful stories)
  • Perfect robots.txt file for Wordpress, blogger [Rules explained]

 

 

wordpress Pagespeed optimization

Digitalocean free $100 Credit

Cloudways Review pricing promo codes

Siteground cpu limits Pricing

Shared Hosting

Managed wordpress Hosting

Managed Hosting Cloud server

VPS Hosting

Cloud Hosting – Unmanaged

Google cloud Pricing

Aws Pricing

Azure pricing

nginx server tutorials

apache server tutorials

linux sysadmin tutorials

mysql Commands list pdf

LEMP Install Ubuntu

Mysql Performance Tuning

Nginx Performance tuning

Linux Performance tuning

Php -fpm performance tuning

Redis Performance tuning

linux server security

nginx security best practices

wordpress security plugins

 

 

wordpress

 

Top 5  WP Google Analytics Plugins

WP Backup Plugins

Wp Comment Plugins

Top wordpress Security Plugins

WP Seo Plugins

WP Caching Plugins

Best Adsense Plugins for WordPress

Wp social Sharing Plugins

autoshare social media plugins

WP speed Optimization Plugins

Speedup WordPress google Score

More Wp tuts

Server Admin Cloud

 

Installing Nginx LEMP On ubuntu

Installing apache Lamp ubuntu

nginx fastcgi cache enable

php – fpm install  & Configuration

Opcache install & Configure

php -fpm pool manager explained

Mysql Install & Configuration

Redis Object cache install & configure

 

Nginx as Reverse Proxy and Load balancer

Load Balance / auto scaling in google cloud

Linux Commands PDF

Mysql Commands Pdf

Letsencrypt tutorial

mysqldump export & import 

Pagespeed Module install & configure

nginx.conf best file

mysql.conf best file

upgrade ubuntu

© 2023 - All Rights Reserved Disclaimer & Privacy Policy