ROBOTS.TXT GENERATOR — ALLOW / BLOCK BOTS + SITEMAP | SEO READY
100% Private: File aapke browser me banti hai — koi upload nahi.
Leading / zaroori — /admin/ likhne se pura admin folder block ho jayega.
Robots.txt me Sitemap: line Google/Bing ko seedha aapka sitemap batati hai.
LIVE FILE PREVIEW
User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
File ko website ke root folder (public_html) me upload karo — example.com/robots.txt pe khulna chahiye.
ROBOTS.TXT — COMPLETE GUIDE (2026)
Robots.txt Generator Kya Hai?
Robots.txt Generator ek free SEO tool hai jo aapki website ke liye ready-to-upload robots.txt file banata hai — bina kisi syntax yaad kiye. Template chuno (WordPress, Blogger, e-commerce), kaunse bots block karne hain select karo, private folders likho, sitemap URL daalo — aur right side me live preview file ka final code dikhata hai. Copy karo ya direct robots.txt download karo aur apne hosting ke root folder me daal do.
Robots.txt Kaise Kaam Karta Hai? (2 Minute Basics)
File ke rules User-agent groups me bante hain. Har group ek bot ka naam batata hai aur uske liye Allow/Disallow paths likhta hai:
User-agent: * ← sabhi bots ke liye Disallow: /admin/ ← admin folder crawl mat karo Allow: /admin/logo.png ← par ye file allowed User-agent: GPTBot ← sirf OpenAI ka bot Disallow: / ← poori site block Sitemap: https://example.com/sitemap.xml
User-agent: * sabke liye default hai. Specific bot ka group (jaise GPTBot) us bot ke liye * ke rules ko override karta hai — isliye AI bots block karne ke liye alag group chahiye hota hai.
Template Comparison — Aapke Liye Kaunsa
| Template | Kya block karta hai | Kiske liye |
|---|---|---|
| WordPress | /wp-admin/, /wp-login.php, /cart/, /checkout/, /my-account/, search results, /xmlrpc.php, ?replytocom | Koi bhi WordPress site — crawl budget bachata hai |
| Blogger | /search, tag/archive pages | Blogspot ya Blogger custom domain |
| E-commerce | Cart, checkout, account, compare, filter URLs (?sort=, ?price=) | Online stores — thin/duplicate pages crawl budget kha jati hain |
| Allow all | Kuch nahi — sirf sitemap | Simple portfolio, landing page |
| Block all | Poori site | Staging, development, private demo sites |
Sabse Bade Myths — Jo Rank Todte Hain
- "Disallow karne se Google se hat jayega" — Galat! Disallow sirf crawl rokta hai. Page index se hatane ke liye
noindexmeta tag ya Search Console ka Removal Tool use karo. - "Googlebot block kar dunga to spam nahi lagega" — Googlebot block = Google aapki site padh hi nahi payega = zero ranking. Kabhi nahi karna.
- "JS/CSS folders block kar do" — Google page render karne ke liye CSS/JS chahiye.
Disallow: /wp-content/jaisi galti rendering tod deti hai. - "Noindex bhi robots.txt me likh sakta hai" — Noindex robots.txt directive nahi hota; wo meta tag ya HTTP header me aata hai.
Upload Ke Baad Verify Kaise Karein
- Browser me
example.com/robots.txtkholo — file ka content dikhna chahiye (404 nahi). - Google Search Console → robots.txt report — live file aur errors dikhte hain.
- Search Console me kisi bhi page ka URL Inspection chalao — "Crawl allowed: yes/no" saaf batata hai.
- Kisi bhi online robots.txt tester me apna URL daal ke specific paths test karo — "/admin/ block ho raha hai ya nahi" turant pata chalega.
Galti Se Poori Site Block Ho Gayi? — Emergency Fix
Agar live site pe galti se Disallow: / chala gaya hai: (1) turant robots.txt me Disallow: (empty) karke re-upload karo, (2) Search Console me robots.txt report pe jaake "Request a crawl" karo — Google file ko minutes me re-fetch karta hai, (3) 24–48 ghante me crawl normal ho jata hai. Panic me robots.txt delete mat karo — empty Disallow wali file hi sahi hai.
Frequently Asked Questions (FAQ)
robots.txt is a plain text file at your website root (example.com/robots.txt) that tells search engine crawlers which parts of your site they may or may not crawl. You need it if you want to hide private folders (admin, CGI, user data), stop crawl-budget waste on thin pages, point crawlers to your sitemap, or throttle aggressive bots. If your whole site should be public, a simple allow-all file is still worth having because it also carries your sitemap pointer.
Yes — completely free, unlimited use, no signup. Everything is generated in your browser, so nothing you type is uploaded or stored anywhere.
Upload it to the root of your website — exactly one folder above where your homepage files live. For example, on Hostinger cPanel/public_html, place robots.txt directly inside public_html. The file must open at example.com/robots.txt directly in a browser; a file in a subfolder or under a different name will not work.
No — this is the biggest robots.txt myth. Disallowed URLs are not crawled, but they can still appear in results if other sites link to them. To actually remove a page from Google, use a noindex meta tag or the URL Removal tool in Google Search Console. robots.txt controls crawling, not indexing.
For most sites: block AI-training crawlers (GPTBot, CCBot, Google-Extended) if you do not want your content used for AI training, block scrapers like AhrefsBot or SemrushBot only if you genuinely care, and always keep Googlebot, Bingbot and Googlebot-Image allowed unless you have a specific reason. Blocking major search engines will kill your traffic.
Crawl-delay tells a bot to wait N seconds between requests. Googlebot ignores this directive (use Search Console settings instead), but Bing and many smaller bots respect it. Set it only if your hosting struggles with bot load — 10 is a common safe value. On shared hosting with normal traffic, you usually do not need it.
Yes — order matters. First write Allow: /folder/allowed-file.html, then Disallow: /folder/. Google and Bing evaluate the most specific (longest) matching rule, so the Allow line wins for that file while the rest of the folder stays blocked. This generator writes the rules in the correct order for you.
Google rechecks robots.txt roughly every 24 hours, and usually within minutes of you submitting or testing it in Search Console’s robots.txt report. After uploading, open example.com/robots.txt in a browser to confirm it serves correctly, then watch the Search Console report for crawl errors in the next day.