Robots.txt Kya Hai — Bots Block, AI Crawlers Aur Sitemap Guide (2026 Hinglish)

Ek chhoti si text file jo aapki poori website ka darwaza control karti hai

Socho — aapki website ka ek aisa darwaza hai jisse Google, Bing, OpenAI ke bots, scrapers, sabse pehle guzarte hain. Us darwaze ka naam hai robots.txt — website root me rakhi ek plain text file, jisme 10-20 lines likh kar aap decide karte ho ki kaunsa bot kahan aa sakta hai aur kahan nahi. Sahi likhi robots.txt aapka crawl budget bachati hai, private folders chhupati hai, sitemap ka raasta batati hai — aur galat likhi file poori site ka Google se rishta tod sakti hai.

Is guide me hum robots.txt ko zero se samjhenge — syntax, user-agents, allow/disallow rules, AI bots blocking, sitemap line, aur sabse zaroori: kaunsi galtiyan kitna nuksan karti hain. Aur haan, hamare free Robots.txt Generator se aap ye file 2 minute me bana sakte ho — 5 ready templates ke saath, bina syntax yaad kiye.

Robots.txt Generator Kya Hai

Robots.txt Generator ek free online tool hai jo aapki website ke liye ready-to-upload robots.txt file banata hai. Aapko sirf ye karna hai: template chuno (WordPress, Blogger, e-commerce, allow-all, block-all), kaunse bots block karne hain tick karo, private folders ki list do, sitemap URL daalo — aur right side me live preview me final file ban-ti jayegi. Copy ya Download dabao, file ko hosting ke root folder me daal do.

Tool ka sabse bada fayda: validation warnings. Agar aapne folder path me leading slash nahi lagaya, ya default policy block-all kar di live site pe — tool turant red warning dega. Ye warnings wo hain jo manually likhte waqt nahi milti, aur jinki kim aap tab bhugatte hain jab traffic girne lagta hai.

Robots.txt Actually Kaise Kaam Karta Hai (Basics — 3 Minute)

Har search engine bot (crawler) website pe aane se pehle ek courtesy follow karta hai: wo pehle example.com/robots.txt fetch karta hai aur dekhta hai ki uske liye kya rules likhe hain. File ka format bahut simple hai — User-agent groups:

User-agent: *            ← sabhi bots ke liye rules
Disallow: /admin/        ← admin folder mat kholo
Allow: /admin/logo.png   ← par ye file allowed hai

User-agent: GPTBot       ← sirf OpenAI ka bot
Disallow: /              ← poori site band

Sitemap: https://example.com/sitemap.xml

Teen core concepts:

• User-agent: Kis bot ke liye rule hai. * ka matlab sabhi bots. Specific naam (GPTBot, Googlebot) us bot ke liye * wale rules ko override karta hai.
• Disallow: Kaunsa path crawl nahi hoga. Empty Disallow (Disallow: ke baad kuch nahi) ka matlab sab kuch allowed.
• Allow: Blocked folder ke andar exception — koi file/folder phir se allowed. Google/Bing sabse lambe (specific) matching rule ko follow karte hain, isliye Allow line kaam karti hai.

Ek baat jo sabse zyada confuse karti hai: robots.txt crawl control hai, index control nahi. Agar aapne koi page Disallow kiya, Google use crawl nahi karega — par agar web pe kahin uska link hai, to wo URL Google results me phir bhi dikh sakta hai (bina content ke). Page ko index se hatane ke liye noindex meta tag chahiye — ye do alag cheezein hain.

5 Templates — Aapke Liye Kaunsa Sahi Hai

WordPress template ka apna logic hai: /wp-admin/ login area hai — crawl karne ka koi SEO fayda nahi; ?s= search results pages thin content hain jo Google ko confuse karti hain; /xmlrpc.php ek purana hack-attempt magnet hai jise crawlers bhi baar-baar maangte hain. Ye sab chhota crawl-budget saving hai jo bade sites pe meaningful hota hai.

AI Bots Ko Block Karna — 2026 Ka Sabse Bada Sawal

Ab har website owner ka ye sawal hai: "kya main OpenAI/Anthropic/CommonCrawl ke bots ko block karun?" Ye bots aapka content AI training me use kar sakte hain. Options:

• GPTBot (OpenAI training), CCBot (Common Crawl — kai AI companies ka data source), Google-Extended (Google ka AI-training feed — Gemini), anthropic-ai (Claude), PerplexityBot — sab robots.txt se blockable hain.
• Block karne ka fayda: aapka content AI models ko training data nahi milega — aapki intellectual property ka control aapke paas.
• Block karne ka nuksan: AI search assistants (ChatGPT search, Perplexity) aapki site ko cite nahi kar payenge — naye traffic sources se aap kat jayenge. Google search pe koi asar nahi (Googlebot alag hai).

Meri practical salah: agar aapka business content/traffic-driven hai (blog, tools, media) to AI-citation traffic future me valuable hai — abhi block mat karo. Agar aapka content proprietary hai (research, paid content, unique data) to GPTBot + CCBot block karna reasonable hai. Robots.txt Generator me ye bots checkboxes se hain — ek click ka decision, aur kabhi bhi badal sakte ho.

Ek honesty note: robots.txt ek courtesy protocol hai — well-behaved bots (Google, OpenAI) ise maante hain, par malicious scrapers ise ignore karte hain. Real protection ke liye rate-limiting aur firewall chahiye. Robots.txt se "polite web" manage hota hai.

Robots.txt Banane Ke 5 Steps

1. Template select karo — apne platform se match karta hua. Ye folders/bots ke defaults set kar dega jo aap baad me customize kar sakte ho.
2. Default policy decide karo — 95% cases me "Allow all" hi rakho aur specific folders Disallow list me daalo. "Block everything" sirf staging ke liye.
3. Private folders add karo — /admin/, /private/, /cgi-bin/, jahan bhi users ya system ka sensitive area hai. Har line ek path, leading / ke saath.
4. Bots choose karo — AI/scrape bots jo aapko block karne hain. Search engines (Googlebot, Bingbot) ko kabhi block mat karo.
5. Sitemap URL + Download — sitemap line sabse neeche aati hai (Google/Bing ise directly padhte hain). Download dabao, file root folder me upload karo, browser me example.com/robots.txt khol ke verify karo.

Upload ke baad Google Search Console → robots.txt report me file ka status 24 ghante me update ho jata hai — wahan se crawl errors bhi dekh sakte ho.

Ek Real Destruction Story — Aur Uska Lesson

Robots.txt ki power samajhne ke liye ek cautionary tale (composite, real patterns se): ek growing startup apni marketing site launch kar rahe the. Developer ne robots.txt likha aur test karte-karte Disallow: / default me chhod diya — socha launch ke baad badal dunga. Launch hui. Website perfect chal rahi thi, design zabardast tha, content solid tha. Par 6 hafte baad bhi Google pe ek bhi page index nahi hua.

Problem dhoondhne me 2 hafte gaye — kisi ko laga sitemap ka issue hai, kisi ne plugins blame kiye. Aakhir me ek SEO freelancer ne 10 second me pakda: robots.txt ki pehli line. File fix hui, request-indexing bheja gaya, aur 3 hafte me pages index hone lage — par wo 2 mahine ki head-start kho gayi, aur launch-window wali PR campaign ka SEO benefit zero hua.

Lesson simple hai: robots.txt 20 lines ki file hai jiska impact poore site pe hota hai — isliye ise serious treat karo. Har deploy ke baad ek baar example.com/robots.txt khol ke dekh lo (10 second ka habit), aur kabhi bhi "temporary" block-all live pe mat chhodo. Robots.txt Generator ki warnings isi liye hain — block-all live policy pe tool khud red flag dikhata hai.

Crawl Budget — Bade Sites Ka Hidden Factor

Googlebot ek site pe limited requests karta hai apne "crawl budget" me — chhoti site pe ye koi issue nahi, par 1000+ pages wali site pe Google ka waqt kahan jata hai, ye matter karta hai. Agar aapke 70% URLs filter/tag/search variations hain, to Google aapke asli content tak aate-aagate thak jayega. Robots.txt in junk URLs ko kaat kar asli pages ka crawl time badhati hai.

Crawl-delay bhi isi family ka concept hai: bot ko har request ke beech N second ka gap dena. Googlebot ise ignore karta hai (Search Console settings use karo), par Bing aur chhote bots maante hain. Shared hosting pe agar bot-load ki problem hai to Crawl-delay: 10 try karo — par zyada mat karna, warna aapki naye pages ki discovery hi slow ho jayegi.

Subdirectories, Subdomains Aur Wildcards — Advanced Rules

Basic allow/disallow ke baad ye 3 advanced concepts aate hain jo badi sites pe daily use hote hain:

• Subdirectory blocking: Disallow: /blog/private/ sirf wo exact path block karta hai — /blog/ ka baaki sab allowed rehta hai. Paths prefix-based matching karte hain: Disallow: /temp /temp/, /temp-files/, /temporary.html — teeno ko block karega. Exact matching chahiye to end me $ lagao: Disallow: /*.pdf$ sirf PDF files ko match karta hai, PDF naam wali folders ko nahi.
• Subdomains: blog.example.com aur www.example.com ke liye alag-alag robots.txt chahiye — har hostname apni root se file serve karta hai. Multi-domain setup me ye yaad rakhna zaroori hai.
• Wildcard patterns: * (any sequence) aur $ (end anchor) — Google/Bing dono support karte hain. Common patterns: /*?utm_ (tracking URLs), /*.pdf$ (PDF files), /search/* (search paths). Par wildcards overuse se asli pages bhi block ho jate hain — ek pattern lagane se pehle uske matches mentally test karo.

Ek aur pro-concept: multiple sitemap lines — robots.txt me ek se zyada Sitemap: lines allowed hain (sitemap index file ho ya alag-alag section sitemaps). Google sab padhta hai. Waise bhi sitemap index file use karna cleaner hai — ek line, saare sitemaps andar.

7 Sabse Bade Robots.txt Myths — Busted

1. "Disallow karne se Google search se page hat jayega" — Nahin. Crawl rukta hai, indexing nahi. Removal ke liye noindex meta tag ya Search Console Removal Tool.
2. "Googlebot block karne se spam nahi lagega" — Googlebot block = Google aapki site padh hi nahi sakta = zero ranking. Ye sabse costly mistake hai.
3. "CSS/JS folders block kar do" — Google mobile-first indexing me page render karta hai — bina CSS/JS ke layout samajh nahi aayega. 2015 ke purane guides mat follow karo.
4. "robots.txt me noindex likh sakta hai" — Noindex robots.txt ka directive hai hi nahi. Google ne ise kabhi officially support nahi kiya.
5. "Block all karke dobara allow karna easy hai" — Block-all wali file live pe pahunchi aur 2 hafte tak gayab rahi — recovery me mahine lagte hain. Staging pe bhi password-protection better hai.
6. "Sitemap robots.txt me zaroori nahi, Search Console me submit kar diya" — Search Console sirf Google ke liye hai; Bing/DuckDuckGo bots robots.txt ki Sitemap line se hi sitemap dhoondhte hain.
7. "Ek hi robots.txt sab subdomains pe chalti hai" — Har subdomain ka apna robots.txt hota hai (blog.example.com ko alag file chahiye).

WordPress Users Ka Special Section — Virtual vs Physical Robots.txt

WordPress me ek interesting technicality hai: WordPress khud koi physical robots.txt file nahi rakhta — wo virtual robots.txt serve karta hai agar physical file na mile. Matlab: agar aapne root me file nahi banayi, to WordPress apne default rules dega (usually sirf /wp-admin/ block + sitemap line if SEO plugin active). Isliye:

• Yoast/RankMath users: Ye plugins "File Editor" me virtual robots.txt edit karne dete hain — par wo tabhi kaam karta hai jab physical file NA ho. Physical file ban gayi to plugin ki virtual file ignore ho jati hai — confusion ka common source.
• Best practice: Physical file hi rakho root me — full control, server-level speed (PHP ko hit hi nahi karna padta), aur plugin hatega to bhi robots.txt intact rahega.
• Multisite: Har sub-site ka apna robots.txt nahi ho sakta WordPress multisite me ek hi root se — custom handling chahiye.

Robots.txt Generator ka WordPress template in defaults ko already include karta hai — aapko bas download karke root me daalna hai, aur plugin ke File Editor me jaake check karna ki wahan koi conflicting rules to nahi.

Upload Kahan Karna Hai — Exact Locations

Verify karne ka tarika: browser me example.com/robots.txt kholo. File ka content seedha text me dikhna chahiye — 404 ya HTML page aa raha hai to location galat hai. Ek aur check: file ka naam exactly robots.txt hai — lowercase, no spelling mistake. Robots.TXT ya robot.txt kaam nahi karega.

Galti Se Poori Site Block Ho Gayi? — Emergency Protocol

Ye panic moment hota hai, par protocol clear hai:

1. Turant fix: robots.txt me Disallow: / hatao — default group me empty Disallow: rakho. File re-upload karo.
2. Fast re-crawl: Search Console → robots.txt report kholo — Google yahan se file ko minutes me re-fetch karta hai. Ya URL Inspection me kisi bhi page ka live test chalao — wo robots.txt bhi re-check karta hai.
3. 24-48 ghante: Crawl resume hone me itna time lagta hai. Panicked ho ke robots.txt delete mat karo — empty-Disallow wali file hi correct state hai.
4. Impact assess karo: Sirf kuch din ka block-all serious long-term damage nahi karta — Google page ki importance dobara evaluate karta hai. 2+ hafte ka block-all deindexing tak ja sakta hai, wahan recovery slow hoti hai.

Prevention better hai: kabhi bhi production site pe direct experiment mat karo. Robots.txt Generator ka live-preview + warnings flow isi liye banaya hai — file ban-ne se pehle hi galti dikh jati hai.

E-commerce Sites Ke Liye Special Rules — Filter URL Jungle

E-commerce ya badi content site chalate ho to robots.txt aapki sabse powerful cost-saving tool hai. Problem ye hai: ek product page 50 tarah se open ho sakta hai — sort by price, filter by color, session IDs, tracking parameters — aur har variation ka alag URL hai. Google inhe duplicate/thin pages samajhta hai aur crawl budget khatakta hai. Robots.txt se ye jungle control karte hain:

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*?price=
Disallow: /*?sessionid=

Wildcard * ka matlab hai: /*?sort= har URL jo beech me kahin ?sort= rakhta hai, block ho jayega. Ye patterns Google aur Bing dono support karte hain. Par ek balance hai: itna aggressive block mat karo ki Google asli product pages tak na pahunch paye — sirf variation URLs block karo, canonical pages khule rakhna.

Ek aur trick: agar query parameters ke saath-saath pretty versions bhi hain (example.com/shoes?color=red aur example.com/shoes/red), to robots.txt block karne se pehle canonical tags se problem solve karne ki koshish karo — robots.txt block karne par Google page ka content nahi dekh payega, aur canonical ko trust nahi karega. Crawl + index dono ka khel samajh ke approach lo.

Real Numbers — Ek Blog Ka 90-Din Ka Experiment

Composite example real-world patterns se: ek Hindi blog jiske 1,800 pages the, par Google sirf 400 ko regularly crawl karta tha. Search Console me "Discovered - currently not indexed" me 900+ URLs pade the. Kya kiya unhone robots.txt se:

1. Week 1: Tag archives, search results (?s=), aur attachment pages block kiye — ye 60% crawl requests kha rahe the.
2. Week 2-4: Googlebot ki activity Search Console Crawl Stats me dekhi — total crawl requests same rahi, par useful pages ka share 30% se 75% ho gaya.
3. Week 6-12: Index coverage dramatically improve hua — purane 900 "Discovered" URLs me se 600+ index hone lage, kyunki Googlebot ab un tak pahunch raha tha.

Yaad rahe: robots.txt ne ek bhi page "improve" nahi kiya — usne sirf Google ka waqt sahi jagah route kiya. Chhoti site (50-100 pages) pe ye effect minimal hoga, par 1000+ pages pe crawl-budget optimization free me milta hai. Isliye bade sites ke SEO checklist me robots.txt hamesha top-3 me hota hai.

10 Golden Rules — Copy Karke Rakh Lo

1. File hamesha root me, naam exactly robots.txt (lowercase).
2. Default policy 95% cases me allow-all + specific Disallows.
3. Googlebot/Bingbot kabhi block nahi — kabhi bhi.
4. CSS/JS folders kabhi Disallow nahi — rendering tod jaati hai.
5. AI bots (GPTBot/CCBot) apne content-strategy se decide karo.
6. Har rule me leading slash — /admin/ sahi, admin/ galat.
7. Sitemap line sabse neeche — full https:// URL.
8. Staging pe block-all theek, par live pe kabhi nahi.
9. Upload ke baad browser + Search Console se verify.
10. Mahine me ek baar robots.txt report me crawl errors dekho.

Robots.txt Testing — Kaise Confirm Kare Ki Rules Kaam Kar Rahe Hain

File live kar di — ab prove karo ki wo actually kaam kar rahi hai. Ye testing ladder follow karo:

1. Browser test: example.com/robots.txt kholo — content dikhna chahiye, 404 nahi.
2. Search Console robots.txt report: Live file dikhegi + parse errors agar koi. "Last downloaded" timestamp bhi milega.
3. URL Inspection (single page): Block kiye huye kisi page ka URL inspect karo — "URL is not available to Google" ya crawl-blocked message aana chahiye. Allowed page pe "Crawl allowed: yes".
4. Crawl Stats: Search Console → Settings → Crawl Stats — dekho Googlebot kahan ja raha hai. Blocked folders ki requests 7-14 din me girna chahiye.
5. Bing Webmaster Tools: Wahan bhi robots.txt tester hai — cross-engine verify free me.

Testing me ek subtle point: robots.txt changes Googlebot ko turant nahi dikhte — Google cached robots.txt use karta hai jo 24 ghante tak stale ho sakti hai (Search Console se force re-fetch kar sakte ho). Isliye testing ke baad 1-2 din dekh ke conclusions lo, same-day panic mat karo.

Robots.txt Glossary — 10 Terms

• User-agent: Bot ka naam jiske liye rule hai.
• Disallow: Blocked path.
• Allow: Exception inside blocked path.
• Crawl Budget: Googlebot ki site pe limited requests ka quota.
• Crawl-delay: Requests ke beech ka gap (seconds).
• Crawling vs Indexing: Padhna vs yaad rakhna — robots.txt sirf crawling control karta hai.
• noindex: Meta tag/HTTP header jo indexing rokta hai — robots.txt se alag.
• Wildcard (*): Pattern matching — /*?sort= sab sort URLs ko match karta hai.
• Sitemap Directive: robots.txt ki last line — sitemap ka location.
• Robots Meta Tag: Page-level instructions (robots.txt se alag, page ke HTML me).

Aksar Puche Jane Wale Sawal

Tool page pe 8 detailed FAQs hain — Robots.txt Generator ke FAQ section me. Short version: robots.txt root me jati hai; ye crawling control karta hai indexing nahi; AI bots checkboxes se blockable hain; crawl-delay sirf kuch bots maante hain; block-all sirf staging ke liye; aur file me sitemap line zaroor dalni chahiye.

Ab Kya? — Technical SEO Ka Agla Kadam

Robots.txt technical SEO ka ek pillar hai. Baaki pillars: sitemap ke liye XML Sitemap Generator, on-page tags ke liye Meta Tag Generator, rich results ke liye Schema Markup Generator, aur content quality ke liye Keyword Density Checker. Saare SEO tools ek jagah: SEO Tools category | All Tools.