
Completed
Posted
Paid on delivery
My server logs are filling up with requests from a non-Google, non-Bing “meta” crawler and the spike is chewing through bandwidth. I need help understanding exactly which user-agents or IP ranges are responsible and putting an end to their drain—whether that means a complete block or a well-tuned rate limit. I already have access to my site’s [login to view URL] and can modify it, so you’re welcome to supply the exact directives there. If a stronger measure (.htaccess rules, nginx directives, firewall entries) is smarter, just outline the commands and I’ll apply them or give you temporary access to do so. Acceptance is simple: after your solution goes live, traffic from the offending crawler must drop to a negligible level and overall bandwidth should stabilise. Please bring any log-analysis tools or best-practice snippets you rely on; I’m happy to follow along so I understand the fix for future incidents.
Project ID: 40491656
10 proposals
Remote project
Active 4 days ago
Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs

Hi! I can stop the crawler drain within 24 hours. My plan: (1) analyze a sample of your access logs to pinpoint the exact user-agents (likely meta-externalagent / facebookexternalhit) and the IP ranges behind them, cross-checking against Meta's published ASN (AS32934) so we block the right traffic and nothing else; (2) deliver ready-to-apply directives in layers: [login to view URL] rules, plus nginx (map + limit_req) or .htaccess equivalents, plus firewall entries for the ASN ranges — so the block holds even if the bot ignores [login to view URL]; (3) verify with you that requests drop to negligible levels and bandwidth stabilises, and leave you a short documented runbook so you can handle any future bot spikes yourself. I work with Linux, Nginx and Apache daily and will explain each step as I go, as you requested. To start I only need a log sample (or temporary access). Delivery within 1 day.
₹1,050 INR in 1 day
0.0
0.0
10 freelancers are bidding on average ₹1,193 INR for this job

I am the perfect freelancer for your project as I have extensive experience dealing with server issues and handling Linux systems - a skill you specifically mentioned in your project description. As a Full-Stack developer, I understand the importance of efficient deployment and can create robust solutions using my Node.js and PHP stack expertise. Moreover, my time spent working with Kubernetes (K8s) and Linux administration has given me a deep understanding of infrastructure-related challenges. I believe this will greatly benefit our collaboration since your project requires technical analysis and careful implementation to properly identify user agents or IP ranges and to efficiently block them. I assure you that my solution won't just be an instant fix, but also something that will empower you to handle any similar incidents in the future. I always strive to share my knowledge with clients so they can be self-reliant. Let's work together to bring your excessive crawler problem to an end, ensuring negligible traffic from the problematic meta crawler and stable bandwidth going forward.
₹1,050 INR in 1 day
3.7
3.7

The surge in bandwidth consumption from a non-Google, non-Bing meta crawler indicates an urgent need for both identification and mitigation. By analyzing your server logs with tools like AWStats or GoAccess, I can pinpoint the specific user-agents and IP ranges generating excessive requests. Implementing tailored .htaccess rules or nginx directives will effectively curtail these crawlers while keeping legitimate traffic intact. Deliverables will be ready within 2 days, ensuring immediate relief from log clutter and bandwidth issues. Want me to sketch a quick action plan so you can see the approach?
₹775 INR in 2 days
2.1
2.1

Hello Dear, I am Jagmohan Kumawat, a multidisciplinary digital expert with 5+ years of experience building luxury websites, mobile apps, digital marketing, automations, intelligent systems and high impact creative assets. My focus is always on delivering results with precision, style and longterm scalability. Signature Excellence • Luxury Websites & Web Systems • iOS & Android Mobile App Development • Custom API Integrations • Advanced Workflow Automation & AI Systems • Salesforce CRM Setup, Flows & Cloud Integrations • Professional Branding & UI/UX • Social Media Content & Digital Strategy • E-Commerce Systems & Automation • Premium Video Creation & Cinematic Editing Technology & Creative Mastery Node.js | React | Laravel | WordPress | PHP | JS | Shopify | Android/iOS | Salesforce Cloud | CRM Automations | REST APIs | Firebase | AI Solutions | Figma | Adobe Suite (Illustrator, Photoshop, Premiere Pro, After Effects) Canva Pro | AutoCAD | Motion Graphics & AI Video | Digital Marketing | Content Writer | SEO/SMO I’m ready to take your project forward with precision, creativity and technical excellence. Feel free to share more details. I will prepare a customize plan for your exact needs. Best Regards Jagmohan Kumawat
₹1,500 INR in 7 days
0.0
0.0

hello, i am interested in working on this project. i have experience following detailed instructions and can accurately review, identify, and handle content according to the provided guidelines. i pay close attention to detail, work efficiently, and can deliver high-quality results within the required timeframe. i am available to start immediately and look forward to contributing to the success of this project. thank you for your consideration.
₹1,050 INR in 7 days
0.0
0.0

Hello, best regards, I can assist you with your request by providing a well-structured solution that implements security measures to ensure only authorized personnel can make connections, with each user having a strong, robust password. I can also provide you with regular advice on best security practices to help prevent future risks. All my work is done on-site, ensuring the confidentiality of the data you provide. I look forward to a positive response. Thank you.
₹1,050 INR in 7 days
0.0
0.0

WE'VE JUST COMPLETED A SIMILAR PROJECT. Our previous client faced a similar issue with excessive meta crawlers flooding their server logs and consuming bandwidth. We implemented tailored directives and measures to swiftly resolve the problem. After the project, the client saw a significant reduction in unwanted crawler traffic and a stable bandwidth usage. Their website performance improved, leading to better user experiences and increased efficiency. This success aligns with your goal of reducing server load and optimizing bandwidth allocation. I'm confident that we can deliver a customized solution to address your specific needs effectively. I'd love to chat about your vision and bring it to life! The worst that can happen is you walk away with a free consultation. Regards, Laurence.
₹750 INR in 7 days
0.0
0.0

Pehowa, India
Payment method verified
Member since Jun 3, 2014
₹12500-37500 INR
$10-30 USD
₹750-1250 INR / hour
₹150000-250000 INR
$10-30 CAD
$250-750 USD
₹600-1500 INR
$250-750 USD
₹600-1500 INR
$30-250 USD
₹75000-150000 INR
₹12500-37500 INR
$30-250 USD
$250-750 USD
$10000-20000 USD
₹1500-12500 INR
₹100-400 INR / hour
$30-250 USD
₹750-1250 INR / hour
₹100-400 INR / hour