![]() |
|
Welcome to the Australian Ford Forums forum. You are currently viewing our boards as a guest which gives you limited access to view most discussions and inserts advertising. By joining our free community you will have access to post topics, communicate privately with other members, respond to polls, upload content and access many other special features without post based advertising banners. Registration is simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. Please Note: All new registrations go through a manual approval queue to keep spammers out. This is checked twice each day so there will be a delay before your registration is activated. |
|
Site Support If something isn't working or you have a suggestion ( a nice one !! ) let us know here. |
![]() |
|
Thread Tools | Display Modes |
![]() |
#1 | ||
Chairman & Administrator
![]() ![]() Join Date: Dec 2004
Location: 1975
Posts: 107,001
|
Good afternoon.
No, I'm not posting about some exotic 8-legged creature but about the web crawlers that have been hammering this site for the last 5 days or so. These crawlers are ignoring the normal directives that robots are supposed to follow and thus we have several hundred of them crawling the site constantly. The impact is that the server CPU's get maxxed out at 100% and overall performance slows to a (no pun intended) crawl. They can normally be kept at bay by blocking them at the server level but for some reason this doesn't appear to be stopping this lot of Huawei crawlers. There have been problems reported around the world from them so we aren't the only ones being impacted as Huawei try to get their search engine up to the standard of Google as part of their ongoing war with each other. We are working with out ISP to find a solution.
__________________
Observatio Facta Rotae
|
||
![]() |
![]() ![]() |
31 users like this post: | 1970XW351, aussiblue, Bam, Beastie, Burnout, Charliewool, Citroënbender, DaveD, DFB FGXR6, DJM83, FairmontGS, FERG_51, five 7, FormulaFG, FoxtrotGolfXray 5.0, Itsme, kcodezd, knight rider, Linz, mad2, MITCHAY, Peuty, Pis-ton broke, pottery beige, Rallye Sport, Silver Ghia, sr71, Tickford., wodahs, Work Horse, yakcam |
![]() |
#2 | ||
DIY Tragic
Join Date: Apr 2018
Location: Sydney, more than not. I hate it.
Posts: 21,782
|
Thank you. I’d assumed the issue ran downstream and was just my problem.
|
||
![]() |
![]() ![]() |
2 users like this post: |
![]() |
#3 | ||
FF.Com.Au Hardcore
Join Date: Oct 2020
Posts: 670
|
How about blocking large swathes of chinese IP's in cpanel, crude but could be a stopgap fix. I doubt youd exclude any real members with this approach.
I realise you may already have done this, thought id mention it. |
||
![]() |
![]() ![]() |
This user likes this post: |
![]() |
#4 | ||
Chairman & Administrator
![]() ![]() Join Date: Dec 2004
Location: 1975
Posts: 107,001
|
Already done that!
__________________
Observatio Facta Rotae
|
||
![]() |
![]() ![]() |
5 users like this post: |
![]() |
#5 | |||
FG XR6 Ute & Sedan
![]() Join Date: Oct 2006
Location: Bibra Lake WA
Posts: 22,893
|
I was going to suggest "nuke em till they glow and then shoot 'em in the dark"
But they are the ones with all the nukes and they just want to be loved it seems https://www.bbc.com/news/world-asia-china-57327177 Quote:
__________________
regards Blue |
|||
![]() |
![]() ![]() |
![]() |
#6 | ||
FG XR6 Ute & Sedan
![]() Join Date: Oct 2006
Location: Bibra Lake WA
Posts: 22,893
|
Any of the comments on this blog helpful https://www.johnlarge.co.uk/blocking...scrapers-bots/ ?
__________________
regards Blue |
||
![]() |
![]() ![]() |
![]() |
#7 | |||
Chairman & Administrator
![]() ![]() Join Date: Dec 2004
Location: 1975
Posts: 107,001
|
Quote:
__________________
Observatio Facta Rotae
|
|||
![]() |
![]() ![]() |
![]() |
#8 | ||
Chairman & Administrator
![]() ![]() Join Date: Dec 2004
Location: 1975
Posts: 107,001
|
It 'might' be resolved but there are quite a few of the spiders still connected so we'll need to wait until their sessions expire before knowing for sure.
__________________
Observatio Facta Rotae
|
||
![]() |
![]() ![]() |
5 users like this post: |
![]() |
#9 | ||
The Terrain Tamer
![]() ![]() Join Date: May 2013
Posts: 36,348
|
Maybe closing the Covid Thread might have an impact...
![]()
__________________
Current Ride : A Ford owned D3... |
||
![]() |
![]() ![]() |
![]() |
#10 | ||
Peter Car
Join Date: Dec 2004
Location: geelong
Posts: 23,145
|
Have you tried aeroguard?
|
||
![]() |
![]() ![]() |
2 users like this post: |
![]() |
#11 | ||
FF.Com.Au Hardcore
Join Date: Oct 2020
Posts: 670
|
|
||
![]() |
![]() ![]() |
This user likes this post: |
![]() |
#12 | ||
FF.Com.Au Hardcore
Join Date: Jul 2005
Location: Melbourne
Posts: 6,890
|
Is it the sheer number of crawlers or are the crawlers doing weird stuff? Asked some of the folks at work, if the former, they were pointing as some form of DDOS protection....but I believe that is big $$$.
|
||
![]() |
![]() ![]() |
This user likes this post: |
![]() |
#13 | ||
🚫⏰4️⃣🐃💩
Join Date: Sep 2012
Posts: 1,901
|
I think I've found what these "Chinese Spiders" are trying to attack...
https://fordforums.com.au/showthread.php?t=11484389 ![]() |
||
![]() |
![]() ![]() |
![]() |
#14 | ||
FF.Com.Au Hardcore
Join Date: Nov 2005
Location: perth
Posts: 4,355
|
![]() ![]() we all know what thread their here (like every one else) for !! https://fordforums.com.au/showthread.php?t=11483655 ![]() ![]()
__________________
yes still (as money n time permit) doing the rebuilding the zh fairlane with a clevo 400m 4v heads injected whipple blown with aode 4 speed trans to a 9" ....... we'll get there eventually just remember don't be afraid to try something new. Remember, amateurs built the Ark...Professionals built the Titanic! I have taken up meditation... at least it's better than sitting around doing nothing !! |
||
![]() |
![]() ![]() |
4 users like this post: |
![]() |
#15 | ||
Banned
Join Date: Jan 2009
Posts: 1,621
|
there jealous because they don't have 351's and xa to xc coupes, oh and not forgetting xy gtho phase 3's.
|
||
![]() |
![]() ![]() |
![]() |
#16 | ||
Donating Member
![]() Join Date: Mar 2007
Location: Heading thru Hell (Corner)
Posts: 8,161
|
I thought they were all just salivating over the delicious delicacies in this thread
![]()
__________________
Labels are for jars, not for people. Life is a journey, not a destination. ~~~~~~~~~~~~~~ Daily: 2013 FGII EcoLPi in Winter White Play: 2015 FG X XR8 in Emperor Show' N Shine thread Gone, but not forgotten: 2015 SZII petrol Titanium Territory in Emperor |
||
![]() |
![]() ![]() |
![]() |
#17 | |||
Chairman & Administrator
![]() ![]() Join Date: Dec 2004
Location: 1975
Posts: 107,001
|
Quote:
The (useless) answer from our ISP was to install a Fortinet Firewall in front of our server for an extra US$50 / month but I've managed to block a number of them and I can still use the Linux iptables firewall if I need to. To give you an idea of the impact (and why it is hurting us) here are some raw numbers.... This time of year our data usage averages 15 Gb / day but since May 28th the average has been 141 Gb / day. Likewise, we would normally serve ~500k pages a day with 700k server hits but the averages since 28th May have been 3.3M pages and 3.5M hits. It is improving gradually and there are currently no Huawei spiders active but there are some other Chinese ones like Baidu that I still need to knock on the head.
__________________
Observatio Facta Rotae
|
|||
![]() |
![]() ![]() |
20 users like this post: |
![]() |
#18 | ||
Regular...with metamusal
![]() Join Date: Oct 2009
Location: Geeeloong
Posts: 6,345
|
is that what happened earlier?
|
||
![]() |
![]() ![]() |
2 users like this post: |
![]() |
#19 | ||
Chairman & Administrator
![]() ![]() Join Date: Dec 2004
Location: 1975
Posts: 107,001
|
No - that was me being stupid and restarting mySQL without confirming that the 'forum' user had access to the database.
I'm going to have to take it down again to fix that problem as the temporary workaround in place isn't very good.
__________________
Observatio Facta Rotae
|
||
![]() |
![]() ![]() |
4 users like this post: |
![]() |
#20 | ||
T3/Sprint8
![]() Join Date: Jan 2005
Location: Australia
Posts: 16,313
|
expect so for couldn't load page up some 10mins ago......edit, ah ok it was Russell stand corrected.
and people made noise about our stance re Huawei. Glad the Poms went back on this as well, they quoted in the end it would run their IT period in time, more so CCP could just shut down/control whenever they wish. Baidu friggin their own google. All power to you Russell.
__________________
Tickfords T3/TS50 '02 Sprint8 manual Sept 24 '16 Daily Macan GTS "Don't believe everything you read on the internet. Abraham Lincoln" |
||
![]() |
![]() ![]() |
![]() |
#21 | ||
Chairman & Administrator
![]() ![]() Join Date: Dec 2004
Location: 1975
Posts: 107,001
|
Just as an FYI, I am going to shut down the server at 3:15 for the 5 minutes it will take to fix the database issue.
Apologies but it is necessary.
__________________
Observatio Facta Rotae
|
||
![]() |
![]() ![]() |
![]() |
#22 | ||
Chairman & Administrator
![]() ![]() Join Date: Dec 2004
Location: 1975
Posts: 107,001
|
Took more like 12 minutes but it's resolved now.
__________________
Observatio Facta Rotae
|
||
![]() |
![]() ![]() |
9 users like this post: |
![]() |
#23 | ||
FF.Com.Au Hardcore
Join Date: Jun 2011
Location: Pt Lincoln far side South Oz
Posts: 5,756
|
![]() ![]() ![]() ![]()
__________________
Dont p i s s off older people. At our age the term Life in Prison is not a deterrent |
||
![]() |
![]() ![]() |
![]() |
#24 | ||
FF.Com.Au Hardcore
Join Date: Apr 2005
Location: Canberra
Posts: 13,372
|
Thanks Russ. I thought something was not quite right.
I'm not sure what model we use here but it could be quite a difference in cost with that sort of demand. I know that Google collect all sort of **** here but as I'm given to understand there is a lot of censorship over there so not sure sure what they are trying to achieve except to compile a hit list ![]() |
||
![]() |
![]() ![]() |
![]() |
#25 | ||
FF.Com.Au Hardcore
Join Date: Jul 2005
Location: Melbourne
Posts: 6,890
|
Have a look at these two references in case it happens again. The first link seems to recommend some plugins and tools which might help. The second link refers to Robot.txt, pretty standard, guessing you've already done it, relies on the crawlers to obey the rules, but obviously not mandatory.
https://codewithhugo.com/importance-crawler-bot-block/ https://help.dreamhost.com/hc/en-us/...s-and-crawlers Interesting, Huawei has been diversifying from their mobile and network equipment business, looks like they have now started their own search engine which would explain the crawlers. |
||
![]() |
![]() ![]() |
![]() |
#26 | |||
FF.Com.Au Hardcore
Join Date: Jul 2005
Location: Melbourne
Posts: 6,890
|
Sorry, first link got screwed.
https://codewithhugo.com/importance-crawler-bot-block/ Here are some helpful tips to prevent malicious bots from attacking your website: Quote:
|
|||
![]() |
![]() ![]() |
![]() |
#27 | ||
Chairman & Administrator
![]() ![]() Join Date: Dec 2004
Location: 1975
Posts: 107,001
|
Yes, we already had robots.txt in use but they ignore it so they have been blocked at the firewall level to keep them out. I've also set up a honey pot and fail2ban to keep hyperactive ones at bay in the future.
__________________
Observatio Facta Rotae
|
||
![]() |
![]() ![]() |