Having a hard time recently with a particular MSP / Security vendor in south-east Asia who keeps auto-generating abuse reports to our upstreams claiming we “illegally” “probed” them on 123/udp.
We serve the very constrained Asia and African regions and don’t really want to stop serving the pool. Some countries we are all alone with cloudflare. Centralization is a bad thing.
These guys with their automated abuse reports are causing tremendous grief.
Reached out to them once and they whitelisted some of our NTP servers so no reports are sent but they are adamant their system is working as intended and after a while they start submitting again.
The open localhost is for a monitoring script on localhost to be able to return data. For the WAN facing side it seems sane:
ntpq -c rv will time out.
ntpdate -dqv will return the time. I think that’s about right to serve the pool but not have a huge DDOS amplification factor ?
In any case these guys seem to operate some public WiFi and business networks and when client devices reach out to the pool and we answer the request their IDS seems to not put 2 and 2 together and claims we are hitting them on 123/udp.
Many TB served each month and only this one entity keeps sending abuse reports. This also lead to our entire domain name being sent to some DNS blacklist that people with AdGuard and pihole like to use.
Greetings Fellow Sys Ad/s Kindly take a look at one or more of your IP addresses that were seen attacking/probing our network. It may be compromised and it could be attacking other networks as well. Our Timezone is GMT +8. Please see the logs below: DateTime Action AttackClass SourceIP Srcport Protocol DestinationIP DestPort 0 22-Nov-2022 04:53:14 DENIED 205.189.160.58 123 UDP 202.91.175.150 123.0 1 23-Nov-2022 00:14:57 DENIED 205.189.160.58 123 UDP 202.91.175.150 123.0 2 23-Nov-2022 02:09:31 DENIED 205.189.160.58 123 UDP 202.91.175.150 123.0 3 23-Nov-2022 03:26:34 DENIED 205.189.160.58 123 UDP 202.91.175.150 123.0 4 23-Nov-2022 04:06:59 DENIED 205.189.160.58 123 UDP 202.91.175.150 123.0 5 23-Nov-2022 04:45:32 DENIED 205.189.160.58 123 UDP 202.91.175.150 123.0 6 23-Nov-2022 06:20:08 DENIED
I know these guys. They sent a similar notice to my ISP, from which I have a virtual server in Singapore (and elsewhere). My ISP relayed the complaint to me, and I responded with:
"A kind reminder that UDP/123 is NTP, Network Time Protocol.
If you go to http://94.237.64.20/ you will notice that the sole service provided by this server is NTP. This server (and a few others I have) are in the NTP pool, www.ntppool.org
This and a few thousand other servers provide accurate time for free for millions of Linux/*IX servers, wireless routers, IoT products, many Android phones and other networked appliances.
As for this complaint in particular, it is therefore very likely that some device within the complainer’s organization has requested time using NTP from pool.ntp.org, and my server has responded to that request. This particular server I’m operating responds to around 35000 time requests per second, ie. around 3000 million requests per day. GeoIP is involved, so most of the requests to this server come from Asia/Oceania.
In short, this was no attack but a legitimate response to a request originating from the complainer’s network. If the complainer does not want to receive responses to NTP queries, perhaps the complainer should block outgoing requests to UDP/123 from their network. This is generally speaking not a good idea, though. A better solution would be to adjust their firewall to not warn about incoming UDP/123 packets if they are related to a request coming from their own network."
This was in August. I don’t know if they sent more complaints to my ISP, but at least my ISP hasn’t contacted me again after this.
I was thinking of doing that. Their ASN has a small number of ranges I could just create a centralized list and have the instances pull it an deny access but its against the spirit of the pool.
There is a general uptick in folks taking an interest in (yay!) but then misinterpreting their firewall logs and creating random abuse reports (meh). I had this beauty the other day:
hey im student1adams and im here to report a ip address coming up on
my logs as a malware site coming from your guys datacenter and i would
like to see if there is a way to get this ip tooken down and or delt
with because i have been having problems with this ip and it keeps
trying to get me to a malware site as its said on my logs \"Device
attempted to connect to ip address 205.189.160.58" and
you would think it would block it and it has but it doesnt stop there
because this ip kept coming back meaning someone on your datacenter
using your services has tried to connect to my ip severel times and
well i was gonna try to contact you guys through the phone which is
best for me but i dont care email is best as well but all i want is
this ip tooken down and well if it doesnt get tooken down then i guess
you guys wouldnt care if i was to say hack this ip with my own brute
force and unless if you want a ip to be breached for doing illegal
shit to my router i recommend you guys take this ip down because i am
getting sick and tired of this ip always coming back and doing this to
my ip so pls and thank you and if i see no work down i guess ill do my
own myself so have a good day and pls let me know when the work has
been done because this ip is getting on my nerves and if you want
proof i have alll of the proof with more then 20 images of this ip
coming back and coming back and coming back so have a good day
When they were told to RFTM and pound sand and “nothing was done to stop this egregious attack” they followed up:
hey so this ip 205.189.160.58 which is a ip coming from your guys
datacenter is doing illegal things to my router needs to be tooken
down and i contacted you guys about this and he has still not been
tooken down so there is the proof that he is still attacking me today
but tried to remote connect to my phone so if you can pls take this ip
down pls and thank you
In this instance the upstream is chill and understands how to filter noise and did not worry about it. But in other markets (Philippines, Singapore) upstream is soul-less automation and will eventually drop us if any stuff comes in. In these markets bandwidth is a seller’s market and you can’t just go stand up a server that serves 10TB of NTP a month pro bono.
For IPv6 we have plenty of in-house space that fixes all of this nonsense but most of the requests are IPv4 and we use other people’s address space so this nonsense goes to support@upstream and it’s frustrating.
BNS hosting.
They seem to be PH/SG based MSP and manage some public WiFI in the region. They have hurt our Singapore, Korea, Japan, Los Angeles, Manila, Cebu and Bangkok NTP pool nodes.
I tried to get them to check what their IDS is doing because I am certain we aren’t dialing out to them on 123/udp from our end. Their devices are making requests and their IDS seems to only see one side of it. They sell their elite IDS capabilities as a service
Their response was arrogant and dismissive. They are adament we are unilaterally connecting to them on 123/udp which is utter nonsense. We connect to in-house Stratum 1 GPS/PPS and some government / university stratum 1 nearby, then serve the pool.
The other day OVH asked me to explain “abusive use originating from my VPS”, without any details provided. It’s just an NTP server, with rate limiting enabled, so not even a 1:1 reflection (unless they are switching between a very large number of addresses).
I’m wondering if it’s related to the broken timesyncd clients flooding public servers for hours or days. Many of them seem to be in PH, more than any other country, at least from what I see on my servers. I tried mailing the corresponding abuse addresses, but never got a response.
Simply send abuse complaints to them for “probing port 123/UDP” on your server. Either they will start blocking traffic to you (so no future abuse reports from both parties) or they start to explain why this is not abuse. And their explaination will become your answer to their future abuse reports.
I’ve gotten two of these false abuse reports from BNS hosting so far. I’ve went ahead and blocked their subnet (202.91.160.0/20) from my NTP servers because I don’t want to risk my hosting provider shutting down my NTP servers because BNS can’t figure out how to properly run their automated firewall log reporting system.
I have been responding to them, and they ignored them, till I added the whois email to it, every time I email with him added (Wilson@dagupan.com) I get a response saying I will be whitelisted, I keep getting the notices every few days.
Last time I told them I had enough of it, and I got another one today, and responded again to them, and I blocked their asn from my routers.
It’s not a question of who runs them, it’s the people that do the lack of security review of the incident reports.
The systems IDS systems all flag any ntp incoming as an issue, cause it isn’t smart enough to know what ones are attacks and what ones are not. Really this issue was resolved a decade ago but well.
So people that do not do security correctly, and just depend on what the automated system says, without customizing it and filtering out false positives and just turning the detection level way up, you get this issue.
It’s just insane they would send these reports out automatically. I am in no fear of being terminated, as I own my ip space.
giant.rocks - AbuseIPDB User Profile seems to be a totally different issue.
It appears they are a pool server, and they are also running a website on the same ip.
Any hit to their website using the pool ntp org is causing them to be flagged as an attack and when it lookup pool ntp org to report the attack, it’s using whatever the dns servers return for that entry at the time, causing this pool user to be reporting all other pool users.
Yep, definitely the same entity. He told me that someone got in contact with him the other day as well to report the problem, and that he actually shut the system down on the 28th to investigate.
Funny how things work out like that, but glad the AbuseIPDB part is being looked into now since this has been happening for like 7 months.
I created an account so I could reply to this thread.
giant.rocks was me.
I have used running Asterisk as an excuse to learn Linux, and although, I have done it now on and off for a number of years, I would hardly call myself any kind of expert. Which means, I have absolutely made a number of stupid mistakes along the way. Including this one.
I am still not reporting again. Until I get a better understanding of fail2ban I won’t be reporting again.
I really don’t want to cause any trouble. I am glad that two people contacted me. I really should have been paying more attention. That’s all on me.
