Looks like the majority of the monitor nodes have problems querying your server. With a score graph like that, it should be fairly simple to reproduce the problem yourself by setting up some monitoring on your own from some other server, with a script like this one.
If you need a second (or third, or fourth…) opinion, here are some stats:
Singapore, Netherlands, Australia, Finland
You can do traceroutes to the respective server names.
As usual, before bothering your ISP too much, make sure your own server is fine. For example, see if there are odd messages in dmesg about conntrack tables filling up.