8 Lessons Learned from Network Bots

At Securly, we have gained a lot of insight into bot behavior based on our own experience seeing this message. Google’s security team has been tremendously helpful in understanding this issue.

We are sharing our findings below hoping they will be helpful for anyone experiencing similar issues. Note that we are intentionally not sharing any insights that Google wouldn’t want bot writers to get access to.

Here are 8 lessons learned from our experience with network bots:

1) Google search generally doesn’t like “onion routing”

Onion routing can happen even unintentionally on many networks when the school has a network or a web-filter with more than one egress points to the public Internet.

If Google detects that the same user(s) is accessing its service from different source IP addresses, it flags the behavior as bad. This is how Tor or HOLA VPN works as well.

We haven’t dealt with HOLA specifically, but it is possible that some students may have installed HOLA VPN to get around web-filtering, leading to the entire district/school IP(s) getting flagged by Google.

2) Google attempts to flag traffic at the user-level first before it bans the entire IP address

To do this, it naturally has to rely on the IP addresses present in the traffic it sees.

If the school network uses proxy-based web-filters that are incapable for preventing onion-routing and additionally are incapable of adding X-Forwarded-For headers to the Google traffic, the school will trigger Google bot alerts in our experience.

3) Repeated Google queries can get the IP flagged

If onion-routing is not involved, even with fixed IP addresses, we have found that bots that are performing repeated Google queries can get the IP flagged.

4) If you use hosted or cloud-based proxies, ensure that these are not open proxies.

In our experience, restricting traffic to registered school IPs helps a lot.

If your school needs to keep proxy access open to even unregistered source IPs (e.g. to support take-home 1:1 iPads being proxied through on-premise web filtering proxy), then you must ensure that X-forwarded-for headers are added.

Again, this is only possible if your web-filters are capable of MITM HTTPS handling. As explained above, without this, the bots would appear to Google to come from the IP address of the school web-filter causing bot alerts to show up for all users behind that IP.

5) Unsolvable Google captchas can be caused by having multiple public IP addresses

If you are seeing a Google captcha that you are unable to solve, in our experience that is a sign that the Google related traffic is exiting your network from more than one public facing IP address. This is common in medium to large sized districts.

What happens in this case is the captcha gets served because Google sees “onion routing”, and once it is served, even when it is solved, Google doesn’t associate the solved captcha from one IP address to the offending IP for which it served the captcha.

For example, google.com/search may have happened from IP address A, but the captcha itself was served from ipv4.google.com which is accessed via IP address B. In our experience, ensuring Google.com and ipv4.google.com get accessed from the same source IP get rids of the issue where the captcha is unsolvable.

6) Bots are smart!

Bots are extremely smart, and we have found even cases where the bots discovered that we were an open-proxy only over the CONNECT + HTTP methods (we have regular HTTP, regular HTTPS and CONNECT HTTPS covered).

7) A good way to detect bots on your network is to look at night-time traffic

Bots in general do not go to sleep like humans do, and for that reason, you should see Google search traffic from these happening even at 2AM local time.

Firewall logs can quickly point to the source of the traffic on your school network.

8) A single infected machine can impact your whole network

Yes, a single infected machine on your network can bring the access to Google search down for the entire district.

Sponsor Ads

• Ad Space Available - Rent it Now!

Report this Page

Created on Dec 31st 1969 18:00. Viewed 0 times.

Share on APSense

Comments