#1: Send 5.5 million webhooks to Splunk per month
Things One, Two and Three log all events real-time into Splunk. Webhooks with a JSON payload.
This is nothing new per say but the OpenCanary instances are around the Internet and I am happy to not have to open ports up for Webhooks as it’s clear someone will find that open port and try to guess the secret.
As a long-time user of Tailscale, I came upon the ideal of sending those Webhooks to my Splunk box using my tailnet. The OpenCanary from Thinkst supports Webhooks in its config file; I tried sending these into Slack and Teams but <insert smiley here> they are not built for what comes when the lights are on.
"WebHook": {
"class": "opencanary.logger.WebhookHandler",
"url":"http://tailscale_ip:webhook_port/your_webhook_goes_here",
"method": "POST",
"data": {"message": "%(message)s"},
"status_code": 200
}
What is clear is the Tailscale stack is up to the task. The three honeypots get between 5 and 6 million hits per month – and report them faithfully into Splunk using the Tailscale layer that binds the hosts together. That’s a considerable amount of traffic!
This makes life very easy with a low friction deployment, no ports open to the Internet and an ability to handle volume reliably.
Which brings us onto…..
#2: Run a Tailscale reverse proxy in the Internet
I expose the above Splunk instance to the Internet via a Caddy server which in turn gives me TLS using ACME certificates and it works really well.
The Caddy sits in Google Cloud and serves TLS connections to my NAS systems, Splunk and PiHole/Adguard webadmin interfaces. With SNI support, I believe the Caddy protects itself well from scanning and the hosts behind do have strong authentication in place. Caddy scores well on TLS testing sites like Qualys’ SSLLabs with a lean towards a minimum of TLS1.2 usually supported. If you want to go really crazy, proxy the connection first through Cloudflare and force TLS1.3!!
Which brings us onto…..
#3: Run DNS over Tailscale to a PiHole or Adguard Home server
Lucky Number Three? If you’re reading this and got this far, you are probably security- and tracking-minded and may duplicate what I have done in your own setup.
PiHole and Adguard Home are both DNS servers that can be shoehorned onto small servers (like a Raspberry Pi or in Docker) and then configured to use blocklists to improve your web experience and reduce tracking/ads. There are many guides to setting this up on the Internet which are fairly reliable – the killer aspect is to use Tailscale to then give you DNS results – no matter where you are – from your own DNS server.
Some might also use SmartDNS services in order to change location when consuming some web services; the Exit Node of Tailscale may well be your friend assuming you set your DNS servers to use those SmartDNS services. Have your DNS server in your fixed location be your exit node and you may find yourself happy with the outcome.
Adguard Home supports DoH and other “modern” DNS protocols out of the box; PiHole needs Cloudflared to get DoH working (but this works well).