This blog covers how you can reduce your cloud administration attack surface using a Wireguard VPN in Google Cloud. In a following blog post we’ll build it with Terraform.
Introduction
A popular use of Virtual Private Networks (VPNs) is to provide additional privacy on the traffic leaving and entering a computer. That and watching “video content” restricted to geographical regions. This article is about neither of these.
One of the things we do at Hexiosec is build cloud infrastructure. Creating cloud infrastructure on the Internet means you need to manage it also from the Internet. And this means private, privileged interfaces to your infrastructure are exposed to everyone else on the Internet. As the UK NCSC recommends, avoiding exposing management interfaces reduces your attack surface. One way this can be achieved is by only allowing connections to the management interfaces from a pre-determined list of IP addresses (enforced by the firewall/security rules). But this leaves us with the problem: ‘what if we don’t have a fixed IP to work from?’. This is what that this blog is about. We will build a locked down relay box with a fixed IP address that we securely route our traffic via where we’re working.
Concept
The following diagram outlines what we’re going to build to reduce the exposure of our cloud infrastructure management interfaces to the Internet.
exploits --
Office <-------------------------------------- |
(fixed IP) ________________ | v ___________________________
| | | x |
Home <--VPN--> | Cloud [fixed IP] <-----------> [IP allow list] Cloud Infrastructure
Office <--VPN--> | VM | x |______________________
---------------- ^
|
scanning --
|__________________________________|
|
This blog
The allow list on the firewall for our cloud infrastructure includes addresses of the places where we have a stable fixed IP address (e.g. our offices in our case). There are many places (such as domestic broadband) where we don’t have control over when the IP address is cycled. So, to work around this, we route our traffic via a relay computer where we have a fixed stable IP address back onto the Internet.
Security Benefit
It might sound like all we’ve done is move the Internet-exposed attack surface to another cloud machine, and we absolutely have - to something with a much smaller attack surface - the only open port is for a Wireguard VPN endpoint which, when port scanned, doesn’t respond without valid encryption on the incoming packet (this makes scanning for attack vectors more difficult). In addition being compromised does not necessarily lead to access to infrastructure of actual value as all existing, good practice authentication systems should all remain in place on management interfaces in line with a zero-trust architecture.
This isn’t far off from the concept of a bastion host or an identity-aware proxy, however, our relay isn’t a gateway or authentication portal to actual resources, it is simply reducing the aperture to a variety of authentication interfaces you might have. It can be considered a layer of your defence in depth.
But now you might be thinking that adding things to your infrastructure to provide more defence in depth is all well and good, but now we have an additional piece of infrastructure to manage. This is true and I am definitely of the opinion that adding complexity can be the enemy of security; if, in the pursuit of security, the complexity introduced leads someone to misconfigure, mismanage or worse bypass something for convenience then you’ve basically defeated what you set out to achieve. Our approach to minimise the burden of this new infrastructure is to use Infrastructure as Code (IaC) to manage it, and use trivial text files for VPN configuration and keys resulting in the following benefits:
- Ease of deployment - running IaC scripts instead of clicking individual settings is less error prone, reliably repeatable, and usually much less time consuming. Should the infrastructure need to be setup again (change, disaster, migration, etc) having the whole setup in code enables you to easily re-deploy.
- Simple maintenance - we’ll abstract settings we want to be able to change over time into parameters, so we only need to tweak these and re-run the code.
- Audit - enshrining your setup in code makes it easier to review and audit (perhaps even allow you to use static code analysis). In addition you can regularly compare the code to what is deployed to confirm nothing has been changed from how you configured it.
- Sharing/handover - The use of code and simple text based configuration files should make it very accessible for someone new to figure out what is going on and work on it. Comments in the code and Git commit messages help to document the infrastructure build.
Technology Choices
To build this we’ve selected the following tools:
- Google Cloud - Google Cloud has cost effective VMs, and the standard tier networking deal offers 200Gb of traffic a month for free using hot-potatoes. They also have a well documented and reliable API (in our view).
- Terraform - We’ll use this to build and maintain the relay, enshrining our configuration in code. Since I’ve personally got my head around Terraform I now actually actively avoid setting up infrastructure manually. Having every detail defined in a text based format provides a very accessible way to understand exactly how something is configured (for when I forget a week later), easy reproducibility should I want to do it again (for when I forget a week later), and documented maintenance by way of tracking of changes in Git (another tool I apply to absolutely all of my work) with the history showing every change, why and when (for when I - you get the picture). In this blog we won’t cover the Terraforming of this infrastructure in order to keep it focussed. Instead there will be another in the near future.
- Wireguard - An open source, well implemented VPN service, that doesn’t respond to any unauthorised traffic (this eliminates opportunistic scanning). It is also “unburdened” by key management - see the next point.
- Text files in a bucket - Sometimes the simplest solutions can be the best. We manage Wireguard keys in plain text files as it’s simple and obvious (and use our corporate password manager to distribute the keys). It should be noted that this probably doesn’t scale too well, and might require revisiting at some point.
Build
Here is what we want to build:
o __________________
-|- Me <-----wireguard----> | Google Cloud VM | <------> Internet
/ \ | with fixed IP |
------------------
^
|
A |
o ___________________________________
-|- | Google Cloud Bucket |
/ \ Admin <----gcloud CLI---->| gs://my-wg-config-bucket/wg0.conf |
-----------------------------------
We will:
- Create the infrastructure (VM, networking, storage bucket) using Terraform. This gives us a relay machine with an IP address on the Internet. The actual Terraform of this is going to be covered in another blog post. However, the inventory and configuration choices are going to be covered in this blog.
- Create some key pairs and Wireguard configurations:
wg0.conffor the server and variouswg0.conffiles for users. The server configuration is uploaded to the bucket. The VM periodically pulls the config from the bucket and configures its Wireguard instance with it (configured within the Terraform build).
A client can then connect to the cloud VM; client traffic goes through the VM and back out to the Internet with the IP address of the VM:
- the user
wg0.confcan be configured to tunnel all traffic (0.0.0.0/0) through the VM, - OR, configured to only tunnel specific IP ranges - in our case the IPs of the infrastructure management interfaces. IP ranges for things like Azure and Google are available in their respective documentation.
Google Cloud Inventory
The elements of Google Cloud we’re going to need are:
- A Google Cloud Project - we recommend creating a dedicated Project (in the Google Cloud sense) as this provides isolation from any other work you have in Google Cloud,
- Storage Bucket - store the Wireguard server configuration file,
- Compute Instance - this is the virtual machine that will be the relay,
- Compute Network - required to provide the Compute Instance network connectivity,
- Compute Firewall - only containing a rule to allow Wireguard traffic in,
- Compute Subnetwork - required as part of a network and will also have a geographic region associated with it,
- Compute Address - a reserved IP address; this resource represents the stable IP address we want,
- Identity and Access Management objects - we want two different roles:
- VPN maintainer - a user that can update keys/config and manage VMs,
- Wireguard Config Reader - read access to the Wireguard config bucket for the Compute Instances.
Regarding the Google Cloud networking components, bear in mind that Compute Instances, Compute Subnetworks and Compute Addresses are geographically Zoned, and they all have to match if you want to attach them together. Your traffic will appear to originate from the geographical area that the Zone is associated with.
Infrastructure Configuration
In this section we’ll cover the configuration choices we made for the Google Cloud resources.
Identity Access Management (IAM)
The only user role we need to create is for VPN maintenance. That is updating and troubleshooting the VMs and buckets. Rather than assign permissions directly to users, we prefer to create groups mapped to a business responsibility, in this case “VPN Maintainer”. This makes it easier to give, audit and revoke multiple permissions. The easier we make it to manage permissions (especially highly privileged ones), the less potential there is for mistakes and compromise. Here is a list of Google Cloud permissions we’ll provide a “VPN Maintainer”:
compute.instances.reset
compute.instances.resume
compute.instances.start
compute.instances.stop
compute.instances.suspend
compute.instances.use
compute.instances.setMetadata # Required to allow provisioning of SSH keys
iap.tunnelInstances.accessViaIAP # SSH via IAP into the VM
roles/viewer
roles/storage.objectUser
roles/iam.serviceAccountUser
We will bundle the first 8 permissions into a custom “role” so we can then then construct a Google IAM Policy which can then be assigned to identities. It’s good practice to manage permissions in a fine-grained way rather than using a primitive role such as Google’s “roles/Owner” or “roles/Editor”, which almost always assign excess permissions.
Wireguard Bucket
We need storage for the Wireguard config, so we use a Google Cloud Storage Bucket. We will configure the following options:
- “Public Access Prevention” - there is no need to have this bucket public; inadvertent public buckets has been a common Cloud Infrastructure misconfiguration.
- “Uniform Bucket Level Access” - in Google Cloud buckets can be configured to use their own access control mechanism or use the Google Project roles and permissions. We generally prefer the latter - this allows us to have a single permissions space to manage ultimately giving us less things to manage for maintenance and audit.
We’ll also need a way for the VMs to reach the data in this bucket so we’ll use a Service Account with read only permissions on the Wireguard configuration bucket. The Google Cloud permission we want to grant this Service Account is roles/storage.objectViewer only.
Networking
Networking is part of Google Cloud’s VPC (Virtual Private Cloud) product. In our inventory we listed multiple related resources:
- Virtual Private Cloud Network
- Disable ‘auto create subnetworks’ - we only want a single subnetwork that we’ll configure ourselves to minimise extraneous resources.
- VPC Network firewall rules
- Add a rule to allow our Wireguard connections on UDP port 51820.
- Add a rule to allow us to SSH into the VMs for trouble shooting - but we’ll restrict this to using SSH over Google IAP by setting ‘source ranges’ to “35.235.240.0/20” as given in the documentation. This way identity and authentication is managed via our existing Google Cloud accounts rather than a whole new set of identities or keys.
- VPC Subnetwork
- Above we disabled “auto-create subnetworks” so that we can add our own, configured for our purposes; set the “IP CIDR range” (e.g. 10.0.1.0/24) and set a region. Make note of the region you select because the External IP and Compute Instance resources need to match.
- External IP address
- This resource gives us a dedicated IP address to specify in our cloud infrastructure firewall allow list. If we don’t use this resource then the Compute Instance will be given one automatically. This automatically assigned IP address will be released if we shutdown the VM for any reason (e.g. updates). Creating an External IP allows us to retain an IP even if the VM isn’t active, and reassign the IP when the VM comes up.
- Configure the ’network tier’ to STANDARD - what you choose here is a performance-cost trade-off. The default is ‘PREMIUM’ which is routed via Google infrastructure around the world at a cost, however, ‘STANDARD’ downgrades it to be routed over normal Internet and gives 250Gb free.
If we want to create multiple VPNs, we can create a dedicated network for each one as this makes the configuration a bit simpler.
Compute
Finally we create the actual Compute Instance(s) that will run Wireguard, which we will use as an exit node to the Internet.
- Choose ’e2-micro’ - we chose this because it’s the cheapest and seems to be sufficient for at least a few users. Of course upgrade this if you need and your purse allows. This is really the only cost involved in this infrastructure.
- Ensure Zone matches that used in the network configuration.
- For boot image we chose ‘debian-12’ as a stable distribution actively patched, that has what we need.
- In the network interface we need to set ’network tier’ to whatever the External IP resource was configured to.
- For the network interface (“nat_ip” in Terraform) we choose our External IP Address resource.
- Attach our Service Account for accessing the Wireguard Configuration Bucket.
For our instances to be able to forward traffic and run Wireguard we use the ‘metadata startup script’ to setup everything in the host on creation. The following can be appended into one long string for this field:
sudo apt-get update; sudo apt-get install -yq sl vim wireguard-tools;cron_wg0 = "echo '* * * * * root gcloud storage cp gs://your-wireguard-config-bucket-name/wg0.conf /etc/wireguard/; wg-quick strip /etc/wireguard/wg0.conf > /tmp/wg0.conf; wg-quick up wg0 || wg syncconf wg0 /tmp/wg0.conf' >> /etc/crontab";- this is a regularly scheduled job that retrieves the Wireguard configuration from the bucket that houses the Wireguard configuration.echo net.ipv4.ip_forward=1 >> /etc/sysctl.conf; sysctl -p; iptables -A FORWARD -j ACCEPT; iptables -t nat -s 10.0.0.0/24 -A POSTROUTING -j MASQUERADE- this is Linux networking configuration to enable re-direction of traffic from one network interface to another; the relay functionality.
Connecting with Wireguard
We need a Wireguard server and client configuration - these are text files. Using wg-quick this is quite straightforward, and man wg-quick has example configs you can use as a template:
# Server wg0.conf
[Interface]
Address = 10.0.0.1/24 # Our internal Wireguard network
PrivateKey = < #### Server Private Key #### > # Server private key - see below
ListenPort = 51820
MTU = 1400
# List all clients here:
# - specify public key
# - give them unique IPs on our wireguard network
[Peer]
PublicKey = < #### Client 1 Public Key #### >
AllowedIPs = 10.0.0.2
[Peer]
PublicKey = < #### Client 2 Public Key #### >
AllowedIPs = 10.0.0.3
Client config for the first peer shown above:
# Client wg0.conf
[Interface]
Address = 10.0.0.2
PrivateKey = < #### Client 1 Private Key #### > # Client private key - see below
DNS = 1.1.1.1 # Optional but sometimes useful
# Server details here:
[Peer]
PublicKey = < #### Server Public Key #### > # Server public key
AllowedIPs = 0.0.0.0/0 # This will shuttle ALL traffic through.
# If you want selective traffic then you
# can list them here instead.
Endpoint = 35.211.12.123:51820 # IP:Port of the VM you want to use.
Key pairs for the server and for the clients can be generated with $ wg genkey | tee private.key | wg pubkey > public.key (see man wg). Copy and paste these into the appropriate fields above.
Then we upload the server wg0.conf to our Wireguard configuration bucket. Use gcloud storage buckets list to get the name of the bucket, then:
$ gcloud storage cp wg0.conf gs://< ### Bucket name here ### >/
Within a minute the VM’s should all query and implement this configuration. Then we can connect to it with a Wireguard client. Example:
$ wg-quick up wg.conf
$ curl ifconfig.io
35.211.12.123
Maintenance
Ongoing maintenance should be quite light. To add a user you:
- generate some keys for the new user,
- update the server config with a new
[Peer]by:- downloading it:
gcloud storage cp gs://wgvpn-config-1234567899/wg0.conf ., - editing it,
- uploading it
gcloud storage cp wg0.conf gs://wgvpn-config-123456789/.
- downloading it:
To remove a user you follow the same process but delete a [Peer] entry from the config file.
Debugging
Computers being computers everything should work first time and you should have no problems. In the unlikely situation everything doesn’t work first time you can SSH into the VM to look around. List your Compute Instances thus:
$ gcloud compute instances list
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
hexiosec-vpn-office us-east1-b e2-micro 10.0.1.2 35.211.XXX.XXX RUNNING
hexiosec-vpn-customer-a europe-west2-a e2-micro 10.0.1.2 35.214.XXX.XXX RUNNING
hexiosec-vpn-xyz europe-west2-a e2-micro 10.0.1.4 35.214.XXX.XXX RUNNING
And then
$ gcloud compute ssh <vm-name>
Conclusion
In this blog we’ve talked about how we can use a Wireguard VPN to reduce the attack surface of our privileged cloud administration interfaces on the Internet, providing us a layer in our defense in depth. To help us with managing this (which in turn also helps our security) we are going to use Terraform which comes with many benefits, crucially, including us having to do less work in the long term to maintain and keep it secure. In a following blog post we’ll talk about why we like Infrastructure as Code so much and build out this blog using Terraform.
Related Posts
