(Packet.net is now Equinix Metal (https://metal.equinix.com/)
Earlier this year we had a workshop with a customer, they where a "very on-prem/in-house setup", They where thinking "Hybrid-Cloud" but wanted to see what Terrafrom was capable of and it's limitations well lets see ;). However no reason why it's not limited to the big 3 public cloud providers (AWS, Azure and GCP). On the 1st February 2020 I had to stay up late to watch the Six Nations live Ireland vs Scotland, so spent the time hacking together some bare-metal provisioning on Packet.net, now an Equinix company ;)
My home setup has the HP ZR30z Pre-4K monitor, an old beast but ((when you got nearly 30"'s of screen-real estate, then a ruby match is a nice addition to the space))
Here's what it looked like, Doc's, Code and Live ruby Stream
Without a huge amount of effort also, and all in time to make the first-half.
It's quite exciting to see such a stack get bootstrapped without very little layers in-between, (e.g, VPS, nested virtualisation,) no this is bare-metal. This brought back memories of way back in the Blade Server days, bootstrapping ESXi on a bunch of Dell Blades in the Dell m1000e blade chassis using internal usb flash disks. One of the existing tools that makes this really work well is Tinkerbell, which Packet recently open-sourced (link)
After some Terraform and 29 mins into the match. We had 2x ESXi 6.5.0 hosts up, vCenter VM provisioned with a nice bit of resources to play with! As you can see the box is a Dell PowerEdge R6415 with AMD EPYC CPU and 64GB's RAM. It's also important to note that #IPv6 was provisioned too without any special configuration I had to manage. The network layer really get provisioned out of the box, also upstream Packet.net build up an LACP on the pair of 10Gbps
Once the Terrafrom apply had completed, just before the 15min mark, vCenter vm was still bootstrapping so the interface wasn't ready yet to connect,
As you can see the hardware is supported by VMware, it detected well the hardware stack below. We can see it's standard server and storage hardware but with the added benefit of access to a well setup enterprise automation.
One major difference running Terraform on bare-metal cloud is the provision of Layer2 services, in this case, VLAN's and LACP on the ports. You never see this low-level in the larger cloud providers. Things like Multicast, VXLAN etc.. is a challange to relisse, where certain applications like TV streaming, (IPTV multicast) streams can't be pushed
Once the new virtual network interfaces where in place, the provisioning of the system could take place, as in Packet you can control the attachments of various VLAN,s LACP groups it's quite flexible to get the rest of the network up an running purely with Terraform (and some bash scripts)
To make this happen in such a fluid way, lets list out the components:
- PXE boot images (for all Linux distros, ESX, ESXi, Zen etc..)
- DHCP Boott, and TFTP server (yes that
- Out-of-band management access to iLO / iDRAC
- some magic middle ware service making all the adjustments to automate the setup, in this case Tinkerbell
- Packets.net API wrapped with the Terraform Packet Provider
Do you want to achieve this yourself? would you like to know how bare-metal provisioning using Terraform, get a "Hybrid-Cloud" setup, Imagine provisioning this across Packet+AWS Cloud that would be cool!. Happy to provide training and understand better what Terraform/Terrragrunt /w Ansible can enabled and together make good candidates for your next automation projects.
Our Services - and more
If you'd like to know more 56K.Cloud and of course the benefits of Cloud Adoption, Containers and DevOps Automation, feel free to reach out to info@56K.Cloud to learn more about our services, training, and solutions.