🐧Linux 🏡 Homelab init
This repo contains the design, details, and planning involved in
creating my home lab suitable for enterprise-level work. I've opted to
publish these details in the hopes is helps others learn and be inspired
to create their own.
⚡ Most people should probably not publish this kind of detail for
hackers to use against them. I have a special case because I've
already chosen to be so public about everything I do on Twitch and
YouTube for the sake of education. Others will probably not want to
take such risks.
Hardware Summary
I have one VM server host and a bunch of hardware left over from
SKILSTAK. I've also added some hardware for
machine learning (eventually). This does not include my gaming/streaming
rigs, laptops, or any other devices in our home.
- 1 x HP Z640 (28 cores, 128 GiB RAM, 512 GiB SSD)
- 20 x 2016 Mac Mini (4 cores, 4 GiB RAM, 512 GiB HD)
- 6 x MSI Trident (4 cores, 8 GiB RAM, 1 TiB HD)
- 6 x Dell Optiplex 7050 (4 cores, 16 GiB RAM, 256 GiB SSD, 3600 IOPS)
- 1 x Mac Pro
- 1 x iMac
- 1 x Raspi 4
- 2 x Raspi 3B
- 2 x nVidia Jetson Nano (2 GiB version)
- 1 x Netgear Nighthawk X6 R8000 (with USB SMB-accessible storage)
- 1 x Netgear 24-Port (12 PoE) GiB Switch
- 1 x TP-Link 24 Port GiB Switch
- 1 x ISP cable modem
(I donated more than a dozen Raspi 2s to friends on stream for projects.)
Eventually, I need to get a beefy, separate (not rack mounted) UPS and
possibly an iSCSI NAS device. The UPS should prevent the need for
dedicated 20Amp circuits.
Planned Use Cases
The purpose of this apartment lab is to facilitate development of
on-prem cloud native infrastructure management software and services
primarily involving Kubernetes and major Kubernetes applications and
running pentesting scans intrusion detection experiments (honeypots).
- Install and maintain a primary Kubernetes cluster on-prem
- Experiment with different separate K8S clusters
- Explore Microk8s and k3s and Talos on Raspberry Pi
- Experiment with communication between clusters (mTLS+gRPC, REST)
- Practice SMB, NFS, and Cephfs K8S storage class providers
- Practice identity management with Keycloak, Kerberos, OpenLDAP
- Practice configuration management with enterprise tools (Ansible)
- Develop infrastructure management software and APIs
- Develop distributed rootkits, botnets, malware targeting cloud
- Automate bug bounty discovery and stealth target scanning
- Practice intrusion detection and develop software to assist
Infrastructure Stack
- Ubuntu Server OS
- KVM/libvirt/Firecracker
- Sidero/Talos OS
- CoreDNS Server (HA, external)
- Ansible
- Icinga2
- etcd (HA, external)
- CRI-O
- Kubernetes
- Calico
- Goldilocks
- MetalLB
- Node Feature Discovery (NFD)
- nVidia GPU Feature Discovery
- NFS Storage Class
- SMB Storage Class
- OpenEBS?
- CephFS Storage Class
- Keycloak
- OpenLDAP/Kerberos
- OPA Gatekeeper
- Harbor/Trivy
- Istio
- Tekton
- ArgoCD
- Elastic Search
- LogStash (or Fluentd?)
- Kibana
- Seldon
- Prometheus
- Thanos
- Graphana
- Github Enterprise ($245/year/person)
Other stuff I still have to learn:
Design Decisions
Ubuntu Server. Red Hat might be big in enterprise, but unnecessary
in my home lab. Ubuntu Server skills are far more important for
containers and developers to master.
No FreeIPA. Primarily a Red Hat technology, can be used on Ubuntu
but swimming upstream to do so. Also, Keycloak is mandatory for things
that will mirror what I do at work.
No Proxmox. Only supports 32 nodes per cluster and falls on it's face
with just 14 (which I tried). Using KVM directly and Virtual Machine
Manager instead, which saves on a lot of Proxmox framework overhead
waste as well.
No VMs for main Kubernetes. One distinct advantage of on-prem
Kubernetes is that the 30% performance penalty from virtualisation
(which all cloud providers have) can be avoided.
etcd "stacked" within Kubernetes.
No disk encryption. I just don't need it so the overhead would be
wasted. Besides, if I did need it, I could just encrypt a loopback
mounted storage block file or just use GPG.
Stages of Personal Home Lab Evolution
Here are the stages of my own personal home lab over a few decades.
I'm at stage #7 now.
- Personal laptop and ssh access to cloud servers
- Built my own on-prem (garage) Linux server machine from scratch
- Added old discarded machines from friends and family with Linux
- Used VMware Player/Workstation and 1-3 headless VMs on “desktop”
- Acquired several Mac Minis and Raspi while at SKILSTAK
- Added dedicated Type 1 Hypervisor server, K8S, plug and play
- Add Enterprise rack and multiple components
- Configured multiple secure virtual and physical networks
Rack Organization
I'm separating the hardware into two racks to make it easier to move
around and, when necessary, locate close to separate circuits (although
I'm shameless enough to run a power extension chord from a good 20A
bathroom circuit breaker if I need).
The 21U rack height allows me to put a monitor/keyboard/mouse on top and
have it be the perfect height to use while standing. Since only the HP
server is fixed (and on rails), I can pull anything out on the rare
occasion when I need to plug them into a monitor, etc.
Rack 1 (21U)
- ON: Dell Monitor, keyboard, mouse
- 1U: 15A (1800W) PDU 10+2 Outlet (w/display)
- 1U: TP-Link 24-Port Switch with VLAN (14W to PDU)
- 1U: 24-Port Cat6 Keystone Patch Panel
- 2U: MSI Trident (shelf) (230W to PDU)
- 2U: MSI Trident (shelf) (230W to PDU)
- 2U: MSI Trident (shelf) (230W to PDU)
- 2U: MSI Trident (shelf) (230W to PDU)
- 2U: MSI Trident (shelf) (230W to PDU)
- 2U: MSI Trident (shelf) (230W to PDU)
- 2U: UPS (w/display) - eventually
- 4U: HP Z640 Mid-Tower Workstation (w/rails) (975W to PDU)
Max Power Consumption:
Devices |
Watts |
Switch |
14W |
HP Server |
975W |
MSI Tridents |
1380W |
Dell Monitor |
33W |
TOTAL |
2402W |
Rack 2 (21U)
- ON: Netgear Wifi Router (5W to Mixed Power)
- 1U: 15A (1800W) PDU 12+2 Outlet (w/display)
- 1U: 15A (1800W) PDU 12+2 Outlet (w/display)
- 1U: 24-Port Cat6 Keystone Patch Panel
- 1U: NETGEAR 24-Port (12 PoE, 100W to PDU1)
- 3U: ISP Modem (5W to Mixed Power)
- xx: Mixed Power strip (to PDU1)
- xx: Protectli Vault (OPNsense) (16W)
- 2U: 2 x Jeston Nanos, Pi4, 2 x Pi3 (self,5x5W to Mixed Power)
- 5U: 10 Mac Minis (shelf) (10x85W to PDU1)
- 5U: 10 Mac Minis (shelf) (10x85W to PDU2)
- xx: Mac Pro (shelf behind minis) (902W to PDU2)
- 2U: 2700W(3000VA) CyberPower OL3000RTXL2U UPS (w/display) - eventually
Max Power Consumption:
Devices |
Watts |
Switch |
100W |
Mixed Power |
51W |
Minis |
1700W |
Mac Pro |
902W |
TOTAL |
2737W |
Related: