proxmox cluster proof of concept

proxmox.md

ProxMox Cluster Proof Of Concept (Work In Progress)

Start Date - First week of August 2023

End Date - none

Last Updated 9/1/2023

note: these are designed to be primarily a re-install guide for myself (writing things down helps me memorize the knowledge), as such don't take any of this on blind faith - some areas are well tested and the docs are very robust (some of the extra credit items, less so). YMMV i wiill regularly update until this become production

Outcomes

1. Hardware and Base Proxmox Install

2. Thunderbolt Mesh Networking Setup

3. Enable IPv4 based OSPF Routing

4. Setup Cluster

5. Setup Ceph and High Availability

6. Create CephFS and storage for ISOs and CT Templates

7. Setup HA Windows Server VM + TPM

8. How to migrate Gen2 Windows VM from Hyper-V to Proxmox

   1. Notes on migrating my real world domain controller 2 VM
   2. Notes on migrating my real world domain controller 1 VM
   3. Notes on migrating a windows admin center VM

9. Create and Migrate a Debian VMs from Hyper-V

   1. Notes on migrating my debian VM based docker swarm

10. Migrating HomeAssistanct VM from Hyper-V

11. Extra Credit (optional):

    1. Enable vGPU Passthrough - windows
    2. Install Lets Encrypt Cert (CloudFlare as DNS Provder
    3. Azure Active Directory Auth
    4. Install Proxmox Backup Server (PBS) on synology with CIFS backend
    5. Send email alerts via O365 using Postfix HA Container

12. Notes & Troubleshootig

TODO

• move my running productin hyper-v vms to proxmox now i know how
• add TLS to the mail relay? with LE certs? maybe? -- revist IPv6 when IPv6 thunderbolt issue is resolved in debian
• maybe send syslog to my syslog server (securely)
• figure out ceph pubic/cluster running on different networks - unclear its needed for this size of install
• figure out igpu access for containers https://bookstack.swigg.net/books/linux/page/lxc-gpu-access (this might be more useful to me than VM igp sriov access)

Purpose of POC

I have been using Hyper-V for my docker swarm cluster VM hosts (see other gists). Original intenttion was to try and get Thunderbolt Networking for a Hyper-V cluster going and clustered storage for the VMs. This turns out to be super hard when using NUCs as cluster nodes due to too few disks. I looked at solar winds as alternative but this was both complex and not pervasive.

I had been watching proxmox for years and thought now was a good time to jump in and see what it is all about. (i had never booted or looked at proxmox UI before doing this - so this documentation is soup to nuts and intended for me to repro if needed)

Goals of POC

1. VMs running on clustered storage {completed}
2. Use of ThunderBolt for ~26Gbe Cluster VM operations (replication, failover etc)
   • Thunderbolt meshs with OSPF routing {completed}
   • Ceph over thunderbolt mesh {completed}
   • VM running with live migration {completed}
   • VM running with HA failove of node failure {completed}
   • Seperate VM/CT Migration network over thunderbolt mesh {not started}
3. Use low powered off the shelf Intel NUCs {completed}
4. Migrate VMs from Hyper-V:
   • Windows Server Domain Controler / DNS / DHCP / CA / AAD SYNC VMs {not started}
   • Debian Dcoker Host (for my 3 running 3 node swarm) VMs {not started}
   • HomeAssistant VM {not started}
5. Sized to last me 5+ years (lol, yeah, right)

Hardware Selected

1. 3x 13th Gen Intel NUCs (NUC13ANHi7):
   • Core i7-1360P Processor(12 Cores, 5.0 GHz, 16 Threads)
   • Intel Iris Xe Graphics
   • 64 GB DDR4 3200 CL22 RAM
   • Samsung 870 EVO SSD 1TB Boot Drive
   • Samsung 980 Pro NVME 2 TB Data Drive
   • 1x Onboard 2.5Gbe LAN Port
   • 2x Onboard Thunderbolt4 Ports
   • 1 x 2.5Gbe usinng Intel NUCIOALUWS nvme epxansion port
2. 3 x OWC TB4 Cables

Key Software Components Used

1. Proxmox v8.x
2. Ceph (included with Proxmox)
3. LLDP (included with Proxmox)
4. Free Range Routing - FRR OSPF - (included with Proxmox)
5. nano ;-)

Key Resources Leveraged

Proxmox/Ceph Guide from packet pushers

Proxmox Forum - several community members were invaluable in providing me a breadcrumb trail.

systemd.link manual pages

udevadm manual

udev manual