31 KiB
31 KiB
Getting Started
Phase 0: Preparation (All 4 VMs)
Before installing anything, ensure the VMs can talk to each other and have the necessary "plumbing."
Assign Static IPs: Ensure your VMs have fixed internal IPs (e.g., 10.0.0.1 through 10.0.0.4).
Hostname Setup: Give them clear names so you don't get confused:
vm-master (Control Plane)
vm-worker-1, vm-worker-2, vm-worker-3
Disable Swap: Kubernetes (and k3s) performs better with swap off.
sudo swapoff -a
# To make it permanent, comment out the 'swap' line in /etc/fstab
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
Phase 1: Deploying Kubernetes (K3s)
Using K3s is highly recommended here because it is lightweight and handles "flaky" or high-latency networks better than standard K8s.
Step 1: Install Control Plane (VM1)
On your first VM in Country B, run:
curl -sfL https://get.k3s.io | sh -
# Get the Token to join other nodes
sudo cat /var/lib/rancher/k3s/server/node-token
Step 2a: Join Worker Nodes (VM2, VM3, VM4)
On the other three VMs, run the join command using the IP of VM1 and the token you just found:
curl -sfL https://get.k3s.io | K3S_URL=https://<VM1_IP>:6443 K3S_TOKEN=<TOKEN> sh -
Ours :
curl -sfL https://get.k3s.io | K3S_URL=https://192.168.3.91:6443 K3S_TOKEN=K10ad78a8da2499c55aeeb901a063ecf6ddd9261d57886e8813b96c6524f8a7b8f1::server:7a4c1e17141af2563287de23f083b07d sh -
Verify the cluster:
sudo kubectl get nodes
Step 2b: Label them, change x-vm2 to your server-name
kubectl label node x-vm2 node-role.kubernetes.io/worker=worker
kubectl label node x-vm3 node-role.kubernetes.io/worker=worker
kubectl label node x-vm4 node-role.kubernetes.io/worker=worker
Step 2c :Create namespaces via your control-plane:
kubectl create namespace infra # For Minio, Redis, NATS
kubectl create namespace db # For YugabyteDB
kubectl create namespace stream # For Redpanda, RisingWave, Pinot
kubectl create namespace apps # For Airflow, Superset, Postgraphile
Step 2d: install helm on your laptop (windows local host)
choco install kubernetes-helm # Installs Helm
choco install kubernetes-cli -y # Installs Kubectl (The CLI to talk to k3s)
Verify
helm version
kubectl version --client
Step 2e: setup your laptop to control the cluster:
sudo cat /etc/rancher/k3s/k3s.yaml # run in control-plane
This gives:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data:
....
server: https://127.0.0.1:6443
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
kind: Config
users:
- name: default
user:
client-certificate-data:
....
- The Critical Edit:
Find the line server: https://127.0.0.1:6443
Change 127.0.0.1 to the IP address you use to SSH into that VM (ideally your Tailscale IP). Our Control-Plane VM has IP 192.165.3.10 on private network
- Copy k3s.yaml and ensure you save it in C:\Users\<YourUser>\.kube\config
- Log back into control plane VM (x-vm1) and restart k3s with the public IP in the "Tls-San" list:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--tls-san 192.165.3.91" sh
or
you can add insecure-skip-tls-verify: true like
apiVersion: v1
clusters:
- cluster:
# certificate-authority-data: LS0tLS1... (REMOVE OR COMMENT THIS OUT)
insecure-skip-tls-verify: true
server: https://192.168.3.10:6443
name: default
contexts:
....
Check:
kubectl cluster-info
kubectl config view
kubectl get nodes
3. Phase 2: Connecting Gitea (The GitOps Flow)
Don't manually "push" code to the cluster. Instead, use ArgoCD. It sits inside your cluster in Country B and "pulls" changes from Gitea in Country A.
Install ArgoCD in your new cluster:
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
or using helm chart:
# Add the repo first if you haven't
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
# Generate the manifest
helm template argocd argo/argo-cd `
--namespace argocd `
--set server.service.type=ClusterIP `
--set "server.extraArgs={--insecure}" `
> argocd/argocd-manifest.yaml
on local machine:
kubectl port-forward svc/argocd-server -n argocd 8388:443
user : admin
pw: x (0mZZafSJi7kyBYsX)
To get x:
From PS:
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | ForEach-Object { [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($_)) }
From WSL:
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 --decode
Delete password:
kubectl delete secret argocd-initial-admin-secret -n argocd
Option: Use a NodePort (Truly Permanent)
If you don't want to run port-forward every time, you can change the Service Type to NodePort. This will open a port on your VM's physical IP address.
Update your helm template command:
--set server.service.type=NodePort `
--set server.service.nodePortHttp=30443
4. Phase 3: Handling the Container Registry
Code is just text; Kubernetes needs Images (Docker images).
Option 1 (Centralized): Enable the Gitea Container Registry on Server A1. Your CI (like Gitea Actions) builds the image in Country A and pushes it to Gitea. Your nodes in Country B then download that image.
Option 2 (Faster): If the images are large, the cross-country download will be slow. Consider setting up a Registry Mirror or a local registry in Country B to cache the images.
Summary Checklist
Network: Install Tailscale on all 4 VMs and Server A1 so they can see each other.
Cluster: Use K3s to turn the 4 VMs into a single cluster.
Deploy: Install ArgoCD on the cluster to pull code from Gitea.
CI/CD: Use Gitea Actions to build your Docker images and store them in the Gitea Registry.
Our Architecture: Country A to Country B
- Challenges: Deploy K8s in my server B1 with 4 virtual machines in a country B, my code lives in gitea hosted in a server A1 in other country A
- Answer :To deploy a Kubernetes cluster across 4 VMs in one country while pulling code from Gitea in another, you need a setup that accounts for latency and cross-border networking.
The most stable and common way to do this is using K3s for the cluster and a GitOps approach (like ArgoCD) for the deployment
Key Components:
Source (Country A): Server A1 hosts Gitea. This is your "Source of Truth."
Destination (Country B): Server B1 runs 4 VMs. You will designate 1 as the Control Plane (Master) and 3 as Worker Nodes.
The Bridge: Since the servers are in different countries, use Tailscale or WireGuard to create a secure "Mesh" network. This allows Country B's VMs to talk to Country A's Gitea as if they were in the same room.
When you run sudo swapoff -a, you are deactivating the use of the hard drive as "fake RAM."
In Docker Compose: If your VM ran out of RAM, the Linux OS would start "swapping" data to the slow disk to keep the container alive. It gets very slow, but it stays up.
In Kubernetes: K8s wants to be the "Boss" of memory. If a node is full, K8s wants to know immediately so it can move a pod to another VM (like from x-vm2 to x-vm3). If Swap is on, the node "lies" to Kubernetes saying "I still have space (on my slow disk!)," which prevents K8s from managing the cluster correctly.
Rule of Thumb: Always keep Swap OFF for K3s/K8s to ensure your databases (Yugabyte/Redpanda) don't suddenly become 100x slower.
What is Argo CD and GitOps?
Think of Argo CD as a "Sync Engine" that lives inside your K3s cluster in Country B.
The Role of GitOps
GitOps is a practice where your Git Repository is the "Single Source of Truth" for your infrastructure.
In traditional CI/CD (like Jenkins), you "push" code to the cluster.
In GitOps, the cluster "pulls" its own configuration from Git.
Why use Argo CD?
Drift Detection: If someone manually deletes a pod in Country B, Argo CD sees it doesn't match the Git repo and automatically recreates it.
Cross-Border Reliability: Since Country B "pulls" from Country A, the cluster doesn't need to be "reachable" from the internet. It only needs to be able to "see" Gitea.
Audit Trail: Every change to your cluster is a Git commit. You know exactly who changed what and when
Why ArgoCD?
If the connection between Country A and B drops for a moment, ArgoCD will simply wait and retry. It ensures your cluster eventually matches your code without you doing anything.
Install ArgoCD in your new cluster:
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
Connect Gitea: In the ArgoCD UI, add your Gitea repository URL (using the private Tailscale IP for security).
Define an "Application": Tell ArgoCD to watch a specific folder in your Gitea repo and deploy any .yaml files it finds there to your 4-node cluster.
Utilizing ArgoCD
Since you now have ArgoCD and your Kustomize structure ready, here is the file that makes ArgoCD "take over."
Create a file named argocd/root-app.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root-cluster-app
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/YOUR_USERNAME/YOUR_REPO.git
targetRevision: HEAD
path: . # Points to your root kustomization.yaml
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
-
You run kubectl apply -k . one last time manually.
-
You apply this root-app.yaml.
-
ArgoCD looks at your Git repo.
-
From now on, whenever you git push a change to Benthos, NATS, or Yugabyte, ArgoCD will see it and update the cluster automatically. You never have to run kubectl apply again.
Docker-compose vs k8s
| Docker Compose Concept | Kubernetes Equivalent |
|---|---|
| container_name: nats | Service name: nats |
| volumes: yb_data | PersistentVolumeClaim (PVC) |
| depends_on | initContainers (K8s doesn't have a native depends_on wait) |
| environment | ConfigMap or Secret |
| "ports: ""8090:8080""" | Service (Type: ClusterIP or LoadBalancer) |
Why no "Network" or "Port Mapping" 4195?
No Network: In Docker Compose, containers are isolated unless they share a network. In Kubernetes, all Pods in the same cluster can talk to each other by default. Your Benthos pod can reach NATS just by using the name nats-cluster:4222.
Port 4195: In Kubernetes, ports inside a Deployment are just "documentation" for the cluster. To actually reach that port from your Windows laptop, you need a Service.
Since you have 4 VMs, an Ingress will allow you to visit benthos.local or nats.local in your browser without having to use port-forward every time.
Converting Docker Compose to K8s: The Strategy
Do not try to convert all 30+ services at once. You should group them into Helm Charts or Kustomize folders in your Git repo.
Step A: Handle the "Files" (ConfigMaps & Secrets)
In Docker Compose, you used volumes: ./prometheus/prometheus.yml. In K8s, files are stored as ConfigMaps.
Create a ConfigMap for risingwave.toml.
Create a ConfigMap for prometheus.yml.
Create a ConfigMap for your Benthos templates. kubectl create configmap benthos-config --from-file=./benthos-configs/
Step B: The "Big Three" (StatefulSets)
Services like YugabyteDB, Redpanda, and Pinot are "Stateful." In Docker Compose, they just used local folders. In K8s, you must use a StatefulSet instead of a Deployment.
Step C: Networking
In Compose, you had nats-network. In K8s, all Pods can talk to each other by default across namespaces using the DNS format: `<service-name>.<namespace>.svc.cluster.local.`
In Docker Compose, you just use the service name (e.g., nats). In Kubernetes, cross-namespace communication uses a Fully Qualified Domain Name (FQDN).
How it works:
If Airflow (in apps) needs to talk to NATS (in infra), instead of just calling nats, it calls:nats.infra.svc.cluster.local
Why you SHOULD use namespaces:
Organization: With a stack as massive as yours, kubectl get pods in a single namespace would result in a "wall of text" that is impossible to read.
Resource Quotas: You can eventually limit how much RAM the stream namespace takes so it doesn't crash your db namespace.
Security: You can set "Network Policies" later to ensure only apps can talk to db.
Verdict: Keep the namespaces. It is a "best practice" for a stack this complex.
1. The Conversion Tool: Kompose
The fastest way to get started is using Kompose (Kubernetes + Compose). It translates docker-compose.yaml into K8s .yaml manifests (Deployments, Services, PersistentVolumeClaims).
Steps to convert:
curl -L https://github.com/kubernetes/kompose/releases/download/v1.31.2/kompose-linux-amd64 -o kompose # Install Kompose
kompose convert -f docker-compose.yaml
2. "Helm" (The Professional Way)
Most of the tools in your list have official "Charts" (install scripts). Instead of writing Yugabyte YAML from scratch, you use Helm:
# Example: Installing Yugabyte via Helm into your 'db' namespace
helm repo add yugabytedb https://charts.yugabyte.com
helm repo update
helm install my-yugabyte yugabytedb/yugabyte --namespace db
3. The Better Strategy: The "Hybrid" Approach
Instead of one giant YAML file, break your deployment into Helm Charts. Most of the tools you are using have official, high-quality Helm charts that handle the complex K8s configuration for you.
Recommendation: Use Official Charts for the "Heavy Lifters"
Don't try to manually write K8s YAML for these; use Helm:
YugabyteDB: Use the Yugabyte Helm Chart.
Airflow: Use the Official Apache Airflow Chart.
Redpanda: Use the Redpanda Operator/Chart.
RisingWave: Use the RisingWave Operator.
Use Custom YAML for your "Logic" layer:
Use your converted Kompose files only for your custom subgraphs and configurations:
subgraph-python
benthos (with ConfigMaps for your configs)
cosmo-router
Why "Blind Conversion" will fail for your stack
Your compose file has several "Cloud Native" complexities that require manual intervention:
Stateful Sets vs. Deployments: In Docker, you just list yugabytedb-1, 2, 3. In K8s, you should use a StatefulSet. This ensures that if yugabytedb-1 restarts, it attaches to the exact same disk it had before.
Healthchecks: Your healthcheck blocks in Compose need to be converted to K8s livenessProbe and readinessProbe.
Networking: In Compose, all services see each other via the container name. In K8s, you must ensure each Deployment has a Service object so that postgraphile can find yugabytedb-2 via DNS.
Initialization: Your airflow-init container should be handled as a K8s Job or an initContainer, rather than a long-running service with a profile.
The Result: It will generate dozens of files. Do not apply them yet. Kompose handles simple web apps well but struggles with complex distributed systems like YugabyteDB, Redpanda, or Airflow.
Recommended Repository Structure (Gitea)
Organize your Gitea repo like this so ArgoCD can read it easily:
/my-cluster-repo
/infra
nats-deployment.yaml
minio-deployment.yaml
redis-configmap.yaml
/db
yugabyte-statefulset.yaml
/streaming
redpanda-statefulset.yaml
risingwave-deployment.yaml
/apps
airflow-helm-values.yaml
The "Missing Link": How K3s creates a cluster from images?
This is the most common point of confusion. The cluster doesn't "create itself" from an image; it downloads the image to run your application. Here is the step-by-step flow:
Gitea Action (CI): Your code changes → Gitea builds a Docker image → It pushes that image to the Gitea Container Registry (on Server A1).
Manifest Update: The Gitea Action also updates a small YAML file in your Git repo (e.g., changing image: v1.0 to image: v1.1).
Argo CD (CD): Argo CD (in Country B) notices the YAML file changed.
The Pull: Argo CD tells K3s: "Update this deployment to use image v1.1."
K3s Execution: K3s looks at the Gitea Registry in Country A, pulls (downloads) the new image, and restarts the pods.
Helm Charts: K3s vs. K8s
Can we use Helm for K3s?
Yes. K3s is a fully certified Kubernetes distribution. Anything that works on standard K8s works on K3s. In fact, K3s comes with a Helm Controller pre-installed, allowing you to deploy Helm charts just by dropping a YAML file into a specific folder (/var/lib/rancher/k3s/server/manifests).
Are Helm Charts identical?
99% of the time, Yes. * The Kubernetes resources (Deployments, Services, Ingress) are identical.
The only difference usually involves the "Ingress Controller." Standard K8s often uses NGINX, while K3s comes with Traefik by default. You might need to adjust your Helm values.yaml to tell it to use Traefik instead of NGINX.
Helm
The Kubernetes package manager
Common actions for Helm:
- helm search: search for charts
- helm pull: download a chart to your local directory to view
- helm install: upload the chart to Kubernetes
- helm list: list releases of charts
Environment variables:
| Name | Description |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| $HELM_CACHE_HOME | set an alternative location for storing cached files. |
| $HELM_CONFIG_HOME | set an alternative location for storing Helm configuration. |
| $HELM_DATA_HOME | set an alternative location for storing Helm data. |
| $HELM_DEBUG | indicate whether or not Helm is running in Debug mode |
| $HELM_DRIVER | set the backend storage driver. Values are: configmap, secret, memory, sql. |
| $HELM_DRIVER_SQL_CONNECTION_STRING | set the connection string the SQL storage driver should use. |
| $HELM_MAX_HISTORY | set the maximum number of helm release history. |
| $HELM_NAMESPACE | set the namespace used for the helm operations. |
| $HELM_NO_PLUGINS | disable plugins. Set HELM_NO_PLUGINS=1 to disable plugins. |
| $HELM_PLUGINS | set the path to the plugins directory |
| $HELM_REGISTRY_CONFIG | set the path to the registry config file. |
| $HELM_REPOSITORY_CACHE | set the path to the repository cache directory |
| $HELM_REPOSITORY_CONFIG | set the path to the repositories file. |
| $KUBECONFIG | set an alternative Kubernetes configuration file (default "~/.kube/config") |
| $HELM_KUBEAPISERVER | set the Kubernetes API Server Endpoint for authentication |
| $HELM_KUBECAFILE | set the Kubernetes certificate authority file. |
| $HELM_KUBEASGROUPS | set the Groups to use for impersonation using a comma-separated list. |
| $HELM_KUBEASUSER | set the Username to impersonate for the operation. |
| $HELM_KUBECONTEXT | set the name of the kubeconfig context. |
| $HELM_KUBETOKEN | set the Bearer KubeToken used for authentication. |
| $HELM_KUBEINSECURE_SKIP_TLS_VERIFY | indicate if the Kubernetes API server's certificate validation should be skipped (insecure) |
| $HELM_KUBETLS_SERVER_NAME | set the server name used to validate the Kubernetes API server certificate |
| $HELM_BURST_LIMIT | set the default burst limit in the case the server contains many CRDs (default 100, -1 to disable) |
| $HELM_QPS | set the Queries Per Second in cases where a high number of calls exceed the option for higher burst values |
| $HELM_COLOR | set color output mode. Allowed values: never, always, auto (default: never) |
| $NO_COLOR | set to any non-empty value to disable all colored output (overrides $HELM_COLOR) |
Helm stores cache, configuration, and data based on the following configuration order:
- If a HELM_*_HOME environment variable is set, it will be used
- Otherwise, on systems supporting the XDG base directory specification, the XDG variables will be used
- When no other location is set a default location will be used based on the operating system
By default, the default directories depend on the Operating System. The defaults are listed below:
| Operating System | Cache Path | Configuration Path | Data Path |
| ---------------- | ------------------------- | ------------------------------ | ----------------------- |
| Linux | $HOME/.cache/helm | $HOME/.config/helm | $HOME/.local/share/helm |
| macOS | $HOME/Library/Caches/helm | $HOME/Library/Preferences/helm | $HOME/Library/helm |
| Windows | %TEMP%\helm | %APPDATA%\helm | %APPDATA%\helm |
Usage:
helm [command]
Available Commands:
completion generate autocompletion scripts for the specified shell
create create a new chart with the given name
dependency manage a chart's dependencies
env helm client environment information
get download extended information of a named release
help Help about any command
history fetch release history
install install a chart
lint examine a chart for possible issues
list list releases
package package a chart directory into a chart archive
plugin install, list, or uninstall Helm plugins
pull download a chart from a repository and (optionally) unpack it in local directory
push push a chart to remote
registry login to or logout from a registry
repo add, list, remove, update, and index chart repositories
rollback roll back a release to a previous revision
search search for a keyword in charts
show show information of a chart
status display the status of the named release
template locally render templates
test run tests for a release
uninstall uninstall a release
upgrade upgrade a release
verify verify that a chart at the given path has been signed and is valid
version print the helm version information
Flags:
--burst-limit int client-side default throttling limit (default 100)
--color string use colored output (never, auto, always) (default "auto")
--colour string use colored output (never, auto, always) (default "auto")
--content-cache string path to the directory containing cached content (e.g. charts) (default "C:\\Users\\wendg2\\AppData\\Local\\Temp\\helm\\content")
--debug enable verbose output
-h, --help help for helm
--kube-apiserver string the address and the port for the Kubernetes API server
--kube-as-group stringArray group to impersonate for the operation, this flag can be repeated to specify multiple groups.
--kube-as-user string username to impersonate for the operation
--kube-ca-file string the certificate authority file for the Kubernetes API server connection
--kube-context string name of the kubeconfig context to use
--kube-insecure-skip-tls-verify if true, the Kubernetes API server's certificate will not be checked for validity. This will make your HTTPS connections insecure
--kube-tls-server-name string server name to use for Kubernetes API server certificate validation. If it is not provided, the hostname used to contact the server is used
--kube-token string bearer token used for authentication
--kubeconfig string path to the kubeconfig file
-n, --namespace string namespace scope for this request
--qps float32 queries per second used when communicating with the Kubernetes API, not including bursting
--registry-config string path to the registry config file (default "C:\\Users\\wendg2\\AppData\\Roaming\\helm\\registry\\config.json")
--repository-cache string path to the directory containing cached repository indexes (default "C:\\Users\\wendg2\\AppData\\Local\\Temp\\helm\\repository")
--repository-config string path to the file containing repository names and URLs (default "C:\\Users\\wendg2\\AppData\\Roaming\\helm\\repositories.yaml")
Use "helm [command] --help" for more information about a command.
What is an Ingress and when to use it?
Think of your 4 VMs as a private gated community.
Pods are the houses.
Services are the internal phone extensions (e.g., dial 4195 for Benthos).
Ingress is the Security Guard at the Main Gate.
When does it make sense?
When you want "Pretty" URLs: Instead of remembering 192.168.1.50:31045, you want to type benthos.local or grafana.local.
SSL/HTTPS: You want to handle all your security certificates in one place (the Ingress) rather than inside every single pod.
Single Entry Point: You only want to open Port 80 (HTTP) and 443 (HTTPS) on your VM firewalls. The Ingress decides which pod gets the traffic based on the "Host" name you typed in your browser.
When NOT to use it?
Internal Traffic: Benthos talking to NATS should never go through an Ingress. They should talk directly via the internal Service name (nats-cluster). Ingress is for human-to-cluster or external-app-to-cluster traffic.
How to use Ingress in K3s
K3s comes with a built-in Ingress controller called Traefik. You don't need to install anything! You just need to define an "Ingress Route."
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: benthos-ingress
namespace: infra
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
rules:
- host: benthos.local # You will add this to your Windows 'hosts' file
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: benthos-ui
port:
number: 4195
The Final Step: The hosts file Since benthos.local isn't a real internet address, you need to tell your Windows laptop where to find it.
Open Notepad as Administrator.
Open C:\Windows\System32\drivers\etc\hosts.
Add a line with the IP of any of your 4 VMs: 192.168.x.x benthos.local
Now, you can just type http://benthos.local in Chrome!
Longhorn
Setup Storage: Install a "Storage Class" on your K3s cluster (like Longhorn or the default Local-Path Provisioner). This allows Yugabyte and ElasticSearch to claim disk space on your VMs.
in Docker Compose, your data stayed on the VM's local folder. In K8s, pods can "float" between VMs. If yugabytedb moves from x-vm2 to x-vm4, it needs its data to follow it.
The Disaster Scenario (No Longhorn):
Your NATS data is saved on x-vm2's hard drive.
x-vm2 crashes or loses power.
Kubernetes moves your NATS pod to x-vm3.
The Problem: x-vm3 doesn't have the data! It's trapped on the dead hard drive of x-vm2. Your database is now empty or corrupted.
The Longhorn Solution:
Longhorn takes the hard drives of all 4 VMs and creates a "Shared Pool." When NATS writes a file, Longhorn instantly copies it to 3 different VMs. If one VM dies, the pod starts on another VM, and Longhorn "plugs in" the replicated data immediately.
Longhorn is a lightweight storage orchestrator. It takes the empty space on all 4 of your VMs and turns it into one big "distributed disk."
In windows:
helm repo add longhorn https://charts.longhorn.io
helm repo update
Install it into your cluster:
kubectl create namespace longhorn-system
helm install longhorn longhorn/longhorn --namespace longhorn-system
kubectl patch storageclass longhorn -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
Connecting to proxmox host via ssh
- Generate ssh key: ssh-keygen -t ed25519
- Push the Key to Proxmox: type $env:USERPROFILE.ssh\id_ed25519.pub | ssh root@192.168.3.50 "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
pveum user token add root@pam terraform-token --privsep 0
1. Cluster Setup (3 Nodes, 12 VMs)
First, you should join your three servers into a Proxmox Cluster. This allows you to manage all three from a single interface and move VMs between them easily.
Create Cluster: Log into Node 1 (192.168.3.70), go to Datacenter > Cluster, and click Create Cluster.
Join Nodes: Copy the "Join Information" and paste it into the Cluster > Join Cluster section on Node 2 and Node 3.
VM Deployment: You can then create 4 VMs on each node.
Pro Tip: Create one "Gold Master" VM with Docker installed, convert it to a Template, and then "Clone" it 12 times to save hours of setup.
2. Remote Access for Developers
Since your IPs (192.x.x.x) are private, developers at other locations cannot see them. Do not use Port Forwarding for Proxmox; it is a security risk.
The "Mesh VPN" Way (Easiest): Install Tailscale or ZeroTier on each VM (or just one "Gateway" VM).
This creates a secure virtual network. Developers simply install the same app, and they can access the VMs as if they were in the same room.
The Reverse Proxy Way: If the apps are web-based, use Nginx Proxy Manager or Cloudflare Tunnels. This allows developers to access apps via a URL (e.g., app1.yourdomain.com) without a VPN.
3. CI/CD Workflow: Gitea → Docker → Argo CD
Argo CD is designed for Kubernetes, but you can use it for Docker deployments if you run a lightweight Kubernetes cluster (like K3s) inside your VMs.
The Architecture
Gitea (The "Source"): Hosts your code and docker-compose.yaml or Kubernetes manifests.
Gitea Actions (The "Builder"): When code is pushed, a runner builds the Docker image and pushes it to a Registry (Gitea has a built-in container registry).
Argo CD (The "Operator"): Watches your Gitea repository. When it sees a change, it automatically pulls the new image and updates the deployment in the VM.
Deployment Steps
Install Gitea: Run Gitea in one of your VMs (using Docker).
Install K3s: On your 12 VMs, install K3s (curl -sfL https://get.k3s.io | sh -). This gives you a tiny Kubernetes environment in each VM.
Install Argo CD: Install Argo CD inside your K3s cluster.
Connect Repo: Point Argo CD to your Gitea repository URL.
Sync: Set the sync policy to "Automatic." Now, every time a developer pushes code to Gitea, Argo CD will update the VM automatically.
free -h # physical RAM installed and recognized df -h # Proxmox-Specific Storage lsblk # Physical Disk Overview fdisk -l # Detailed Hardware Info htop # pre installed dmidecode -t memory | grep -i size # hardware info pveperf