### Getting started kubectl apply -k . # from root project folder if not detected: kubectl get pods -n db kubectl apply -f db/yugabytedb.yaml -n db - Find services and its ports kubectl get svc -n apps - Port forwarding local to cosmo-router,argo-cd kubectl port-forward svc/cosmo-router 3002:3002 -n apps kubectl port-forward svc/argocd-server -n argocd 8480:443 - Open yugabyte kubectl exec -it yb-tserver-0 -n db -- ysqlsh -h yb-tserver-0 - Open nats kubectl run nats-tools --image=natsio/nats-box -n infra --rm -it -- nats stream info request_stream --server=nats://nats-cluster:4222 if already exist: kubectl exec -it nats-box -n infra -- nats stream info request_stream -s nats://nats-cluster:4222 - list stream: kubectl exec -it nats-box -n infra -- nats -s nats://nats-cluster.infra.svc.cluster.local:4222 stream ls - **add stream**: kubectl exec -it nats-box nats -n infra -- /bin/sh nats -s nats://nats-cluster.infra.svc.cluster.local:4222 stream add request_stream --subjects "input_request_logs" --ack --storage file --retention limits --max-msgs=-1 --max-bytes=-1 --max-age=1y --replicas 3 check: kubectl describe pod nats-cluster-1 -n infra - Check benthos kubectl logs -f benthos-556f97988-cgd7f -n infra - Redeploy For cosmo-router under namespace apps: ``` to apply changes: helm upgrade cosmo-router ./cosmo-router -n apps # from folder apps we can not use this command to apply changes: kubectl rollout restart deployment cosmo-router -n apps ``` ### Install Linkerd on local machine Linkerd exe is installed in your folder of choice, and set environment variables path to refer to this. ### Some linkerd commands Some commands: linkerd viz dashboard linkerd version --client linkerd check --linkerd-namespace apps linkerd -n apps stat deployments linkerd check --pre ### Deinstalling Linkerd: - Deleting Linkerd (depends on namespace) : ``` - kubectl delete deployment linkerd-destination linkerd-identity linkerd-proxy-injector -n apps - kubectl delete deployment metrics-api prometheus tap tap-injector web -n apps - Delete the Mutating Webhooks (CRITICAL): - kubectl delete mutatingwebhookconfiguration linkerd-proxy-injector-webhook-config - kubectl delete mutatingwebhookconfiguration linkerd-tap-injector-webhook-config - kubectl delete validatingwebhookconfiguration linkerd-sp-validator-webhook-config - Remove the annotation from your namespace: kubectl annotate namespace apps linkerd.io/inject- - Restart your apps to strip the sidecars: kubectl rollout restart deployment cosmo-router -n apps kubectl rollout restart statefulset nats-cluster -n infra ``` - Deleting Linkerd using Helm installation: ``` # 1. Delete the current broken Linkerd installations helm uninstall linkerd-viz -n apps helm uninstall linkerd-control-plane -n apps helm uninstall linkerd-crds -n apps # 2. IMPORTANT: Delete any leftover Linkerd webhooks (these often block restarts) kubectl delete mutatingwebhookconfiguration linkerd-proxy-injector-webhook-config kubectl delete validatingwebhookconfiguration linkerd-sp-validator-webhook-config ``` - Deleting Linkerd using linkerd CLI: linkerd uninstall --namespace apps | kubectl delete -f - ### Open source Helm chart Most of the time we do not need to write k8s yaml files. If we have helm chart available from the provider we generate the files needed using helm template and refer the generated file from kustomization.yaml #### Using helm chart vs kustomize if you want to move your cluster to new VMs, you just run: To run with helm chart: kubectl apply -k . Kubernetes will create the namespaces, then deploy NATS, then Benthos, then Yugabyte all in one go |Method|Pro|Con| |--|---|--| |helm install|Fast and easy for one-offs.|Hard to remember settings; hard to track in Git.| |Kustomize + Helm|Single source of truth; everything in Git.|Requires --enable-helm flag; slightly more setup.| - Manual way with helm chart We also have 2 ways by using helm template vs helm install ``` helm template yugabytedb yugabytedb/yugabyte ` --namespace db ` --set storage.master.storageClass=longhorn ` --set storage.tserver.storageClass=longhorn ` --set replicas.master=3 ` --set replicas.tserver=3 ` --set enableLoadBalancer=false ` --set gflags.master.max_clock_skew_usec=2000000 ` --set gflags.tserver.max_clock_skew_usec=2000000 ` --set gflags.master.time_source=system ` --set gflags.tserver.time_source=system ` --set gflags.tserver.start_pgsql_proxy=true ` > db/yugabytedb.yaml OR helm install yugabytedb yugabytedb/yugabyte ` --namespace db ` --set storage.master.storageClass=longhorn ` --set storage.tserver.storageClass=longhorn ` --set replicas.master=3 ` --set replicas.tserver=4 ` --set gflags.master.max_clock_skew_usec=2000000 ` --set gflags.tserver.max_clock_skew_usec=2000000 ` --set gflags.master.time_source=system ` --set enableLoadBalancer=false ` --set gflags.tserver.start_pgsql_proxy=true ` --set gflags.tserver.time_source=system ``` |Feature|helm template|helm install| |---|---|----| |Action|Local Only. Generates raw Kubernetes YAML and prints it |to a file.|Live. Sends the YAML directly to your cluster API.| |Result|"You get a file (db/yugabytedb.yaml) that you can |inspect| edit| or commit to Git."|The database starts running |immediately in your db namespace.| |Usage|Best for GitOps/Kustomize. You use the generated file as |a resource in Kustomize.|Best for Quick deployment. Harder to track changes over time in Git.| - Kustomize way but required installing standalone Kustomization binary file on windows `./kustomize build . --enable-helm | kubectl apply -f -` We do not use Kustomize here #### NATS ``` helm repo add nats https://nats-io.github.io/k8s/helm/charts/ helm repo update - Basic nats-values.yaml config: jetstream: enabled: true fileStore: pvc: size: 5Gi # Reserve 5GB for JetStream or - HA # To increase replicas for High Availability replicaCount: 3 # JetStream Clustering (The "Replica Factor") config: cluster: enabled: true jetstream: enabled: true # This ensures your data survives if x-vm2 goes down fileStore: pvc: enabled: true storageClassName: "local-path" # k3s default storage size: 10Gi Check: # Create the namespace first kubectl create namespace infra # Install NATS using the chart and your values file helm install nats-cluster nats/nats --namespace infra -f nats-values.yaml or with helm template helm template nats-cluster nats/nats ` --namespace infra ` --set replicaCount=3 ` --set config.cluster.enabled=true ` --set config.jetstream.enabled=true ` --set config.jetstream.fileStore.pvc.enabled=true ` --set config.jetstream.fileStore.pvc.storageClassName="longhorn" ` --set config.jetstream.fileStore.pvc.size=10Gi ` > infra/nats-cluster.yaml Apply the template (without creating cluster) kubectl apply -f infra/nats-cluster.yaml Apply the template( with creating cluster pods, service, etc): helm install nats-cluster nats/nats --namespace infra -f nats-values.yaml After modification of other files you can apply the changes by: helm upgrade nats-cluster nats/nats --namespace infra -f nats-values.yaml Remove nats-cluster helm uninstall nats-cluster -n infra Remove pvc of nats-cluster (first uninstall nats-cluster using helm-uninstall) kubectl delete pvc -n infra -l app.kubernetes.io/instance=nats-cluster Verify: kubectl get all -n infra kubectl get statefulset -n infra kubectl get pods -n infra -w kubectl get pvc -n infra kubectl get sc # verify longhorn is ready and status healthy To avoid typing -n infra every time, you can switch your "active room" to infra permanently: kubectl config set-context --current --namespace=infra Apply changes: helm upgrade nats-cluster nats/nats -n infra -f nats-values.yaml Testing: first terminal: kubectl run nats-box --image=natsio/nats-box:latest -n infra -it --rm nats sub -s nats-cluster test.topic new terminal: # Instead of 'run', we 'exec' into the pod that is already there kubectl exec -n infra -it nats-box -- /bin/sh nats pub -s nats-cluster test.topic "Hello from Country B!" or kubectl exec -it nats-box -n infra -- nats -s nats-cluster:4222 pub input_request_logs '{"content": {"hash": "test-123"}, "message": "Hello Benthos!"}' Delete: kubectl delete statefulset nats-cluster -n infra kubectl delete all -l app.kubernetes.io/instance=nats-cluster -n default ``` #### Intrepretation ``` kubectl get all -n infra NAME READY STATUS RESTARTS AGE pod/nats-box 1/1 Running 0 13m pod/nats-cluster-0 2/2 Running 0 87s pod/nats-cluster-1 2/2 Running 0 87s pod/nats-cluster-2 2/2 Running 0 87s pod/nats-cluster-box-868cc6c48b-dlvtn 1/1 Running 0 26m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nats-cluster ClusterIP 10.43.208.167 4222/TCP 26m service/nats-cluster-headless ClusterIP None 4222/TCP,6222/TCP,8222/TCP 26m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nats-cluster-box 1/1 1 1 26m NAME DESIRED CURRENT READY AGE replicaset.apps/nats-cluster-box-868cc6c48b 1 1 1 26m NAME READY AGE statefulset.apps/nats-cluster 3/3 88s Interpretation: The "Power Trio" (StatefulSet) pod/nats-cluster-0 pod/nats-cluster-1 pod/nats-cluster-2 Why: The official Helm chart defaults to replicaCount: 3. This is for High Availability. Since you have 4 VMs, Kubernetes likely spread these across your different nodes. If one VM in Country B fails, NATS will keep running because the other two pods have a "quorum" (majority) and won't lose your data. The "Utility" Pods (Box) pod/nats-box: This is the temporary pod you created manually with kubectl run to test the connection. It stays there until you delete it or it finishes. pod/nats-cluster-box-868cc6c48b-dlvtn: This was created automatically by the Helm chart. The NATS team includes a permanent "box" deployment so you always have a toolset inside the cluster to check the status of the NATS stream. What about the "2/2" READY status? You'll notice the nats-cluster-x pods say 2/2. This means each pod actually contains two containers: The NATS Server: The actual engine. The NATS Config Reloader: A "sidecar" container that watches for changes to your settings and tells the NATS server to refresh without restarting. ``` #### Rules of Engagement Creating stream with Benthos for how NATS stores your data ``` ? Retention Policy Limits, WorkQueue, Interest ? Discard Policy Old vs New In NATS, -1 means "Infinite" or "No Limit." ? Stream Messages Limit -1 ? Per Subject Messages Limit -1 ? Total Stream Size -1 ? Message TTL -1 ? Max Message Size -1 ? Duplicate tracking time window 2m0s ? Allow message Roll-ups Yes ? Allow message deletion Yes ? Allow purging subjects or the entire stream (Y/n) nats -s nats-cluster:4222 stream add request_stream --subjects "input_request_logs" --ack --storage file --retention limits --max-msgs=-1 --max-bytes=-1 --max-age=1y --replicas 3 nats -s nats-cluster:4222 stream add transaction_stream --subjects "input_transaction_logs" --ack --storage file --replicas 3 ``` ### Benthos ``` # Running namespaces kubectl apply -f namespaces.yaml # run this command from the root of your repo # Running Kustomization Deploy configs kubectl apply -k . -n infra # from infra\benthos folder kubectl apply -k .\benthos\ -n infra # from infra folder # Manual creation of config map kubectl create configmap benthos-config --from-file=./benthos-configs/ # Verify ConfigMaps Check if your files were successfully "packed" into Kubernetes: kubectl get configmap -n infra # To see the actual content of your streams inside K8s: kubectl describe configmap benthos-streams -n infra # Verify the 3 Replicas kubectl get pods -n infra -l app=benthos # Verify the Dashboard (Port-Forward) To see the Benthos UI on your laptop: kubectl port-forward svc/benthos-ui -n infra 4195:4195 # Peek inside the running Pod: kubectl exec -it -n infra -- ls /configs/streams kubectl exec -it -n infra -- ls -R /configs # Verify kubectl get svc -n infra. # Create Stream: kubectl exec -n infra -it nats-box -- /bin/sh ``` #### Debugging ``` kubectl describe pod benthos-68cb959d58-47prd -n infra kubectl logs benthos-86f5c886b7-5fr5t -n infra If needed delete only deployment and redeploy using kustomization Depends on the your position kubectl delete deployment benthos -n infra kubectl apply -k .\infra\benthos\ -n infra or only without namespaces kubectl apply -k .\infra\benthos\ or only (if you are inside \infra\benthos\) kubectl apply -k . # You must not delete configmap since it get refreshed everytime you changed. Optional: kubectl delete configmap --all -n infra (Only if only Benthos is in infra) # Check output after redeploy kubectl get pod -n infra -o yaml kubectl kustomize . ``` ### Installing Longhorn on each VMs - Prerequisite You need to run the following command on all 4 VMs (Master and all Workers) to ensure Longhorn can communicate with the disks: ``` sudo apt install open-iscsi nfs-common util-linux -y sudo systemctl enable --now iscsid Why do we need this? Longhorn creates "Block Devices." To do that, the Linux kernel on your VM needs open-iscsi to "attach" to the virtual disks Longhorn creates. Without this, your pods will be stuck in ContainerCreating forever 1. Add the repo helm repo add longhorn https://charts.longhorn.io helm repo update 2. Install into a new namespace Longhorn is a "System Service" not Application Infrastructure, requires high privileges helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace - Check : kubectl get pods -n longhorn-system 3. Running the StorageClass Patch from Local Windows Laptop Once installed, tell K3s to use Longhorn for every database deployed: kubectl patch storageclass longhorn -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' or kubectl patch storageclass longhorn -p "{\"metadata\": {\"annotations\":{\"storageclass.kubernetes.io/is-default-class\":\"true\"}}}" or Run kubectl edit storageclass longhorn Look for the metadata: section. Add this line under annotations: storageclass.kubernetes.io/is-default-class: "true" Save and exit. - Notes Because you have 4 VMs, Longhorn will default to 3 replicas for your data. This is perfect. It means: Data is written to VM1. Longhorn clones it to VM2 and VM3. VM4 stays as a "spare" or handles other data. If any one VM catches fire, your data is still 100% safe and available. 4. UI Longhorn: A. Update your Windows Hosts file: - Open Notepad (Make sure you have right to write) - Open C:\Windows\System32\drivers\etc\hosts. - Add the IP of your Master VM: 192.168.x.x longhorn.local (Replace with your actual VM IP) B. Create longhorn.ingress.yaml and run: /country-b-cluster-ops /system <-- NEW FOLDER for cluster-wide tools longhorn-ingress.yaml /infra <-- For your messaging/processing (NATS, Benthos) /db <-- For your databases (Yugabyte) kubectl apply -f system/longhorn-ingress.yaml C. Open http://longhorn.local from your laptop ``` #### Additional Configuration? There is one common issue with K3s and Longhorn. K3s stores its data in /var/lib/rancher/k3s/storage, but Longhorn defaults to /var/lib/longhorn. Recommendation: If you have a specific large hard drive or partition on your VMs where you want the data to live, you can configure that in the Longhorn UI. If you just have one big / partition, the default is fine. ### Yugabyte ``` helm repo add yugabytedb https://charts.yugabyte.com helm repo update helm install yugabytedb yugabytedb/yugabyte ` --namespace db ` --set storage.master.storageClass=longhorn ` --set storage.tserver.storageClass=longhorn ` --set replicas.master=3 ` --set replicas.tserver=4 ` --set gflags.master.max_clock_skew_usec=2000000 ` --set gflags.tserver.max_clock_skew_usec=2000000 ` --set gflags.master.time_source=system ` --set enableLoadBalancer=false ` --set gflags.tserver.start_pgsql_proxy=true ` --set gflags.tserver.time_source=system or helm template yugabytedb yugabytedb/yugabyte ` --namespace db ` --set storage.master.storageClass=longhorn ` --set storage.tserver.storageClass=longhorn ` --set replicas.master=3 ` --set replicas.tserver=4 ` --set gflags.master.max_clock_skew_usec=2000000 ` --set gflags.tserver.max_clock_skew_usec=2000000 ` --set gflags.master.time_source=system ` --set enableLoadBalancer=false ` --set gflags.tserver.start_pgsql_proxy=true ` --set gflags.tserver.time_source=system > yugabytedb.yaml ``` - set gflags.tserver.start_pgsql_proxy=true? This flag tells the Yugabyte T-Server to turn on the YSQL API layer - Fully Qualified Domain Name FQDN for communication - yugabytedb.db.svc.cluster.local - The pattern is: `[service-name].[namespace].svc.cluster.local` #### Optimal Node Distribution for 4 VMs ``` For a 4-node cluster, you have enough overhead to ensure High Availability (HA) while maximizing resource use. Master Count: 3 Why: Yugabyte Masters use the Raft consensus algorithm. You need an odd number to avoid "split-brain" scenarios. 3 masters can tolerate 1 node failure. Since you have 4 VMs, you should stick with 3. T-Server Count: 3 or 4 Option 3 (Balanced): Keep 3 T-Servers. This leaves 1 VM entirely free for Benthos, NATS, and Redpanda. Option 4 (High Performance): Run 4 T-Servers. This spreads your data shards across all available hardware. Recommendation: Go with 3 Masters and 3 T-Servers. This keeps the "Replication Factor" (RF) at 3, which is the standard. Adding a 4th T-Server with RF3 is possible, but 3/3 is more predictable for a small cluster. ``` ### Linkerd, Viz, openssl on k8s ``` 1. Create the Trust Anchor (Root CA) # Generate private key for the Root a.openssl ecparam -name prime256v1 -genkey -noout -out ca.key b.openssl ecparam -name prime256v1 -genkey -noout -out ca.key # Generate the self-signed Root Certificate (Valid for 10 years) a.openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 \ -out ca.crt -subj "/CN=root.linkerd.cluster.local" \ -addext "basicConstraints=critical,CA:TRUE" b.openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.crt -subj "/CN=root.linkerd.cluster.local" -addext "basicConstraints=critical,CA:TRUE" 2. Create the Identity Issuer (Intermediate CA) # Generate private key for the Issuer a/b. openssl ecparam -name prime256v1 -genkey -noout -out issuer.key # Create a CSR (Certificate Signing Request) a/b. openssl req -new -key issuer.key -out issuer.csr -subj "/CN=identity.linkerd.cluster.local" # Create a config file for the intermediate CA extensions a. cat > issuer.ext < ext.txt echo "keyUsage=critical,digitalSignature,keyCertSign,cRLSign" >> ext.txt openssl x509 -req -in issuer.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out issuer.crt -days 365 -sha256 -extfile ext.txt 3. Generate k3s Manifests helm repo add linkerd https://helm.linkerd.io/stable && helm repo update 4. Generate CRDs Linkerd Custom Resource Definitions must be installed first. helm template linkerd-crds linkerd/linkerd-crds --namespace linkerd > linkerd-crds.yaml 5. Generate Control Plane We will bake your certificates directly into the generated YAML: ``` helm template linkerd-control-plane linkerd/linkerd-control-plane --namespace linkerd --set-file identityTrustAnchorsPEM=ca.crt --set-file identity.issuer.tls.crtPEM=issuer.crt --set-file identity.issuer.tls.keyPEM=issuer.key --set identity.issuer.scheme=kubernetes.io/tls > linkerd-control-plane.yaml or helm install linkerd-control-plane linkerd/linkerd-control-plane -n linkerd --set-file identityTrustAnchorsPEM=ca.crt --set-file identity.issuer.tls.crtPEM=issuer.crt --set-file identity.issuer.tls.keyPEM=issuer.key ``` 6. Generate Viz (Dashboard) Default namespaces are linkerd and linkerd-viz but we save them under our namespace apps ``` a. helm template linkerd-viz linkerd/linkerd-viz --namespace linkerd-viz > linkerd-viz.yaml (standard namespace) b. helm template linkerd-viz linkerd/linkerd-viz \ --namespace apps \ --set linkerdNamespace=apps \ --set tap.namespace=apps \ --set dashboard.namespace=apps \ --set prometheus.namespace=apps > linkerd-viz.yaml or using c. helm install linkerd-viz linkerd/linkerd-viz -n linkerd-viz --create-namespace Uninstalling: helm ls -n apps helm uninstall linkerd-viz -n apps helm uninstall linkerd-control-plane -n apps helm uninstall linkerd-crds -n apps ``` 7. Creating k8s Secret in our namespace the pod is looking for secret in the apps namespace before the pod can start: kubectl create secret tls linkerd-identity-issuer --cert=issuer.crt --key=issuer.key --namespace=linkerd or kubectl create secret tls linkerd-identity-issuer --cert=issuer.crt --key=issuer.key --namespace=apps 8.. Apply CRDs first kubectl apply -f linkerd-crds.yaml 9. Apply Control Plane kubectl apply -f linkerd-control-plane.yaml 10. Apply Viz kubectl apply -f linkerd-viz.yaml 11. Activating the Connection (Injection) kubectl annotate namespace apps linkerd.io/inject=enabled 12. Check pods status kubectl get pods -n apps Redeploy if error linkerd-destination-6b69957545-9b9tz 0/4 CrashLoopBackOff 720 (2m23s ago) 12h linkerd-destination-74cc587f5-gvgqd 0/4 CrashLoopBackOff 712 (50s ago) 12h linkerd-identity-68858f6c75-t2lk9 2/2 Running 0 14h linkerd-proxy-injector-54bc495b55-sgk6l 0/2 CrashLoopBackOff 309 (2m23s ago) 12h linkerd-proxy-injector-584f78bf9-86r64 0/2 CrashLoopBackOff 306 (60s ago) 12h - destination: The main service discovery logic. -policy: Handles authorization policies. -linkerd-proxy: The "sidecar" that handles the pod's own traffic. -linkerd-init: (Finished) Sets up network rules. 13.Rollout Restart: For existing services (like Yugabyte or your Python subgraph) to join the mesh, they need to be restarted: kubectl rollout restart deployment cosmo-router -n apps kubectl rollout restart statefulset yugabytedb -n db 12. How to Verify To verify the connection and see your Yugabyte/Redpanda traffic: Check Status: linkerd check (Requires Linkerd CLI: curl -sL https://run.linkerd.io/install | sh). Open Dashboard: linkerd viz dashboard. Check mTLS: In the dashboard, look for the shield icon next to your cosmo-router to subgraph traffic. This confirms your OpenSSL certificates are working. Debugging: kubectl describe pod -n apps -l linkerd.io/control-plane-component=identity openssl x509 -in ca.crt -text -noout kubectl logs -n apps deployment/linkerd-identity ``` #### Linkerd Activation / Deactivation ``` # Run these commands to tell Linkerd that these namespaces are now part of the mesh: kubectl annotate namespace apps linkerd.io/inject=enabled kubectl annotate namespace infra linkerd.io/inject=enabled kubectl annotate namespace db linkerd.io/inject=enabled kubectl annotate namespace stream linkerd.io/inject=enabled # Restart everything in those namespaces kubectl rollout restart deployment -n apps kubectl rollout restart deployment -n infra kubectl rollout restart statefulset -n infra # For your NATS cluster kubectl rollout restart statefulset -n db # For your Yugabyte/DB kubectl rollout restart deployment -n stream # Remove the annotation to deactivate kubectl annotate namespace apps linkerd.io/inject- kubectl annotate namespace infra linkerd.io/inject- kubectl annotate namespace db linkerd.io/inject- kubectl annotate namespace stream linkerd.io/inject- # Clean the pods kubectl rollout restart deployment -n apps kubectl rollout restart deployment -n infra kubectl rollout restart statefulset -n infra kubectl rollout restart statefulset -n db ``` ### Cosmo router ``` 1. Check cosmo-router/ ├── config.yaml <-- Your router config ├── router.json <-- Your composed graph ├── values.yaml <-- Updated with Ingress and Middleware annotations └── templates/ ├── _helpers.tpl <-- Required for names ├── configmap.yaml <-- Uses .Files.Get(our custom config) ├── deployment.yaml <-- Incl Checksum & Linkerd injection (The brain) ├── service.yaml (The external network) ├── ingress.yaml (The internal network) └── middleware.yaml <-- The Gzip logic └── hpa.yaml optional Delete: a. httproute.yaml (Causes the current error) b.serviceaccount.yaml (Caused the previous error) c. tests/ (The default test folder often causes similar issues) 2. Test it helm template cosmo-router ./cosmo-router 3. Run cosmo-router # Deploy it helm install cosmo-router ./cosmo-router -n apps or Redeploy helm upgrade cosmo-router ./cosmo-router -n apps or If needed uninstall helm uninstall linkerd-crds -n apps helm uninstall cosmo-router -n apps Some usefult commands: kubectl delete pods -n apps -l linkerd.io/extension=viz kubectl annotate namespace apps linkerd.io/inject=enables kubectl rollout restart deployment cosmo-router -n apps kubectl logs -n apps -l app.kubernetes.io/name=cosmo-router ``` #### Helm chart: Which files to Keep, Adapt, or Delete? Since we ran helm create, you have a lot of "boilerplate." ``` 5. Keep and Adapt: deployment.yaml: Ensure the volumeMounts and volumes match the ConfigMap name above. service.yaml: Keep it to allow other pods (or the Ingress) to find the router. values.yaml: Use this for your image tags and environment-specific toggles. _helpers.tpl: Do not delete. This generates the names (like cosmo-router.fullname) used in every other file. Keep (but leave disabled): hpa.yaml: Useful later for auto-scaling, but set autoscaling.enabled: false in values.yaml for now. ingress.yaml: Keep this if you want to access the router from outside the cluster (e.g., router.example.com). Delete: httproute.yaml: This is for the "Gateway API." Unless you have a specific Gateway controller installed, standard ingress.yaml is what you'll use. NOTES.txt: Usually just contains generic text. ``` #### What ToDo after deleting the primary node that bootstrapped the cluster. ``` ssh to other k3s control plane sudo systemctl status k3s sudo systemctl stop k3s sudo k3s server --cluster-reset sudo systemctl start k3s sudo kubectl get nodes sudo nano /etc/systemd/system/k3s.service from ExecStart=/usr/local/bin/k3s server --server https://192.168.3.91:6443 --tls-san 192.168.3.156 to ExecStart=/usr/local/bin/k3s server --cluster-init --tls-san 192.168.3.156 ``` #### Delete stale nodes ``` sudo kubectl delete node invixel-vm1 sudo kubectl delete node invixel-vm6 sudo kubectl delete node invixel-vm7 sudo kubectl delete node invixel-ubuntu1 ```