k8s-infra-cluster/manuals/deploy_k8s_components.md

Getting started

kubectl apply -k . # from root project folder if not detected: kubectl get pods -n db kubectl apply -f db/yugabytedb.yaml -n db

• Find services and its ports kubectl get svc -n apps

• Port forwarding local to cosmo-router,argo-cd kubectl port-forward svc/cosmo-router 3002:3002 -n apps

kubectl port-forward svc/argocd-server -n argocd 8480:443

• Open yugabyte kubectl exec -it yb-tserver-0 -n db -- ysqlsh -h yb-tserver-0

• Open nats kubectl run nats-tools --image=natsio/nats-box -n infra --rm -it -- nats stream info request_stream --server=nats://nats-cluster:4222

  if already exist:

  kubectl exec -it nats-box -n infra -- nats stream info request_stream -s nats://nats-cluster:4222

  • list stream: kubectl exec -it nats-box -n infra -- nats -s nats://nats-cluster.infra.svc.cluster.local:4222 stream ls
  • add stream: kubectl exec -it nats-box nats -n infra -- /bin/sh

  nats -s nats://nats-cluster.infra.svc.cluster.local:4222 stream add request_stream --subjects "input_request_logs" --ack --storage file --retention limits --max-msgs=-1 --max-bytes=-1 --max-age=1y --replicas 3

  check: kubectl describe pod nats-cluster-1 -n infra

• Check benthos kubectl logs -f benthos-556f97988-cgd7f -n infra

• Redeploy For cosmo-router under namespace apps:

to apply changes:
  helm upgrade cosmo-router ./cosmo-router -n apps # from folder apps
we can not use this command to apply changes:
  kubectl rollout restart deployment cosmo-router -n apps

Install Linkerd on local machine

Linkerd exe is installed in your folder of choice, and set environment variables path to refer to this.

Some linkerd commands

Some commands: linkerd viz dashboard linkerd version --client linkerd check --linkerd-namespace apps linkerd -n apps stat deployments linkerd check --pre

Deinstalling Linkerd:

• Deleting Linkerd (depends on namespace) :

- kubectl delete deployment linkerd-destination linkerd-identity linkerd-proxy-injector -n apps
- kubectl delete deployment metrics-api prometheus tap tap-injector web -n apps
- Delete the Mutating Webhooks (CRITICAL): 
  - kubectl delete mutatingwebhookconfiguration linkerd-proxy-injector-webhook-config
  - kubectl delete mutatingwebhookconfiguration linkerd-tap-injector-webhook-config
  - kubectl delete validatingwebhookconfiguration linkerd-sp-validator-webhook-config 
- Remove the annotation from your namespace: 
  kubectl annotate namespace apps linkerd.io/inject-
- Restart your apps to strip the sidecars:
  kubectl rollout restart deployment cosmo-router -n apps
  kubectl rollout restart statefulset nats-cluster -n infra

• Deleting Linkerd using Helm installation:

# 1. Delete the current broken Linkerd installations
helm uninstall linkerd-viz -n apps
helm uninstall linkerd-control-plane -n apps
helm uninstall linkerd-crds -n apps

# 2. IMPORTANT: Delete any leftover Linkerd webhooks (these often block restarts)
kubectl delete mutatingwebhookconfiguration linkerd-proxy-injector-webhook-config
kubectl delete validatingwebhookconfiguration linkerd-sp-validator-webhook-config

• Deleting Linkerd using linkerd CLI: linkerd uninstall --namespace apps | kubectl delete -f -

Open source Helm chart

Most of the time we do not need to write k8s yaml files. If we have helm chart available from the provider we generate the files needed using helm template and refer the generated file from kustomization.yaml

Using helm chart vs kustomize

if you want to move your cluster to new VMs, you just run: To run with helm chart: kubectl apply -k .

Kubernetes will create the namespaces, then deploy NATS, then Benthos, then Yugabyte all in one go

Method	Pro	Con
helm install	Fast and easy for one-offs.	Hard to remember settings; hard to track in Git.
Kustomize + Helm	Single source of truth; everything in Git.	Requires --enable-helm flag; slightly more setup.

• Manual way with helm chart We also have 2 ways by using helm template vs helm install

helm template yugabytedb yugabytedb/yugabyte `
  --namespace db `
  --set storage.master.storageClass=longhorn `
  --set storage.tserver.storageClass=longhorn `
  --set replicas.master=3 `
  --set replicas.tserver=3 `
  --set enableLoadBalancer=false `
  --set gflags.master.max_clock_skew_usec=2000000 `
  --set gflags.tserver.max_clock_skew_usec=2000000 `
  --set gflags.master.time_source=system `
  --set gflags.tserver.time_source=system `
  --set gflags.tserver.start_pgsql_proxy=true `
  > db/yugabytedb.yaml

  OR
helm install yugabytedb yugabytedb/yugabyte `
  --namespace db `
  --set storage.master.storageClass=longhorn `
  --set storage.tserver.storageClass=longhorn `
  --set replicas.master=3 `
  --set replicas.tserver=4 `
  --set gflags.master.max_clock_skew_usec=2000000 `
  --set gflags.tserver.max_clock_skew_usec=2000000 `
  --set gflags.master.time_source=system `
  --set enableLoadBalancer=false `
  --set gflags.tserver.start_pgsql_proxy=true `
  --set gflags.tserver.time_source=system

Feature	helm template	helm install
Action	Local Only. Generates raw Kubernetes YAML and prints it	to a file.
Result	"You get a file (db/yugabytedb.yaml) that you can	inspect
Usage	Best for GitOps/Kustomize. You use the generated file as	a resource in Kustomize.

• Kustomize way but required installing standalone Kustomization binary file on windows ./kustomize build . --enable-helm | kubectl apply -f -

We do not use Kustomize here

NATS

helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm repo update

- Basic
nats-values.yaml
config:
  jetstream:
    enabled: true
    fileStore:
      pvc:
        size: 5Gi  # Reserve 5GB for JetStream

or 

- HA
# To increase replicas for High Availability
replicaCount: 3

# JetStream Clustering (The "Replica Factor")
config:
  cluster:
    enabled: true
  jetstream:
    enabled: true
    # This ensures your data survives if x-vm2 goes down
    fileStore:
      pvc:
        enabled: true
        storageClassName: "local-path" # k3s default storage
        size: 10Gi

Check:
# Create the namespace first
kubectl create namespace infra

# Install NATS using the chart and your values file
helm install nats-cluster nats/nats --namespace infra -f nats-values.yaml

or with helm template
helm template nats-cluster nats/nats `
  --namespace infra `
  --set replicaCount=3 `
  --set config.cluster.enabled=true `
  --set config.jetstream.enabled=true `
  --set config.jetstream.fileStore.pvc.enabled=true `
  --set config.jetstream.fileStore.pvc.storageClassName="longhorn" `
  --set config.jetstream.fileStore.pvc.size=10Gi `
  > infra/nats-cluster.yaml

Apply the template (without creating cluster)
kubectl apply -f infra/nats-cluster.yaml

Apply the template( with creating cluster pods, service, etc):
helm install nats-cluster nats/nats --namespace infra -f nats-values.yaml

After modification of other files you can apply the changes by:
helm upgrade nats-cluster nats/nats --namespace infra -f nats-values.yaml

Remove nats-cluster
helm uninstall nats-cluster -n infra

Remove pvc of nats-cluster (first uninstall nats-cluster using helm-uninstall)
kubectl delete pvc -n infra -l app.kubernetes.io/instance=nats-cluster

Verify:
kubectl get all -n infra
kubectl get statefulset -n infra

kubectl get pods -n infra -w
kubectl get pvc -n infra

kubectl get sc # verify longhorn is ready and status healthy

To avoid typing -n infra every time, you can switch your "active room" to infra permanently:
kubectl config set-context --current --namespace=infra


Apply changes:
helm upgrade nats-cluster nats/nats -n infra -f nats-values.yaml

Testing:
first terminal:
kubectl run nats-box --image=natsio/nats-box:latest -n infra -it --rm
nats sub -s nats-cluster test.topic
new terminal:
# Instead of 'run', we 'exec' into the pod that is already there
kubectl exec -n infra -it nats-box -- /bin/sh
nats pub -s nats-cluster test.topic "Hello from Country B!"

or 

kubectl exec -it nats-box -n infra -- nats -s nats-cluster:4222 pub input_request_logs '{"content": {"hash": "test-123"}, "message": "Hello Benthos!"}'

Delete:
kubectl delete statefulset nats-cluster -n infra
kubectl delete all -l app.kubernetes.io/instance=nats-cluster -n default

Intrepretation

kubectl get all -n infra
NAME                                    READY   STATUS    RESTARTS   AGE
pod/nats-box                            1/1     Running   0          13m
pod/nats-cluster-0                      2/2     Running   0          87s
pod/nats-cluster-1                      2/2     Running   0          87s
pod/nats-cluster-2                      2/2     Running   0          87s
pod/nats-cluster-box-868cc6c48b-dlvtn   1/1     Running   0          26m

NAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/nats-cluster            ClusterIP   10.43.208.167   <none>        4222/TCP                     26m
service/nats-cluster-headless   ClusterIP   None            <none>        4222/TCP,6222/TCP,8222/TCP   26m

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nats-cluster-box   1/1     1            1           26m

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/nats-cluster-box-868cc6c48b   1         1         1       26m

NAME                            READY   AGE
statefulset.apps/nats-cluster   3/3     88s

Interpretation:
The "Power Trio" (StatefulSet)
pod/nats-cluster-0
pod/nats-cluster-1
pod/nats-cluster-2

Why: The official Helm chart defaults to replicaCount: 3. This is for High Availability. Since you have 4 VMs, Kubernetes likely spread these across your different nodes. If one VM in Country B fails, NATS will keep running because the other two pods have a "quorum" (majority) and won't lose your data.

The "Utility" Pods (Box)
pod/nats-box: This is the temporary pod you created manually with kubectl run to test the connection. It stays there until you delete it or it finishes.

pod/nats-cluster-box-868cc6c48b-dlvtn: This was created automatically by the Helm chart. The NATS team includes a permanent "box" deployment so you always have a toolset inside the cluster to check the status of the NATS stream.

What about the "2/2" READY status?
You'll notice the nats-cluster-x pods say 2/2. This means each pod actually contains two containers:

The NATS Server: The actual engine.

The NATS Config Reloader: A "sidecar" container that watches for changes to your settings and tells the NATS server to refresh without restarting.

Rules of Engagement

Creating stream with Benthos for how NATS stores your data

? Retention Policy Limits, WorkQueue, Interest
? Discard Policy Old vs New  
In NATS, -1 means "Infinite" or "No Limit."
? Stream Messages Limit -1
? Per Subject Messages Limit -1
? Total Stream Size -1
? Message TTL -1  
? Max Message Size -1
? Duplicate tracking time window 2m0s
? Allow message Roll-ups Yes
? Allow message deletion Yes
? Allow purging subjects or the entire stream (Y/n)

nats -s nats-cluster:4222 stream add request_stream --subjects "input_request_logs" --ack --storage file --retention limits --max-msgs=-1 --max-bytes=-1 --max-age=1y --replicas 3

nats -s nats-cluster:4222 stream add transaction_stream --subjects "input_transaction_logs" --ack --storage file --replicas 3

Benthos

# Running namespaces
kubectl apply -f namespaces.yaml # run this command from the root of your repo

# Running Kustomization Deploy configs
kubectl apply -k . -n infra # from infra\benthos folder
kubectl apply -k .\benthos\ -n infra # from infra folder

# Manual creation of config map
kubectl create configmap benthos-config --from-file=./benthos-configs/

# Verify ConfigMaps
Check if your files were successfully "packed" into Kubernetes:
kubectl get configmap -n infra


# To see the actual content of your streams inside K8s:
kubectl describe configmap benthos-streams -n infra
# Verify the 3 Replicas
kubectl get pods -n infra -l app=benthos
# Verify the Dashboard (Port-Forward) To see the Benthos UI on your laptop:
kubectl port-forward svc/benthos-ui -n infra 4195:4195

# Peek inside the running Pod:
kubectl exec -it <pod-name> -n infra -- ls /configs/streams
kubectl exec -it <pod-name> -n infra -- ls -R /configs
# Verify
kubectl get svc -n infra.
# Create Stream:
kubectl exec -n infra -it nats-box -- /bin/sh

Debugging

kubectl describe pod benthos-68cb959d58-47prd -n infra

kubectl logs benthos-86f5c886b7-5fr5t -n infra

If needed delete only deployment and redeploy using kustomization 
Depends on the your position
kubectl delete deployment benthos -n infra
kubectl apply -k .\infra\benthos\ -n infra 
or only without namespaces
kubectl apply -k .\infra\benthos\ 
or only (if you are inside \infra\benthos\)
kubectl apply -k . 

# You must not delete configmap since it get refreshed everytime you changed. Optional:

kubectl delete configmap --all -n infra (Only if only Benthos is in infra)
# Check output after redeploy
kubectl get pod -n infra -o yaml
kubectl kustomize .

Installing Longhorn on each VMs

• Prerequisite You need to run the following command on all 4 VMs (Master and all Workers) to ensure Longhorn can communicate with the disks:

sudo apt install open-iscsi nfs-common util-linux -y
sudo systemctl enable --now iscsid

Why do we need this? Longhorn creates "Block Devices." To do that, the Linux kernel on your VM needs open-iscsi to "attach" to the virtual disks Longhorn creates. Without this, your pods will be stuck in ContainerCreating forever


1. Add the repo
helm repo add longhorn https://charts.longhorn.io
helm repo update

2. Install into a new namespace

Longhorn is a "System Service" not Application Infrastructure, requires high privileges
helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace

- Check : kubectl get pods -n longhorn-system

3. Running the StorageClass Patch from Local Windows Laptop
Once installed, tell K3s to use Longhorn for every database deployed:

kubectl patch storageclass longhorn -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

or 
kubectl patch storageclass longhorn -p "{\"metadata\": {\"annotations\":{\"storageclass.kubernetes.io/is-default-class\":\"true\"}}}" 

or
  Run kubectl edit storageclass longhorn

  Look for the metadata: section.

  Add this line under annotations:
  storageclass.kubernetes.io/is-default-class: "true"

  Save and exit.

- Notes
Because you have 4 VMs, Longhorn will default to 3 replicas for your data. This is perfect. It means:

Data is written to VM1.

Longhorn clones it to VM2 and VM3.

VM4 stays as a "spare" or handles other data.

If any one VM catches fire, your data is still 100% safe and available.


4. UI Longhorn:
A. Update your Windows Hosts file: 
- Open Notepad (Make sure you have right to write)

- Open C:\Windows\System32\drivers\etc\hosts.

- Add the IP of your Master VM:
192.168.x.x  longhorn.local (Replace with your actual VM IP)

B. Create longhorn.ingress.yaml and run:
/country-b-cluster-ops
  /system             <-- NEW FOLDER for cluster-wide tools
    longhorn-ingress.yaml
  /infra              <-- For your messaging/processing (NATS, Benthos)
  /db                 <-- For your databases (Yugabyte)
kubectl apply -f system/longhorn-ingress.yaml

C. Open http://longhorn.local from your laptop

Additional Configuration?

There is one common issue with K3s and Longhorn. K3s stores its data in /var/lib/rancher/k3s/storage, but Longhorn defaults to /var/lib/longhorn.

Recommendation: If you have a specific large hard drive or partition on your VMs where you want the data to live, you can configure that in the Longhorn UI. If you just have one big / partition, the default is fine.

Yugabyte

helm repo add yugabytedb https://charts.yugabyte.com
helm repo update
helm install yugabytedb yugabytedb/yugabyte `
  --namespace db `
  --set storage.master.storageClass=longhorn `
  --set storage.tserver.storageClass=longhorn `
  --set replicas.master=3 `
  --set replicas.tserver=4 `
  --set gflags.master.max_clock_skew_usec=2000000 `
  --set gflags.tserver.max_clock_skew_usec=2000000 `
  --set gflags.master.time_source=system `
  --set enableLoadBalancer=false `
  --set gflags.tserver.start_pgsql_proxy=true `
  --set gflags.tserver.time_source=system


or 

helm template yugabytedb yugabytedb/yugabyte `
  --namespace db `
  --set storage.master.storageClass=longhorn `
  --set storage.tserver.storageClass=longhorn `
  --set replicas.master=3 `
  --set replicas.tserver=4 `
  --set gflags.master.max_clock_skew_usec=2000000 `
  --set gflags.tserver.max_clock_skew_usec=2000000 `
  --set gflags.master.time_source=system `
  --set enableLoadBalancer=false `
  --set gflags.tserver.start_pgsql_proxy=true `
  --set gflags.tserver.time_source=system > yugabytedb.yaml

• set gflags.tserver.start_pgsql_proxy=true? This flag tells the Yugabyte T-Server to turn on the YSQL API layer

• Fully Qualified Domain Name FQDN for communication

  • yugabytedb.db.svc.cluster.local

  • The pattern is: [service-name].[namespace].svc.cluster.local

Optimal Node Distribution for 4 VMs

For a 4-node cluster, you have enough overhead to ensure High Availability (HA) while maximizing resource use.

Master Count: 3

Why: Yugabyte Masters use the Raft consensus algorithm. You need an odd number to avoid "split-brain" scenarios. 3 masters can tolerate 1 node failure. Since you have 4 VMs, you should stick with 3.

T-Server Count: 3 or 4

Option 3 (Balanced): Keep 3 T-Servers. This leaves 1 VM entirely free for Benthos, NATS, and Redpanda.

Option 4 (High Performance): Run 4 T-Servers. This spreads your data shards across all available hardware.

Recommendation: Go with 3 Masters and 3 T-Servers. This keeps the "Replication Factor" (RF) at 3, which is the standard. Adding a 4th T-Server with RF3 is possible, but 3/3 is more predictable for a small cluster.

Linkerd, Viz, openssl on k8s

1. Create the Trust Anchor (Root CA)
# Generate private key for the Root
a.openssl ecparam -name prime256v1 -genkey -noout -out ca.key
b.openssl ecparam -name prime256v1 -genkey -noout -out ca.key
# Generate the self-signed Root Certificate (Valid for 10 years)
a.openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 \
  -out ca.crt -subj "/CN=root.linkerd.cluster.local" \
  -addext "basicConstraints=critical,CA:TRUE"
b.openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.crt -subj "/CN=root.linkerd.cluster.local" -addext "basicConstraints=critical,CA:TRUE"

2. Create the Identity Issuer (Intermediate CA)
# Generate private key for the Issuer
a/b. openssl ecparam -name prime256v1 -genkey -noout -out issuer.key

# Create a CSR (Certificate Signing Request)
a/b. openssl req -new -key issuer.key -out issuer.csr -subj "/CN=identity.linkerd.cluster.local"

# Create a config file for the intermediate CA extensions
a. cat > issuer.ext <<EOF
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,digitalSignature,keyCertSign,cRLSign
EOF

# Sign the Issuer certificate with your Root CA (Valid for 1 year)
a. openssl x509 -req -in issuer.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out issuer.crt -days 365 -sha256 -extfile issuer.ext
b.# Sign Issuer (Critical: must have CA:TRUE)
echo "basicConstraints=critical,CA:TRUE,pathlen:0" > ext.txt
echo "keyUsage=critical,digitalSignature,keyCertSign,cRLSign" >> ext.txt
openssl x509 -req -in issuer.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out issuer.crt -days 365 -sha256 -extfile ext.txt

3. Generate k3s Manifests
helm repo add linkerd https://helm.linkerd.io/stable && helm repo update

4. Generate CRDs
Linkerd Custom Resource Definitions must be installed first.
helm template linkerd-crds linkerd/linkerd-crds --namespace linkerd > linkerd-crds.yaml

5. Generate Control Plane
We will bake your certificates directly into the generated YAML:

helm template linkerd-control-plane linkerd/linkerd-control-plane --namespace linkerd --set-file identityTrustAnchorsPEM=ca.crt --set-file identity.issuer.tls.crtPEM=issuer.crt --set-file identity.issuer.tls.keyPEM=issuer.key --set identity.issuer.scheme=kubernetes.io/tls > linkerd-control-plane.yaml

or

helm install linkerd-control-plane linkerd/linkerd-control-plane -n linkerd --set-file identityTrustAnchorsPEM=ca.crt --set-file identity.issuer.tls.crtPEM=issuer.crt --set-file identity.issuer.tls.keyPEM=issuer.key

6. Generate Viz (Dashboard)
Default namespaces are linkerd and linkerd-viz but we save them under our namespace apps

a. helm template linkerd-viz linkerd/linkerd-viz --namespace linkerd-viz > linkerd-viz.yaml (standard namespace)

b. helm template linkerd-viz linkerd/linkerd-viz
--namespace apps
--set linkerdNamespace=apps
--set tap.namespace=apps
--set dashboard.namespace=apps
--set prometheus.namespace=apps > linkerd-viz.yaml

or using c. helm install linkerd-viz linkerd/linkerd-viz -n linkerd-viz --create-namespace

Uninstalling: helm ls -n apps

helm uninstall linkerd-viz -n apps helm uninstall linkerd-control-plane -n apps helm uninstall linkerd-crds -n apps

7. Creating k8s Secret in our namespace
the pod is looking for secret in the apps namespace before the pod can start:
kubectl create secret tls linkerd-identity-issuer --cert=issuer.crt  --key=issuer.key  --namespace=linkerd
or 
kubectl create secret tls linkerd-identity-issuer --cert=issuer.crt  --key=issuer.key  --namespace=apps

8.. Apply CRDs first
kubectl apply -f linkerd-crds.yaml 

9. Apply Control Plane
kubectl apply -f linkerd-control-plane.yaml 
10. Apply Viz
kubectl apply -f linkerd-viz.yaml 

11. Activating the Connection (Injection)
kubectl annotate namespace apps linkerd.io/inject=enabled

12. Check pods status

kubectl get pods -n apps
Redeploy if error

linkerd-destination-6b69957545-9b9tz      0/4     CrashLoopBackOff   720 (2m23s ago)   12h
linkerd-destination-74cc587f5-gvgqd       0/4     CrashLoopBackOff   712 (50s ago)     12h
linkerd-identity-68858f6c75-t2lk9         2/2     Running            0                 14h
linkerd-proxy-injector-54bc495b55-sgk6l   0/2     CrashLoopBackOff   309 (2m23s ago)   12h
linkerd-proxy-injector-584f78bf9-86r64    0/2     CrashLoopBackOff   306 (60s ago)     12h

- destination: The main service discovery logic.
-policy: Handles authorization policies.
-linkerd-proxy: The "sidecar" that handles the pod's own traffic.
-linkerd-init: (Finished) Sets up network rules.

13.Rollout Restart:
For existing services (like Yugabyte or your Python subgraph) to join the mesh, they need to be restarted:

kubectl rollout restart deployment cosmo-router -n apps
kubectl rollout restart statefulset yugabytedb -n db

12. How to Verify
To verify the connection and see your Yugabyte/Redpanda traffic:

Check Status: linkerd check (Requires Linkerd CLI: curl -sL https://run.linkerd.io/install | sh).

Open Dashboard: linkerd viz dashboard.

Check mTLS: In the dashboard, look for the shield icon next to your cosmo-router to subgraph traffic. This confirms your OpenSSL certificates are working.

Debugging:

kubectl describe pod -n apps -l linkerd.io/control-plane-component=identity
openssl x509 -in ca.crt -text -noout
kubectl logs -n apps deployment/linkerd-identity

Linkerd Activation / Deactivation

# Run these commands to tell Linkerd that these namespaces are now part of the mesh:
kubectl annotate namespace apps linkerd.io/inject=enabled
kubectl annotate namespace infra linkerd.io/inject=enabled
kubectl annotate namespace db linkerd.io/inject=enabled
kubectl annotate namespace stream linkerd.io/inject=enabled

# Restart everything in those namespaces
kubectl rollout restart deployment -n apps
kubectl rollout restart deployment -n infra
kubectl rollout restart statefulset -n infra  # For your NATS cluster
kubectl rollout restart statefulset -n db     # For your Yugabyte/DB
kubectl rollout restart deployment -n stream

# Remove the annotation to deactivate
kubectl annotate namespace apps linkerd.io/inject-
kubectl annotate namespace infra linkerd.io/inject-
kubectl annotate namespace db linkerd.io/inject-
kubectl annotate namespace stream linkerd.io/inject-

# Clean the pods
kubectl rollout restart deployment -n apps
kubectl rollout restart deployment -n infra
kubectl rollout restart statefulset -n infra
kubectl rollout restart statefulset -n db

Cosmo router

1. Check
cosmo-router/
├── config.yaml          <-- Your router config
├── router.json          <-- Your composed graph
├── values.yaml          <-- Updated with Ingress and Middleware annotations
└── templates/
    ├── _helpers.tpl <-- Required for names
    ├── configmap.yaml  <-- Uses .Files.Get(our custom config)
    ├── deployment.yaml <-- Incl Checksum & Linkerd injection (The brain)
    ├── service.yaml (The external network)
    ├── ingress.yaml (The internal network)
    └── middleware.yaml  <-- The Gzip logic
    └── hpa.yaml  optional
    
Delete: 
a. httproute.yaml (Causes the current error)
b.serviceaccount.yaml (Caused the previous error)
c. tests/ (The default test folder often causes similar issues)

2. Test it
helm template cosmo-router ./cosmo-router

3. Run cosmo-router
# Deploy it
helm install cosmo-router ./cosmo-router -n apps

or Redeploy
helm upgrade cosmo-router ./cosmo-router -n apps

or If needed uninstall
helm uninstall linkerd-crds -n apps
helm uninstall cosmo-router -n apps

Some usefult commands:
kubectl delete pods -n apps -l linkerd.io/extension=viz
kubectl annotate namespace apps linkerd.io/inject=enables
kubectl rollout restart deployment cosmo-router -n apps
kubectl logs -n apps -l app.kubernetes.io/name=cosmo-router

Helm chart: Which files to Keep, Adapt, or Delete?

Since we ran helm create, you have a lot of "boilerplate."

5. Keep and Adapt:
deployment.yaml: Ensure the volumeMounts and volumes match the ConfigMap name above.

service.yaml: Keep it to allow other pods (or the Ingress) to find the router.

values.yaml: Use this for your image tags and environment-specific toggles.

_helpers.tpl: Do not delete. This generates the names (like cosmo-router.fullname) used in every other file.

Keep (but leave disabled):
hpa.yaml: Useful later for auto-scaling, but set autoscaling.enabled: false in values.yaml for now.

ingress.yaml: Keep this if you want to access the router from outside the cluster (e.g., router.example.com).

Delete:
httproute.yaml: This is for the "Gateway API." Unless you have a specific Gateway controller installed, standard ingress.yaml is what you'll use.

NOTES.txt: Usually just contains generic text.

What ToDo after deleting the primary node that bootstrapped the cluster.

ssh to other k3s control plane

sudo systemctl status k3s
sudo systemctl stop k3s
sudo k3s server --cluster-reset
sudo systemctl start k3s
sudo kubectl get nodes

sudo nano /etc/systemd/system/k3s.service

from
ExecStart=/usr/local/bin/k3s server --server https://192.168.3.91:6443 --tls-san 192.168.3.156

to 
ExecStart=/usr/local/bin/k3s server --cluster-init --tls-san 192.168.3.156

Delete stale nodes

sudo kubectl delete node invixel-vm1
sudo kubectl delete node invixel-vm6
sudo kubectl delete node invixel-vm7
sudo kubectl delete node invixel-ubuntu1