``` kubectl run nats-box--image=natsio/nats-box -n infra --rm --overrides='{\"metadata\": {\"annotations\": {\"linkerd.io/inject\": \"enabled\"}}}' --stdin --tty --nats -s nats://nats-cluster:4222 stream add request_stream --subjects \"input_request_logs\" --ack --storage file --retention limits --max-msgs=-1 --max-bytes=-1 --max-age=1y --replicas 3 kubectl run nats-box --image=natsio/nats-box -n infra --rm -it --overrides='{\"metadata\": {\"annotations\": {\"linkerd.io/inject\": \"enabled\"}}}' -- nats stream info request_stream -s nats://nats-cluster:4222 (kubectl run nats-box --image=natsio/nats-box -n infra --dry-run=client -o yaml -- nats -s nats://nats-cluster:4222 stream add request_stream --subjects "input_request_logs" --ack --storage file --retention limits --max-msgs=-1 --max-bytes=-1 --max-age=1y --replicas 3 | linkerd inject - | kubectl apply -f -) Listen to topics: kubectl exec -it nats-box -n infra -- nats -s nats://nats-cluster:4222 sub "input_request_logs" --stream request_stream To view existing messages: (nats) stream view request_stream --subject input_request_logs. To check consumer status: (nats) consumer info request_stream . Info of clusters kubectl exec -it nats-cluster-0 -n infra -c nats -- /bin/sh # Exec into the FIRST pod (0) kubectl exec -it nats-box -n infra -- nats -s nats://nats-cluster:4222 stream ls kubectl exec -it nats-box -n infra -- nats -s nats://nats-cluster.infra.svc.cluster.local:4222 stream ls kubectl exec -it nats-box -n infra -- nats -s nats://nats-cluster:4222 stream info request_stream kubectl exec -it nats-cluster-0 -n infra -c nats -- netstat -tuln kubectl get pod nats-cluster-0 -n infra -o wide kubectl get svc --all-namespaces kubectl get endpoints nats-cluster -n infra kubectl exec -it nats-box -n infra -- nats stream add request_stream --subjects "input_request_logs" --ack --storage file --retention limits --max-msgs=-1 --max-bytes=-1 --max-age=1y --replicas 3 --server nats://nats-cluster:4222 Apply modification: helm upgrade nats-cluster nats/nats --namespace infra -f nats-values.yaml Some commands: kubectl get pods -n infra --show-labels kubectl delete pod nats-cluster-0 nats-cluster-1 nats-cluster-2 -n infra kubectl get volumes.longhorn.io -n longhorn-system kubectl get sc # for longhorn kubectl describe pod -n apps cosmo-router-675c8666cb-cmt82 kubectl logs -n apps -l linkerd.io/control-plane-component=identity kubectl logs -n apps -l app.kubernetes.io/name=cosmo-router kubectl delete pods -n apps -l linkerd.io/extension=viz kubectl delete all -l app.kubernetes.io/instance=nats-cluster -n default kubectl delete pod nats-box -n infra --force kubectl delete pvc -n infra -l app.kubernetes.io/instance=nats-cluster Activate linkerd: kubectl annotate namespace apps linkerd.io/inject=enabled kubectl annotate namespace infra linkerd.io/inject=enabled kubectl annotate namespace db linkerd.io/inject=enabled kubectl annotate namespace stream linkerd.io/inject=enabled Inject into pods: kubectl rollout restart deployment cosmo-router -n apps # only injects to cosmo-router # Restart everything in those namespaces kubectl rollout restart deployment -n apps kubectl rollout restart deployment -n infra kubectl rollout restart statefulset -n infra # For your NATS cluster kubectl rollout restart statefulset -n db # For your Yugabyte/DB kubectl rollout restart deployment -n stream Example: After updating benthos streams or templates kubectl annotate pod -l app=benthos linkerd.io/inject=disabled --overwrite -n infra kubectl rollout restart deployment benthos -n infra Deactivate linkerd: kubectl annotate namespace apps linkerd.io/inject- kubectl annotate namespace infra linkerd.io/inject- kubectl annotate namespace db linkerd.io/inject- kubectl annotate namespace stream linkerd.io/inject- Clean the pods: kubectl rollout restart deployment -n apps kubectl rollout restart deployment -n infra kubectl rollout restart statefulset -n infra kubectl rollout restart statefulset -n db Check which namespaces/pods are set for the mesh: kubectl get ns -o custom-columns=NAME:.metadata.name,INJECT:.metadata.annotations."linkerd\.io/inject" kubectl get pods -A -o custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,MESHED:.metadata.annotations."linkerd\.io/inject" kubectl get namespaces -L linkerd.io/inject kubectl get pod -n apps -o yaml | grep "linkerd.io/inject" kubectl get pods -A -l linkerd.io/control-plane-ns Linkerd commands: linkerd check # Ensure the Linkerd system itself is healthy. linkerd check --linkerd-namespace apps linkerd -n apps check --proxy linkerd stat ns linkerd viz dashboard # View the dashboard in your browser linkerd viz stat namespace apps Check stats for your specific namespace linkerd viz edge -n apps # See real-time traffic between Cosmo-Router and NATS linkerd viz tap -n infra deploy/benthos linkerd viz stat -n apps deploy/cosmo-router linkerd viz tap deployment/benthos -n infra --to deployment/nats-cluster # Run this to see if the Linkerd proxies are actually passing traffic: Important: The "Skip" Configuration metadata: annotations: linkerd.io/inject: disabled kubectl exec -it deployment/benthos -n infra -- env | findstr ECOM_PASS kubectl logs deployment/benthos -n infra | findstr "sql_poller" first run kubectl exec -it nats-box-temp nats -n infra -- /bin/sh Then the followings: nats str view fact_stream -s nats://nats-cluster:4222 nats consumer next fact_stream my_consumer_name -s nats://nats-cluster:4222 nats sub "output_request_logs" -s nats://nats-cluster:4222 nats stream ls -s nats://nats-cluster:4222 nats stream info fact_stream -s nats://nats-cluster:4222 # Check if other pods on that node can reach the internet kubectl run net-test --image=alpine --restart=Never -n infra -- nslookup google.com kubectl exec -it yb-tserver-0 -n db -- ysqlsh -h kubectl port-forward svc/cosmo-router 3002:3002 -n apps kubectl port-forward svc/postgres-ha-rw 5436:5432 -n db # (Best Practice für Zero-Downtime): Evakuiere die alten Nodes zuerst manuell (Drain): Für die alten dbworker und nworker ausführen: kubectl drain --ignore-daemonsets --delete-emptydir-data ``` #### The Difference: NATS Core vs. JetStream - NATS Core (nats sub): This is "Fire and Forget." If no one is listening when Benthos sends the message, the message disappears forever. - NATS JetStream: This is a Stream. It saves the messages to disk. To get data from a stream, you don't use sub; you use a Consumer.