### Getting Started #### Join new nodes ``` # retrieve token on control-plane node sudo cat /var/lib/rancher/k3s/server/node-token K10094c77f9748b705ce6fd128b916dd96793160853577fdcba986c57de9b6f4a3c::server:b5743ed19b035f78f59607b2fa3bda2e # join via curl -sfL https://get.k3s.io | K3S_URL=https://:6443 K3S_TOKEN= sh - K3S_URL: The address of your control plane node curl -sfL https://get.k3s.io | K3S_URL=https://192.168.3.91:6443 K3S_TOKEN=K10094c77f9748b705ce6fd128b916dd96793160853577fdcba986c57de9b6f4a3c::server:b5743ed19b035f78f59607b2fa3bda2e sh - ``` #### 1. Using ansible ``` a. ansible-playbook -i hosts.ini ansible-prepare-k3s.yml if not working: export ANSIBLE_CONFIG=./ansible.cfg sed -i 's/\r$//' hosts.ini b. copy content of file from control-plane (sudo cat /etc/rancher/k3s/k3s.yaml) to config inside C:\Users\wendg2\.kube c. kustomize build --enable-helm . | kubectl apply --server-side --force-conflicts -f - d. ansible-playbook -i hosts.ini deploy-k3s.yml ``` #### 2. Install kustomize - Download version > v4.x fromhttps://github.com/kubernetes-sigs/kustomize/releases and set the path in your environment variables - Test kustomize #### 3. Setup k3s remote control export ANSIBLE_CONFIG=./ansible.cfg ssh-keygen -R 192.168.3.91 # in PS ssh-keygen -f '/home/josh/.ssh/known_hosts' -R '192.168.3.91' # in wsl #### How Ansible finds its Config (The Hierarchy) Ansible looks for its configuration in a very specific order. It stops at the first one it finds: ``` 1. ANSIBLE_CONFIG (The environment variable): This is the "Nuclear Option." It overrides everything else. 2. ansible.cfg (In the current directory): This is what you were trying to use, but it was being ignored due to the permissions. 3. ~/.ansible.cfg (In your home directory): Personal user settings. 4. /etc/ansible/ansible.cfg: The global system default ``` #### 4. using kustomize and helm together - For yugabyte there must be enough space kustomize build . --enable-helm | kubectl apply -f - kustomize build . | kubectl apply --server-side --force-conflicts -f - # from root folder kustomize build . | kubectl apply --server-side --force-conflicts -f - # from some folder for longhorn-system, argocd, linkerd, etc - For postgres we use cnpg operator ``` We must run twice: kustomize build . --enable-helm | kubectl apply -f - kubectl get pods -n cnpg-system kubectl get cluster -n db -w kubectl get pods -n db ``` #### 5.Start our apps ``` a. kubectl port-forward svc/cosmo-router 3002:3002 -n apps b. kubectl run nats-box-temp --image=natsio/nats-box -n infra --rm -it -- nats -s nats://nats-cluster:4222 sub "input_request_logs" c. kubectl exec -it nats-box nats -n infra -- /bin/sh nats -s nats://nats-cluster.infra.svc.cluster.local:4222 stream add request_stream --subjects "input_request_logs" --ack --storage file --retention limits --max-msgs=-1 --max-bytes=-1 - -max-age=1y --replicas 3 FQDN with pattern as `[service-name].[namespace].svc.cluster.local` - find service name with : kubectl get svc -n db - find service name is to be found through labels : service-name for nats-cluster from app.kubernetes.io/instance nats-cluster in labels app.kubernetes.io/instance=nats-cluster d. kubectl logs -f benthos-6ff4b9dfb5-7klqr -n infra ``` #### Apply Changes - After modifying Benthos ( directly from benthos folder ): ``` kustomize build . | kubectl apply -f - or kubectl apply -f benthos-deployment.yaml ``` - After modifying cosmo router: helm upgrade cosmo-router . -n apps #### Find secret kubectl exec -it deployment/benthos -n infra -- env | findstr ECOM_PASS kubectl get secret postgres-ha-app -n infra kubectl get secrets -n infra kubectl get secret postgres-ha-app -n db -o jsonpath='{.data}' #### Create secret #### Force-copy secrets 1. This copies the secret so the 'infra' namespace can see it kubectl get secret postgres-ha-app -n db -o yaml | %{ $_ -replace "namespace: db", "namespace: infra" } | kubectl apply -f - 2. force-copy from db to infra and clean up the metadata so Kubernetes accepts them as "new" secrets in the infrastructure namespace. ``` Method 2a.Copy the app secret to the infra namespace $secret = kubectl get secret postgres-ha-app -n db -o json | ConvertFrom-Json $secret.metadata.psobject.Properties.Remove('namespace') $secret.metadata.psobject.Properties.Remove('resourceVersion') $secret.metadata.psobject.Properties.Remove('uid') $secret | ConvertTo-Json | kubectl apply -n infra -f - Method 2b. Copy the app secret from 'db' to 'infra' kubectl get secret postgres-ha-app --namespace=db -o yaml | ` ForEach-Object { $_ -replace 'namespace: db', 'namespace: infra' } | ` kubectl apply -f - # Copy the superuser secret from 'db' to 'infra' kubectl get secret postgres-ha-superuser --namespace=db -o yaml | ` ForEach-Object { $_ -replace 'namespace: db', 'namespace: infra' } | ` kubectl apply -f - or Method 2c.Export from DB namespace, strip IDs, and apply to INFRA namespace kubectl get secret postgres-ha-app -n db -o yaml | ` Select-String -Pattern "ownerReferences|resourceVersion|uid|creationTimestamp|namespace: db" -NotMatch | ` Out-String | ForEach-Object { $_ + "`n namespace: infra" } | kubectl apply -f - kubectl get secret postgres-ha-superuser -n db -o yaml | ` Select-String -Pattern "ownerReferences|resourceVersion|uid|creationTimestamp|namespace: db" -NotMatch | ` Out-String | ForEach-Object { $_ + "`n namespace: infra" } | kubectl apply -f - or Mehod 2d. Copy the app secret kubectl get secret postgres-ha-app -n db -o json | jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid)' | kubectl apply -n infra -f - # Copy the superuser secret kubectl get secret postgres-ha-superuser -n db -o json | jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid)' | kubectl apply -n infra -f - ``` #### Avoiding sudo - Unable to read Error while executing kubectl command option ``` # Option 1: Change permissions (allows your user to read) sudo chmod 644 /etc/rancher/k3s/k3s.yaml # Option 2: Copy the config to your user's directory mkdir -p ~/.kube sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config sudo chown $(id -u):$(id -g) ~/.kube/config chmod 600 ~/.kube/config ``` #### Debug ``` kubectl get all -A kubectl describe pod yb-master-0 -n db kubectl cluster-info kubectl get cluster -n db -w # cnpg operator kubectl get nodes.longhorn.io -n longhorn-system kubectl get pods -n longhorn-system -o wide kubectl get svc -n db kubectl logs -f deployment/benthos -n infra --all-containers NATS: kubectl exec -it nats-box-temp nats -n infra -- /bin/sh nats -s nats://nats-cluster:4222 sub "output_request_logs" ``` #### Delete Mostly for db use all and -l label kubectl get pods -n db --show-labels ``` From the related directory: kustomize build . | kubectl delete -f - kubectl delete all -l app=yb-master -n db kubectl delete all -l app=yb-tserver -n db kubectl delete all -l cnpg.io/cluster=postgres-ha -n db For shrinking volumes in postgres: kubectl delete cluster postgres-ha -n db kubectl delete pvc --all -n db ``` #### Known error/Bugs ``` 1. In wsl Running helm command or kubectl Error: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp 127.0.0.1:8080: a. Create the directory if it doesn't exist mkdir -p ~/.kube b. Link your Windows kubeconfig to your WSL home directory ln -s /mnt/c/Users//.kube/config ~/.kube/config This fix got executed in deploy-k3s 2. with httproutes.gateway.networking.k8s.io kubectl delete crd httproutes.gateway.networking.k8s.io kubectl edit crd httproutes.gateway.networking.k8s.io (add v1 back into the spec.versions list) kubectl get crd | grep gateway.networking.k8s.io 3. pods exist in wsl but not in PS sudo chmod 644 /etc/rancher/k3s/k3s.yaml Run in PS kustomize build . --enable-helm | kubectl apply --server-side --force-conflicts -f - 4. pods exist in PS but not in wsl -- Join/Query from Laptop to k3s servers # On WSL mkdir -p ~/.kube sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config sudo chown $(id -u):$(id -g) ~/.kube/config export KUBECONFIG=~/.kube/config change IP manually from 127.0.0.1 to 192.168.3.91 or mkdir -p ~/.kube scp master-1@192.168.3.91:/etc/rancher/k3s/k3s.yaml ~/.kube/config sed -i 's/127.0.0.1/192.168.3.91/g' ~/.kube/config chmod 600 ~/.kube/config # Testen kubectl get nodes # on PS ssh master@192.168.3.156 "sudo cat /etc/rancher/k3s/k3s.yaml" > "$HOME\.kube\config" # Localhost-IP durch die echte Master-IP ersetzen (Get-Content "$HOME\.kube\config") -replace '127.0.0.1', '192.168.3.156' | Set-Content "$HOME\.kube\config" # Testen kubectl get nodes ``` #### Login to db ``` Getting user name $userBase64 = kubectl get secret postgres-ha-app -n db -o jsonpath='{.data.username}' [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($userBase64)) Getting password for user invixel_admin: For Powershell - $pass = kubectl get secret postgres-ha-app -n db -o jsonpath='{.data.password}' - [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($pass)) Ours: 0lkzPxlwj6JVOXwwoLYROZJsONJoPK3MtrqkxnH3iaXUs0gFg0WL78RxyDdB86Sk Getting password for super user $pass = kubectl get secret postgres-ha-superuser -n db -o jsonpath='{.data.password}' [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($pass)) Ours: CuUwr9dPXJibVFygh7oGastvIJb4syMZXKljsf0dbEl91TwuYLqvEW35hN98ytKe 1. From inside the cluster (Quick Check) kubectl exec -it postgres-ha-1 -n db -- psql -U postgres 2. From windows machine: kubectl port-forward svc/postgres-ha-rw -n db 5432:5432 Open your DB Tool and connect to: Host: localhost Port: 5432 User: postgres Password: (The one you decoded in step 2) ``` #### Primary Key & Sequential SELECT increment_by, cache_size, last_value FROM pg_sequences WHERE schemaname = 'public' AND sequencename = 'request_logs_id_seq'; ALTER SEQUENCE request_logs_id_seq INCREMENT BY 1; #### Alternative using ansible --- - name: Deploy Longhorn to k3s Cluster hosts: master become: yes vars: # Path to k3s config on the master node kubeconfig: /etc/rancher/k3s/k3s.yaml longhorn_namespace: longhorn-system tasks: - name: Add Longhorn Helm repo kubernetes.core.helm_repository: name: longhorn repo_url: "https://charts.longhorn.io" - name: Create Longhorn namespace kubernetes.core.k8s: name: "{{ longhorn_namespace }}" kind: Namespace state: present kubeconfig: "{{ kubeconfig }}" - name: Install Longhorn via Helm kubernetes.core.helm: name: longhorn chart_ref: longhorn/longhorn release_namespace: "{{ longhorn_namespace }}" kubeconfig: "{{ kubeconfig }}" wait: yes # Longhorn requires open-iscsi and nfs-client on nodes # Ensure those are installed on your workers first! - name: Set Longhorn as default StorageClass kubernetes.core.k8s_json_patch: kind: StorageClass name: longhorn kubeconfig: "{{ kubeconfig }}" patch: - op: add path: /metadata/annotations/storageclass.kubernetes.io~1is-default-class value: "true"