admin/k8s-infra-cluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
k8s-infra-cluster/manuals.md
wendi gunawan 4119e44d83 setup k3s cluster
2026-03-25 11:46:39 +01:00

9.5 KiB
Raw Blame History

Getting Started

1. Using ansible

a.  ansible-playbook -i hosts.ini ansible-prepare-k3s.yml
if not working:
export ANSIBLE_CONFIG=./ansible.cfg
sed -i 's/\r$//' hosts.ini

b. copy content of file from control-plane (sudo cat /etc/rancher/k3s/k3s.yaml) to config inside C:\Users\wendg2\.kube 
c. kustomize build --enable-helm . | kubectl apply --server-side --force-conflicts -f -
d. ansible-playbook -i hosts.ini deploy-k3s.yml

2. Install kustomize

3. Setup k3s remote control

export ANSIBLE_CONFIG=./ansible.cfg ssh-keygen -R 192.168.3.91 # in PS ssh-keygen -f '/home/josh/.ssh/known_hosts' -R '192.168.3.91' # in wsl

How Ansible finds its Config (The Hierarchy)

Ansible looks for its configuration in a very specific order. It stops at the first one it finds:

1. ANSIBLE_CONFIG (The environment variable): This is the "Nuclear Option." It overrides everything else.

2. ansible.cfg (In the current directory): This is what you were trying to use, but it was being ignored due to the permissions.

3. ~/.ansible.cfg (In your home directory): Personal user settings.

4. /etc/ansible/ansible.cfg: The global system default

4. using kustomize and helm together

  • For yugabyte there must be enough space kustomize build . --enable-helm | kubectl apply -f - kustomize build . | kubectl apply --server-side --force-conflicts -f - # from root folder kustomize build . | kubectl apply --server-side --force-conflicts -f - # from some folder

for longhorn-system, argocd, linkerd, etc

  • For postgres we use cnpg operator 
    We must run twice: 
kustomize build . --enable-helm | kubectl apply -f -
kubectl get pods -n cnpg-system
kubectl get cluster -n db -w
kubectl get pods -n db

5.Start our apps

a. kubectl port-forward svc/cosmo-router 3002:3002 -n apps
b. kubectl run nats-box-temp --image=natsio/nats-box -n infra --rm -it -- nats -s nats://nats-cluster:4222 sub "input_request_logs"
c. kubectl exec -it nats-box nats -n infra -- /bin/sh

nats -s nats://nats-cluster.infra.svc.cluster.local:4222 stream add request_stream --subjects "input_request_logs" --ack --storage file --retention limits --max-msgs=-1 --max-bytes=-1 -
-max-age=1y --replicas 3

FQDN with pattern as `[service-name].[namespace].svc.cluster.local`
- find service name  with : kubectl get svc -n db
- find service name is to be found through labels :
service-name for nats-cluster from app.kubernetes.io/instance
nats-cluster in labels app.kubernetes.io/instance=nats-cluster
d. kubectl logs -f benthos-6ff4b9dfb5-7klqr -n infra

Apply Changes

  • After modifying Benthos ( directly from benthos folder ): 
    kustomize build . | kubectl apply -f -     or
kubectl apply -f benthos-deployment.yaml
  • After modifying cosmo router: helm upgrade cosmo-router . -n apps

Find secret

kubectl exec -it deployment/benthos -n infra -- env | findstr ECOM_PASS kubectl get secret postgres-ha-app -n infra kubectl get secrets -n infra kubectl get secret postgres-ha-app -n db -o jsonpath='{.data}'

Create secret

Force-copy secrets

  1. This copies the secret so the 'infra' namespace can see it kubectl get secret postgres-ha-app -n db -o yaml | %{ $_ -replace "namespace: db", "namespace: infra" } | kubectl apply -f -
  2. force-copy from db to infra and clean up the metadata so Kubernetes accepts them as "new" secrets in the infrastructure namespace.
Method 2a.Copy the app secret to the infra namespace
$secret = kubectl get secret postgres-ha-app -n db -o json | ConvertFrom-Json
$secret.metadata.psobject.Properties.Remove('namespace')
$secret.metadata.psobject.Properties.Remove('resourceVersion')
$secret.metadata.psobject.Properties.Remove('uid')
$secret | ConvertTo-Json | kubectl apply -n infra -f -


Method 2b. Copy the app secret from 'db' to 'infra'
kubectl get secret postgres-ha-app --namespace=db -o yaml | `
    ForEach-Object { $_ -replace 'namespace: db', 'namespace: infra' } | `
    kubectl apply -f -

# Copy the superuser secret from 'db' to 'infra'
kubectl get secret postgres-ha-superuser --namespace=db -o yaml | `
    ForEach-Object { $_ -replace 'namespace: db', 'namespace: infra' } | `
    kubectl apply -f -

or

Method 2c.Export from DB namespace, strip IDs, and apply to INFRA namespace
kubectl get secret postgres-ha-app -n db -o yaml | `
  Select-String -Pattern "ownerReferences|resourceVersion|uid|creationTimestamp|namespace: db" -NotMatch | `
  Out-String | ForEach-Object { $_ + "`n  namespace: infra" } | kubectl apply -f -

kubectl get secret postgres-ha-superuser -n db -o yaml | `
  Select-String -Pattern "ownerReferences|resourceVersion|uid|creationTimestamp|namespace: db" -NotMatch | `
  Out-String | ForEach-Object { $_ + "`n  namespace: infra" } | kubectl apply -f -

or 
Mehod 2d. Copy the app secret
kubectl get secret postgres-ha-app -n db -o json | jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid)' | kubectl apply -n infra -f -

# Copy the superuser secret
kubectl get secret postgres-ha-superuser -n db -o json | jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid)' | kubectl apply -n infra -f -

Debug

kubectl get all -A 
kubectl describe pod yb-master-0 -n db
kubectl cluster-info
kubectl get cluster -n db -w # cnpg operator
kubectl get nodes.longhorn.io -n longhorn-system
kubectl get pods -n longhorn-system -o wide
kubectl get svc -n db
kubectl logs -f deployment/benthos -n infra --all-containers
NATS:
kubectl exec -it nats-box-temp nats -n infra -- /bin/sh
  nats -s nats://nats-cluster:4222 sub "output_request_logs"

Delete

Mostly for db use all and -l label kubectl get pods -n db --show-labels

From the related directory: kustomize build . | kubectl delete -f -
kubectl delete all -l app=yb-master -n db
kubectl delete all -l app=yb-tserver -n db
kubectl delete all -l cnpg.io/cluster=postgres-ha -n db
For shrinking volumes in postgres:
kubectl delete cluster postgres-ha -n db
kubectl delete pvc --all -n db

Known error/Bugs

1. In wsl Running helm command or kubectl

Error: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp 127.0.0.1:8080:
a. Create the directory if it doesn't exist
mkdir -p ~/.kube

b. Link your Windows kubeconfig to your WSL home directory
ln -s /mnt/c/Users/<YourWindowsUsername>/.kube/config ~/.kube/config

This fix got executed in deploy-k3s 

2. with httproutes.gateway.networking.k8s.io
kubectl delete crd httproutes.gateway.networking.k8s.io
kubectl edit crd httproutes.gateway.networking.k8s.io (add v1 back into the spec.versions list)
kubectl get crd | grep gateway.networking.k8s.io

3. pods exist in wsl but not in PS

sudo chmod 644 /etc/rancher/k3s/k3s.yaml
Run in PS
kustomize build . --enable-helm | kubectl apply --server-side --force-conflicts -f -

4. pods exist in PS but not in wsl

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
export KUBECONFIG=~/.kube/config

Login to db

Getting user name
$userBase64 = kubectl get secret postgres-ha-app -n db -o jsonpath='{.data.username}'
[System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($userBase64))

Getting password for user invixel_admin:
For Powershell
- $pass = kubectl get secret postgres-ha-app -n db -o jsonpath='{.data.password}'
- [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($pass))

Ours: 
0lkzPxlwj6JVOXwwoLYROZJsONJoPK3MtrqkxnH3iaXUs0gFg0WL78RxyDdB86Sk
Getting password for super user
$pass = kubectl get secret postgres-ha-superuser -n db -o jsonpath='{.data.password}'
[System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($pass))
Ours:
CuUwr9dPXJibVFygh7oGastvIJb4syMZXKljsf0dbEl91TwuYLqvEW35hN98ytKe

1. From inside the cluster (Quick Check)
kubectl exec -it postgres-ha-1 -n db -- psql -U postgres

2. From windows machine:
kubectl port-forward svc/postgres-ha-rw -n db 5432:5432
Open your DB Tool and connect to:

Host: localhost

Port: 5432
User: postgres
Password: (The one you decoded in step 2)

Primary Key & Sequential

SELECT increment_by, cache_size, last_value FROM pg_sequences WHERE schemaname = 'public' AND sequencename = 'request_logs_id_seq';

ALTER SEQUENCE request_logs_id_seq INCREMENT BY 1;

Alternative using ansible

  • name: Deploy Longhorn to k3s Cluster hosts: master become: yes vars:

    Path to k3s config on the master node

    kubeconfig: /etc/rancher/k3s/k3s.yaml longhorn_namespace: longhorn-system

    tasks:

    • name: Add Longhorn Helm repo kubernetes.core.helm_repository: name: longhorn repo_url: "https://charts.longhorn.io"

    • name: Create Longhorn namespace kubernetes.core.k8s: name: "{{ longhorn_namespace }}" kind: Namespace state: present kubeconfig: "{{ kubeconfig }}"

    • name: Install Longhorn via Helm kubernetes.core.helm: name: longhorn chart_ref: longhorn/longhorn release_namespace: "{{ longhorn_namespace }}" kubeconfig: "{{ kubeconfig }}" wait: yes

      Longhorn requires open-iscsi and nfs-client on nodes

      Ensure those are installed on your workers first!

    • name: Set Longhorn as default StorageClass kubernetes.core.k8s_json_patch: kind: StorageClass name: longhorn kubeconfig: "{{ kubeconfig }}" patch: - op: add path: /metadata/annotations/storageclass.kubernetes.io~1is-default-class value: "true"