318 lines
11 KiB
Markdown
318 lines
11 KiB
Markdown
### Getting Started
|
|
#### Join new nodes
|
|
```
|
|
# retrieve token on control-plane node
|
|
sudo cat /var/lib/rancher/k3s/server/node-token
|
|
K10094c77f9748b705ce6fd128b916dd96793160853577fdcba986c57de9b6f4a3c::server:b5743ed19b035f78f59607b2fa3bda2e
|
|
|
|
# join via
|
|
curl -sfL https://get.k3s.io | K3S_URL=https://<your-server-ip>:6443 K3S_TOKEN=<NodeToken> sh -
|
|
|
|
K3S_URL: The address of your control plane node
|
|
|
|
curl -sfL https://get.k3s.io | K3S_URL=https://192.168.3.91:6443 K3S_TOKEN=K10094c77f9748b705ce6fd128b916dd96793160853577fdcba986c57de9b6f4a3c::server:b5743ed19b035f78f59607b2fa3bda2e sh -
|
|
|
|
```
|
|
#### 1. Using ansible
|
|
```
|
|
a. ansible-playbook -i hosts.ini ansible-prepare-k3s.yml
|
|
if not working:
|
|
export ANSIBLE_CONFIG=./ansible.cfg
|
|
sed -i 's/\r$//' hosts.ini
|
|
|
|
b. copy content of file from control-plane (sudo cat /etc/rancher/k3s/k3s.yaml) to config inside C:\Users\wendg2\.kube
|
|
c. kustomize build --enable-helm . | kubectl apply --server-side --force-conflicts -f -
|
|
d. ansible-playbook -i hosts.ini deploy-k3s.yml
|
|
```
|
|
#### 2. Install kustomize
|
|
- Download version > v4.x fromhttps://github.com/kubernetes-sigs/kustomize/releases and set the path in your environment variables
|
|
- Test kustomize
|
|
|
|
|
|
#### 3. Setup k3s remote control
|
|
export ANSIBLE_CONFIG=./ansible.cfg
|
|
ssh-keygen -R 192.168.3.91 # in PS
|
|
ssh-keygen -f '/home/josh/.ssh/known_hosts' -R '192.168.3.91' # in wsl
|
|
|
|
#### How Ansible finds its Config (The Hierarchy)
|
|
Ansible looks for its configuration in a very specific order. It stops at the first one it finds:
|
|
```
|
|
1. ANSIBLE_CONFIG (The environment variable): This is the "Nuclear Option." It overrides everything else.
|
|
|
|
2. ansible.cfg (In the current directory): This is what you were trying to use, but it was being ignored due to the permissions.
|
|
|
|
3. ~/.ansible.cfg (In your home directory): Personal user settings.
|
|
|
|
4. /etc/ansible/ansible.cfg: The global system default
|
|
```
|
|
#### 4. using kustomize and helm together
|
|
- For yugabyte there must be enough space
|
|
kustomize build . --enable-helm | kubectl apply -f -
|
|
kustomize build . | kubectl apply --server-side --force-conflicts -f - # from root folder
|
|
kustomize build . | kubectl apply --server-side --force-conflicts -f - # from some folder
|
|
|
|
for longhorn-system, argocd, linkerd, etc
|
|
|
|
- For postgres we use cnpg operator
|
|
```
|
|
We must run twice:
|
|
kustomize build . --enable-helm | kubectl apply -f -
|
|
kubectl get pods -n cnpg-system
|
|
kubectl get cluster -n db -w
|
|
kubectl get pods -n db
|
|
```
|
|
#### 5.Start our apps
|
|
```
|
|
a. kubectl port-forward svc/cosmo-router 3002:3002 -n apps
|
|
b. kubectl run nats-box-temp --image=natsio/nats-box -n infra --rm -it -- nats -s nats://nats-cluster:4222 sub "input_request_logs"
|
|
c. kubectl exec -it nats-box nats -n infra -- /bin/sh
|
|
|
|
nats -s nats://nats-cluster.infra.svc.cluster.local:4222 stream add request_stream --subjects "input_request_logs" --ack --storage file --retention limits --max-msgs=-1 --max-bytes=-1 -
|
|
-max-age=1y --replicas 3
|
|
|
|
FQDN with pattern as `[service-name].[namespace].svc.cluster.local`
|
|
- find service name with : kubectl get svc -n db
|
|
- find service name is to be found through labels :
|
|
service-name for nats-cluster from app.kubernetes.io/instance
|
|
nats-cluster in labels app.kubernetes.io/instance=nats-cluster
|
|
d. kubectl logs -f benthos-6ff4b9dfb5-7klqr -n infra
|
|
```
|
|
|
|
#### Apply Changes
|
|
|
|
- After modifying Benthos ( directly from benthos folder ):
|
|
```
|
|
kustomize build . | kubectl apply -f - or
|
|
kubectl apply -f benthos-deployment.yaml
|
|
```
|
|
- After modifying cosmo router: helm upgrade cosmo-router . -n apps
|
|
|
|
|
|
|
|
#### Find secret
|
|
kubectl exec -it deployment/benthos -n infra -- env | findstr ECOM_PASS
|
|
kubectl get secret postgres-ha-app -n infra
|
|
kubectl get secrets -n infra
|
|
kubectl get secret postgres-ha-app -n db -o jsonpath='{.data}'
|
|
|
|
|
|
#### Create secret
|
|
|
|
#### Force-copy secrets
|
|
1. This copies the secret so the 'infra' namespace can see it
|
|
kubectl get secret postgres-ha-app -n db -o yaml | %{ $_ -replace "namespace: db", "namespace: infra" } | kubectl apply -f -
|
|
2. force-copy from db to infra and clean up the metadata so Kubernetes accepts them as "new" secrets in the infrastructure namespace.
|
|
```
|
|
Method 2a.Copy the app secret to the infra namespace
|
|
$secret = kubectl get secret postgres-ha-app -n db -o json | ConvertFrom-Json
|
|
$secret.metadata.psobject.Properties.Remove('namespace')
|
|
$secret.metadata.psobject.Properties.Remove('resourceVersion')
|
|
$secret.metadata.psobject.Properties.Remove('uid')
|
|
$secret | ConvertTo-Json | kubectl apply -n infra -f -
|
|
|
|
|
|
Method 2b. Copy the app secret from 'db' to 'infra'
|
|
kubectl get secret postgres-ha-app --namespace=db -o yaml | `
|
|
ForEach-Object { $_ -replace 'namespace: db', 'namespace: infra' } | `
|
|
kubectl apply -f -
|
|
|
|
# Copy the superuser secret from 'db' to 'infra'
|
|
kubectl get secret postgres-ha-superuser --namespace=db -o yaml | `
|
|
ForEach-Object { $_ -replace 'namespace: db', 'namespace: infra' } | `
|
|
kubectl apply -f -
|
|
|
|
or
|
|
|
|
Method 2c.Export from DB namespace, strip IDs, and apply to INFRA namespace
|
|
kubectl get secret postgres-ha-app -n db -o yaml | `
|
|
Select-String -Pattern "ownerReferences|resourceVersion|uid|creationTimestamp|namespace: db" -NotMatch | `
|
|
Out-String | ForEach-Object { $_ + "`n namespace: infra" } | kubectl apply -f -
|
|
|
|
kubectl get secret postgres-ha-superuser -n db -o yaml | `
|
|
Select-String -Pattern "ownerReferences|resourceVersion|uid|creationTimestamp|namespace: db" -NotMatch | `
|
|
Out-String | ForEach-Object { $_ + "`n namespace: infra" } | kubectl apply -f -
|
|
|
|
or
|
|
Mehod 2d. Copy the app secret
|
|
kubectl get secret postgres-ha-app -n db -o json | jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid)' | kubectl apply -n infra -f -
|
|
|
|
# Copy the superuser secret
|
|
kubectl get secret postgres-ha-superuser -n db -o json | jq 'del(.metadata.namespace,.metadata.resourceVersion,.metadata.uid)' | kubectl apply -n infra -f -
|
|
|
|
```
|
|
#### Avoiding sudo - Unable to read Error while executing kubectl command option
|
|
```
|
|
# Option 1: Change permissions (allows your user to read)
|
|
sudo chmod 644 /etc/rancher/k3s/k3s.yaml
|
|
|
|
# Option 2: Copy the config to your user's directory
|
|
mkdir -p ~/.kube
|
|
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
|
|
sudo chown $(id -u):$(id -g) ~/.kube/config
|
|
chmod 600 ~/.kube/config
|
|
```
|
|
|
|
#### Debug
|
|
```
|
|
kubectl get all -A
|
|
kubectl describe pod yb-master-0 -n db
|
|
kubectl cluster-info
|
|
kubectl get cluster -n db -w # cnpg operator
|
|
kubectl get nodes.longhorn.io -n longhorn-system
|
|
kubectl get pods -n longhorn-system -o wide
|
|
kubectl get svc -n db
|
|
kubectl logs -f deployment/benthos -n infra --all-containers
|
|
NATS:
|
|
kubectl exec -it nats-box-temp nats -n infra -- /bin/sh
|
|
nats -s nats://nats-cluster:4222 sub "output_request_logs"
|
|
```
|
|
#### Delete
|
|
Mostly for db use all and -l label
|
|
kubectl get pods -n db --show-labels
|
|
```
|
|
From the related directory: kustomize build . | kubectl delete -f -
|
|
kubectl delete all -l app=yb-master -n db
|
|
kubectl delete all -l app=yb-tserver -n db
|
|
kubectl delete all -l cnpg.io/cluster=postgres-ha -n db
|
|
For shrinking volumes in postgres:
|
|
kubectl delete cluster postgres-ha -n db
|
|
kubectl delete pvc --all -n db
|
|
```
|
|
#### Known error/Bugs
|
|
```
|
|
1. In wsl Running helm command or kubectl
|
|
|
|
Error: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp 127.0.0.1:8080:
|
|
a. Create the directory if it doesn't exist
|
|
mkdir -p ~/.kube
|
|
|
|
b. Link your Windows kubeconfig to your WSL home directory
|
|
ln -s /mnt/c/Users/<YourWindowsUsername>/.kube/config ~/.kube/config
|
|
|
|
This fix got executed in deploy-k3s
|
|
|
|
2. with httproutes.gateway.networking.k8s.io
|
|
kubectl delete crd httproutes.gateway.networking.k8s.io
|
|
kubectl edit crd httproutes.gateway.networking.k8s.io (add v1 back into the spec.versions list)
|
|
kubectl get crd | grep gateway.networking.k8s.io
|
|
|
|
3. pods exist in wsl but not in PS
|
|
|
|
sudo chmod 644 /etc/rancher/k3s/k3s.yaml
|
|
Run in PS
|
|
kustomize build . --enable-helm | kubectl apply --server-side --force-conflicts -f -
|
|
|
|
4. pods exist in PS but not in wsl -- Join/Query from Laptop to k3s servers
|
|
|
|
# On WSL
|
|
mkdir -p ~/.kube
|
|
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
|
|
sudo chown $(id -u):$(id -g) ~/.kube/config
|
|
export KUBECONFIG=~/.kube/config
|
|
|
|
change IP manually from 127.0.0.1 to 192.168.3.91
|
|
|
|
or
|
|
|
|
mkdir -p ~/.kube
|
|
scp master-1@192.168.3.91:/etc/rancher/k3s/k3s.yaml ~/.kube/config
|
|
sed -i 's/127.0.0.1/192.168.3.91/g' ~/.kube/config
|
|
chmod 600 ~/.kube/config
|
|
|
|
# Testen
|
|
kubectl get nodes
|
|
|
|
# on PS
|
|
ssh master@192.168.3.156 "sudo cat /etc/rancher/k3s/k3s.yaml" > "$HOME\.kube\config"
|
|
|
|
# Localhost-IP durch die echte Master-IP ersetzen
|
|
(Get-Content "$HOME\.kube\config") -replace '127.0.0.1', '192.168.3.156' | Set-Content "$HOME\.kube\config"
|
|
|
|
# Testen
|
|
kubectl get nodes
|
|
```
|
|
|
|
|
|
#### Login to db
|
|
```
|
|
Getting user name
|
|
$userBase64 = kubectl get secret postgres-ha-app -n db -o jsonpath='{.data.username}'
|
|
[System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($userBase64))
|
|
|
|
Getting password for user invixel_admin:
|
|
For Powershell
|
|
- $pass = kubectl get secret postgres-ha-app -n db -o jsonpath='{.data.password}'
|
|
- [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($pass))
|
|
|
|
Ours:
|
|
0lkzPxlwj6JVOXwwoLYROZJsONJoPK3MtrqkxnH3iaXUs0gFg0WL78RxyDdB86Sk
|
|
Getting password for super user
|
|
$pass = kubectl get secret postgres-ha-superuser -n db -o jsonpath='{.data.password}'
|
|
[System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($pass))
|
|
Ours:
|
|
CuUwr9dPXJibVFygh7oGastvIJb4syMZXKljsf0dbEl91TwuYLqvEW35hN98ytKe
|
|
|
|
1. From inside the cluster (Quick Check)
|
|
kubectl exec -it postgres-ha-1 -n db -- psql -U postgres
|
|
|
|
2. From windows machine:
|
|
kubectl port-forward svc/postgres-ha-rw -n db 5432:5432
|
|
Open your DB Tool and connect to:
|
|
|
|
Host: localhost
|
|
|
|
Port: 5432
|
|
User: postgres
|
|
Password: (The one you decoded in step 2)
|
|
```
|
|
|
|
#### Primary Key & Sequential
|
|
SELECT increment_by, cache_size, last_value
|
|
FROM pg_sequences
|
|
WHERE schemaname = 'public' AND sequencename = 'request_logs_id_seq';
|
|
|
|
ALTER SEQUENCE request_logs_id_seq INCREMENT BY 1;
|
|
|
|
|
|
#### Alternative using ansible
|
|
---
|
|
- name: Deploy Longhorn to k3s Cluster
|
|
hosts: master
|
|
become: yes
|
|
vars:
|
|
# Path to k3s config on the master node
|
|
kubeconfig: /etc/rancher/k3s/k3s.yaml
|
|
longhorn_namespace: longhorn-system
|
|
|
|
tasks:
|
|
- name: Add Longhorn Helm repo
|
|
kubernetes.core.helm_repository:
|
|
name: longhorn
|
|
repo_url: "https://charts.longhorn.io"
|
|
|
|
- name: Create Longhorn namespace
|
|
kubernetes.core.k8s:
|
|
name: "{{ longhorn_namespace }}"
|
|
kind: Namespace
|
|
state: present
|
|
kubeconfig: "{{ kubeconfig }}"
|
|
|
|
- name: Install Longhorn via Helm
|
|
kubernetes.core.helm:
|
|
name: longhorn
|
|
chart_ref: longhorn/longhorn
|
|
release_namespace: "{{ longhorn_namespace }}"
|
|
kubeconfig: "{{ kubeconfig }}"
|
|
wait: yes
|
|
# Longhorn requires open-iscsi and nfs-client on nodes
|
|
# Ensure those are installed on your workers first!
|
|
|
|
- name: Set Longhorn as default StorageClass
|
|
kubernetes.core.k8s_json_patch:
|
|
kind: StorageClass
|
|
name: longhorn
|
|
kubeconfig: "{{ kubeconfig }}"
|
|
patch:
|
|
- op: add
|
|
path: /metadata/annotations/storageclass.kubernetes.io~1is-default-class
|
|
value: "true" |