kubernetes(k8s)설치 - kubespray

la -l .ssh/
total 8
-rw------- 1 kakao kakao 1679 Mar  4 23:08 id_rsa
-rw-r--r-- 1 kakao kakao  395 Mar  4 23:08 id_rsa.pub

ssh-copy-id 192.168.55.3
ssh-copy-id 192.168.55.203
ssh-copy-id 192.168.55.223
ssh-copy-id 192.168.55.109
ssh-copy-id 192.168.55.249

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL) NOPASSWD:ALL

sundo@k8s-1:~$ sudo pip3 install --upgrade pip

Collecting pip
Downloading pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
|████████████████████████████████| 1.5 MB 1.3 MB/s
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.0.2
Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
Can't uninstall 'pip'. No files were found to uninstall.
Successfully installed pip-20.3.3

user@k8s-1:~/kubespray$ sudo pip install -r requirements.txt
Collecting ansible==2.9.6
  Downloading ansible-2.9.6.tar.gz (14.2 MB)
     |████████████████████████████████| 14.2 MB 7.9 MB/s
Requirement already satisfied: PyYAML in /usr/lib/python3/dist-packages (from ansible==2.9.6->-r requirements.txt (line 1)) (5.3.1)
Requirement already satisfied: cryptography in /usr/lib/python3/dist-packages (from ansible==2.9.6->-r requirements.txt (line 1)) (2.8)
Collecting jinja2==2.11.1
  Downloading Jinja2-2.11.1-py2.py3-none-any.whl (126 kB)
     |████████████████████████████████| 126 kB 12.9 MB/s
Requirement already satisfied: MarkupSafe>=0.23 in /usr/lib/python3/dist-packages (from jinja2==2.11.1->-r requirements.txt (line 2)) (1.1.0)
Collecting jmespath==0.9.5
  Downloading jmespath-0.9.5-py2.py3-none-any.whl (24 kB)
Collecting netaddr==0.7.19
  Downloading netaddr-0.7.19-py2.py3-none-any.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 11.2 MB/s
Collecting pbr==5.4.4
  Downloading pbr-5.4.4-py2.py3-none-any.whl (110 kB)
     |████████████████████████████████| 110 kB 1.7 MB/s
Collecting ruamel.yaml==0.16.10
  Downloading ruamel.yaml-0.16.10-py2.py3-none-any.whl (111 kB)
     |████████████████████████████████| 111 kB 11.9 MB/s
Collecting ruamel.yaml.clib>=0.1.2
  Downloading ruamel.yaml.clib-0.2.2-cp38-cp38-manylinux1_x86_64.whl (578 kB)
     |████████████████████████████████| 578 kB 12.4 MB/s
Building wheels for collected packages: ansible
  Building wheel for ansible (setup.py) ... done
  Created wheel for ansible: filename=ansible-2.9.6-py3-none-any.whl size=16164186 sha256=0e4c703cf1639de65d09967cfffb84368b674f686ccc73963a8c1cfa34aaecef
  Stored in directory: /root/.cache/pip/wheels/b6/4a/7c/2ca8a5588b8df841aabfbc38fffcc8145406cbd092a340a243
Successfully built ansible
Installing collected packages: ruamel.yaml.clib, jinja2, ruamel.yaml, pbr, netaddr, jmespath, ansible
  Attempting uninstall: jinja2
    Found existing installation: Jinja2 2.10.1
    Uninstalling Jinja2-2.10.1:
      Successfully uninstalled Jinja2-2.10.1
Successfully installed ansible-2.9.6 jinja2-2.11.1 jmespath-0.9.5 netaddr-0.7.19 pbr-5.4.4 ruamel.yaml-0.16.10 ruamel.yaml.clib-0.2.2

user@k8s-1:~/kubespray$ ansible --version
ansible 2.9.6
  config file = /home/sundo/kubespray/ansible.cfg
  configured module search path = ['/home/sundo/kubespray/library']
  ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.5 (default, Jul 28 2020, 12:59:40) [GCC 9.3.0]

~/kubespray$ CONFIG_FILE=inventory/mycluster/hosts.yml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
DEBUG: Adding group all
DEBUG: Adding group kube-master
DEBUG: Adding group kube-node
DEBUG: Adding group etcd
DEBUG: Adding group k8s-cluster
DEBUG: Adding group calico-rr
Usage: inventory.py ip1 [ip2 ...]
Examples: inventory.py 10.10.1.3 10.10.1.4 10.10.1.5

Available commands:
help - Display this message
print_cfg - Write inventory file to stdout
print_ips - Write a space-delimited list of IPs from "all" group
print_hostnames - Write a space-delimited list of Hostnames from "all" group

Advanced usage:
Add another host after initial creation: inventory.py 10.10.1.5
Add range of hosts: inventory.py 10.10.1.3-10.10.1.5
Add hosts with different ip and access ip: inventory.py 10.0.0.1,192.168.10.1 10.0.0.2,192.168.10.2 10.0.0.3,192.168.10.3
Add hosts with a specific hostname, ip, and optional access ip: first,10.0.0.1,192.168.10.1 second,10.0.0.2 last,10.0.0.3
Delete a host: inventory.py -10.10.1.3
Delete a host by id: inventory.py -node1

Configurable env vars:
DEBUG                   Enable debug printing. Default: True
CONFIG_FILE             File to write config to Default: ./inventory/sample/hosts.yaml
HOST_PREFIX             Host prefix for generated hosts. Default: node
SCALE_THRESHOLD         Separate ETCD role if # of nodes >= 50
MASSIVE_SCALE_THRESHOLD Separate K8s master and ETCD if # of nodes >= 200

ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/home/kakao/.ssh/id_rsa):
Created directory '/home/kakao/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/kakao/.ssh/id_rsa.
Your public key has been saved in /home/kakao/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:jf2Hdsglf6RZupEHLOiwL942CFh/B5HzO41GJpfKsu4 kakao@kakao-1
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
+----[SHA256]-----+

la -l .ssh/
total 8
-rw------- 1 kakao kakao 1679 Mar  4 23:08 id_rsa
-rw-r--r-- 1 kakao kakao  395 Mar  4 23:08 id_rsa.pub

cat .ssh/id_rsa.pub

$ssh-copy-id kakao@192.168.55.206 # 마스터 노드 #1
$ssh-copy-id kakao@192.168.55.117
$ssh-copy-id kakao@192.168.55.183
$ssh-copy-id kakao@192.168.55.105
$ssh-copy-id kakao@192.168.55.199

# 작업 피시
$ssh-copy-id kakao@192.168.55.131

ssh-copy-id root@192.168.55.117
ssh-copy-id root@192.168.55.183
ssh-copy-id root@192.168.55.105
ssh-copy-id root@192.168.55.199

kakao@kakao-2:~$ cat .ssh/authorized_keys

sudo apt -y install python-pip

pip install --upgrade pip

$git clone https://github.com/kubernetes-sigs/kubespray.git
Cloning into 'kubespray'...
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 40892 (delta 6), reused 0 (delta 0), pack-reused 40879
Receiving objects: 100% (40892/40892), 11.85 MiB | 4.24 MiB/s, done.
Resolving deltas: 100% (22701/22701), done.
Checking connectivity... done.

$sudo pip install -r requirements.txt

WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
WARNING: The directory '/home/kakao/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, u'EOF occurred in violation of protocol (_ssl.c:590)'),)': /simple/ansible/
Collecting ansible==2.7.12
  Downloading ansible-2.7.12.tar.gz (11.9 MB)
     |████████████████████████████████| 11.9 MB 4.8 MB/s
Collecting jinja2==2.10.1
  Downloading Jinja2-2.10.1-py2.py3-none-any.whl (124 kB)
     |████████████████████████████████| 124 kB 192 kB/s
Collecting netaddr==0.7.19
  Downloading netaddr-0.7.19-py2.py3-none-any.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 756 kB/s
Collecting pbr==5.2.0
  Downloading pbr-5.2.0-py2.py3-none-any.whl (107 kB)
     |████████████████████████████████| 107 kB 189 kB/s
Collecting hvac==0.8.2
  Downloading hvac-0.8.2-py2.py3-none-any.whl (91 kB)
     |████████████████████████████████| 91 kB 132 kB/s
Collecting jmespath==0.9.4
  Downloading jmespath-0.9.4-py2.py3-none-any.whl (24 kB)
Collecting ruamel.yaml==0.15.96
  Downloading ruamel.yaml-0.15.96-cp27-cp27mu-manylinux1_x86_64.whl (601 kB)
     |████████████████████████████████| 601 kB 380 kB/s
Collecting PyYAML
  Downloading PyYAML-5.3.tar.gz (268 kB)
     |████████████████████████████████| 268 kB 747 kB/s
Collecting paramiko
  Downloading paramiko-2.7.1-py2.py3-none-any.whl (206 kB)
     |████████████████████████████████| 206 kB 712 kB/s
Collecting cryptography
  Downloading cryptography-2.8-cp27-cp27mu-manylinux2010_x86_64.whl (2.3 MB)
     |████████████████████████████████| 2.3 MB 337 kB/s
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages (from ansible==2.7.12->-r requirements.txt (line 1)) (20.7.0)
Collecting MarkupSafe>=0.23
  Downloading MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl (24 kB)
Collecting requests>=2.21.0
  Downloading requests-2.23.0-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 23.1 MB/s
Collecting ruamel.ordereddict; platform_python_implementation == "CPython" and python_version <= "2.7"
  Downloading ruamel.ordereddict-0.4.14-cp27-cp27mu-manylinux1_x86_64.whl (93 kB)
     |████████████████████████████████| 93 kB 380 kB/s
Collecting pynacl>=1.0.1
  Downloading PyNaCl-1.3.0-cp27-cp27mu-manylinux1_x86_64.whl (762 kB)
     |████████████████████████████████| 762 kB 336 kB/s
Collecting bcrypt>=3.1.3
  Downloading bcrypt-3.1.7-cp27-cp27mu-manylinux1_x86_64.whl (59 kB)
     |████████████████████████████████| 59 kB 23.3 MB/s
Collecting six>=1.4.1
  Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting cffi!=1.11.3,>=1.8
  Downloading cffi-1.14.0-cp27-cp27mu-manylinux1_x86_64.whl (387 kB)
     |████████████████████████████████| 387 kB 315 kB/s
Collecting ipaddress; python_version < "3"
  Downloading ipaddress-1.0.23-py2.py3-none-any.whl (18 kB)
Collecting enum34; python_version < "3"
  Downloading enum34-1.1.9-py2-none-any.whl (11 kB)
Collecting certifi>=2017.4.17
  Downloading certifi-2019.11.28-py2.py3-none-any.whl (156 kB)
     |████████████████████████████████| 156 kB 549 kB/s
Collecting idna<3,>=2.5
  Downloading idna-2.9-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 22.1 MB/s
Collecting chardet<4,>=3.0.2
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 647 kB/s
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Downloading urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
     |████████████████████████████████| 125 kB 682 kB/s
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
     |████████████████████████████████| 112 kB 516 kB/s
Building wheels for collected packages: ansible, PyYAML
  Building wheel for ansible (setup.py) ... done
  Created wheel for ansible: filename=ansible-2.7.12-py2-none-any.whl size=9430239 sha256=dae0a74b4e52c8a75f6f4d32ebca19920335a8da22fdb8a27129c8ce10763307
  Stored in directory: /tmp/pip-ephem-wheel-cache-IRMPOh/wheels/35/16/79/a507de0ed355e1b8030e0bf6f19b469dfc3600cd0ce4567406
  Building wheel for PyYAML (setup.py) ... done
  Created wheel for PyYAML: filename=PyYAML-5.3-cp27-cp27mu-linux_x86_64.whl size=46466 sha256=eca64dfe0ad2946927bbe688fea1b1bbefe6aad4c85867527dae9d6871ca46cb
  Stored in directory: /tmp/pip-ephem-wheel-cache-IRMPOh/wheels/06/0f/01/cdac9a236377daed23bce2de82d8cc036abc5717f63bd8268c
Successfully built ansible PyYAML
Installing collected packages: MarkupSafe, jinja2, PyYAML, six, pycparser, cffi, ipaddress, enum34, cryptography, pynacl, bcrypt, paramiko, ansible, netaddr, pbr, certifi, idna, chardet, urllib3, requests, hvac, jmespath, ruamel.ordereddict, ruamel.yaml
Successfully installed MarkupSafe-1.1.1 PyYAML-5.3 ansible-2.7.12 bcrypt-3.1.7 certifi-2019.11.28 cffi-1.14.0 chardet-3.0.4 cryptography-2.8 enum34-1.1.9 hvac-0.8.2 idna-2.9 ipaddress-1.0.23 jinja2-2.10.1 jmespath-0.9.4 netaddr-0.7.19 paramiko-2.7.1 pbr-5.2.0 pycparser-2.20 pynacl-1.3.0 requests-2.23.0 ruamel.ordereddict-0.4.14 ruamel.yaml-0.15.96 six-1.14.0 urllib3-1.25.8

vi inventory/mycluster/inventory.ini

## Configure 'ip' variable to bind kubernetes services on a
# ## different ip than the default iface
# ## We should set etcd_member_name for etcd cluster. The node that is not a etcd member do not need to set the value, or can set the empty string value.
[all]
kakao-1 ansible_host=192.168.55.206 # ip=192.168.55.206 etcd_member_name=etcd1
kakao-2 ansible_host=192.168.55.117 # ip=192.168.55.117 etcd_member_name=etcd1
kakao-3 ansible_host=192.168.55.183 # ip=192.168.55.183 etcd_member_name=etcd1
kakao-4 ansible_host=192.168.55.105 # ip=192.168.55.105 etcd_member_name=etcd1
kakao-5 ansible_host=192.168.55.199 # ip=192.168.55.199 etcd_member_name=etcd1


# ## configure a bastion host if your nodes are not directly reachable
# bastion ansible_host=x.x.x.x ansible_user=some_user

[kube-master]
# node1
# node2
kakao-1
kakao-2
kakao-3

[etcd]
kakao-1
kakao-2
kakao-3


[kube-node]
kakao-4
kakao-5

[calico-rr]

[k8s-cluster:children]
kube-master
kube-node
calico-rr

root@kakao-1:~# kubectl get node
NAME      STATUS   ROLES    AGE     VERSION
kakao-1   Ready    master   5m52s   v1.16.6
kakao-2   Ready    master   5m1s    v1.16.6
kakao-3   Ready    master   5m1s    v1.16.6
kakao-4   Ready    <none>   3m18s   v1.16.6
kakao-5   Ready    <none>   3m18s   v1.16.6

SK [download : download_container | Download image if required] *********************************************************************
fatal: [node1 -> 192.168.55.51]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/bin/docker", "pull", "quay.io/coreos/etcd@sha256:None"], "delta": "0:00:00.064617", "end": "2021-01-08 13:49:02.595504", "msg": "non-zero return code", "rc": 1, "start": "2021-01-08 13:49:02.530887", "stderr": "invalid reference format", "stderr_lines": ["invalid reference format"], "stdout": "", "stdout_lines": []}
FAILED - RETRYING: download_container | Download image if required (2 retries left).
FAILED - RETRYING: download_container | Download image if required (2 retries left).
FAILED - RETRYING: download_container | Download image if required (1 retries left).
FAILED - RETRYING: download_container | Download image if required (1 retries left).
fatal: [node2 -> 192.168.55.76]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/bin/docker", "pull", "quay.io/coreos/etcd@sha256:None"], "delta": "0:00:00.052715", "end": "2021-01-08 13:49:18.261204", "msg": "non-zero return code", "rc": 1, "start": "2021-01-08 13:49:18.208489", "stderr": "invalid reference format", "stderr_lines": ["invalid reference format"], "stdout": "", "stdout_lines": []}
fatal: [node3 -> 192.168.55.87]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/bin/docker", "pull", "quay.io/coreos/etcd@sha256:None"], "delta": "0:00:00.053943", "end": "2021-01-08 13:49:18.326764", "msg": "non-zero return code", "rc": 1, "start": "2021-01-08 13:49:18.272821", "stderr": "invalid reference format", "stderr_lines": ["invalid reference format"], "stdout": "", "stdout_lines": []}