kubernetes实战(十)：k8s使用Helm安装harbor
1、基本概念
对于复杂的应用中间件，需要设置镜像运行的需求、环境变量，并且需要定制存储、网络等设置，最后设计和编写Deployment、Configmap、Service及Ingress等相关yaml配置文件，再提交给kubernetes进行部署。这些复杂的过程将逐步被Helm应用包管理工具实现。
Helm是一个由CNCF孵化和管理的项目，用于对需要在k8s上部署复杂应用进行定义、安装和更新。Helm以Chart的方式对应用软件进行描述，可以方便地创建、版本化、共享和发布复杂的应用软件。
Chart：一个Helm包，其中包含了运行一个应用所需要的工具和资源定义，还可能包含kubernetes集群中的服务定义，类似于Homebrew中的formula、apt中的dpkg或者yum中的rpm文件。
Release：在K8S集群上运行一个Chart实例。在同一个集群上，一个Chart可以安装多次，例如有一个MySQL Chart，如果想在服务器上运行两个数据库，就可以基于这个Chart安装两次。每次安装都会生成新的Release，会有独立的Release名称。
Repository：用于存放和共享Chart的仓库。
简单来说，Helm的任务是在仓库中查找需要的Chart，然后将Chart以Release的形式安装到K8S集群中。
Harbor基本概念：此篇文章很不错
2、Helm安装
Helm由两个组件组成：
- HelmClinet：客户端，拥有对Repository、Chart、Release等对象的管理能力。
TillerServer：负责客户端指令和k8s集群之间的交互，根据Chart定义，生成和管理各种k8s的资源对象。
安装HelmClient：可以通过二进制文件或脚本方式进行安装。
下载最新版二进制文件：https://github.com/helm/helm/releases
复制代码
[root@k8s-master01 ~]# tar xf helm-v2.11.0-linux-amd64.tar.gz
[root@k8s-master01 ~]# cp linux-amd64/helm linux-amd64/tiller /usr/local/bin/
复制代码
复制代码
[root@k8s-master01 ~]# helm version
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Error: could not find tiller
# 因为没有安装tillerServer所以会报找不到tiller
复制代码
安装TillerServer
所有节点下载tiller:v[helm-version]镜像，helm-version为上面helm的版本2.11.0
docker pull dotbalo/tiller:v2.11.0
使用helm init安装tiller
复制代码
[root@k8s-master01 ~]# helm init --tiller-image dotbalo/tiller:v2.11.0
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
复制代码
再次查看helm version及pod状态
复制代码
[root@k8s-master01 ~]# helm version
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
复制代码
复制代码
[root@k8s-master01 ~]# kubectl get pod -n kube-system | grep tiller
tiller-deploy-5d7c8fcd59-d4djx 1/1 Running 0 49s
[root@k8s-master01 ~]# kubectl get pod,svc -n kube-system | grep tiller
pod/tiller-deploy-5d7c8fcd59-d4djx 1/1 Running 0 3m
service/tiller-deploy ClusterIP 10.106.28.190 44134/TCP 5m
复制代码
3、Helm使用
3.1 helm search：搜索可用的Chart
Helm初始化完成之后，默认配置为使用官方的k8s chart仓库
通过search查找可用的Chart
复制代码
[root@k8s-master01 ~]# helm search gitlab
NAME CHART VERSION APP VERSION DESCRIPTION
stable/gitlab-ce 0.2.2 9.4.1 GitLab Community Edition
stable/gitlab-ee 0.2.2 9.4.1 GitLab Enterprise Edition
[root@k8s-master01 ~]# helm search | more
NAME CHART VERSION APP VERSION DESCRIPTION
stable/acs-engine-autoscaler 2.2.0 2.1.1 Scales worker n
odes within agent pools
stable/aerospike 0.1.7 v3.14.1.2 A Helm chart fo
r Aerospike in Kubernetes
stable/anchore-engine 0.9.0 0.3.0 Anchore contain
er analysis and policy evaluation engine s...
stable/apm-server 0.1.0 6.2.4 The server rece
ives data from the Elastic APM agents and ...
stable/ark 1.2.2 0.9.1 A Helm chart fo
r ark
stable/artifactory 7.3.1 6.1.0 DEPRECATED Univ
ersal Repository Manager supporting all ma...
stable/artifactory-ha 0.4.1 6.2.0 DEPRECATED Univ
ersal Repository Manager supporting all ma...
stable/auditbeat 0.3.1 6.4.3 A lightweight s
hipper to audit the activities of users an...
--More--
复制代码
查看详细信息
复制代码
[root@k8s-master01 ~]# helm search gitlab
NAME CHART VERSION APP VERSION DESCRIPTION
stable/gitlab-ce 0.2.2 9.4.1 GitLab Community Edition
stable/gitlab-ee 0.2.2 9.4.1 GitLab Enterprise Edition
[root@k8s-master01 ~]# helm inspect stable/gitlab-ce
复制代码
3.2 Helm install harbor
使用helm repo remove和add删除repository和添加aliyun的repository
复制代码
[root@k8s-master01 harbor-helm]# helm repo list
NAME URL
aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
复制代码
下载harbor，并checkout到0.3.0分支
git clone https://github.com/goharbor/harbor-helm.git
更改requirement.yaml如下
复制代码
[root@k8s-master01 harbor-helm]# cat requirements.yaml
dependencies:
- name: redis
version: 1.1.15
repository: https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
#repository: https://kubernetes-charts.storage.googleapis.com
复制代码
下载依赖
复制代码
[root@k8s-master01 harbor-helm]# helm dependency update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "aliyun" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading redis from repo https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
Deleting outdated charts
复制代码
所有节点下载相关镜像
复制代码
docker pull goharbor/chartmuseum-photon:v0.7.1-v1.6.0
docker pull goharbor/harbor-adminserver:v1.6.0
docker pull goharbor/harbor-jobservice:v1.6.0
docker pull goharbor/harbor-ui:v1.6.0
docker pull goharbor/harbor-db:v1.6.0
docker pull goharbor/registry-photon:v2.6.2-v1.6.0
docker pull goharbor/chartmuseum-photon:v0.7.1-v1.6.0
docker pull goharbor/clair-photon:v2.0.5-v1.6.0
docker pull goharbor/notary-server-photon:v0.5.1-v1.6.0
docker pull goharbor/notary-signer-photon:v0.5.1-v1.6.0
docker pull bitnami/redis:4.0.8-r2
复制代码
更改values.yaml所有的storageClass为storageClass: "gluster-heketi"
注意修改values.yaml的redis默认配置，添加port至master
复制代码
master:
port: 6379
复制代码
注意修改charts/redis-1.1.15.tgz 里面的redis的values.yaml的storageClass也为"gluster-heketi"，usePassword为 false
注意修改charts/redis-1.1.15.tgz 里面的redis下template下的svc的name: {{ template "redis.fullname" . }}-master
注意修改相关存储空间的大小，比如registry。
安装harbor
helm install --name harbor-v1 . --wait --timeout 1500 --debug --namespace harbor
如果报forbidden的错误，需要创建serveraccount
复制代码
[root@k8s-master01 harbor-helm]# helm install --name harbor-v1 . --set externalDomain=harbor.xxx.net --wait --timeout 1500 --debug --namespace harbor
[debug] Created tunnel using local port: '35557'
[debug] SERVER: "127.0.0.1:35557"
[debug] Original chart version: ""
[debug] CHART PATH: /root/harbor-helm
Error: release harbor-v1 failed: namespaces "harbor" is forbidden: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "harbor"
复制代码
解决：
复制代码
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
复制代码
再次部署：
复制代码
......
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
harbor-v1-redis-84dffd8574-xzrsh 0/1 Running 0
harbor-v1-harbor-adminserver-5b59c684b4-g6cjc 1/1 Running 0
harbor-v1-harbor-chartmuseum-699cf6599-q6vfw 1/1 Running 0
harbor-v1-harbor-clair-6d9bb84485-2p52v 1/1 Running 0
harbor-v1-harbor-jobservice-5c9496775d-sj6mb 1/1 Running 0
harbor-v1-harbor-notary-server-5fb65b6866-dnnnk 1/1 Running 0
harbor-v1-harbor-notary-signer-5bfcfcd5cf-j774t 1/1 Running 0
harbor-v1-harbor-registry-75c9b6b457-pqxj6 1/1 Running 0
harbor-v1-harbor-ui-5974bd5549-zl9nj 1/1 Running 0
harbor-v1-harbor-database-0 1/1 Running 0
==> v1/Secret
NAME AGE
harbor-v1-harbor-adminserver
harbor-v1-harbor-chartmuseum
harbor-v1-harbor-database
harbor-v1-harbor-ingress
harbor-v1-harbor-jobservice
harbor-v1-harbor-registry
harbor-v1-harbor-ui
NOTES:
Please wait for several minutes for Harbor deployment to complete.
Then you should be able to visit the UI portal at https://core.harbor.domain.
For more details, please visit https://github.com/goharbor/harbor.
......
复制代码
查看pod
复制代码
[root@k8s-master01 harbor-helm]# kubectl get pod -n harbor
NAME READY STATUS RESTARTS AGE
harbor-v1-harbor-adminserver-5b59c684b4-g6cjc 1/1 Running 1 2m
harbor-v1-harbor-chartmuseum-699cf6599-q6vfw 1/1 Running 0 2m
harbor-v1-harbor-clair-6d9bb84485-2p52v 1/1 Running 1 2m
harbor-v1-harbor-database-0 1/1 Running 0 2m
harbor-v1-harbor-jobservice-5c9496775d-sj6mb 1/1 Running 1 2m
harbor-v1-harbor-notary-server-5fb65b6866-dnnnk 1/1 Running 0 2m
harbor-v1-harbor-notary-signer-5bfcfcd5cf-j774t 1/1 Running 0 2m
harbor-v1-harbor-registry-75c9b6b457-pqxj6 1/1 Running 0 2m
harbor-v1-harbor-ui-5974bd5549-zl9nj 1/1 Running 2 2m
harbor-v1-redis-84dffd8574-xzrsh 1/1 Running 0 2m
复制代码
查看service
复制代码
[root@k8s-master01 harbor-helm]# kubectl get svc -n harbor
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
glusterfs-dynamic-database-data-harbor-v1-harbor-database-0 ClusterIP 10.101.10.82 1/TCP 2h
glusterfs-dynamic-harbor-v1-harbor-chartmuseum ClusterIP 10.97.114.51 1/TCP 36s
glusterfs-dynamic-harbor-v1-harbor-registry ClusterIP 10.98.207.16 1/TCP 36s
glusterfs-dynamic-harbor-v1-redis ClusterIP 10.105.214.102 1/TCP 31s
harbor-v1-harbor-adminserver ClusterIP 10.99.152.38 80/TCP 3m
harbor-v1-harbor-chartmuseum ClusterIP 10.99.237.224 80/TCP 3m
harbor-v1-harbor-clair ClusterIP 10.98.217.176 6060/TCP 3m
harbor-v1-harbor-database ClusterIP 10.111.182.188 5432/TCP 3m
harbor-v1-harbor-jobservice ClusterIP 10.98.202.61 80/TCP 3m
harbor-v1-harbor-notary-server ClusterIP 10.110.72.98 4443/TCP 3m
harbor-v1-harbor-notary-signer ClusterIP 10.106.234.19 7899/TCP 3m
harbor-v1-harbor-registry ClusterIP 10.98.80.141 5000/TCP 3m
harbor-v1-harbor-ui ClusterIP 10.98.240.15 80/TCP 3m
harbor-v1-redis ClusterIP 10.107.234.107 6379/TCP 3m
复制代码
查看pv和pvc
复制代码
[root@k8s-master01 harbor-helm]# kubectl get pv,pvc -n harbor | grep harbor
persistentvolume/pvc-080d1242-e990-11e8-8a89-000c293ad492 1Gi RWO Delete Bound harbor/database-data-harbor-v1-harbor-database-0 gluster-heketi 2h
persistentvolume/pvc-f573b165-e9a3-11e8-882f-000c293bfe27 8Gi RWO Delete Bound harbor/harbor-v1-redis gluster-heketi 1m
persistentvolume/pvc-f575855d-e9a3-11e8-882f-000c293bfe27 5Gi RWO Delete Bound harbor/harbor-v1-harbor-chartmuseum gluster-heketi 1m
persistentvolume/pvc-f577371b-e9a3-11e8-882f-000c293bfe27 10Gi RWO Delete Bound harbor/harbor-v1-harbor-registry gluster-heketi 1m
persistentvolumeclaim/database-data-harbor-v1-harbor-database-0 Bound pvc-080d1242-e990-11e8-8a89-000c293ad492 1Gi RWO gluster-heketi 2h
persistentvolumeclaim/harbor-v1-harbor-chartmuseum
