This document introduces how to use the StarRocks Operator to automate the deployment and management of a StarRocks cluster on a Kubernetes cluster.
It includes the following parts:
[!NOTE]
The StarRocks k8s operator was designed to be a level 2 operator. See https://sdk.operatorframework.io/docs/overview/operator-capabilities/ to understand more about the capabilities of a level 2 operator.
It includes the following main steps:
StarRockCluster CRD is a custom resource definition (CRD) that defines the StarRocks cluster. It is used to create and manage StarRocks clusters by using the StarRocks Operator. Please refer to api.md for the detailed description of the StarRockCluster CRD.
Apply the StarRockCluster CRD by using the following command:
kubectl apply -f https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/starrocks.com_starrocksclusters.yaml
You can choose to deploy the StarRocks Operator by using a default configuration file or a custom configuration file.
kubectl apply -f https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/operator.yaml
The StarRocks Operator is deployed to the namespace starrocks and manages all StarRocks clusters under all
namespaces. After operator.yaml is applied, The following resources will be created:
namespace/starrocks created
serviceaccount/starrocks created
clusterrole.rbac.authorization.k8s.io/kube-starrocks-operator created
clusterrolebinding.rbac.authorization.k8s.io/kube-starrocks-operator created
role.rbac.authorization.k8s.io/cn-leader-election-role created
rolebinding.rbac.authorization.k8s.io/cn-leader-election-rolebinding created
deployment.apps/kube-starrocks-operator created
curl -O https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/operator.yaml
kubectl apply -f operator.yaml
Check the running status of the StarRocks Operator. If the pod is in the Running state and all containers
inside the pod are READY, the StarRocks Operator is running as expected.
$ kubectl -n starrocks get pods
NAME READY STATUS RESTARTS AGE
starrocks-controller-65bb8679-jkbtg 1/1 Running 0 5m6s
You need to prepare a separate yaml file to deploy the StarRocks FE, BE and CN components. You can directly use the sample configuration files provided by StarRocks to deploy a StarRocks cluster (an object instantiated by using the custom resource StarRocks Cluster). For example, you can use starrocks-fe-and-be.yaml to deploy a StarRocks cluster that consists of three FE nodes and three BE nodes.
kubectl apply -f https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/examples/starrocks/starrocks-fe-and-be.yaml
The following table describes a few important fields in the starrocks-fe-and-be.yaml file.
| Field | Description |
|---|---|
| Kind | The resource type of the object. The value must be StarRocksCluster. |
| Metadata | Metadata, in which the following sub-fields are nested:<ul><li>name: the name of the object. Each object name uniquely identifies an object of the same resource type.</li><li>namespace: the namespace to which the object belongs.</li></ul> |
| Spec | The expected status of the object. Valid values are starRocksFeSpec, starRocksBeSpec, and starRocksCnSpec. |
You can also deploy the StarRocks cluster by using a modified configuration file. For supported fields and detailed descriptions, see api.md.
Deploying the StarRocks cluster takes a while. During this period, you can use the
command kubectl -n starrocks get pods to check the starting status of the StarRocks cluster. If all the pods are in
the Running state and all containers inside the pods are READY, the StarRocks cluster is running as expected.
NOTE
If you customize the namespace in which the StarRocks cluster is located, you need to replace
starrockswith the name of your customized namespace.
$ kubectl -n starrocks get pods
NAME READY STATUS RESTARTS AGE
starrocks-controller-65bb8679-jkbtg 1/1 Running 0 22h
starrockscluster-sample-be-0 1/1 Running 0 23h
starrockscluster-sample-be-1 1/1 Running 0 23h
starrockscluster-sample-be-2 1/1 Running 0 22h
starrockscluster-sample-fe-0 1/1 Running 0 21h
starrockscluster-sample-fe-1 1/1 Running 0 21h
starrockscluster-sample-fe-2 1/1 Running 0 22h
Note
If some pods cannot be up after a long period of time, you can use
kubectl logs -n starrocks <pod_name>to view the log information or usekubectl -n starrocks describe pod <pod_name>to view the event information to address the problem.
The components of the StarRocks cluster can be accessed through their associated Services, such as the FE Service. For detailed descriptions of Services and their access addresses, see api.md and Services.
The following table describes the FE Services of the StarRocks cluster. starrockscluster-sample-fe-service is the
Service that user can configure it from StarRocksCluster CR, and user should only use it to access the StarRocks.
starrockscluster-sample-fe-search is the internal Service that is used by StarRocks Cluster to discover the FE nodes.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
starrockscluster-sample-fe-search ClusterIP None <none> 9030/TCP 76s
starrockscluster-sample-fe-service ClusterIP 10.96.26.146 <none> 8030/TCP,9020/TCP,9030/TCP,9010/TCP 76s
From within the Kubernetes cluster, the StarRocks cluster can be accessed through the FE Service’s ClusterIP.
Obtain the internal virtual IP address CLUSTER-IP and port PORT(S) of the FE Service.
$ kubectl -n starrocks get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
starrockscluster-sample-be-search ClusterIP None <none> 9050/TCP 66s
starrockscluster-sample-be-service ClusterIP 10.96.86.207 <none> 9060/TCP,8040/TCP,9050/TCP,8060/TCP 66s
starrockscluster-sample-fe-search ClusterIP None <none> 9030/TCP 2m27s
starrockscluster-sample-fe-service ClusterIP 10.96.26.146 <none> 8030/TCP,9020/TCP,9030/TCP,9010/TCP 2m27s
Access the StarRocks cluster by using the MySQL client from within the Kubernetes cluster.
mysql -h 10.100.162.xxx -P 9030 -uroot
Upon deploying a fresh StarRocks cluster, the root user’s password remains unset, potentially posing a security
risk. See Change root user password HOWTO for details on how to set
the root user’s password.
From outside the Kubernetes cluster, you can access the StarRocks cluster through the FE Service’s LoadBalancer or NodePort. This topic uses LoadBalancer as an example:
Run the command kubectl -n starrocks edit src starrockscluster-sample to update the StarRocks cluster configuration
file, and add service field to the starRocksFeSpec field.
spec:
starRocksFeSpec:
service:
type: LoadBalancer # specified as LoadBalancer
Obtain the IP address EXTERNAL-IP and port PORT(S) that the FE Service exposes to the outside.
$ kubectl -n starrocks get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
starrockscluster-sample-be-search ClusterIP None <none> 9050/TCP 6m39s
starrockscluster-sample-be-service ClusterIP 10.96.86.207 <none> 9060/TCP,8040/TCP,9050/TCP,8060/TCP 6m39s
starrockscluster-sample-fe-search ClusterIP None <none> 9030/TCP 8m
starrockscluster-sample-fe-service LoadBalancer 10.96.26.146 a7509284bf3784983a596c6eec7fc212-618xxxxxx.us-west-2.elb.amazonaws.com 8030:30028/TCP,9020:32241/TCP,9030:32640/TCP,9010:32384/TCP 8m
Log in to your machine host and access the StarRocks cluster by using the MySQL client.
mysql -h a7509284bf3784983a596c6eec7fc212-618xxxxxx.us-west-2.elb.amazonaws.com -P9030 -uroot
From outside the Kubernetes cluster, you can access the StarRocks cluster through the FE Service’s port forwarding.
kubectl command-line tool and configured access to the Kubernetes cluster.kubectl -n starrocks port-forward service/starrockscluster-sample-fe-service 9030:9030 to forward
local port 9030 to FE Service’s port 9030. mysql -h 127.0.0.1 -P9030 -uroot
Run the following command to specify a new BE image file, such as starrocks/be-ubuntu:latest:
kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksBeSpec":{"image":"starrocks/be-ubuntu:latest"}}}'
Run the following command to specify a new FE image file, such as starrocks/fe-ubuntu:latest:
kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksFeSpec":{"image":"starrocks/fe-ubuntu:latest"}}}'
The upgrade process lasts for a while. You can run the command kubectl -n starrocks get pods to view the upgrade
progress.
This topic takes scaling out the BE and FE clusters as examples.
Run the following command to scale out the BE cluster to 9 nodes:
kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksBeSpec":{"replicas":9}}}'
Run the following command to scale out the FE cluster to 4 nodes:
kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksFeSpec":{"replicas":4}}}'
The scaling process lasts for a while. You can use the command kubectl -n starrocks get pods to view the scaling
progress.
Add cautions on scale-in FE nodes:
FE nodes can be scaled-in, but there are some limitations:
The official images contains default application configuration file, however, they can be overwritten by configuring kubernetes configmap deployment crd.
You can generate the configmap from an StarRocks configuration file.
Below is an example of creating a Kubernetes configmap fe-config-map from the fe.conf configuration file. You can do
the same with BE and CN.
# create fe-config-map from starrocks/fe/conf/fe.conf file
kubectl create configmap fe-config-map --from-file=starrocks/fe/conf/fe.conf
Once the configmap is created, you can reference the configmap in the yaml file. For example:
# fe use configmap example
starRocksFeSpec:
configMapInfo:
configMapName: fe-config-map
resolveKey: fe.conf
# cn use configmap example
starRocksCnSpec:
configMapInfo:
configMapName: cn-config-map
resolveKey: cn.conf
# be use configmap example
starRocksBeSpec:
configMapInfo:
configMapName: be-config-map
resolveKey: be.conf
Issue description: When a custom resource StarRocksCluster is installed using kubectl apply -f xxx, an error is
returned The CustomResourceDefinition 'starrocksclusters.starrocks.com' is invalid: metadata.annotations: Too long: must have at most 262144 bytes.
Cause analysis: Whenever kubectl apply -f xxx is used to create or update resources, a metadata
annotation kubectl.kubernetes.io/last-applied-configuration is added. This metadata annotation is in JSON format and
records the last-applied-configuration. kubectl apply -f xxx” is suitable for most cases, but in rare situations ,
such as when the configuration file for the custom resource is too large, it may cause the size of the metadata
annotation to exceed the limit.
Solution: If you install the custom resource StarRocksCluster for the first time, it is recommended to
use kubectl create -f xxx. If the custom resource StarRocksCluster is already installed in the environment, and you
need to update its configuration, it is recommended to use kubectl replace -f xxx.