✘ zayden@Zaydenui-MacBookPro ~ gcloud container clusters get-credentials autopilot-cluster-1 --region asia-east1 --project kubernetes-project-386708
WARNING: Python 3.5-3.7 will be deprecated on August 8th, 2023. Please use Python version 3.8 and up.
If you have a compatible Python interpreter installed, you can use it by setting
the CLOUDSDK_PYTHON environment variable to point to it.
Fetching cluster endpoint and auth data.
ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=403, message=Required "container.clusters.get" permission(s) for "projects/kubernetes-project-386708/locations/asia-east1/clusters/autopilot-cluster-1".
계정 인증 및 프로젝트 연동
✘ zayden@Zaydenui-MacBookPro ~ gcloud init
WARNING: Python 3.5-3.7 will be deprecated on August 8th, 2023. Please use Python version 3.8 and up.
If you have a compatible Python interpreter installed, you can use it by setting
the CLOUDSDK_PYTHON environment variable to point to it.
Welcome! This command will take you through the configuration of gcloud.
Settings from your current configuration [default] are:
core:
account: seojeonghyeon6958@gmail.com
disable_usage_reporting: 'False'
project: kubernetes-project-386708
Pick configuration to use:
[1] Re-initialize this configuration [default] with new settings
[2] Create a new configuration
Please enter your numeric choice: 2
Enter configuration name. Names start with a lower case letter and contain only lower case letters a-z, digits 0-9, and hyphens '-': autoever.seojeonghyeon0630@gmail.com
ERROR: (gcloud.init) Invalid name [autoever.seojeonghyeon0630@gmail.com] for a configuration. Except for special cases (NONE), configuration names start with a lower case letter and contain only lower case letters a-z, digits 0-9, and hyphens '-'.
✘ zayden@Zaydenui-MacBookPro ~ gcloud init
WARNING: Python 3.5-3.7 will be deprecated on August 8th, 2023. Please use Python version 3.8 and up.
If you have a compatible Python interpreter installed, you can use it by setting
the CLOUDSDK_PYTHON environment variable to point to it.
Welcome! This command will take you through the configuration of gcloud.
Settings from your current configuration [default] are:
core:
account: seojeonghyeon6958@gmail.com
disable_usage_reporting: 'False'
project: kubernetes-project-386708
Pick configuration to use:
[1] Re-initialize this configuration [default] with new settings
[2] Create a new configuration
Please enter your numeric choice: 2
Enter configuration name. Names start with a lower case letter and contain only lower case letters a-z, digits 0-9, and hyphens '-':autoever.seojeonghyeon0630
ERROR: (gcloud.init) Invalid name [autoever.seojeonghyeon0630] for a configuration. Except for special cases (NONE), configuration names start with a lower case letter and contain only lower case letters a-z, digits 0-9, and hyphens '-'.
✘ zayden@Zaydenui-MacBookPro ~ gcloud config configurations list
WARNING: Python 3.5-3.7 will be deprecated on August 8th, 2023. Please use Python version 3.8 and up.
If you have a compatible Python interpreter installed, you can use it by setting
the CLOUDSDK_PYTHON environment variable to point to it.
NAME IS_ACTIVE ACCOUNT PROJECT COMPUTE_DEFAULT_ZONE COMPUTE_DEFAULT_REGION
default True seojeonghyeon6958@gmail.com kubernetes-project-386708
zayden@Zaydenui-MacBookPro ~ gcloud init
WARNING: Python 3.5-3.7 will be deprecated on August 8th, 2023. Please use Python version 3.8 and up.
If you have a compatible Python interpreter installed, you can use it by setting
the CLOUDSDK_PYTHON environment variable to point to it.
Welcome! This command will take you through the configuration of gcloud.
Settings from your current configuration [default] are:
core:
account: seojeonghyeon6958@gmail.com
disable_usage_reporting: 'False'
project: kubernetes-project-386708
Pick configuration to use:
[1] Re-initialize this configuration [default] with new settings
[2] Create a new configuration
Please enter your numeric choice: 2
Enter configuration name. Names start with a lower case letter and contain only lower case
letters a-z, digits 0-9, and hyphens '-': company-account
Your current configuration has been set to: [company-account]
You can skip diagnostics next time by using the following flag:
gcloud init --skip-diagnostics
Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).
Choose the account you would like to use to perform operations for this configuration:
[1] seojeonghyeon6958@gmail.com
[2] Log in with a new account
Please enter your numeric choice: 2
Your browser has been opened to visit:
<https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=32555940559.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8085%2F&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fappengine.admin+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.login+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcompute+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&state=pPTwqKVbMQBYlyxw2lM3wQ8RtJxcZk&access_type=offline&code_challenge=QJAKwIrUBr0O7ocO4XMNib6dX_Jy8TYsEL9b6OmfZ7M&code_challenge_method=S256>
You are logged in as: [autoever.seojeonghyeon0630@gmail.com].
Pick cloud project to use:
[1] kubernetes-project-386708
[2] tokyo-baton-386708
[3] Enter a project ID
[4] Create a new project
Please enter numeric choice or text value (must exactly match list item): 1
Your current project has been set to: [kubernetes-project-386708].
Do you want to configure a default Compute Region and Zone? (Y/n)? Y
Which Google Compute Engine zone would you like to use as project default?
If you do not specify a zone via a command line flag while working with Compute Engine
resources, the default is assumed.
[1] us-east1-b
[2] us-east1-c
[3] us-east1-d
[4] us-east4-c
[5] us-east4-b
[6] us-east4-a
[7] us-central1-c
[8] us-central1-a
[9] us-central1-f
[10] us-central1-b
[11] us-west1-b
[12] us-west1-c
[13] us-west1-a
[14] europe-west4-a
[15] europe-west4-b
[16] europe-west4-c
[17] europe-west1-b
[18] europe-west1-d
[19] europe-west1-c
[20] europe-west3-c
[21] europe-west3-a
[22] europe-west3-b
[23] europe-west2-c
[24] europe-west2-b
[25] europe-west2-a
[26] asia-east1-b
[27] asia-east1-a
[28] asia-east1-c
[29] asia-southeast1-b
[30] asia-southeast1-a
[31] asia-southeast1-c
[32] asia-northeast1-b
[33] asia-northeast1-c
[34] asia-northeast1-a
[35] asia-south1-c
[36] asia-south1-b
[37] asia-south1-a
[38] australia-southeast1-b
[39] australia-southeast1-c
[40] australia-southeast1-a
[41] southamerica-east1-b
[42] southamerica-east1-c
[43] southamerica-east1-a
[44] asia-east2-a
[45] asia-east2-b
[46] asia-east2-c
[47] asia-northeast2-a
[48] asia-northeast2-b
[49] asia-northeast2-c
[50] asia-northeast3-a
Did not print [63] options.
Too many options [113]. Enter "list" at prompt to print choices fully.
Please enter numeric choice or text value (must exactly match list item): 26
Your project default Compute Engine zone has been set to [asia-east1-b].
You can change it by running [gcloud config set compute/zone NAME].
Your project default Compute Engine region has been set to [asia-east1].
You can change it by running [gcloud config set compute/region NAME].
Your Google Cloud SDK is configured and ready to use!
* Commands that require authentication will use autoever.seojeonghyeon0630@gmail.com by default
* Commands will reference project `kubernetes-project-386708` by default
* Compute Engine commands will use region `asia-east1` by default
* Compute Engine commands will use zone `asia-east1-b` by default
Run `gcloud help config` to learn how to change individual settings
This gcloud configuration is called [company-account]. You can create additional configurations if you work with multiple accounts and/or projects.
Run `gcloud topic configurations` to learn more.
Some things to try next:
* Run `gcloud --help` to see the Cloud Platform services you can interact with. And run `gcloud help COMMAND` to get help on any gcloud command.
* Run `gcloud topic --help` to learn about advanced features of the SDK like arg files and output formatting
* Run `gcloud cheat-sheet` to see a roster of go-to `gcloud` commands.
zayden@Zaydenui-MacBookPro ~ gcloud config configurations list
WARNING: Python 3.5-3.7 will be deprecated on August 8th, 2023. Please use Python version 3.8 and up.
If you have a compatible Python interpreter installed, you can use it by setting
the CLOUDSDK_PYTHON environment variable to point to it.
NAME IS_ACTIVE ACCOUNT PROJECT COMPUTE_DEFAULT_ZONE COMPUTE_DEFAULT_REGION
company-account True autoever.seojeonghyeon0630@gmail.com kubernetes-project-386708 asia-east1-b asia-east1
default False seojeonghyeon6958@gmail.com kubernetes-project-386708
연동 결과
zayden@Zaydenui-MacBookPro ~ gcloud container clusters get-credentials autopilot-cluster-1 --region asia-east1
WARNING: Python 3.5-3.7 will be deprecated on August 8th, 2023. Please use Python version 3.8 and up.
If you have a compatible Python interpreter installed, you can use it by setting
the CLOUDSDK_PYTHON environment variable to point to it.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for autopilot-cluster-1.
# Cluster에 대한 인증정보 가져오기,
gcloud container clusters get-credentials 클러스터명 \\
--region 지역명
gcloud container clusters get-credentials autopilot-cluster-1 \\
--region asia-south1
#Google Cloud CLI에서 Default Project 설정 gcloud config set project 프로젝트ID
gcloud config set project kubernetestest-369102
정상적으로 GKE와 연동되었는지 확인
# gcloud에서 파드 정보를 가져오는 것을 확인
zayden@Zaydenui-MacBookPro ~/Documents/Work/GoogleCloud kubectl get pods
W1119 12:52:37.896920 9670 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
No resources found in default namespace.
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ vi logapp.yml
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl create -f logapp.yml -n gw
deployment.apps/logapp created
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get all -n gw
NAME READY STATUS RESTARTS AGE
pod/logapp-7cb56cb45f-5jf62 0/1 ContainerCreating 0 4m3s
pod/logapp-7cb56cb45f-p8hjw 1/1 Running 0 4m3s
pod/logapp-7cb56cb45f-rk6vp 0/1 ContainerCreating 0 4m3s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/logapp 1/3 3 1 4m5s
NAME DESIRED CURRENT READY AGE
replicaset.apps/logapp-7cb56cb45f 3 3 1 4m5s
3) PV에 잘 저장되는지 어떻게 확인할까?
PV에 저장이 된다면 LogApp Deployment를 삭제했을 때에도 Log를 저장하는 위치에 이전 로그가 남아있어야 합니다.
3. Filebeat
1) Filebeat 설정을 수정하기 위해 yml파일 Download
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ curl -O <https://raw.githubusercontent.com/elastic/helm-charts/main/filebeat/values.yaml>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 7133 100 7133 0 0 14985 0 --:--:-- --:--:-- --:--:-- 14985
2) Deployment로 Filebeat 생성하도록 yml파일 작성(Daemonset과 Deployment 중 선택 가능, yml 파일 참고)
---
daemonset:
# Annotations to apply to the daemonset
annotations: {}
# additionals labels
labels: {}
affinity: {}
# Include the daemonset
enabled: false
# Extra environment variables for Filebeat container.
envFrom: []
# - configMapRef:
# name: config-secret
extraEnvs: []
#- name: "ELASTICSEARCH_USERNAME"
#valueFrom:
#secretKeyRef:
#name: elasticsearch-master-credentials
#key: username
#- name: "ELASTICSEARCH_PASSWORD"
#valueFrom:
#secretKeyRef:
#name: elasticsearch-master-credentials
#key: password
# Allows you to add any config files in /usr/share/filebeat
extraVolumes: []
# - name: extras
# emptyDir: {}
extraVolumeMounts: []
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
hostNetworking: false
# Allows you to add any config files in /usr/share/filebeat
# such as filebeat.yml for daemonset
filebeatConfig:
filebeat.yml: |
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
output.elasticsearch:
host: '${NODE_NAME}'
hosts: '["{ELASTICSEARCH_HOSTS:elasticsearch-master:9200}"]'
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]
# Only used when updateStrategy is set to "RollingUpdate"
maxUnavailable: 1
nodeSelector: {}
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security other sensitive values
secretMounts: {}
#- name: elasticsearch-master-certs
#secretName: elasticsearch-master-certs
#path: /usr/share/filebeat/certs/
# - name: filebeat-certificates
# secretName: filebeat-certificates
# path: /usr/share/filebeat/certs
# Various pod security context settings. Bear in mind that many of these have an impact on Filebeat functioning properly.
#
# - User that the container will execute as. Typically necessary to run as root (0) in order to properly collect host container logs.
# - Whether to execute the Filebeat containers as privileged containers. Typically not necessarily unless running within environments such as OpenShift.
securityContext:
runAsUser: 0
privileged: false
resources:
requests:
cpu: "100m"
memory: "100Mi"
limits:
cpu: "1000m"
memory: "200Mi"
tolerations: []
**deployment:**
# Annotations to apply to the deployment
annotations: {}
# additionals labels
labels: {}
affinity: {}
# Include the deployment
**enabled: true**
# Extra environment variables for Filebeat container.
envFrom: []
# - configMapRef:
# name: config-secret
extraEnvs: []
#- name: "ELASTICSEARCH_USERNAME"
#valueFrom:
#secretKeyRef:
#name: elasticsearch-master-credentials
#key: username
#- name: "ELASTICSEARCH_PASSWORD"
#valueFrom:
#secretKeyRef:
#name: elasticsearch-master-credentials
#key: password
# Allows you to add any config files in /usr/share/filebeat
**extraVolumes:
- name: extras
persistentVolumeClaim:
claimName: logstorage-pvc**
**extraVolumeMounts:
- name: extras
mountPath: /usr/share/extras
readOnly: true**
**filebeatConfig:
filebeat.yml: |
filebeat.inputs:
- type: log
paths :
- /usr/share/extras/transaction.log
fields :
log_format : "json"
kafka_topic : "kafkastudy-gkelog-transaction"
filebeat.config.modules :
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.kafka:
hosts: ["kafka-master.messagequeue.svc:9092"]
topic: '%{[fields.kafka_topic]}'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: snappy
max_message_bytes: 1000000
processors:
- add_host_metadata: ~**
nodeSelector: {}
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security other sensitive values
secretMounts: {}
#- name: elasticsearch-master-certs
#secretName: elasticsearch-master-certs
#path: /usr/share/filebeat/certs/
# - name: filebeat-certificates
# secretName: filebeat-certificates
# path: /usr/share/filebeat/certs
#
# - User that the container will execute as.
# Not necessary to run as root (0) as the Filebeat Deployment use cases do not need access to Kubernetes Node internals
# - Typically not necessarily unless running within environments such as OpenShift.
securityContext:
runAsUser: 0
privileged: false
resources:
requests:
cpu: "100m"
memory: "100Mi"
limits:
cpu: "1000m"
memory: "200Mi"
tolerations: []
# Replicas being used for the filebeat deployment
replicas: 1
extraContainers: ""
# - name: dummy-init
# image: busybox
# command: ['echo', 'hey']
extraInitContainers: []
# - name: dummy-init
# Root directory where Filebeat will write data to in order to persist registry data across pod restarts (file position and other metadata).
hostPathRoot: /var/lib
dnsConfig: {}
# options:
# - name: ndots
# value: "2"
hostAliases: []
#- ip: "127.0.0.1"
# hostnames:
# - "foo.local"
# - "bar.local"
image: "docker.elastic.co/beats/filebeat"
imageTag: "8.5.1"
imagePullPolicy: "IfNotPresent"
imagePullSecrets: []
livenessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
curl --fail 127.0.0.1:5066
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
filebeat test output
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
# Whether this chart should self-manage its service account, role, and associated role binding.
managedServiceAccount: true
clusterRoleRules:
- apiGroups:
- ""
resources:
- namespaces
- nodes
- pods
verbs:
- get
- list
- watch
- apiGroups:
- "apps"
resources:
- replicasets
verbs:
- get
- list
- watch
podAnnotations: {}
# iam.amazonaws.com/role: es-cluster
# Custom service account override that the pod will use
serviceAccount: ""
# Annotations to add to the ServiceAccount that is created if the serviceAccount value isn't set.
serviceAccountAnnotations: {}
# eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/k8s.clustername.namespace.serviceaccount
# How long to wait for Filebeat pods to stop gracefully
terminationGracePeriod: 30
# This is the PriorityClass settings as defined in
# <https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass>
priorityClassName: ""
updateStrategy: RollingUpdate
# Override various naming aspects of this chart
# Only edit these if you know what you're doing
nameOverride: ""
fullnameOverride: ""
*현재 생성한 LogApp과 Filebeat는 Kafka와 다른 Namespace에 존재하게 된다. 실제 운영에서도 Application과 ELK 담당자는 나눠져 있다. Filebeat가 다른 Namespace에 접근하기 위해 kafka-master.es.svc 로 표기함으로 어느 Namespace에 Service가 존재하는 지 알 수 있게 해준다.
3) Filebeat 생성 및 상태 확인
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get all -n gw
NAME READY STATUS RESTARTS AGE
**pod/filebeat-filebeat-86b7b849df-df9zv 0/1 Running 0 16s**
pod/logapp-5fb9fbc98b-2l5cp 0/1 ContainerCreating 0 77m
pod/logapp-5fb9fbc98b-5mp8h 0/1 ContainerCreating 0 77m
pod/logapp-5fb9fbc98b-q68m6 1/1 Running 0 77m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 19h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/filebeat-filebeat 0/1 1 0 16s
deployment.apps/logapp 1/3 3 1 77m
NAME DESIRED CURRENT READY AGE
replicaset.apps/filebeat-filebeat-86b7b849df 1 1 0 16s
replicaset.apps/logapp-5fb9fbc98b 3 3 1 77m
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl describe pod filebeat-filebeat-86b7b849df-df9zv -n gw
Name: filebeat-filebeat-86b7b849df-df9zv
Namespace: default
Priority: 0
Service Account: filebeat-filebeat
Node: gke-elk-cluster-default-pool-4bfe4779-314z/10.140.0.8
Start Time: Sun, 21 May 2023 01:51:39 +0000
Labels: app=filebeat-filebeat
chart=filebeat-8.5.1
pod-template-hash=86b7b849df
release=filebeat
Annotations: configChecksum: 9f77fcccf152c8e496e8ccf15fdb152aea28689be218150b114bf7ddc4cd034
Status: Running
IP: 10.92.1.16
IPs:
IP: 10.92.1.16
Controlled By: ReplicaSet/filebeat-filebeat-86b7b849df
Containers:
filebeat:
Container ID: containerd://61eaac49d9b50452af1d0ad4600a8cf9a2c3b3e475ce3e968887ee97f52f5cff
Image: docker.elastic.co/beats/filebeat:8.5.1
Image ID: docker.elastic.co/beats/filebeat@sha256:2e7a537a8a2c3d2767009ee65faadce572bc7ec5e571aba730f11a236da93a3a
Port: <none>
Host Port: <none>
Args:
-e
-E
http.enabled=true
State: Running
Started: Sun, 21 May 2023 01:51:40 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 200Mi
Requests:
cpu: 100m
memory: 100Mi
Liveness: exec [sh -c #!/usr/bin/env bash -e
curl --fail 127.0.0.1:5066
] delay=10s timeout=5s period=10s #success=1 #failure=3
Readiness: exec [sh -c #!/usr/bin/env bash -e
filebeat test output
] delay=10s timeout=5s period=10s #success=1 #failure=3
Environment:
POD_NAMESPACE: default (v1:metadata.namespace)
Mounts:
/usr/share/extras from extras (ro)
/usr/share/filebeat/filebeat.yml from filebeat-config (ro,path="filebeat.yml")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-csdsk (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
filebeat-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: filebeat-filebeat-deployment-config
Optional: false
**extras:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: logstorage-pvc
ReadOnly: false**
kube-api-access-csdsk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 33s default-scheduler Successfully assigned default/filebeat-filebeat-86b7b849df-df9zv to gke-elk-cluster-default-pool-4bfe4779-314z
Normal Pulled 33s kubelet Container image "docker.elastic.co/beats/filebeat:8.5.1" already present on machine
Normal Created 32s kubelet Created container filebeat
Normal Started 32s kubelet Started container filebeat
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get all -n gw
NAME READY STATUS RESTARTS AGE
**pod/filebeat-filebeat-86b7b849df-df9zv 1/1 Running 0 56s**
pod/logapp-5fb9fbc98b-2l5cp 0/1 ContainerCreating 0 78m
pod/logapp-5fb9fbc98b-5mp8h 0/1 ContainerCreating 0 78m
pod/logapp-5fb9fbc98b-q68m6 1/1 Running 0 78m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 19h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/filebeat-filebeat 1/1 1 1 56s
deployment.apps/logapp 1/3 3 1 78m
NAME DESIRED CURRENT READY AGE
replicaset.apps/filebeat-filebeat-86b7b849df 1 1 1 56s
replicaset.apps/logapp-5fb9fbc98b 3 3 1 78m
4) Kafka까지 인입이 잘되는 지 확인
I have no name!@kafka-master-client:/$ kafka-console-consumer.sh --bootstrap-server kafka-master:9092 --topic kafkastudy-gkelog-transaction
{"@timestamp":"2023-05-21T01:59:55.443Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.5.1"},"fields":{"log_format":"json","kafka_topic":"kafkastudy-gkelog-transaction"},"host":{"os":{"codename":"focal","type":"linux","platform":"ubuntu","version":"20.04.5 LTS (Focal Fossa)","family":"debian","name":"Ubuntu","kernel":"5.15.89+"},"name":"filebeat-filebeat-86b7b849df-df9zv","containerized":true,"ip":["10.92.1.16"],"mac":["2A-6C-BF-8F-4E-34"],"hostname":"filebeat-filebeat-86b7b849df-df9zv","architecture":"x86_64"},"agent":{"version":"8.5.1","ephemeral_id":"e00c0689-2c64-46cb-b4d0-5feda709b2b3","id":"4fe495d7-56ed-4c04-9561-4251184e5227","name":"filebeat-filebeat-86b7b849df-df9zv","type":"filebeat"},"ecs":{"version":"8.0.0"},"log":{"file":{"path":"/usr/share/extras/transaction.log"},"offset":1355},"message":"{\\"transactionid\\":\\"5c8e23f3-3020-425a-a614-06d0c96fa2cf\\",\\"userId\\":\\"seojeonghyeon\\",\\"message\\":\\"Hello\\",\\"currentTime\\":\\"2023-05-21T10:59:54.597872686\\"}","input":{"type":"log"}}
{"@timestamp":"2023-05-21T02:01:00.449Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.5.1"},"agent":{"ephemeral_id":"e00c0689-2c64-46cb-b4d0-5feda709b2b3","id":"4fe495d7-56ed-4c04-9561-4251184e5227","name":"filebeat-filebeat-86b7b849df-df9zv","type":"filebeat","version":"8.5.1"},"ecs":{"version":"8.0.0"},"host":{"os":{"family":"debian","name":"Ubuntu","kernel":"5.15.89+","codename":"focal","type":"linux","platform":"ubuntu","version":"20.04.5 LTS (Focal Fossa)"},"containerized":true,"ip":["10.92.1.16"],"mac":["2A-6C-BF-8F-4E-34"],"name":"filebeat-filebeat-86b7b849df-df9zv","hostname":"filebeat-filebeat-86b7b849df-df9zv","architecture":"x86_64"},"log":{"file":{"path":"/usr/share/extras/transaction.log"},"offset":1501},"message":"{\\"transactionid\\":\\"c438b18a-c28f-4f60-9fc4-1f7044465d0c\\",\\"userId\\":\\"Kristen\\",\\"message\\":\\"How are you?\\",\\"currentTime\\":\\"2023-05-21T11:00:50.490580785\\"}","input":{"type":"log"},"fields":{"log_format":"json","kafka_topic":"kafkastudy-gkelog-transaction"}}
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ vi logapp.yml
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl create -f logapp.yml -n gw
deployment.apps/logapp created
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get all -n gw
NAME READY STATUS RESTARTS AGE
pod/logapp-7cb56cb45f-5jf62 0/1 ContainerCreating 0 4m3s
pod/logapp-7cb56cb45f-p8hjw 1/1 Running 0 4m3s
pod/logapp-7cb56cb45f-rk6vp 0/1 ContainerCreating 0 4m3s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/logapp 1/3 3 1 4m5s
NAME DESIRED CURRENT READY AGE
replicaset.apps/logapp-7cb56cb45f 3 3 1 4m5s
3) PV에 잘 저장되는지 어떻게 확인할까?
PV에 저장이 된다면 LogApp Deployment를 삭제했을 때에도 Log를 저장하는 위치에 이전 로그가 남아있어야 합니다.
3. Filebeat
1) Filebeat 설정을 수정하기 위해 yml파일 Download
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ curl -O <https://raw.githubusercontent.com/elastic/helm-charts/main/filebeat/values.yaml>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 7133 100 7133 0 0 14985 0 --:--:-- --:--:-- --:--:-- 14985
2) Deployment로 Filebeat 생성하도록 yml파일 작성(Daemonset과 Deployment 중 선택 가능, yml 파일 참고)
---
daemonset:
# Annotations to apply to the daemonset
annotations: {}
# additionals labels
labels: {}
affinity: {}
# Include the daemonset
enabled: false
# Extra environment variables for Filebeat container.
envFrom: []
# - configMapRef:
# name: config-secret
extraEnvs: []
#- name: "ELASTICSEARCH_USERNAME"
#valueFrom:
#secretKeyRef:
#name: elasticsearch-master-credentials
#key: username
#- name: "ELASTICSEARCH_PASSWORD"
#valueFrom:
#secretKeyRef:
#name: elasticsearch-master-credentials
#key: password
# Allows you to add any config files in /usr/share/filebeat
extraVolumes: []
# - name: extras
# emptyDir: {}
extraVolumeMounts: []
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
hostNetworking: false
# Allows you to add any config files in /usr/share/filebeat
# such as filebeat.yml for daemonset
filebeatConfig:
filebeat.yml: |
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
output.elasticsearch:
host: '${NODE_NAME}'
hosts: '["{ELASTICSEARCH_HOSTS:elasticsearch-master:9200}"]'
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]
# Only used when updateStrategy is set to "RollingUpdate"
maxUnavailable: 1
nodeSelector: {}
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security other sensitive values
secretMounts: {}
#- name: elasticsearch-master-certs
#secretName: elasticsearch-master-certs
#path: /usr/share/filebeat/certs/
# - name: filebeat-certificates
# secretName: filebeat-certificates
# path: /usr/share/filebeat/certs
# Various pod security context settings. Bear in mind that many of these have an impact on Filebeat functioning properly.
#
# - User that the container will execute as. Typically necessary to run as root (0) in order to properly collect host container logs.
# - Whether to execute the Filebeat containers as privileged containers. Typically not necessarily unless running within environments such as OpenShift.
securityContext:
runAsUser: 0
privileged: false
resources:
requests:
cpu: "100m"
memory: "100Mi"
limits:
cpu: "1000m"
memory: "200Mi"
tolerations: []
**deployment:**
# Annotations to apply to the deployment
annotations: {}
# additionals labels
labels: {}
affinity: {}
# Include the deployment
**enabled: true**
# Extra environment variables for Filebeat container.
envFrom: []
# - configMapRef:
# name: config-secret
extraEnvs: []
#- name: "ELASTICSEARCH_USERNAME"
#valueFrom:
#secretKeyRef:
#name: elasticsearch-master-credentials
#key: username
#- name: "ELASTICSEARCH_PASSWORD"
#valueFrom:
#secretKeyRef:
#name: elasticsearch-master-credentials
#key: password
# Allows you to add any config files in /usr/share/filebeat
**extraVolumes:
- name: extras
persistentVolumeClaim:
claimName: logstorage-pvc**
**extraVolumeMounts:
- name: extras
mountPath: /usr/share/extras
readOnly: true**
**filebeatConfig:
filebeat.yml: |
filebeat.inputs:
- type: log
paths :
- /usr/share/extras/transaction.log
fields :
log_format : "json"
kafka_topic : "kafkastudy-gkelog-transaction"
filebeat.config.modules :
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.kafka:
hosts: ["kafka-master.messagequeue.svc:9092"]
topic: '%{[fields.kafka_topic]}'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: snappy
max_message_bytes: 1000000
processors:
- add_host_metadata: ~**
nodeSelector: {}
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security other sensitive values
secretMounts: {}
#- name: elasticsearch-master-certs
#secretName: elasticsearch-master-certs
#path: /usr/share/filebeat/certs/
# - name: filebeat-certificates
# secretName: filebeat-certificates
# path: /usr/share/filebeat/certs
#
# - User that the container will execute as.
# Not necessary to run as root (0) as the Filebeat Deployment use cases do not need access to Kubernetes Node internals
# - Typically not necessarily unless running within environments such as OpenShift.
securityContext:
runAsUser: 0
privileged: false
resources:
requests:
cpu: "100m"
memory: "100Mi"
limits:
cpu: "1000m"
memory: "200Mi"
tolerations: []
# Replicas being used for the filebeat deployment
replicas: 1
extraContainers: ""
# - name: dummy-init
# image: busybox
# command: ['echo', 'hey']
extraInitContainers: []
# - name: dummy-init
# Root directory where Filebeat will write data to in order to persist registry data across pod restarts (file position and other metadata).
hostPathRoot: /var/lib
dnsConfig: {}
# options:
# - name: ndots
# value: "2"
hostAliases: []
#- ip: "127.0.0.1"
# hostnames:
# - "foo.local"
# - "bar.local"
image: "docker.elastic.co/beats/filebeat"
imageTag: "8.5.1"
imagePullPolicy: "IfNotPresent"
imagePullSecrets: []
livenessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
curl --fail 127.0.0.1:5066
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
filebeat test output
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
# Whether this chart should self-manage its service account, role, and associated role binding.
managedServiceAccount: true
clusterRoleRules:
- apiGroups:
- ""
resources:
- namespaces
- nodes
- pods
verbs:
- get
- list
- watch
- apiGroups:
- "apps"
resources:
- replicasets
verbs:
- get
- list
- watch
podAnnotations: {}
# iam.amazonaws.com/role: es-cluster
# Custom service account override that the pod will use
serviceAccount: ""
# Annotations to add to the ServiceAccount that is created if the serviceAccount value isn't set.
serviceAccountAnnotations: {}
# eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/k8s.clustername.namespace.serviceaccount
# How long to wait for Filebeat pods to stop gracefully
terminationGracePeriod: 30
# This is the PriorityClass settings as defined in
# <https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass>
priorityClassName: ""
updateStrategy: RollingUpdate
# Override various naming aspects of this chart
# Only edit these if you know what you're doing
nameOverride: ""
fullnameOverride: ""
*현재 생성한 LogApp과 Filebeat는 Kafka와 다른 Namespace에 존재하게 된다. 실제 운영에서도 Application과 ELK 담당자는 나눠져 있다. Filebeat가 다른 Namespace에 접근하기 위해 kafka-master.es.svc 로 표기함으로 어느 Namespace에 Service가 존재하는 지 알 수 있게 해준다.
3) Filebeat 생성 및 상태 확인
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get all -n gw
NAME READY STATUS RESTARTS AGE
**pod/filebeat-filebeat-86b7b849df-df9zv 0/1 Running 0 16s**
pod/logapp-5fb9fbc98b-2l5cp 0/1 ContainerCreating 0 77m
pod/logapp-5fb9fbc98b-5mp8h 0/1 ContainerCreating 0 77m
pod/logapp-5fb9fbc98b-q68m6 1/1 Running 0 77m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 19h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/filebeat-filebeat 0/1 1 0 16s
deployment.apps/logapp 1/3 3 1 77m
NAME DESIRED CURRENT READY AGE
replicaset.apps/filebeat-filebeat-86b7b849df 1 1 0 16s
replicaset.apps/logapp-5fb9fbc98b 3 3 1 77m
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl describe pod filebeat-filebeat-86b7b849df-df9zv -n gw
Name: filebeat-filebeat-86b7b849df-df9zv
Namespace: default
Priority: 0
Service Account: filebeat-filebeat
Node: gke-elk-cluster-default-pool-4bfe4779-314z/10.140.0.8
Start Time: Sun, 21 May 2023 01:51:39 +0000
Labels: app=filebeat-filebeat
chart=filebeat-8.5.1
pod-template-hash=86b7b849df
release=filebeat
Annotations: configChecksum: 9f77fcccf152c8e496e8ccf15fdb152aea28689be218150b114bf7ddc4cd034
Status: Running
IP: 10.92.1.16
IPs:
IP: 10.92.1.16
Controlled By: ReplicaSet/filebeat-filebeat-86b7b849df
Containers:
filebeat:
Container ID: containerd://61eaac49d9b50452af1d0ad4600a8cf9a2c3b3e475ce3e968887ee97f52f5cff
Image: docker.elastic.co/beats/filebeat:8.5.1
Image ID: docker.elastic.co/beats/filebeat@sha256:2e7a537a8a2c3d2767009ee65faadce572bc7ec5e571aba730f11a236da93a3a
Port: <none>
Host Port: <none>
Args:
-e
-E
http.enabled=true
State: Running
Started: Sun, 21 May 2023 01:51:40 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 200Mi
Requests:
cpu: 100m
memory: 100Mi
Liveness: exec [sh -c #!/usr/bin/env bash -e
curl --fail 127.0.0.1:5066
] delay=10s timeout=5s period=10s #success=1 #failure=3
Readiness: exec [sh -c #!/usr/bin/env bash -e
filebeat test output
] delay=10s timeout=5s period=10s #success=1 #failure=3
Environment:
POD_NAMESPACE: default (v1:metadata.namespace)
Mounts:
/usr/share/extras from extras (ro)
/usr/share/filebeat/filebeat.yml from filebeat-config (ro,path="filebeat.yml")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-csdsk (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
filebeat-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: filebeat-filebeat-deployment-config
Optional: false
**extras:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: logstorage-pvc
ReadOnly: false**
kube-api-access-csdsk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 33s default-scheduler Successfully assigned default/filebeat-filebeat-86b7b849df-df9zv to gke-elk-cluster-default-pool-4bfe4779-314z
Normal Pulled 33s kubelet Container image "docker.elastic.co/beats/filebeat:8.5.1" already present on machine
Normal Created 32s kubelet Created container filebeat
Normal Started 32s kubelet Started container filebeat
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get all -n gw
NAME READY STATUS RESTARTS AGE
**pod/filebeat-filebeat-86b7b849df-df9zv 1/1 Running 0 56s**
pod/logapp-5fb9fbc98b-2l5cp 0/1 ContainerCreating 0 78m
pod/logapp-5fb9fbc98b-5mp8h 0/1 ContainerCreating 0 78m
pod/logapp-5fb9fbc98b-q68m6 1/1 Running 0 78m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 19h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/filebeat-filebeat 1/1 1 1 56s
deployment.apps/logapp 1/3 3 1 78m
NAME DESIRED CURRENT READY AGE
replicaset.apps/filebeat-filebeat-86b7b849df 1 1 1 56s
replicaset.apps/logapp-5fb9fbc98b 3 3 1 78m
4) Kafka까지 인입이 잘되는 지 확인
I have no name!@kafka-master-client:/$ kafka-console-consumer.sh --bootstrap-server kafka-master:9092 --topic kafkastudy-gkelog-transaction
{"@timestamp":"2023-05-21T01:59:55.443Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.5.1"},"fields":{"log_format":"json","kafka_topic":"kafkastudy-gkelog-transaction"},"host":{"os":{"codename":"focal","type":"linux","platform":"ubuntu","version":"20.04.5 LTS (Focal Fossa)","family":"debian","name":"Ubuntu","kernel":"5.15.89+"},"name":"filebeat-filebeat-86b7b849df-df9zv","containerized":true,"ip":["10.92.1.16"],"mac":["2A-6C-BF-8F-4E-34"],"hostname":"filebeat-filebeat-86b7b849df-df9zv","architecture":"x86_64"},"agent":{"version":"8.5.1","ephemeral_id":"e00c0689-2c64-46cb-b4d0-5feda709b2b3","id":"4fe495d7-56ed-4c04-9561-4251184e5227","name":"filebeat-filebeat-86b7b849df-df9zv","type":"filebeat"},"ecs":{"version":"8.0.0"},"log":{"file":{"path":"/usr/share/extras/transaction.log"},"offset":1355},"message":"{\\"transactionid\\":\\"5c8e23f3-3020-425a-a614-06d0c96fa2cf\\",\\"userId\\":\\"seojeonghyeon\\",\\"message\\":\\"Hello\\",\\"currentTime\\":\\"2023-05-21T10:59:54.597872686\\"}","input":{"type":"log"}}
{"@timestamp":"2023-05-21T02:01:00.449Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.5.1"},"agent":{"ephemeral_id":"e00c0689-2c64-46cb-b4d0-5feda709b2b3","id":"4fe495d7-56ed-4c04-9561-4251184e5227","name":"filebeat-filebeat-86b7b849df-df9zv","type":"filebeat","version":"8.5.1"},"ecs":{"version":"8.0.0"},"host":{"os":{"family":"debian","name":"Ubuntu","kernel":"5.15.89+","codename":"focal","type":"linux","platform":"ubuntu","version":"20.04.5 LTS (Focal Fossa)"},"containerized":true,"ip":["10.92.1.16"],"mac":["2A-6C-BF-8F-4E-34"],"name":"filebeat-filebeat-86b7b849df-df9zv","hostname":"filebeat-filebeat-86b7b849df-df9zv","architecture":"x86_64"},"log":{"file":{"path":"/usr/share/extras/transaction.log"},"offset":1501},"message":"{\\"transactionid\\":\\"c438b18a-c28f-4f60-9fc4-1f7044465d0c\\",\\"userId\\":\\"Kristen\\",\\"message\\":\\"How are you?\\",\\"currentTime\\":\\"2023-05-21T11:00:50.490580785\\"}","input":{"type":"log"},"fields":{"log_format":"json","kafka_topic":"kafkastudy-gkelog-transaction"}}
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ vi logapp.yml
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl create -f logapp.yml -n gw
deployment.apps/logapp created
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get all -n gw
NAME READY STATUS RESTARTS AGE
pod/logapp-7cb56cb45f-5jf62 0/1 ContainerCreating 0 4m3s
pod/logapp-7cb56cb45f-p8hjw 1/1 Running 0 4m3s
pod/logapp-7cb56cb45f-rk6vp 0/1 ContainerCreating 0 4m3s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/logapp 1/3 3 1 4m5s
NAME DESIRED CURRENT READY AGE
replicaset.apps/logapp-7cb56cb45f 3 3 1 4m5s
3) PV에 잘 저장되는지 어떻게 확인할까?
PV에 저장이 된다면 LogApp Deployment를 삭제했을 때에도 Log를 저장하는 위치에 이전 로그가 남아있어야 합니다.
3. Filebeat
1) Filebeat 설정을 수정하기 위해 yml파일 Download
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ curl -O <https://raw.githubusercontent.com/elastic/helm-charts/main/filebeat/values.yaml>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 7133 100 7133 0 0 14985 0 --:--:-- --:--:-- --:--:-- 14985
2) Deployment로 Filebeat 생성하도록 yml파일 작성(Daemonset과 Deployment 중 선택 가능, yml 파일 참고)
---
daemonset:
# Annotations to apply to the daemonset
annotations: {}
# additionals labels
labels: {}
affinity: {}
# Include the daemonset
enabled: false
# Extra environment variables for Filebeat container.
envFrom: []
# - configMapRef:
# name: config-secret
extraEnvs: []
#- name: "ELASTICSEARCH_USERNAME"
#valueFrom:
#secretKeyRef:
#name: elasticsearch-master-credentials
#key: username
#- name: "ELASTICSEARCH_PASSWORD"
#valueFrom:
#secretKeyRef:
#name: elasticsearch-master-credentials
#key: password
# Allows you to add any config files in /usr/share/filebeat
extraVolumes: []
# - name: extras
# emptyDir: {}
extraVolumeMounts: []
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
hostNetworking: false
# Allows you to add any config files in /usr/share/filebeat
# such as filebeat.yml for daemonset
filebeatConfig:
filebeat.yml: |
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
output.elasticsearch:
host: '${NODE_NAME}'
hosts: '["{ELASTICSEARCH_HOSTS:elasticsearch-master:9200}"]'
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]
# Only used when updateStrategy is set to "RollingUpdate"
maxUnavailable: 1
nodeSelector: {}
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security other sensitive values
secretMounts: {}
#- name: elasticsearch-master-certs
#secretName: elasticsearch-master-certs
#path: /usr/share/filebeat/certs/
# - name: filebeat-certificates
# secretName: filebeat-certificates
# path: /usr/share/filebeat/certs
# Various pod security context settings. Bear in mind that many of these have an impact on Filebeat functioning properly.
#
# - User that the container will execute as. Typically necessary to run as root (0) in order to properly collect host container logs.
# - Whether to execute the Filebeat containers as privileged containers. Typically not necessarily unless running within environments such as OpenShift.
securityContext:
runAsUser: 0
privileged: false
resources:
requests:
cpu: "100m"
memory: "100Mi"
limits:
cpu: "1000m"
memory: "200Mi"
tolerations: []
**deployment:**
# Annotations to apply to the deployment
annotations: {}
# additionals labels
labels: {}
affinity: {}
# Include the deployment
**enabled: true**
# Extra environment variables for Filebeat container.
envFrom: []
# - configMapRef:
# name: config-secret
extraEnvs: []
#- name: "ELASTICSEARCH_USERNAME"
#valueFrom:
#secretKeyRef:
#name: elasticsearch-master-credentials
#key: username
#- name: "ELASTICSEARCH_PASSWORD"
#valueFrom:
#secretKeyRef:
#name: elasticsearch-master-credentials
#key: password
# Allows you to add any config files in /usr/share/filebeat
**extraVolumes:
- name: extras
persistentVolumeClaim:
claimName: logstorage-pvc**
**extraVolumeMounts:
- name: extras
mountPath: /usr/share/extras
readOnly: true**
**filebeatConfig:
filebeat.yml: |
filebeat.inputs:
- type: log
paths :
- /usr/share/extras/transaction.log
fields :
log_format : "json"
kafka_topic : "kafkastudy-gkelog-transaction"
filebeat.config.modules :
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.kafka:
hosts: ["kafka-master.messagequeue.svc:9092"]
topic: '%{[fields.kafka_topic]}'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: snappy
max_message_bytes: 1000000
processors:
- add_host_metadata: ~**
nodeSelector: {}
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security other sensitive values
secretMounts: {}
#- name: elasticsearch-master-certs
#secretName: elasticsearch-master-certs
#path: /usr/share/filebeat/certs/
# - name: filebeat-certificates
# secretName: filebeat-certificates
# path: /usr/share/filebeat/certs
#
# - User that the container will execute as.
# Not necessary to run as root (0) as the Filebeat Deployment use cases do not need access to Kubernetes Node internals
# - Typically not necessarily unless running within environments such as OpenShift.
securityContext:
runAsUser: 0
privileged: false
resources:
requests:
cpu: "100m"
memory: "100Mi"
limits:
cpu: "1000m"
memory: "200Mi"
tolerations: []
# Replicas being used for the filebeat deployment
replicas: 1
extraContainers: ""
# - name: dummy-init
# image: busybox
# command: ['echo', 'hey']
extraInitContainers: []
# - name: dummy-init
# Root directory where Filebeat will write data to in order to persist registry data across pod restarts (file position and other metadata).
hostPathRoot: /var/lib
dnsConfig: {}
# options:
# - name: ndots
# value: "2"
hostAliases: []
#- ip: "127.0.0.1"
# hostnames:
# - "foo.local"
# - "bar.local"
image: "docker.elastic.co/beats/filebeat"
imageTag: "8.5.1"
imagePullPolicy: "IfNotPresent"
imagePullSecrets: []
livenessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
curl --fail 127.0.0.1:5066
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
filebeat test output
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
# Whether this chart should self-manage its service account, role, and associated role binding.
managedServiceAccount: true
clusterRoleRules:
- apiGroups:
- ""
resources:
- namespaces
- nodes
- pods
verbs:
- get
- list
- watch
- apiGroups:
- "apps"
resources:
- replicasets
verbs:
- get
- list
- watch
podAnnotations: {}
# iam.amazonaws.com/role: es-cluster
# Custom service account override that the pod will use
serviceAccount: ""
# Annotations to add to the ServiceAccount that is created if the serviceAccount value isn't set.
serviceAccountAnnotations: {}
# eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/k8s.clustername.namespace.serviceaccount
# How long to wait for Filebeat pods to stop gracefully
terminationGracePeriod: 30
# This is the PriorityClass settings as defined in
# <https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass>
priorityClassName: ""
updateStrategy: RollingUpdate
# Override various naming aspects of this chart
# Only edit these if you know what you're doing
nameOverride: ""
fullnameOverride: ""
*현재 생성한 LogApp과 Filebeat는 Kafka와 다른 Namespace에 존재하게 된다. 실제 운영에서도 Application과 ELK 담당자는 나눠져 있다. Filebeat가 다른 Namespace에 접근하기 위해 kafka-master.es.svc 로 표기함으로 어느 Namespace에 Service가 존재하는 지 알 수 있게 해준다.
3) Filebeat 생성 및 상태 확인
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get all -n gw
NAME READY STATUS RESTARTS AGE
**pod/filebeat-filebeat-86b7b849df-df9zv 0/1 Running 0 16s**
pod/logapp-5fb9fbc98b-2l5cp 0/1 ContainerCreating 0 77m
pod/logapp-5fb9fbc98b-5mp8h 0/1 ContainerCreating 0 77m
pod/logapp-5fb9fbc98b-q68m6 1/1 Running 0 77m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 19h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/filebeat-filebeat 0/1 1 0 16s
deployment.apps/logapp 1/3 3 1 77m
NAME DESIRED CURRENT READY AGE
replicaset.apps/filebeat-filebeat-86b7b849df 1 1 0 16s
replicaset.apps/logapp-5fb9fbc98b 3 3 1 77m
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl describe pod filebeat-filebeat-86b7b849df-df9zv -n gw
Name: filebeat-filebeat-86b7b849df-df9zv
Namespace: default
Priority: 0
Service Account: filebeat-filebeat
Node: gke-elk-cluster-default-pool-4bfe4779-314z/10.140.0.8
Start Time: Sun, 21 May 2023 01:51:39 +0000
Labels: app=filebeat-filebeat
chart=filebeat-8.5.1
pod-template-hash=86b7b849df
release=filebeat
Annotations: configChecksum: 9f77fcccf152c8e496e8ccf15fdb152aea28689be218150b114bf7ddc4cd034
Status: Running
IP: 10.92.1.16
IPs:
IP: 10.92.1.16
Controlled By: ReplicaSet/filebeat-filebeat-86b7b849df
Containers:
filebeat:
Container ID: containerd://61eaac49d9b50452af1d0ad4600a8cf9a2c3b3e475ce3e968887ee97f52f5cff
Image: docker.elastic.co/beats/filebeat:8.5.1
Image ID: docker.elastic.co/beats/filebeat@sha256:2e7a537a8a2c3d2767009ee65faadce572bc7ec5e571aba730f11a236da93a3a
Port: <none>
Host Port: <none>
Args:
-e
-E
http.enabled=true
State: Running
Started: Sun, 21 May 2023 01:51:40 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 200Mi
Requests:
cpu: 100m
memory: 100Mi
Liveness: exec [sh -c #!/usr/bin/env bash -e
curl --fail 127.0.0.1:5066
] delay=10s timeout=5s period=10s #success=1 #failure=3
Readiness: exec [sh -c #!/usr/bin/env bash -e
filebeat test output
] delay=10s timeout=5s period=10s #success=1 #failure=3
Environment:
POD_NAMESPACE: default (v1:metadata.namespace)
Mounts:
/usr/share/extras from extras (ro)
/usr/share/filebeat/filebeat.yml from filebeat-config (ro,path="filebeat.yml")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-csdsk (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
filebeat-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: filebeat-filebeat-deployment-config
Optional: false
**extras:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: logstorage-pvc
ReadOnly: false**
kube-api-access-csdsk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 33s default-scheduler Successfully assigned default/filebeat-filebeat-86b7b849df-df9zv to gke-elk-cluster-default-pool-4bfe4779-314z
Normal Pulled 33s kubelet Container image "docker.elastic.co/beats/filebeat:8.5.1" already present on machine
Normal Created 32s kubelet Created container filebeat
Normal Started 32s kubelet Started container filebeat
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get all -n gw
NAME READY STATUS RESTARTS AGE
**pod/filebeat-filebeat-86b7b849df-df9zv 1/1 Running 0 56s**
pod/logapp-5fb9fbc98b-2l5cp 0/1 ContainerCreating 0 78m
pod/logapp-5fb9fbc98b-5mp8h 0/1 ContainerCreating 0 78m
pod/logapp-5fb9fbc98b-q68m6 1/1 Running 0 78m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 19h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/filebeat-filebeat 1/1 1 1 56s
deployment.apps/logapp 1/3 3 1 78m
NAME DESIRED CURRENT READY AGE
replicaset.apps/filebeat-filebeat-86b7b849df 1 1 1 56s
replicaset.apps/logapp-5fb9fbc98b 3 3 1 78m
4) Kafka까지 인입이 잘되는 지 확인
I have no name!@kafka-master-client:/$ kafka-console-consumer.sh --bootstrap-server kafka-master:9092 --topic kafkastudy-gkelog-transaction
{"@timestamp":"2023-05-21T01:59:55.443Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.5.1"},"fields":{"log_format":"json","kafka_topic":"kafkastudy-gkelog-transaction"},"host":{"os":{"codename":"focal","type":"linux","platform":"ubuntu","version":"20.04.5 LTS (Focal Fossa)","family":"debian","name":"Ubuntu","kernel":"5.15.89+"},"name":"filebeat-filebeat-86b7b849df-df9zv","containerized":true,"ip":["10.92.1.16"],"mac":["2A-6C-BF-8F-4E-34"],"hostname":"filebeat-filebeat-86b7b849df-df9zv","architecture":"x86_64"},"agent":{"version":"8.5.1","ephemeral_id":"e00c0689-2c64-46cb-b4d0-5feda709b2b3","id":"4fe495d7-56ed-4c04-9561-4251184e5227","name":"filebeat-filebeat-86b7b849df-df9zv","type":"filebeat"},"ecs":{"version":"8.0.0"},"log":{"file":{"path":"/usr/share/extras/transaction.log"},"offset":1355},"message":"{\\"transactionid\\":\\"5c8e23f3-3020-425a-a614-06d0c96fa2cf\\",\\"userId\\":\\"seojeonghyeon\\",\\"message\\":\\"Hello\\",\\"currentTime\\":\\"2023-05-21T10:59:54.597872686\\"}","input":{"type":"log"}}
{"@timestamp":"2023-05-21T02:01:00.449Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.5.1"},"agent":{"ephemeral_id":"e00c0689-2c64-46cb-b4d0-5feda709b2b3","id":"4fe495d7-56ed-4c04-9561-4251184e5227","name":"filebeat-filebeat-86b7b849df-df9zv","type":"filebeat","version":"8.5.1"},"ecs":{"version":"8.0.0"},"host":{"os":{"family":"debian","name":"Ubuntu","kernel":"5.15.89+","codename":"focal","type":"linux","platform":"ubuntu","version":"20.04.5 LTS (Focal Fossa)"},"containerized":true,"ip":["10.92.1.16"],"mac":["2A-6C-BF-8F-4E-34"],"name":"filebeat-filebeat-86b7b849df-df9zv","hostname":"filebeat-filebeat-86b7b849df-df9zv","architecture":"x86_64"},"log":{"file":{"path":"/usr/share/extras/transaction.log"},"offset":1501},"message":"{\\"transactionid\\":\\"c438b18a-c28f-4f60-9fc4-1f7044465d0c\\",\\"userId\\":\\"Kristen\\",\\"message\\":\\"How are you?\\",\\"currentTime\\":\\"2023-05-21T11:00:50.490580785\\"}","input":{"type":"log"},"fields":{"log_format":"json","kafka_topic":"kafkastudy-gkelog-transaction"}}
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ curl <https://baltocdn.com/helm/signing.asc> | sudo apt-key add -
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
100 1699 100 1699 0 0 5720 0 --:--:-- --:--:-- --:--:-- 5720
OK
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ sudo apt-get install apt-transport-https --yes
********************************************************************************
You are running apt-get inside of Cloud Shell. Note that your Cloud Shell
machine is ephemeral and no system-wide change will persist beyond session end.
To suppress this warning, create an empty ~/.cloudshell/no-apt-get-warning file.
The command will automatically proceed in 5 seconds or on any key.
Visit <https://cloud.google.com/shell/help> for more information.
********************************************************************************
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
apt-transport-https is already the newest version (2.2.4).
The following package was automatically installed and is no longer required:
libpcre2-posix2
Use 'sudo apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 10 not upgraded.
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ echo "deb <https://baltocdn.com/helm/stable/debian/> all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
deb <https://baltocdn.com/helm/stable/debian/> all main
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ sudo apt-get update
********************************************************************************
You are running apt-get inside of Cloud Shell. Note that your Cloud Shell
machine is ephemeral and no system-wide change will persist beyond session end.
To suppress this warning, create an empty ~/.cloudshell/no-apt-get-warning file.
The command will automatically proceed in 5 seconds or on any key.
Visit <https://cloud.google.com/shell/help> for more information.
********************************************************************************
Hit:1 <http://deb.debian.org/debian> bullseye InRelease
Get:2 <http://deb.debian.org/debian> bullseye-updates InRelease [44.1 kB]
Get:3 <https://download.docker.com/linux/debian> bullseye InRelease [43.3 kB]
Get:4 <http://deb.debian.org/debian-security> bullseye-security InRelease [48.4 kB]
Get:5 <https://apt.releases.hashicorp.com> bullseye InRelease [12.9 kB]
Hit:6 <https://repo.mysql.com/apt/debian> bullseye InRelease
Hit:7 <https://storage.googleapis.com/bazel-apt> stable InRelease
Get:8 <https://download.docker.com/linux/debian> bullseye/stable amd64 Packages [27.8 kB]
Get:9 <http://deb.debian.org/debian-security> bullseye-security/main Sources [318 kB]
Get:10 <http://deb.debian.org/debian-security> bullseye-security/main amd64 Packages [304 kB]
Hit:11 <https://apt.llvm.org/bullseye> llvm-toolchain-bullseye-13 InRelease
Get:12 <https://apt.releases.hashicorp.com> bullseye/main amd64 Packages [104 kB]
Ign:13 <https://cli.github.com/packages> bullseye InRelease
Get:14 <https://packages.cloud.google.com/apt> apt-transport-artifact-registry-stable InRelease [5,094 B]
Get:15 <https://packages.cloud.google.com/apt> gcsfuse-bullseye InRelease [5,008 B]
Get:16 <https://packages.cloud.google.com/apt> cloud-sdk-bullseye InRelease [6,403 B]
Hit:13 <https://cli.github.com/packages> bullseye InRelease
Get:17 <https://baltocdn.com/helm/stable/debian> all InRelease [7,652 B]
Get:18 <https://packages.microsoft.com/debian/11/prod> bullseye InRelease [3,629 B]
Get:19 <https://baltocdn.com/helm/stable/debian> all/main amd64 Packages [3,907 B]
Get:20 <https://us-apt.pkg.dev/projects/demosite-images> demosite-apt InRelease [1,094 B]
Get:21 <https://packages.microsoft.com/debian/11/prod> bullseye/main armhf Packages [13.9 kB]
Get:22 <https://packages.microsoft.com/debian/11/prod> bullseye/main amd64 Packages [85.4 kB]
Get:23 <https://apt.postgresql.org/pub/repos/apt> bullseye-pgdg InRelease [117 kB]
Get:24 <https://packages.microsoft.com/debian/11/prod> bullseye/main arm64 Packages [15.0 kB]
Get:25 <https://packages.sury.org/php> bullseye InRelease [6,841 B]
Get:26 <https://packages.sury.org/php> bullseye/main amd64 Packages [376 kB]
Fetched 1,549 kB in 8s (202 kB/s)
Reading package lists... Done
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ sudo apt-get install helm
********************************************************************************
You are running apt-get inside of Cloud Shell. Note that your Cloud Shell
machine is ephemeral and no system-wide change will persist beyond session end.
To suppress this warning, create an empty ~/.cloudshell/no-apt-get-warning file.
The command will automatically proceed in 5 seconds or on any key.
Visit <https://cloud.google.com/shell/help> for more information.
********************************************************************************
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
libpcre2-posix2
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
helm
0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded.
Need to get 16.0 MB of archives.
After this operation, 50.6 MB of additional disk space will be used.
Get:1 <https://baltocdn.com/helm/stable/debian> all/main amd64 helm amd64 3.12.0-1 [16.0 MB]
Fetched 16.0 MB in 1s (18.5 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package helm.
(Reading database ... 142154 files and directories currently installed.)
Preparing to unpack .../helm_3.12.0-1_amd64.deb ...
Unpacking helm (3.12.0-1) ...
Setting up helm (3.12.0-1) ...
Processing triggers for man-db (2.9.4-2) ...
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ helm version
version.BuildInfo{Version:"v3.9.3", GitCommit:"414ff28d4029ae8c8b05d62aa06c7fe3dee2bc58", GitTreeState:"clean", GoVersion:"go1.17.13"}
2) Helm Repo 추가
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ helm repo add elastic <https://helm.elastic.co>
"elastic" has been added to your repositories
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ helm repo add bitnami <https://charts.bitnami.com/bitnami>
"bitnami" has been added to your repositories
2. Elasticsearch
1) Helm을 통해 Elasticsearch 설치(별도 yml파일 수정 없이 설치)
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ helm install elasticsearch elastic/elasticsearch -n es
NAME: elasticsearch
LAST DEPLOYED: Sat May 20 06:11:41 2023
NAMESPACE: es
STATUS: deployed
REVISION: 1
NOTES:
1. Watch all cluster members come up.
$ kubectl get pods --namespace=es -l app=elasticsearch-master -w
2. Retrieve elastic user's password.
$ kubectl get secrets --namespace=es elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d
3. Test cluster health using Helm test.
$ helm --namespace=es test elasticsearch
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get pods -n es
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 0/1 Init:0/1 0 23s
elasticsearch-master-1 0/1 Init:0/1 0 23s
elasticsearch-master-2 0/1 Init:0/1 0 23s
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get pods -n es
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 1/1 Running 0 2m33s
elasticsearch-master-1 1/1 Running 0 2m33s
elasticsearch-master-2 1/1 Running 0 2m33s
*별도 yml파일 수정 없이 설치하게 되면 3개의 Node 모두 Master, Injest, Coordinate, Data Node의 역할 모두를 가지게 된다.(yml 파일 내 설정 확인) Node를 분리해서 구축하고 싶다면 yml 파일 내려받아 수정 후 설치 진행하면 된다.
2) Elasticsearch 설정 정보 확인
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get po elasticsearch-master-0 -n es -o yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2023-05-20T06:11:44Z"
generateName: elasticsearch-master-
labels:
app: elasticsearch-master
chart: elasticsearch
controller-revision-hash: elasticsearch-master-d469cccdf
release: elasticsearch
statefulset.kubernetes.io/pod-name: elasticsearch-master-0
name: elasticsearch-master-0
namespace: es
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
**kind: StatefulSet**
name: elasticsearch-master
uid: a749825b-c73d-41e2-b179-fca640b59dc8
resourceVersion: "4382"
uid: 4b99ce8d-2f32-4fd3-9b34-fcda281d0a4e
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- elasticsearch-master
topologyKey: kubernetes.io/hostname
automountServiceAccountToken: true
containers:
- env:
- name: node.name
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: cluster.initial_master_nodes
value: elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,
- name: node.roles
value: master,data,data_content,data_hot,data_warm,data_cold,ingest,ml,remote_cluster_client,transform,
- name: discovery.seed_hosts
value: elasticsearch-master-headless
**- name: cluster.name
value: elasticsearch
- name: network.host
value: 0.0.0.0
- name: ELASTIC_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: elasticsearch-master-credentials**
**- name: xpack.security.enabled
value: "true"
- name: xpack.security.transport.ssl.enabled
value: "true"
- name: xpack.security.http.ssl.enabled
value: "true"**
**- name: xpack.security.transport.ssl.verification_mode
value: certificate**
**- name: xpack.security.transport.ssl.key
value: /usr/share/elasticsearch/config/certs/tls.key
- name: xpack.security.transport.ssl.certificate
value: /usr/share/elasticsearch/config/certs/tls.crt
- name: xpack.security.transport.ssl.certificate_authorities
value: /usr/share/elasticsearch/config/certs/ca.crt
- name: xpack.security.http.ssl.key**
**value: /usr/share/elasticsearch/config/certs/tls.key
- name: xpack.security.http.ssl.certificate
value: /usr/share/elasticsearch/config/certs/tls.crt
- name: xpack.security.http.ssl.certificate_authorities
value: /usr/share/elasticsearch/config/certs/ca.crt**
image: docker.elastic.co/elasticsearch/elasticsearch:8.5.1
imagePullPolicy: IfNotPresent
name: elasticsearch
ports:
- containerPort: 9200
name: http
protocol: TCP
- containerPort: 9300
name: transport
protocol: TCP
readinessProbe:
exec:
command:
- bash
- -c
- |
set -e
# Exit if ELASTIC_PASSWORD in unset
if [ -z "${ELASTIC_PASSWORD}" ]; then
echo "ELASTIC_PASSWORD variable is missing, exiting"
exit 1
fi
# If the node is starting up wait for the cluster to be ready (request params: "wait_for_status=green&timeout=1s" )
# Once it has started only check that the node itself is responding
START_FILE=/tmp/.es_start_file
# Disable nss cache to avoid filling dentry cache when calling curl
# This is required with Elasticsearch Docker using nss < 3.52
export NSS_SDB_USE_CACHE=no
http () {
local path="${1}"
local args="${2}"
set -- -XGET -s
if [ "$args" != "" ]; then
set -- "$@" $args
fi
set -- "$@" -u "elastic:${ELASTIC_PASSWORD}"
curl --output /dev/null -k "$@" "https://127.0.0.1:9200${path}"
}
if [ -f "${START_FILE}" ]; then
echo 'Elasticsearch is already running, lets check the node is healthy'
HTTP_CODE=$(http "/" "-w %{http_code}")
RC=$?
if [[ ${RC} -ne 0 ]]; then
echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \\${BASIC_AUTH} <https://127.0.0.1:9200/> failed with RC ${RC}"
exit ${RC}
fi
# ready if HTTP code 200, 503 is tolerable if ES version is 6.x
if [[ ${HTTP_CODE} == "200" ]]; then
exit 0
elif [[ ${HTTP_CODE} == "503" && "8" == "6" ]]; then
exit 0
else
echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \\${BASIC_AUTH} <https://127.0.0.1:9200/> failed with HTTP code ${HTTP_CODE}"
exit 1
fi
else
echo 'Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )'
if http "/_cluster/health?wait_for_status=green&timeout=1s" "--fail" ; then
touch ${START_FILE}
exit 0
else
echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
exit 1
fi
fi
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 3
timeoutSeconds: 5
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: "1"
memory: 2Gi
securityContext:
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
**- mountPath: /usr/share/elasticsearch/data
name: elasticsearch-master
- mountPath: /usr/share/elasticsearch/config/certs
name: elasticsearch-certs
readOnly: true**
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-frqnq
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostname: elasticsearch-master-0
initContainers:
- command:
- sysctl
- -w
- vm.max_map_count=262144
image: docker.elastic.co/elasticsearch/elasticsearch:8.5.1
imagePullPolicy: IfNotPresent
name: configure-sysctl
resources: {}
securityContext:
privileged: true
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
**volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-frqnq
readOnly: true**
nodeName: gke-elk-cluster-default-pool-4bfe4779-cf8q
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1000
runAsUser: 1000
serviceAccount: default
serviceAccountName: default
subdomain: elasticsearch-master-headless
terminationGracePeriodSeconds: 120
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: elasticsearch-master
persistentVolumeClaim:
claimName: elasticsearch-master-elasticsearch-master-0
- name: elasticsearch-certs
secret:
defaultMode: 420
secretName: elasticsearch-master-certs
- name: kube-api-access-frqnq
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2023-05-20T06:12:50Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2023-05-20T06:13:58Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2023-05-20T06:13:58Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2023-05-20T06:11:48Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://b55b3deb69e8ad24b04f32c57ec1cc8f0e2bafb2a9ec8e3ccf3a346834bcd5e5
image: docker.elastic.co/elasticsearch/elasticsearch:8.5.1
imageID: docker.elastic.co/elasticsearch/elasticsearch@sha256:d784066422aec9f66ae424f692d2416057e78853ab015915a04530570c955cc8
lastState: {}
name: elasticsearch
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2023-05-20T06:12:50Z"
hostIP: 10.140.0.9
initContainerStatuses:
- containerID: containerd://680ac3ec01e71048465a6a986bd0351f72cb3a8ba6b86e86860adf0ab14269e0
image: docker.elastic.co/elasticsearch/elasticsearch:8.5.1
imageID: docker.elastic.co/elasticsearch/elasticsearch@sha256:d784066422aec9f66ae424f692d2416057e78853ab015915a04530570c955cc8
lastState: {}
name: configure-sysctl
ready: true
restartCount: 0
state:
terminated:
containerID: containerd://680ac3ec01e71048465a6a986bd0351f72cb3a8ba6b86e86860adf0ab14269e0
exitCode: 0
finishedAt: "2023-05-20T06:12:43Z"
reason: Completed
startedAt: "2023-05-20T06:12:43Z"
phase: Running
podIP: 10.92.2.5
podIPs:
- ip: 10.92.2.5
qosClass: Burstable
startTime: "2023-05-20T06:11:48Z"
Statefulset는 상태를 저장하는 Stateful Application에 적합하며 Deployment는 상태를 저장하지 않는 Stateless Application에 적합하다.(Statefulset은 Deployment와 달리 각 Pod가 독자성을 가지고 생성, 독자성을 띄기 위해 Statefulset은 Headless 서비스를 사용하고 Volume을 특정 Pod에 고정한다.)
Elasticsearch Application은 관측 가능성 지표나 각종 상태를 저장해야 하는 Stateful Application이므로 Statefulset Object로 생성하게 된다.
3. Kafka
1) Helm을 통해 Kafka 설치(Broker = 3)
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ helm install kafka-master --set replicaCount=3 bitnami/kafka -n messagequeue
NAME: kafka-master
LAST DEPLOYED: Sat May 27 01:55:40 2023
NAMESPACE: messagequeue
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: kafka
CHART VERSION: 22.1.2
APP VERSION: 3.4.0
** Please be patient while the chart is being deployed **
Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:
kafka-master.messagequeue.svc.cluster.local
Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:
kafka-master-0.kafka-master-headless.messagequeue.svc.cluster.local:9092
kafka-master-1.kafka-master-headless.messagequeue.svc.cluster.local:9092
kafka-master-2.kafka-master-headless.messagequeue.svc.cluster.local:9092
To create a pod that you can use as a Kafka client run the following commands:
kubectl run kafka-master-client --restart='Never' --image docker.io/bitnami/kafka:3.4.0-debian-11-r28 --namespace messagequeue --command -- sleep infinity
kubectl exec --tty -i kafka-master-client --namespace messagequeue -- bash
PRODUCER:
kafka-console-producer.sh \\
--broker-list kafka-master-0.kafka-master-headless.messagequeue.svc.cluster.local:9092,kafka-master-1.kafka-master-headless.messagequeue.svc.cluster.local:9092,kafka-master-2.kafka-master-headless.messagequeue.svc.cluster.local:9092 \\
--topic test
CONSUMER:
kafka-console-consumer.sh \\
--bootstrap-server kafka-master.messagequeue.svc.cluster.local:9092 \\
--topic test \\
--from-beginning
2) Kafka-Topic 생성(Topic Name : kafkastudy-gkelog-transaction)
I have no name!@kafka-master-client:/$ kafka-topics.sh --bootstrap-server kafka-master:9092 --topic kafkastudy-gkelog-transaction --partitions 1 --replication-factor 3 --create
Created topic kafkastudy-gkelog-transaction.
I have no name!@kafka-master-client:/$ kafka-topics.sh --bootstrap-server kafka-master:9092 --topic kafkastudy-gkelog-transaction --describe
Topic: kafkastudy-gkelog-transaction PartitionCount: 1 ReplicationFactor: 3 Configs: flush.ms=1000,segment.bytes=1073741824,flush.messages=10000,max.message.bytes=1000012,retention.bytes=1073741824
Topic: kafkastudy-gkelog-transaction Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
*Logstash를 1개만 생성할 것이기 때문에 Partition은 1개만 생성하였다. 이후 Traffic이 많아지면 Logstash를 Scale-out 시키고 Parittion Count를 증가시킨다. Partition은 줄일 수 없기 때문에 주의해서 증가시킬 수 있도록 한다.
3) 데이터 인입 검증을 위한 환경
Shell을 하나 더 켜서 kafka-master-client를 준비하고 Consumer로 메세지가 정상적으로 들어오는 지 검증 가능하다.
4. 중간 정리(생성된 Object 확인)
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get all -n es
NAME READY STATUS RESTARTS AGE
pod/elasticsearch-master-0 1/1 Running 0 90m
pod/elasticsearch-master-1 1/1 Running 0 90m
pod/elasticsearch-master-2 1/1 Running 0 90m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearch-master ClusterIP 10.96.15.136 <none> 9200/TCP,9300/TCP 90m
service/elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 90m
NAME READY AGE
statefulset.apps/elasticsearch-master 3/3 90m
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get all -n messagequeue
NAME READY STATUS RESTARTS AGE
pod/kafka-master-0 1/1 Running 0 55s
pod/kafka-master-1 1/1 Running 0 54s
pod/kafka-master-2 1/1 Running 0 54s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kafka-master ClusterIP 10.96.10.236 <none> 9092/TCP 56s
service/kafka-master-headless ClusterIP None <none> 9092/TCP,9094/TCP,9093/TCP 56s
NAME READY AGE
statefulset.apps/kafka-master 3/3 56s
---
replicas: 1
# Allows you to add any config files in /usr/share/logstash/config/
# such as logstash.yml and log4j2.properties
#
# Note that when overriding logstash.yml, `http.host: 0.0.0.0` should always be included
# to make default probes work.
logstashConfig:
**logstash.yml: |
http.host: 0.0.0.0
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: '${ELASTICSEARCH_USERNAME}'
xpack.monitoring.elasticsearch.password: '${ELASTICSEARCH_PASSWORD}'
xpack.monitoring.elasticsearch.hosts: [""]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /usr/share/logstash/config/certs/ca.crt**
# logstash.yml: |
# key:
# nestedkey: value
# log4j2.properties: |
# key = value
# Allows you to add any pipeline files in /usr/share/logstash/pipeline/
### ***warn*** there is a hardcoded logstash.conf in the image, override it first
**logstashPipeline:
logstash.conf: |
input {
kafka{
bootstrap_servers => "kafka-master.messagequeue.svc:9092"
topics => ["kafkastudy-gkelog-transaction"]
group_id => "gkelog"
consumer_threads => 1
codec => json
decorate_events => true
}
}
filter {
mutate{
gsub => ['message', '\\"', '']
}
grok{
match => {
message => "transactionid:%{DATA:transactionid},userId:%{DATA:userId},message:%{DATA:messageContent},currentTime:%{TIMESTAMP_ISO8601:currentTime}"
}
}
mutate{
remove_field => [agent, ecs, host, log, input, fields, event, original, message]
}
}
output {
file {
path => "/usr/share/logstash/logstash.log"
codec => rubydebug
}
elasticsearch {
hosts => [""]
index => "kafkastudy-gkelog-transaction"
action => "update"
cacert => "/usr/share/logstash/config/certs/ca.crt"
user => '${ELASTICSEARCH_USERNAME}'
password => '${ELASTICSEARCH_PASSWORD}'
doc_as_upsert => true
document_id => "%{transactionid}"
}
}**
# logstash.conf: |
# input {
# exec {
# command => "uptime"
# interval => 30
# }
# }
# output { stdout { } }
# Allows you to add any pattern files in your custom pattern dir
logstashPatternDir: "/usr/share/logstash/patterns/"
logstashPattern: {}
# pattern.conf: |
# DPKG_VERSION [-+~<>\\.0-9a-zA-Z]+
# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
**extraEnvs:
- name: "ELASTICSEARCH_USERNAME"
valueFrom:
secretKeyRef:
name: elasticsearch-master-credentials
key: username
- name: "ELASTICSEARCH_PASSWORD"
valueFrom:
secretKeyRef:
name: elasticsearch-master-credentials
key: password**
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
# Allows you to load environment variables from kubernetes secret or config map
envFrom: []
# - secretRef:
# name: env-secret
# - configMapRef:
# name: config-map
# Add sensitive data to k8s secrets
secrets: []
# - name: "env"
# value:
# ELASTICSEARCH_PASSWORD: "LS1CRUdJTiBgUFJJVkFURSB"
# api_key: ui2CsdUadTiBasRJRkl9tvNnw
# - name: "tls"
# value:
# ca.crt: |
# LS0tLS1CRUdJT0K
# LS0tLS1CRUdJT0K
# LS0tLS1CRUdJT0K
# LS0tLS1CRUdJT0K
# cert.crt: "LS0tLS1CRUdJTiBlRJRklDQVRFLS0tLS0K"
# cert.key.filepath: "secrets.crt" # The path to file should be relative to the `values.yaml` file.
# A list of secrets and their paths to mount inside the pod
**secretMounts:
- name: elasticsearch-master-certs
secretName: elasticsearch-master-certs
path: /usr/share/logstash/config/certs**
hostAliases: []
#- ip: "127.0.0.1"
# hostnames:
# - "foo.local"
# - "bar.local"
image: "docker.elastic.co/logstash/logstash"
imageTag: "8.5.1"
imagePullPolicy: "IfNotPresent"
imagePullSecrets: []
podAnnotations: {}
# additionals labels
labels: {}
logstashJavaOpts: "-Xmx1g -Xms1g"
resources:
requests:
cpu: "100m"
memory: "1536Mi"
limits:
cpu: "1000m"
memory: "1536Mi"
volumeClaimTemplate:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
rbac:
create: false
serviceAccountAnnotations: {}
serviceAccountName: ""
annotations:
{}
#annotation1: "value1"
#annotation2: "value2"
#annotation3: "value3"
podSecurityPolicy:
create: false
name: ""
spec:
privileged: false
fsGroup:
rule: RunAsAny
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- secret
- configMap
- persistentVolumeClaim
persistence:
enabled: false
annotations: {}
extraVolumes:
[]
# - name: extras
# emptyDir: {}
extraVolumeMounts:
[]
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
extraContainers:
[]
# - name: do-something
# image: busybox
# command: ['do', 'something']
extraInitContainers:
[]
# - name: do-something
# image: busybox
# command: ['do', 'something']
# This is the PriorityClass settings as defined in
# <https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass>
priorityClassName: ""
# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: "kubernetes.io/hostname"
# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this "best effort"
antiAffinity: "hard"
# This is the node affinity settings as defined in
# <https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity>
nodeAffinity: {}
# This is inter-pod affinity settings as defined in
# <https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity>
podAffinity: {}
# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"
httpPort: 9600
# Custom ports to add to logstash
extraPorts:
[]
# - name: beats
# containerPort: 5001
updateStrategy: RollingUpdate
# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1
podSecurityContext:
fsGroup: 1000
runAsUser: 1000
securityContext:
capabilities:
drop:
- ALL
# readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
# How long to wait for logstash to stop gracefully
terminationGracePeriod: 120
# Probes
# Default probes are using `httpGet` which requires that `http.host: 0.0.0.0` is part of
# `logstash.yml`. If needed probes can be disabled or overridden using the following syntaxes:
#
# disable livenessProbe
# livenessProbe: null
#
# replace httpGet default readinessProbe by some exec probe
# readinessProbe:
# httpGet: null
# exec:
# command:
# - curl
# - localhost:9600
livenessProbe:
httpGet:
path: /
port: http
initialDelaySeconds: 300
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
successThreshold: 1
readinessProbe:
httpGet:
path: /
port: http
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
successThreshold: 3
## Use an alternate scheduler.
## ref: <https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/>
##
schedulerName: ""
nodeSelector: {}
tolerations: []
nameOverride: ""
fullnameOverride: ""
lifecycle:
{}
# preStop:
# exec:
# command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
# postStart:
# exec:
# command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
service:
{}
# annotations: {}
# type: ClusterIP
# loadBalancerIP: ""
# ports:
# - name: beats
# port: 5044
# protocol: TCP
# targetPort: 5044
# - name: http
# port: 8080
# protocol: TCP
# targetPort: 8080
ingress:
enabled: false
annotations:
{}
# kubernetes.io/tls-acme: "true"
className: "nginx"
pathtype: ImplementationSpecific
hosts:
- host: logstash-example.local
paths:
- path: /beats
servicePort: 5044
- path: /http
servicePort: 8080
tls: []
# - secretName: logstash-example-tls
# hosts:
# - logstash-example.local
3) Logstash 설치
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ vi values.yaml
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ helm install logstash elastic/logstash -n es -f values.yaml
NAME: logstash
LAST DEPLOYED: Mon May 22 05:04:32 2023
NAMESPACE: es
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Watch all cluster members come up.
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl get all -n es
NAME READY STATUS RESTARTS AGE
pod/elasticsearch-master-0 1/1 Running 0 105m
pod/elasticsearch-master-1 1/1 Running 0 105m
pod/elasticsearch-master-2 1/1 Running 0 105m
pod/kibana-kibana-7759ccf877-vqpmf 1/1 Running 0 101m
pod/logstash-logstash-0 0/1 ContainerCreating 0 7s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearch-master ClusterIP 10.96.2.194 <none> 9200/TCP,9300/TCP 105m
service/elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 105m
service/kibana-kibana ClusterIP 10.96.9.203 <none> 5601/TCP 27h
service/logstash-logstash-headless ClusterIP None <none> 9600/TCP 7s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kibana-kibana 1/1 1 1 27h
NAME DESIRED CURRENT READY AGE
replicaset.apps/kibana-kibana-7759ccf877 1 1 1 27h
NAME READY AGE
statefulset.apps/elasticsearch-master 3/3 105m
statefulset.apps/logstash-logstash 0/1 7s
autoever_seojeonghyeon0630@cloudshell:~ (kubernetes-project-386708)$ kubectl describe pod logstash-logstash-0 -n es
Name: logstash-logstash-0
Namespace: es
Priority: 0
Service Account: default
Node: gke-elk-cluster-default-pool-4bfe4779-cf8q/10.140.0.9
Start Time: Mon, 22 May 2023 05:04:32 +0000
Labels: app=logstash-logstash
chart=logstash
controller-revision-hash=logstash-logstash-6fccdbcd99
heritage=Helm
release=logstash
statefulset.kubernetes.io/pod-name=logstash-logstash-0
Annotations: configchecksum: 70c86d885b248b6c40a481430f7377102a9b1f108299b2e262450e4575c596b
pipelinechecksum: 062b99a376822f4f20c6280bb278531e56e692abc70a75980d6d286c683223f
Status: Running
IP: 10.92.2.71
IPs:
IP: 10.92.2.71
**Controlled By: StatefulSet/logstash-logstash**
Containers:
logstash:
Container ID: containerd://c0a1b957d9fb9665cfd1c6d814514399e59ade9ad4b896fefdf1b6b1af565f9d
Image: docker.elastic.co/logstash/logstash:8.5.1
Image ID: docker.elastic.co/logstash/logstash@sha256:b82b9d00f0ec6714f1424e39406a15d1024fa4636c54718edb10cca8e1f2664e
**Port: 9600/TCP**
Host Port: 0/TCP
State: Running
Started: Mon, 22 May 2023 05:04:33 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 1536Mi
Requests:
cpu: 100m
memory: 1536Mi
**Liveness: http-get http://:http/ delay=300s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get http://:http/ delay=60s timeout=5s period=10s #success=3 #failure=3**
Environment:
LS_JAVA_OPTS: -Xmx1g -Xms1g
**ELASTICSEARCH_USERNAME: <set to the key 'username' in secret 'elasticsearch-master-credentials'> Optional: false
ELASTICSEARCH_PASSWORD: <set to the key 'password' in secret 'elasticsearch-master-credentials'> Optional: false**
Mounts:
**/usr/share/logstash/config/certs from elasticsearch-master-certs (rw)
/usr/share/logstash/config/logstash.yml from logstashconfig (rw,path="logstash.yml")
/usr/share/logstash/pipeline/logstash.conf from logstashpipeline (rw,path="logstash.conf")**
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r5qhv (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
**Volumes:
elasticsearch-master-certs:
Type: Secret (a volume populated by a Secret)
SecretName: elasticsearch-master-certs
Optional: false
logstashconfig:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: logstash-logstash-config
Optional: false
logstashpipeline:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: logstash-logstash-pipeline
Optional: false**
kube-api-access-r5qhv:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 35m default-scheduler Successfully assigned es/logstash-logstash-0 to gke-elk-cluster-default-pool-4bfe4779-cf8q
Normal Pulled 35m kubelet Container image "docker.elastic.co/logstash/logstash:8.5.1" already present on machine
Normal Created 35m kubelet Created container logstash
Normal Started 35m kubelet Started container logstash
4) Logstash 설정파일(pipeline/logstash.conf) 검증은 어떻게 가능할까?
yml 파일 내 logstashConfig 아래 log4j2.properties 설정이 가능하다. 해당 설정을 통해 Log 파일을 생성하도록 한다.
yml 파일 내 logstashConfig 아래 logstash.conf의 output에 아래와 같이 설정을 넣어주게 되면 들어온 데이터에 대해 Log파일로 생성(Log파일 생성되는 위치 주의, 권한이 없는 위치라면 Pod가 정지된다)되게 되며 오류에 대한 로그 또한 확인 가능하다
Kibana 내 DevTools 도구에는 Grok Debugger라는 도구가 존재한다. 해당 도구를 이용하면 아래 이미지와 같이 맛깔나게 Grok에 대해 디자인 가능하다.
