Kubernetes 1.27: Query Node Logs Using The Kubelet API
source link: https://kubernetes.io/blog/2023/04/21/node-log-query-alpha/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Kubernetes 1.27: Query Node Logs Using The Kubelet API
Author: Aravindh Puthiyaparambil (Red Hat)
Kubernetes 1.27 introduced a new feature called Node log query that allows viewing logs of services running on the node.
What problem does it solve?
Cluster administrators face issues when debugging malfunctioning services running on the node. They usually have to SSH or RDP into the node to view the logs of the service to debug the issue. The Node log query feature helps with this scenario by allowing the cluster administrator to view the logs using kubectl. This is especially useful with Windows nodes where you run into the issue of the node going to the ready state but containers not coming up due to CNI misconfigurations and other issues that are not easily identifiable by looking at the Pod status.
How does it work?
The kubelet already has a /var/log/ viewer that is accessible via the node
proxy endpoint. The feature supplements this endpoint with a shim that shells
out to journalctl
, on Linux nodes, and the Get-WinEvent
cmdlet on Windows
nodes. It then uses the existing filters provided by the commands to allow
filtering the logs. The kubelet also uses heuristics to retrieve the logs.
If the user is not aware if a given system services logs to a file or to the
native system logger, the heuristics first checks the native operating system
logger and if that is not available it attempts to retrieve the first logs
from /var/log/<servicename>
or /var/log/<servicename>.log
or
/var/log/<servicename>/<servicename>.log
.
On Linux we assume that service logs are available via journald, and that
journalctl
is installed. On Windows we assume that service logs are available
in the application log provider. Also note that fetching node logs is only
available if you are authorized to do so (in RBAC, that's get and
create access to nodes/proxy
). The privileges that you need to fetch node
logs also allow elevation-of-privilege attacks, so be careful about how you
manage them.
How do I use it?
To use the feature, ensure that the NodeLogQuery
feature gate is
enabled for that node, and that the kubelet configuration options
enableSystemLogHandler
and enableSystemLogQuery
are both set to true. You can
then query the logs from all your nodes or just a subset. Here is an example to
retrieve the kubelet service logs from a node:
# Fetch kubelet logs from a node named node-1.example
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet"
You can further filter the query to narrow down the results:
# Fetch kubelet logs from a node named node-1.example that have the word "error"
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet&pattern=error"
You can also fetch files from /var/log/
on a Linux node:
kubectl get --raw "/api/v1/nodes/<insert-node-name-here>/proxy/logs/?query=/<insert-log-file-name-here>"
You can read the documentation for all the available options.
How do I help?
Please use the feature and provide feedback by opening GitHub issues or reaching out to us on the #sig-windows channel on the Kubernetes Slack or the SIG Windows mailing list.
Recommend
-
85
Virtual Kubelet Virtual Kubelet is an open source Kubernetes kubelet implementation that masquerades as a kubelet for the purposes of connecting Kuberne...
-
32
本文通过记录kubelet连接apiserver超时问题的原因及修复办法。 上篇文章回顾:一文读懂HBase多租户 背 景 kubernetes是ma...
-
28
驱逐管理是kubelet自我保护的一种机制,当资源缺乏的时候,通过阈值和策略主动驱逐部分的Pod从而保持系统的资源水位在一个合理的范围内,本身的策略官方文档上已经写的很详细,今天只是看下其在代码上的一些设计 1. 设计臆测
-
31
近期,腾讯游戏Tenc容器团队对外开源了Kubernetes(K8s)多集群调度方案 tensile-kube , 本文将简单介绍tensile-kube。 https://github.com/virtual-kubelet/tensile-kube tens...
-
14
Version k8s 版本 v1.18.3 CNI CNI 严格来说不是 kubelet 的一部分,而是容器...
-
9
30 March 2021 / Kubernetes #Kubernetes failed to run Kubelet: running with swap on is not supported, please disable swap 安装完Kubernetes以后...
-
4
Collect and Query your Kubernetes Cluster Logs with Grafana Loki 6 days ago2021-07-19T00:00:00+02:00 by Wolfgang Ofner 14 minSo far I have talked about the development, deployment, and config...
-
9
后Kubernetes时代的虚拟机管理技术之Virtual-Kubelet篇 ...
-
1
kubelet CRI 创建 Pod 调用流程 本文以 kubelet dockershim 创建 Pod 调用流程为例做分析。 kubelet 通过调用 dockershim 来创建并启动容器,而 dockershim 则调用 Docker 来创建并启动容器,并调用 CNI 来构建 Pod 网络。
-
6
Kubernetes源码分析——kubelet | 李乾坤的博客 go 程序中大量使用channel 一个是消灭了观察者模式 很多功能组件得以独立。以...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK