본문 바로가기
OS/Linux

[Linux] Prometheus & Grafana 소스 컴파일 설치 및 Trouble Shooting

by Yoon_estar 2025. 2. 21.
728x90

🚀 개요

외부 인터넷이 차단된 환경에서 PrometheusGrafana를 소스 컴파일을 통해 설치하고, 이를 활용하여 서버의 CPU 및 시스템 온도를 모니터링하는 방법을 정리합니다. 또한 설치 과정에서 발생한 오류와 이를 해결한 과정을 포함합니다.

 


🖥️설치 파일 다운로드 

1️⃣Prometheus

https://github.com/prometheus/prometheus/releases/tag/v3.1.0

 

Release 3.1.0 / 2025-01-02 · prometheus/prometheus

What's Changed [SECURITY] upgrade golang.org/x/crypto to address reported CVE-2024-45337. #15691 [CHANGE] Notifier: Increment prometheus_notifications_errors_total by the number of affected alerts...

github.com

 

# wget https://github.com/prometheus/prometheus/releases/download/v3.1.0/prometheus-3.1.0.linux-amd64.tar.gz

2️⃣Node Exporter

https://github.com/prometheus/node_exporter/releases

 

Releases · prometheus/node_exporter

Exporter for machine metrics. Contribute to prometheus/node_exporter development by creating an account on GitHub.

github.com

 

# wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz

3️⃣ Alertmanager

https://github.com/prometheus/alertmanager

 

GitHub - prometheus/alertmanager: Prometheus Alertmanager

Prometheus Alertmanager. Contribute to prometheus/alertmanager development by creating an account on GitHub.

github.com

 

# wget https://github.com/prometheus/alertmanager/archive/refs/heads/main.zip

4️⃣ Grafana 

https://grafana.com/docs/grafana/latest/setup-grafana/installation/redhat-rhel-fedora/

 

Install Grafana on RHEL or Fedora | Grafana documentation

Install Grafana on RHEL or Fedora This topic explains how to install Grafana dependencies, install Grafana on RHEL or Fedora, and start the Grafana server on your system. You can install Grafana from the RPM repository, from standalone RPM, or with the bin

grafana.com

 

# wget https://dl.grafana.com/enterprise/release/grafana-enterprise-11.2.0-1.x86_64.rpm

5️⃣ ipmitool

2025.02.06 - [OS/Linux] - [Linux] Dell 서버 iDRAC(IPMI) 관리용 ipmitool 소스 컴파일 설치 가이드

 


🔗Prometheus 소스 컴파일 및 설치

1️⃣Prometheus 소스 다운로드 및 압축 해제

# wget https://github.com/prometheus/prometheus/releases/download/v3.1.0/prometheus-3.1.0.linux-amd64.tar.gz
#  tar xvf prometheus-2.51.0.linux-amd64.tar.gz
#  mv prometheus-2.51.0.linux-amd64 /opt/prometheus

 

2️⃣Prometheus 서비스 등록

sudo tee /etc/systemd/system/prometheus.service <<EOF
[Unit]
Description=Prometheus Server
After=network.target

[Service]
User=root
ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.path=/opt/prometheus/data

[Install]
WantedBy=default.target
EOF

 

3️⃣서비스 활성화 및 실행

# sudo systemctl daemon-reload
# sudo systemctl enable prometheus
# sudo systemctl start prometheus
# systemctl status prometheus.service
● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2025-02-06 17:48:57 KST; 15min ago
 Main PID: 4072398 (prometheus)
    Tasks: 11 (limit: 49442)
   Memory: 23.9M
   CGroup: /system.slice/prometheus.service
           └─4072398 /opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml

Feb 06 17:48:57 final prometheus[4072398]: time=2025-02-06T08:48:57.875Z level=INFO source=head.go:723 msg="Replaying WAL, t>
Feb 06 17:48:57 final prometheus[4072398]: time=2025-02-06T08:48:57.876Z level=INFO source=head.go:795 msg="WAL segment load>
Feb 06 17:48:57 final prometheus[4072398]: time=2025-02-06T08:48:57.876Z level=INFO source=head.go:832 msg="WAL replay compl>
Feb 06 17:48:57 final prometheus[4072398]: time=2025-02-06T08:48:57.880Z level=INFO source=main.go:1245 msg="filesystem info>
Feb 06 17:48:57 final prometheus[4072398]: time=2025-02-06T08:48:57.880Z level=INFO source=main.go:1248 msg="TSDB started"
Feb 06 17:48:57 final prometheus[4072398]: time=2025-02-06T08:48:57.880Z level=INFO source=main.go:1433 msg="Loading configu>
Feb 06 17:48:57 final prometheus[4072398]: time=2025-02-06T08:48:57.881Z level=INFO source=main.go:1472 msg="updated GOGC" o>
Feb 06 17:48:57 final prometheus[4072398]: time=2025-02-06T08:48:57.881Z level=INFO source=main.go:1482 msg="Completed loadi>
Feb 06 17:48:57 final prometheus[4072398]: time=2025-02-06T08:48:57.881Z level=INFO source=main.go:1209 msg="Server is ready>
Feb 06 17:48:57 final prometheus[4072398]: time=2025-02-06T08:48:57.881Z level=INFO source=manager.go:169 msg="Starting rule>

 


🔗 Node Exporter 소스 컴파일 및 설치

  • Node Exporter는 서버의 CPU, 메모리, 디스크 상태를 Prometheus로 전송하는 역할을 합니다.

1️⃣Node Exporter 다운로드 및 설치

# wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
# tar xvf node_exporter-1.8.1.linux-amd64.tar.gz
# sudo mv node_exporter-1.8.1.linux-amd64 /opt/node_exporter

2️⃣Node Exporter 서비스 등록

# tee /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
User=root
ExecStart=/opt/node_exporter/node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

[Install]
WantedBy=default.target
EOF

 

3️⃣서비스 활성화 및 실행

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
# sudo systemctl enable node_exporter
Created symlink /etc/systemd/system/default.target.wants/node_exporter.service → /etc/systemd/system/node_exporter.service.
[root@final prometheus_setup]# sudo systemctl start node_exporter
[root@final prometheus_setup]# systemctl status node_exporter.service
● node_exporter.service - Prometheus Node Exporter
   Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2025-02-06 18:07:30 KST; 6s ago
 Main PID: 4086838 (node_exporter)
    Tasks: 5 (limit: 49442)
   Memory: 2.8M
   CGroup: /system.slice/node_exporter.service
           └─4086838 /opt/node_exporter/node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile_collect>

Feb 06 18:07:30 final node_exporter[4086838]: ts=2025-02-06T09:07:30.741Z caller=node_exporter.go:118 level=info collector=t>
Feb 06 18:07:30 final node_exporter[4086838]: ts=2025-02-06T09:07:30.741Z caller=node_exporter.go:118 level=info collector=t>
Feb 06 18:07:30 final node_exporter[4086838]: ts=2025-02-06T09:07:30.741Z caller=node_exporter.go:118 level=info collector=u>
Feb 06 18:07:30 final node_exporter[4086838]: ts=2025-02-06T09:07:30.741Z caller=node_exporter.go:118 level=info collector=u>
Feb 06 18:07:30 final node_exporter[4086838]: ts=2025-02-06T09:07:30.741Z caller=node_exporter.go:118 level=info collector=v>
Feb 06 18:07:30 final node_exporter[4086838]: ts=2025-02-06T09:07:30.741Z caller=node_exporter.go:118 level=info collector=w>
Feb 06 18:07:30 final node_exporter[4086838]: ts=2025-02-06T09:07:30.741Z caller=node_exporter.go:118 level=info collector=x>
Feb 06 18:07:30 final node_exporter[4086838]: ts=2025-02-06T09:07:30.741Z caller=node_exporter.go:118 level=info collector=z>
Feb 06 18:07:30 final node_exporter[4086838]: ts=2025-02-06T09:07:30.742Z caller=tls_config.go:313 level=info msg="Listening>
Feb 06 18:07:30 final node_exporter[4086838]: ts=2025-02-06T09:07:30.742Z caller=tls_config.go:316 level=info msg="TLS is di>

 

🔗 lm-sensors로 CPU 온도 데이터 수집

# sensors
acpitz-virtual-0
Adapter: Virtual device
temp1:        +27.8°C  (crit = +105.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +49.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:        +46.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:        +47.0°C  (high = +80.0°C, crit = +100.0°C)
Core 2:        +46.0°C  (high = +80.0°C, crit = +100.0°C)
Core 3:        +46.0°C  (high = +80.0°C, crit = +100.0°C)
Core 4:        +47.0°C  (high = +80.0°C, crit = +100.0°C)
Core 5:        +49.0°C  (high = +80.0°C, crit = +100.0°C)

 

🔗CPU 온도 데이터를 Prometheus에 추가

1️⃣ 온도 데이터를 Node Exporter에 연결

# mkdir -p /var/lib/node_exporter/textfile_collector
# tee /var/lib/node_exporter/textfile_collector/cpu_temp.prom <<EOF
> # HELP node_cpu_temperature CPU temperature
> # TYPE node_cpu_temperature gauge
> node_cpu_temperature{core="0"} 50.0
> node_cpu_temperature{core="1"} 52.0
> EOF
# HELP node_cpu_temperature CPU temperature
# TYPE node_cpu_temperature gauge
node_cpu_temperature{core="0"} 50.0
node_cpu_temperature{core="1"} 52.0

# sudo systemctl restart node_exporter
# systemctl status node_exporter.service
● node_exporter.service - Prometheus Node Exporter
   Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2025-02-06 18:15:11 KST; 5s ago
 Main PID: 4092545 (node_exporter)
    Tasks: 5 (limit: 49442)
   Memory: 6.2M
   CGroup: /system.slice/node_exporter.service
           └─4092545 /opt/node_exporter/node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile_collect>

Feb 06 18:15:12 final node_exporter[4092545]: ts=2025-02-06T09:15:12.024Z caller=node_exporter.go:118 level=info collector=t>
Feb 06 18:15:12 final node_exporter[4092545]: ts=2025-02-06T09:15:12.024Z caller=node_exporter.go:118 level=info collector=t>
Feb 06 18:15:12 final node_exporter[4092545]: ts=2025-02-06T09:15:12.024Z caller=node_exporter.go:118 level=info collector=u>
Feb 06 18:15:12 final node_exporter[4092545]: ts=2025-02-06T09:15:12.024Z caller=node_exporter.go:118 level=info collector=u>
Feb 06 18:15:12 final node_exporter[4092545]: ts=2025-02-06T09:15:12.024Z caller=node_exporter.go:118 level=info collector=v>
Feb 06 18:15:12 final node_exporter[4092545]: ts=2025-02-06T09:15:12.024Z caller=node_exporter.go:118 level=info collector=w>
Feb 06 18:15:12 final node_exporter[4092545]: ts=2025-02-06T09:15:12.024Z caller=node_exporter.go:118 level=info collector=x>
Feb 06 18:15:12 final node_exporter[4092545]: ts=2025-02-06T09:15:12.024Z caller=node_exporter.go:118 level=info collector=z>
Feb 06 18:15:12 final node_exporter[4092545]: ts=2025-02-06T09:15:12.025Z caller=tls_config.go:313 level=info msg="Listening>
Feb 06 18:15:12 final node_exporter[4092545]: ts=2025-02-06T09:15:12.025Z caller=tls_config.go:316 level=info msg="TLS is di>

2️⃣Prometheus 설정 파일 수정

# tee /opt/prometheus/prometheus.yml <<EOF
> global:
>   scrape_interval: 15s
>
> scrape_configs:
>   - job_name: 'node_exporter'
>     static_configs:
>       - targets: ['localhost:9100']
> EOF
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

# sudo systemctl restart prometheus
# systemctl status prometheus.service
● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2025-02-06 18:16:57 KST; 5s ago
 Main PID: 4093540 (prometheus)
    Tasks: 17 (limit: 49442)
   Memory: 24.7M
   CGroup: /system.slice/prometheus.service
           └─4093540 /opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml

Feb 06 18:16:57 final prometheus[4093540]: time=2025-02-06T09:16:57.214Z level=INFO source=head.go:795 msg="WAL segment load>
Feb 06 18:16:57 final prometheus[4093540]: time=2025-02-06T09:16:57.215Z level=INFO source=head.go:795 msg="WAL segment load>
Feb 06 18:16:57 final prometheus[4093540]: time=2025-02-06T09:16:57.215Z level=INFO source=head.go:832 msg="WAL replay compl>
Feb 06 18:16:57 final prometheus[4093540]: time=2025-02-06T09:16:57.217Z level=INFO source=main.go:1245 msg="filesystem info>
Feb 06 18:16:57 final prometheus[4093540]: time=2025-02-06T09:16:57.217Z level=INFO source=main.go:1248 msg="TSDB started"
Feb 06 18:16:57 final prometheus[4093540]: time=2025-02-06T09:16:57.217Z level=INFO source=main.go:1433 msg="Loading configu>
Feb 06 18:16:57 final prometheus[4093540]: time=2025-02-06T09:16:57.218Z level=INFO source=main.go:1472 msg="updated GOGC" o>
Feb 06 18:16:57 final prometheus[4093540]: time=2025-02-06T09:16:57.219Z level=INFO source=main.go:1482 msg="Completed loadi>
Feb 06 18:16:57 final prometheus[4093540]: time=2025-02-06T09:16:57.219Z level=INFO source=main.go:1209 msg="Server is ready>
Feb 06 18:16:57 final prometheus[4093540]: time=2025-02-06T09:16:57.219Z level=INFO source=manager.go:169 msg="Starting rule>

 

🔗Grafana 설치

1️⃣Grafana 다운로드 및 설치

# wget https://dl.grafana.com/enterprise/release/grafana-enterprise-10.0.1.linux-amd64.tar.gz
# tar zxvf grafana-enterprise-10.0.1.linux-amd64.tar.gz
# cd grafana-10.0.1/
# mv grafana-10.0.1 /opt/grafana

2️⃣Grafana 실행 스크립트 설정

# sudo tee /etc/systemd/system/grafana.service <<EOF
[Unit]
Description=Grafana Server
After=network.target

[Service]
User=root
ExecStart=/opt/grafana/bin/grafana-server --homepath /opt/grafana --config /opt/grafana/conf/grafana.ini

[Install]
WantedBy=multi-user.target
EOF
# sudo systemctl daemon-reload
# sudo systemctl enable grafana
# sudo systemctl start grafana
# sudo systemctl status grafana

⚠️ Troubleshooting

1️⃣Grafana 실행 오류(Could not find config defaults)

오류 메세지

Grafana-server Init Failed: Could not find config defaults, make sure homepath is set correctly.

 

해결 방법

# sudo cp /opt/grafana/conf/sample.ini /opt/grafana/conf/grafana.ini
# sudo chown -R grafana:grafana /opt/grafana

-------------------------------------------------------

# vi 
```
[paths]
home = /opt/grafana
data = /opt/grafana/data
logs = /opt/grafana/logs
plugins = /opt/grafana/plugins
provisioning = /opt/grafana/conf/provisioning

[server]
http_port = 3000

--------------------------------------------------------------

# sudo chmod -R 755 /opt/grafana
# sudo systemctl restart grafana

🔗Web UI 접속

1️⃣Grafana

2️⃣Prometheus

  • admin / admin으로 로그인 후 PW 변경