前言

上一小節描述了metrics、traces，本小節來把log也加進去，並且做一個traces與log的聯動

• 當查看日誌的時候，可以同時跳轉到對應的jaeger，查看分段trace情況

應用服務

• 本次要測試的應用服務架構為 a.py-->b.py
• 業務服務會往對應的目錄打印日誌，並且日誌包含了trace_id

a.py

import tornado.httpserver as httpserver
import tornado.web
from tornado.ioloop import IOLoop
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.trace import get_tracer
from opentelemetry.propagate import inject
import requests
import logging


trace.set_tracer_provider(
    TracerProvider(resource=Resource.create({SERVICE_NAME: "line-a"}))
)
span_processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://127.0.0.1:4318/v1/traces"))
trace.get_tracer_provider().add_span_processor(span_processor)

logger = logging.getLogger('a.py')
logger.setLevel(logging.INFO)
formatter = logging.Formatter( '%(asctime)s - %(name)s - %(levelname)s - %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
file_handler = logging.FileHandler('logs/access-a.log', mode='a', encoding='utf-8')
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)

def traced(name):
    def decorator(func):
        def wrapper(*args, **kwargs):
            tracer = get_tracer(__name__)
            with tracer.start_as_current_span(name):
                trace_id = format(trace.get_current_span().get_span_context().trace_id, '032x')
                logger.info('{} {}'.format(name, trace_id))
                return func(*args, **kwargs)
        return wrapper
    return decorator

class TestFlow(tornado.web.RequestHandler):
    def get(self):
        views()
        self.finish('hello world')

@traced("phase-1")
def views():
    views_2()
    headers = {}
    inject(headers)
    requests.get("http://127.0.0.1:20000", headers=headers)

@traced("phase-2")
def views_2():
    pass

def applications():
    urls = []
    urls.append([r'/', TestFlow])
    return tornado.web.Application(urls)

def main():
    app = applications()
    server = httpserver.HTTPServer(app)
    server.bind(10000, '0.0.0.0')
    server.start(1)
    IOLoop.current().start()


if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt as e:
        IOLoop.current().stop()
    finally:
        IOLoop.current().close()

同理配置b.py

...
trace.set_tracer_provider(
    TracerProvider(resource=Resource.create({SERVICE_NAME: "line-b"}))
)

logger = logging.getLogger('b.py')
logger.setLevel(logging.INFO)
formatter = logging.Formatter( '%(asctime)s - %(name)s - %(levelname)s - %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
file_handler = logging.FileHandler('logs/access-b.log', mode='a', encoding='utf-8')

...
@traced("phase-3")
def views(headers):
    pass

...
def main():
    app = applications()
    server = httpserver.HTTPServer(app)
    server.bind(20000, '0.0.0.0')
    server.start(1)
    IOLoop.current().start()
...

訪問curl 127.0.0.1:10000之後，就會往日誌文件中寫入日誌

▶ cat access-a.log
2025-08-21 11:22:18 - a.py - INFO - phase-1 38f0e8cef3e331993eca1675fa8c956b
2025-08-21 11:22:18 - a.py - INFO - phase-2 38f0e8cef3e331993eca1675fa8c956b

▶ cat access-b.log
2025-08-21 11:22:18 - b.py - INFO - phase-3 38f0e8cef3e331993eca1675fa8c956b

日誌系統loki

1）安裝loki

loki.yaml

auth_enabled: false

server:
  http_listen_port: 3100

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

docker run -d --name grafana-loki \
  -p 3100:3100 \
  -v ./loki.yaml:/etc/loki/local-config.yaml \
  grafana/loki:v3.5.3

2）安裝promtail

promtail.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://10.22.12.178:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - labels:
      job: access_log
      __path__: /var/log/access*.log

docker run -d --name grafana-promtail \
  -v ./promtail.yaml:/etc/promtail/config.yml \
  -v ./logs:/var/log:ro \
  grafana/promtail:3.5

3）安裝grafana

docker run -d --name grafana-grafana \
  -p 3000:3000 \
  grafana/grafana:12.1.1

4）在grafana上配置loki

登錄grafana --> connections --> data sources --> 選擇loki --> 開始配置

最關鍵的步驟就是把connection改成loki的地址10.22.12.178:3100

5）查看下剛才寫入的日誌

日誌已經進來了

logs跳轉至traces

之前已經將traces採集進jaeger，現在又擁有了日誌，將他們通過traceid聯動起來，在grafana找到日誌，並且通過traceid字段跳轉去jaeger查詢詳細信息

登錄grafana --> connections --> data sources --> 選擇剛才配置好的loki --> Derived fields

• Name：隨便取一個
• Type：Regex in log line
• Regex：([0-9a-f]{32,})
• URL：配置jaeger的URL，http://10.22.12.178:16686/trace/${__value.raw}

配置完成之後查看日誌，有會個跳轉按鈕，點擊之後就來帶jaeger-UI的查詢頁面

聯繫我

• 聯繫我，做深入的交流
  

至此，本文結束
在下才疏學淺，有撒湯漏水的，請各位不吝賜教...