Skip to content

Add Centralized Logging and Monitoring Stack (Prometheus + Grafana) #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
berkal749 wants to merge 2 commits into
base: dev
Choose a base branch
from
Open

Add Centralized Logging and Monitoring Stack (Prometheus + Grafana) #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5fb904b
Merge pull request #41 from MicroClub-USTHB/dev
YounesBensafia Mar 21, 2026
27b2143
(idk bro but i did somthing) Add Prometheus and Grafana services to D…
berkal749 Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions docker-compose.dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ services:
--reload-dir /app/src
depends_on:
- qdrant
networks:
- maicro-network
healthcheck:
test:
["CMD-SHELL", "curl -f http://localhost:8000/api/v1/health || exit 1"]
Expand All @@ -44,8 +46,49 @@ services:
- "6334:6334"
volumes:
- qdrant_data:/qdrant/storage
networks:
- maicro-network
healthcheck:
disable: true

prometheus:
image: prom/prometheus:latest
container_name: prometheus_dev
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml

Copilot AI Mar 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Prometheus config bind-mount isn’t read-only here (unlike docker-compose.yml which uses :ro). Making it read-only helps prevent accidental in-container edits and keeps behavior consistent across environments.

Suggested change
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

Copilot uses AI. Check for mistakes.
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
networks:
- maicro-network
depends_on:
- maicro

grafana:
image: grafana/grafana:latest
container_name: grafana_dev
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:-admin}
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-admin}
Comment on lines +76 to +77

Copilot AI Mar 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even in the dev compose, defaulting Grafana admin credentials to admin/admin can be risky (e.g., if ports are exposed on a shared network). Consider removing the defaults and documenting required env vars, or binding Grafana to localhost only when defaults are used.

Suggested change
- GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:-admin}
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-admin}
- GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:?GRAFANA_ADMIN_USER not set}
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:?GRAFANA_ADMIN_PASSWORD not set}

Copilot uses AI. Check for mistakes.
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning:ro
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
networks:
- maicro-network
depends_on:
- prometheus

volumes:
qdrant_data:
prometheus_data:
grafana_data:

networks:
maicro-network:
driver: bridge
45 changes: 45 additions & 0 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ services:
env_file:
- .env
restart: unless-stopped
networks:
- maicro-network
healthcheck:
test:
[
Expand All @@ -20,3 +22,46 @@ services:
interval: 30s
timeout: 10s
retries: 3

prometheus:
image: prom/prometheus:latest
container_name: prometheus
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus_data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
depends_on:
- maicro
restart: unless-stopped
networks:
- maicro-network

grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
environment:
GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:-admin}
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-admin}
Comment on lines +49 to +50

Copilot AI Mar 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grafana admin credentials default to admin/admin via ${...:-admin}. If this compose file is used beyond purely local dev, this creates an easily exploitable default. Prefer requiring these env vars (no insecure default), or inject via Docker secrets / a local-only override file.

Suggested change
GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:-admin}
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-admin}
GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:?GRAFANA_ADMIN_USER must be set}
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?GRAFANA_ADMIN_PASSWORD must be set}

Copilot uses AI. Check for mistakes.
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning:ro
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
depends_on:
- prometheus
restart: unless-stopped
networks:
- maicro-network

volumes:
prometheus_data:
grafana_data:

networks:
maicro-network:
driver: bridge
197 changes: 197 additions & 0 deletions grafana/dashboards/maicro-overview.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"links": [],
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"

Copilot AI Mar 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This dashboard references the Prometheus datasource via uid: "${DS_PROMETHEUS}", but there is no templating/input variable defined in the JSON (templating.list is empty) and the provisioned datasource file doesn’t set a fixed uid. This will usually make the dashboard fail to bind to the datasource on import/provisioning. Set an explicit datasource uid in Grafana provisioning and reference that literal uid in the dashboard JSON (or add the proper __inputs/templating variable wiring).

Suggested change
"uid": "${DS_PROMETHEUS}"
"uid": "prometheus"

Copilot uses AI. Check for mistakes.
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "red",
"value": 0
},
{
"color": "green",
"value": 1
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"textMode": "auto"
},
"pluginVersion": "11.0.0",
"targets": [
{
"editorMode": "code",
"expr": "up{job=\"maicro\"}",
"legendFormat": "",
"range": true,
"refId": "A"
}
],
"title": "mAIcro Up",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 2,
"pointSize": 4,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 16,
"x": 8,
"y": 0
},
"id": 2,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "11.0.0",
"targets": [
{
"editorMode": "code",
"expr": "up{job=\"maicro\"}",
"legendFormat": "up",
"range": true,
"refId": "A"
}
],
"title": "mAIcro Availability (Time Series)",
"type": "timeseries"
}
],
"refresh": "10s",
"schemaVersion": 39,
"style": "dark",
"tags": [
"maicro",
"observability"
],
"templating": {
"list": []
},
"time": {
"from": "now-15m",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "mAIcro Overview",
"uid": "maicro-overview",
"version": 1,
"weekStart": ""
}
11 changes: 11 additions & 0 deletions grafana/provisioning/dashboards/dashboards.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
apiVersion: 1

providers:
- name: mAIcro Dashboards
orgId: 1
folder: mAIcro
type: file
disableDeletion: false
editable: true
options:
path: /var/lib/grafana/dashboards
9 changes: 9 additions & 0 deletions grafana/provisioning/datasources/prometheus.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
apiVersion: 1

datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: true
Comment on lines +3 to +9

Copilot AI Mar 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provisioned Prometheus datasource doesn’t define a stable uid. Since dashboards often reference datasources by uid, provisioning without one can break pre-provisioned dashboards (including the one added in this PR that uses ${DS_PROMETHEUS}). Add an explicit uid here and keep dashboards aligned to it.

Copilot uses AI. Check for mistakes.
13 changes: 13 additions & 0 deletions prometheus.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
global:
scrape_interval: 15s
evaluation_interval: 15s

scrape_configs:
- job_name: 'maicro'
static_configs:
- targets: ['maicro:8000']
metrics_path: '/metrics'

Comment on lines +6 to +10

Copilot AI Mar 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prometheus is configured to scrape maicro:8000/metrics, but the FastAPI app currently doesn't expose a /metrics endpoint (no metrics/instrumentation found in src/). As-is, this scrape job will continuously 404; either add metrics instrumentation (e.g., Prometheus client/instrumentator) or adjust/remove this scrape config until the endpoint exists.

Suggested change
- job_name: 'maicro'
static_configs:
- targets: ['maicro:8000']
metrics_path: '/metrics'

Copilot uses AI. Check for mistakes.
- job_name: 'qdrant'
static_configs:
- targets: ['qdrant:6333']
Comment on lines +9 to +13

Copilot AI Mar 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prometheus.yml includes a scrape job targeting qdrant:6333, but the non-dev docker-compose.yml doesn’t define a qdrant service/network alias. In that environment Prometheus will log repeated DNS/connection failures. Either add qdrant to docker-compose.yml (if intended), or split Prometheus config per environment so prod doesn’t reference a missing target.

Suggested change
metrics_path: '/metrics'
- job_name: 'qdrant'
static_configs:
- targets: ['qdrant:6333']
metrics_path: '/metrics'

Copilot uses AI. Check for mistakes.
14 changes: 14 additions & 0 deletions src/core/log.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@


import logging


logging.basicConfig(
level=logging.INFO,
format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
)


def get_logger(name: str) -> logging.Logger:
"""Get a logger instance with the app's default configuration."""
Comment on lines +6 to +13

Copilot AI Mar 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Configuring logging via logging.basicConfig(...) at import time is fragile under uvicorn (it typically installs handlers before importing the app, so basicConfig becomes a no-op). Consider moving configuration into an explicit configure_logging() that is invoked by the entrypoint, or configure a dedicated app logger/handler (or provide a uvicorn log_config) so formatting/level are reliably applied without global import side effects.

Suggested change
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
)
def get_logger(name: str) -> logging.Logger:
"""Get a logger instance with the app's default configuration."""
_LOGGING_CONFIGURED = False
def configure_logging() -> None:
"""
Configure application logging.
This function is safe to call multiple times. If the root logger already
has handlers configured (for example, by uvicorn or another framework),
this function will not override that configuration.
"""
global _LOGGING_CONFIGURED
if _LOGGING_CONFIGURED:
return
root_logger = logging.getLogger()
# If handlers are already present (e.g., installed by uvicorn), respect
# the existing configuration and do not call basicConfig.
if not root_logger.handlers:
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
)
_LOGGING_CONFIGURED = True
def get_logger(name: str) -> logging.Logger:
"""Get a logger instance with the app's default configuration."""
configure_logging()

Copilot uses AI. Check for mistakes.
return logging.getLogger(name)
9 changes: 2 additions & 7 deletions src/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from contextlib import asynccontextmanager

from fastapi import FastAPI
from core.log import get_logger

from api.error_handlers import register_exception_handlers
from api.routes import router
Expand All @@ -12,13 +13,7 @@
from core.ingestion import ingest_from_discord, run_startup_audit
import logging

Copilot AI Mar 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

import logging is now unused after switching to get_logger; remove it to avoid ruff/linters failing and to reduce confusion about which logging config is active.

Suggested change
import logging

Copilot uses AI. Check for mistakes.

logging.basicConfig(
level=logging.INFO,
format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
)

logger = logging.getLogger(__name__)

logger = get_logger(__name__)

@asynccontextmanager
async def lifespan(app: FastAPI):
Expand Down
Loading