Prometheus and Grafana handle metrics. Loki handles logs - the third pillar of observability. Together they answer: "what happened, when, and why."
Pods (frontend + backend)
↓ writes logs to stdout
Promtail (DaemonSet - one per node)
↓ tails container logs, ships them
Loki (log storage, time-indexed like Prometheus)
↓ queried by
Grafana (Explore tab)
Loki only indexes metadata (labels), not full log content - dramatically cheaper storage and simpler operation than Elasticsearch. It's designed to pair natively with Grafana and Prometheus, using the same query philosophy (PromQL-like LogQL).
Loki needs persistent storage for its single-binary deployment. EKS doesn't include this by default on newer versions - must be installed as an addon.
Defined in modules/eks/main.tf:
resource "aws_iam_role" "ebs_csi_driver" {
name = "${var.project}-${var.environment}-ebs-csi-driver-role"
assume_role_policy = data.aws_iam_policy_document.ebs_csi_assume_role.json
}
resource "aws_iam_role_policy_attachment" "ebs_csi_driver" {
role = aws_iam_role.ebs_csi_driver.name
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}
resource "aws_eks_addon" "ebs_csi_driver" {
cluster_name = aws_eks_cluster.main.name
addon_name = "aws-ebs-csi-driver"
service_account_role_arn = aws_iam_role.ebs_csi_driver.arn
depends_on = [aws_eks_node_group.main]
}
This uses IRSA (IAM Roles for Service Accounts) - the CSI driver assumes a scoped IAM role via OIDC, not broad node permissions.
Verify it's running:
kubectl get pods -n kube-system | grep ebs
Expected:
ebs-csi-controller-xxx 6/6 Running
ebs-csi-controller-xxx 6/6 Running
ebs-csi-node-xxx 3/3 Running (one per node)
kubectl patch storageclass gp2 \
-p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
Verify: