Name the log pipe: journald, then Loki or OpenSearch

Grafana’s Loki docs mark Promtail end of life on 2 March 2026. I opened that page. Loki 3.7.6 shipped 5 August 2026. OpenSearch 3.7.0 went GA on 9 June 2026 on the project’s releases page, last updated 23 July 2026. A post that lists twelve SIEM logos and never names the unit that writes stdout is a vendor dump. This page names one pipe.

The pipe is a local journal, a shipper whose config is in git, a store you can name, and a retention number you can defend. Pair host files with the Ubuntu page. Pair what the app may log with input validation so a request body never becomes a secret in a line. Pair HTTP response policy with the headers guide.

journald is the local source

On Ubuntu 24.04 the unit you already wrote for webapp sends stdout and stderr to systemd. journald stores that with a machine id, a unit name, a priority, and a cursor. You do not tail a file under /var/log/webapp unless the process cannot speak stdout. Prefer the journal. A file you then forget to rotate is a disk incident, not a pipeline.

# /etc/systemd/system/webapp.service
[Unit]
Description=webapp
After=network.target

[Service]
User=webapp
Group=webapp
WorkingDirectory=/srv/webapp
ExecStart=/srv/webapp/bin/webapp
Restart=on-failure
StandardOutput=journal
StandardError=journal
SyslogIdentifier=webapp

[Install]
WantedBy=multi-user.target

The app writes one JSON object per line to stdout. No timestamps you invent if journald already has _SOURCE_REALTIME_TIMESTAMP. No request bodies. No Authorization values. A login failure is {"event":"login_fail","email_hash":"..."}, not the password.

function logLine(event, fields) {
  process.stdout.write(JSON.stringify({ event, ...fields }) + "\n");
}

app.post("/login", loginLimiter, express.urlencoded({ limit: "8kb", extended: false }), async (req, res) => {
  const email = req.body.email;
  if (typeof email !== "string") {
    logLine("login_bad", { reason: "type" });
    res.status(400).send("bad login");
    return;
  }
  const user = await users.findByEmail(email);
  if (!user) {
    logLine("login_fail", { reason: "unknown" });
    res.status(401).send("no");
    return;
  }
  logLine("login_ok", { userId: user.id });
  req.session.userId = user.id;
  res.status(204).end();
});

loginLimiter is the same identifier the DoS and server-prevention pages use. This page does not re-teach the quota. It needs the line that lands after a 401. Persist the journal so a reboot does not erase the last week:

# /etc/systemd/journald.conf.d/00-persist.conf
[Journal]
Storage=persistent
SystemMaxUse=1G
MaxRetentionSec=14day

Then systemctl restart systemd-journald. journalctl -u webapp -n 20 -o json-pretty is the local proof before any shipper exists. If that is empty, Alloy has nothing to read. Fix the unit first. When the disk is tight, journalctl --vacuum-size=800M is the local trim. Do not vacuum to zero during an incident.

sshd, Caddy, and nginx already speak the journal if they are systemd units. You do not need a second file tail for “security logs” on a single VPS. journalctl -u ssh -n 50 is the auth noise. journalctl -u webapp is the hasher. A third file under /var/log is how the shipper and the human look at different truths.

Alloy ships it. Promtail does not.

Grafana documented the Promtail deprecation in the Loki 3.4 announcement and put the EOL date on the Promtail page: 2 March 2026, no further support. Alloy is the collector that page names. I opened loki.source.journal. When path is empty, Alloy reads /var/log/journal and /run/log/journal. The process needs to be in group systemd-journal.

// /etc/alloy/config.alloy
loki.relabel "journal" {
  forward_to = []

  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label  = "unit"
  }
  rule {
    source_labels = ["__journal_priority"]
    target_label  = "priority"
  }
}

loki.source.journal "read" {
  max_age       = "12h"
  forward_to    = [loki.write.local.receiver]
  relabel_rules = loki.relabel.journal.rules
  labels        = {
    job  = "journald",
    host = "app-a",
  }
}

loki.write "local" {
  endpoint {
    url = "http://127.0.0.1:3100/loki/api/v1/push"
  }
}

Identifiers stay loki.source.journal "read", loki.write "local", labels job, host, unit, priority. Do not map __journal_pid or a request id to a Loki label. That is cardinality. A new process id every second will blow the index you thought was cheap.

If you still have a Promtail yaml, Grafana’s convert path is alloy convert --source-format=promtail. I am citing the Alloy migrate page. Do not leave Promtail installed “until we have time.” The EOL date is already five months behind August 2026.

Loki when a label is enough

Loki indexes labels. It does not build an inverted index of the line. That is the whole product. LogQL is “pick a stream, then grep the bytes.” WinonaRyder’s 2020 comment is the shape: JSON in, labels for unit and host, Explore for a night of login_fail.

I opened the Loki 3.7.6 release on GitHub, published 6 August 2026. Run that, or the 3.7.x your distro tracks. A single-binary or simple scalable deploy on one box is enough for a handful of VPS journals. Object storage comes later, when the disk on that box is the constraint.

# logcli against Loki you admin. Same labels as config.alloy.
logcli query '{job="journald", unit="webapp.service"} |= "login_fail"' \
  --addr=http://127.0.0.1:3100 \
  --since=24h

Useful queries stay on labels you already emit:

{job="journald", unit="ssh.service"} |= "Failed"
{job="journald", unit="webapp.service"} |= "login_fail"
{job="journald", unit="webapp.service"} | json | event="login_ok"

If every question you ask is “show me webapp on app-a since 18:00” or “lines that contain login_fail,” Loki is the store. If the question is “every line in the last 90 days that contains this one email address, across units I cannot list,” you want a word index. That is the next section, not a Grafana plugin you have not measured.

OpenSearch when you need a word

OpenSearch 3.0.0 GA was 6 May 2025. 3.7.0 is dated 9 June 2026 on the releases page I opened. 3.8.0’s window closed 4 August 2026. I could not confirm a 3.8.0 row in the history table on 23 July 2026’s page. Pin 3.7.0, or the 3.x your cluster already runs. 2.19.6, dated 2 July 2026, is the maintenance line if you are not ready to move majors.

OpenSearch builds a full-text index. That is why it costs RAM and why it answers “find this token anywhere.” Use it for forensic search, for a compliance export, for a security queue that must filter on a field you did not label at ingest. Do not use it as the default home for nginx access logs you only ever filter by status and host. That workload is Loki.

Alloy’s first-party journal source writes to Loki. The shipper I can name for the word index is Fluent Bit’s systemd input and opensearch output, or OpenSearch Data Prepper if you already run that collector. A minimal Fluent Bit file that reuses the same unit filter:

# /etc/fluent-bit/fluent-bit.conf
[SERVICE]
    Flush         5
    Daemon        Off

[INPUT]
    Name              systemd
    Tag               host.webapp
    Systemd_Filter    _SYSTEMD_UNIT=webapp.service
    DB                /var/lib/fluent-bit/systemd.db

[OUTPUT]
    Name              opensearch
    Match             host.webapp
    Host              127.0.0.1
    Port              9200
    Index             journald-webapp
    Suppress_Type_Name On

Create the index template so event is a keyword and the raw line is text. Then a query you can rerun:

curl -sS "http://127.0.0.1:9200/journald-webapp/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{"query":{"term":{"event":"login_fail"}},"size":20}'

Retention, redaction, one index field

A pipe without a delete is a leak with a delay. journald’s MaxRetentionSec=14day is the local cap. Loki needs a retention_period in the compactor config you actually loaded. OpenSearch needs an ISM policy that deletes journald-webapp-* after the number you wrote down. The policy name is journald-webapp-14d. If that name does not exist, the cluster will keep the index until the disk fills.

Redact at the app, not in a regex you will forget. If a library dumps headers on error, wrap the logger. If nginx access_log includes query strings with tokens, change the log format before Alloy tails that file. journald of the webapp unit is enough for the security questions on this page. Access logs are a second stream with a second retention if you need them.

Do not forward the journal to a UDP syslog you do not encrypt. The 1980s BSD payload the old article celebrated has no TLS and no year. If you must speak syslog, IETF syslog over TLS to a collector you admin. journald plus Alloy on localhost is the shorter 2026 path.

webapp.service
   stdout JSON
        |
        v
   journald   Storage=persistent  14d
        |
        +-- Alloy  loki.source.journal "read"
        |      labels job, host, unit
        |      -> Loki :3100
        |
        +-- Fluent Bit  systemd input
               -> OpenSearch  journald-webapp
               only if you need a word index

Prove a failed login arrives

You are not shipping other people’s logs. You are proving a 401 you caused on your own app showed up in the store you named.

1. journalctl -u webapp -n 5 -o cat shows a login_fail line after you POST /login with a bad password against your own origin.
2. alloy is running and its log shows journal reads, not permission denied. The process is in systemd-journal.
3. logcli against http://127.0.0.1:3100 returns that line under {job="journald", unit="webapp.service"}.
4. If you enabled OpenSearch, the term query on event: login_fail returns the same moment.

journalctl -u webapp -n 5 -o cat
# Expect a line: {"event":"login_fail","reason":"unknown"}

logcli query '{job="journald", unit="webapp.service"} |= "login_fail"' \
  --addr=http://127.0.0.1:3100 --since=1h

If journald has the line and Loki does not, Alloy is the gap: group, path, or the push URL. If Loki has it and you expected OpenSearch, you are looking at the wrong store. Pick one for the on-call query and keep the other, if you have it, for the rare word search. Do not build a third “SIEM” because a PDF listed one.