# Tame systemd journal logs across a fleet

The systemd journal will happily grow until it fills the disk. On an FR201 with a modest M.2 SSD, that's a real concern — a chatty service or a tight-loop error can produce gigabytes of logs in a few days. The fix is two-part:

1. **One-time cleanup**: vacuum the existing journal to reclaim space now.
2. **Persistent cap**: tell `journald` how much disk it's allowed to use, so it self-prunes from here on.

If you have one device, you SSH in and run a few commands. If you have twenty, you want automation. This guide covers both with a small PowerShell driver that pushes a bash configuration script to a range of devices in parallel.

## What the automation does

Two files do the work:

- journal-automation/
  - automate.ps1
  - configure_journald.sh
- `configure_journald.sh` runs on each FR201. It vacuums the journal down to 100 MB, sets a 1 GB on-disk cap and a 100 MB tmpfs cap in `/etc/systemd/journald.conf`, and restarts `systemd-journald` to apply the change.
- `automate.ps1` runs on your workstation. It walks a range of IP addresses, copies `configure_journald.sh` to each device with `scp`, runs it with `sudo`, and logs success or failure for each host.

## Prerequisites

- PowerShell 7+ on your workstation (the driver uses cross-platform OpenSSH; Windows PowerShell 5.1 also works).
- SSH access to every target device with key-based auth set up. Password auth will work but you'll be typing your password a lot.
- A user account on each device (the example uses `pi`) with passwordless `sudo` for `systemctl` and journal commands.
- `configure_journald.sh` saved with **LF line endings**. CRLF will break the script silently on the device — see the [dos2unix guide](/guides/onlogic-fr201/dos2unix/) if you're not sure.

## The configuration script

This is the script that runs on each device.

```bash title="configure_journald.sh"
#!/bin/bash
set -euo pipefail

# Must run as root.
if [[ $EUID -ne 0 ]]; then
    echo "This script must be run as root." >&2
    exit 1
fi

echo "Vacuuming existing journal entries..."
journalctl --vacuum-size=100M

echo "Updating /etc/systemd/journald.conf..."
CONF=/etc/systemd/journald.conf

# Replace or append SystemMaxUse and RuntimeMaxUse.
for setting in "SystemMaxUse=1G" "RuntimeMaxUse=100M"; do
    key="${setting%%=*}"
    if grep -qE "^\s*#?\s*${key}=" "$CONF"; then
        sed -i "s|^\s*#\?\s*${key}=.*|${setting}|" "$CONF"
    else
        echo "$setting" >> "$CONF"
    fi
done

echo "Restarting systemd-journald..."
systemctl restart systemd-journald

echo "Done. Current journal usage:"
journalctl --disk-usage
```

A few notes:

- `set -euo pipefail` makes the script bail on the first error. Without it, a partial run can leave the configuration half-applied.
- The `for` loop replaces the `SystemMaxUse` and `RuntimeMaxUse` lines if they already exist (commented or not), and appends them otherwise. That makes the script idempotent — running it twice is safe.
- `journalctl --disk-usage` at the end is a nice sanity check; you'll see in the PowerShell output what each device ended up at.

## The PowerShell driver

This runs on your workstation and fans out to the devices.

```powershell title="automate.ps1"
# Adjust to match your fleet.
$User       = "pi"
$IpRange    = 50..69 | ForEach-Object { "192.168.1.$_" }
$ScriptPath = Join-Path $PSScriptRoot "configure_journald.sh"
$RemoteTmp  = "/tmp/configure_journald.sh"
$Timeout    = 5

$SuccessLog = Join-Path $PSScriptRoot "journal_config_success.log"
$FailLog    = Join-Path $PSScriptRoot "journal_config_fail.log"

# Reset the logs each run.
"" | Set-Content $SuccessLog
"" | Set-Content $FailLog

foreach ($ip in $IpRange) {
    Write-Host "[$ip] copying script..." -ForegroundColor Cyan

    $scp = & scp -o "ConnectTimeout=$Timeout" -o "StrictHostKeyChecking=accept-new" `
                 $ScriptPath "${User}@${ip}:${RemoteTmp}" 2>&1
    if ($LASTEXITCODE -ne 0) {
        "[$ip] scp failed: $scp" | Tee-Object -FilePath $FailLog -Append
        continue
    }

    Write-Host "[$ip] running script..." -ForegroundColor Cyan
    $ssh = & ssh -o "ConnectTimeout=$Timeout" -o "StrictHostKeyChecking=accept-new" `
                 "${User}@${ip}" "chmod +x $RemoteTmp && sudo $RemoteTmp && rm $RemoteTmp" 2>&1
    if ($LASTEXITCODE -eq 0) {
        "[$ip] OK`n$ssh" | Tee-Object -FilePath $SuccessLog -Append
    } else {
        "[$ip] ssh failed:`n$ssh" | Tee-Object -FilePath $FailLog -Append
    }
}

Write-Host "Done. See $SuccessLog and $FailLog." -ForegroundColor Green
```

What it's doing:

- Builds a list of IPs from a range (`192.168.1.50` through `192.168.1.69` in this example).
- For each device, copies the bash script to `/tmp`, makes it executable, and runs it with `sudo`.
- 5-second connect timeout on both `scp` and `ssh` so the loop doesn't hang on a powered-down device.
- Tees the per-host output to `journal_config_success.log` or `journal_config_fail.log` so you have a record without scrolling back through the terminal.

## Run it

1. Drop both files into a folder on your workstation:

    - journal-automation/
      - automate.ps1
      - configure_journald.sh
2. Make sure `configure_journald.sh` has Unix line endings. From PowerShell:

    ```powershell title="Force LF line endings"
    (Get-Content .\configure_journald.sh -Raw) -replace "`r`n", "`n" |
        Set-Content -NoNewline .\configure_journald.sh
    ```

3. Edit the top of `automate.ps1` to match your network — username, IP range, timeout.

4. Run it:

    ```powershell title="Run the deployment"
    .\automate.ps1
    ```

5. Review the logs:

    - journal-automation/
      - journal_config_success.log
      - journal_config_fail.log
    ## Tuning the caps

The defaults in the script — 1 GB on-disk, 100 MB on tmpfs — are reasonable for a typical FR201 install. If you have less storage, drop them. If you want more retention for forensic reasons, raise them. Adjust these two lines in `configure_journald.sh`:

```bash title="configure_journald.sh"
for setting in "SystemMaxUse=1G" "RuntimeMaxUse=100M"; do
```

`SystemMaxUse` controls the persistent journal under `/var/log/journal/`. `RuntimeMaxUse` controls the volatile journal in `/run/log/journal/` (used when persistent storage isn't available).

## Troubleshooting

**SSH connection failures**: confirm key-based auth works against one host with `ssh pi@192.168.1.50` before fanning out. The 5-second timeout means a misconfigured device will fail fast rather than hanging the whole run.

**Permission errors on the device**: the user needs passwordless `sudo` for at least `systemctl` and `journalctl`. Add a sudoers rule scoped to those binaries instead of granting blanket `NOPASSWD: ALL`.

**No effect after the script runs**: confirm `systemd-journald` actually restarted (`systemctl status systemd-journald`) and that `/etc/systemd/journald.conf` shows the new values. A CRLF in the bash script can cause it to silently fail partway through.

## Security notes

- The driver requires SSH access and `sudo` rights — the same access you'd use to log in by hand. Use SSH keys, not passwords, and scope `sudo` rules narrowly.
- The script writes to `/tmp` and removes itself after running, so nothing is left behind on the device.
- All connections use connect timeouts so a hung host can't stall the rollout.