Skip to content

Institutional stacks

4. Automated Fixity & Audit Engine

Silent data corruption (bit rot) is the single greatest threat to long-term digital retention. Implement automated systemd timers to verify SHA-256 checksum manifests against live disk states.

This audit engine exports native metrics to Prometheus (via Node Exporter textfile collector), logs events to systemd-journald/syslog for SIEM ingestion (Splunk, Elastic), and supports enterprise notification triggers (SMTP email-to-ticket or PagerDuty webhooks).

Enterprise Fixity Audit Script (/opt/scripts/fixity_audit.sh)

#!/usr/bin/env bash
set -euo pipefail

# --- CONFIGURATION ---
MANIFEST="/tank/archival-masters/manifest.sha256"
LOG_TAG="fixity-audit"
PROM_TEXTFILE_DIR="/var/lib/node_exporter/textfile_collector"
PROM_METRIC_FILE="${PROM_TEXTFILE_DIR}/archival_fixity.prom"

# Optional Alerting Endpoints (Set via environment or leave empty)
ALERT_EMAIL="${ALERT_EMAIL:-sysadmin@institution.edu}"
GENERIC_WEBHOOK_URL="${GENERIC_WEBHOOK_URL:-}" # PagerDuty, Opsgenie, Teams, ServiceNow

# Log directly to host systemd journald / syslog
log_info()  { logger -t "$LOG_TAG" -p daemon.info "$1"; }
log_error() { logger -t "$LOG_TAG" -p daemon.err "$1"; }

log_info "Starting scheduled fixity audit..."

if [ ! -f "$MANIFEST" ]; then
    log_error "CRITICAL: Manifest file missing at $MANIFEST"
    exit 1
fi

START_TIME=$(date +%s)
FAILED_COUNT=0

# Run hash verification
if ! Output=$(sha256sum -c "$MANIFEST" 2>&1); then
    # Calculate failed files count
    FAILED_COUNT=$(echo "$Output" | grep -c "FAILED" || true)
fi

END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))

# --- EXPORT PROMETHEUS METRICS ---
if [ -d "$PROM_TEXTFILE_DIR" ]; then
    cat <<EOF> "${PROM_METRIC_FILE}.tmp"
# HELP archival_fixity_failed_files Count of files failing SHA-256 fixity audit.
# TYPE archival_fixity_failed_files gauge
archival_fixity_failed_files ${FAILED_COUNT}
# HELP archival_fixity_last_run_timestamp Epoch timestamp of last fixity audit.
# TYPE archival_fixity_last_run_timestamp gauge
archival_fixity_last_run_timestamp ${END_TIME}
# HELP archival_fixity_duration_seconds Execution time of fixity audit.
# TYPE archival_fixity_duration_seconds gauge
archival_fixity_duration_seconds ${DURATION}
EOF
    mv "${PROM_METRIC_FILE}.tmp" "$PROM_METRIC_FILE"
fi

# --- HANDLE AUDIT RESULT ---
if [ "$FAILED_COUNT" -eq 0 ]; then
    log_info "SUCCESS: Fixity audit passed. All archival objects verified intact."
    exit 0
else
    ALERT_MSG="CRITICAL ALERT: Archival Fixity Audit FAILED! ${FAILED_COUNT} file(s) failed SHA-256 integrity checks."
    log_error "$ALERT_MSG"

    # 1. Dispatch POSIX Mail / Ticketing Alert (ServiceNow / Jira Ingest)
    if command -v mailx &> /dev/null && [ -n "$ALERT_EMAIL" ]; then
        echo -e "Fixity Audit Failure Report\n\n$Output" | mailx -s "[CRITICAL] Fixity Failure - Archival Vault" "$ALERT_EMAIL"
    fi

    # 2. Dispatch Enterprise Webhook (PagerDuty / Webex / Teams / Generic API)
    if [ -n "$GENERIC_WEBHOOK_URL" ]; then
        curl -s -X POST -H "Content-Type: application/json" \
             -d "{\"summary\": \"$ALERT_MSG\", \"severity\": \"critical\", \"source\": \"$(hostname)\"}" \
             "$GENERIC_WEBHOOK_URL" || true
    fi

    exit 1
fi

Make the script executable: 

chmod +x /opt/scripts/fixity_audit.sh

Systemd Automated Timer

Create /etc/systemd/system/fixity-audit.service:

[Unit]
Description=HistoryLabs Archival Fixity Audit Service
After=network.target

[Service]
Type=oneshot
ExecStart=/opt/scripts/fixity_audit.sh
StandardOutput=journal
StandardError=journal

Create /etc/systemd/system/fixity-audit.timer:

[Unit]
Description=Run Archival Fixity Audit Weekly

[Timer]
OnCalendar=Sun *-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

Enable the timer on your host: 

sudo systemctl daemon-reload
sudo systemctl enable --now fixity-audit.timer