Troubleshooting Common SyslogSend ErrorsSyslogSend is a lightweight utility (or feature in many logging frameworks) used to forward log messages from applications and devices to a syslog server. While generally reliable, problems can arise at many points along the path — from local configuration to network transport to the receiving syslog daemon. This article covers common SyslogSend errors, how to diagnose their root causes, and practical fixes and preventive measures.
1. Common error categories and symptoms
- Network/connectivity failures — connection refused, timeouts, unreachable host.
- Authentication and encryption problems — TLS handshake failures, certificate errors.
- Message formatting and parsing issues — malformed messages, dropped structured data.
- Rate limiting and throttling — messages silently dropped or delayed.
- Local configuration mistakes — wrong destination, port, facility/severity mapping.
- Receiver-side issues — syslog server rejecting or filtering messages, disk space problems.
2. Collecting diagnostic information
Before changing configs, gather evidence:
- Local logs: check SyslogSend’s own logs (if available) and local syslog/journald entries.
- Network traces: use tcpdump, tshark, or Wireshark to capture traffic between the sender and syslog server.
- Connection tests: ping, traceroute, and nc (netcat) or openssl s_client for TLS.
- Receiver logs: inspect the syslog server (rsyslog, syslog-ng, Graylog, or SIEM) for incoming connection attempts, parsing errors, or rejections.
- Timing and rate data: record timestamps and counts to spot bursts or rate-limited drops.
3. Network/connectivity errors
Symptoms: “Connection refused”, “No route to host”, “Connection timed out.”
Diagnosis:
- Confirm destination IP/hostname and port (UDP typically 514, TCP often 514 or 6514 for TLS).
- Use ping and traceroute to verify basic reachability.
- Use nc or socat to test TCP connectivity: nc -v
- Capture packets with tcpdump: sudo tcpdump -i any host
and inspect whether UDP/TCP packets leave your host and whether responses arrive.
Common fixes:
- Correct hostname/IP and port in SyslogSend configuration.
- Open firewall rules on sender, receiver, and any intermediate firewalls for the chosen protocol/port.
- For UDP, remember it’s connectionless — “no response” doesn’t always mean failure; inspect packet counts on the server.
- For TCP, check that the syslog server’s listener is running and bound to the expected interface.
Example tcpdump command:
sudo tcpdump -n -i eth0 host 10.0.0.5 and port 514 4. TLS and certificate issues
Symptoms: “TLS handshake failed”, “certificate verify failed”, protocol version mismatch.
Diagnosis:
- Use openssl s_client to test TLS: openssl s_client -connect
: -showcerts - Check logs on both sides for handshake errors (unsupported ciphers, expired certs).
- Verify the certificate chain and that the sender trusts the CA used by the server.
Common fixes:
- Update or replace expired certificates.
- Ensure the SyslogSend client trusts the server’s CA (add CA to trust store or specify CA file).
- Align TLS protocol versions and ciphers between client and server.
- For mutual TLS (mTLS), confirm client certificate is present and valid.
Example openssl command:
openssl s_client -connect syslog.example.com:6514 -CAfile /etc/ssl/certs/ca-bundle.crt 5. Message formatting and parsing errors
Symptoms: Logs show “malformed message” or structured data missing; syslog server reports parsing failures.
Diagnosis:
- Inspect raw messages (tcpdump -A or Wireshark) to see exact payload.
- Confirm whether the sender uses RFC 3164 (BSD syslog), RFC 5424 (modern syslog), or JSON/CEF/LEEF structured formats.
- Check for null bytes, incorrect PRI values, or missing timestamp/hostname fields.
Common fixes:
- Configure SyslogSend to emit the format expected by your syslog server (RFC 5424 vs RFC 3164).
- Encode structured data properly (e.g., JSON must be valid and the receiver configured to parse JSON).
- Sanitize application messages to remove control characters or excessive length that could break parsers.
6. Rate limiting, throttling, and message loss
Symptoms: Bursts of logs disappear; “dropped X messages” entries in logs.
Diagnosis:
- Check sender and receiver for rate-limit or drop counters.
- Network devices (load balancers, firewalls) may also apply rate limits.
- Use a packet capture to see if UDP packets are transmitted but not received.
Common fixes:
- Switch to TCP or TLS if reliable delivery is required.
- Implement local buffering or queuing in SyslogSend to handle bursts.
- Tune rate-limiting parameters on the receiver or remove overly aggressive firewall rate limits.
- Use batching or compressing when supported to reduce per-message overhead.
7. Local configuration mistakes
Symptoms: No traffic, logs not arriving, messages labeled with wrong host/facility/severity.
Diagnosis:
- Re-check SyslogSend config file for destination, port, protocol, tag, facility, and severity mappings.
- Ensure service is running and enabled; check systemctl status (if applicable).
- Validate syntax with any provided config-checking tools or by restarting the service and watching logs.
Common fixes:
- Correct mistyped hostnames/ports; ensure correct protocol selection (UDP vs TCP vs TLS).
- Set correct hostname or use “app-name” tagging if you want source identification.
- Ensure the process has permission to bind to privileged ports (or use higher ports and NAT).
Example systemctl check:
sudo systemctl status syslogsend.service sudo journalctl -u syslogsend -f 8. Receiver-side problems
Symptoms: Messages arrive but are rejected, filtered, or not indexed.
Diagnosis:
- Inspect receiver logs for parsing errors, permission or quota issues, disk full, or retention policy hits.
- Confirm listener configuration (rsyslog/syslog-ng) — is it expecting TLS? Is it bound to the right interface/port?
Common fixes:
- Adjust receiver filters/inputs to accept the incoming format and source.
- Free disk space or increase quotas for logging storage.
- Tune parser rules or add multiline parsers if messages are split across lines.
9. Debugging examples and step-by-step checks
Quick checklist to run in order:
- Verify service is running: systemctl status syslogsend.
- Confirm configuration: check destination, port, protocol, and format.
- Test network reachability: ping/traceroute.
- Test port connectivity: nc -v host port (TCP) or use hping3 for UDP tests.
- Capture traffic: tcpdump -n -i any host
and inspect payloads. - Check TLS: openssl s_client -connect host:port -showcerts.
- Review sender and receiver logs simultaneously to correlate timestamps.
- Temporarily route logs to a local file to ensure the sender generates entries.
10. Preventive measures and best practices
- Use TCP/TLS for reliable, secure transport where possible.
- Implement retries and local persistence (disk-based queue) in the sender.
- Standardize on RFC 5424 and include structured data when helpful.
- Monitor drop counters and set alerts for message loss or high latency.
- Maintain certificate lifecycles and automate renewals (e.g., certbot, ACME).
- Keep time synchronized (NTP/chrony) to avoid timestamp-related parsing issues.
- Document your logging topology (senders, collectors, filters) and configurations.
11. When to escalate
- Persistent TLS handshake failures after validating certs and ciphers — involve server-side admins.
- High-volume unexplained drops indicating network equipment rate limiting — involve network team.
- Suspected bugs in SyslogSend itself — check project issue tracker, update to latest stable release, or contact vendor/support.
12. Quick reference commands
- Check service: sudo systemctl status syslogsend
- Capture packets: sudo tcpdump -n -i any host
and port <514/6514> - Test TCP connect: nc -v
- Test TLS: openssl s_client -connect
: -CAfile /path/to/ca.pem - Follow logs: sudo journalctl -u syslogsend -f
Troubleshooting SyslogSend usually follows standard logging and network debugging practices: gather evidence, isolate layer (application, transport, network, receiver), and apply targeted fixes. With systematic checks and proper logging architecture (TLS, reliable transport, local buffering), most common errors can be prevented or resolved quickly.
Leave a Reply