Explanation:
| From the Syslog configuration, remove the facilities that send CEF messages. | CEF1 | | From the Log Analytics agent, disable Syslog synchronization. | Server2 | The goal is to eliminate duplicate events in the Azure Sentinel workspace (SW1). Duplication typically occurs when the same log source is sending data to Azure Sentinel via multiple collection methods.
Analysis of the Environment
* SW1 is the Azure Sentinel (now Microsoft Sentinel) workspace, which is the final destination for all logs.
* CEF1 is a Linux server configured as a log forwarder (often called a CEF collector) for Microsoft Sentinel. It uses the Log Analytics agent (or the newer Azure Monitor Agent) to ingest logs and is specifically configured to forward Common Event Format (CEF) logs to SW1.
* Server1 sends CEF logs to CEF1. This is the intended, single collection path for Server1's CEF logs:
Server1 CEF1 SW1. No duplication is inherent here.
* Server2 sends Syslog logs to CEF1. This path is: Server2 CEF1 SW1.
* Since CEF1 is running the Log Analytics agent (required to forward logs to SW1) and is configured to collect Syslog data (to receive Server2's logs), the Log Analytics agent on CEF1 will also attempt to ingest the Syslog messages it receives into SW1.
* However, the Log Analytics agent itself can also be used to collect Syslog/CEF logs directly from the source server.
Addressing Duplication
Duplication is most likely to occur if a server is sending the same logs to a forwarder AND also has the Log Analytics agent configured to send the same logs directly to SW1.
Action 1: From the Syslog configuration, remove the facilities that send CEF messages.
* Resource: CEF1
* Reasoning: CEF1 is a Linux server running the Log Analytics agent and is acting as the collector.
Server1 sends CEF logs to CEF1. These CEF logs are transmitted using Syslog (specifically, a custom Syslog format). If the Log Analytics agent on CEF1 is configured to collect all Syslog facilities, it will ingest the raw CEF Syslog messages it receives from Server1 AND also ingest the parsed CEF messages via its custom forwarding logic. To prevent the Syslog collector on CEF1 from ingesting the raw CEF messages that it is supposed to be forwarding, you must modify its Syslog configuration (e.g., in /etc/rsyslog.conf or equivalent) to ignore the facilities/log files used by the incoming CEF messages from Server1. The primary purpose of CEF1 is to receive and forward CEF, not to have its Log Analytics agent ingest the raw Syslog that transports the CEF payload.
Action 2: From the Log Analytics agent, disable Syslog synchronization.
* Resource: Server2
* Reasoning: Server2 is configured to send Syslog logs to CEF1 (Server2 CEF1 SW1). Since Server2 is a Linux server, it may also have the Log Analytics agent installed for other monitoring purposes. If the Log Analytics agent on Server2 is installed, it is configured by default to collect Syslog logs directly and send them to SW1 (Server2 SW1). This creates a duplicate path for the Syslog data:
* Path A (Intended): Server2 Syslog CEF1 SW1
* Path B (Duplication): Server2 Log Analytics Agent Syslog SW1
* According to Microsoft Sentinel documentation on log ingestion, when using a dedicated forwarder (like CEF1) for Syslog/CEF, you must disable the Syslog collection on the Log Analytics agent of the source machine (Server2) to prevent this duplication. This is typically done by disabling Syslog synchronization in the Log Analytics agent configuration or removing the Syslog entry from the agent's data sources.
References: Microsoft Sentinel documentation on data connectors for Syslog and CEF, specifically the sections discussing the deployment of the Log Analytics agent and forwarders, which repeatedly warn about the need to prevent dual-ingestion of the same log type (Syslog or CEF) from both the source server's agent and a dedicated collector/forwarder.