Forum Discussion

Deepanshu_Marwah's avatar
Deepanshu_Marwah
Brass Contributor
Jun 23, 2021
Solved

Urgent !! CEF Syslog duplication Issue

Hi All 

I have configured a Fortinet integration with Azure sentinel on local7 facility. My current configuration is ingesting Fortinet logs in both the tables `CommonSecurityLog` and `syslog`. The number of logs are quite large and duplication may cost double cost. Please help to fix this issue 

 

 

Actions tried:

 

1. Performed steps as mentioned in the documentation.

 

 

 

 

Using the same machine to forward both plain Syslog and CEF messages

If you plan to use this log forwarder machine to forward Syslog messages as well as CEF, then in order to avoid the duplication of events to the Syslog and CommonSecurityLog tables:

On each source machine that sends logs to the forwarder in CEF format, you must edit the Syslog configuration file to remove the facilities that are being used to send CEF messages. This way, the facilities that are sent in CEF won't also be sent in Syslog. See Configure Syslog on Linux agent for detailed instructions on how to do this.

You must run the following command on those machines to disable the synchronization of the agent with the Syslog configuration in Azure Sentinel. This ensures that the configuration change you made in the previous step does not get overwritten.
sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/OMS_MetaConfigHelper.py --disable'

 

 

 

 

 

2. Tried removing syslog facilities on log analytics workspace => agent configuration => syslog

 

 

  • RogerS's avatar
    RogerS
    Copper Contributor
    Jan 28, 2025

    I have created a new file in /etc/rsyslog.d/ called 10-filter-CEF.conf and I've tried all sorts of combinations to stop the CEF records from Fortigate firewall being duplicated into the syslog without success.
    Some of the combinations I've tried are below; none of which seem to work. Help!!!
    #:programname, contains, "CEF" /var/log/CEF.log
    #& stop
    #:hostname, contains, "prdvmfwall01" /var/log/CEF.log
    #& stop
    #:msg, contains, "prdvmfwall01 CEF" /var/log/CEF.log
    #& stop
    #:msg, contains, "0|Fortinet|Fortigate|" /var/log/CEF.log
    #& stop
    #:rawmsg, contains, "CEF:" /var/log/CEF.log
    #& stop
    #:rawmsg, contains, "CEF:" ~
    #if ($rawmsg contains "CEF:") then stop
    #$rawmsg contains "CEF:" then @@127.0.0.1:25226
    #& stop
    #:msg, contains, "CEF:" ~
    if ($programname == "CEF" or $rawmsg contains "CEF:") then stop
    #if ($rawmsg contains "CEF:") then stop

    • RogerS's avatar
      RogerS
      Copper Contributor
      Jan 30, 2025

      I'm now getting closer.

      I've created a file /etc/rsyslog.d/05-filter-CEF.conf that contains the following:
      if ($programname == "CEF") then @@127.0.0.1:28330 
      & stop

      By doing this I still get CEF records in the syslog.

      As suggested by some other articles I changed the port to 25226 but then don't get CEF records in syslog or CommonSecurityLog.

      Please can anybody help

    • RogerS's avatar
      RogerS
      Copper Contributor
      Jan 29, 2025

      I found the problem, I had to prefix the file with 05 so that it got processed before 10-azuremonitoragent-omfwd.conf. With 05 prefix most of the combinations above would work

  • CliveWatson's avatar
    CliveWatson
    Icon for Microsoft rankMicrosoft
    Jun 23, 2021
    For option 2, did you press SAVE after making the change? After that point (as it wont delete already ingested data) are you still getting 'new' duplicates into the Syslog table?

Resources