Preparing for Log Collection

To send your logs to the Insight Platform, you can forward them from a Security Information and Event Management system (SIEM) or you can collect the log events directly from the log sources.  If you will be forwarding your logs from a SIEM, please read the Forwarding Logs from a SIEM section.  Otherwise, skip this section and continue on to the section for each type of log data that you will have the Insight Platform ingest.  Please note that you can combine these two methods and forward some log event types from the SIEM and then collect the rest directly.

To see a list of event sources, please see Supported Event Sources.

A Rapid7 collector requires each stream of syslog logs to be sent to it on a unique TCP or UDP port.  You will need to configure each device that will send logs using syslog to send the logs over a TCP or UDP port that is unique on that collector.  It is common to start sending the logs using port 10000, although you may use any open unique port. Also, for Linux collectors, the ports used must be higher than 1024.

Forwarding logs from a SIEM

InsightIDR supports the forwarding of logs from the following SIEM/log aggregation products:

  • HP ArcSight
  • LogRhythm
  • McAfee Enterprise Security Manager (formerly Nitrosecurity)
  • Splunk
  • IBM QRadar
  • FireEye Threat Analytics Platform (TAP)

For all SIEM/log aggregation productions, you will need to follow the vendor documentation to forward the log/event data to a collector using standard syslog for both the log format and also the transport methodology.  Additional information regarding Splunk and QRadar is provided below.

Before your Insight deployment, if you will be forwarding logs from your SIEM, you should be prepared to perform the necessary steps on the SIEM.  You can either complete the setup before the deployment or complete the setup with your Rapid7 Consultant during the deployment.

Splunk

You will need to install and configure the Rapid7 App on each node of your index cluster. 

To learn how to get the Rapid7 App for Splunk Enterprise, please see:  https://community.rapid7.com/docs/DOC-3130.

For more information on Splunk, you can visit:

IBM QRadar

There are two steps for adding a collector as a forward destination in QRadar.

  1. Follow IBM’s documentation to add a collector as a forward destination: http://www-01.ibm.com/support/knowledgecenter/SS42VS_7.2.1/com.ibm.qradar.doc_7.2.1/t_qradar_adm_add_frwrd_dest.html?cp=SS42VS_7.2.1%2F4-0-15-0.
  2. After you have added a collector as a forwad destination, you will need to perform only one of the following steps. 

Preparing User Attribution event sources

 User attribution event sources are required for user behavioral analytics, which is a critical component of InsightIDR.  You should plan to ingest all of these log sources in InsightIDR. 

User attribution event sources include:

  • LDAP
  • Active Directory (domain controller security logs)
  • DHCP

Configuring LDAP event sources

Adding a Lightweight Directory Access Protocol (LDAP) server allows Insight to track the users, admins, and security groups contained in the domain.  LDAP automatically mirrors data across all LDAP servers; thus, even if you have multiple LDAP servers, you will only need to configure one LDAP event source, unless you have manually disabled the auto-mirror feature.

To add the LDAP event source, you will need:

  • An Active Directory account that has read access to user and group objects in the domain.  You may use the same account that you are using to collect Active Directory event sources.
  • Port 636 (LDAPS) to be open between the Collector and the LDAP server.

Configuring Active Directory (AD) event sources

The Insight Platform can collect significant events from the security log on domain controllers.  You should add in one Active Directory (AD) event source for each domain controller in your organization.

To prepare to collect Active Directory event sources, you need to:

  • Open ports 135, 139, and 445 between the collector and the AD event source for each domain controller.
  • Access to a domain account that is a member of the Domain Admins group

For more information on AD sources, please see Setting Up Active Directory Event Sources.

Configuring Microsoft DHCP event sources

The Insight Platform can collect DHCP audit logs.  To prepare to collect the DHCP audit trail, you need:

  • The DHCP logs to be written into a folder that the collector can connect to as a network share.  This folder should be changed from the default location and should contain only the DHCP logs. 

For more information on Microsoft DHCP and DNS event sources, please see Setting Up DHCP and DNS Event Sources.

Configuring other (Non-Microsoft) DHCP event sources

The Insight Platform can collect DHCP logs from the following types of DHCP servers:

  • Alcatel-Lucent VitalQIP
  • Bluecat DNS/DHCP
  • Cisco IOS
  • Cisco Meraki DHCP
  • InfoBlox Trinzic
  • ISC DHCP
  • MicroTik DHCP Server
  • Sophos UTM

For all of these event source types, the DHCP host must be able to send the DHCP logs using standard syslog. 

Adding a syslog DHCP event source

In order to add a syslog DHCP event source, you will need to:

  • Ensure that the DHCP host is logging all DHCP activity.
  • Configure the DHCP host to send logs to a collector on a unique UDP or TCP port (above 1024) and by specifying it as a syslog server.

Preparing Additional Event Sources

 Besides the user attribution event sources, you can ingest additional high-value logs into your InsightIDR platform.  These additional events allow you to search and analyze data across your entire environment.  You should ingest all of these types of event sources if possible.

Configuring Microsoft DNS Event Sources

The Insight Platform can collect Microsoft DNS audit logs. 

To prepare to collect the logs, you need the DNS log to be written into a folder that the collector can connect to as a network share.  For more information on Microsoft DHCP and DNS event sources, please see Setting Up DHCP and DNS Event Sources.

Configuring Non-Microsoft DNS Event Sources

Besides Microsoft DNS, the Insight Platform can collect DNS logs from these types of event sources:

  • ISC Bind9
  • PowerDNS
  • InfoBlox Trinzic*

For these event source types, the DNS host must be able to send the DNS logs using standard syslog. 

In order to add a syslog DNS event source, you will need to prepare the following:

  • Ensure that the DNS host is logging all DNS activity
  • Configure the DNS host to send logs to a collector on a unique UDP or TCP port (above 1024) and by specifying it as a syslog server

Please review the specific vendor documentation on forwarding logs using syslog to configure the sending devices.

If your InfoBlox system is providing both DNS and DHCP services in your environment, send all DHCP and DNS logs to the same UDP/TCP port.  In Insight, you will configure only a “DHCP” event source for Infoblox, and this DHCP event source will parse both DHCP and DNS events.

Configuring Firewall Event Sources

To collect firewall traffic and other log data, you need to:

  • Configure the firewall to send syslog to the collector on a unique UDP or TCP port (above 1024)
  • If you have a Checkpoint firewall, see the Checkpoint OPSEC LEA guide:  https://community.rapid7.com/docs/DOC-3128. Also, please note that Checkpoint firewall logs can only be collected by a collector running on a Windows OS; it is not possible to collect Checkpoint logs from a collector running on Linux.

If the firewall has additional modules for VPN, malware detection, IDS/IPS, web proxy, etc., you should enable logging on the firewall for all of these modules and configure the logs to be sent using syslog.  You can either send all of the log data from the device to the same port, in which case you will have one event source in Insight for the device, or you can send each type of log data to a different port, in which case you will have a separate event source for each type of log data.  Technically, it does not matter which way you send the log data to Insight; using one port for all traffic means you will have one event source and using different ports means you will have multiple event sources.

Similarly, if you have several firewalls that are the same manufacturer and model, you can configure all of them to send their log data over the same port or you can configure each firewall to send to a unique port.  If you configure all the firewalls to send log data to the same port, such as UDP port 10000, then you have one event source in Insight for all of the firewalls.  On the other hand, if you configure each firewall to send to a different, unique port, for example Firewall1 sends on UDP port 10001 and Firewall2 sends on UDP port 10002, etc, then you have a separate event source for each firewall.  The first method is usually easier to configure but more difficult to manage in InsightIDR as it is more difficult to troubleshoot issues with the event sources if there is only one for all firewalls.

Configuring VPN Event Sources

The Insight Platform supports these types of VPN logs and collection methods:

Device Type Can Forward Using Syslog Can Forward from a SIEM or Log Aggregator Can Read Logs from a Folder

Cisco ASA VPN

Yes

Yes

No

F5 Networks FirePass

Yes

Yes

No

Juniper SA

Yes

Yes

No

Microsoft IAS (RADIUS)

Yes

Yes

Yes

Microsoft Network Policy Server

Yes

Yes

Yes

Microsoft Remote Web Access

Yes

Yes

Yes

MobilityGuard OneGate

Yes

Yes

No

NetScaler VPN

Yes

Yes

No

OpenVPN

Yes

Yes

No

VMware Horizon

Yes

Yes

No

Collecting VPN logs with syslog

To collect VPN logs using sylog:

  • Configure the VPN device to send syslog to the collector on a unique UDP or TCP port (above 1024)
  • Document the IP address ranges used by VPN (this will be entered into the Insight Platform during the deployment)

To collect VPN logs from a folder for those device types that support it, the folder contains the logs from the device.  To prepare to collect the VPN logs using this method, you need t he VPN logs to be written into a folder that the collector can connect to as a network share. Please review the specific vendor documentation on forwarding logs using syslog to configure the sending devices.

Configuring IDS/IPS event sources

The Insight Platform can collect events from these types of IDS/IPS devices:

  • Corero IPS
  • Dell iSensor
  • HP TippingPoint
  • McAfee IDS
  • Metaflows IDS
  • Security Onion
  • Snort IDS
  • Sourcefire 3D

To collect IDS/IPS events, you need to configure the device to send syslog to the collector on a unique UDP or TCP port (above 1024).

Configuring Email/ActiveSync event sources

Adding in email and ActiveSync logs helps Insight track your user’s devices, track user locations with ActiveSync and OWA, and investigate malicious links from emails.

Integrating Cloud Services Event Sources

The Insight Platform can ingest logs from cloud services.  The following cloud event source types are supported:

  • AWS Cloud Trail
  • Box.com
  • Duo Security
  • Google Apps
  • Office 365
  • Okta.com
  • Salesforce.com

To prepare your cloud services, please see the Cloud Services Integration Guide:https://community.rapid7.com/docs/DOC-3273.

Configuring Antivirus event sources

Collecting antivirus events allows for more contextual information to be added to an asset.  The only type of AV event that is parsed in Insight is when a virus is detected by the AV software.  Collecting the AV events let you view viruses found on an asset when looking at the asset in Insight. 

These types of Antivirus servers are supported:

  • Cylance Protect
  • ESET Antivirus
  • F-Secure
  • McAfee ePO
  • Sophos Enduser Protection
  • Symantec Endpoint Protection
  • Trend Micro Control Manager
  • Trend Micro OfficeScan

For other antivirus products, use the vendor documentation to configure the antivirus server to send syslog to the collector on a unique UDP or TCP port (above 1024).

McAfee ePO

The only way to ingest ePO logs into Insight is to forward them from a log aggregator or SIEM.  Configure the log aggregator or SIEM to forward the logs in standard syslog format.

Sophos Enduser Protection

Sophos events are stored in a SQL server database on the Sophos server.  To pull the events from the database, you need:

  • A credential that has permissions to read the Sophos database.
  • The name of the Sophos database. Sophos changes the database name with each version of the product.
  • To configure the SQL Server to support all protocols: shared memory, named pipes, and TCP/IP on the Sophos server.

For TrendMicro OfficeScan:

TrendMicro cannot forward logs natively.  However, you can forward them into Insight from a log aggregator or SIEM.  Configure the log aggregator or SIEM to forward the logs in standard syslog format.

You can use the free tool Nxlog to read the Application log on the Trend server and forward the antivirus events to Insight from it.  More information on Nxlog can be obtained from: http://nxlog.org/products/nxlog-community-edition.

Integrating advanced malware event sources

At this time, the Insight Platform has a log parser for FireEye NX. However, if you have a malware detection module as part of your firewall, those events can forwarded as part of the regular firewall traffic.

To collect the FireEye events, you will need to configure the device to send syslog to the collector on a unique UDP or TCP port (above 1024).  Check the FireEye documentation for specifics on how to configure the FireEye.

Integrating Web Proxy event sources

To collect web proxy events, you need to configure the device to send syslog to the collector on a unique TCP or UDP port (above 1024).

The Insight Platform can ingest logs from these web proxies:

  • Barracuda Web Filter
  • Blue Coat ProxySG*
  • Cisco IronPort
  • McAfee Web Reporter
  • Sophos Secure Web Gateway
  • Squid Web Proxy
  • WebSense Web Security Gateway
  • Zscaler NSS**

* Blue Coat web proxies log using ELFF format, which allows you to define the fields that get logged in any order.  Insight requires the following fields to be at the front of the log for the parser to work.  Any extra fields can be added after these fields and will not affect the parsing of the log:

date time c-ip s-action sc-status cs-method cs-uri-scheme cs-uri cs(User-Agent) cs-bytes sc-bytes sc-filter-result

** Zscaler NSS supports multiple logging formats; however InsightIDR currently has a parser for the QRadar LEEF format only.

Database Monitoring Event Sources

Database administrative activity can be tracked for Microsoft SQL Server.  In order to collect this activity, you must install a Snare agent on each SQL Server.  Both an enterprise and a free, open-source Snare agent is available; the Enterprise Snare agent is recommended. To learn more about thr differences between the Enterprise and open-source agents, please visithttps://www.intersectalliance.com/our-product/snare-agent/enterprise-vs-opensource/.

To download the free, open-source Snare agent, go to https://www.intersectalliance.com/try-snare-opensource-now/.

To learn how to install the Snare agent, please see https://community.rapid7.com/docs/DOC-3150.

What's Next?