Quick Log Collection Troubleshooting

Posted on Updated on

We already discussed about how configure log sources, and how configure QRadar to receive the logs. Let’s say that everything is ready, you are in front of the customer, and the logs doesn’t show up, do you know how to troubleshoot it? Here is some quick troubleshooting tips, that can help you in those situations:

  • Verify the connectivity between the log source and the QRadar collector:
    • You can simply ping from the log source to the collector;
    • By default, the IP-Tables from QRadar drop pings, so you will need to stop the iptables process in the QRadar collector. You can do it opening the terminal (or ssh) in the QRadar and using the following command:
      services iptables stop ;
    • If you cannot even ping the QRadar server from your log source, the issue is the network;
    • Don’t forget to restart the IPtables after testing, just use the following command:
      services iptables start ;
  • Verify the firewalls between the log source and the QRadar:
    • The firewalls should allow the ports used to collect. For example, for collecting syslog, the firewalls should allow the port 514/UDP;
    • If you have no access to the firewall, a simple way to test the firewall is using the telnet command from the logsource to the QRadar:  telnet [IP] [PORT]
      Example: telnet 10.1.1.1 514
    • If the telnet doesn’t work, some firewall is dropping the packets on the specified port, you should ask for a firewall rule allowing the traffic;
  • Verify the flows coming in the QRadar collector:
    • You can use the command tcpdump in the QRadar to verify if the packets are being received in the QRadar;
    • Syntax: tcpdump -i [INTERFACE] src host [IP-LOGSOURCE] port [PORT]
    • Example: tcpdump -i eth0 src host 10.2.2.2 port 514
    • If nothing shows up, there is some network issue dropping the packets or the log source is not properly configured;
  • Verify the QRadar Logs:
    • The QRadar logs are stored in the following folder: /var/log/
    • The main log is named qradar.log
    • You can simple access and monitor the log using the following command: tail –f  /var/log/qradar.log
    • You can verify the current EPS using the following command:
      tail –f  /var/log/qradar.log | grep ‘Events per Second’

I hope this post help you guys to troubleshoot collecting problems on QRadar. If you have any question or suggestion, please leave us a comment!

Creating a Cold Backup

Posted on Updated on

The QRadar solution offers two types of license by default: High Availability and Disaster Recovery. These licenses can be very useful in medium-large environments making the system more reliable. Both of licenses need to be purchased separately from the base QRadar license, and we know that most of the cases the clients want a solution to reach the compliance levels but with no extra cost. So one possible solution is creating a Cold Backup. You will not find it on the regular IBM documentation, so make sure that you follow carefully the steps of this post, the procedure is easy for people who already have QRadar experience.

Just a quick stop to explain what is a cold backup: Basically is a “clone” from the primary server, and it has the same configuration than the main server but stay always powered off. In case of some failure in the main server the staff should manually power the cold backup on. After restoring the primary server (and before turning it on), the cold backup should be powered off manually. This solution in most of the cases don’t need an extra license, you can use the same than the primary server but should NEVER have the both servers online at same time. Please consult your IBM sales representative before considering the cold backup, the laws can change between countries.

So, here is the high level steps. If you have any question on the steps, please leave a comment. Make sure that you understand all the process before doing it.

  1. Verify and take note of all the network configuration of the Primary server. You should have: IP, DNS, Gateway, hostname, email server, etc;
  2. Create an configuration backup of your primary QRadar;
  3. Turn off the primary QRadar server;
  4. Install (or re-install) the QRadar in the cold backup server using the network information gathered on the step 1;
  5. Apply on the cold backup the same license file than the primary server;
  6. After the finish of the installation, access the web interface of the cold backup, and import the backup generated on the step 2;
  7. Verify the logs collection and all the imported configuration;
  8. Turn off the cold backup server;
  9. Turn on the primary server;
  10. Finish;

Some considerations about this cold backup solution:

  • The primary and cold backup should NEVER be on at same time. Make sure that you power off one server before turning on the other;
  • All the transition process is manual, so when you have a failure in the primary you should manually turn the primary off and turn the cold backup on;
  • The cold backup server may be not supported by the IBM official support. You should always consider buying a High-Availability or Disaster-Recovery license;
  • The log data from the Primary will not be on the cold backup, and the log data in the cold backup will not be in the primary. To synchronize the data between the two servers you can use a external storage or manually import the data;
  • Please make sure that you understand the whole process before doing it, we are not responsible for any misconfiguration issue;
  • The both servers need to be in the same subnet;
  • Remember that every configuration done in the Primary server should be replicated to the cold backup. To do it, just export the configuration from the primary (step 2) and import on the cold backup (step 6);
  • Once a month run the updates on the ColdBackup to keep it updated;
  • And again: Ensure that the both servers are never online at same time;

And as always, if you have any question or sugestion, let us know in the comments!

Configuring the Log Sources

Posted on Updated on

When implementing a large QRadar environment we can face several types of log sources across the network. QRadar support more than one hundred type of devices out-of-the-box and can integrate with any another log source using customized parsers. The log source parsers are known in QRadar as Device Support Modules (DSMs).

A personal recommendation to integrate log sources with QRadar is: always use syslog when it is possible. Why? Basically syslog is the standard log protocol for many devices, and QRadar can easily collect, identify and receive logs using this protocol. The syslog typically uses UDP connections, so make the log collection more fast and with almost zero latency. Make sure that all the firewalls of your environment allow traffic to QRadar in the port 514 (default syslog port).

IBM provide a good documentation explaining thorougly how to configure each type of device to send logs to QRadar. You can find the DSM configuration guide in the following link:

QRadar_721_DSMConfigurationGuide

Do you have another tips to configure your devices? Share with us!

Storage Sizing

Posted on Updated on

In the last post we discussed how to calculate the EPS of our environment. Now lets discuss how to calculate the required size of the storage, since with the EPS in hands it turns way easier to calculate the size of our database. In this scenario we will consider only the log storage, not considering the network flows storage.

First of all, we need to understand how the data is stored on QRadar. Basically, you have 3 types of data:

  • Online live data: All the events can be accessed with no latency. In this case the data is not compacted;
  • Online compacted data: All the events can be accessed but with a small latency because the data is compacted. The avarage compression rate is 10:1;
  • Offline data: All the events cannot be accessed instantly because all the data is in a external backup server. To access this data the user should import the backup into the QRadar (or into a QRadar Virtual Machine) for analysis;

After understanding which each type of  data represents, we can start to calculate the storage based on the requirements of the project. In the sizing, we only use the Online data, the offline backup is not considered (since it is a external independent server).

To make an easy explanation, lets use the following requirements:
[Online Live Data: 7 days; Online Compacted: 180 days; EPS: 2500]

Steps to calculate:

  • Calculate how much data is generated each second: Multiply the EPS by 300 bytes (the average size of an log):
    In the example: 2500 x 300 = 750000 bytes = 732.5 kb/s
  • With the Data Per Second, we can calculate how much data we have in one day (1 day = 86400 seconds):
    In the example: 732.5 * 86400 = 63288000 kb/day = 61804.7 Mb/day = 60.4 Gb/day
  • Now that we know how much data is generated in one day, lets calculate the Online Live Data size (non-compacted):
    In the example: 60.4Gb/day * 7 = 422.8Gb
  • Now, lets calculate the Online Compacted Data. Note that the average compression rate is 10:1 :
    In the example: 180 days – 7 days (online live data) = 173 days
    173 days * 60.4Gb = 10449.2 Gb
    10449.2Gb * 0,1 (compression rate) = 1044.92Gb
  • We have the size of the online live and the online compacted data. Now we just need to sum both and we have the final size:
    In the example: 422.8Gb + 1044.92Gb = 1467.72Gb = 1.43Tb

Following this basic steps we can have a accurate approximation of the necessary storage size. A good practice is using a storage 20% bigger than the estimated.

Do you have any another experience with storage sizing? Let us know in the comments!

UPDATE: According to one of our readers (see comments), starting from the version 7.2.7, the stored data will always be compressed. So, if you are sizing your environment for the latest QRadar version, you should use only the “compressed data” calculations.

QRadar Sizing – Determining EPS

Posted on Updated on

One of the biggest challenges when sizing a QRadar implementation is estimating the Events Per Second (aka. EPS) of the environment, specially because in the most of the cases we don’t have full access to the log sources to precisely determine the EPS. So in this post we will review some tips about how to estimate the EPS.

Determining the EPS of one event source with access to the system or access to the logfiles.

# Dump the log in a file and delete all the log not from the past 24h. Leave only the last 24h of logs
– If the system generate syslog, follow these steps:
    a. Configure the logsource to send the logs to any linux server
    b. In the destination linux server execute the following command: tcpdump -i eth0 src host SOURCE_IP dst port 514
c. Run the command for exactly 24 hours in a regular day and verify how many log packets you got.

# Verify the number of logs in the file.
– If there is just one log per line, simple open the file on notepad and verify how many lines you have;
– If the logs are not one per line, verify the whole size of the file (in bytes) and divide by 250 (the avarage size of a log line). Example: File with 3Mb = 3145728 bytes / 250 = 12583 Log packets

# Divide the number of packets by 86400, the result will be the EPS of the log source

Determining the EPS without access to logs or the system:
# From my previous experience, a good approximation of EPS is:

Device Type EPS
Active Directory 15
IIS or Exchange 10
General Windows Server 2
General Windows Workstation 0,5
UNIX/Linux Server 0,5
DNS or DHCP 15
AntiVirus Server 20
Database 1
Proxy 25
Core/Border Firewall 150
Small Firewall 20
IPS, IDS or DAM 5
VPN 5
Routers/Switches 0,25

Calculating the EPS of the whole environment:
# Multiply the number of each device by the estimated EPS
# Sum the EPS of all kind of devices and you will have the EPS of your whole environment
– Example:
     3 Core Routers + 2 IPS = 3x 150 + 2x 5 = 460 EPS
Remember to always consider at least 20% margin for buying your license.

Do you have any another tips to calculate EPS? Let us know in the comments!

SIEM Magic Quadrant 2013

Posted on Updated on

The Gartner Group published in June of this year the result of the Magic Quadrand for SIEM solutions. For the folks that don’t know what is a magic quadrant, it is a chart comparison between all the SIEM solutions in therms of “ability to execute” and “completeness of vision”, dividing the competitors in 4 categories: Niche Players, Challengers, Visionaries and Leaders.

Since IBM aquired the Q1Labs, every year the QRadar is classified as Leader, and in 2013 it was not different. The following image is the Gatner Magic Quadrant for SIEM solutions of 2013:

Magic-Quadrant-for-Security-Information-and-Event-Management_IBM-Q1-Labs-HP-Arcsight-McAfee

Just to remember, this chart evaluate only the SIEM solution, not considering the several another features from QRadar (Risk Management, Vulnerability Assessment/Management, Network Analysis, etc).

To read more about the results of 2013, read this article in the Security Intelligence blog.

What is your opinion about the QRadar facing the competitors? Leave a comment!

Migration from IBM TSIEM to QRadar

Posted on Updated on

A couple of months ago one friend ask me how to migrate from the IBM TSIEM solution to the QRadar.

Pause: For the folks that don’t know what is the Tivoli Security Information and Event Management (aka. TSIEM), it was the old SIEM solution from IBM that was discontinued in 2011, when IBM acquired the Q1Labs (and the QRadar). Who only worked with TSIEM will be impressed about how simple QRadar is compared to the TSIEM.

After some research, I found this documentation from IBM explaining thoroughly how to migrate from the old TSIEM solution to the QRadar. The documentation is based on the version  7.0 of the QRadar, but it can be easily used as base for the migration for any QRadar version (including the new 7.2).