Today I was reading about the new QRadar integration with the IBM BigData solution. Instead of writing down here, I decided to share with you guys a very nice video that summarize the benefits of this integration.
(Part 1) QRadar Basics and Big Data
(Part 2) QRadar BigData Extension:
I hope you guys enjoy the videos. You can also check more from the author in his youtube channel.
IBM recently released the new “IBM Security QRadar Certified Deployment Professional” or also called ” IBM Security QRadar SIEM V7.1 Implementation”. For the most of the people certifications are just accomplishments to attach on their CV, but the real value of the certification is not the paper itself, but is the study to get the certification. Even people that work years with the product, when studying to the certification discover new features or new ways to work with the solution, and being certified (after the proper study) gives you the necessary confidence that at least you already saw all the features of the product and you are able to use the tool in its best way.
The new certification (code C2150-196) consists in a 90-minutes test containing 64 questions involving all the phases of the project. From installing the hardware to tuning the rules. As mentioned in the first paragraph, studying and getting certified will give you a broader vision about the product, not only the tasks that you are used. The test passing score is 70%, a high score compared to another certifications from IBM, and as it involves all phases of the project, you should dedicate part of you time to study the tool.
The best way to prepare yourself to the certification is exploring the tool. Don’t try to go to the certification having never even logged on QRadar. Another good source of information, is the study guide from IBM that you can find on this link. It basically provides you with all the topics of the certification.
A personal tip to you is focus in the following categories: Difference between the versions (SIEM, LogManager, etc); theory behind the offences (how it is generates, how to configure the rules, etc); Interface usage (where can you find the features, how to do things in the interface, etc); and Solution Architecture (Components).
Another suggestion for people who have budget for it, go for the IBM classes. I went to two QRadar courses (2 years ago) and both were very helpful and practical. The courses were filled with useful exercises and hands-on activities. The bad point is the prices, but usually the companies pays for the training. To learn more about the IBM QRadar course, check this link out.
After studying the study guide (or attending the official training), exploring the tool and practicing the theory, you will be good to go for the certification. To get more information about how to schedule your certification visit the official IBM learning center.
In the last post we discussed how to calculate the EPS of our environment. Now lets discuss how to calculate the required size of the storage, since with the EPS in hands it turns way easier to calculate the size of our database. In this scenario we will consider only the log storage, not considering the network flows storage.
First of all, we need to understand how the data is stored on QRadar. Basically, you have 3 types of data:
- Online live data: All the events can be accessed with no latency. In this case the data is not compacted;
- Online compacted data: All the events can be accessed but with a small latency because the data is compacted. The avarage compression rate is 10:1;
- Offline data: All the events cannot be accessed instantly because all the data is in a external backup server. To access this data the user should import the backup into the QRadar (or into a QRadar Virtual Machine) for analysis;
After understanding which each type of data represents, we can start to calculate the storage based on the requirements of the project. In the sizing, we only use the Online data, the offline backup is not considered (since it is a external independent server).
To make an easy explanation, lets use the following requirements:
[Online Live Data: 7 days; Online Compacted: 180 days; EPS: 2500]
Steps to calculate:
- Calculate how much data is generated each second: Multiply the EPS by 300 bytes (the average size of an log):
In the example: 2500 x 300 = 750000 bytes = 732.5 kb/s
- With the Data Per Second, we can calculate how much data we have in one day (1 day = 86400 seconds):
In the example: 732.5 * 86400 = 63288000 kb/day = 61804.7 Mb/day = 60.4 Gb/day
- Now that we know how much data is generated in one day, lets calculate the Online Live Data size (non-compacted):
In the example: 60.4Gb/day * 7 = 422.8Gb
- Now, lets calculate the Online Compacted Data. Note that the average compression rate is 10:1 :
In the example: 180 days – 7 days (online live data) = 173 days
173 days * 60.4Gb = 10449.2 Gb
10449.2Gb * 0,1 (compression rate) = 1044.92Gb
- We have the size of the online live and the online compacted data. Now we just need to sum both and we have the final size:
In the example: 422.8Gb + 1044.92Gb = 1467.72Gb = 1.43Tb
Following this basic steps we can have a accurate approximation of the necessary storage size. A good practice is using a storage 20% bigger than the estimated.
Do you have any another experience with storage sizing? Let us know in the comments!
UPDATE: According to one of our readers (see comments), starting from the version 7.2.7, the stored data will always be compressed. So, if you are sizing your environment for the latest QRadar version, you should use only the “compressed data” calculations.
One of the biggest challenges when sizing a QRadar implementation is estimating the Events Per Second (aka. EPS) of the environment, specially because in the most of the cases we don’t have full access to the log sources to precisely determine the EPS. So in this post we will review some tips about how to estimate the EPS.
Determining the EPS of one event source with access to the system or access to the logfiles.
# Dump the log in a file and delete all the log not from the past 24h. Leave only the last 24h of logs
– If the system generate syslog, follow these steps:
a. Configure the logsource to send the logs to any linux server
b. In the destination linux server execute the following command: tcpdump -i eth0 src host SOURCE_IP dst port 514
c. Run the command for exactly 24 hours in a regular day and verify how many log packets you got.
# Verify the number of logs in the file.
– If there is just one log per line, simple open the file on notepad and verify how many lines you have;
– If the logs are not one per line, verify the whole size of the file (in bytes) and divide by 250 (the avarage size of a log line). Example: File with 3Mb = 3145728 bytes / 250 = 12583 Log packets
# Divide the number of packets by 86400, the result will be the EPS of the log source
Determining the EPS without access to logs or the system:
# From my previous experience, a good approximation of EPS is:
|IIS or Exchange||10|
|General Windows Server||2|
|General Windows Workstation||0,5|
|DNS or DHCP||15|
|IPS, IDS or DAM||5|
Calculating the EPS of the whole environment:
# Multiply the number of each device by the estimated EPS
# Sum the EPS of all kind of devices and you will have the EPS of your whole environment
3 Core Routers + 2 IPS = 3x 150 + 2x 5 = 460 EPS
# Remember to always consider at least 20% margin for buying your license.
Do you have any another tips to calculate EPS? Let us know in the comments!
The Gartner Group published in June of this year the result of the Magic Quadrand for SIEM solutions. For the folks that don’t know what is a magic quadrant, it is a chart comparison between all the SIEM solutions in therms of “ability to execute” and “completeness of vision”, dividing the competitors in 4 categories: Niche Players, Challengers, Visionaries and Leaders.
Since IBM aquired the Q1Labs, every year the QRadar is classified as Leader, and in 2013 it was not different. The following image is the Gatner Magic Quadrant for SIEM solutions of 2013:
Just to remember, this chart evaluate only the SIEM solution, not considering the several another features from QRadar (Risk Management, Vulnerability Assessment/Management, Network Analysis, etc).
To read more about the results of 2013, read this article in the Security Intelligence blog.
What is your opinion about the QRadar facing the competitors? Leave a comment!