In the last post we discussed how to calculate the EPS of our environment. Now lets discuss how to calculate the required size of the storage, since with the EPS in hands it turns way easier to calculate the size of our database. In this scenario we will consider only the log storage, not considering the network flows storage.
First of all, we need to understand how the data is stored on QRadar. Basically, you have 3 types of data:
- Online live data: All the events can be accessed with no latency. In this case the data is not compacted;
- Online compacted data: All the events can be accessed but with a small latency because the data is compacted. The avarage compression rate is 10:1;
- Offline data: All the events cannot be accessed instantly because all the data is in a external backup server. To access this data the user should import the backup into the QRadar (or into a QRadar Virtual Machine) for analysis;
After understanding which each type of data represents, we can start to calculate the storage based on the requirements of the project. In the sizing, we only use the Online data, the offline backup is not considered (since it is a external independent server).
To make an easy explanation, lets use the following requirements:
[Online Live Data: 7 days; Online Compacted: 180 days; EPS: 2500]
Steps to calculate:
- Calculate how much data is generated each second: Multiply the EPS by 300 bytes (the average size of an log):
In the example: 2500 x 300 = 750000 bytes = 732.5 kb/s
- With the Data Per Second, we can calculate how much data we have in one day (1 day = 86400 seconds):
In the example: 732.5 * 86400 = 63288000 kb/day = 61804.7 Mb/day = 60.4 Gb/day
- Now that we know how much data is generated in one day, lets calculate the Online Live Data size (non-compacted):
In the example: 60.4Gb/day * 7 = 422.8Gb
- Now, lets calculate the Online Compacted Data. Note that the average compression rate is 10:1 :
In the example: 180 days – 7 days (online live data) = 173 days
173 days * 60.4Gb = 10449.2 Gb
10449.2Gb * 0,1 (compression rate) = 1044.92Gb
- We have the size of the online live and the online compacted data. Now we just need to sum both and we have the final size:
In the example: 422.8Gb + 1044.92Gb = 1467.72Gb = 1.43Tb
Following this basic steps we can have a accurate approximation of the necessary storage size. A good practice is using a storage 20% bigger than the estimated.
Do you have any another experience with storage sizing? Let us know in the comments!
UPDATE: According to one of our readers (see comments), starting from the version 7.2.7, the stored data will always be compressed. So, if you are sizing your environment for the latest QRadar version, you should use only the “compressed data” calculations.
One of the biggest challenges when sizing a QRadar implementation is estimating the Events Per Second (aka. EPS) of the environment, specially because in the most of the cases we don’t have full access to the log sources to precisely determine the EPS. So in this post we will review some tips about how to estimate the EPS.
Determining the EPS of one event source with access to the system or access to the logfiles.
# Dump the log in a file and delete all the log not from the past 24h. Leave only the last 24h of logs
– If the system generate syslog, follow these steps:
a. Configure the logsource to send the logs to any linux server
b. In the destination linux server execute the following command: tcpdump -i eth0 src host SOURCE_IP dst port 514
c. Run the command for exactly 24 hours in a regular day and verify how many log packets you got.
# Verify the number of logs in the file.
– If there is just one log per line, simple open the file on notepad and verify how many lines you have;
– If the logs are not one per line, verify the whole size of the file (in bytes) and divide by 250 (the avarage size of a log line). Example: File with 3Mb = 3145728 bytes / 250 = 12583 Log packets
# Divide the number of packets by 86400, the result will be the EPS of the log source
Determining the EPS without access to logs or the system:
# From my previous experience, a good approximation of EPS is:
|IIS or Exchange||10|
|General Windows Server||2|
|General Windows Workstation||0,5|
|DNS or DHCP||15|
|IPS, IDS or DAM||5|
Calculating the EPS of the whole environment:
# Multiply the number of each device by the estimated EPS
# Sum the EPS of all kind of devices and you will have the EPS of your whole environment
3 Core Routers + 2 IPS = 3x 150 + 2x 5 = 460 EPS
# Remember to always consider at least 20% margin for buying your license.
Do you have any another tips to calculate EPS? Let us know in the comments!
The Gartner Group published in June of this year the result of the Magic Quadrand for SIEM solutions. For the folks that don’t know what is a magic quadrant, it is a chart comparison between all the SIEM solutions in therms of “ability to execute” and “completeness of vision”, dividing the competitors in 4 categories: Niche Players, Challengers, Visionaries and Leaders.
Since IBM aquired the Q1Labs, every year the QRadar is classified as Leader, and in 2013 it was not different. The following image is the Gatner Magic Quadrant for SIEM solutions of 2013:
Just to remember, this chart evaluate only the SIEM solution, not considering the several another features from QRadar (Risk Management, Vulnerability Assessment/Management, Network Analysis, etc).
To read more about the results of 2013, read this article in the Security Intelligence blog.
What is your opinion about the QRadar facing the competitors? Leave a comment!
A couple of months ago one friend ask me how to migrate from the IBM TSIEM solution to the QRadar.
Pause: For the folks that don’t know what is the Tivoli Security Information and Event Management (aka. TSIEM), it was the old SIEM solution from IBM that was discontinued in 2011, when IBM acquired the Q1Labs (and the QRadar). Who only worked with TSIEM will be impressed about how simple QRadar is compared to the TSIEM.
After some research, I found this documentation from IBM explaining thoroughly how to migrate from the old TSIEM solution to the QRadar. The documentation is based on the version 7.0 of the QRadar, but it can be easily used as base for the migration for any QRadar version (including the new 7.2).
Finding the official documentation sometimes is a painful task. In this post you can find the IBM official product documentation for all the recent QRadar versions.
Current IBM QRadar 7.2.1 Documentation:
IBM QRadar 7.2.1 SIEM: All the documents related with the SIEM solution, including administration guide, user guide, etc.
IBM QRadar 7.2.1 Vulnerability Manager: All the documentation related with the new Vulnerability Manager feature.
IBM QRadar 7.2.1 Risk Manager: Documentation regarding with the Risk Manager feature, part of the QRadar framework.
IBM QRadar 7.1 MR2 SIEM: All the documents related with the SIEM solution version 7.1.
IBM QRadar 7.0 SIEM: All the documents related with the SIEM solution version 7.1.
For more QRadar documentation, please visit the IBM QRadar Documentation Centre.
Found some broken link? Couldn’t find the documentation that you are looking for? Contact us!