Troubleshooting
Proactively identifying performance issues with the HCF plugin
In the last post, we talked about the Health Check Framework (HCF) and its benefits. Since I’ve been using the plugin for over a month I was able to collect useful performance information and identify some potential performance issues even before they occur. In this post, you will learn how to proactively monitor your system performance and prevent potential performance issues from happening.
First, you will need to install the Health Check Framework plugin. The installation process is quite straightforward: all you need is to go to your IBM app store on your QRadar environment, search for “Health Check Framework” and install it following the steps on the screen. With the plugin installed, you can start by browsing the plugin interface and extracting a report about your system performance. In this report, you will see a lot of details about your system, such as CPU Usage, Disk Usage, EPS, FPM, heavy reports, current users, etc.
In my case, we are planning to expand the scope of servers monitored by QRadar, so I wanted to understand if we would need any hardware upgrades. For testing purposes, I disabled the log collection of 30 Windows servers that were currently being monitored and I noticed that the RAM memory usage reduced by around 5% (see images below). Obviously, this number will vary according to the server usage, but this test gave me a rough estimation that for each 30 servers my RAM memory usage will increase around 5%. So with the help of the HCF plugin, I was able to identify hardware upgrades to accommodate the monitoring of new servers, avoiding system outages due to lack of resources during the scope expansion.
Even if you’re not planning to expand your scope, you can use historical performance data to proactively identify issues. For example, let’s say, your QRadar monitors a new e-commerce website. The number of logs you get depends on the traffic your website has. With the historical data, you will be able to identify a performance trend: as the e-commerce website becomes more popular, the EPS increases and the CPU/Memory usage also increases. With this data, you will be able to estimate at which point in time you will need a hardware upgrade, avoiding any unexpected system outages due to lack of resources.
Another very interesting data that I found in this report was the “Event Average Payload Size”, which as the name says, tells you the average size of logs received. This can be very useful to identify hard drive requirements when expanding your EPS.
Using the same plugin I was also able to identify heavy reports and rules that were severely impacting the performance of my QRadar environment. After reviewing and fixing the queries of the reports and rules, it was noticed a considerable reduction in the CPU usage.
Monitoring the performance of your system puts you in a proactive posture in relation to your environment. Being proactive means that you will not be firefighting issues as before, but instead, monitoring and planning upgrades ahead to avoid issues even before they happen.
QRadar Apps: Health Check Framework
One of the most interesting features introduced on QRadar 7.2.6 is the AppExchange, which allow you to install plugins (or also called, QRadar Apps) within just few clicks. Last week I came across a very interesting app called Health Check Framework (HCF) that allows you to perform health checks on your QRadar platform. What I found interesting is that the plugin brings you information that you would need to spend hours trying to find in the complicated QRadar log files or that you would need to be manually running scripts on the server. The plugin also creates for you real-time dashboards showing information around data compression, index file usage, EPS usage, and others.
The plugin, which can be easily installed through the IBM AppExchange, is developed by a company called Science Soft, you can check their app details in their website, but here’s the main features that caught my attention:
- HCF provides a 360-degree view of all essential characteristics of QRadar operation. It indicates deviations, which allows security officers to take urgent steps to fix them.
- QRadar health check can be both scheduled and started manually on demand, and its results are provided as a report.
- HCF assesses QRadar’s state with 60+ operational metrics that are configured into 25 health markers showing either ‘OK’ or ‘Failed’ and reported in an email to HCF subscribers.
- The reports describe how well the security system components are connected to QRadar and if there are security events that are not classified.
- Ability to create a system-health baseline using the called “health markers”, which are are like a snapshot of QRadar, allowing you to compare the health status over a period of time.
- ‘Failed’ markers are followed by recommendations on further actions.
The HCF app has been helping me to troubleshoot performance issues and helped me to proactively identify some issues (for example, that my server was using almost all of its RAM memory in a constant trend). The app also reduced the amount of time I spend analyzing the QRadar log files, so it may be useful for you to keep control of your SIEM environment and increase the ROI of your SIEM investment.
If you would like to know more about this plugin, check out the developer’s website or the IBM QRadar AppExchange! If you have used this app, tell us on the comments your experience!
Checking if GUI is working on the IMM
Hi guys, sometimes you’re not able to log in to the IMM web interface and check many information regarding event processor/ collector. If the GUI is down you can go through the following easy steps:
- ssh to the IMM,
- check if the server is running: system> ssl
- if it’s running try to reset the IMM.
- if the server is not running just turn it on: system> ssl se on
Easy way to copy a file to all managed hosts
Hi guys, today I’d like to present you another very useful command which you can use on a daily basics. Sometimes there is a need of coping a file to your all managed hosts. There is a very easy way to do this by using:
/opt/qradar/support/all_servers.sh -p file
… this will copy the “file” to the /tmp directory of all appliances.
After the file transfer, you can use the tips on this post to run commands all across your environment regarding with the new files.
One example of using this command, is transferring your security policies to all the environment and after deploying the configuration using the all_servers script.
Do you have any use cases for this feature? Drop us a line in the comments!
Who added a new account ?
Sometimes is necessary to audit the configurations of the QRadar and find the people involved on the changes in the system. Those changes can be verified inside the “events” tab of QRadar (and filtering by the events from the QRadar device). Another quick way to find the audit information about the QRadar is checking it own logs. For example, to check the history of accounts added on the system, there is a quick command that you can execute to check who and when added new account. To get this information just execute the following command :
[root@MY_RADAR]# cat /var/log/audit/audit.log | grep ‘AccountAdded’ | less
You will get log lines such as the example below :
Jun 12 14:34:37 127.0.0.1 X&Y (7638) /console/JSON-RPC/QRadar.saveUser QRadar.saveUser | [Configuration] [UserAccount] [AccountAdded] ID: 24 | Username: ABC | Email: ABC@DEF | Description: | Role ID: 2 | Security Profile ID: 1
By analyzing the log line, we can verify that X&Y added new account ABC with the e-mail address of ABC@DEF at Jun the 12th at 14:43.
QRadar Insights 