Troubleshooting

Proactively identifying performance issues with the HCF plugin

Posted on Updated on

In the last post, we talked about the Health Check Framework (HCF) and its benefits. Since I’ve been using the plugin for over a month I was able to collect useful performance information and identify some potential performance issues even before they occur. In this post, you will learn how to proactively monitor your system performance and prevent potential performance issues from happening.

First, you will need to install the Health Check Framework plugin. The installation process is quite straightforward: all you need is to go to your IBM app store on your QRadar environment, search for “Health Check Framework” and install it following the steps on the screen. With the plugin installed, you can start by browsing the plugin interface and extracting a report about your system performance. In this report, you will see a lot of details about your system, such as CPU Usage, Disk Usage, EPS, FPM, heavy reports, current users, etc.

Report generated by the Health Check Framework (HFC) – A holistic view of your environment, note the number of tabs on the report

 

In my case, we are planning to expand the scope of servers monitored by QRadar, so I wanted to understand if we would need any hardware upgrades. For testing purposes, I disabled the log collection of 30 Windows servers that were currently being monitored and I noticed that the RAM memory usage reduced by around 5% (see images below). Obviously, this number will vary according to the server usage, but this test gave me a rough estimation that for each 30 servers my RAM memory usage will increase around 5%. So with the help of the HCF plugin, I was able to identify hardware upgrades to accommodate the monitoring of new servers, avoiding system outages due to lack of resources during the scope expansion.

Before disabling Windows test servers – Memory used ~40GB

 

After disabling the Windows servers – Memory used: ~38GB

 

Even if you’re not planning to expand your scope, you can use historical performance data to proactively identify issues. For example, let’s say, your QRadar monitors a new e-commerce website. The number of logs you get depends on the traffic your website has. With the historical data, you will be able to identify a performance trend: as the e-commerce website becomes more popular, the EPS increases and the CPU/Memory usage also increases. With this data, you will be able to estimate at which point in time you will need a hardware upgrade, avoiding any unexpected system outages due to lack of resources.

Another very interesting data that I found in this report was the “Event Average Payload Size”, which as the name says, tells you the average size of logs received. This can be very useful to identify hard drive requirements when expanding your EPS.

Using the same plugin I was also able to identify heavy reports and rules that were severely impacting the performance of my QRadar environment. After reviewing and fixing the queries of the reports and rules, it was noticed a considerable reduction in the CPU usage.

Chart extracted from the HCF report – Identifying heavy rules

 

Monitoring the performance of your system puts you in a proactive posture in relation to your environment. Being proactive means that you will not be firefighting issues as before, but instead, monitoring and planning upgrades ahead to avoid issues even before they happen.

QRadar Apps: Health Check Framework

Posted on

One of the most interesting features introduced on QRadar 7.2.6 is the AppExchange, which allow you to install plugins (or also called, QRadar Apps) within just few clicks. Last week I came across a very interesting app called Health Check Framework (HCF) that allows you to perform health checks on your QRadar platform. What I found interesting is that the plugin brings you information that you would need to spend hours trying to find in the complicated QRadar log files or that you would need to be manually running scripts on the server. The plugin also creates for you real-time dashboards showing information around data compression, index file usage, EPS usage, and others.

dashboard1
QRadar Health Check Framework (HCF) Dashboard

 

The plugin, which can be easily installed through the IBM AppExchange, is developed by a company called Science Soft, you can check their app details in their website, but here’s the main features that caught my attention:

 

  • HCF provides a 360-degree view of all essential characteristics of QRadar operation. It indicates deviations, which allows security officers to take urgent steps to fix them.
  • QRadar health check can be both scheduled and started manually on demand, and its results are provided as a report.
  • HCF assesses QRadar’s state with 60+ operational metrics that are configured into 25 health markers showing either ‘OK’ or ‘Failed’ and reported in an email to HCF subscribers.
  • The reports describe how well the security system components are connected to QRadar and if there are security events that are not classified.
  • Ability to create a system-health baseline using the called “health markers”, which are are like a snapshot of QRadar, allowing you to compare the health status over a period of time.
  • ‘Failed’ markers are followed by recommendations on further actions.

 

dashboard2
Sample of the analytics dashboard on QRadar Health Check Framework

 

The HCF app has been helping me to troubleshoot performance issues and helped me to proactively identify some issues (for example, that my server was using almost all of its RAM memory in a constant trend). The app also reduced the amount of time I spend analyzing the QRadar log files, so it may be useful for you to keep control of your SIEM environment and increase the ROI of your SIEM investment.

If you would like to know more about this plugin, check out the developer’s website or the IBM QRadar AppExchange! If you have used this app, tell us on the comments your experience!

Checking if GUI is working on the IMM

Posted on

Hi guys, sometimes you’re not able to log in to the IMM web interface and check many information regarding event processor/ collector. If the GUI is down you can go through the following easy steps:

  •  ssh to the IMM,
  • check if the server is running:     system>     ssl
    • if it’s running try to reset the IMM.
    • if the server is not running just turn it on:          system>   ssl se on

IMM Troubleshooting

Easy way to copy a file to all managed hosts

Posted on

Hi guys, today I’d like to present you another very useful command which you can use on a daily basics. Sometimes there is a need of coping a file to your all managed hosts. There is a very easy way to do this by using:

/opt/qradar/support/all_servers.sh -p file

… this will copy the “file” to the /tmp directory of all appliances.

After the file transfer, you can use the tips on this post to run commands all across your environment regarding with the new files.

One example of using this command, is transferring your  security policies to all the environment and after deploying the configuration using the all_servers script.

Do you have any use cases for this feature? Drop us a line in the comments!

Who added a new account ?

Posted on Updated on

Sometimes is necessary to audit the configurations of the QRadar and find the people involved on the changes in the system. Those changes can be verified inside the “events” tab of QRadar (and filtering by the events from the QRadar device). Another quick way to find the audit information about the QRadar is checking it own logs. For example, to check the history of accounts added on the system, there is a quick command that you can execute to check who and when added new account. To get this information just execute the following command :

[root@MY_RADAR]# cat /var/log/audit/audit.log | grep ‘AccountAdded’ | less

You will get log lines such as the example below :

Jun 12 14:34:37 127.0.0.1 X&Y (7638) /console/JSON-RPC/QRadar.saveUser QRadar.saveUser | [Configuration] [UserAccount] [AccountAdded] ID: 24 | Username: ABC | Email: ABC@DEF | Description: | Role ID: 2 | Security Profile ID: 1

By analyzing the log line, we can verify that X&Y added new account ABC with the e-mail address of ABC@DEF at Jun the 12th at 14:43.

Changing firewall rules

Posted on Updated on

By the original QRadar configuration, all the appliances comes with a pre-configured firewall rules in the OS. For testing purposes we can simple deactivate the firewall using the command “service iptables stop” (to stop the firewall) and “service iptables start” (to turn it back). But sometimes we need to update the firewall configuration aiming permanent changes.

In order to change firewall rules on your appliance you need to follow the below steps:

  • Connect through SSH to the appliance that you want to make modifications;
  • Login using ‘root’ account;
  • Edit one of the following files:
    • /opt/qradar/conf/iptables.pre
    • /opt/qradar/conf/iptables.post
    • /opt/qradar/conf/iptables-nat.post
  • Add your firewall rules in the file, for example:
    • -A INPUT -i eth0 -s x.x.x.x -j ACCEPT
  • Save the file with the ‘ :wq ‘;
  • Run /opt/qradar/bin/iptables_update.pl so your changes take effect;

With those steps your firewall configuration is now changed and will persist even in rebooting cases.

Running commands across the environment – Part 2

Posted on Updated on

Continuing the post about running commands across the environment, today we’d like to present you another very useful and powerful command. Gathering information about the appliances and servers can be a painful task, but QRadar can provide us with some good scripts to make this task easy and automated. For example, if you execute on your QRadar Console:

 [root@MY_RADAR]# /opt/qradar/bin/myver -v

…you’ll get a lot of  information about you appliance like :

  • Appliance type,
  • Core version of the system,
  • Patch number,
  • Is the QRM enabled,
  • Is the appliance you ran this command is a console,
  • What’s the IP address,
  • What’s the kernel architecture,
  • Information about CPU, Operating System and if this is HA host or not.

And here’s the tricky part: to get this information from all your QRadar servers and appliances, you can combine it with the “/opt/qradar/support/all_servers.sh” command, presented in the another post, and gather this valuable information from all your managed hosts. For example, we can run this command across all the servers and input the result in a text file:

[root@MY_RADAR]# /opt/qradar/support/all_servers.sh “/opt/qradar/bin/myver -v” > /root/info.txt

As you can see, with just one line we can gather information of all our servers and generate a raw report of our QRadar environment. Simple, isn’t it?