How To
Summary
This article describes steps for copying event and flow data from between QRadar hosts. The most common method for users to copy data between appliances is when they are migrating to new hardware. This article describes how to move data with the support tool syncAriel.sh or manually moving data with rsync, then regenerate indexes on the new appliance.
Environment
Steps
Method A: Use the syncAriel.sh utility (recommended)
- Ensure you restored your latest nightly configuration backup from your old appliance to your new appliance before you attempt to migrate any data on disk. For more information, see Migration procedures.
- Download the syncAriel.sh utility.
1. Set up iptables rules on the destination
- Use SSH to log in to the QRadar Console as the root user.
- Open an SSH session to the managed host that is the destination for the event and flow data (new appliance).
- On the destination host, type:
vi /opt/qradar/conf/iptables.pre
- Add the following iptables rule and save your changes:
-A INPUT -p tcp --dport 22 -j ACCEPT --src <IP Address (data source)>
- Save your changes to the file.
- Run the iptables update script to commit the change.
/opt/qradar/bin/iptables_update.pl
- To list iptables rules, type:
iptables -L -v -n
- Review the INPUT chain to confirm the new rule is listed for port 22 to the source host.
- You can now test connectivity from the source host CLI by running:
ssh <destination IP>
For more information, see How to edit iptables rules in QRadar.
Results
You are now ready to set up a nonpassword authentication key (public key) as described in the next step. As multiple SSH sessions are opened, the nonpassword authentication procedure allows the migration to continue without prompting for the root password during the data migration.
2. Setup nonpassword authentication (public key)
- To copy the SSH ID, type:
ssh-copy-id <Destination IP>
- Type the root password for the destination.
This command updates the authorized_keys file of the destination to match the id_rsa.pub key from the source. - Test the connection to ensure no password is needed when you SSH from the source host CLI to the destination IP.
- If the destination prompts for a password, see Step 4. In recent QRadar versions the SSH configuration is more strict. Administrators might need to generate n rsa-sha2-512 public key before you run the ssh-copy-id command.
- If you successfully connect without a password prompt, see Using syncAriel.sh to copy data to a destination.
- If your initial attempt to run the ssh-copy-id command failed to allow nonpassword authentication, type:
ssh-keygen -t rsa -b 2048 -E sha512
- When prompted, overwrite the existing id_rsa key. For example,
Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): /root/.ssh/id_rsa already exists. Overwrite (y/n)? y Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub.
- Repeat this procedure to copy the SSH ID and verify whether you can successfully SSH without a password prompt as described in step 3b.
- Copy the syncAriel.sh script to the source host's "/root" directory
- The set permissions on the utility, type:
chmod +x syncAriel.sh
- To start a screen session, type:
screen -S syncariel
Note: If your session is interrupted, SSH to the source host. You can reconnect to the screen session with the command screen -r syncariel. - To sync data between QRadar hosts, type:
sh syncAriel.sh -i <destination IP>
- Wait for the transfer to complete. Depending on the amount of data or network bandwidth limitations, the transfer might take hours or days to complete. The command prompt returns when the data migration is complete.
Note: If you experience 'No such file or directory' errors while rsync runs, the error message is caused when files and folders are no longer present. The most likely cause of this error is when data is removed by event or flow retention before the rsync was able to move the data. You can ignore these error messages. - After your rsync is complete, type the following command to close the screen session:
exit
- Open an SSH session to the destination host.
- Remove the iptables rule for the legacy server:
-A INPUT -p tcp --dport 22 -j ACCEPT --src <IP Address (data source)>
- Save your changes to the file.
- To update iptables and commit your change, type:
/opt/qradar/bin/iptables_update.pl
- To confirm the iptables rules for your old appliance is removed, type:
iptables -L -v -n
Results
After the data is migrated to the new appliance, you must reindex the data to ensure searches perform as expected.
Method B: Use rsync to manually copy data (alternate)
- Ensure you restored your latest nightly configuration backup from you old appliance to your new appliance before you attempt to migrate any data on disk. For more information, see Migration procedures.
- If the copy process is interrupted rsync does not track what files were moved. The entire process must then be restarted.
- You must ensure that your rsync command is correct or your data could end up in the wrong directory.
- Use SSH to log in to the QRadar Console as the root user.
- Open an SSH session to the managed host that is the destination for the event and flow data (new appliance):
- On the destination host, type:
vi /opt/qradar/conf/iptables.pre
- Add the following iptables rule and save your changes:
-A INPUT -p tcp --dport 22 -j ACCEPT --src <IP Address (data source)>
- To update iptables and commit your change, type:
/opt/qradar/bin/iptables_update.pl
- To confirm the iptables rules for your old appliance is removed, type:
iptables -L -v -n
- Review the INPUT chain to confirm the new rule is listed for port 22 for the source host's IP address.
- You can now test connectivity from the source host CLI by running:
ssh <destination IP>
For more information, see How to edit iptables rules in QRadar.
- Use SSH to log in to the QRadar Console as the root user.
- Open an SSH session to the legacy host that has the data to migrate.
- To start a screen session, type:
screen -S rsync
Note: If your session is interrupted, SSH to the source host. You can reconnect to the screen session with the command screen -r rsync. - Use rsync to move the required files to the new appliance. For example,
- Use rsync to copy all data (not recommended, see syncAriel.sh)
If administrators want to use rsync to copy all data from one appliance to another, type the following command:
rsync -avz /store/ariel/ root@<destination IP>:/store/ariel
- How to copy a specific event or flow directory with rsync
To rsync a specific directory in /store/ariel, administrators must copy both payloads and records directories. For example, to copy events and flows from Aug 2021 to a new appliance, type the following commands:
rsync -avz /store/ariel/events/payloads/2021/8/ root@<destination IP>
:/store/ariel/events/payloads/2021/8 rsync -avz /store/ariel/events/records/2021/8/ root@
<destination IP> :/store/ariel/events/records/2021/8
rsync -avz /store/ariel/flows/records/2021/8/ root@
<destination IP> :/store/ariel/flows/records/2021/8 - Moving certificates
Appliances that collect events need to have their trusted certificates migrated to the new appliance. If this directory is not copied, then log sources on the appliance that attempt to connect to a remote host can fail to handshake or collect data.
rsync -avz /opt/qradar/conf/trusted_certificates/ root@<destination IP>:/opt/qradar/conf/trusted_certificates
- Crossover cables and rsync
If crossover cables are configured between the two appliances, use rsync -av instead of rsync -avz.
rsync -av /store/ariel/ root@<destination IP>:/store/ariel
- Use rsync to copy all data (not recommended, see syncAriel.sh)
- After your rsync is complete, type the following command to close the screen session:
exit
- Open an SSH session to the destination host.
- Remove the iptables rule for the legacy server:
-A INPUT -p tcp --dport 22 -j ACCEPT --src <IP Address (data source)>
- Save your changes to the file.
- To update iptables to commit the change, type:
/opt/qradar/bin/iptables_update.pl
- To confirm the iptables rules for your old appliance is removed, type:
iptables -L -v -n
Results
After the data is migrated to the new appliance, you might need to reindex the data to ensure searches perform as expected.
Post-migration: Reindexing your data
Each QRadar appliance that stores event or flow data creates local index files on the appliance to improve search speed. When you move /store/ariel data manually between appliances, reindexing is necessary to ensure old indexes are removed and updated. Indexes allow QRadar running on the host to determine where on disk the data resides so results return quickly. When indexes are not available, a direct scan of the raw data is performed, which can create unnecessary disk (I/O) and CPU load and degrade search speed.
Reindexing your data is required in the following scenarios:
- If data migrated for a timeframe that already has data on the destination host.
- If data migrated from multiple hosts to a single host where the data has an overlapping time frame.
NOTE: Depending on the amount of data on the host, reindexing data might take a considerable amount of time. It is recommended to use the "screen" command as noted in previous steps to avoid interruptions related to network issues.
The time period in the example procedure updates the last two days of data to ensure recent searches are quick for recently migrated data. Administrators must update the date and time in the example commands to ensure the offline indexer utility re-creates indexes.
- Use SSH to log in to the QRadar Console as the root user.
- Open an SSH session to the host that contains your copied data.
- To start a screen session, type:
screen -S indexer
- Remove any 1-minute indexes.
/opt/qradar/bin/ariel_offline_indexer.sh -n events -v -t '2021/10/25 00:00' -d 2880 -a -r
- To rebuild 1-minute indexes for your events, type:
/opt/qradar/bin/ariel_offline_indexer.sh -n events -v -t '2021/10/25 00:00' -d 2880 -a
- Remove the super indexes for the same period:
/opt/qradar/bin/ariel_offline_indexer.sh -n events -v -t '2021/10/25 00:00' -d 2880 -s -r
- To rebuild your super indexes, type:
/opt/qradar/bin/ariel_offline_indexer.sh -n events -v -t '2021/08/09 00:00' -d 2880 -s
After the indexes are rebuilt, the data migration is complete.
Related Information
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
21 September 2023
UID
ibm16488441