Ingest historical data into your target platform

In previous articles, you selected a target platform for your historical data. You also selected a data transfer tool for migration ingestion and stored the historical data in a staging location. You can now start to ingest the data into the target platform.

This article describes how to ingest your historical data into your selected target platform.

Export data from the legacy SIEM

In general, SIEMs can export or dump data to a file in your local file system, so you can use this method to extract the historical data. It’s also important to set up a staging location for your exported files. The tool you use to transfer the data ingestion can copy the files from the staging location to the target platform. For information about the tools you can use for data ingestion, see Select a target Microsoft platform to host the exported historical data.

This diagram shows the high-level export and ingestion process.

Diagram illustrating steps involved in export and ingestion.

To export data from your current SIEM, see one of the following sections:

Ingest data to Microsoft Sentinel data lake

Microsoft Sentinel data lake is the native data layer of the Microsoft Sentinel platform. It's the simplest path to a unified, queryable history of your security data inside Microsoft Sentinel and the recommended platform for long-term data retention.

To ingest your historical data into Microsoft Sentinel data lake (option 1 in the diagram above) use the Custom Log Ingestion tool.

Ingest to Azure Data Explorer

To ingest your historical data into Azure Data Explorer (ADX) (option 2 in the diagram above):

  1. Install and configure LightIngest on the system where logs are exported, or install LightIngest on another system that has access to the exported logs. LightIngest supports Windows only.
  2. If you don't have an existing ADX cluster, create a new cluster and copy the connection string. Learn how to set up ADX.
  3. In ADX, create tables and define a schema for the CSV or JSON format (for QRadar). Learn how to create a table and define a schema with sample data or without sample data.
  4. Run LightIngest with the folder path that includes the exported logs as the path, and the ADX connection string as the output. When you run LightIngest, ensure that you provide the target ADX table name, that the argument pattern is set to *.csv, and the format is set to .csv (or json for QRadar).

Ingest to Azure Blob Storage

To ingest your historical data into Azure Blob Storage (option 3 in the diagram above):

  1. Install and configure AzCopy on the system to which you exported the logs. Alternatively, install AzCopy on another system that has access to the exported logs.
  2. Create an Azure Blob Storage account and copy the authorized Microsoft Entra ID credentials or Shared Access Signature token.
  3. Run AzCopy with the folder path that includes the exported logs as the source, and the Azure Blob Storage connection string as the output.

Next step

Now that you ingested historical data into your target platform, convert your dashboards to workbooks.

Convert your dashboards to workbooks

  • Last updated on