Collect your Apache Spark applications logs and metrics using Azure Storage account

This article describes the Azure Blob Storage destination for Fabric Apache Spark Diagnostic Emitter.

Fabric Apache Spark Diagnostic Emitter provides a common configuration model for Spark diagnostics across destinations. In this tutorial, you configure that model to route Apache Spark logs, event logs, and metrics to Azure Blob Storage.

For emitter architecture and destination selection guidance, see Fabric Apache Spark Diagnostic Emitter overview.

Collect logs and metrics to storage account

Step 1: Create a storage account

To collect diagnostic logs and metrics, you can use an existing Azure Storage account. If you don't have one, you can create an Azure blob storage account or create a storage account to use with Azure Data Lake Storage Gen2.

Step 2: Create a Fabric Environment Item with Apache Spark Configuration

Option 1: Configure with Azure Storage URI and Access key

1. Create an environment item in Fabric

2. Add the following Spark properties with the appropriate values to the environment item, or select Add from .yml in the ribbon to download the sample yaml file, which already containing the following properties.

   spark.synapse.diagnostic.emitters: MyStorageBlob
   spark.synapse.diagnostic.emitter.MyStorageBlob.type: "AzureStorage"
   spark.synapse.diagnostic.emitter.MyStorageBlob.categories: "DriverLog,ExecutorLog,EventLog,Metrics"
   spark.synapse.diagnostic.emitter.MyStorageBlob.uri:  "https://<my-blob-storage>.blob.core.windows.net/<container-name>/<folder-name>"
   spark.synapse.diagnostic.emitter.MyStorageBlob.auth: "AccessKey"
   spark.synapse.diagnostic.emitter.MyStorageBlob.secret: <storage-access-key>
   spark.fabric.pools.skipStarterPools: "true" //Add this Spark property when using the default pool.
   

   Fill in the following parameters in the configuration file: <my-blob-storage>, <container-name>, <folder-name>, <storage-access-key>. For more details on these parameters, see Azure Storage configurations.

Option 2: Configure with Azure Key Vault

To configure Azure Key Vault for storing the workspace key:

1. Create and go to your key vault in the Azure portal.

2. On the settings page for the key vault, select Secrets, then Generate/Import.

3. On the Create a secret screen, choose the following values:

   • Name: Enter a name for the secret.
   • Value: Enter the <storage-access-key> for the secret.
   • Leave the other values to their defaults. Then select Create.

4. Create an environment item in Fabric.

5. Add the following Spark properties. Or select Add from .yml on the ribbon to upload the sample yaml file which includes following Spark properties.

   spark.synapse.diagnostic.emitters: <MyStorageBlob>
   spark.synapse.diagnostic.emitter.MyStorageBlob.type: "AzureStorage"
   spark.synapse.diagnostic.emitter.MyStorageBlob.categories: "DriverLog,ExecutorLog,EventLog,Metrics"
   spark.synapse.diagnostic.emitter.MyStorageBlob.uri:  "https://<my-blob-storage>.blob.core.windows.net/<container-name>/<folder-name>"
   spark.synapse.diagnostic.emitter.MyStorageBlob.auth: "AccessKey"
   spark.synapse.diagnostic.emitter.MyStorageBlob.secret.keyVault: <AZURE_KEY_VAULT_URI>
   spark.synapse.diagnostic.emitter.MyStorageBlob.secret.keyVault.secretName: <AZURE_KEY_VAULT_SECRET_KEY_NAME>
   spark.fabric.pools.skipStarterPools: "true" //Add this Spark property when using the default pool.
   

   Fill in the following parameters in the configuration file: <my-blob-storage>, <container-name>, <folder-name>, <AZURE_KEY_VAULT_URI>, <AZURE_KEY_VAULT_SECRET_KEY_NAME>. For more details on these parameters, see Azure Storage configurations.

6. Save and publish changes.

Option 3: Configure with service principal certificate authentication

Use this option to authenticate to Azure Storage with a Microsoft Entra service principal and a certificate stored in Azure Key Vault.

Before configuring the Spark properties:

• Create or import a certificate in Azure Key Vault. The certificate must contain an exportable private key.
• Download only the public certificate in CER or PEM format, and upload it to the Microsoft Entra app registration under Certificates & secrets > Certificates.
• Assign the Storage Blob Data Contributor role to the service principal on the target storage account or container.
• Assign the Key Vault Certificate User role on the Azure Key Vault to the signed-in Fabric user who starts the Spark session.

Add the following Spark properties to the Fabric environment:

spark.synapse.diagnostic.emitters: MyStorageBlob 
spark.synapse.diagnostic.emitter.MyStorageBlob.type: "AzureStorage" 
spark.synapse.diagnostic.emitter.MyStorageBlob.categories: "DriverLog,ExecutorLog,EventLog,Metrics" 
spark.synapse.diagnostic.emitter.MyStorageBlob.uri: "https://<STORAGE_ACCOUNT>.blob.core.windows.net/<CONTAINER>/<FOLDER>" 
spark.synapse.diagnostic.emitter.MyStorageBlob.auth: "ServicePrincipalCert" 
spark.synapse.diagnostic.emitter.MyStorageBlob.certificate.keyVault.certificateName: "<CERTIFICATE_NAME>" 
spark.synapse.diagnostic.emitter.MyStorageBlob.certificate.keyVault: "https://<KEY_VAULT_NAME>.vault.azure.net/" 
spark.synapse.diagnostic.emitter.MyStorageBlob.tenantId: "<SERVICE_PRINCIPAL_TENANT_ID>" 
spark.synapse.diagnostic.emitter.MyStorageBlob.clientId: "<SERVICE_PRINCIPAL_CLIENT_ID>" 
spark.fabric.pools.skipStarterPools: "true" 

The uri property identifies the destination Blob Storage container and optional folder. The certificate name must exactly match the certificate name in Azure Key Vault.

Step 3: Attach the environment item to notebooks or spark job definitions, or set it as the workspace default

To attach the environment to Notebooks or Spark job definitions:

1. Navigate to the specific notebook or Spark job definition in Fabric.
2. Select the Environment menu on the Home tab and select the environment with the configured diagnostics Spark properties.
3. The configuration is applied when you start a Spark session.

To set the environment as the workspace default:

1. Navigate to workspace settings in Fabric.
2. Find Spark settings in workspace settings (Workspace setting > Data Engineering/Science > Spark settings).
3. Select Environment tab and choose the environment with diagnostics spark properties configured, and click Save.

Step 4. Submit an Apache Spark application and view the logs and metrics

You can use the Apache Log4j library to write custom logs.

Example for Scala:

	   %%spark
	   val logger = org.apache.log4j.LogManager.getLogger("com.contoso.LoggerExample")
	   logger.info("info message")
	   logger.warn("warn message")
	   logger.error("error message")
	   //log exception
	   try {
	         1/0
	   } catch {
	         case e:Exception =>logger.warn("Exception", e)
	   }
	   // run job for task level metrics
	   val data = sc.parallelize(Seq(1,2,3,4)).toDF().count()

Example for PySpark:

	   %%pyspark
	   logger = sc._jvm.org.apache.log4j.LogManager.getLogger("com.contoso.PythonLoggerExample")
	   logger.info("info message")
	   logger.warn("warn message")
	   logger.error("error message")

Step 5: View the logs files in Azure storage account

After submitting a job to the configured Spark session, you can view the logs and metrics files in the destination storage account. The logs are stored in corresponding paths based on different applications, identified by <workspaceId>.<fabricLivyId>. All log files are in JSON Lines format (also known as newline-delimited JSON or ndjson), which is convenient for data processing.

Available configurations

Log data sample

Here's a sample log record in JSON format:

{
  "timestamp": "2025-02-28T09:13:57.978Z",
  "category": "Log|EventLog|Metrics",
  "fabricLivyId": "<fabric-livy-id>",
  "applicationId": "<application-id>",
  "applicationName": "<application-name>",
  "executorId": "<driver-or-executor-id>",
  "userId": "<the-submitter-user-id>",
  "fabricTenantId": "<my-fabric-tenant-id>",
  "capacityId": "<my-fabric-capacity-id>",
  "artifactType": "SynapseNotebook|SparkJobDefinition",
  "artifactId": "<my-fabric-item-id>",
  "fabricWorkspaceId": "<my-fabric-workspace-id>",
  "fabricEnvId": "<my-fabric-environment-id>",
  "executorMin": "<executor-min>",
  "executorMax": "<executor-max>",
  "isHighConcurrencyEnabled": "true|false",
  "properties": {
    // The message properties of logs, events and metrics.
    "timestamp": "2025-02-28T09:13:57.941Z",
    "message": "ApplicationAttemptId: appattempt_1740734011890_0001_000001",
    "logger_name": "org.apache.spark.deploy.yarn.ApplicationMaster",
    "level": "INFO",
    "thread_name": "main"
    // ...
  }
}

Fabric workspaces with Managed virtual network

Create a managed private endpoint for the target Azure Blob Storage. For detailed instructions, refer to Create and use managed private endpoints in Fabric.

Once the managed private endpoint is approved, users can begin emitting logs and metrics to the target Azure Blob Storage.

Next steps

• Create Apache spark job definition
• Create, configure, and use an environment in Fabric
• Create and use managed private endpoints in Fabric
• Develop, execute, and manage Fabric notebooks
• Monitor Spark Applications