Collect Apache Spark applications logs and metrics using Azure Event Hubs

This article describes the Azure Event Hubs destination for Fabric Apache Spark Diagnostic Emitter.

Fabric Apache Spark Diagnostic Emitter provides a common configuration model for Spark diagnostics across destinations. In this article, you configure that model to route Apache Spark logs, event logs, and metrics to Azure Event Hubs.

For emitter architecture and destination selection guidance, see Fabric Apache Spark Diagnostic Emitter overview.

Collect logs and metrics to Azure Event Hubs

Step 1: Create an Azure Event Hubs Instance

To collect diagnostic logs and metrics, you can use an existing Azure Event Hubs instance. If you don't have one, you can create an event hub.

Step 2: Create a Fabric Environment Item with Apache Spark Configuration

Option 1: Configure with Azure Event Hubs Connection String

1. Create an environment item in Fabric

2. Add the following Spark properties with the appropriate values to the environment item, or select Add from .yml in the ribbon to download the sample yaml file which already containing the following properties.

   spark.synapse.diagnostic.emitters: MyEventHub
   spark.synapse.diagnostic.emitter.MyEventHub.type: "AzureEventHub"
   spark.synapse.diagnostic.emitter.MyEventHub.categories: "Log,EventLog,Metrics"
   spark.synapse.diagnostic.emitter.MyEventHub.secret: <connection-string>
   spark.fabric.pools.skipStarterPools: "true" //Add this Spark property when using the default pool.
   

   Fill in the <connection-string> parameters in the configuration file. For more information, see Azure Event Hubs configurations.

Option 2: Configure with Azure Key Vault

To configure Azure Key Vault for storing the workspace key:

1. Create and go to your key vault in the Azure portal.

2. On the settings page for the key vault, select Secrets, then Generate/Import.

3. On the Create a secret screen, choose the following values:

   • Name: Enter a name for the secret.
   • Value: Enter the <connection-string> for the secret.
   • Leave the other values to their defaults. Then select Create.

4. Create an environment item in Fabric.

5. Add the following Spark properties. Or select Add from .yml on the ribbon to download the sample yaml file, which includes following Spark properties.

   spark.synapse.diagnostic.emitters: MyEventHub
   spark.synapse.diagnostic.emitter.MyEventHub.type: "AzureEventHub"
   spark.synapse.diagnostic.emitter.MyEventHub.categories: "Log,EventLog,Metrics"
   spark.synapse.diagnostic.emitter.MyEventHub.secret.keyVault: <AZURE_KEY_VAULT_URI>
   spark.synapse.diagnostic.emitter.MyEventHub.secret.keyVault.secretName: <AZURE_KEY_VAULT_SECRET_KEY_NAME>
   spark.fabric.pools.skipStarterPools: "true" //Add this Spark property when using the default pool.
   

   Fill in the following parameters in the configuration file: <AZURE_KEY_VAULT_URI>, <AZURE_KEY_VAULT_SECRET_KEY_NAME>. For more details on these parameters, refer to Azure Event Hubs configurations.

6. Save and publish changes.

Option 3: Configure with service principal certificate authentication

Use this option to authenticate to Azure Event Hubs with a Microsoft Entra service principal and a certificate stored in Azure Key Vault. For more information, see Create a service principal containing a certificate using Azure CLI.

Before configuring the Spark properties:

• Create or import a certificate in Azure Key Vault. The certificate must contain an exportable private key.
• Download only the public certificate in CER or PEM format, and upload it to the Microsoft Entra app registration under Certificates & secrets > Certificates.
• Record the Azure Key Vault URI and certificate name. You use these values in the Spark properties.
• Assign the Azure Event Hubs Data Sender role to the service principal on the target Event Hubs namespace or Event Hub instance.
• Assign the Key Vault Certificate User role on the Azure Key Vault to the signed-in Fabric user who starts the Spark session.

Add the following Spark properties to the Fabric environment:

spark.synapse.diagnostic.emitters: MyEventHub 
spark.synapse.diagnostic.emitter.MyEventHub.type: "AzureEventHub" 
spark.synapse.diagnostic.emitter.MyEventHub.categories: "DriverLog,ExecutorLog,EventLog,Metrics" 
spark.synapse.diagnostic.emitter.MyEventHub.hostName: "<EVENT_HUB_NAMESPACE>.servicebus.windows.net" 
spark.synapse.diagnostic.emitter.MyEventHub.entityPath: "<EVENT_HUB_NAME>" 
spark.synapse.diagnostic.emitter.MyEventHub.certificate.keyVault.certificateName: "<CERTIFICATE_NAME>" 
spark.synapse.diagnostic.emitter.MyEventHub.certificate.keyVault: "https://<KEY_VAULT_NAME>.vault.azure.net/" 
spark.synapse.diagnostic.emitter.MyEventHub.tenantId: "<SERVICE_PRINCIPAL_TENANT_ID>" 
spark.synapse.diagnostic.emitter.MyEventHub.clientId: "<SERVICE_PRINCIPAL_CLIENT_ID>" 
spark.fabric.pools.skipStarterPools: "true" 

For certificate-based authentication, hostName is the fully qualified domain name of the Event Hubs namespace, without the sb:// prefix. entityPath is the name of the target Event Hub instance. The certificate name must exactly match the certificate name in Azure Key Vault.

Step 3: Attach the Environment Item to Notebooks or Spark Job Definitions, or Set It as the Workspace Default

To attach the environment to Notebooks or Spark job definitions:

1. Navigate to the specific notebook or Spark job definition in Fabric.
2. Select the Environment menu on the Home tab and select the environment with the configured diagnostics Spark properties.
3. The configuration is applied when you start a Spark session.

To set the environment as the workspace default:

1. Navigate to workspace settings in Fabric.
2. Find Spark settings in workspace settings (Workspace setting > Data Engineering/Science > Spark settings).
3. Select Environment tab and choose the environment with diagnostics spark properties configured, and select Save.

Step 4. Submit an Apache Spark application and view the logs and metrics

You can use the Apache Log4j library to write custom logs.

Example for Scala:

	   %%spark
	   val logger = org.apache.log4j.LogManager.getLogger("com.contoso.LoggerExample")
	   logger.info("info message")
	   logger.warn("warn message")
	   logger.error("error message")
	   //log exception
	   try {
	      1/0
	   } catch {
	      case e:Exception =>logger.warn("Exception", e)
	   }
	   // run job for task level metrics
	   val data = sc.parallelize(Seq(1,2,3,4)).toDF().count()

Example for PySpark:

	   %%pyspark
	   logger = sc._jvm.org.apache.log4j.LogManager.getLogger("com.contoso.PythonLoggerExample")
	   logger.info("info message")
	   logger.warn("warn message")
	   logger.error("error message")

Available configurations

Log data sample

Here's a sample log record in JSON format:

{
  "timestamp": "2025-02-28T09:13:57.978Z",
  "category": "Log|EventLog|Metrics",
  "fabricLivyId": "<fabric-livy-id>",
  "applicationId": "<application-id>",
  "applicationName": "<application-name>",
  "executorId": "<driver-or-executor-id>",
  "userId": "<the-submitter-user-id>",
  "fabricTenantId": "<my-fabric-tenant-id>",
  "capacityId": "<my-fabric-capacity-id>",
  "artifactType": "SynapseNotebook|SparkJobDefinition",
  "artifactId": "<my-fabric-item-id>",
  "fabricWorkspaceId": "<my-fabric-workspace-id>",
  "fabricEnvId": "<my-fabric-environment-id>",
  "executorMin": "<executor-min>",
  "executorMax": "<executor-max>",
  "isHighConcurrencyEnabled": "true|false",
  "properties": {
    // The message properties of logs, events and metrics.
    "timestamp": "2025-02-28T09:13:57.941Z",
    "message": "ApplicationAttemptId: appattempt_1740734011890_0001_000001",
    "logger_name": "org.apache.spark.deploy.yarn.ApplicationMaster",
    "level": "INFO",
    "thread_name": "main"
    // ...
  }
}

Once diagnostics are emitted to Azure Event Hubs, you can use that event hub as a source in a Fabric eventstream to process or route the data.

Fabric workspaces with Managed virtual network

Create a managed private endpoint for the target Azure Event Hubs. For detailed instructions, refer to Create and use managed private endpoints in Fabric.

Once the managed private endpoint is approved, users can begin emitting logs and metrics to the target Azure Event Hubs.

Next steps

• Create Apache Spark job definition
• Create, configure, and use an environment in Fabric
• Create and use managed private endpoints in Fabric
• Develop, execute, and manage Fabric notebooks
• Monitor Spark Applications