Install a different version of SynapseML on Fabric

SynapseML (Synapse Machine Learning) is preinstalled on Microsoft Fabric. If you need a specific version, you can override it by using the %%configure magic command in a Fabric notebook.

Quick start

Prerequisites

Before you begin, make sure you have the following resources:

SynapseML and Fabric runtime compatibility

Choose the SynapseML version and Spark Avro JAR that match your Fabric runtime Spark version:

You can find available SynapseML versions on Maven Central.

Install SynapseML with %%configure

The %%configure -f command configures Spark session properties, including Maven package coordinates. The -f flag forces a restart of the existing session.

Example for Fabric Runtime 1.2 (Spark 3.4)

Paste this code into the first cell of a new Fabric notebook and run the cell:

%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:1.0.8,org.apache.spark:spark-avro_2.12:3.4.1",
      "spark.jars.repositories": "https://mmlspark.azureedge.net/maven",
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind",
      "spark.yarn.user.classpath.first": "true",
      "spark.sql.parquet.enableVectorizedReader": "false",
      "spark.sql.legacy.replaceDatabricksSparkAvro.enabled": "true"
  }
}

To install a different version, replace the version numbers in spark.jars.packages:

• synapseml_2.12:<version>: the SynapseML version you want (for example, 1.0.4).
• spark-avro_2.12:<spark-version>: must match your Fabric runtime Spark version (see compatibility table).

Example for Fabric Runtime 1.1 (Spark 3.3)

%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:0.11.4,org.apache.spark:spark-avro_2.12:3.3.1",
      "spark.jars.repositories": "https://mmlspark.azureedge.net/maven",
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind",
      "spark.yarn.user.classpath.first": "true",
      "spark.sql.parquet.enableVectorizedReader": "false",
      "spark.sql.legacy.replaceDatabricksSparkAvro.enabled": "true"
  }
}

Verify the Spark configuration

After the cell runs, verify that the Spark session accepted the configuration. Run this code in a new cell:

print(spark.conf.get("spark.jars.packages"))

Expected output (for the Runtime 1.2 example):

com.microsoft.azure:synapseml_2.12:1.0.8,org.apache.spark:spark-avro_2.12:3.4.1

Check SynapseML version

To verify the installation, run the following code in a new cell. The version number should match the version you installed.

import synapse.ml.lightgbm
print(f"SynapseML version: {synapse.ml.lightgbm.__version__}")

Expected output (for the Runtime 1.2 example):

SynapseML version: 1.0.8

Revert to the default version

To revert to the preinstalled SynapseML version, remove the %%configure cell and restart the notebook session:

1. Delete or comment out the %%configure cell.
2. Select Session > Stop session in the notebook toolbar.
3. Run any cell to start a new session with the default configuration.

Configuration reference

The %%configure block includes these Spark properties:

Troubleshooting

%%configure cell doesn't take effect

Cause: The %%configure cell wasn't the first cell to run, or a Spark session was already active.

Fix: Select Session > Stop session, then run the %%configure cell first.

JAR download fails or times out

Cause: The Maven repository is unreachable, or the version coordinates are incorrect.

Fix: Verify the SynapseML version exists on Maven Central. Check that the spark-avro version matches your Spark runtime version.

ClassNotFoundException or NoSuchMethodError at runtime

Cause: The SynapseML version isn't compatible with the Fabric runtime Spark version.

Fix: Use the compatibility table to select the correct version combination.

ImportError: No module named 'synapse.ml'

Cause: The SynapseML Python wrapper package isn't installed in the notebook environment.

Fix: In a new cell, run:

%pip install synapseml==1.0.8

Replace 1.0.8 with the version that matches the JAR you installed.

Version number doesn't match the installed version

Cause: The Python __version__ attribute comes from the synapseml pip package, which might differ from the JAR version installed by %%configure.

Fix: Verify both the Spark config and the pip package version:

# Check the JAR version from Spark config
print("JAR packages:", spark.conf.get("spark.jars.packages"))

# Check the pip package version
import synapse.ml.lightgbm
print("Pip package version:", synapse.ml.lightgbm.__version__)

If the versions don't match, install the matching pip package:

%pip install synapseml==1.0.8

Related content

• How to use LightGBM with SynapseML
• How to use AI services with SynapseML
• How to perform the same classification task with and without SynapseML
• Manage Apache Spark libraries in Fabric