Synapse Pipeline and Spark Job Delays across West US2 and East US2, from 5/28 11PM to 5/29 1AM

Christine Han 140 Reputation points Microsoft Employee
2026-05-29T18:41:21.0133333+00:00

Hi Synapse supporting:
We found large amount of pipeline runs and spark jobs created from 5/28 11PM to 5/29 1AM were stuck for 6 to 24 hours. These jobs resumed randomly after long delays. This issue happened across all of our Synapse workspaces in both West US2 and East US2. It has caused many problems from yesterday through this morning.

The issue caused:

  1. Failures in bronze/silver data processing pipelines due to conflicts on parquet files.
  2. Spark pool capacity drain, because stale pipelines started running again several hours later at the same time as regularly scheduled pipelines.

Two sample issues are below:

  1. Long-running Spark job. The Spark job started at 5/28 7:56 UTC. The actual data processing completed within 30 minutes, however the Spark job itself was not completed until 24 hours later.
    User's image

From the log, you can see the job was sleeping for one day!User's image

  1. Long-running "Lookup" activity in pipeline: The Lookup activity was running for 6 hours. Pipeline runs and Spark jobs
    User's image

Please let us know is there outage event happening from Synapse side, and what's the preventive actions?

Azure Synapse Analytics
Azure Synapse Analytics

An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.

Answer accepted by question author
Pilladi Padma Sai Manisha 11,705 Reputation points Microsoft External Staff Moderator
2026-06-10T05:45:03.1866667+00:00

Hi @Christine Han
Issue Summary Customers experienced connectivity issues when accessing Azure Synapse Studio through private network connectivity, resulting in slow page loads, timeouts, and workspace access failures.

Root Cause A recent platform configuration change unintentionally affected request routing for Azure Synapse Studio when using private connectivity. As a result, some client requests were directed to invalid service endpoints, causing connectivity failures before reaching the backend services.

Resolution The engineering team identified the routing issue, rolled back the affected configuration, and implemented the necessary corrections. Connectivity was validated after the fix, and normal service functionality was restored.

Preventive Actions Microsoft has enhanced validation processes for private connectivity scenarios, improved automated testing coverage for end-to-end workspace access, strengthened deployment safeguards and monitoring, and completed a Post Incident Review (PIR) to help prevent similar issues in the future.

Was this answer helpful?

2 people found this answer helpful.
0 comments

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Sina Salam 31,456 Reputation points Volunteer Moderator
    2026-05-30T15:20:11.05+00:00

    Hello Christine Han,

    Welcome to the Microsoft Q&A and thank you for posting your questions here.

    I understand that you are having Synapse Pipeline and Spark Job Delays across West US2 and East US2, from 5/28 11PM to 5/29 1AM.

    This strongly indicates a Synapse orchestration/Spark control-plane delay or Azure service-side degradation. Azure public status also recorded a May 29, 2026 “Multiple Services experiencing degradation in West US 2” incident, tracking ID GHRP-84G; however, East US 2 impact must be confirmed from Azure Service Health for the affected subscriptions. - https://azure.status.microsoft/en-us/status/history/, https://azure.microsoft.com/en-us/blog/get-an-official-service-issue-root-cause-analysis-with-azure-service-health/

    The immediate fix is to collect run-level evidence, cancel unsafe stale runs, pause schedules where necessary, and prevent overlapping parquet writes. The permanent mitigation is to enforce pipeline concurrency, idempotent staging-to-final parquet writes, separate recovery Spark pools, and configure Azure Monitor plus Service Health alerts.

    The below steps are resolution and associated links for more reading and steps:

    • Validate the incident from Azure Service Health > Health history for the affected subscriptions, services, regions, and timestamps.
    • Open a Microsoft support case with workspace names, pipeline run IDs, activity run IDs, Spark application IDs, Spark pool names, UTC timestamps, and screenshots from Synapse Monitor.
    • Cancel or quarantine unsafe stale pipeline runs before allowing normal schedules to continue.
    • Enforce pipeline concurrency and idempotent parquet writes so late-resumed runs cannot overwrite or conflict with current bronze/silver processing.
    • Separate recovery/backfill Spark workloads from normal production Spark pools and configure autoscale based on backlog recovery requirements, not only normal workload size.
    • Add Azure Monitor and Service Health alerts for long-running pipeline activities, abnormal Spark application duration, stale runs, and regional Azure service incidents.

    Regarding your questions use the below as a reference and steps:

    I hope this is helpful! Do not hesitate to let me know if you have any other questions, steps or clarifications.

    Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

    Was this answer helpful?

    1 person found this answer helpful.

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.