• Home
  • AWS
  • Run interactive workloads on Amazon EMR on EC2 with Spark Connect

Amazon EMR on EC2 now supports interactive Apache Spark sessions with Spark Connect. Data engineers and data scientists can develop and debug Apache Spark applications interactively from managed notebooks in Amazon SageMaker Unified Studio and their own IDEs, such as Jupyter and Visual Studio Code, with each session running on dedicated EMR on EC2 clusters. You can also monitor and debug active and completed sessions in the EMR console.

 

An interactive session provides a persistent Spark context that spans across cells and scripts, letting you blend local Python code execution with remote Spark operations. Spark Connect’s client-server architecture decouples your application client from the Spark driver and allows you to maintain your preferred development environment and tooling while Spark infrastructure runs on the cluster. This architecture supports workflows including ad hoc data exploration, iterative step-by-step debugging, and incremental PySpark job development before deploying to production. For observability, you get real-time session monitoring via the Spark UI, history tracking through the Spark History Server, and session management from the EMR console or API/CLI/SDK.

 

Interactive Sessions is available on Amazon EMR on EC2 with AWS runtime for Apache Spark (emr-spark-8.0) and later, in all AWS Regions where Amazon EMR is available, except the AWS GovCloud Regions and the China Regions. The Amazon SageMaker Unified Studio experience is available in supported regions. To get started, visit the Interactive sessions with Spark Connect guide or the Amazon SageMaker Unified Studio Getting Started guide.

Source Link: https://educronix.com/run-interactive-workloads-on-amazon-emr-on-ec2-with-spark-connect/ Author: aws - Published on: 2026-08-04 20:29:00
This post was originally published on this site
Share this post

Subscribe to our newsletter

Keep up with the latest blog posts by staying updated. No spamming: we promise.
By clicking Sign Up you’re confirming that you agree with our Terms and Conditions.

Related posts

☎
New Educronix Product

Educronix Softphone

Free WebRTC desktop softphone for Windows and macOS. Connects directly to your PBX — voice and video calls, Call Waiting, DND, live call quality and more. Choose your edition and platform:

100% WebRTC — built on the JsSIP library.

Standard Edition
Call Center Edition
🎙 AI Assistant(voice)