Summary
How can a PySpark application be developed in the browser and run in OCI Data Flow?
The tutorial prepares OCI authentication and Spark in Cloud Shell, creates a Code Editor project and uses Spark Oracle Datasource to read an Oracle Autonomous Database table. The wallet is stored in OCI Object Storage, and the application configures database connection details.
The script is first tested in Cloud Shell with Run Locally, checking the first five records. The Python artifact is then uploaded to Object Storage, and a Data Flow application is created to run it on managed resources.
Resource Principal is configured for OCI resource access while the example database credentials remain in use. After the Data Flow execution, Spark UI stages and the DAG are reviewed to understand job dependencies.
Start
OCI Data Flow is introduced for distributed processing applications, with Code Editor for browser-based development. The walkthrough prepares the environment, builds a PySpark application that reads an Autonomous table, tests it locally and publishes it for Data Flow execution. Spark UI reviews the resulting job.
Requirements and authentication for Code Editor
Code Editor and Cloud Shell prerequisites include access policies, OCI user configuration and Spark environment tools. The presenter prepares an API Key, downloads its key file and keeps the configuration preview for later steps.
Create the project at Code Editor
Code Editor opens from Data Flow, and a project is created from scratch. Its name is defined, it is located in the selected compartment, and its terminal opens. The project provides the workspace for preparing credentials and the Python file.
Configure OCI's Credentials
OCI configuration is prepared with profile values and the private key path. Files are placed in the user configuration directory, and the profile is checked for correct reading. The configuration and key copies initially created inside the project are then removed.
Prepare Spark at Cloud Shell
The script for configuring Spark in Cloud Shell for Data Flow is introduced. The presenter explains that the environment is already prepared; setup needs to finish before application execution. This connects OCI authentication with the project processing tools.
Create PySpark with Spark Oracle Datasource
A Python file creates a Spark session with PySpark. Oracle examples provide the Autonomous read using Spark Oracle Datasource and a wallet in Object Storage. The connector enablement option is reviewed, and the code is adapted to the example connection.
Autonomous Database's Wallet at Object Storage
The wallet is downloaded from Autonomous and uploaded to an Object Storage bucket. The script sets the filename, namespace and bucket, together with connection identifier, schema and table. Database username and password are also configured, followed by a preview of the first five records.
Local Test with Run Locally
Code Editor selects Run Locally, Python and the application file. Spark Oracle Datasource is enabled, and the test starts in Cloud Shell. The log shows the first five table records, checking the read before creating the managed Data Flow execution.
Publish artifact at Object Storage
Upload Artifact uploads the Python application from the project. The Object Storage bucket path is specified, and the bucket view is refreshed to verify the script. This artifact becomes the Data Flow application input.
Create an application at OCI Data Flow
A Data Flow application is created with a name, Spark version and Python language. The example retains default driver and executor settings. The published script bucket and path are selected, and the log destination is configured before advanced options are reviewed.
Authentication with Resource Principal
Advanced options enable Resource Principal and the Spark Oracle Datasource property. Resource Principal authenticates OCI resource access from the execution. The Autonomous connection also uses the wallet and database credentials configured in the example. These options complete application creation.
Run the application at Data Flow
Run starts the managed application. Data Flow provisions Spark cluster resources and executes the published script. The walkthrough waits for completion before opening monitoring tools. This Data Flow execution follows the earlier local Cloud Shell test.
Spark UI and DAG display
Spark UI opens after completion to inspect job stages. The DAG visualization represents dependencies between stages and tasks, helping review distributed dataset processing. The closing connects this view with analysis and debugging of Spark applications developed in Code Editor.