Summary
Take your Bronze data to Gold and connect them with analysis and governance at Oracle Cloud. In this video I show you how to combine OCI Object Storage, Oracle AI Data Platform Workbench, OCI Data Catalog and Oracle Analytics Cloud into a makehouse architecture with medallion layers.
We'll go through the buckets, Master Catalog, tables Delta and the organisation of the workspace. We review computer configuration, Spark UI metrics and workflow orchestration with parameters to process specific dates and work with incremental or complete loads. We also explore audit, login and access roles.
We have been trying three useful cases: to add a column to a Delta table by keeping historic data, mask information from customers with OCI Language or regular expressions, and run SQL consultations about the Gold Layer to get reports.
Then we organize the business context with a glossary at OCI Data Catalog and connect Oracle Analytics Cloud with AIDP to select tables, link data and prepare visualizations. Thus you can follow the path from storage and transformation to analytical consumption.
Start
Following the introduction, a tour with OCI Object Storage, Oracle AI Data Platform Workbench, OCI Data Catalog and Oracle Analytics Cloud is presented. The aim of the example is to track data from the medallion layers to their business organisation and their analytical consumption.
OCI Object Storage and medallion layers
The lab's buckets and the separation of data by layers are scattering. The Bronze cape contains traffic files, references and examples to prove masking and schematic evolution. The storytelling also distinguishes the Landing area, which can be organized in another comparison according to the design of the environment.
Access to Oracle AI Data Platform Workbench
The Oracle Cloud console opens the instance of Oracle AI Data Platform Workbench. Access requires confirmation of authentication in mobile application. after entering the environment where catalogs, projects and resources from the lab are registered.
Master Catalog and tables Delta
Master Catalog organizes the Bronze, Silver and Gold layers, with the tables reference and monitoring. columns and data types of tables Delta are inspected, including examples of traffic, masking and column growth. They also review the exchange history and options to incorporate other catalogs.
Workspace, notebooks and project organization
The workspace grouping notebooks and archives according to the cape of the process. The Shared folder for the work of the computer is distinguished and intake from Landing, reusable references and consumption queries are found at Gold. The notebooks and jobs are organized to later associate them with a execution cluster.
Compute and cluster configuration
The computer options that will run notebooks, jobs and queries are reviewed. The example shows a standard cluster, resources alternatives and inactivity shutdown with an interval of 120 minutes, as well as an option to keep it up and up. From your connection options you get the file that will be used later in Analytics.
Spark UI and execution metrics
Spark UI allows to review the processes run in the cluster, use and metrics of their nodes and executors. The explanation relates these views to the monitoring of loads: as well as confirming that a process has been completed, they allow us to observe how they consumed resources during their execution.
Workflows, parameters and load modes
The traffic workflow links the intake and transformations of Bronze, Silver and Gold. Parameters are shown to select a processing date and an incremental or complete Loading Mode. The flow should also be associated with the infrastructure where its tasks will be performed.
Centralised audit and login
The centralized view brings together the audit and execution records of notebooks, jobs, tasks and calls to UPIs of the environment. That point allows you to place activity messages without having to review each resource separately.
Roes and collaborative access
The management of roles and users will be dedicated to organizing access to the environment. The explanation puts these permissions into collaborative work: different participants can come into the project and run their respective processes.
Retirement and incremental Loading
The test starts with a synthetic table and runs a workflow that incorporates an additional column without eliminating the history. The details, output and time line of his tasks are reviewed and the Bronze, Silver and Gold tables are compared. The result allows us to observe how the scheme grows and how earlier stage records are represented.
Data Mapping with OCI Language and regex
The workflow processes client data and puts a masked version into Gold. Email, phone and free text are reviewed and the output indicates whether the detection was made using OCI Language or regular expressions. The tables and final consultation allow us to check the data that have been passed through each layer with the values that will be shown to consumers.
SQL Consultation about Gold Layer
A report SQL about Gold data is run from the workspace. A complete execution and an output limited to ten records are reviewed to explore the result. The demonstration shows how a consumer consultation can coexist with the transformation notebooks within the project.
OCI Data Catalog and business glossary
OCI Data Catalog contributes the glossary with sources, tables, columns, categories and lab relations. The tour includes upgrade jobs and examples of references that link objects from Bronze and Silver. The aim is for the equipment to be able to track a data, understand its context and keep its relationships documented before applying or re-using information.
Connection Oracle Analytics Cloud with AIDP
Oracle Analytics Cloud creates a connection with AIDP using the configuration file obtained from the cluster. The explanation indicates that this computer caters to different users of the lab: notebooks, jobs and analytics. The connection allows you to gain access to the schemes and to prepare an analysis about available tables.
Creation and selection of tables dataset
A dataset is created from the AIDP connection and Gold and reference tables are dragged as antennas and traffic aggregates. When an analysis requires more details, Silver information will also be incorporated. The selection of tables prepares information that will be related to the next instalment.
Relationship between tables and data profiling
The dataset allows to review the ratio between tables and a column profile with at least metric, maximum, unique values and date types. The cache and upgrade options are also discussed. These views help to check the data structure before saving the set and building visualizations.
Visibility and closure
The dataset with its production will create visualities. The closure brings together the complete course: storage at Object Storage, AIDP medallion processing, Data Catalog governance and Oracle Analytics Cloud consumption.