Training Pillars

Recognize the tools responsible for bringing data into Foundry and transforming it into reliable, governed datasets. Understand what each tool does and where it sits in the data journey. Navigate a real project's lineage, interpret build failures, and use AIP Assist to investigate transform logic.

Recognize the full output side of the Foundry platform - how governed data becomes ontology objects, operational applications, analytics surfaces, AI-powered workflows, and managed infrastructure. Build a basic Contour analysis using a provided dataset, applying filters and groupings to answer a defined business question.

Understand Foundry's connection architecture and differentiate between Direct and Agent-based connections. Configure import and export controls at the connection level. Request and approve egress policies through the proper governance workflow. Create and compare Snapshot, Update, and Incremental syncs, and automate them using UI-based schedules and custom cron expressions.

Configure REST API connections and register inbound webhooks for event-driven ingestion. Establish FTP and SFTP connections for file-based data transfer. Distinguish between Datasets and Virtual Tables and select the appropriate representation based on performance and governance requirements. Configure write-back workflows to push data from Foundry back to a source system. Evaluate when to use Listeners for inbound low-throughput event streams where the source system cannot be customized.

Understand HyperAuto's architecture and its role as Foundry's purpose-built ERP integration layer. Recognize how HyperAuto maps ERP modules and fields to Foundry datasets. Evaluate when HyperAuto is the appropriate path versus a generic Data Connection for ERP data extraction. Configure an Agent Connection to an ERP-like source system as a practical alternative when HyperAuto is unavailable.

Understand the role and responsibilities of a Foundry data engineer within a delivery team. Internalize core data governance principles including data ownership, naming conventions, data contracts, and documentation standards. Identify and distinguish Foundry's primitive data structures - Data Sources, structured Datasets, unstructured Datasets, Timeseries Syncs, and Media Sets. Understand the architectural distinction between Datasets and Object Types and when data graduates from one to the other. Recognize the Raw→Clean→Ontology layering pattern and articulate why each layer exists. Navigate both Pipeline Builder and Code Repositories at an orientation level.

Design and build complete data pipelines using Pipeline Builder's full node set - Transform, Split, Join, Union, Use LLM, Generate, Explain, and Add output. Create and apply Custom Functions to encapsulate reusable logic. Design pipelines that produce clean, traceable Data Lineage. Apply schema management practices and handle schema drift defensively. Make informed dataset partitioning decisions based on downstream query patterns. Apply data governance principles and architectural best practices for visual pipelines, including naming conventions, documentation, and avoiding common design pitfalls.

Navigate Code Repositories using either the classic IDE or VS Code editor surface. Understand Foundry's branching model - how it diverges from standard git, how to isolate work in branches, and how branches interact with builds. Apply Python fundamentals for data engineering including data types, control flow, list comprehensions, and error handling. Write production-ready transforms using the `@transform` and `@transform_df` decorators. Apply common PySpark transformation techniques including select, filter, withColumn, cast, join, union, groupBy, agg, window functions, and sorting. Preview transform outputs during development. Export Pipeline Builder pipelines to code and convert the resulting Java to Python. Use Continue for agentic development assistance within Code Repositories.

Select the appropriate compute library - PySpark, Polars, or Pandas - for a given dataset size and transformation pattern. Apply advanced decorators: `@transform_pandas`, `@transform_polars`, `@configure`, `@incremental`, and `@lightweight`. Consume external APIs from within transforms using External Systems in Code Repositories. Read and parse unstructured datasets from the Foundry filesystem. Apply common performance optimizations per library including vectorization, lazy evaluation, Spark profile configuration, coalescing, and repartitioning. Design dataset partitioning strategies based on downstream query patterns. Detect and handle schema drift defensively in code. Write unit tests for transform logic. Apply production best practices - avoiding full scans, leveraging lazy transformations, and structuring pipelines for maintainability and governance.

Engineer production-ready pipelines for Foundry's most complex data structure types. Structure timeseries datasets correctly using the soho format for optimal loading and Time Series Catalog registration. Work with geospatial data - ingesting, transforming, and querying geopoints and polygons, applying spatial joins and proximity filters. Handle advanced structural types - nested structs, arrays, maps, and semi-structured JSON payloads - both in Pipeline Builder and Code Repositories transforms.