What are Pipelines?
Pipelines chain multiple tasks together into automated workflows. Use them to:- Run tasks in sequence
- Schedule recurring data jobs
- Handle errors and retries
- Build end-to-end data processes
Creating a Pipeline
1
Go to Pipelines
Click Pipelines in the sidebar, then click + New Pipeline
2
Name Your Pipeline
Give it a descriptive name like “Daily Sales ETL”
3
Add Tasks
Drag tasks from the left panel onto the canvas
4
Connect Tasks
Draw lines between tasks to define the execution order
5
Configure Schedule (Optional)
Set up when the pipeline should run automatically
6
Save
Click Save to save your pipeline
The Pipeline Canvas
The visual canvas lets you design your workflow:- Tasks appear as boxes
- Arrows show data flow direction
- Colors indicate task status when running
Adding Tasks
- Click + Add Task or drag from the task list
- Select an existing task or create a new one
- Position it on the canvas
Connecting Tasks
- Hover over a task’s right edge
- Drag to create a connection
- Drop on another task’s left edge
Scheduling Pipelines
Run pipelines automatically on a schedule:Setting a Schedule
- Open your pipeline
- Click Schedule tab
- Choose frequency
- Set the time
- Save
Running Pipelines
Manual Run
- Open the pipeline
- Click Run Now
- Monitor progress on the canvas
Viewing Run History
- Go to the pipeline
- Click Runs tab
- See all past executions with status
Error Handling
When a task fails:- The pipeline stops at that task
- You see an error message
- Fix the issue
- Click Retry to continue from the failed task
Example Pipeline
Daily Customer Report- Fetch Orders: SQL query to get today’s orders
- Calculate Metrics: Python script to compute KPIs
- Generate Report: Format data as HTML report
- Send Email: Deliver to stakeholders
Best Practices
Start Simple
Build with 2-3 tasks, then expand
Test Each Task
Verify tasks work individually first
Add Descriptions
Document what the pipeline does
Monitor Runs
Check run history for failures