Overview:
Pipeline Manager is the part of Bright Analytics where your data is brought in from your connected sources. It covers connecting to a source, running the jobs that sync data on a schedule, and monitoring whether those jobs succeed.
Users with granted permissions can access the Pipeline Manager tool from the home screen.
The ETL component
Every job in Pipeline Manager follows the same underlying process - Extract, Transform, Load (ETL):
Extract: data is pulled from a connected source, using one of your integrations.
Transform: data is reshaped or standardised into a usable format. This can happen automatically as part of a job, or manually through a Custom Transform job for cases that need specific logic.
Load: the finished data is stored in Bright Analytics, ready to be used in your Semantic Layer and reports.
Monitoring your jobs:
The Jobs screen gives you a live view of every job connected to your instance, with tabs to filter by status: All, Active, Paused, Inactive, and Failures, each showing a count.
Each job is listed with:
Source / job name: for example, "CM360 Data Import #1 / DCM Performance Reports."
Status indicator: shows whether a job is running, paused, or inactive.
Frequency: how often the job runs, e.g. every 24 hours.
First run: the time of day the job's first run of the cycle starts.
Lookback count: how many days back each run checks for updated data.
Lookback window: an additional buffer applied to the lookback count.
Run history bar: a visual record of recent runs for that job.
From here, you can also create a New job or add a New integration using the buttons in the top right.
Connections:
The connections tab shows you all the current integrations that are feeding into your reports.
Clicking into a specific integration will open up the connection settings and options, where users can amend and update the values such as account IDs.
