A "Business Data Pipeline" (BDP) is a structured pathway that facilitates safe and efficient data sharing and integration across BYU departments. It describes data moving from a system of record like Workday or AIM to another system like Tableau or the locker rental app.
BDPs are secure channels designed for accessing and consuming BYU’s domain-specific data. Authorized users can request access to these data products through a Data Sharing Agreement (DSA). Only authorized users, who have obtained a DSA, can access the BDPs. The authorization process is managed through Alation, a catalog tool that helps users locate available information within the pipeline.
How a BDP Works at BYU
A good BDP connects the following activities:
- Collects and assembles data
- Securely transfers and stores data
- Accounts for and provides access to data
- Uses and reports data
BDPs provide both transactional data (records of events) and analytical data (used for analysis and insights). The platform can cater to various use cases depending on the domain’s needs.
Purpose and Benefits of BDPs
BDPs exist to improve how data is shared, governed, and used across BYU.
Key Benefits
- Standardization and Normalization: BDPs ensure data is consistently formatted and standardized. Users can trace the origin and calculation methods of the data, ensuring accuracy and reliability.
- Abstraction Layer: The pipeline allows data to be connected and disconnected without disrupting systems, enabling smooth integration and updates.
- Enhanced Data Governance: Data owners have more control over their data, allowing them to govern, standardize, and manage it effectively.
- Efficient Data Access: By meshing information from different sources, BDPs make fragmented data cohesively accessible across multiple organizations, saving time and resources.
When data stewards request a dataset be included in a BDP, OIT provides expert data modeling to ensure high data quality and secure dataset storage. Pre-building datasets simplifies the data catalog and request process. Review and approval of data sharing agreements are easier with pre-built datasets than with ad hoc data fields, and pre-built datasets improve overall data quality.
Sometimes, pre-built datasets in BDPs need to be blended in order to facilitate data analytics and reporting. This is especially true when different domains must be combined, such as student data, human resource data, and financial data. Blending uses the same expert data modeling, secure dataset storage, and governance steps.
Data Governance and Access
As BYU collects and assembles data, the data steward determines what datasets will be shared in the BDP. These datasets are classified as public, internal, confidential, or restricted.
To receive access to a dataset in the BDP, the requestor must have a data sharing agreement approved by the data steward. Data stewards are appointed by data trustees (Vice Presidents).
A data workgroup serves as the “front door” to a Business Data Pipeline. A data workgroup is typically made up of representatives from a campus unit (those needing data) and OIT’s data team. Through these workgroups, OIT helps campus units with:
- Submitting a data sharing agreement to the data steward for an existing dataset
- Requesting a new dataset to be built (with approval of the data steward)
- Requesting a new BDP
Components of a BDP
BDPs are built from several core components:
- Domains: Categorized BYU data, each headed by a trustee and managed by stewards. These stewards appoint business experts who collaborate with OIT to meet data needs.
- Data Products: A catalog for finding data, usage agreements to set sharing boundaries, usage trackers to monitor data usage, and performance metrics.
- Templates: Include datasets, interfaces, and pipelines for consistent data handling.
- Provisioning: Involves version control, updates, and configurations to maintain data integrity.
- Contracts: Detail usage agreements, terms, and billing policies, with OIT funding the platform for campus-wide use.
Technology Involved in a BDP
Leading off-the-shelf software is used in a BDP.
- To model datasets and securely transfer and store them, OIT uses Informatica and Amazon Web Services.
- To account for and provide access to datasets, OIT uses Dremio and Tyk.
- OIT has teams of employees trained in these software tools and maintains strong vendor relationships.
Data Quality
Good data quality maximizes:
- Accuracy
- Completeness
- Consistency
- Validity
- Timeliness
Data Accuracy
Data that reflects the real world has good accuracy. Example: A student moves to a new apartment and the new address is reflected in the student information system.
Data Completeness
Good completeness reduces missing information. Example: A student’s phone number includes all ten digits.
Data Consistency
Good consistency means source data and delivered data match. Example: Information in the financial system moves through a business data pipeline and matches across systems.
Data Validity
Information that fits defined parameters is considered valid. Example: A number-only field contains only numbers.
Data Timeliness
Data snapshots that refresh within the defined frequency are timely. Example: A snapshot refreshes daily at 4 a.m.