Using Azure Data Factory for data integration with a Data Vault data wareho...
Event Information
Description
Approx. Running order for the night
- 6:15 - 6:30: Registration & Intro
- 6:15 - 7:15: Part 1.
- 7:15 - 7:30: Break and Refreshments
- 7:30 - 8:30: Part 2.
Featured Presentation:
Using Azure Data Factory for data integration with a Data Vault data warehouse
Dan Galavan, Data Architect GSD
Summary
In this session you will receive an introduction to both Azure Data Factory v2 and the Data Vault data modelling methodology. We will start with an overview of both followed by creating a Data Vault data warehouse along with an associated virtualized access layer in Azure. Finally, the pipelines needed to integrate data from an upstream application database will be implemented using Azure Data Factory v2.
Azure Data Factory
Azure Data Factory is a cloud-based data integration service that allows you to create workflows for orchestrating and automating data movement and data transformation. The data-driven pipelines (workflows) can ingest data from disparate data stores. Azure Data Factory can process and transform data by using compute services such as Azure HDInsight Hadoop, Spark, Azure Data Lake Analytics, and Azure Machine Learning. Additionally, you can publish data to data stores for business intelligence (BI) applications to consume. Ultimately, through Azure Data Factory, raw data can be organized into meaningful data stores for better business decisions.
Data Vault data modelling:
Over 80% of data warehouses in the Netherlands have been implemented using the Data Vault methodology, and has been touted as the next generation data modelling approach. The Data Vault makes a clear distinction between data acquisition & auditing vs. data interpretation & interpolation. This talk will show you how a single version of the facts may just edge out a single version of the truth. The knock-on effect is high adaptably to changing business requirements, something that can be a challenge with traditional data warehousing approaches. This also leads to faster sprints. Additionally, the Hub (core business concept), Link (relationship) and Satellite (context and history) design lends itself well to massively parallel ETL processing.
About Dan:
A Data Architect who has been working with databases since 1998. Has delivered solutions in a variety of industries including Banking, Investment Banking, Aerospace, CRM, Social Protection, Telecoms, and Health Product Regulation - primarily using the Microsoft Data Platform. While living and working abroad, on St. Patrick’s Day 2003 Dan came face to face with MS SQL Server for the first time - it’s not a national holiday in Germany for some reason! Starting with SQL Server Database design and development, Dan branched into ETL, Reporting, Data Modelling, DB optimization, Data Quality Management and Master Data Management. Regular contributor to the Dublin SQL Server User Group.
Organiser Belfast SQL Server User Group
Organiser of Using Azure Data Factory for data integration with a Data Vault data warehouse
The Belfast SQL Server User Group was started in August of 2012.
I had been at a SQLBits conference a few months before and saw there are a number of SQL Server User Groups based in the UK. One key thing to note is that although there is a user group in Dubin, there were none in Northern Ireland.
I made some enquiries into what was involved in starting a SQL Server User Group within the greater Belfast area, contacting the SQL User Group (http://sqlserverfaq.com/).
After a bit of time I was able to secure a Venue (big thank you to Liberty IT), some speakers and promotion for the events.
If you would like to help with any events, have any particular topics you would like to hear about or have another questions please contact us at sqlbelfast@gmail.com.
