Is your customer looking for drag and drop reports or capabilities inside of Excel. SQL Server Analysis Services (SSAS) is the answer for you then. You’ll be amazed how quickly you can develop sophisticated reports after watching the basics of this session.

SSAS is its own server. It is not part of the SQL Server service.

• IIS
• XMLA between server and client

MDX – SQL Language for Cubes
BIDS – Building the Cubes
SSMS – Managing SSAS

Analysis Services

What are cubes?

• What are measures? (See Erik Veerman Dimensional modeling session)
  • The value you are measuring, e.g. the count of users with blue eyes, from WA state, etc.
  • The question you are trying to ask
• What are measure groups?
  • Groupings of these measures within your cube
  • Typically define them by your business problem (e.g. Sales, HR, etc.)
• Cubes are a grouping of measure groups.

Analysis Services Dimensions

What are dimensions and hierarchies? (See Erik Veerman Dimensional modeling session)

• What is a dimension?
  • It is what you want to categorize or pivot against
  • e.g. How many users are from WA state?
  • The dimension is geography, “WA” is the member
• •What is a hierarchy?
  • How do you want to organize the members of your dimension
  • e.g. Geography has numerous levels such as country, state, and city
  • Hierarchy is the organization of those levels such as City ->State ->Country

Concrete examples like these are helpful for my learning and understanding. Thanks!

User Interfaces

• Query language is MDX
• BIDS for developers
• Excel for most users is adequate or Reporting Services
• SharePoint for web users
• 3rdparty applications like Proclarity or Panorama
• Controls you can purchase and build into your own apps

Create User defined hierarchies.

• Remember that in 2008 the hierarchy can be thought of as reversed.
• Looks graphically like date -> month -> qtr -> year
  • Means: to get to a date you must go through year, qtr, month, date

Deal with problematic keys.

Most end users will use Excel. 2007 provides additional features.

In this session, you’ll learn how to load a typical data warehouse in SSIS efficiently. You’ll start by seeing some of the strengths and weaknesses of the Slowly Changing Dimension (SCD) Wizard in SSIS and how you can get around some of the weaknesses including your own home-brewed solution. You’ll then see how to load a fact table using SSIS and how to make the common components scale.

Dimensional Modeling (See Dimensional Modeling 101 Notes)

• Data Separated into fact and dimension tables
• Dimension tables answer the pivot or where clause
  • Make as wide and descriptive as possible
  • Surrogate keys operate as unique ID for each row
  • Keep surrogate keys as small as possible
• Fact tables answer the what questions or select statement
  • Intersect all dimension tables
  • Surrogate keys from each dimension in this table
  • Measures are the “what” like Price, Quantity, Duration

The challenge is how you move the data from the OLTP (Relational) DB into the Data Warehouse.

Discusses SCD (Slowly Changing Data) Dimension types (Already here)

Don’t make everything Type two or your DB will bloat significantly. Also your reports would be more difficult to write.

Probably want to fix NULL values to be something. Use a Derived Column transform. Makes Nulls to be something like 0, Unknown, etc…

SCD Wizard Strengths

• SSIS transform that creates many other transforms conditionally
• Reduces design time of SCD load by 80%-90% to minutes per dimension
• Can be customized easily
• Compares differences between source and destination to find changes and new records
• Outputs:
• Type 0,1,2 update
• Inferred members
• New rows
• Duplicate rows

Historical Attribute Options – How do you want to set the expiration of historical records.

Problem with SCD Wizard is that any time you go back and change the configuration, all the output logic below it gets re-written. You lose what you created.

SCD Wizard Weaknesses

• Scalability -Generally up to about 50,000 records into the transform but varies based on number of updates
• Maintainability -After you customize, rerunning the wizard recreates all the transforms
• Uses OLE DB Command transforms for updates is row-level. Creates scalability issue here if lots of updates.

Making Your Own SCD Wizard

• Can use a Merge Join or Lookup Transform
  • If no match found, it is an insert (Ignore Errors)
• Lookup Transform will scale better than Merge Join but lacks parameterization
• Add a Conditional Split transform after Lookup to direct to insert, duplicate or update path

Additional Scalability

• Watch your Lookup Transformation for scalability issues (don’t cache too much!)
  • Potentially cache only the last 1 years worth of data with Partial Caching
  • Only cache columns needed
• Additional scalability can be reached by landing updates into a staging table
  • Then set-based update with an Execute SQL task.
• Checksum Transform can be used to detect changes across many columns
  • Or HASHBYTES T-SQL statement

Inferred Members

• Created during the fact load
  • A new Dim record is created using the value of “unknown” or NULL as a placeholder
  • The record is flagged as an inferred member

Slowly Changing Dimension Wizard

• SSIS transform that creates many other transforms conditionally
• Handles:
• Type 0 (fixed attribute)
• Type 1 (changing attribute)
• Type 2 (historical attribute)
• Inferred members
• Typically can address 80% of the business scenarios

SCD Wizard Strengths

• SSIS transform that creates many other transforms conditionally
• Reduces design time of SCD load by 80%-90% to minutes per dimension
• Can be customized easily
• Compares differences between source and destination to find changes and new records
• Outputs:
• Type 0,1,2 update
• Inferred members
• New rows
• Duplicate rows

SCD Wizard Weaknesses

• Scalability -Generally up to about 50,000 records into the transform but varies based on number of updates
• Maintainability -After you customize, rerunning the wizard recreates all the transforms
• Uses OLE DB Command transforms for updates is row-level. Creates scalability issue here if lots of updates.

Lookup Transform – Lookup source against target dimension table. Select ALL available lookup fields and alias at TARGET_ so you can match against them. Link by primary key. Ignore the failure of matches.

Then use a conditional split – If target PK is NULL it is a new record, else Update.

“Pretty much a pain in the butt to write.” – Brian Knight in reference to the else UPDATE piece from the conditional split.

Ugh…no transcript and this info is not in the slides. ARGH! Can’t remember it all.

Brian uses the Checksum source vs. Checksum destination to deal with the UPDATE referred to above, but not guaranteed to be unique.

Hashbytes on the other hand is more unique if you wish. But it does not work on numeric fields. Need to cast them.

Fact Table Loads

• Series of Lookup Transforms
  • In Type 2 Dimensions add WHERE EndDate IS NOT NULL
• Measures created Derived Column Transforms
• Aggregate transform to roll up the grain
• Lookup failure would create an inferred member or set to unknown

In a Business Intelligence solution, the fact tables hold the core data that you are analyzing – facts (also called measures). Therefore, fact tables are a critical component to get right the first time. Poor fact table design will lead to poor performance and difficult calculations. This session dives into fact table and considers the basic column types, measure aggregation types, fact table types, and volume considerations.

Facts

• The fact itself
  • The “measure” that is being tracked. The thing
  • Quantity, count, amount, percent
  • Most always numerical, continuous values
    • e.g., price of a product, quantity sold, budget value, count of customers
• Facts (or measures) can be classified by…
  • Numerical data type
  • Aggregation type
  • Additive nature
  • Granularity – level of detail stored
• Fact tables
  • Capture measures/facts
  • Association with dimensions (using surrogate key as foreign key in fact table)
    • No dimensions attributes!
  • Some tracking information included
• Different types of fact tables
  • Transactional – Additive facts tracking events over time (Star Schema)
    • Most common type of fact table
      • Track the occurrence of events, each detailed event is captured into a row in the fact table
      • Measures are typically additive across all dimensions
      • Common transactional fact table types
        • Sales, Visits, Web-page hits, Account transactions
  • Snapshot or inventory – Pictures in time of levels or balances
    • Periodic
    • Accumulating
    • Known as inventory level fact tables
      • Time dimension used to identify grain
      • Non additive measures across time, but typically additive across all other dimensions
      • Common transactional fact table types
        • Inventory levels, Event booking levels, Chart of account balance levels
  • Factless – Dimensionality relationships
    • No measured facts!
      • Are useful to describe events and coverage
      • Information that something has or has not happened
        • Often used to represent many-to-many relationships
        • Contain only dimension keys
        • Common factless fact tables:
      • Class attendance, Event tracking, Coverage tables, Promotion or campaign facts
• Fact Table Granularity
  • Never mix the grain of the table!
  • The level of detail of data contained in the fact table
  • The description of a single instance (a record) of the fact table
  • Typically includes a time level and a distinct combinations of other dimensions
    • e.g. Daily item totals by product, by store, Weekly snapshot of store inventory by product

Maybe include ETL load date/time in the fact table.

Measures – Additive Nature

• Additive: Facts that can be summed up/aggregated across all of the dimensions in the fact table
  • e.g. discrete numerical measures of activity, i.e. quantity sold, dollars sold
• Semi-Additive: Facts that can be summed up for some of the dimensions in the fact table, but not the others
  • e.g. numerical measures of intensity, i.e. account balance, inventory level, distinct counts
• Non-Additive: Facts that cannot be summed up for any of the dimensions present in the fact table.
  • e.g. room temp

Aggregations

• Aggregation (Aggs): A summarization of base-level fact table records
  • Common aggregation scenarios
    • Category product by store by day
    • District store by product by day
    • Monthly sales by product by store
    • Category product by store district by day
    • Category product by store district by month
• Aggregations need to account for the additive nature of the measures
  • Aggregations can be created on-the-fly or by the process of pre-aggregation
  • Common aggregations
    • Sum
    • Count, Distinct Count
    • Max, Min
    • Average
    • Semi-additive: Last Child, Last Non-empty Child

http://www.microsoft.com/fasttrack
SQL Server Fast Track Data Warehouse accelerates your data warehouse roadmap with new SQL Server 2008 Enterprise scalable reference architectures for HP, Dell and Bull. Reduce costs, save time and reduce risk with reliable, pre-tested hardware and best practices for warehousing. Read more.

Design with Additive in Mind

• Think dimensionally!
• Complex requirements don’t need to be designed with complex queries
• Many times new fact tables can be designed that can answer specific questions, such as date attributes and ranges

Getting Started

• Step 1: Identify high-value business process to model (orders, invoices, shipments, inventory)
  • Confirm data source availability
  • Understand value vs. complexity
• Step 2: Identify reporting grain of the business process
  • The grains is the level of detail at which the data should be represented for analytics
  • This may not be the same grain as the source!
  • For snapshot facts, determine what time level will be captured for each snapshot (daily, weekly, monthly)
• Step 3: Identify dimensionality that will apply to each fact table
  • Time, product, customer, store, etc.
  • Some dimensions are not grain-identifying
  • Validate the source can associate to the fact table
• Step 4: Identify measured facts that will populate fact table
  • Validate the base measures are identifiable from the source
  • Some measures may be derived
  • Measure examples: product count, quantity sold, dollars sold, inventory quantity
• Identify business questions:
  • How much total business did my newly remodeled stores do compared with the chain average?
  • How did leather goods items costing less than $5 do with my most frequent shoppers?
  • What was the revenue comparison of non-holiday weekend days to holiday weekend days?
• Analyze questions to assist design!

I really like these “Getting There” bullets. Erik does a good job summarize and giving you something practical to walk away with. Nice stuff.

Dimension Tables are one of the core components in a dimensional design and it is critical to design your dimension tables correctly in order to lay a solid foundation to a Business Intelligence system. This session dives into the dimension design techniques and considers the core components of a dimension table, surrogate keys, attributes, and hierarchies. In addition, we will consider advanced hierarchy types such as parent-child hierarchies, snowflake designs and unbalanced hierarchies. Finally, we will consider the best practices in tracking changes historically.

This session was intense so I just could not keep up with the slides. Here are just notes apart from the slides once I fell behind. Erik presents excellently and at a level appropriate for a 101 session without being condescending. Great job!

Dimensions (review from Dimensional Modeling 101 Session)

Dimensions – Qualitative
Facts (Measures) – Quantitative

• Dimensions (Qualitative information)
  • Business perspective from which data is looked upon
  • Collection of text attributes that are highly correlated
  • e.g. Product, Store, Time, Manager of Store
• Conformity (How Dimensions relate to each other)
  • Shared with multiple fact relationships
  • Provides data correlation
• Attributes (many times is related to a table column, not always)
  • Descriptive characteristics of an entity
  • Building blocks of dimensions, describe each instance
  • Usually text fields, with discrete values
  • e.g., the flavor of a product, the size of a product
• Hierarchies
  • drill-paths within the dimension
  • Allows top-down analysis
  • Natural groupings of data relationships
  • e.g., date hierarchy is most common. Year to qtr to month to day for example)
• Dimension Keys
  • Surrogate Keys (Primary key, DW created)
  • Candidate Business Keys/Alternate key (Source system keys, uniqueness)
• Dimension Granularity
  • Granularity in general is the level of detail of data contained in an entity (lowest level detail)
  • A dimensions granularity is the lowest level object which uniquely identifies a member
  • Typically the identifying name of a dimension

Name surrogate key with SK_
Dimension naming DIM_
Fact naming FACT_

http://www.codeplex.com/MSFTDBProdSamples to download the AdventureWorksDW example.

Hierarchies

ALL level is usually the top level

All -> Region -> Country -> City -> Office

Seeing one we could use would related to the various Bible translation project locations currently under way. Need to keep in mind project sensitivities.

A dimension can have multiple hierarchies.

Standard Hierarchies

• All levels have values.

Ragged Hierarchies

• Missing members at the mid-levels.
  • All -> Country -> State -> City and then you might be missing a state is the country is Israel since there are not states there.

Unbalanced Hierarchies

• Multiple grains in the hierarchy
• May not always be able to drill down to the lowest detail level since the branch may stop short. (Think Org Chart)

Demo: SSMS connected to a SSAS Cube

Product – Standard hierarchy

Sales Territory – example of unbalanced hierarchy

Employee – another example of an unbalanced hierarchy

Dimension Keys

Business Keys

• Column(s) identify the unique instance of a business record form the source system.
• You may have multiple records with the same Business Key. This is right as it allows historical tracking.
• Used in the process that ties fact records with dimension members
  • Business key is used to find the right surrogate key reference

Surrogate Keys

• • Defined the dimension’s primary key
  • Usually an integer
    • Important to pick the right column width.
    • 2,4,8 byte
      • SmallInt – 2 bytes go for up to around 10k records
      • BigInt – 8 bytes, for REALLY large dimensions
  • Consolidate multi-value business keys
  • Allows tracking of dimension history
  • Standardizes dimension tables
    • All are structured in the same way
      • Business Key
      • Surrogate key

Don’t put attributes and business keys in your fact table.

Design Practices

• Avoid smart keys
• Avoid production keys
• production may decide to reuse keys
• the company may acquire a competitor and thereby change the key building rules changed record, but deliberately not changed key

Types of Dimensions

Different dimensions types should be used in different scenarios.

Basic Dimension Types

• Standard Star dimension
  • Single table
  • Usually date
• Snowflake
  • More than one table in a cascading
• Parent Child (unbalanced hierarchy)
  • Self referencing surrogate to parent surrogate key relationship
  • Self referencing business key relationship
  • e.g. Org Chart, Chart of Accounts

Advanced Types

• Degenerate (Seldom used. Be careful)
  • Dimensions business key with no corresponding dimension table
  • Are embedded in the fact table
  • Typically use only when your Dim record count is about the same as the fact table record count.
  • Usually in line item oriented tables like sales tables
• Profile or Junk Dimensions
  • Convenient grouping of flags and attributes to get them out of the fact table into a useful dimensional framework.
  • Good for one-off lookups. Put them into a single dimension rather than a bunch of small dimensions
    • True/False type attributes
• Role Playing Dimension
  • A single dimension used for multiple purposes
  • Typical example is the date dimension or the geography dimension (outrigger)
• Time Dimension
  • Multiple calendars
    • Fiscal
    • Natural
  • Usually a single table

Tracking History

Industry best practices.

Changing Dimensions

• Slowly changing Dimensions
  • o – No change
    • Birthdate
  • 1 – Not interested in history (Updating a row/record)
  • 2 – Slow changes. Adds new row/record
  • 3 – Fast changes. Adds new column

• Rapidly changing Dimensions
  • Large dimensions
    • Limit type 2’s

Good example graphical slides of Type 1 versus Type 2

Getting There

• Understand dimension hierarchies and drill-paths
• Confirm historically tracked attribute req’s.
  • Don’t be afraid to push back a little
• Check source system data integrity, cleanliness
• Review current reports

This session focuses on the basic design patterns for building relational database structures for Business Intelligence applications on SQL Server. By laying the foundation to dimensional modeling, this session provides an overview of dimension modeling theory and the justification of dimension modeling by reviewing the core structures involved in data marts and data warehouses and contrasting these design techniques to other types of systems. If you are new to Business Intelligence or feel you need a refresher on dimensional modeling, this session is for you.

We are not talking about transactional design., this would be a poor design for BI.

Data Warehousing

• A relational database repository that contains current and historical enterprise-wide business data structured in a way that is optimized for data retrieval and enables business decisions.

Data Mart

R. Kimball -”a data mart is a flexible set of data, ideally based on the most atomic (granular) data possible to extract from operational source, and presented in a symmetric (dimensional) model that is resilient when faced with unexpected user queries”

• “in its most simplistic form a data mart represent data from a single business process” Business process= purchase order, store inventory, etc”

OLAP = On-line Analytical Processing

A reporting system designed to allow different flexible analysis in real time, on-line, with data structures designed for fast retrieval, with redundancy included to support performance.

Note: “On-line” doesn’t indicate data from on-line systems, rather on-the-fly

Business Intelligence

• Forrester definition: A process of transforming data into information and making it available to users in time to make a difference.
  • Focus on delivery of information to the user
  • May or may not be created at a corporate level

Don’t approach this with tunnel vision. Ensure you REALLY understand what the business user needs.

Three Type of Business Intelligence

• Strategic Business Intelligence (Strategic Decisions)
  • Who: strategic leaders
  • What: formulate strategy and monitor corporate performance
  • Examples: Balance scorecard, Strategic Planning
• Analytical Business Intelligence (They want to know Why. Love Excel)
  • Who: analysts, knowledge worker, controller
  • What: ad-hoc analysis
  • Examples: Financial and Sales Analysis, Customer Segmentation, Click stream analysis
• Operational Business Intelligence (Operational, parameterized reports)
  • Who: operational managers
  • What: execution of strategy against objectives
  • Examples: Budgeting, Sales forecasting

Example Questions to be answered for various BI types.

• Strategic Questions
  • Is the overall product margin meeting the planned targets?
  • Are we meeting our quarterly revenue objectives and what is the trend?
  • Are our vendor backorders affecting overall product sales?
• Analysis Questions
  • What factors affect order processing time?
  • How did each product line (or product) contribute to a district’s profit (or store) last quarter (or month, or year)?
  • Which products have the lowest Gross Margin Return on Inventory (GMROI)?
• Operational Questions
  • When did that order ship?
  • What was the revenue for a sales district last quarter?
  • What was the average inventory level for a product last year?

Strategy map. Visually see how an organization and KPI’s look.

What is Dimensional Modeling?

• The process and outcome of designing logical database schemas created to support OLAP and Data Warehousing solutions

Not about Normal Forms. Short, squatty tables, not tall, wide tables.

Transactional vs. Analytical

• Production/Transactional supports (HR and Financial Systems)
  • Granular transactions
  • Real time production systems
  • Current, changing data
• Business Intelligence/Data Warehousing supports (current and historical data)
  • Summarized queries
  • Consistent, heterogeneous data
  • Voluminous, historical, stable data
• Transactional and DW applications require different design and storage

Transactional (OLTP) is about speed, efficiency, normal forms, reduced redundancy.

AdventureWorks transactional example

AdeventureWorksDW for data warehouse example

Reporting Challenges with OLTP

• Schema doesn’t clearly call out subjects, objects, events, states…
  • Difficult to prepare reports and analysis views
  • Requires multiple joins
  • Indexes not optimized for reporting
• Models business process, not information (IN not OUT)
• Levels show only current state, history is not tracked

Data Warehousing, The Solution

• Schema designed with reporting and analysis in mind
• With redundant data, specially prepared for analysis, we can do more:
  • Prepare data over time
  • Prepare aggregates
  • Add data from other sources, not only OLTP
  • Sales value shows much more if we know also market capacity and our market share

You will create some redundant data. That’s okay and part of the point. Easy reporting and aggregation

Excellent introduction by Erik. I think I can see why he would be a good SQL mentor on the subject.

Dimensional Modeling

• Used by most contemporary BI solutions
  • “Right” mix of normalization and denormalization often called Dimensional Normalization
  • Some use for full data warehouse design
  • Others use for data mart designs
• Consists of two primary types of tables
  • Dimension tables
  • Fact tables

Getting the “right” mix of normalization and denormalization can be tricky and can be more art than science.

Dimensional Vs. Transactional

Dimension Tables

• Contain attributes related to business entities
  • Customers, vendors, employees
  • Products, materials, even invoices (attributes!)
  • Dates and sometimes time (hours, minutes, etc.)
• Often employ surrogate keys (newly created primary key of the table typically)
  • Defined within the dimensional model
  • Not the same as source system primary, alternate, or business keys
  • Usually an identity integer
• Not uncommon to have many, many columns (60,70,80 are okay)

Fact Tables

(The numbers that describe to quantitive data. Also called Measures)

• Contain numbers and other business metrics
  • Define the basic measures users want to analyze
  • Numbers are then aggregated according to related dimensions
  • (quantities, prices, counts)
• Fact tables contain dimension keys
  • Defines relationship between measures and dimensions using surrogate keys
• Typically narrow tables, but often very large (mostly numeric)

Star Schema Design

Entity diagram of the DW schema

• Fact table holds measures for events, levels and states
  • Provides the relationship between dimension table
  • Highly normalized structure
• Dimension tables track attributes such as subjects and objects
  • Star Schema dimension tables all connect directly to one or more fact tables
  • Star Schema dimension tables are highly denormalized to reduce joins

Snowflake Schema Design

• Snowflake schema has normalized dimensions
  • Cascading hierarchy of tables for a single dimension with several 1-M relationships
  • More complicated schema
  • Allows dimension to be used in fact tables with different grains
  • Often easier management of attributes

SQL Server 2008 can optimize joins between dimension and fact tables.

Why Dimensional Modeling?

• Logical model is easy to understand
• Optimized for performance
• Historical tracking of information