In this session, you’ll learn how to load a typical data warehouse in SSIS efficiently. You’ll start by seeing some of the strengths and weaknesses of the Slowly Changing Dimension (SCD) Wizard in SSIS and how you can get around some of the weaknesses including your own home-brewed solution. You’ll then see how to load a fact table using SSIS and how to make the common components scale.

Dimensional Modeling (See Dimensional Modeling 101 Notes)

• Data Separated into fact and dimension tables
• Dimension tables answer the pivot or where clause
  • Make as wide and descriptive as possible
  • Surrogate keys operate as unique ID for each row
  • Keep surrogate keys as small as possible
• Fact tables answer the what questions or select statement
  • Intersect all dimension tables
  • Surrogate keys from each dimension in this table
  • Measures are the “what” like Price, Quantity, Duration

The challenge is how you move the data from the OLTP (Relational) DB into the Data Warehouse.

Discusses SCD (Slowly Changing Data) Dimension types (Already here)

Don’t make everything Type two or your DB will bloat significantly. Also your reports would be more difficult to write.

Probably want to fix NULL values to be something. Use a Derived Column transform. Makes Nulls to be something like 0, Unknown, etc…

SCD Wizard Strengths

• SSIS transform that creates many other transforms conditionally
• Reduces design time of SCD load by 80%-90% to minutes per dimension
• Can be customized easily
• Compares differences between source and destination to find changes and new records
• Outputs:
• Type 0,1,2 update
• Inferred members
• New rows
• Duplicate rows

Historical Attribute Options – How do you want to set the expiration of historical records.

Problem with SCD Wizard is that any time you go back and change the configuration, all the output logic below it gets re-written. You lose what you created.

SCD Wizard Weaknesses

• Scalability -Generally up to about 50,000 records into the transform but varies based on number of updates
• Maintainability -After you customize, rerunning the wizard recreates all the transforms
• Uses OLE DB Command transforms for updates is row-level. Creates scalability issue here if lots of updates.

Making Your Own SCD Wizard

• Can use a Merge Join or Lookup Transform
  • If no match found, it is an insert (Ignore Errors)
• Lookup Transform will scale better than Merge Join but lacks parameterization
• Add a Conditional Split transform after Lookup to direct to insert, duplicate or update path

Additional Scalability

• Watch your Lookup Transformation for scalability issues (don’t cache too much!)
  • Potentially cache only the last 1 years worth of data with Partial Caching
  • Only cache columns needed
• Additional scalability can be reached by landing updates into a staging table
  • Then set-based update with an Execute SQL task.
• Checksum Transform can be used to detect changes across many columns
  • Or HASHBYTES T-SQL statement

Inferred Members

• Created during the fact load
  • A new Dim record is created using the value of “unknown” or NULL as a placeholder
  • The record is flagged as an inferred member

Slowly Changing Dimension Wizard

• SSIS transform that creates many other transforms conditionally
• Handles:
• Type 0 (fixed attribute)
• Type 1 (changing attribute)
• Type 2 (historical attribute)
• Inferred members
• Typically can address 80% of the business scenarios

SCD Wizard Strengths

• SSIS transform that creates many other transforms conditionally
• Reduces design time of SCD load by 80%-90% to minutes per dimension
• Can be customized easily
• Compares differences between source and destination to find changes and new records
• Outputs:
• Type 0,1,2 update
• Inferred members
• New rows
• Duplicate rows

SCD Wizard Weaknesses

• Scalability -Generally up to about 50,000 records into the transform but varies based on number of updates
• Maintainability -After you customize, rerunning the wizard recreates all the transforms
• Uses OLE DB Command transforms for updates is row-level. Creates scalability issue here if lots of updates.

Lookup Transform – Lookup source against target dimension table. Select ALL available lookup fields and alias at TARGET_ so you can match against them. Link by primary key. Ignore the failure of matches.

Then use a conditional split – If target PK is NULL it is a new record, else Update.

“Pretty much a pain in the butt to write.” – Brian Knight in reference to the else UPDATE piece from the conditional split.

Ugh…no transcript and this info is not in the slides. ARGH! Can’t remember it all.

Brian uses the Checksum source vs. Checksum destination to deal with the UPDATE referred to above, but not guaranteed to be unique.

Hashbytes on the other hand is more unique if you wish. But it does not work on numeric fields. Need to cast them.

Fact Table Loads

• Series of Lookup Transforms
  • In Type 2 Dimensions add WHERE EndDate IS NOT NULL
• Measures created Derived Column Transforms
• Aggregate transform to roll up the grain
• Lookup failure would create an inferred member or set to unknown

DMV – Dynamic Management Views (and functions)

• Returns server state information that can be used to monitor the health of a server instance, diagnose problems, and tune performance.* MSFT – 2008 BOL

DMV Categories (Not all are going to be in this session)

• Change Data Capture
• Query Notifications
• Common Language Runtime
• Replication
• Database Mirroring
• Resource Governor
• Database Related
• Service Broker
• Execution
• SQL Server Extended Events
• Full-Text Search
• SQL Server Operating System
• Index
• Transaction
• I/O
• Security
• Object

DMV’s help with the ever growing infrastructure and complexities. Allow you to monitor and manage your environment.

Key DMV’s

• Sys.dm_os_sys_info
  • General Server Info
• Sys.dm_exec_requests
  • Sp_who(2)
• Sys.dm_db_index_operational_stats
  • Sp_lock
• Sys.dm_db_index_usage_stats
  • Base DMV for Index Analysis and Recommendations
• Msdb.dbo.backupset
  • Base table for backup history
• Sys.dm_exec_cached_plans
  • Shows query plans that are cached by SQL Server
  • For query plans, this DMV maps to the syscacheobjects table in SQL Server 2000
• Sys.dm_exec_query_stats
  • Performance statistics for cached query plans
  • Top 10
• Sys.dm_io_virtual_file_stats
  • I/O stats for data and log files
• Sys.dm_os_memory_pools
  • Monitors Cache memory
• Sys.dm_exec_sql_text
  • Returns the text of the SQL batch that is identified by the specified sql_handle.
  • This table-valued function replaces the system function fn_get_sql
  • Obtained from:
    • Sys.dm_exec_query_stats
    • Sys.dm_exec_requests
    • Sys.dm_exec_cursors
    • Sys.dm_exec_xml_handles
    • Sys.dm_exec_query_memory_grants
    • Sys.dm_exec_connections

“Use it early, use it often” – Jeremy 

Love it that every technical person presenting a demo has hit at least one minor glitch. Let’s me know it’s not just me. Ha!

Resources

• http://www.sswug.org
• Local User Group
• Books On Line
  • SQL Server 2008 Performance Studio
    • https://msevents.microsoft.com/cui/WebCastEventDetails.aspx?culture=en-US&EventID=1032349947&CountryCode=US
    • http://www.microsoft.com/midsizebusiness/business-goals/business-operations/

Seems that when comparing the HTML output to the client between logged in and logged out, the GMAP script is missing from the logged out HTML.

<script src="http://maps.google.com/maps?file=api&v=2.115&key=BeSureToGetYourOwnCodeBecauseIRemovedMineForThisPost=en" type="text/javascript">

On a hunch, I disabled block caching in the Admin –> Performance settings. Voila! Problem solved. So this appears to be a block caching issue. For now, I’ve just turned block caching off and will have to wait for time to dig into it more.

Just thought I’d post it here for posterity sake.

Blessings.

BTW Details are:
Drupal: 6.6
GMAP_Location module: 6.x-1.0-rc2
Location module: 6.x-3.x-dev