Presenter: Matt Masson
http://blogs.msdn.com/mattm/

Starting up a new project with SQL Server Integration Services? Save time and money by learning about how to avoid common issues that users hit when implementing their ETL solutions. Drawing on the most frequently asked questions in the MSDN Integration Services forum and some of the more interesting packages we’ve seen over the years, well highlight common mistakes, clear up some misconceptions, and provide best practices to help you efficiently and effectively use Integration Services.

How Does the Data Flow Work?

• In-memory ETL engine
  • Really ELT rather than ETL
• Metadata set at design time, not runtime
  • Allows pre-calculations of sizes, etc…
• Data is moved through the pipeline in buffers

Chose the Right Transforms

• Synchronous
  • 1 output row for every input row
• Asynchronous
  • Any changes to the buffer shape. Like a Union ALL
• Beware of blocking transforms like Sort and Aggregate
• Take advantage of your database engine
  • ORDER BY …
  • SELECT INTO …

Don’t do lots of sorts in SSIS, do them in the Source DB input. The DB server is much more efficient at this.

What are Configurations?

• Allow you to externalize settings
• Change behavior without modifying the package
• Multiple configuration sources
  • XML and SQL Server are the most common

Configurations can be added after project creation. Launch the configuration wizard. These are design time configurations.

Using /SET (Used with Runtime configurations)

• DTEXEC /SET
• Override values that don’t already have a configuration set
• Set connection strings
• Pass arguments to a package

Why Can’t I Share My Package?

• Package protection level
  • Encrypt {all|sensitive} with User Key
  • Encrypt {all|sensitive} with Password
  • Don’t Save Sensitive
  • Server Storage

How Do I Deploy My Packages?

Deployment – File System

• Deploy files using an MSI, or copied to a share
• XML configuration

Deployment – SQL Server (*Recommended*)

• Files are stored in MSDB
• Execution typically done with SQL Agent
  • Make use of Proxy Accounts
• Store configurations in SQL Server

Deployment – Package Store

• Deploys to the SSIS service
• Front end to File and SQL storage

What is the SSIS Service For?

• Not required for execution
  • Tracks running packages
  • Component cache
• Can manage multiple File or SQL locations
• Remote connections to the service will require DCOM security changes

How should I do Logging?

• Logging has to be enabled on the package
• Multiple logging destinations
  • SQL Server is most common
• Custom logging solutions

Package Execution Reports

• Available from Microsoft as a download for Reporting Services (SSRS)

How do I Handle Errors?

• Control Flow
  • Precedence constraints
  • Event Handlers
  • Events “bubble up” to parent
• Data Flow
  • Redirect error output

Using Expressions

• Set package properties
• Variable values
• Derived Column transform
• Conditional Split transform

Data Flow Expressions

• 32bit vs. 64bit
  • SSIS will install both 64bit and 32bit runtimes
  • BIDS is 32bit only
    • Only 32bit drivers will appear at design time
    • Use project setting for 64bit execution
  • Use configurations to switch providers
• Other Stuff
  • There are no MS 64bit Excel drivers
    • Use 32bit or 3rdparty
  • SSIS is an extensible platform

Resources

• SSIS Team Blog
  • http://blogs.msdn.com/mattm/
• Community Samples
  • http://www.codeplex.com/SQLSrvIntegrationSrv
• MSDN Portal Page
  • http://msdn.microsoft.com/en-us/sqlserver/cc511477.aspx

Is your customer looking for drag and drop reports or capabilities inside of Excel. SQL Server Analysis Services (SSAS) is the answer for you then. You’ll be amazed how quickly you can develop sophisticated reports after watching the basics of this session.

SSAS is its own server. It is not part of the SQL Server service.

• IIS
• XMLA between server and client

MDX – SQL Language for Cubes
BIDS – Building the Cubes
SSMS – Managing SSAS

Analysis Services

What are cubes?

• What are measures? (See Erik Veerman Dimensional modeling session)
  • The value you are measuring, e.g. the count of users with blue eyes, from WA state, etc.
  • The question you are trying to ask
• What are measure groups?
  • Groupings of these measures within your cube
  • Typically define them by your business problem (e.g. Sales, HR, etc.)
• Cubes are a grouping of measure groups.

Analysis Services Dimensions

What are dimensions and hierarchies? (See Erik Veerman Dimensional modeling session)

• What is a dimension?
  • It is what you want to categorize or pivot against
  • e.g. How many users are from WA state?
  • The dimension is geography, “WA” is the member
• •What is a hierarchy?
  • How do you want to organize the members of your dimension
  • e.g. Geography has numerous levels such as country, state, and city
  • Hierarchy is the organization of those levels such as City ->State ->Country

Concrete examples like these are helpful for my learning and understanding. Thanks!

User Interfaces

• Query language is MDX
• BIDS for developers
• Excel for most users is adequate or Reporting Services
• SharePoint for web users
• 3rdparty applications like Proclarity or Panorama
• Controls you can purchase and build into your own apps

Create User defined hierarchies.

• Remember that in 2008 the hierarchy can be thought of as reversed.
• Looks graphically like date -> month -> qtr -> year
  • Means: to get to a date you must go through year, qtr, month, date

Deal with problematic keys.

Most end users will use Excel. 2007 provides additional features.

This talk describes the new MERGE statement and Change Data Capture (CDC) feature introduced in SQL Server 2008, and how to take advantage of them in SQL Server Integration Services. Well cover the traditional ways of doing incremental data loads, and see how these new key features make the process a whole lot easier. If you’re interested in learning best practices for this new SQL Server 2008 functionality, you won’t want to miss this session!

CDC and Merge are new in SQL Server 2008

Also 2008 has improved SCD processing

Options to Capture Change

• Audit columns – created via triggers
• Log Scraping
• Full DB Diff compare

(New Options) Change Data Capture in SQL Server

• Changes captured from the log asynchronously
  • Really just integrated log scraping
• Tells you what changed at the source
• Enabled per table or DB
• CDC API’s provide access to change data

CDC Components

• CDC Tables
• Capture Job
• Cleanup Job
• API

Change Tracking

• Captures that a row was changed, but not the changed data
• Synchronous capture mechanism
• Less overhead for applications that do not require the historical information

Turn CDC on:

• sp_cdc_enable_db
• sp_cdc_enable_table
• LSN = Log Sequence Number
• Sp_cdc_generate_wrapper_function
  • Creates a custom function DDL for reading CDC data

Loading SSIS Package

• Source from the CDC Wrapper function
• Split on CDC operation
  • Insert
  • Update
  • Delete

Lookup Changes in SQL 2008

• Explicit cache modes
  • Full
  • Partial
  • None
• Cache connection manager

External Reference Data

• Joins can work but can be difficult to manage
• Instead, use the SSIS Lookup transform.

The MERGE Statement

• Single statement can deal with Inserts, Updates and Deletes all at once
• Use Cases
  • Conditionally insert or update rows in a target table
  • Synchronize two tables
• Built to perform

Using MERGE in SSIS (Execute SQL Tasks in SSIS)

• Stage
  • Store data in a temp table
• Optimize
  • Create indexes on join cols
  • Provides addtl perf benefits
• Execute
  • Run the MERGE SQL statement

When doing the insert, you must drop the FK constraints in the destination table.

Taking Advantage of MERGE

• Consume CDC data
• Update Dimensions
• Alternative to the Slowly Changing Dimension Wizard

Additional Resources

• SSIS Team Blog
  • http://blogs.msdn.com/mattm
• Comparing Change Data Capture and Change Tracking
  • http://msdn.microsoft.com/en-us/library/cc280519.aspx
• Optimizing MERGE Statement Performance
  • http://msdn.microsoft.com/en-us/library/cc879317.aspx
• MERGE Destination
  • http://www.codeplex.com/SQLSrvIntegrationSrv/Release/ProjectReleases.aspx?ReleaseId=19048

• Slipstream – You are now able to integrate the base installation with service packs (or Hotfixes) and install in a single step.
• Service Pack Uninstall – You are now able to uninstall only the Service Pack (without removing the whole instance)
• Report Builder 2.0 Click Once capability

Beware, known issues to check BEFORE installing. 

This series of articles present our findings and processes as we performed them. These processes may not be optimal nor preferred in other installations, but they represent the best case for our particular assignment. As such, we certainly welcome feedback.

Part I – Building the Test Environment

Before performing the upgrade we wanted to ensure that our processes would work. Additionally, we have a deference for knowing what roadblocks we might run into ahead of time. When dealing with databases, most times I am going to bet that people or system will be impacted by the DB services being unavailable. Therefore, time is important. Testing before hand can shrink the outage windows and help keep customers happier.

As such, we decided to replicate the key pieces of the infrastructure into a virtual environment. We did this using the VMWare ESXi platform which we are familiar with from our own server infrastructure environment. If you are unfamiliar with ESXi, it is a free version of the VMWare virtual infrastructure product line. You do have to register with the VMWare.com website to download and obtain a license key, but you probably want to be able to review the knowledge base and community forums anyhow. For an SMB like us, it is a great virtualization platform.

Testing Environment ESXi Server Setup

First, we needed the VMWare ESXi server. So we downloaded the latest ESXi bootable ISO from VMWare.com, burned the disc and booted our test server up. We won’t go into all the setup screens, but you can find ESXi installation instructions in a variety of places including this one. Essentially we just took the default installation, rebooted and then configured the management interface with a static IP, mask and gateway. Finally we setup a root password.

Converting or Moving Production to Test

Now whether your production servers are already virtual machines on VMWare or are real, physical servers you are going to need to create an exact copy of them for your test environment. VMWare has a great little, free, tool that allows you to do this. Although they have bigger, better (not free) tools to do this, our needs are such that the free tools work. The VMWare Converter tool is the key. Download it from VMWare.com. Then, we simply install the converter application on the servers that we need to replicate to test. [Note: Just found out that version 4.0 of VMWare converter was released on 2/12/2009. We were using version 3.0]

The conversion process takes the current server and converts it to a compatible VM (virtual machine) on the test server. This include all O/S, network, disk, CPU and memory settings. One note, you’ll want to ensure you have enough disk space on the test ESXi server to accommodate all the disks you are converting for all the VM’s.

[Note: We learned the hard way, having to rebuild the array on the test ESXi server from RAID 5 to RAID 0 in order to get the last bit of disk space we needed. Lesson:  sum all your disk space needs for all the servers you will be converting, ahead of time.]

The process is quite simple but may take numerous hours depending on the data needing to be migrated. So let’s walk through one example of converting one of our production machines in the following video.

1) This is the VMWare converter. Start the converter and choose to start a new conversion.

2) This is the new conversion project wizard splash screen. Click Next.

3) We will be stepping through setting up the source for the conversion. Click Next.

4) Source type will be physical machine regardless if source is physical or another VMWare VM.

5) Choose "local machine" and the converter will convert the current machine using the current users credentials.

6) Choose the disks to be converted. Be sure you include the O/S drive and any data drives you might need.

7) We will define the destination VMWare ESXi server.

8) Our destination type will be VMWare Infrastructure Virtual Machine since we want to convert his server to a VM that will run on VMWare ESXi.

9) This is the destination VMWare ESXi server name or IP and the VMWare login credentials.

10) Type the name you wish to give the new VM when it is converted. This is not the O/S name but the VM name that shows up in the VM inventory.

11) The datastore where you want the converted VM files to be stored on the VMWare ESXi server.

12) The number of NICs that will be created in the destination VM.

13) Place checkmarks on these two options to ignore hibernation and page files and install the VMare tools on the destination VM.

14) Do you want to VM to automatically power on after the conversion is complete? Click finish to start the conversion.

When the conversion completes, pat yourself on the back as you just converted your first server. Now just repeat this process for each of the servers you need to complete your test environment.

Just a quick observation and statement of the obvious. The length of time that a conversion takes is primarily dependent upon two factors, network and disk space. On the network side, I would highly recommend using Gigabit networks for converting larger servers. We found a many fold reduction in conversion time simply by moving from 100 megabit to Gigabit networks. Additionally, the amount of used disk space on the source server will also play a role in the conversion time. The more disk space in use on the source server that is needing to be converted, the more time the conversion will take.

Keep in mind I said “used” space. Unusued space on the disks being converted seems to play very little into the conversion times. As such, clean out your temp files and such before doing a conversion. It might save you some time when converting.

Configuring the Isolated Test Network

Doing this allows us to connect to the management network on the ESXi. Once installed, point your browser to the IP address of the ESXi server via https and you can download the VMWare Virtual Infrastructure client. (https://192.168.1.134 or whatever IP you gave your ESXi server). This client application currently only runs on Windows, sorry Linux folks. This is what you will use to manage your virtual clients.

In our case,we wanted the virtual servers to be able to connect to themselves but not the “real” LAN so as not to cause duplicate IP and Active Directory issues. To do this, we created an isolated virtual network. From the client application, we created a second virtual switch. When we created the virtual switch, we did not link a physical network card to the virtual switch definition. This essential makes a virtual switch that can talk only to other devices on the same virtual switch.This is a plus for testing of an isolated network environment where you do not want the test servers to talk to your LAN.  On the downside, your servers on this new virtual switch can’t get to the Internet either. (at least not without some other fancy networking tricks).

Here’s some screenshots of the process.

1) Create new virtual switch.

2) Select the Create a Virtual Switch option and click Next.

3) Give your new network a name and click Next.

Again, congratulations. You just finished creating you own host only virtual network switch for isolated server testing. Now you need to move your VM’s to use this new network. You’ll need to edit each VM’s setting and change the network that the NIC connects to be your newly create host only network.

1) Edit the VM settings.

2) Change the network to your host only network you created above. Now Ok to save.

That’s the majority of it. Again, we used this setup to convert 5 of our production servers including SQL 2000, SQL 2005, Terminal Services, File Services and Domain Controller all to this isolated test environment. Then we made a snapshot of each converted server which allowed us to go back to the fresh conversion state as many times as we wanted.

From here, we embarked upon our SQL Server 2008 upgrade testing.

Blessings.

Presentation and slide content by IT Mentors

1. SSRS 2008 Architecture/Runtime
2. Configuration
3. New Controls
a. Tablix
b. Gauges and Indicators
4. Deployment

### Benefits of Eliminating IIS from the SSRS Architecture
* Easier configuration
* IIS settings for other applications impact SSRS
* Better resource management
* IIS designed for static or dynamic HTML pages, not report execution for large reports to many concurrent users.
* Memory management easier to implement outside of IIS
* Consolidation of two services into one
* Communication process between services eliminated
* More efficient allocation of memory across server
* Elimination of deployment obstacles
* SQL Server DBAs lacking IIS skills
* IT policies prevent IIS and SQL Server on same server

### Architecture
* HTTP Listener
* Uses HTTP.SYS directly from the operating system
* Accepts requests directed to the configured URL and port
* Authentication Layer
* Confirms identity associated with request
* Supports Windows, Basic, or anonymous Access with a custom authentication
* Application Domains
* Provide a front-end component with Report Manager
* Process reports interactively with Report Server Web service
* Support scheduled operations with background processing
* Application Domain Management
* Replaces IIS functionality

Notes: Speaker does not discuss the complications that running IIS and SSRS 2008 on the same server. Instead he talks about how they both can run on the same server. Might want to be sure you understand the complications that can arise from both on the same server, especially when it comes to SSL needs.

### Configuration
* Memory Management
* Disabling Features

Notes: Speaker states the obvious about the importance of these items. Disabling features = reducing surface area.

### Memory Management
* Memory Broker monitors and responds to memory pressure fluctuations
* Memory events trigger switch from RAM to file system cache as needed
* Configurable thresholds determine report server response to memory pressure changes

Notes: Good simplistic explanation of these by the speaker. Might want to research a bit more if you really need to tweak memory optimization for peak performance.

* Low Memory Pressure
* Current requests continue
* New Requests accepted
* Background low priority
* Medium Memory Pressure
* Current requests continue
* New requests _might_ be accepted
* Memory allocations reduced for all applications
* Background items get the largest reduction
* High Memory Pressure
* Current requests continue
* No new requests accepted

Notes: Good explanation and slide for this concept.

### Configure Memory Settings

Notes: Speaker shows you the what to change in the config file but neglects to say which config files is being changed and where it is located. The correlating MSDN article on this topic is located here.

Information on the RSReportServer.config file can also be found on MSDN.

### Disabling Features
* Report Manager
* RSReportServer.config
* IsReportManagerEnabled
* On Demand Processing
* RSReportServer.config
* IsWebServiceEnabled
* Scheduled Events and Report Delivery
* RSReportServer.config
* IsSchedulingService
* IsNotificationService
* IsEventService
* Report Builder
* SQL Server Management Studio
* Server Properties
* Report Server Windows Service
* SQL Server Configuration Manager

Notes: One might want to consider the “Why?” on each of these to determine appropriate use cases for enabling or disabling. Now the speaker shows where the config file is.

Demos were done for memory configuration settings and the Tablix control.

Lots of good gauges and indicators available thanks to the new Dundas obtained controls.

Finally discussion about the rs command line tool for deployment.

First I needed to check and see if there is a supported upgrade path from SQL Server 2008 Enterprise Edition Evaluation to the full licensed product. Indeed there is according to this MSDN article. That may make an in place test to production scenario plausible.

Another MSDN article provide a bunch of food for thought regarding the upgrading of Reporting Services.

In our case, we are needing to upgrade and move servers. From what I have read so far, I am thinking that it would be best to upgrade the existing SSRS 2005 server to 2008 and then move Reporting Services to the new server. This MSDN Article refers to the migration process.

Apparently the Reporting Services database creates stored procedures during its creation that refer to the actual Reporting Services DB name. Because of this, you cannot easily change the Reporting Services database name when moving SSRS to another server.

“You cannot rename the report server database. The identity of the database is recorded in report server stored procedures when the database is created. Renaming either the report server primary or temporary databases will cause errors to occur when the procedures run, invalidating your report server installation.”

Another quirk to our installation is that we use SSL for report delivery. This TechNet article describes the SSL certificate configuration process in SSRS 2008. Keep in mind that since SSRS no longer runs on IIS, having IIS installed on the server running SSRS may complicate life. Specifically when running SSL, also running IIS causes a service dependency.

Well, take a look at the partner site at vConferenceOnline.com. They’ve got a great setup going. Essentially the pre-tape each session in the studio. That way the quality is controlled and ends up great. (Next one is slated to be in HD). Then they have created an asp based solution (Gotta wonder why ASP?  ) that handles making the appropriate “rooms” available at the appropriate times during the actual virtual conference. All from the comfort of home, the office, the library or even your favorite coffee shop.

This time, they ran 3 rooms simultaneously for the Business Intelligence conference I attended. The format allows the first half of the day to have these sessions running. The second half of the day the re-run the first half of the day. This way you get a chance to see secondary sessions that may have been running at the same time in the morning.

There were actually 4 conferences going on at the same time. A SQL Server, Business Intelligence, .Net Developer and MS Sharepoint were all available. Each conference was a separate registration fee but for the price you cannot beat it.

Well, take a look at the partner site at vConferenceOnline.com. They’ve got a great setup going. Essentially the pre-tape each session in the studio. That way the quality is controlled and ends up great. (Next one is slated to be in HD). Then they have created an asp based solution (Gotta wonder why ASP?  ) that handles making the appropriate “rooms” available at the appropriate times during the actual virtual conference.

The video quality is very good and continues to improve. The content distribution network they are using seems to be quite well suited for such multimedia distribution. Note, you’re going to need some decent bandwidth to get the best quality.

There are set times for exhibitor booth visits which has gone through a great overhaul this time. On demand vendor videos, downloads and of course vendor chat rooms.

Here’s the best part. 3 days for one conference was $100, actually $10 less since I’ve been to the previous two. That’s a BARGAIN when you consider typical conference costs, meals, lodging, travel and more. You certainly do miss out on the personal interaction however they are working on ideas for that. Currently during each presentation, appropriate chat rooms are available for participants and the speaker.

So, yup, I’ll continue to go as long as the quality of the experience and information remains high. Kudos to those who helped put it on.

Kudos Stephen Wynkoop, Chris Shaw and the rest of the gang.