Enabling and reviewing My Reports

As described in Part 1 the My Reports folder needs to be enabled in order to use the folder or display it in the Open Report dialogue. The RC0 version had a documentation bug which has been rectified (https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=366413)

Getting ready

In order to enable the My Reports folder you need to carry out a few tasks. This will require authentication and working with the SQL Server Management Studio. These tasks are listed here:

The Server Properties window is displayed with a navigation list on the left consisting of the following:

In the General page the name, version, edition, authentication mode, and URL of Reporting Service is displayed. Download of an ActiveX Client Print control is enabled by default. In order to work with Report Builder effectively and provide a My Reports folder for each user, you need to place a check mark for the check box Enable a My Reports folder for each user. The My Reports feature has been turned on as shown in the next screenshot.

In the Execution page there is choice for report timeout execution, with the default set such that the report execution expires after 1800 seconds.

In the History page there is choice between keeping an unlimited number of snapshots in the report history (default) or to limit the copies allowing you to specify how many to be kept.

In the Logging page, report execution logging is enabled and the log entries older than 60 days are removed by default. This can be changed if desired.

In the Security page, both Windows integrated security for report data sources and ad hoc report executions are enabled by default.

The Advanced page shows several more items including the ones described thus far as shown in the next figure.

The Advanced page is displayed as shown:

The description of everything that a user with the assignment My Reports role can do is as follows:

May publish reports and linked reports, manage folders, reports, and resources in a users My Reports folder.

Report Builder 2.0 is displayed.

The Open Report dialogue appears as shown. When the report Server is offline, the default location is My Documents, like Microsoft products Excel and MS Access.

The Report server that is active should get displayed here as shown:

All the folders and files on the server become accessible as shown:

Verify that a My Reports folder is created for the user (current user).

1. Make sure the Report Server has started.
2. Make sure you have adequate permissions to access the Servers.
3. Open the Microsoft SQL Server Management Studio as described previously.
4. Connect to the Reporting Services after making sure you have started the Reporting Services.
5. Right-click the Report Server node.
   • General
   • Execution
   • History
   • Logging
   • Security
   • Advanced
6. In the General page enable the My Reports feature by placing a check mark.
7. Click on the Advanced list item in the left.
8. Now expand the Security node of Reporting Services and you will see that the My Reports role is present in the list of roles as shown. This is also added to the ReportServer database.
9. Now bring up Report Builder 2.0 by clicking Start | All Programs | Microsoft SQL Server 2008 Report Builder | Report Builder 2.0.
10. Click on Office Button | Open.
11. Choose the Recent sites and Servers.
12. Highlight the Server URL and click Open.
13. Open the Report Manager by providing its URL address.

There could be slight differences in the look of the interface depending on whether you are using the RTM or the final version of SQL Server 2008 Enterprise edition.

A step-by-step guide to getting the most of Microsoft SQL Server Reporting Services 2008   Everything you need to create and deliver data-rich reports with SQL Server 2008 Reporting Services as quickly as possible Packed with hands-on-examples to learn and improve your skills Connect and report from databases, spreadsheets, XML Data, and more No experience of SQL Server Reporting Services required http://www.packtpub.com/learning-sql-server-2008-reporting-services/book

Modifying a basic report

In this exercise, the report created will be modified to illustrate the formatting, layout, and other capabilities built into the Report Builder.

Getting ready

This hands-on will be using a MS Access report that was imported using Visual Studio and hosted on the Report Server. The MS Access report will be modified to use the new Report Items in Report Builder 2.0.

Follow the steps

You will be carrying out the following steps:

1. Open Report Builder and open the ByOrders.rdl report.
2. Review the imported MS Access report.
3. Modify the properties.

Open Report Builder and open the ByOrders.rdl report

You will be accessing the ByOrders.rdl file from the Report Builder in order to modify it in the Report Builder. The steps are listed here:

The ByOrders report gets displayed in the Report Builder as shown:

1. Start Report Builder from its shortcut.
2. Click the Office button and in the drop-down window click on Open. The Open dialogue is displayed.
3. Click on Recent Sites and Servers in the left navigation area. The Report Servers' URL is displayed.
4. Highlight the Report Server URL and click on the Open button.
5. Click on the MS Access folder and click on the Open button.
6. Highlight the report ByOrders and click on the Open button.

Review the imported MS Access report

The dataset for the report is DataSet1 on the left and the report body is in the design area. The various report items and their data binding will be examined for one such control, the CompanyName. It will be instructive to study the others as well. In order to review the report we will follow these steps:

The properties of CompanyName are displayed as shown. CompanyName is inside a container textbox inside the Tablix and it is a place holder. You should also notice the large square bracket on the left ranging three rows. This is the grouping symbol.

This opens the Placeholder's Properties page as shown:

You have added the HTML tags <u> and </u> on either side of the original CompanyName data that came from the dataset. Now it is slightly more than the data and in the design view (CompanyName) it is replaced by this expression. A placeholder is the holding place of an expression. You can make a textbox into a placeholder by designating the textbox to hold an expression. For example there are two place holders for the time Now () and the expression ="Page" & Globals.PageNumber & " of "& Globals.TotalPages.

The report gets processed and you may need to provide the login for this report. The username is Admin and there is no password.

The report is displayed in the Report Builder as shown:

The Address field is highlighted in the report as shown:

You can see that this represents the detail (the data that comes in each of the rows). It is also a placeholder and its value is that of "Address". You can apply Rich Text formatting by using HTML tags for textboxes as well as place-holder values.

Note that only certain HTML tags and CSS attributes are supported. For the supported HTML tags refer to http://msdn.microsoft.com/en-us/library/dd207048.aspx. For an example of CSS rendering, refer to http://hodentek.blogspot.com/2009/01/can-you-use-css-style-attributes-in.html.

Both vertical and horizontal movements of objects can be very smooth and can be changed in Points as shown:

1. Highlight CompanyName and right–click on it.
2. Click on Placeholder Properties….
3. Change the Markup type to HTML – Interpret HTML tags as styles.
4. Click on the fx symbol (which opens an Expression window) along  the Value.
5. In the Expression window that is displayed, modify the expression as shown in the following screenshot:
6. Click on the OK button on the Expression window as well as the Placeholder Properties' window.
7. Click on the Run button in the Home menu.
8. Click on the View Report button.
9. Click on the Address field in the report design.
10. Change Report title ByOrders to Orders by editing the textbox.
11. Extend the length of the <<Expr>> directly below the report title to accommodate a longer string.
12. Make the EmployeeID feld left aligned as well as formatted. Set the font weight Bold.
13. Click on the table that has the Address, City, and so on. Extend it to the right by dragging the table handles so that the Required Date can be fully displayed.
14. Rearrange the positions of objects and the size of textboxes to fully display the data.

Highlighted objects can be expanded and moved using the Ctrl or Shift keys together with the arrow keys.

The modified report design is shown in the following screenshot:

Summary

The Report Builder 2.0 tool is described in full, starting from the top to the bottom of this interface. Creating reports with this tool is described using an existing report and modifying it as well as creating a new report from data. Embedding charts and gauges are also described. Some of the interactive and rich text features are  also discussed.

Report Builder 1.0 is briefly mentioned but its utility for reports models created using VS 2008 or BIDS did not warrant an example due to current limitations. It may be possible to use this tool with Report Models created with an earlier version of SQL Server. One of the main features of Report Builder 2.0 is its ability to create Ad Hoc reports. The reader may notice slight differences in the look of the interfaces slightly at variance depending on the version of the SQL Server 2008 and Report Builder 2.0 used.

The Microsoft SQL Server 2008 Reporting Services Report Builder 2.0 tool can be installed from a standalone installer available at this Microsoft site, http://download.microsoft.com/download/a/f/6/af64f194-8b7e-4118-b040-4c515a7dbc46/ReportBuilder.msi. The same file is also available from a collection of download files when you access the Microsoft SQL Server 2008 Feature Pack, October 2008 at http://www.microsoft.com/downloads/details.aspx?FamilyId=228DE03F-3B5A-428A-923F-58A033D316E1&displaylang=en.

Report Builder overview

In the present version of SQL Server 2008 [Enterprise Evaluation edition] there  are two Report Builders available. Report Builder 1.0, which has remained as a program that can be launched from the Report Manager, and the new Report Builder 2.0, which is a stand alone report authoring tool that needs to be independently launched.

Although Report Builder 1.0 can access Report Models built with Visual Studio 2008 and the Report Manager, it cannot be used to create reports using those models. It also does not work with Reports generated by Visual Studio 2008/BIDS/Report Builder 2.0. The errors can be summarized as follows:

Specifying credentials in a URL is not supported

System.IO.StreamReader: The Report element was not found

MemoryStream length must be non-negative and less than 2^31-1-origin. Parameter name: offset; Remote GDI stream version: ?. Expected version: 11.0.1

• When you try to access the Report Server 2008 from the link provided on the Report Builder 1.0 interface you get the following error message:
• When you try to open a report created using VS2008/BIDS/ReportBuilder2.0 using the Open Report… and Open File… navigational items in Report Builder 1.0 you get the following error message:
• Report Builder 1.0 allows you to access Report Models created with VS2008/BIDS/Report Manager and even allows you create a report in design view but this report cannot be processed on the Report Server. If you try to do so, you get the following error message:

In this article the Report Builder 2.0 interface will be described along with the new features that are incorporated into this version. Report Builder 2.0 is admirably suited to address all items in the Report Definition Language of 2008.

One of the important features of Report Builder 2.0 is the empowerment it provides business users to create ad hoc reports using the Report Models built on the databases they use.

In this article you will be learning mostly about the Report Builder 2.0  interface details and working with it to create reports or modify them. It may be noted that Report Builder generates 2008 compliant RDL files as described in http://download.microsoft.com/download/6/5/7/6575f1c8-4607-48d2-941d-c69622e11c32/RDL_spec_08.pdf and therefore, cannot work with reports generated using 2005 technology.

Report Builder 2.0 user interface description

Report Builder is a report authoring tool and the basic procedure for authoring a report consists of the following steps:

• Report planning
• Connecting to a source of data
• Extracting a dataset from source
• Designing the report and data binding
• Previewing the report

Although deploying the report is not included in the above, Report Builder can deploy the report as well. It is not always necessary to deploy a completed report, as any part of a report definition file can be deployed. This makes modifying a report on the server very flexible.

In the following sections, the various parts of the Report Builder interface will be described starting at the very top and going to the bottom of the interface

The menu for file operations

Report Builder 2.0 can be accessed from Start | All Programs | Microsoft SQL Server 2008 Report Builder | Report Builder 2.0.

This brings up the Report Builder Interface 2.0 as shown with the design area containing two icons: Table or Matrix and Chart. Each of these will launch a  related wizard which will step you through the various tasks. The Report Builder 2.0 interface is very similar to Office 2007. More than one instance of Report Builder can be launched.

At the very top of the following screen shown you have the undo and redo controls as well as a save icon.

When you click on the save icon the Save as Report window gets displayed as shown. Here you provide a name for the report. The default save extension is  *.rdl and it will be saved to the report server. It may also be persisted to a folder on your machine.

Clicking on the Office Button (top left) opens a drop-down window shown in the following screenshot:

In this window, you can carry out a number of tasks such as creating a new report, opening an existing report, saving a report, and saving a report with a different name.

The Save button saves it to the default location seen earlier and Save as invokes the same window to save the report with a different name as seen earlier displying the report server instance as the Save to location.

The Recent Documents pane shows the more recent reports created with this tool. New allows you to create a new report. When you click on Open, the following Open Report window gets displayed with the default location http://Hodentek2:8080/ReportServer_SANGAM/My Reports. You will also notice the message: This folder is not available because the My Reports feature is not enabled on the computer. Also the Open Reports window allows you look for reports with the extension .rdl.

Therefore, unless the My Reports feature is enabled, this window is unusable. This is supposed to be possible from Report Manager but there are no controls in Report Manager that would do this. An alternative was suggested by one of the MSDN forum moderators (see http://social.msdn.microsoft.com/forums/en-US/sqlreportingservices/thread/6c695160-29e8-4185-be6d-5fe027a6975c/). Hands-on exercise (Part 2) will describe how you may enable My Reports. The idea of My Reports is similar to My Documents where each user can keep his reports.

When the Options button (in the previous screenshot) is clicked it opens the window Report Builder Options window with two tabbed pages Settings and Resource shown as follows:

Here you can view, as well as modify, Report Builder settings. The defaults are more than adequate to work with the examples in this book.

Clicking on the Resources button brings up this interesting window which enables you to interact with Microsoft regarding SSRS activities, concerns, community, and so on. If you are serious about Reporting Services, these are very valuable links. The About button when clicked can provide you with Report Builder version information.

The ribbon

The main menu consists of Home, Insert, and View menu items which are part of the "ribbon". The ribbon introduced by Microsoft in Office 2007 is actually a container for other toolbar items. The ribbon is the replacement for the classic menus, toolbars, and is supposed to be more efficient and discoverable by the user. In fact you see a lot more on the "ribbon" than in the classic menu.

Home

The next figure shows the Home menu with its toolbar arranged from left to right and divided into sections. The Run toolbar item with the title Views when clicked would run the report open in the design view (in fact, even without a report open in the design view, the report can be run. The result would be the current date and time getting displayed in the center of the screen of an untitled report which has just ExecutionTime as the only item in the report).

The Font, Paragraph, Border, and Number toolbar sections become enabled if parts of a report need editing. The formatting of textboxes in the report, the formatting of numbers in the report, and the alignment of components in the layout can all be independently managed using these toolbar items.

Insert

When you click on the Insert menu item on the "ribbon", the tabbed page for this item is displayed as shown in the following screenshot:

It has four sections: Data Regions, Report Items, Subreports, and Header & Footer. These are all the normal items that are used either individually or together to make up a report. There can be more than one data region in a report.

A step-by-step guide to getting the most of Microsoft SQL Server Reporting Services 2008   Everything you need to create and deliver data-rich reports with SQL Server 2008 Reporting Services as quickly as possible Packed with hands-on-examples to learn and improve your skills Connect and report from databases, spreadsheets, XML Data, and more No experience of SQL Server Reporting Services required http://www.packtpub.com/learning-sql-server-2008-reporting-services/book

Data Regions

In the Data Regions section you have both the Tablix (Table, Matrix, and List) and the graphic controls that can be bound to data—the Chart and the Gauge. Gauge is new in SQL Server Reporting Services 2008. Chart and gauge implementations are the off shoot of collaboration with Dundas (http://www.dundas.com/). Report Builder is built in such a way that the dataset must be defined before any of the data regions are added to the report body. For the purpose of describing the various data regions in this section, it is assumed (in order to get the screen shots shown here) that a dataset has been defined and the default wizards on the design surface have been removed.

Table

The Table is meant for displaying data retrieved from a database either all data detailed in groups or a combination (some grouped and some detailed) of both. It has a fixed number of columns which can be adjusted at design time. The table length expands to accommodate the rows.

Data can be grouped by a single field or by multiple fields. Expression designer can be used in grouping as well. The grouping is carried out by creating row groups. Static rows can be added for row headings (labels) and totals. Aggregates for groups can be added. Both detailed data as well as grouped data can be hidden initially and the user can interactively reveal the data needed by drill downs.

When you click on Insert | Table | Insert Table and then click on the design surface you can add a table to the design area. The table appears as shown with handles to adjust its dimensions. The table can be dragged to any other location on the design surface (the body of the report) as well.

After placing the table, which by default has three columns and two rows, when you click on any other part of the design area you will see the table as shown. When you hover over the cell marked Data on the table you will see a little icon. This icon is a minimized version of the dataset fields. The grayed out feature that surrounds the table indicate the position of the rows and columns of the table. It also shows such other features as whether it is a detail, or whether it is a group. In the case of group, within a group the feature would indicate the nesting schematically as well. When you want to increase the size of a column or a row you can drag the double headed arrow that gets displayed when your cursor is placed between two columns or between two cells as shown.

When you click on the dataset icon in the cell Data you get a drop-down list containing the fields in the dataset as shown. You can choose any of the fields to occupy the cell you clicked and the corresponding header will be added to the table. In this particular dataset there are nine fields and you can choose any of them to occupy the cell.

When you right-click on a cell, a drop-down menu will be available. It can be used for the following:

• Work with the highlighted textbox (each cell of the table is a textbox) including to copy, cut, delete, and paste contents.
• Work with the properties of the Textbox.
• Populate the textbox with an expression using the expression builder. The expression builder gets displayed when fx Expression is clicked.
• Use Select to select the body or the Tablix.
• Insert a new column or a new row. Columns can be added to the right or the left of the clicked cell and rows can be added above or below the clicked cell.
• Delete columns and rows.
• Add a group. Both row and column groups can be added.

When you click on the properties of the textbox, the Text Box Properties window is displayed. The textbox has several properties which are arranged on the left as a list with each item having its own page as shown. The Help button on any of the pages will take you directly to the definition of the properties and is extremely useful.

In the General page, you can make changes to the elements in the Name, Value, and Sizing options page as shown. The Value is one which you choose among the column values (from the drop-down) from the dataset. You may also add a text for the ToolTip, which will display this text when the report is generated and this cell is accessed by hovering over it in the report. Alternatively you can set the Value and Tooltip using fx—the button that brings up the Expression window.

In the Number page you can set the number and date data type formatting options for the cell that contains a number or a date. This is what you normally would find in most Microsoft products such as Excel and Access.

In the Alignment page you can choose the vertical and horizontal alignments as well as the padding of the textbox content from the edges of the cell.

Similarly the Font and Border properties are the same ones you find in most Microsoft products.

The Fill property lets you add or change background color to the report as well as add a graphic element. The graphic element can be embedded, external, or originate from a database (being one of the fields accessed). Expressions can be developed to set a desired color for the Fill.

The Visibility of the textbox can be any of Show, Hide, Show or Hide based on an expression. In each of these cases the visibility can be toggled when another table cell is clicked (which can be chosen). This page also gives access to the Expression window which is similar to the MS Access expression builder.

The Interactive Sorting page allows you to define interactive sorting options on  the textbox.

Matrix

Matrix provides a similar functionality (roughly speaking rows against columns) to cross-tab reports in MS Access (http://aspalliance.com/1041_Creating_a_Crosstab_Report_in_Visual_Studio_2005_Using_Crystal_Reports.all) and Pivot Table dynamic views (http://www.aspfree.com/c/a/MS-SQL-Server/On-Accessing-Data-From-An-OLAP-Server-Using-MS-Excel/3/). The matrix should have at least one row group and one column group. The matrix can expand both ways to accommodate the data, horizontally for column groups and vertically for row groups. The matrix cells (intersection of rows and columns) display summary information (aggregates).

When you click on Insert Matrix in the Insert menu and drop it on the design area of Report Builder 2.0, it gets displayed as shown in the following figure:

Now if you click inside the boundary of the (2×2) empty matrix you will see more features of the matrix as shown in the following screenshot. The basic elements are the ColumnGroup (Column Groups), the RowGroup (Row Groups), and the Data. The group information is also displayed as shown by overlaid lines pointing to them. There needs to be a minimum of one group and one column for the matrix and there could be a hierarchy of column and row groups.

The row and column group cells have their own properties which can be displayed when you right-click on them as shown in the next screenshot for the row group. When you right-click on the cell marked Rows, the following drop-down menu  pops up.

In addition to the properties that you can set for the textbox in that cell, you have additional submenu items that work with the grouping and totaling. These are part of representing data in a matrix.

Each of the Tablix for the Rows and Columns has the additional submenu items which are shown here for the Rows. Similar ones apply for the Columns as well. These are useful when you want to create nested groups. With the Matrix design interface in SQL Server 2005 this would not have been possible.

Add Group

• Row Group
  • Parent Group…
  • Child Group…
  • ——————–
  • Adjacent Above
  • Adjacent Below

Row Group

• Delete Group
• Group Properties

Add Total

• Before
• After

In addition to the above, each of the items Rows and Columns cells has the following items as well. These specify how new columns and rows are inserted with reference to the current cell as shown. The differences are due to the geometrical positions that are allowed for the new columns or rows as shown.

For the "Columns" cell:

Insert Column

• Inside Group-Left
• Inside Group-Right
• ——————
• Outside Group-Left
• Outside Group-Right

Insert Row

• Inside Group-Above
• Inside Group-Below
• ——————
• Outside Group_Above

For the "Rows" cell:

Insert Column

• Inside Group-Left
• Inside Group-Right
• ——————
• Outside Group-Left

Insert Row

• Inside Group-Above
• Inside Group-Below
• ——————
• Outside Group_Above
• Outside Group_Below

Besides using a cell as a starting point, one could also use the rows as a whole or column as a whole to add further structure as shown in the next figure. Of course you need to use the proper submenu option to arrive at a particular matrix structure. Clicking at the indicated points would let you choose the structure you want for your matrix. If you click at the location shown for the Tablix you could choose to the delete the whole matrix. The Tablix graphical arrangement gives you the maximum flexibility in extending the matrix in 2-dimensions.

List

The list data region repeats for each row of data. List element provides a single container for the data which can be used to generate what are called Free Form Reports. In this kind of report there is no rigid structure such as a table for the data. You can also place a list inside another list or even a chart inside a list. You can drag a column from a dataset and drop it into the list. You can work with the list using the properties of the Rectangle it contains as well as its Tablix properties.

As described earlier, the design interface is very flexible and you can leverage all features provided by the Tablix structure like displaying details and adding groups either independent, or nested. The properties pages described earlier allow you to sort and filter grouped data.

When you drop a List on the design surface you will see just a single cell as shown. You can change its dimensions to suit your needs.

When you click on the List you can access its handles as shown:

When you add a List, there is one column and one row (just one cell). This can be extended in both directions by choosing the appropriate submenu items. These can be displayed by right-clicking on the handles as shown:

Chart

A picture tells lot more than a bunch of numbers, and charts (graphs) aggregate the whole range of data that is highly informative and aesthetically pleasing. Charts have a myriad of properties, both with regard to how they are linked to data as well as their visual properties that it is hard to justify describing them in an abbreviated fashion.

Charts are basically used while creating a graph from the data to summarize important and relevant information. There are two ways you can work with chart in Report Builder 2.0. The easiest is to use the Chart Wizard. The other way is to start with a chart template and then associate it with a dataset.

There are many chart types such as bar charts, column charts, line charts, pie charts, area charts, polar charts, range charts, scatter charts and so on. The following screenshot schematically shows the various supported chart types. The chart type would depend on the data that it represents. Generally a chart should help visualize the data. A chart has its data region as well, its Tablix properties.

You can insert a chart on the report body by clicking Insert | Chart |Insert Chart and clicking again on the body of the report. This brings up the Select Chart Type window as shown:

Here you have number of options as shown. For each of these choices you have a number of other options as shown on the right-hand side when you choose the Column type.

Assuming you choose the default (the one highlighted in the above figure) and click on the OK button, the type of chart you chose gets added to the design area of the report as shown. You can increase the size of the chart both ways by dragging the handles.

When you double-click inside the chart, you see the drop-zones on its three sides as shown. These are the areas into which you can drag-and-drop columns from the Dataset or use the minimized dataset icon that gets displayed when you hover over this area.

When you right-click on any of these areas, you can access the various ways you can work with the charts as shown:

The important chart related items are category fields, series fields, and data fields. You can work with all properties of the chart from this drop-down menu and even change the type of chart you want to develop.

Gauge

This is new in Reporting Services 2008. Like chart, gauge is also a data region. Gauge has only a single data region unlike a chart. It has the look of any industrial meters (measuring instrument) with a range of values and the indicator showing the present value or some configurable value. They can be used together with both table and matrix elements.

When you add a gauge to a report, it comes up positioned within a gauge panel. The properties of this can be accessed when you click outside the boundary of the gauge. When you want the gauge to display data, you should associate its data property with the dataset. Gauge, like chart also has a myriad of properties which can be accessed from its Properties window.

In Hands-on exercise, (Part 2) you will be adding a radial gauge to display the data and a linear gauge to display an average.

Report Items

Textbox, image, line, and rectangle are the items you find in the Report Items section of Insert. You will be using textbox and image both of which may be bound to related as well as unrelated database variables. In the case of textbox, you have an option to use a static text, a field from the built-in fields, or connected to one of the fields from the dataset. A table added to a report has textboxes in its cells, but you can add a textbox outside the table and bind it to an aggregate value related to  the dataset.

Similar to the textbox, an image added to the report can be embedded, originate from an external source; or being one of the dataset fields. An image can be added to the current report by right-clicking the Images folder in Report Builder and picking an image from your hard drive which can then be simply embedded in the report using the Image report item. This way you can add a logo to your report.

A line cannot be bound to the dataset. Its purpose is purely to provide support as a graphic separator element. The rectangle is also used to improve the visible appeal. However, it can contain other items, even data regions. You can control the rendering behaviour of items placed inside a rectangle (parent control) for the controls placed inside the rectangle. The rectangle will be an anchor for the items placed within it and move when the rectangle is moved in the report design.

When the line or the rectangle is used inside data regions you may use their RepeatWith property to allow them to be rendered when the report gets displayed.

Subreports

A subreport is a child of a main report. The main report is a container for the subreport(s). The Subreport section in the Insert menu item of the "ribbon" allows you to add a subreport. The parent report and the subreport are stored usually in the same folder on the report server. The main report can be designed to pass parameters to the subreport. The parameter then filters the subreport for it to be displayed in a data region of the main report.

Subreports can be separated from main data region of the main report or they can be placed within the data region of the main report. A report can contain more than one subreport.

As described above, the subreport may be placed inside the main report several ways as shown in the following screenshot:

Header and footer

A report page can contain a header and footer. In Report Builder 2.0 there is page footer by default which contains the built-in parameter ExecutionTime.

Some of the common features of page headers and footers are the following:

• Headers go at the top of the page and footers go to the bottom and they repeat on each page of the report.
• They both can contain static text, images, lines and rectangles, borders, expressions (lookup the properties of these in Report Builder 2.0).The expressions can include field references for reports from the dataset.
• You can easily add headers and footers from the Insert menu item in the "ribbon". You can just, as easily, remove them by right-clicking the item and choose to remove the item.
• The most common use of headers and footers is to display page numbers, report titles, and so on. There are number of built-in fields such as Page Number, Execution Time, Report Name, Total Pages and so on, which can be dragged-and-dropped on to headers and footers.
• To display variable data from a dataset in headers and footers you place a textbox and set the value for the textbox using an expression. Choose the appropriate field from the dataset. In a similar manner you can display aggregate values from the expression builder. For data from multiple datasets you cannot reference the fields, rather you should reference the objects in the report.
• You can suppress these on the first and last pages of a report using the PrintOnFirstPage, PrintOnLastPage properties which can be accessed from their properties.
• Report headers and footer are not the same as the page headers and footers.
• Reports that you see with a browser are rendered by the HTML renderer, but the report can be delivered in different formats. Each of them has their own renderer and you should optimize the report for the format you want to deliver.

View

The View menu has very few items as shown:

Report Data and Grouping are checked by default. If you want to review or make changes to the properties of objects you place on the report, you should frst place a checkmark for the Properties. Otherwise you will see only the Properties pop-up window that comes up when you right-click an object to look at its properties.

Place checkmark for the Rulers if you want to design measured placement of objects on the report. The rulers become visible when you move around or adjust object sizes.

Report Data, the Report Designer, and the properties

These are the items that you will find just below the "ribbon" assuming you have not disabled Report Data and Properties (these are enabled by default) as shown in the following screenshot. The properties will be that of the object highlighted in the report body.

Report Data

As seen in the previous figure, you can create New report data; Edit an existing report data (the figure above has an existing report data in the C_Main. It also has three main folders. The built-in fields we saw earlier. The parameters will contain parameters associated with the report. The images folder contains images that can be imported into this folder, which you can embed in a report from your hard drive.

The Report Data is an excellent starting point to create a report from scratch without using wizards displayed in default. When you click on the drop-down handle New the following menu will be displayed.

All you need to do is to go and configure each of the items from top to bottom as follows:

• The Data Source… will bring up a window where you establish the connection to the datasource.
• After this, you click on Dataset… to create a query to extract a set from the database which you want to display in your report. When you click on Dataset… you will display the properties of Dataset which you can use to design a query visually (only for SQL Server databases), an SQL Statement or import a saved SQL query or RDL file.
• You can then define parameter(s) by clicking on Parameters… to create a parameter from its Properties window. You add a parameter if you want to filter your query further to produce a smaller and more manageable set of pertinent information.
• Click on Images… to choose images from your machine if you need to embed them in the report.

On the other hand, if you already have these you can edit them or delete them.

Report Designer pane

The various parts of this pane are shown in the following screenshot. This has been described earlier. The choices you make in the View pane will show or hide the Groupings at the bottom as well as in the Ruler. When you want to create a report from scratch and do not want to use the wizards, you can delete everything on the report and start from scratch.

For displaying groups of data, the underlying data must support this structure and it is necessary that there exists hierarchical relationships within the data. An example is shown in the following screenshot from the TestNorthwind database used in the exercises.

In Report Builder 2.0, groupings show both row groupings as well as column groupings. If you are starting from the wizard, you will be setting up the row groups as well as column groups as seen in the Arrange Fields page of the wizard. The available fields were moved into column groups and row groups. One of the shortcomings of this wizard is that once you move a field from the available to any of the other three, you cannot move it back. However you can only move between the three. If you go back and return, you have the same arrangement. If you Cancel, you need to start the wizard again. Another problem with this wizard step is that you must add a field to the Value field. What if one wants to show only a few columns of data in a table? However, it is well suited for matrix design. The above shortcomings in the earlier SQL Server 2008 RC0 version have been rectified in the latest version of Report Builder 2.0 bundled with the Feature Pack (https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=371356).

The above arrangement would lead to a report's grouping as shown here:

Each of the groups shown above has its own properties which can be accessed  by right-clicking the group. For example, the row group [ProductName] in the above. You can then review its Group Properties window as shown in the two following screenshots:

When you click on the drop-down item, Group Properties…, the Group Properties window shows up.

Clicking on the Next button in the New Table or Matrix wizard's Arrange  Fields page takes you to the window where you can arrange to show the group aggregates. You can show them in several ways depending on the choice you make as shown in the following screenshot:

Properties

This window appears, by default, at the right of the report designer pane. It shows all configurable properties of the objects on the report body. You only need to click on the object. Most of the properties become effective as soon as you complete the property and leave that property or move to the next property in the Properties window. If you add custom assemblies, these are also shown in this window.

Server status and tools

At the very bottom of the Report Builder you will get an indication as to the Report Server you are connected to (present case there is only one running) and its status as shown in the following screenshot. You can also change from design to run (preview) by clicking on the respective icons in this figure. You can also enlarge or reduce the size of the report both in design and in preview using the zoom slide.

Summary

In this part of the article, we had a look at the Report Builder overview. We described the Report Builder 2.0 interface along with the new features that are incorporated into this version. In the next part, we will discuss about Enabling and reviewing My Reports, and will see how to Modify a basic report.

Indexes

Index design is a large concern when designing your data structure. SQL Server indexes are similar to the indexes you would find in the library or the tabs in a dictionary.

There are four main types of indexes:

• Primary Keys (Creates a unique index to enforce a primary key)
• Unique (Creates a unique index but not a primary key) – Clustered (Which are indexes based on the actual table data)
• Non-Clustered (Not based on the table data)

I'm going to touch on how setting your indexes can affect performance. To retrieve data from a table, a SQL statement is passed and the table is searched in a linear mode until the data that is needed is come across. Indexes eliminate whole table scans by allowing queries to access data by using the index directly. This allows queries to perform lookups much more efficiently.

Indexes help with table SELECTS, UPDATES, and DELETES, but with INSERTS, indexes will slow the performance down. The reason for this is simple if you think about it. Inserting a record in a table with a lot of indexes requires the insert to traverse all the indexes in the table. This is where the design aspect comes into play. If the majority of work running against a table is lookups then you will want to use indexes, if the majority of work done against a table is inserting then you probably do not want to use indexes. Another consideration you might need to consider is table size. If a table is small then it may be more efficient to do a table scan rather than using indexes.

To summarize, when thinking about indexes and performance you must considers the following:

1. What type of actions will this table be primarily used for? If you will be using the table for things other than INSERTS then indexes are a good idea.
2. Can an index be created that will represent the majority of data you will be pulling? If so then the queries need to look no further than the index itself for the data. This eliminates the queries from having to hit the actual data tables.
3. Is the table large enough to require an index? Indexing small tables can slow down your performance.

Stored Procedures

Another "surefire" way to improve your response times and performance is to use stored procedures whenever possible. Stored procedures are prebuilt, precompiled SQL statements that are stored within the database itself. Stored procedures can accept parameters as input and return values as output. Stored procedures are efficient for many reasons. First off, stored procedures exist as part of the database itself. Because of this, they are precompiled and prechecked for errors. SQL statements that are going to be used frequently such as table inserts, retrievals, deletions, and updates should be built as a stored procedure. Data validation can be built into stored procedures to ensure that business rules are followed and data integrity is maintained. When executing a stored procedure it is only necessary to pass an execute command. For example, to call a stored procedure called sp_AddCustomer, you would send

     EXEC sp_AddCustomer, ParameterList

to the database along with any parameters needed. This is much more efficient than sending entire SQL statements over the network, especially if there are many users performing the same actions.

Let's say we have a table called orders. We may want to add orders frequently (we would hope!). We could build a stored procedure that will INSERT a row into the orders table based on a list of parameters. For example:

     CREATE PROCEDURE sp_AddOrder @customername char,
     @Customeraddress char, @ItemNum Int, @OrderDate Date
     As
     Declare @InsertString NVARCHAR(500)

     Set @InsertString ='INSERT INTO ORDERS
     (customername,customeraddress,itemnum,orderdate) values
     (@customername, @Customeraddress, @ItemNum, NOW)'
     GO

Then to execute:

     EXEC sp_AddOrder InsertString,customername,
     Customeraddress, ItemNum

Stored Procedure make database design quick , scalable and maintainable. When deciding to use stored procedures you can follow the rules below.

1. Stored procedures execute by simply passing an execute command to the database. This relieves the network from having to pass an entire SQL statement over thus saving bandwidth and response time.
2. Stored procedures are precompiled. Thus all error checking is done at the time it is built. This ensures solid SQL Statements and fast execution time.
3. Stored procedures are fantastic for maintaining data integrity. Good data integrity always translates into good response times.

Triggers

Triggers are very similar to stored procedures in that they also reside in the database itself. They too are precompiled and have a very similar structure.

So what are the differences? Triggers execute automatically based on a particular action to a table. INSERT, DELETE, UPDATE are all common catalyst to a trigger.

Why would you want an additional statement to run against the database if you are already passing one? The answer – triggers can be used for anything from audit trails to data validation. A trigger on a table is considered to be part of any statement run against that table. Because of this, a ROLLBACK in a trigger will cause the entire SQL statement to be rolled back. It becomes almost impossible for someone to corrupt your data. As we said before, good data means good response times.

If you need to maintain data integrity, you can move all validation into triggers and take that burden off the client machine. In a large-scale environment this can save valuable resources. If you need to maintain an audit trail of sensitive data, you no longer need to put that processing on a client application or on the burden of the person maintaining the database. Every time you update data on an orders table for example, you could move the old data into a table called audit_orders.

All this work which would normally bee done on the client side or by the DBA, is now handled automatically by a TRIGGER. This not only takes the burden away from others, but also assures that there can be no mistakes in tracking the history of the data. The general structure for designing a trigger is below.

     If exists (select * from sysobjects where type = 'TR'
     name='ti_orders')
     Drop trigger ti_orders
     Go
     Create trigger ti_orders on orders for UPDATE
     As
     Insert into audit_orders (parm1,parm2,parm3,parm4,etc,)

The above example will move a copy of the data into the audit_orders column before it is modified. This will give you a snapshot of how the data existed through time. And because this is prebuilt, you only need to write it once and not worry again about maintaining the audit table.

Summary

In an enterprise situation it is often difficult to say what vehicles are going to be used to access or modify your data. There may be more than one group of people, more that one application, or more than one DBA.

Because of this, strong validation on the database side in the form of Stored Procedures,Triggers and Indexes will ensure your database remains scaleable, your data remains safe, and response times remain high.

Poorly designed data models can only frustrate your users and yourself. If you follow the steps above when designing your database, you will find you end up with a very scalable and responsive structure you can build upon.

Phillip Elbert

This inefficiency, materialized in the form of slow performance, is one of the primary causes of system abandonment. This is particularly troublesome in the SQL Server world where many systems servicing large user bases often run on low cost server boxes that leave little margin for performance waste. Many organizations have tossed out their SQL Server solution running on a $3000 PC because the performance wasn’t satisfactory (not achieving so-called “Enterprise” performance), to replace it with a multi-million dollar mainframe solution, overcoming embarrassing inefficiency with brute force.

Reward

Several years back, in a moment of nerdish bravado, I made a foolish blanket statement that I could reduce the runtime of virtually any element of a non-trivial SQL Server database solution by 95% (thus improving the performance by about 20x), doing so through some rudimentary changes requiring nothing more than some analysis, minor code changes (changing the underlying code, but not the functionality), indexing, and file group changes. To my surprise, and even greater dismay, this number actually proved to be remarkably accurate: From giant multi-hour organization wide reports, to simple security procedures run hundreds of times a minute, the obvious low hanging fruit alone often improved performance by 10x or more. With a little bit of elbow grease it has proven extraordinarily common to improve performance by 20x or more, significantly improving responsiveness and load handling of the respective systems at minimal cost.

The remarkable thing is that these weren’t systems implemented by bad developers – many of them were extraordinary developers who implemented a lot of tricks and techniques that I’ve co-opted and added to my own bag of techniques. Instead there seems to be a dearth of real information on developing for performance in SQL Server, leaving many to guess about the best approach, not to mention that there isn’t enough attention paid to performance efficiency in enterprise solutions. Many seem to be under the false impression that gross inefficiency requiring massive clusters to perform trivial tasks merits a capital-E Enterprise designation.

Motivation

In software development there’s an oft-referenced vice known as “premature optimization”. This is the tendency to prematurely focus on code performance while code is still young and awkwardly growing, before the critical performance weaknesses have been identified and measured. The end result of this misguided effort is often convoluted code that is difficult to understand and maintain (for instance code including inline assembly or using specialized system hacks in seldom called edge functions). This is often a mistake of inexperienced programmers that haven’t had the perfectionist engineering streak beaten out of them.

Consider also that performance truly isn't a concern for the vast majority of code in most client-side applications – it likely doesn't matter if the code that validates an input box in a Windows Forms application takes 3ms or 70ms to complete. As the processing is decentralized and isn’t impacting other users who might be running the application elsewhere on the planet, it is basically making use of “free” clock cycles available on the client PC, and generally is imperceptible to the user. If one thousand different users were running the application simultaneously, they’re running it on a thousand powerful PCs, effectively throwing a massive “cluster” at the problem. In other words, you can overcome application inefficiency on the client side through massive computational excess and a endless ability to scale-out. Even in cases where there is worthwhile performance issues identified, for example an image processing algorithms that takes several seconds to perform an operation, it's often best to wait until the project nears a release and the code has settled, at which point you can send a commando performance team to profile and then selectively improve the slowest sections of code that will have the most beneficial impact, focusing on the lowest hanging fruit, yielding a bounty of quick wins. (Taking one for the team because there's no I in team, and no cliché unworthy)

Enterprise databases, or any centralized system for that matter, are entirely different beasts – performance is one of the critical elements of these systems, and performance problems are one of the primary reasons why solutions are abandoned or re-architected. Consider that every clock cycle wasted on a shared resource, such as a database server, impacts the performance of the overall system and every other user. In most environments there is a massive asymmetry between the computational capability of client machines, and the computational capability of a shared system, such as a database server. There are usually some fixed financial and technological limits to the amount of hardware that a system can scale to, so your database server running on a lowly Dell two-way server is desperately trying to keep up with the demands of 500 user workstations pounding away at it. Even though Google is clustered on purportedly thousands of machines, they still have to develop efficiently to be able to economically service millions of users in a timely manner.

Thus, while it might seem irrelevant when taken alone that your stored procedure saturates the resource, taking 200ms to return a simple list of values to populate a drop list for Joe User, imagine 100 users all opening that form at the same time putting a shared demand on the database system. The performance impact starts to become significant and adversely affects the usability (and credibility) of the system. This is exacerbated by the fact that simultaneous performance demands aren’t merely additive on shared resources, but rather contention and task sharing often means that these issues snowball into much more than the sum of the parts.

Wisdom

You should consider the performance of your database from day one with every table you add, every index you create, every trigger you concoct, and every relationship you define. While the misguided will argue that this amounts to premature optimization (as Ralph Waldo Emerson observed, a foolish consistency is the hobgolbin of little minds, and the belief that any performance concerns are premature is just such a foolish consistency), the reality is that the performance of a database system is largely defined by the fundamental design of the system, and as the system grows it becomes much more difficult and costly to solve fundamental performance problems. Furthermore, once an enterprise system reaches production even the simplest performance change, such as adding an index, requires complex analysis to determine how it impacts other parts of the system, or that it satisfies what could be hundreds of procedures accessing the object.

The cynical will wonder how one can predict the future when designing a database system, but the reality is that the access patterns are usually obvious by the time you're starting designing tables – you know how the tables relate, what data will be searched, how often you'll be selecting the records versus modifying them, and how big the fields and records should be. Use this information effectively when developing the tables to choose the appropriate clustered and secondary indexes, to minimize the size of each record, and to write efficient SQL. Don't leave it for a maintenance programmer to reverse engineer the system and apply best guesses in a moment of crisis in the future.

The Basics? We’re All Professionals Here!

Much of the material covered in this outing will be old hat for a lot of developers, but is nonetheless worth a recap – Even among the pros there remains misunderstandings and conflicting information about the fundamentals of databases, and the true magnitude of impact they have on systems. I’ve intentionally authored this series conversationally, as opposed to a “high impact hit list!”, however if you’d just like a brief summary list you can find one at the end.

Data Size

This needs to be addressed as there is a growing camp of “data wasters” that erroneously believe that the larger the amount of waste, the more Enterprise Ready a solution is (capitalization used derisively). I step into this quagmire knowing full well that this section will yield me some “you’re a dummy!” responses from some newly earned adversaries, however that’s a price I’m willing to pay if I can save but one byte tree.

Minimize the size of your data. Don’t use GUIDs where they aren't necessary (e.g. where you don’t really need global uniqueness/replication), and don’t use a bigint where an int or a smallint will suffice. Don’t use a smallint where a tinyint will suffice or an nvarchar where a varchar would be fine. Use the smallest type that is reasonable for the field. Don’t invent vague packing technicalities or native type size issues in an attempt to justify oversized data.

Of course you should plan for realistic growth, and I’m not advocating that you use a tinyint to store your CustomerID field, but keep the rational, real world in mind when designing your applications – are you really going to exceed 2 billion users? Is there going to be more than 32767 languages in your application? Is it likely that we’re going to a new calendaring system that might have 2 billion months?

Evaluate if it might be an acceptable compromise to simply use these large types in your façade while actually storing smaller types in the actual database. This would give you improved performance, and would allow you to easily upsize your data types in the future in the unlikely event that it becomes necessary.

Clearly there are cases where large data types are legimately warranted, however too many database architects abscond themselves of any responsibility for efficiency by making everything a GUID or a bigint “just in case” (GUIDs have a substantial creation cost as well, in addition to the obvious storage and I/O costs. While GUIDs once used the available network card MAC address as the foundation and generally sequentially increased in value on each new GUID, in current Windows variants GUIDs are basically random numbers — used as a clustered primary key they can lead to endless data reordering).

Why does it matter? In real enterprise apps before you know it there are tens or hundreds of millions of rows throughout your database, and these rows need to be read from, and written to, the glacially slow storage subsystem constantly – given this finite resource, doesn’t it seem logical that 1MB of I/O carrying 20,000 records is better than 1MB of I/O carrying only 5,000 records? Of course it is. Isn’t it better that 10% of your database can fit in the memory cache rather than just 5%? Of course.

Don’t be lulled into a false justification of large data types by running ridiculously small benchmarks, where all of the data exists in the memory cache and the I/O is dwarfed by the computational element of the query, yielding "only" a performance hit of 10% or so with larger types – when your database gets to real enterprise size, size really does matter. That cluster indexed GUID primary key not only makes the row bigger, it makes every non-clustered index bigger (and thus slower) as well, and when the weakest link SAN is running at 100%, you'll regret every wasteful byte.

Indexes

Many SQL Server performance problems are rooted in missing or inappropriate indexes, or alternately unused indexes. This is often true for databases thrown together by front-end experts unhappily tasked with supplying the back-end database, just as it’s often the case for those carefully crafted by highly-focused database professionals.

An understanding of indexes, and a focus on their application, is paramount for high performance databases. Not only is it critical to create the right indexes, it’s important to craft your access to properly utilize the indexes that are there.

A Bookshelf Full of Examples

Non-Clustered Indexes

Indexes in reference books serve the same purpose (and share many of the same traits) as those in the database world – by referencing the index you can follow a shortcut to a particular piece of information, seeking directly to that specific page, versus going from page to page scanning the contents. In the SQL Server world these sorts of indexes are called non-clustered indexes (or secondary indexes) – they are a subset of the data ordered for a specific purpose, containing a pointer to where the real data row can be found.

In the Northwind database an example of a non-clustered index is ShipPostalCode on the Orders table. This index sorts by the ShipPostalCode, and may be used for a query such as the following.

SELECT * FROM Orders WHERE ShipPostalCode = '05022'

If you take a look at the execution plan (Ctrl-K or choose the option “Show Execution Plan” under the Query drop-down in Query Analyzer. When you execute a query with this option enabled a new tab, Execution Plan, will appear beside the results tab), you can see that an index seek took place, and then a bookmark lookup to find the actual data row. If we run SET STATISTICS IO ON on the connection before running the above query, we’ll get some statistics on the IO used to satisfy the query, which will be as follows.

Table 'Orders'. Scan count 1, logical reads 4, physical reads 0, read-ahead reads 0.

Compare this to the following query, which simulates having no index.

SELECT * FROM Orders WITH(INDEX(0)) WHERE ShipPostalCode = '05022'

In this case the execution shows a full table scan, and our IO statistics reports the following.

Table 'Orders'. Scan count 1, logical reads 21, physical reads 0, read-ahead reads 0.

Without the index there was significantly more I/O, and the differential would be vastly worse if this were a large enterprise-sized table instead of a small sample table. To make matters much, much worse such a table scan will block on every single exclusive page or row lock on the table, waiting for the data to be unlocked just to be able to verify if the wanted information is contained within, while the index seek knows that the locked data isn't the data that it's looking for and is unaffected. Try running the above two queries in two separate query analyzer windows while the following script is running in yet another (increase the WAITFOR delay if you can't jump between them all within 30 seconds).

BEGIN TRAN

ROLLBACK TRAN

The first query, using the index, instantly returns the result regardless of the row lock, which is logical given that the row being locked is not what the query is looking for, while the second query, not using the index (which can happen because no index exists, or the index isn’t deemed the best choice) blocks until the other connection’s lock is released. Not using an index is not only vastly less efficient, it can significantly worsen blocking problems as a database scales (or rather tries to scale).

Clustered Indexes

Returning to our analogy, many books take it a step further and order the content, making the data itself an index of sorts. A cookbook might sort by main category and then dish name, while a phone book famously sorts by [city, last name, first name] (in that order). Thus if you want to search based upon the sorted data, it’s extremely efficient – in the example of the phone book you can very quickly seek to the desired city and last name, scanning a small number of records for the desired person. In the SQL Server world this sort of index is called a clustered index (the sorting of the data itself), and for obvious reasons you can only have one clustered index in a book, or on a table. The primary benefit of a clustered index is that all of the table data is immediately available for every index match – no dereferencing is necessary.

In the Northwind database consider the following query.

SELECT * FROM Orders JOIN [Order Details] ON Orders.OrderID = [Order Details].OrderID WHERE Orders.ShipPostalCode = '05022'

If you look at the execution plan, the Order Details data is grabbed via a very efficient clustered index seek.

Clustered indexes aren’t all milk and honey, though. For instance imagine that you’re the hard working typesetter maintaining the layout of the phone book, and you’ve carefully arranged all of the entries on the respective pages. Every time a new entry comes in and doesn’t coincidentally fit right at the end of the sort order, or someone changes the information on an existing row in a way that alters the sort order (“Smith” changes his name to “Jones”), you need to reorganize some pages to make space. This same data-churn problem occurs with both non-clustered and clustered indexes, but clustered indexes exacerbate the problem given that it contains the entirety of the row data.

Of course you could plan for this by keeping a bit of blank space on each of your pages to facilitate at least a couple of changes, which is what the fillfactor is used for in SQL Server (a lower fillfactor leaves more empty space but reduces the true data density – insert performance is improved, but read performance is diminished. A high fill factor increases the real data density and thus read performance, but increases the likelihood of inserts requiring page splits. Note that fillfactor only applies on index creation, and whenever you defragment/rebuild your indexes), however this can be a serious performance issue for out-of-order inserted data, or frequently changing cluster indexed fields. It’s for this reason that many developers use the monotonically increasing identity field as their cluster index. Historically there was a worry that having multiple inserts all going to the same "place" in the index, at the end, would lead to scalability killing contention at this hot spot. SQL Server has logic to deal with identity fields and effectively eliminates this hot spot issue.

Another problem with clustered indexes is their girth (they contain the entire row data). This is largely irrelevant if you’re seeking to specific records, or where you actually plan on using all of the data after a lookup, however if you are querying a range of data (for instance the first name of all of the people with the last name “Forbes” in the city of “Oakville”) the query engine will read in the entire row contents for each matching record using a range scan, extracting only the requested data. In our example phone book there is so little extra data that it’s a minor overhead, however in many large real-world tables this can have a serious performance impact.

Consider if instead we had a secondary index that was sorted by City, Last Name, First Name, the query engine could very efficiently scan past only the small index entries.

Covering Indexes

This brings up a very important point – Some indexes contain enough information that you don’t even need to go to the content, your query being satisfied by the index itself. Consider an index in a tour book that sorts famous attractions by their name, and the country and city that they were located in, pointing to the page where further information regarding it could be found. If you just want to know what city the Accademia dell' Arte del Disegno in Italy is found in, a quick seek through the index will tell you that it’s in Florence. In this instance the index was a “covering index”, in that it fully covered our request and we didn’t need to dereference to the complete topical information. This is often the most efficient query mechanism of all.

Consider the following variation of a query we ran earlier.

SELECT OrderID FROM Orders WHERE ShipPostalCode = '05022'

This will efficiently use the ShipPostalCode index to find the specific record, and because the query is fully satisfied by the index itself, the costly bookmark lookup is avoided, and IO is minimal.

Table 'Orders'. Scan count 1, logical reads 2, physical reads 0, read-ahead reads 0.

The observant will note that OrderID isn’t actually in the index ShipPostalCode, or at least it doesn’t appear to be. The trick is that all of the fields that are the sort fields for the clustered index, if one is defined, are automatically added as data fields to every other index on the table. This can be blessing in cases where you want one of the clustered fields and suddenly the non-clustered index is a covering index, but it also needs to be weighed against the fact that it makes every other index larger, and thus less “data dense”.

Small covering index seeks are the most efficient method of pulling data, and it’s a good reason to ensure that you are only pulling the specific fields that you actually need from any given table, preferably with a covering index. Range scans are also highly efficient in many situations, and are usually used when the wanted entries in an index are consecutive, such as when you search BETWEEN two dates against a date index, though because of the previously mentioned bookmark lookup costs range scans are generally only seen if the index is fully covering, or against the clustered index.

Even in the case where a full scan is necessary, indexes might still be fully covering. Consider if the following index were added to the Customer table.

CREATE INDEX CountryCity ON Customers(Country,City,Address)

This index of course contains Country, City, Address, but as mentioned above it also contains CustomerID because it’s in the clustered index. Of course if we query on Country, or Country and City, or Country, City, and Address, an efficient seek or range scan might be used to pull the matching records.

SELECT CustomerID FROM Customers WHERE Country = 'Canada'

What if instead we wanted to search only on the address? In that case the index can’t be searched in order because it sorts by country and then city and address, and thus a particular address could exist anywhere in the index.

SELECT CustomerID FROM Customers WHERE Address = '43 rue St. Laurent'

You might be surprised to see that it still used the index, albeit this time it’s an inefficient scan rather than a seek. The index was used because it covered the query (all predicates and returned columns), and because the index is only a subset of the data it requires less I/O to scan the entire index than it does to scan the entire table data.

Statistics and Bookmarks

In a prior example we ran a query that required a bookmark lookup to satisfy the query (thus it did not have a covering index). The query was as follows.

SELECT * FROM Orders WHERE ShipPostalCode = '05022'

If you look at the execution plan for this query you can see that it seeks the “lookups” in the index, and then does a bookmark lookup against the clustered index (which is the actual table data). In this case there is only a single row to return, but even still the bookmark lookup cost is estimated to account for 50% of the cost of the query.

The cost of bookmark lookups, where an item is found in the index but it isn’t a covering index, is the reason why many people are surprised to find that SQL Server has ignored what they believe are perfect indexes and instead table scanned (“Why isn’t it using my index! ARGHHH!!!”). Consider the following query.

SELECT * FROM Orders WHERE ShipPostalCode = '24100'

Looking at the query plan you can see that it actually did a cluster index scan (which is a table scan on a table with a clustered index) instead of using our index, and the subtree cost is 0.0530.

This might seem perplexing because we seem to have a perfectly satisfactory index, however let’s do the same query again, this time using a query hint to force it to use our index.

SELECT * FROM Orders WITH(INDEX(ShipPostalCode)) WHERE ShipPostalCode = '24100'

If you look at the query plan you can see that it used our index, as demanded, but this time the bookmark lookups account for 80% of the query time. Our subtree cost comes in at 0.0564 – more than it was doing a table scan!

In this case this was only 10 records of a total of 830 (1.2%) yet still it opted to do a full table scan rather than using our index. Many developers have been perplexed in this situation, wondering why SQL Server was avoiding their beautiful index, but it’s doing it for a very valuable reason – it was cheaper than indirectly looking up each piece of data through bookmarks.

Of course we could have avoided bookmark lookups by using the index as a covering index if possible by using a query like the following, presuming this was all the data we needed to extract from the table.

SELECT OrderID FROM Orders WHERE ShipPostalCode = '24100'

Now it uses our index, is super efficient, and has a subtree cost of only 0.0064. In larger databases the difference can be the tremendous by avoiding both the bookmark lookup and the table scan.

So how did the query engine guess how many rows would match a criteria to choose which method (whether by index and bookmark lookup, or table scan) to use to most efficiently satisfy the query? That’s where something called distribution statistics comes into play. Statistics are a representative set of the data that are used by the query engine to make a best-guess plan for how to most efficiently serve the data. You can view the statistics for a given index, in the following case for the index ShipPostalCode, via the DBCC SHOW_STATISTICS command.

DBCC SHOW_STATISTICS('Orders','ShipPostalCode')

Due to the limited number of discrete ShipPostalCode values in the table, the statistics are entirely accurate in this case. In a more realistic database, with thousands or millions of rows, statistics start to become much more of an estimation (with an ever increasing margin of error). These estimations can lead to entirely wrong assumptions by the query engine in some edge cases, such as where it thinks a given set of predicates will yield thousands of rows when really it might yield only a couple.

Statistics can also fail for multi-field indexes. In this case the selectivity of the first field is used, so in the case of the index we created earlier (Country, City, Address), due to the fact that the country has a low selectivity (there are lots of entries for each country), the index will often be ignored, even though the city and address combination is highly unique. For this reason it is generally recommended that the most selective field comes first in your index, so in the case of that index the fields would be Address, City, and then Country. This is debateable because it also makes the index more single purpose – it no longer serves an efficient purpose for less-granular searches like just Country, or Country/City. This needs to be evaluated on a case by case basis, and truly wouldn’t be an issue for fully-normalized tables.

It should also be noted that ensuring that your statistics are as accurate as possible is critical. SQL Server includes automatic statistic updates, on by default, where it will attempt to do data sampling and update statistics when it feels they are out of date. Nonetheless it is a best practice to schedule full statistic updating at regular intervals (at a minimum weekly), preferably using the WITH FULLSCAN option so it is as accurate as possible. The standard database maintenance plan includes a step for statistic updating, and allows you to choose the amount of data to sample.

Regardless of all of the above, there will be cases where you may find that your statistics are up to date, your indexes are optimal, yet SQL Server is still incorrectly choosing not to use your index. In this case it may be an unfortunate reality that an index hint needs to be added to the query to politely (or rather sternly) request that it reconsider. Obviously this should be a last resort.

Actually Using Indexes

So you’ve created beautiful indexes, and you’ve ensured that your query only pulls the necessary data from each table, using covering indexes where possible to avoid costly bookmark lookups. You pull up the execution plan to find…that the query engine is entirely ignoring your index. There are several reasons why this could happen.

Consider the following query.

SELECT OrderID FROM Orders WHERE LEFT(ShipPostalCode,4) = '0502'

Fairly simple query, and from the looks of it one might think it’d be an efficient covered index seek. Upon execution you’ll discover that actually it was an inefficient scan. Consider the following instead.

SELECT OrderID FROM Orders WHERE ShipPostalCode LIKE '0502%'

In this case the query is executed as a highly efficient index seek. I have had cases where this tiny difference reduced an enterprise report from running for literally hours to a matter of seconds.

The reason is that in the former the indexed field was hidden within a function. The query engine can’t predict what the result of the function will be, so it’s forced to evaluate it for every row to see what pops out. LIKE is a first class comparison, it knows how it behaves, so the query engine can actually optimize against it. There are countless cases where people hide criteria fields in functions unnecessarily, and the result is massive, unexpected inefficiency.

The most common example of this mistake is using DATEADD/DATEDIFF to pull rows within a certain period of time – instead of pre-calculated a fixed demarcation (i.e. precalculating GetDate() – 3 years) and then doing a direct comparison with the row data, developers are forcing whole table scans with wasteful date computations on every single row. For instance consider a query to report news items that have occurred within the past 12 hours from a hypothetical news table .

DECLARE @CurrentTime

SELECT * FROM NewsStories WHERE DATEDIFF(hh,NewsDate,@CurrentTime)<12

Guaranteed to be terribly inefficient, yet it's overwhelmingly common. The query engine can much more effectively optimize the following variant.

DECLARE @StartTime

SELECT * FROM NewsStories WHERE NewsDate > @StartTime

Database Cheat Sheet

Indexed Computed Columns

At the outset I advocated that you minimize space usage (increasing real data density). The goal wasn’t to try to fit that database on a floppy disk, but rather to minimize the amount of I/O necessary to satisfy a given query, as I/O is the weakest link of most enterprise systems. There are design choices, such as adding additional indexes, that actually increase the size of your database on disk yet reduce the I/O necessary for certain queries, and these are usually very worthwhile trade-offs.

Another powerful technique you can use to trade disk space for improved database performance is indexed calculated columns. There are countless variations, but I’ll cover one scenario that is fairly commonly used – report counts by month for a given year. In the case of the Orders table this could be achieved via the following query.

SELECT YEAR(OrderDate) AS [Year], MONTH(OrderDate) AS [Month], COUNT(*) AS [Monthly Orders]
FROM Orders
WHERE YEAR(OrderDate)=1997
GROUP BY YEAR(OrderDate), MONTH(OrderDate)

Instead of adhoc decomposing the date into month and year constituents, consider adding them as computed columns.

ALTER TABLE dbo.Orders ADD
            OrderDateYear  AS CONVERT(smallint,YEAR(OrderDate)),
            OrderDateMonth  AS CONVERT(tinyint,MONTH(OrderDate))


Now we can change our query to the following.

SELECT OrderDateYear AS [Year], OrderDateMonth AS [Month], COUNT(*) AS [Monthly Orders]
FROM Orders
WHERE OrderDateYear=1997
GROUP BY OrderDateYear, OrderDateMonth

By itself we’ve done nothing for the query efficiency (in fact it is actually less efficient as it’s applying the where predicate after building the set), though we’ve achieved a bit of “code re-use”. However we now have the foundations for some powerful indexed computed columns.

CREATE NONCLUSTERED INDEX IX_OrderDateDecomposed ON dbo.Orders
            (
            OrderDateYear,
            OrderDateMonth
            ) ON [PRIMARY]
GO


Now the query referencing these computed columns is dramatically more efficient. Even better, these indexed computed columns haven’t decreased the real data density of the table because they’re only materialized in the index. NOTE: Ensure that queries that don’t need these computed fields don’t pull them explicitly or implicitly via the wasteful * column selector, as it’ll unnecessarily calculate each of the computed fields for each row.

Indexed Views

[This functionality only exists in the Enterprise and Developer edition of SQL Server 2000]

An indexed view, sometimes referred to as a materialized view, is a sort of indexed computed columns on steroids, taking the idea of storing computed results to the next level. In a previous example we improved the performance of some data aggregation logic by grouping on some indexed computed columns. We can take a variant of that and create a view out of it.

CREATE VIEW dbo.OrdersByMonth
WITH SCHEMABINDING
AS
SELECT OrderDateYear AS [Year], OrderDateMonth AS [Month], COUNT_BIG(*) AS [Monthly Orders]
FROM dbo.Orders
GROUP BY OrderDateYear, OrderDateMonth

As it is the view is acting as nothing more than a template for queries against the underlying table, and is only of benefit for code reuse (which itself is a very worthwhile goal). We can take that a step further by materializing this view so the actual results are stored, and changes to the underlying table is automatically reflected in the aggregates. The following command creates the indexed view.

CREATE UNIQUE CLUSTERED INDEX IX_OrdersByMonth ON OrdersByMonth(Year,Month)

Now the following query will be satisfied by the indexed view, using the pre-computed values, rather than recalculating for every query.

SELECT [Year],[Month],[Monthly Orders] FROM OrdersByMonth WHERE Year=1997

In the end the resource usage for our monthly order count query has dropped by about 90% over the original query, and this is for a tiny sample database. In the real-world the differential can be extraordinary.

Indexed views do have some downsides, such as the automatic maintenance that occurs whenever the underlying data changes, however they can be an extraordinarily powerful tool in your arsenal and deserve further research if your platform supports it.

Summary

• Keep your rows as small as possible to maximize real data density.
• All tables should have a clustered index except in rare exception situations. Normally this will be a single field primary key.
• Small clustered indexes keep non-clustered indexes small, increasing real data density.
• Clustered indexes help make other indexes covering indexes.
• Queries serviced by covering indexes are extremely efficient.
• Avoid hiding criteria fields in functions – indexes will not be used for them.
• Consider indexed computed columns where appropriate
• If you shelled out the cash for Enterprise Edition, seriously evaluate how indexed views fit in your solutions
• Index, index, index! Only in extremely rare cases are the additional update and insertion costs associated with maintaining indexes heavier than the benefits.
• Understand execution plans. Evaluate them regularly.

Examples given in this series will reference the sample database “Northwind”. It can be installed via the script found on the Microsoft website

Dennis W. Forbes

XML into SQL Server 2005 Example

To load data from an XML into SQL Server, can be done in 3 easy steps.

1. Setup SQLXML.

2. Application to perform loading.

3. Develop Mapping File.

1. Setup Instructions (SQLXML)

[Install]

1. Download SQLXML 3.0
    Note: if using SQL Server 2005, then SQLXML4.0 comes with it.

[Convert to .NET Compatible]

2. To use this library, it must be convert to a type library that is .NET compliant. To do this.

     a. Using a Command Prompt browse to C:\Program Files\Common Files\System\Ole DB

     b. Verify that xblkld3.dll or xblkld4.dll exists.

     c. Assuming you have some version of Visual Studio .NET installed, run

             [Drive]:\[Path of Visual Studio]\Bin\tlbimp.exe [xblkld3.dll or xblkld4.dll]

          This will create a new library called SQLXMLBULKLOADLib.dll.

3. In Visual Studio, select "Add Reference" in the Solution Explorer

4. Browse to Location mentioned in 2.a

[Add to Component Service (COM)]

5. Goto Start > Control Panel > Administrative Tools > Component Services

6. Under Components, Add new Application called "Sql Xml"

7. Open newly created application, and drag-n-drop, the xblkld3.dll or xblkld4.dll into application

[Reference in .NET Solution]

8. Add a reference to SQLXMLBULKLOADLib.dll to your project.

2. Loader Application

The loader application takes 3 things to execute properly – connectionstring, mapping file location, and data file location.  The example that I have provided allows for the user to provide 2 and assumes the connection string is fixed.  Since SqlXml requires a connection string, it is assumed that a database exists and that you have rights to the database based on NTLM credentials of SQL user credentials provided in the connection string.  For this example we have created an empty database named "SqlXmlDemo", and set SqlXml Component to do the following

objBL.CheckConstraints = True  // Check for Database Constraints and throw appropriate errors
objBL.SGDropTables = True      // Drop Tables referenced in Schema prior to loading
objBL.SchemaGen = True         // Create the Database Schema, if it does not exist      

C#: Loader Application

using System;
using System.IO;
using System.Collections;
using SQLXMLBULKLOADLib;
using System.Data.OleDb;
using System.Diagnostics;

namespace SQLXmlExample
{
    class Program
    {
        [STAThread]
        static void Main(string[] args)
        {
            string schema = string.Empty;
            string datafile = string.Empty;
            string connectionString = @"provider=SQLOLEDB;data source=localhost;database=SqlXmlDemo;Integrated Security=SSPI;";

            for (int i = 0; i < args.Length; i++)
            {
                switch (args[i].ToLower())
                {
                    case "-schema":
                            schema = args[i + 1];
                            break;
                    case "-datafile":
                            datafile = args[i + 1];
                            break;
                }
            }
            if (schema == string.Empty || datafile == string.Empty)
            {
                Console.WriteLine("Missing Schema or Data File.  Format: SqlXmlExample -datafile [filename] -schema [filename]");
                return;
            }

            Load(datafile, schema, connectionString);
        }

        static public void Load(string XMLFilename, string XMLMappingFilename, string ConnectionString)
        {
            SQLXMLBULKLOADLib.SQLXMLBulkLoadClass loader = new SQLXMLBULKLOADLib.SQLXMLBulkLoadClass();
            loader.CheckConstraints = true;
            loader.XMLFragment = true;
            loader.SchemaGen = true;
            loader.SGDropTables = false;
            loader.Transaction = false;
            loader.ConnectionString = ConnectionString;
            loader.Execute(XMLMappingFilename, XMLFilename);
        }
    }
}

VBScript: Loader Application

Dim objBL
Set objBL = CreateObject("SQLXMLBulkLoad.SQLXMLBulkLoad.3.0")
objBL.ConnectionString = "provider=SQLOLEDB;data source=localhost;database=SqlXmlDemo;Integrated Security=SSPI;"
objBL.ErrorLogFile = "C:\ImportSample\error.log"

objBL.CheckConstraints = True
objBL.XMLFragment = True
objBL.SchemaGen = True
objBL.SGDropTables = True

objBL.Execute "C:\ImportSample\MappingFile.xml", "C:\ImportSample\DataFile.xml"
Set objBL = Nothing

3. Develop Mapping File.

The maintenance aspect of this process is 95% related to the mapping file.  The mapping file tells the application how the XML data elements map to the SQL Database Table.  The sample data file below is contains information about 3 people.  The mapping file show below the data file. outlines how this data is mapping to the database.  Since the application has been instructed to drop tables and recreate them, the mapping file is in controll of hte structure of the tables in which the data will be loaded.

XML Data File

<?xml version="1.0" encoding="UTF-8" ?>
<Data>
    <Record>
        <TABLE_NM>CLIENTS</TABLE_NM>
        <ERR>0</ERR>
        <FIRST_NAME>John</FIRST_NAME>
        <LAST_NAME>Smith</LAST_NAME>
        <CITY>Houston</CITY>
        <STATE>TX</STATE>
        <ZIP>77082</ZIP>
        <DOB>1973-07-13</DOB>
    </Record>
  <Record>
        <TABLE_NM>PROSPECTS</TABLE_NM>
        <ERR>-1</ERR>
        <FIRST_NAME>James</FIRST_NAME>
        <LAST_NAME>Walton</LAST_NAME>
        <CITY>Dallas</CITY>
        <STATE>TX</STATE>
        <ZIP>77563</ZIP>
        <DOB>1965-01-15</DOB>
    </Record>
  <Record>
    <TABLE_NM>CLIENTS</TABLE_NM>
    <ERR>0</ERR>
    <FIRST_NAME>Jane</FIRST_NAME>
    <LAST_NAME>Doe</LAST_NAME>
    <CITY>Austin</CITY>
    <STATE>TX</STATE>
    <ZIP>77342</ZIP>
    <DOB>1961-12-31</DOB>
  </Record>
</Data>

XML Mapping File

<?xml version="1.0" ?>
<Schema xmlns="urn:schemas-microsoft-com:xml-data"
        xmlns:dt="urn:schemas-microsoft-com:xml:datatypes"
        xmlns:sql="urn:schemas-microsoft-com:xml-sql" >

   <ElementType name="TABLE_NM"        dt:type="string" />
   <ElementType name="ERR"            dt:type="int" />
   <ElementType name="FIRST_NAME"    dt:type="string" />
   <ElementType name="LAST_NAME"    dt:type="string" />
   <ElementType name="ADDRESS"        dt:type="string" />
   <ElementType name="CITY"            dt:type="string" />
   <ElementType name="STATE"        dt:type="string" sql:datatype="nvarchar(2)"/>
   <ElementType name="ZIP"            dt:type="string" />
   <ElementType name="DOB"            dt:type="date" />

   <ElementType name="ROOT"            sql:is-constant="1">
      <element type="Record" />
   </ElementType>

   <ElementType name="Record"       sql:relation="PERSON">
      <element type="TABLE_NM"        sql:field="TABLE_NM" />
      <element type="ERR"            sql:field="ERROR_ID" />
      <element type="FIRST_NAME"    sql:field="NAME_FIRST" />
      <element type="LAST_NAME"        sql:field="NAME_LAST" />
      <element type="ADDRESS"        sql:field="ADDRESS_LINE1" />
      <element type="CITY"            sql:field="CITY" />
      <element type="STATE"            sql:field="ST" />
      <element type="ZIP"            sql:field="ZIP" />
      <element type="DOB"            sql:field="BIRTH_DT" />
   </ElementType>

</Schema>

Result

The result is a database with a newly created "PERSON" table and the data from the data file above.

Database Summary from SQL Server 2005

PERSON Table after execution Loader Application

Brad Merrell