If you are reading this article, you should already have a working knowledge of SQL and are familiar with filtering your data with a WHERE clause, or limiting your result set from joining other tables with an ON clause.

The part where this article comes in is when you start to see a significant amount of time passing while your once simple query is now hitting your production tables which are multitudes bigger than your test data.

This article is not just for those of you who are seeing your queries start to take 30 seconds or more, but everyone could probably benefit from this simple article.

First of All – What is Slow?

In my opinion, 98% of the queries you are running should fire in less than 1 second. That may sound bold (and perhaps to some even foolish) but in truth if you know what you are doing your queries themselves should be very fast and your slow down will be your network traffic if you are hitting a different server for your data.

I have created and worked with databases ranging from just a few thousand records to the hundreds of millions. And my “lookup” queries for finding records (whether customers or products or nearby store locations) all take sub-second. The key, is not just a good index… but rather, a good set of indexes. And more importantly the right index for the right job.

They Clustered my What?

If you are using Microsoft Enterprise Manager for SQL Server 2000 databases, or Microsoft SQL Server Management Studio for 2005 databases, or even just using Visual Studio for small SQL Express databases (such as the one used for this website) then you will likely notice the “Primary Key” button that sets your currently selected column into the primary key column for your table. A primary key is a “unique” index, but it also defaults to being a “clustered” index in the above mentioned IDE’s and probably more.

Why is that important to know? For starters, you only get one clustered index per table. Secondly, your clustered index will also determine the default sorting of your select statements. It also can be your biggest performance loss when you are inserting or updating data in that table.

If the data in your table will rarely change and you use it basically to join to get a friendly name, or to filter out records not in a certain group, then you would just leave the clustered primary key alone. Even if your table changes fairly often such as a “store locations” table where you might have a few thousand stores across the country and they change their phone numbers or address etc., you would still likely just leave the index alone.

So When Do I Cluster?

If you had a table filled with customer data, and you need to find a record based on “Last Name” and “Zip Code” then you might find yourself waiting a good while (or worse yet timing out) while your query is being processed. Now is the time to remove the “cluster” from your primary key field (remember you only get one cluster so you have to remove it from the primary key). Now that you have done that you can create a new index, and of course you are going to set it as a “clustered” index. The columns, in this order, that you will add is “LastName,FirstName”. Now you might be asking “why didn’t we cluster ‘LastName,ZipCode’ since that is what we are searching on?”

That’s a great question, and the answer is this: You really only need to cluster “LastName” as that will be more than good enough to drop your search query down to sub second even if you have millions of records. Adding “Zip Code” to your search will simply help you limit the result set for someone searching for ‘Smith’ which would return just too many records to look at. So, we know we don’t need to add “ZipCode” to the cluster, but why did we add “FirstName”? The answer to that is something I mentioned above. And I’ll explain now.

Default Sort Benefits

Your clustered index will tell SQL Server how to sort results given to you by default. So if you do a “SELECT * FROM dbo.MyTable” then you’d simply get all the records from that table as they were entered. But, if you had a clustered index on “LastName,FirstName” then you’d get all the records sorted by “LastName” then “FirstName”. So in our scenario above of the ‘Customer Search’ we now get the benefit of displaying the results to the user for his search in alphabetical order which helps him to find the record he wanted a lot faster.

There are a lot of other benefits of clustered indexes, and there is so much to go over, but for now I’ll leave it at that. Remember your primary key column is probably defaulted as “clustered” which is fine for most scenarios, but you might want to change it one day. Also, keep in mind that you only get one clustered index per table, so use it wisely. Lastly, keep in mind that fiddling with clustered indexes could effect your INSERT or UPDATE code, so make sure to check the performance of these if it is a factor for you.

Source:singingeels.com

Since we will also want to keep a complete history of the customer’s payments, we can’t just add a single column to our table called ‘IsActive’. (Also, as a side point, proper naming conventions will help you to understand that if a column or property name starts with the word “Is” then you know right away that this is a derived or calculated value.) In our case, we are going to create a VIEW that will include all of the customer’s basic data and also a field indicating that the customer has paid within the past 30 days. To accomplish this, we will need to simple TABLEs.

First – The Customers TABLE

To start off easy, we are going to create a table to hold all of our customer data. We will call this table ‘Customers’. This table will contain an ID field of our customer as well as his name and address information. Our ID field will be the PRIMARY KEY for this table, and for now it can also be the CLUSTERED INDEX as well. (This is the default behavior if you are using the SQL Server 2000 or SQL Server 2005 IDE and you click the “Primary Key” button.) Here is our Customers table definition:

Second – The PaymentHistory TABLE

Now that we have our simple Customers table, we are going to create they PaymentHistory table. This will keep track of all payments for all customers. The fields we are going to need are an ID field for each PaymentHistory record, the CustomerID to tie back to the customer and a DateTime field to let us know when the payment was made. Here is the SQL statment to create the PaymentHistory table.

Joining The Tables Together

This is the PaymentHistory table that will be used to track when a customer has payed his bill, and will therefore have another 30 days of service. Notice that we didn’t use the default clustered index on the pimary key column for this table. The reason for this is that we will be joining to this table using the CustomerID field. By clustering this table first on CustomerID then on ID we will gain significant performance boosts when running joining or running aggregates using CustomerID as a parameter. We also chose to add the ID column as the second part of the clustered index so that our events will sort in order of payment history by default. This technique is discussed in another article on SingingEels.com – “SQL Performance – Clustered Indexes”.

What remains is for us to create a view that ties the two tables together. Our goal is to have all the customer information plus we want to know if the customer has paid within the last 30 days which indicates his being ‘active’. There are a few good ways to do this; we will look at 2 options and discuss the differences.

Our first method of tying the data together won’t involve using a JOIN statement. What we are going to do is simply select all the Customer data and add one column that will be the most recent payment date for the particular client. This will be accomplished using the MAX function to get the latest date and using a reference to the “parent” table (which in this case would be the Customers table) to ensure we only get the latest record for the specific customer in each record.

This method above is fast and easy to do. The problem here comes in when you want more than one piece of data from the PaymentHistory table. To do that, you wouldn’t want to do another sub-query to the same table as this is now adding needless processing on the SQL side. Instead, to pull back multiple fields from the PaymentHisotry table we are going to use a JOIN. Keep in mind though that we don’t want all the records for each customer, but rather all we want is the latest record. This slight modification to the code above will allow us to get the last payment date as well as the ID of the payment record.

In order to get the latest payment data for each customer we made an inline sub-query inside of our JOIN clause. This subquery basically acts just like a VIEW. In fact, if you made that query into a VIEW and called it LatestPayments, then you would be doing the exact same thing that SQL Server is doing in the background. What remains now is to use this logic to make our “IsActive” column based on whether or not the customer has made a payment within the past 30 days. This will be done using a CASE statement.

This query above reads this way in english: “Get all the records from the customers table, and when the latest payment for the customer is less than 30 days old, then the customer is active, otherwise he is inactive.” Now you can make this query into a view and display it on a customer management page on your web application. You could also use this view to dis-allow users from accessing certain pages due to no longer being an active member of your site.

This pattern is easy to follow and it gives much flexibility with minimal code. You could adapt this to work with a “case management” solution where you need to track case status changes, a customer payment system or many other scenarios.

Source:singingeels.com

The reason why I say that the @@IDENTITY and SCOPE_IDENTITY() options are limited is because 1) they only return numeric fields (so “uniqueidentifier” fields won’t work), 2) they can only return the field marked as the table ‘identity’ (so you can’t get a different field that you may be using as the ID), 3) they cannot give back any other auto-generated data such as a timestamp field. Another huge failing is in the case of multiple inserts in one statement. The @@IDENTITY and SCOPE_IDENTITY() options will only return the last record insereted. So if your insert statement added five records, you would only get the ID of the last one.

Intruducing The OUTPUT Clause

SQL Server 2005 intruduces a really amazing new clause that can be used in DELETE, UPDATE and INSERT statements called ‘OUTPUT’. The syntax is easy to follow as you basically just append it to the end of your statement and treat it like a SELECT. This small bit of SQL will show how to get the ID column back from an INSERT using SCOPE_IDENTITY() compared to the OUTPUT clause.

At first glance, these three examples above may all seem equal. They all take the same amount of code, and they all return the identity field of the record that was just inserted. However, if we change the desired results slightly we can see the major failings of the first two, and we’ll see why the OUTPUT clause is so powerful.

Let’s say instead of just wanting to return the new ID to the user, we want to add the EntryDate field to the results that we send back to the user. With the first two options you would have to store the ID in a variable and re-query the table causing much unneeded overhead and code. Whereas with the OUTPUT clause, getting these results are very simple.

Using @@IDENTITY or SCOPE_IDENTITY()

Using The OUTPUT Clause

If that is all the OUTPUT clause could do, it would be worth using. But keep in mind that you can also use it to return multiple results if your insert was more than one record. Another great benefit of the OUTPUT clause is that you can use it for DELETEs and UPDATEs as well. So if you wanted to delete some records based on user input and then show them in a GridView what records have been removed, you could do it in one clean step.

The syntax for UPDATEs is a little different, but even more powerful. There is no “UPDATED” object, but rather you can use “DELETED” or “INSERTED” to represent the list of columns before the change and after the change respectively. Example:

While these other flavors of SQL implimented pagination only for the sake of pagination – meaning the absolutely only reason for the functions is to limit results sets into ‘pages’ of data – SQL Server 2005′s ranking functions provide a lot of power and information. However, for this article, we are only going to demonstrate how to use these abilities for the sake of pagination.

RowNumber In SQL

First of all, we’ll need to get the “row number” from a result set. This is the basis of pagination because you need to know what row you are on to begin demanding only a subset from SQL. First we’ll do just a basic query and look at the results:

This query above will display the following results:

Now we’ll make a slight modification to add a “RowNumber” field.

This syntax may look a little odd, but it’s not that difficult. First of all, the ROW_NUMBER() function (just like all other ‘ranking functions’ in SQL SERVER 2005) needs an OVER clause. Notice that we moved the ORDER BY clause inside the OVER clause. Basically what this is doing is telling SQL how we want the ROW_NUMBER() function to number each record. So we are telling SQL to number the records 1, 2, 3, 4 in the order of LastName then FirstName. Here are the results:

So now that we have a “RowNumber” field, we can add a simple WHERE clause to get only a limited number of records. We will have to wrap query in another SELECT statment so that we can use all of our dynamic columns (such as our RowNumber column) by their names that we assigned them. Also, I’m going to use the TOP clause to limit the query to only return 2 records at a time. Lets see the query, and the results:

Notice I ‘aliased’ the first query and called it “_myResults”. This basically treats that inner query as a VIEW so that I could pull out that first column by it’s name, RowNumber. Here are the results:

Conclusion

Though it may seem like you have to do a lot more work in order to get pagination in SQL Server 2005, the pay-off is that not only does the code make sense, but you gain a lot more power with the ranking functions TSQL. Try playing around with the parameters to see how they affect your queries and what you can do with them.

Source:singingeels.com

The best way to learn how to use SQL properly is by using a real world problem and then solving it step by step. The scenario we’re going to work with is an ecommerce website. Let’s keep it simple and say that this site sells t-shirts, hats and mugs. They have a small customer base, and they want to try to learn more about their audience, so let’s build our reports.

Our Example Schema

We need to quickly define the schema (or design) of our database. This way you’ll be able to understand the queries below as we piece together our reports. We are going to have a few tables:

• Customers
• Products
• Orders
• OrderItems

Here is what each table will look like:

I realize that’s a lot of SQL, and you might not be used to building tables from scratch, but it’s ok if you don’t understand that code above right now. Basically, the schema is there so that you can refer to it as you continue the article to see how / why we are doing things.

SELECTing Basic Data

Our first report is going to be a basic one: who are our customers. Lets examine the query that would give us a list of our customers:

There’s not much to this query. If you didn’t already know, the * (STAR or asterisk) tells SQL to return all of the columns in the table. Let’s change this up a bit by putting the columns in the order we want.

So I’ve done two things here; 1) I put the LastName column in front of the FirstName column, and 2) I decided not to show the ID column. If you specify the columns manually, you have the power to move them around or leave them out all together. There is one problem though with this query above. Just because we put the LastName column first, doesn’t mean our results are going to be in ORDER we want. So let’s now change the query a bit to sort the results alphabetically by a customer’s last name, then first name.

Getting Some Useful Data

OK, so far this information isn’t very useful. In fact, you can’t really call it a report as it’s just a list of your customers. So we are going to step up our query a bit. We are going to add a column that will give you the number of all the orders that each customer has ever ordered. How you might ask? Just watch:

Do you see how simple that was? We added a “subquery” right inside of our main query. We used the COUNT function which will do exactly as it sounds, count how ever many rows meet your criteria. The criteria in this case (our WHERE clause) said to count all the records in the Orders table where the CustomerID field matched the ID field in the Customers table in our main query.

If you are new to subqueries (or sub-selects), or if you’ve heard before that this is a performance nightmare, don’t worry. This is not hard to understand, and there is no performance problem. The performance questions belong in another article, but as for trying to understand what SQL is doing, here goes.

If you are a programmer, you can think of your query as a “for loop” and the sub-select is like a function that is being called inside the loop to get a piece of data from somewhere else. But if you’re not a programmer you can think of it like shopping for a car. As you look at each car you turn to the salesman next to you and ask “how much does this one cost?” SQL is basically doing the same thing. It’s looking through one table, but you told it to get a piece of information from another table for each record.

JOINing Multiple Tables Together

The last piece of the puzzle that we are going to cover is JOINing two (or more) tables together to get relational data. We want to see all of the sales that were made by a customer. We want to know when the order was made, and what items were bought. To do that, we are going to join the Orders table (which has the customer’s ID and the order date) and the OrderItems table (which has the actual products that were purchased).

This is a simple query, but again, if you’re not familiar with JOINs then spend some time looking it over. Let’s examine the query backwards. What that query is doing is it’s finding all the orders for customer “2″. Then, it will find all the order items that belong to each of his orders. Finally, we tell SQL what columns to display (in our SELECT clause at the top).

The results will look something like this:

Let’s extend this query a little bit more to JOIN to another table, the Products table. This will give us the friendly name of the products that were ordered instead of just the ProductID. Watch how easy it is to get a little more information in our report:

Before I show you the result, I want to touch on something I did above. I “aliased” (or renamed) the column “Name” and called it “ProductName”. This is because a column called “Name” in this report might be misleading, but ProductName let’s you know exactly what to find in that column.

Here is the result:

The End – For Now

This is the end of the first segment of the “Understanding SQL” series. You should have learned something if you are new to SQL, or even if you’ve been using it for a little while. This article (and the next few to follow in this series) is designed to help you think like a database, and not like a human.

There is still a lot left under the topic of SELECTing (or querying) data. There are more complex joins, nested subqueries, aggregated results and so much more. We’ll cover those topics in the next “Understanding SQL” article.

Source:singingeels.com

If you are not very familiar with SQL and you haven’t already done so, please read Understanding SQL: SELECT The Data You Want. It will give you a bit more than the basics and you should be able to understand where we take things in this article. Also, we are going to continue with the “ecommerce web site” example from that article.

What is a GROUP BY

Before we can go on, you need to know what a GROUP BY clause is. To put it simply, you can tell SQL how to “GROUP” records together. For example, if you wanted to know how many people had the same name, you could make a query:

That would return a result set of all unique names in your Customers table, and how many times each name is found. Let’s look at another query to help us understand what a GROUP BY is used for in SQL.

This query is telling SQL that you want to know the highest ID of all users if they were grouped together by their names. So all the Bobs would get together, and whoever had the highest ID would come forward in the result set. Likewise, all the Timothys and all the Jonathans would do the same thing. Now let’s continue on to more advanced queries.

On The Fly Views

Often times you may need to treat a small portion of a TABLE as if it were its own unique TABLE (or VIEW). But creating a VIEW for one query is a bit ridiculous. A good “for instance” would be if you wanted a report that displayed the latest purchase made by all customers. You wouldn’t want to simply JOIN the Customers table to the Orders table as you would get all orders for every customer, not just their latest.

You have a few options here: Full JOIN then aggregate subqueries in the WHERE clause or a CTE (Common Table Expression). But the solution we are going to choose is the CTE, and I’ll explain why in a bit. First of all, let’s look at the first option (which is the option likely to be taken by most people). Here is what most people would do:

What they don’t realize is that every column that they have to add to the GROUP BY clause is going to cause more and more strain on SQL. Think about it, as each record comes in, SQL has to check to see if their is already a record with that same ID, FirstName, LastName, Address, City, State and Zip. And imagine if there were more columns in the Customers table; you’d be killing SQL for no reason.

Now like I said above, a CTE can treat a part of a table as if it were it’s own unique table. Meaning, wouldn’t it be nice if we had a table of only the most recent orders for every customer? Lets see how we can make a CTE that has the data we need, and include it in our query so that we don’t have to GROUP BY so many fields.

You may think that this query is more expensive (harder on the processor) than the one above using the massive GROUP BY, but it’s not. In fact, it’s about 100% faster and less processor intensive. If I added more records and got to have a large table, the CTE approach could be hundreds of times faster.

Something to keep in mind though is that the CTE is only available in SQL Server 2005, so if you are using SQL Server 2000 you have to do this the old way. The query will actually ‘cost’ the same to SQL (because it’s really the same thing), but it will look a little uglier. Here it is using a subquery in the WHERE clause.

While this may seem easier than the CTE, you should remember that I’m keeping the query simple for this article. The CTE is a lot cleaner for another developer to come behind because you define your “fake table” at the top of your query, so people know right away what you are trying to do. Also, you give it a name such as “LatestOrders” which explains what you are using it for. Lastly, you can put it in your JOIN clause just like a real table.

Concluding Comments

One thing of note is that a CTE can only be used once in your query. So you can’t declare your CTE at the top, then do multiple queries against it. However, you can make multiple CTE’s and then use them together in one query. Also, everything has it’s place, so if you find that you are building the same CTE multiple times for different reports / queries, then you might want to turn that into a VIEW instead.

If you’ve read the previous article mentioned above, and now this one, you should be able to make some useful queries (or reports). Try to build your queries logically, first explaining to yourself what you want the result to be. Then piece by piece add the code together until you reach the end result. If you find that by adding another JOIN or column to your output suddently breaks what was coming together nicely, then remove it and rethink what you did.

Here is a final example report that you should be able to build with the information in these articles. We’ll make a “Customer Activity Report” that will show the latest purchase date, and how many items have been purchased total for each customer. Here’s the code:

Your results should look something like this:

Source:singingeels.com

There are many situations that would call for a many-to-many schema. You might have a table of “Stores” and a table of “Managers” and your company might to allow managers to switch between stores. This would require that a store could have more than one manger, and a manger could belong to more than one store (thus, many to many). Or you could consider this scenario:

You’re building an application for a customer support center that takes phone calls for a cable company. When a customer calls in to complain about their fuzzy T.V. reception and also they want to upgrade their high-speed internet access, the customer service representative will need to open up two “tickets”; one for the fuzzy reception problem, and one for the high-speed upgrade.

In this scenario we are dealing with two things: phone calls and support tickets. So when the customer calls back the next day about the internet upgrade, we will want to associate that phone call with the second ticket we opened. Likewise, when they call back again the next day about the fuzzy reception, we want to associate that phone call with the first ticket we opened. Let’s see how we can accomplish this many-to-many relationship in SQL. We’ll learn:

• How to create tables that support a many-to-many relationship.
• What indexes should be created for the best performance.
• How to report by phone calls or by tickets (either side of the relationship).

How to Make a Many-to-Many Schema

The first piece in making our tables is to define the two entities (or tables) and then we can join them together by their IDs in a third “linker” table. For the example we are going to use we’ll need to very simple tables: PhoneCalls and Tickets. We’ll keep it simple so as not to get lost in the illustration, but instead to focus on the many-to-many relationship concept. Here is the schema of the two tables:

If we only wanted to support one-to-one functionality we would simply put a “PhoneCallID” field in the Tickets table and make the rule that you can’t have a ticket without a phone call. But instead, we want to allow multiple tickets to be created on one phone call, and we want to allow many phone calls to be associated with a single ticket. To do that we’ll need to build this linker table:

That’s all we need to make our many-to-many schema work. First we would insert a record into the PhoneCalls table. Then, when the caller explains the situation, the customer service rep would enter a record into the Tickets table and the system would insert a link (the ID of the phone call and the ID of the ticket) into the linker table. Eventually though this table is going to be big, and will begin to perform slowly in views and reports. Let’s now look at how to make the perfect indexes for performance on both sides of the relationship:

Perfect Many-to-Many Indexes

Without a proper understanding of SQL indexes, most people would either add an arbitrary column to the linker table and make that the primary key (which doesn’t help us at all), or they might make a composite primary key of PhoneCallID and TicketID. This index will help a lot; it makes sure that there are no duplicates and it gives us a lot of speed when searching or joining to the PhoneCalls table. But what if we wanted to query by the TicketID?

The problem with just one index when dealing with a many-to-many linker table is that you are only tuning performance for one side of the relationship (which makes sense: one index only helps one side). So how do we add an index that will give us the best performance possible? Well, a regular index is ok, but nothing beats a unique index in terms of speed. Here’s the trick!

A linker table (a table of IDs from two other tables for the purpose of a many-to-many relationship) is always unique whether you look at it from left to right or right to left. Meaning, PhoneCallID_TicketID is unique and TicketID_PhoneCallID is also unique, so we can add another unique index to the table!

How Does Another Unique Index Help?

Well, think about your table as if it was a library filled with thousands of books. How would you find a book called “SQL Performance Tips”? Well, you would use the library’s index and skip ahead to the letter “S” and continue in alphabetical order until you found your book. But what if you wanted to find a book by the author “Billy McGee”? You can’t use the index that you were just in because that is sorted by book title (which doesn’t help you because you only know the author’s name). So you would use the library’s other index, the one that is sorted by author’s name.

The same is true for SQL. If you do a query like this:

Then SQL will realize that it should use the PhoneCallID_TicketID index to find your results. But if you change the query only slightly to do this:

Then SQL will understand that the first index doesn’t help, so it will use the TicketID_PhoneCallID index to find your results. That’s how to achieve the best performance with many-to-many relationships.

How Do I JOIN These Tables Together?

When you are dealing with JOINing more than two tables in a query it can sometimes get confusing. So when you’re dealing with a many-to-many query (which requires three tables), it helps to first break up your FROM/JOIN clauses and explain them out loud to yourself as you type. The JOINs should make sense to SQL and to you. Let’s look at a query that would JOIN both of our tables together with the help of the linker table.

You can also use the linker table in your WHERE clause. An example of this would be if you wanted to see all calls associated with a ticket, or all tickets associated with a call. Here is how you would achieve those two queries:

Closing Remarks

Now that we have examined the facts we realize that many-to-many relationships are not difficult to build, and they can perform very well when indexed properly. Also, if you name your tables appropriately, then you’re queries will also be easy to read. So the next time you are designing a database schema for one of your projects, try to squeeze in some of this powerful functionality. Remember, you can easily support a one-to-many relationship with a many-to-many schema, but you can’t go the other way around. So if you think you might need it, why not support it from the ground up.

Source:singingeels.com