If you are reading this article, you should already have a working knowledge of SQL and are familiar with filtering your data with a WHERE clause, or limiting your result set from joining other tables with an ON clause.

The part where this article comes in is when you start to see a significant amount of time passing while your once simple query is now hitting your production tables which are multitudes bigger than your test data.

This article is not just for those of you who are seeing your queries start to take 30 seconds or more, but everyone could probably benefit from this simple article.

First of All – What is Slow?

In my opinion, 98% of the queries you are running should fire in less than 1 second. That may sound bold (and perhaps to some even foolish) but in truth if you know what you are doing your queries themselves should be very fast and your slow down will be your network traffic if you are hitting a different server for your data.

I have created and worked with databases ranging from just a few thousand records to the hundreds of millions. And my “lookup” queries for finding records (whether customers or products or nearby store locations) all take sub-second. The key, is not just a good index… but rather, a good set of indexes. And more importantly the right index for the right job.

They Clustered my What?

If you are using Microsoft Enterprise Manager for SQL Server 2000 databases, or Microsoft SQL Server Management Studio for 2005 databases, or even just using Visual Studio for small SQL Express databases (such as the one used for this website) then you will likely notice the “Primary Key” button that sets your currently selected column into the primary key column for your table. A primary key is a “unique” index, but it also defaults to being a “clustered” index in the above mentioned IDE’s and probably more.

Why is that important to know? For starters, you only get one clustered index per table. Secondly, your clustered index will also determine the default sorting of your select statements. It also can be your biggest performance loss when you are inserting or updating data in that table.

If the data in your table will rarely change and you use it basically to join to get a friendly name, or to filter out records not in a certain group, then you would just leave the clustered primary key alone. Even if your table changes fairly often such as a “store locations” table where you might have a few thousand stores across the country and they change their phone numbers or address etc., you would still likely just leave the index alone.

So When Do I Cluster?

If you had a table filled with customer data, and you need to find a record based on “Last Name” and “Zip Code” then you might find yourself waiting a good while (or worse yet timing out) while your query is being processed. Now is the time to remove the “cluster” from your primary key field (remember you only get one cluster so you have to remove it from the primary key). Now that you have done that you can create a new index, and of course you are going to set it as a “clustered” index. The columns, in this order, that you will add is “LastName,FirstName”. Now you might be asking “why didn’t we cluster ‘LastName,ZipCode’ since that is what we are searching on?”

That’s a great question, and the answer is this: You really only need to cluster “LastName” as that will be more than good enough to drop your search query down to sub second even if you have millions of records. Adding “Zip Code” to your search will simply help you limit the result set for someone searching for ‘Smith’ which would return just too many records to look at. So, we know we don’t need to add “ZipCode” to the cluster, but why did we add “FirstName”? The answer to that is something I mentioned above. And I’ll explain now.

Default Sort Benefits

Your clustered index will tell SQL Server how to sort results given to you by default. So if you do a “SELECT * FROM dbo.MyTable” then you’d simply get all the records from that table as they were entered. But, if you had a clustered index on “LastName,FirstName” then you’d get all the records sorted by “LastName” then “FirstName”. So in our scenario above of the ‘Customer Search’ we now get the benefit of displaying the results to the user for his search in alphabetical order which helps him to find the record he wanted a lot faster.

There are a lot of other benefits of clustered indexes, and there is so much to go over, but for now I’ll leave it at that. Remember your primary key column is probably defaulted as “clustered” which is fine for most scenarios, but you might want to change it one day. Also, keep in mind that you only get one clustered index per table, so use it wisely. Lastly, keep in mind that fiddling with clustered indexes could effect your INSERT or UPDATE code, so make sure to check the performance of these if it is a factor for you.

Source:singingeels.com

Since we will also want to keep a complete history of the customer’s payments, we can’t just add a single column to our table called ‘IsActive’. (Also, as a side point, proper naming conventions will help you to understand that if a column or property name starts with the word “Is” then you know right away that this is a derived or calculated value.) In our case, we are going to create a VIEW that will include all of the customer’s basic data and also a field indicating that the customer has paid within the past 30 days. To accomplish this, we will need to simple TABLEs.

First – The Customers TABLE

To start off easy, we are going to create a table to hold all of our customer data. We will call this table ‘Customers’. This table will contain an ID field of our customer as well as his name and address information. Our ID field will be the PRIMARY KEY for this table, and for now it can also be the CLUSTERED INDEX as well. (This is the default behavior if you are using the SQL Server 2000 or SQL Server 2005 IDE and you click the “Primary Key” button.) Here is our Customers table definition:

Second – The PaymentHistory TABLE

Now that we have our simple Customers table, we are going to create they PaymentHistory table. This will keep track of all payments for all customers. The fields we are going to need are an ID field for each PaymentHistory record, the CustomerID to tie back to the customer and a DateTime field to let us know when the payment was made. Here is the SQL statment to create the PaymentHistory table.

Joining The Tables Together

This is the PaymentHistory table that will be used to track when a customer has payed his bill, and will therefore have another 30 days of service. Notice that we didn’t use the default clustered index on the pimary key column for this table. The reason for this is that we will be joining to this table using the CustomerID field. By clustering this table first on CustomerID then on ID we will gain significant performance boosts when running joining or running aggregates using CustomerID as a parameter. We also chose to add the ID column as the second part of the clustered index so that our events will sort in order of payment history by default. This technique is discussed in another article on SingingEels.com – “SQL Performance – Clustered Indexes”.

What remains is for us to create a view that ties the two tables together. Our goal is to have all the customer information plus we want to know if the customer has paid within the last 30 days which indicates his being ‘active’. There are a few good ways to do this; we will look at 2 options and discuss the differences.

Our first method of tying the data together won’t involve using a JOIN statement. What we are going to do is simply select all the Customer data and add one column that will be the most recent payment date for the particular client. This will be accomplished using the MAX function to get the latest date and using a reference to the “parent” table (which in this case would be the Customers table) to ensure we only get the latest record for the specific customer in each record.

This method above is fast and easy to do. The problem here comes in when you want more than one piece of data from the PaymentHistory table. To do that, you wouldn’t want to do another sub-query to the same table as this is now adding needless processing on the SQL side. Instead, to pull back multiple fields from the PaymentHisotry table we are going to use a JOIN. Keep in mind though that we don’t want all the records for each customer, but rather all we want is the latest record. This slight modification to the code above will allow us to get the last payment date as well as the ID of the payment record.

In order to get the latest payment data for each customer we made an inline sub-query inside of our JOIN clause. This subquery basically acts just like a VIEW. In fact, if you made that query into a VIEW and called it LatestPayments, then you would be doing the exact same thing that SQL Server is doing in the background. What remains now is to use this logic to make our “IsActive” column based on whether or not the customer has made a payment within the past 30 days. This will be done using a CASE statement.

This query above reads this way in english: “Get all the records from the customers table, and when the latest payment for the customer is less than 30 days old, then the customer is active, otherwise he is inactive.” Now you can make this query into a view and display it on a customer management page on your web application. You could also use this view to dis-allow users from accessing certain pages due to no longer being an active member of your site.

This pattern is easy to follow and it gives much flexibility with minimal code. You could adapt this to work with a “case management” solution where you need to track case status changes, a customer payment system or many other scenarios.

Source:singingeels.com

The reason why I say that the @@IDENTITY and SCOPE_IDENTITY() options are limited is because 1) they only return numeric fields (so “uniqueidentifier” fields won’t work), 2) they can only return the field marked as the table ‘identity’ (so you can’t get a different field that you may be using as the ID), 3) they cannot give back any other auto-generated data such as a timestamp field. Another huge failing is in the case of multiple inserts in one statement. The @@IDENTITY and SCOPE_IDENTITY() options will only return the last record insereted. So if your insert statement added five records, you would only get the ID of the last one.

Intruducing The OUTPUT Clause

SQL Server 2005 intruduces a really amazing new clause that can be used in DELETE, UPDATE and INSERT statements called ‘OUTPUT’. The syntax is easy to follow as you basically just append it to the end of your statement and treat it like a SELECT. This small bit of SQL will show how to get the ID column back from an INSERT using SCOPE_IDENTITY() compared to the OUTPUT clause.

At first glance, these three examples above may all seem equal. They all take the same amount of code, and they all return the identity field of the record that was just inserted. However, if we change the desired results slightly we can see the major failings of the first two, and we’ll see why the OUTPUT clause is so powerful.

Let’s say instead of just wanting to return the new ID to the user, we want to add the EntryDate field to the results that we send back to the user. With the first two options you would have to store the ID in a variable and re-query the table causing much unneeded overhead and code. Whereas with the OUTPUT clause, getting these results are very simple.

Using @@IDENTITY or SCOPE_IDENTITY()

Using The OUTPUT Clause

If that is all the OUTPUT clause could do, it would be worth using. But keep in mind that you can also use it to return multiple results if your insert was more than one record. Another great benefit of the OUTPUT clause is that you can use it for DELETEs and UPDATEs as well. So if you wanted to delete some records based on user input and then show them in a GridView what records have been removed, you could do it in one clean step.

The syntax for UPDATEs is a little different, but even more powerful. There is no “UPDATED” object, but rather you can use “DELETED” or “INSERTED” to represent the list of columns before the change and after the change respectively. Example:

While these other flavors of SQL implimented pagination only for the sake of pagination – meaning the absolutely only reason for the functions is to limit results sets into ‘pages’ of data – SQL Server 2005′s ranking functions provide a lot of power and information. However, for this article, we are only going to demonstrate how to use these abilities for the sake of pagination.

RowNumber In SQL

First of all, we’ll need to get the “row number” from a result set. This is the basis of pagination because you need to know what row you are on to begin demanding only a subset from SQL. First we’ll do just a basic query and look at the results:

This query above will display the following results:

Now we’ll make a slight modification to add a “RowNumber” field.

This syntax may look a little odd, but it’s not that difficult. First of all, the ROW_NUMBER() function (just like all other ‘ranking functions’ in SQL SERVER 2005) needs an OVER clause. Notice that we moved the ORDER BY clause inside the OVER clause. Basically what this is doing is telling SQL how we want the ROW_NUMBER() function to number each record. So we are telling SQL to number the records 1, 2, 3, 4 in the order of LastName then FirstName. Here are the results:

So now that we have a “RowNumber” field, we can add a simple WHERE clause to get only a limited number of records. We will have to wrap query in another SELECT statment so that we can use all of our dynamic columns (such as our RowNumber column) by their names that we assigned them. Also, I’m going to use the TOP clause to limit the query to only return 2 records at a time. Lets see the query, and the results:

Notice I ‘aliased’ the first query and called it “_myResults”. This basically treats that inner query as a VIEW so that I could pull out that first column by it’s name, RowNumber. Here are the results:

Conclusion

Though it may seem like you have to do a lot more work in order to get pagination in SQL Server 2005, the pay-off is that not only does the code make sense, but you gain a lot more power with the ranking functions TSQL. Try playing around with the parameters to see how they affect your queries and what you can do with them.

Source:singingeels.com

The best way to learn how to use SQL properly is by using a real world problem and then solving it step by step. The scenario we’re going to work with is an ecommerce website. Let’s keep it simple and say that this site sells t-shirts, hats and mugs. They have a small customer base, and they want to try to learn more about their audience, so let’s build our reports.

Our Example Schema

We need to quickly define the schema (or design) of our database. This way you’ll be able to understand the queries below as we piece together our reports. We are going to have a few tables:

• Customers
• Products
• Orders
• OrderItems

Here is what each table will look like:

I realize that’s a lot of SQL, and you might not be used to building tables from scratch, but it’s ok if you don’t understand that code above right now. Basically, the schema is there so that you can refer to it as you continue the article to see how / why we are doing things.

SELECTing Basic Data

Our first report is going to be a basic one: who are our customers. Lets examine the query that would give us a list of our customers:

There’s not much to this query. If you didn’t already know, the * (STAR or asterisk) tells SQL to return all of the columns in the table. Let’s change this up a bit by putting the columns in the order we want.

So I’ve done two things here; 1) I put the LastName column in front of the FirstName column, and 2) I decided not to show the ID column. If you specify the columns manually, you have the power to move them around or leave them out all together. There is one problem though with this query above. Just because we put the LastName column first, doesn’t mean our results are going to be in ORDER we want. So let’s now change the query a bit to sort the results alphabetically by a customer’s last name, then first name.

Getting Some Useful Data

OK, so far this information isn’t very useful. In fact, you can’t really call it a report as it’s just a list of your customers. So we are going to step up our query a bit. We are going to add a column that will give you the number of all the orders that each customer has ever ordered. How you might ask? Just watch:

Do you see how simple that was? We added a “subquery” right inside of our main query. We used the COUNT function which will do exactly as it sounds, count how ever many rows meet your criteria. The criteria in this case (our WHERE clause) said to count all the records in the Orders table where the CustomerID field matched the ID field in the Customers table in our main query.

If you are new to subqueries (or sub-selects), or if you’ve heard before that this is a performance nightmare, don’t worry. This is not hard to understand, and there is no performance problem. The performance questions belong in another article, but as for trying to understand what SQL is doing, here goes.

If you are a programmer, you can think of your query as a “for loop” and the sub-select is like a function that is being called inside the loop to get a piece of data from somewhere else. But if you’re not a programmer you can think of it like shopping for a car. As you look at each car you turn to the salesman next to you and ask “how much does this one cost?” SQL is basically doing the same thing. It’s looking through one table, but you told it to get a piece of information from another table for each record.

JOINing Multiple Tables Together

The last piece of the puzzle that we are going to cover is JOINing two (or more) tables together to get relational data. We want to see all of the sales that were made by a customer. We want to know when the order was made, and what items were bought. To do that, we are going to join the Orders table (which has the customer’s ID and the order date) and the OrderItems table (which has the actual products that were purchased).

This is a simple query, but again, if you’re not familiar with JOINs then spend some time looking it over. Let’s examine the query backwards. What that query is doing is it’s finding all the orders for customer “2″. Then, it will find all the order items that belong to each of his orders. Finally, we tell SQL what columns to display (in our SELECT clause at the top).

The results will look something like this:

Let’s extend this query a little bit more to JOIN to another table, the Products table. This will give us the friendly name of the products that were ordered instead of just the ProductID. Watch how easy it is to get a little more information in our report:

Before I show you the result, I want to touch on something I did above. I “aliased” (or renamed) the column “Name” and called it “ProductName”. This is because a column called “Name” in this report might be misleading, but ProductName let’s you know exactly what to find in that column.

Here is the result:

The End – For Now

This is the end of the first segment of the “Understanding SQL” series. You should have learned something if you are new to SQL, or even if you’ve been using it for a little while. This article (and the next few to follow in this series) is designed to help you think like a database, and not like a human.

There is still a lot left under the topic of SELECTing (or querying) data. There are more complex joins, nested subqueries, aggregated results and so much more. We’ll cover those topics in the next “Understanding SQL” article.

Source:singingeels.com

If you are not very familiar with SQL and you haven’t already done so, please read Understanding SQL: SELECT The Data You Want. It will give you a bit more than the basics and you should be able to understand where we take things in this article. Also, we are going to continue with the “ecommerce web site” example from that article.

What is a GROUP BY

Before we can go on, you need to know what a GROUP BY clause is. To put it simply, you can tell SQL how to “GROUP” records together. For example, if you wanted to know how many people had the same name, you could make a query:

That would return a result set of all unique names in your Customers table, and how many times each name is found. Let’s look at another query to help us understand what a GROUP BY is used for in SQL.

This query is telling SQL that you want to know the highest ID of all users if they were grouped together by their names. So all the Bobs would get together, and whoever had the highest ID would come forward in the result set. Likewise, all the Timothys and all the Jonathans would do the same thing. Now let’s continue on to more advanced queries.

On The Fly Views

Often times you may need to treat a small portion of a TABLE as if it were its own unique TABLE (or VIEW). But creating a VIEW for one query is a bit ridiculous. A good “for instance” would be if you wanted a report that displayed the latest purchase made by all customers. You wouldn’t want to simply JOIN the Customers table to the Orders table as you would get all orders for every customer, not just their latest.

You have a few options here: Full JOIN then aggregate subqueries in the WHERE clause or a CTE (Common Table Expression). But the solution we are going to choose is the CTE, and I’ll explain why in a bit. First of all, let’s look at the first option (which is the option likely to be taken by most people). Here is what most people would do:

What they don’t realize is that every column that they have to add to the GROUP BY clause is going to cause more and more strain on SQL. Think about it, as each record comes in, SQL has to check to see if their is already a record with that same ID, FirstName, LastName, Address, City, State and Zip. And imagine if there were more columns in the Customers table; you’d be killing SQL for no reason.

Now like I said above, a CTE can treat a part of a table as if it were it’s own unique table. Meaning, wouldn’t it be nice if we had a table of only the most recent orders for every customer? Lets see how we can make a CTE that has the data we need, and include it in our query so that we don’t have to GROUP BY so many fields.

You may think that this query is more expensive (harder on the processor) than the one above using the massive GROUP BY, but it’s not. In fact, it’s about 100% faster and less processor intensive. If I added more records and got to have a large table, the CTE approach could be hundreds of times faster.

Something to keep in mind though is that the CTE is only available in SQL Server 2005, so if you are using SQL Server 2000 you have to do this the old way. The query will actually ‘cost’ the same to SQL (because it’s really the same thing), but it will look a little uglier. Here it is using a subquery in the WHERE clause.

While this may seem easier than the CTE, you should remember that I’m keeping the query simple for this article. The CTE is a lot cleaner for another developer to come behind because you define your “fake table” at the top of your query, so people know right away what you are trying to do. Also, you give it a name such as “LatestOrders” which explains what you are using it for. Lastly, you can put it in your JOIN clause just like a real table.

Concluding Comments

One thing of note is that a CTE can only be used once in your query. So you can’t declare your CTE at the top, then do multiple queries against it. However, you can make multiple CTE’s and then use them together in one query. Also, everything has it’s place, so if you find that you are building the same CTE multiple times for different reports / queries, then you might want to turn that into a VIEW instead.

If you’ve read the previous article mentioned above, and now this one, you should be able to make some useful queries (or reports). Try to build your queries logically, first explaining to yourself what you want the result to be. Then piece by piece add the code together until you reach the end result. If you find that by adding another JOIN or column to your output suddently breaks what was coming together nicely, then remove it and rethink what you did.

Here is a final example report that you should be able to build with the information in these articles. We’ll make a “Customer Activity Report” that will show the latest purchase date, and how many items have been purchased total for each customer. Here’s the code:

Your results should look something like this:

Source:singingeels.com

There are many situations that would call for a many-to-many schema. You might have a table of “Stores” and a table of “Managers” and your company might to allow managers to switch between stores. This would require that a store could have more than one manger, and a manger could belong to more than one store (thus, many to many). Or you could consider this scenario:

You’re building an application for a customer support center that takes phone calls for a cable company. When a customer calls in to complain about their fuzzy T.V. reception and also they want to upgrade their high-speed internet access, the customer service representative will need to open up two “tickets”; one for the fuzzy reception problem, and one for the high-speed upgrade.

In this scenario we are dealing with two things: phone calls and support tickets. So when the customer calls back the next day about the internet upgrade, we will want to associate that phone call with the second ticket we opened. Likewise, when they call back again the next day about the fuzzy reception, we want to associate that phone call with the first ticket we opened. Let’s see how we can accomplish this many-to-many relationship in SQL. We’ll learn:

• How to create tables that support a many-to-many relationship.
• What indexes should be created for the best performance.
• How to report by phone calls or by tickets (either side of the relationship).

How to Make a Many-to-Many Schema

The first piece in making our tables is to define the two entities (or tables) and then we can join them together by their IDs in a third “linker” table. For the example we are going to use we’ll need to very simple tables: PhoneCalls and Tickets. We’ll keep it simple so as not to get lost in the illustration, but instead to focus on the many-to-many relationship concept. Here is the schema of the two tables:

If we only wanted to support one-to-one functionality we would simply put a “PhoneCallID” field in the Tickets table and make the rule that you can’t have a ticket without a phone call. But instead, we want to allow multiple tickets to be created on one phone call, and we want to allow many phone calls to be associated with a single ticket. To do that we’ll need to build this linker table:

That’s all we need to make our many-to-many schema work. First we would insert a record into the PhoneCalls table. Then, when the caller explains the situation, the customer service rep would enter a record into the Tickets table and the system would insert a link (the ID of the phone call and the ID of the ticket) into the linker table. Eventually though this table is going to be big, and will begin to perform slowly in views and reports. Let’s now look at how to make the perfect indexes for performance on both sides of the relationship:

Perfect Many-to-Many Indexes

Without a proper understanding of SQL indexes, most people would either add an arbitrary column to the linker table and make that the primary key (which doesn’t help us at all), or they might make a composite primary key of PhoneCallID and TicketID. This index will help a lot; it makes sure that there are no duplicates and it gives us a lot of speed when searching or joining to the PhoneCalls table. But what if we wanted to query by the TicketID?

The problem with just one index when dealing with a many-to-many linker table is that you are only tuning performance for one side of the relationship (which makes sense: one index only helps one side). So how do we add an index that will give us the best performance possible? Well, a regular index is ok, but nothing beats a unique index in terms of speed. Here’s the trick!

A linker table (a table of IDs from two other tables for the purpose of a many-to-many relationship) is always unique whether you look at it from left to right or right to left. Meaning, PhoneCallID_TicketID is unique and TicketID_PhoneCallID is also unique, so we can add another unique index to the table!

How Does Another Unique Index Help?

Well, think about your table as if it was a library filled with thousands of books. How would you find a book called “SQL Performance Tips”? Well, you would use the library’s index and skip ahead to the letter “S” and continue in alphabetical order until you found your book. But what if you wanted to find a book by the author “Billy McGee”? You can’t use the index that you were just in because that is sorted by book title (which doesn’t help you because you only know the author’s name). So you would use the library’s other index, the one that is sorted by author’s name.

The same is true for SQL. If you do a query like this:

Then SQL will realize that it should use the PhoneCallID_TicketID index to find your results. But if you change the query only slightly to do this:

Then SQL will understand that the first index doesn’t help, so it will use the TicketID_PhoneCallID index to find your results. That’s how to achieve the best performance with many-to-many relationships.

How Do I JOIN These Tables Together?

When you are dealing with JOINing more than two tables in a query it can sometimes get confusing. So when you’re dealing with a many-to-many query (which requires three tables), it helps to first break up your FROM/JOIN clauses and explain them out loud to yourself as you type. The JOINs should make sense to SQL and to you. Let’s look at a query that would JOIN both of our tables together with the help of the linker table.

You can also use the linker table in your WHERE clause. An example of this would be if you wanted to see all calls associated with a ticket, or all tickets associated with a call. Here is how you would achieve those two queries:

Closing Remarks

Now that we have examined the facts we realize that many-to-many relationships are not difficult to build, and they can perform very well when indexed properly. Also, if you name your tables appropriately, then you’re queries will also be easy to read. So the next time you are designing a database schema for one of your projects, try to squeeze in some of this powerful functionality. Remember, you can easily support a one-to-many relationship with a many-to-many schema, but you can’t go the other way around. So if you think you might need it, why not support it from the ground up.

Source:singingeels.com

First of all, it’s important to realize what LINQ is and what it isn’t. LINQ is a language feature you’ll see in Visual Studio 2008 (Visual Studio Orcas) and beyond. It’s not dependant on any particular .NET language as you can use it in C# or VB.NET. LINQ stands for “Language Integrated Query”, but its usage goes beyond language capabilities alone.

It is true that the language speeds up development by allowing you to use a simple query-syntax in C# or VB.NET to query data from a HashTable, XML file or SQL Database. And it’s also true that you can have Visual Studio 2008 write tons of code for you to build classes for your data tables (or even to build SQL tables from your classes). However, this article will demonstrate runtime performance gains you’ll see by using some built in features that come with the language as well.

The benefits we’ll discuss are:

• Intelligent query translations (such as faster data pagination).
• Redundant query checking to prevent needless data hits.
• Redundant update checking to ensure minimal data transfers.

Intelligent Query Translations

One of the built-in benefits of using LINQ is that methods that are deemed “best practice” are used in the underlying technologies. For instance, if you were to use a standard ASP.NET 2.0 SqlDataSource, and you bound that data to a GridView with pagination turned on, you might not realize that you’re transferring your entire data table just to use a few rows at a time in your paginated GridView. This results in some heavy queries, heavy data transfers and often times slow page loads.

The best place to do pagination would be on the SQL server itself. That way you are only transferring the few rows that are desired, saving you much network bandwidth and time. Ultimately, LINQ isn’t doing anything that you couldn’t do yourself, but most developers do not know how to do Pagination In SQL Server 2005, or they may not have time to change all of their queries to do so.

Again, if you take that simple “SqlDataSource” example above, but this time bind two ASP.NET controls to it (let’s say a GridView and a DropDownList), then you’ll be hitting your SQL server twice, and download all that data twice. This brings us to the next performance gain in LINQ.

Redundant Query Checking

Some of the biggest performance gains that I’ve benefited from by switching to LINQ is it’s ability to look at the “context” of your request to determine if it needs to hit the database again. If you’ve already started to use LINQ yourself, but you don’t know what I’m talking about, then you’ll probably have a new appreciation for LINQ in a few minutes.

LINQ uses a “DataContext” object to determine what connection string to use, what objects to map to and other internal things. But you should be careful not to think of the LINQ data context class as you would think of a traditional “SqlConnection”. If you are transitioning from using SqlConnections and SqlCommands you’ll probably code something like this:

If you translated the above method into LINQ, then your method might look something like this:

This at first seems like an acceptable conversion from using a SqlConnection to using LINQ, but you’re missing out on so much (which you’ll see in a minute).

This is pretty much a 1 to 1 switch from writing your own SQL query to writing a query in LINQ. It is noteworthy that it takes a lot less code to query a database. However, these two code blocks will do the same thing, and you won’t gain any performance from calling the LINQ method instead of calling the straight SQL method. Even if you call both of these functions twice in a row, you won’t gain any performance because we are not utilizing the LINQ context correctly.

If we extend the range of our context to last for the entire web request (not just for the life of the function call) then we will begin to see performance increases. Before we examine how to do that, let’s look at this next example that should help explain how the context works. We’ll change our above method, “GetMyData”, so that we can pass in our own context. Here’s the code:

The way to achieve the performance benefits of using the same data context during the entire page request is to store your data context instance in the HttpContext Items collection (which is alive for the duration of the thread). We’ll make a static property that will give us a data context instance, but because we are using the current HttpContext, we won’t have cross thread issues. You can read a little more about the HttpContext in this article: Connect Your Site With HttpContext Items. Let’s look at the code to see how to achieve this:

Let’s now look at the last area of performance gain from using LINQ that we will discuss.

Redundant Update Checking

The last place where you can see performance gains is with writing back to your data source. The LINQ data context not only remembers if you have already queried some data before, but it also keeps track of any changes that are made. This means that if you call the “save” function, but no changes were made first, then LINQ will ignore the request to save. Likewise, if you only change one property (such as “FirstName”), then LINQ will build an UPDATE statement that only contains that one column (again saving time and bandwidth).

That’s a Wrap!

New technologies may sometimes seem dauting (or even pointless). Hopefully this article has helped you to appreciate the clear benefits of using LINQ. Download Visual Studio Orcas (beta 2 at this point) and try out some of the examples in this article. I hope you grow to love LINQ as I have.

Source:singingeels.com

The Need for LINQ

At first, when I didn’t understand LINQ, I couldn’t appreciate why we needed (or why anyone wanted) it. LINQ seemed very limited in usefulness, and I, like many of my colleagues, bashed it in ignorance. Now, due to understanding the semantics and benefits of the language, I’ve come to love it and wonder how I lived without it.

LINQ does not allow us to do anything new, as far as functionality goes. Really, if you wanted to make the argument of a ‘real programmer’, you could do all of this same functionality by writing encyclopedias of C code (heck, might as well just write it in assembly for “performance” reasons). But, LINQ does create a new way of looking at enumerations of data, and gives out-of-the-box functionality for them.

I realize that the above paragraphs might seem vague, so I’ll get right into real examples that I have used as well as some odd examples that you might not have thought about as being ‘queryable objects’.

Baby Steps – Basic LINQ Queries

Let’s look at some very simple examples of LINQ so that we can have a basis to grow on. Let’s pretend that we have a user control that has a variable amount of TextBoxes added to it (based on some database query or whatever). Our user control might have other controls (Buttons, Literals and the like), and we want to disable all of the TextBoxes if the user doesn’t have security permissions to edit these controls. Please keep in mind that this is a hypothetical scenario at this point just so we can see some basic LINQ (in this case, LINQ to Objects).

Now, you can do this a few ways. Because the number of TextBoxes is dynamic we would possibly do something like this:

The above code isn’t really that bad, but I for one don’t like that I have to loop through all of my controls, check the type, cast it to the type that I just checked for, and then set my property. So, let’s see how LINQ will not only “pretty up” my code, but will also make more sense to us programmers:

This LINQ approach is much cleaner. It’s clear from the start that you want to work with a list of TextBoxes only. Secondly, you aren’t type checking, then re-type casting. Now, you might argue that you could avoid the second type-cast by using an “as” keyword and checking for nulls, or you could say that you don’t like how many lines of code this takes. Those are some valid claims, but if you are thinking that, then apparently you have a short attention span (as I mentioned above that this is just to get you a little familiar with the language syntax).

Step Two – Querying Unconventional Data Sources

Once you begin to use LINQ a bit more in you’re C# (or VB), you’ll begin to think of ‘data sources’ in a different way. For instance, you may see a directory (a folder on your computer) as a data source of files. You can even start to look at your list of running processes as a data source. Really, no enumeration is safe!

What if you were to make a web application that allowed you to upload and download files from your home machine? Let’s say you had a DropDownList that would show a list of the files in the specified directory, perhaps with a button that lets you download it. How would you go about populating that DropDownList with clean file names for the “Text” property and perhaps the path for the “Value” property? I’ll go a step further and say that this is what you’re DropDownList looks like in ASP.NET:

Well, you could create you’re own custom class that has a FileName property and a FullPath property. Then you could make a generic list of that class, loop through the files in a Directory.GetFiles method, add an entry for each file, set the list as the DataSource of the DropDownList and data bind. That’s a whole lot of nasty that I’m not even going to show the code for (really, because I never want to write nasty code like that again). Here’s what I would do using LINQ:

All I can say after that is “Go LINQ GO!” and of course “case closed!”

Extension Methods and Lambda Expressions

One more part to LINQ is the heavy use of Extension Methods and Lambda expressions to accomplish this beautiful language enhancement. If you remember our original example of disabling the TextBoxes, then you’re going to love one of the 50 STANDARD OPERATORS that are brought to you just by including a reference to the System.Linq namespace! Check this out:

Don’t check you’re MSDN… there is no method called “OfType” on the ControlCollection class. But, because the ControlCollection class implements the IEnumerable interface, it automatically gets all of the extension methods that LINQ has kindly given to any enumeration (such as “OfType” which I am using above).

LINQ can be done one of two ways (or a combination of both): query expressions or method expressions. The difference between the two is only a syntax difference, as they ultimately compile into the same thing. Example:

Lambda expressions come into play when you want to perform another standard operation such as “SUM” or “MAX”. Here’s an example that I have used recently. I wanted to make a property that displayed the “TotalBill” of an object that had child “orders” in it. Here’s how lambda helped out:

If you’re not quite familiar with lambda as of yet, you can get a better explaination of the above in a blog post I made not long ago: LINQ, Lamda and Extension Methods Work Together!

Conclusion

There is much more to say about LINQ, but I hope for now you have a basic understanding and appreciation for it. As I get more time, I’ll talk about some deeper subjects with LINQ such as how to use it in an enterprise level application and how to gain performance and improve multi-developer cohesion with it.

Source:singingeels.com

First, it’s important to explain what I mean when I say that the GridView doesn’t have sorting capabilities. You’re probably saying to yourself – “I know the GridView has sorting built in because it has a ‘Sort’ method and ‘Sorting’ and ‘Sorted’ events.” And that’s true, but those are merely placeholders that allow you to write your own sorting code.

Also, if you use the designer to build your GridViews, and bind them to SQL data sources, then you might also be confused about the above statements. But rest assured that the sorting magic is all happening in the terribly inefficient SqlDataSource control, and not the actual GridView itself.

If you’re not already familiar with the basics of LINQ, then you might want to first read this article: Learn The Basics of LINQ. To sum up LINQ in one sentence (for the purpose of moving on) I would say: “LINQ is a language feature in .NET that provides a ton of out-of-the-box functionality for querying data.”

The Problem – Why The GridView Can’t Sort

So why is it that the GridView control doesn’t have the ability to sort by default? The problem lies in the fact that the data source is very dynamic, could be of any type and could even be just a paged subset of the ultimate source.

Another problem is performance. If your data source was from a SQL table, then you would want to pass the sorting functionality on to the SQL Server itself to gain the most performance. Because the GridView doesn’t know what your source will be, it can’t really provide the needed functionality.

The Solution – LINQ

Since LINQ provides querying capabilities on a generic level, the above problems are no problem. Because LINQ to objects can handle *any* enumerable objects, and LINQ to SQL is smart enough to pass the sorting logic on to SQL Server, you win no matter what.

For example, imagine if the data source was a collection of MembershipUsers (via the Membership.GetUsers method). Here’s how easy it would be to sort using LINQ and the OrderBy method:

That was very easy to do, but that’s because we know the data type of our source at compile time. But if we are going to make a generic “self sorting GridView”, then we have to build our functionality with the limitation of not knowing the data type of the data source or the name of the field that is being sorted. So we’ll need to reconstruct this functionality at runtime by building “expression trees” (the power behind LINQ).

Step One – Build Our GridView Control

To begin with, let’s build a basic control that inherits from GridView , and stub out a placeholder to put our sorting logic.

Now that we have our custom GridView control ready to go, we can move on to step two.

Step Two – Creating an Expression to Sort With

As was stated earlier, we need to build an expression that will tell LINQ how we want to sort our data source. There are a few tricky issues here that we need to identify and overcome. If we try to copy the example found in this blog post to reproduce our sorting example at the top of this article, we would get half-way there:

So we’ve solved one issue – that of finding the property to sort with by it’s name. The “Expression.Property” method above did that for us. Now, we have to solve the second issue – not knowing what the data type of the data source will be until runtime. To solve this in the fewest lines of code, I’m going to make a custom generic class that will create the sort expression for us so that we can simply supply the type at runtime.

Beautiful! Notice that because we are using a generic class, we can tell the compiler that we want to use the type “T” (whatever “T” will be at runtime, we don’t care)! So, if I wanted to create an instance of my GenericSorter class at runtime, passing in a type that I won’t know until runtime, and then call the “Sort” method, I would just do some simple reflection as will be shown below.

For simplicities sake, the data source is stored in a variable called “_data”. You can download the whole source code at the end of the article to see exactly how it’s done. Here’s a sample of the code:

Conclusion

At first glance, it may seem like we’re doing a lot of code just for sorting functionality. You may even be thinking that this is an extremely convoluted way of sorting a GridView, but really it’s not. Once you understand what LINQ is doing, and how to build your own expression trees, you’ll really appreciate this new language feature in .NET.

I’ll include a web project that has the full source code and some examples of different objects that can be sorted here. If you first don’t understand what all is going on, I’d recommend that you download the source code and step through it in the debugger.

I hope this article has excited you about LINQ and expression trees. If you’re not excited, I hope your informed. If you don’t feel informed, I hope at least that you’re not now afraid of LINQ.

Here’s the source code and sample project: SingingEels_LinqSelfSortingGridView.zip

Source:singingeels.com