Mind

http://www.facebook.com/photo.php?pid=1602042&l=73b1b85027&id=663340749

(I had to get clearance to use my camera in the building!)

Oh yeah, I also touched something that has been in orbit (briefly). WOO. It was a fragment of the honeycombed aluminium casing that shields the final stage during its ascent.

Back from Toronto. More details later. Have another thing to go to tonight.

http://www.thrfeed.com/2009/03/stewart-vs-cramer-video-daily-show.html

(This is the unedited interview)

Advance warning: This is fairly simple, but very geeky post.

One of the supposed advantages of the EF is that it abstracts you from your database implementation. This is considered to be a good thing. It lets me move between different database products (SQL Server, Oracle, MySQL, etc) without having to rewrite anything the business tier of my application. I just need to hook up a different provider to the entity framework and I'm done. All of my business logic and query construction stays at the application layer.

Ironically, this paradigm, which comes from the object oriented world, fails to recognize one of the most basic tenents of object oriented design. The separation of interface from implementation!

First of all, I'd better point out the reason for this post. Why is it a bad thing to put the query construction into automated tools like the entity framework or nHibernate, or into the hands of the application layer more generally?

The short answer is performance. Databases aren't given much respect in many organizations. It's just a place to persist information, right? No, of course not, that's completely wrong. A database is an extremely complex system of physical constructs like indexes, partitions, columns composed of data types, keys, etc. SQL databases also have a standardized (though vendor extended) language for data retrieval and modification. This, of course, is SQL.

SQL is not a procedural language. SQL is a declarative language. Procedural programming fu do not translate to SQL-fu, and vice versa.

A person focused on database development generally knows at least a little something about database design, indexes, partitions, keys, and of course SQL. There's a lot to know. I've been doing it for 7 years and still learn new and complex technical details about SQL Server every week, if not every day. A person focused on database development also, as a rule, knows a little something about SQL.

A procedural programmer, as a rule does not. Or at least, obviously someone who has spent 10 years mostly focused on, say, .NET won't know as much about it as someone who has spent the same amount of time focused on, say, SQL Server. The people who wrote an object-relational mapping tool might know a little, but they don't necessarily. And more importantly, the tool certainly doesn't know anything about your specific indexes, partitions, performance requirements, storage system, etc.

Long story short, this means that the ability to write sensible and efficient SQL for your specific database will only ever be possible by someone with knowledge of the logical and physical structure of that database, and never by an automated tool, or a tool which translates one language (like LINQ) to another (like SQL).

Now here's the key to this post. An ORM layer which dictates certain database requirements and builds your queries for you is COMPLETELY UNNECESSARY in order to achieve the goal of database vendor independence. The belief that it accomplishes this disregards the separation of interface from implementation.

Your database developers can, and should, do this for you by providing the application code with an API. From an OOAD point of view, this is simply the separation of public methods from private data, and has been around FOREVER. How do we accomplish this? With stored procedures, or routines. If we switch vendors the only thing we have to do is ensure that the database provides the same set of routines to the application layer. The code inside those methods is irrelevant to the API itself. I can build a GetCustomer routine (stored proc) in SQL. If a decision is made to move to Oracle, all that matters is that the GetCustomer routine is provided, taking the same parameters and returning the same results. The fact that internally it is built using PL/SQL instead of T-SQL is completely hidden from the application layer.

This does mean that the code INSIDE the routines may have to be rewritten. I say may because if standards compliant SQL is used, it doesn't. You can literally do a copy and paste. But the thing is, you probably want to rewrite that code anyway. Why? For the reasons I discussed above. The performance of your queries depends extremely heavily on your logical data model, your physical database structures, and requires a deep knowledge of same. That's precisely the problem with automated query generation tools... they don't. Your performance can be orders of magnitude worse if you don't tune your queries based on your structures, and vice versa.

Need a concrete example? What about nHibernate. GUID's are usually needed for object creation, so each object gets it's own unique GUID which is then used for lookup. Using GUIDs in a database has the potential for MASSIVE performance loss. I'm talking orders of magnitude on query times, as well as vastly (VASTLY) increasing the physical size of the database on disk. Why? The reasons are complex, which is exactly the point. Care in how they are used is required to avoid this. But do the application programmers know the pitfalls? No, they're application programmers! They don't have to know anything about databases, that's what nHibernate is for, right?

Here's a few things nHibernate doesn't know about that a true SQL developer would (or should), all of which impact database size, or speed, or concurrency, or all of these:

Logical index fragmentation
Physical index fragmentation
Clustered index structures
Non clustered index structures
Index selectivity
Covering indexes
Natural vs surrogate keys
Primary and foreign keys
Key lookups
Heaps
Corelated subqueries
join vs subquery performance
Partitioning
Hotspots
Locking mechanics
Optimizer mechanics
Optimizer hints
Page and extent structures
Temporary database objects
Variable length data types

Tonight I went to the monthly Microsoft presentation. The speaker was Julie Lerman, talking about the Entity Framework, about which she has just written a book.

As a database guy, I was a little taken aback by this technology. It really steps on the toes of database design and DBA's.

But what was really good was that a few of us, including the speaker and organizers, went out for a drink or two afterwards. Now, I've discussed various different aspects of my work with people who have written books on the subject a few times, but usually on usenet. Never in person over a drink or two. That was great.

I've been offered that .NET position.

*shrug*

Had an interview today for a .NET position.

Boy did I screw that up. Here's how stupid I was. There was a C# coding question to be done on paper that required a test of the equality between 3 different variables, and I wrote it as

x == y == z instead of x == y && y == z.

:/

I didn't realize it until I was on the bus and headed home. I don't think I've ever felt quite this stupid and pathetic before. I can certainly explain it as a combination of nerves and a habit of using paper for pseudocode only, but that doesn't really make me feel any better about it. There's no way I'll get the position, but I still had to send off an email saying that I realized what I'd done. I still won't be able to sleep tonight, I've got this horrible knotted stomach feeling that won't go away at least until I get a reply to my email.

This is the kind of thing that haunts me. And I mean for years. I now realize I am in fact afraid of something. Terrified, even. I'm terrified of being wrong.

There was also a complex question at the end about navigating a maze. A question of logic, not syntax. I couldn't think of the solution (apparently only two people had ever gotten it and both had been hired), and it really bugged me. I walked out of the building and thought of the answer almost immediately, and had to go back in and tell them.

Nerves! We hates them!

The daily show and the colbert report are back on. I almost didn't survive the break.

With decent music too!

The codex of alchemical engineering

Tutorial here (not a walkthough, just a video instruction on how to play):

http://www.youtube.com/watch?v=xPqUQtfe9-4

Notes that aren't clear in the tutorial either:

Two claws can't be holding an element or compound at the same time even if they're not moving.

You can drop atoms on the ground to pick up later.

You don't need to let go of an element and then pick it up again when using a changing component. Just passing over the correct position is enough.

Fun ones so far that you can load. These aren't the solutions with the shortest programs or fewest pieces, they're just the ones I've made that seem elegant to me.

Haematite:
p,9,11;b,6,9;m,9,13,180,1,crolww;m,10,13,0,2,cdloru;m,7,11,270,1,rwwclo;m,3,13,180,1,crolwwwwwwww;m,3,9,90,1,wwcluwdorwww;m,6,6,90,2,wcrr11ollwww;

Brimstone:
c,1,11;c,13,11;t,3,9;b,6,11;t,9,11;t,11,9;m,6,13,180,1,wucdrolwww;m,7,7,270,3,owwrrwwcrr;m,3,11,90,1,crrudruodr;m,11,11,90,1,cllluudodl;