ETL

ETL stands for Extract, transform, load as is described in wikipedia article.

I was reading many of Ayende Rahien posts about ETL, but never understood what's is that ETL about. Now I know it:

Extract, Transform, and Load (ETL) is a process in data warehousing that involves
* extracting data from outside sources,
* transforming it to fit business needs (which can include quality levels), and ultimately
* loading it into the end target, i.e. the data warehouse.


Basically what ETL stands is a process of getting oen source of information into destination. This might be an integration of two systems, say a CMS system and a Payment system, or that might be just importing data from spreadsheet into your database.


We have started doing our own implementation of import/export tool a month ago. I don't think this is a waste. The insight we gained from creating our own solution is very valuable, and now we have a choice of either stick with our solution, or use Rhino.ETL . Before that we had no choice, and that's bad

Rhino.ETL can be found here

A vision of enterprise platform Hot & Distributed Deployment

I am to lazy to write my own ideas, but I am not lazy to share _good_ ideas.

Ayende now has a month of a vision of enterprise systems I think it's really cool what he writes.

Please read it: A vision of enterprise platform Hot & Distributed Deployment

Scheduling solutions

Just for my own reference:
Scheduling solutions @ Ayende Rahien

He puts his opinion about four solutions:

* Castle.Components.Scheduling
* Quartz.NET
* Rhino Scheduling
* Task Manager

How to build a document plugin system

Currently we are building a documnt plugin system for our app.

Which way do you prefer to specify documents:
Example number 1:

[DocumentRepository]
public class DocumentRepository
{
 [Documents]
 public void RegisterSheets()
 {
  m_documents.Add(typeof(Client[]), "ClientBook.odt", "Books/second level book", "Client book", "Print client book");
   
  m_documents.Add(typeof(Contact), "ClientSheet.odt", "Sheets", "Contact sheet", "Print contact sheet");
 }
}

Or number two:

[DocumentAttribute(typeof(Client[]), "ClientBook.odt", "Books/second level book")]
public class PrintClientBookClone2 : BasePrintingCommand
{
 public PrintClientBookClone2() : base("Client book", "Print client book")
 {
 }
  
 protected override void Invoke(object sender, EventArgs e)
 {
  WindowOpener.Default.OpenDocument(new DLC.Core.UI.Documents.GenericBookDocument(this, DataRoot));
 }
}
 
[DocumentAttribute(typeof(Contact), "ClientSheet.odt", "Sheets")]
public class PrintContactSheetClone2 : BasePrintingCommand
{
 public PrintContactSheetClone2()
  : base("Client sheet", "print client sheet")
 {
 }
  
 protected override void Invoke(object sender, EventArgs e)
 {
  WindowOpener.Default.OpenDocument(new GenericDocumentForm(this, DataRoot));
 }
}

Both examples achieve the same functionality: two print buttons are created. One is placed under menu "Books/second level book/Client book", the other under "Sheets/Client sheet". Slash here denotes that the menu is composed of some sub-menus.

So which do you choose?

I choose both cases, since the first one is extremely short, and allows you to add documents very quickly. However, the other, while being cumbersome and ugly, allows you to achieve maximum flexibility in case you need it. And we know bussiness people need it alot.

Microsoft to release source code of part of .Net libraries

Today throughout whole world was sent a message that Microsoft will release the source code to the class libraries that make up .NET.

We'll begin by offering the source code (with source file comments included) for the .NET Base Class Libraries (System, System.IO, System.Collections, System.Configuration, System.Threading, System.Net, System.Security, System.Runtime, System.Text, etc), ASP.NET (System.Web), Windows Forms (System.Windows.Forms), ADO.NET (System.Data), XML (System.Xml), and WPF (System.Windows). We'll then be adding more libraries in the months ahead (including WCF, Workflow, and LINQ).

You can deep-scan slashdot and google for the details. I just want to state my opinion about this:

First, it is not an open source license at all. You can SEE the source, but you can not do anything with it. It's just for your eyes, studying, debugging, etc.

If there were programmers who did not get what are the differences between open source and free software, now can see it that here is a big coneyed meaning difference. In essence, open source and free software are 99 percent the same.

However, when people talk about free software, they tend to think that this is about
the cost of the program, which is not true. Free software is about freedom.

When business men talk about open source, they often convey deliberately or mistakenly that open source is all about source code, that is, if you see source code, then it is open source. That is SO untrue. Open source is also about freedom. You not only see source code, but also can modify it, ehance it, fix bugs and release the code to the world.

Sorry folks, but .Net libraries are not open source. But i still hope they sometimes will be.

Second, you can not copy-paste code from .Net libraries to your projects, since .Net source is not released under open source, or free software license.

Third, it still a good thing

Fourth, I strongly believe that Mono will soon reach such a level, that we could move away from .Net platfrom and go into Mono platform. Why to do this? Simple - I see bussiness value in it. And the value comes primarily from the ability to choose. With Mono, our customers will get more options as where to run their software. Should they choose Linux or Windows, it will run on both machines.

Fifth, open source profoundly increases both productivity and research in IT industry. Personally, I can not see myself programming with libraries, where I can not see the code, and all of our libraries except .Net framework are open source.

Why use NHibernate?

In NHibernate users forum i got a question

"To summarize, what I want is ability to construct queries using ICriteria API, and to bypass hydration - instead of real objects i want to get a raw sql result table"

Yeah for that you do want projection. But I'm wondering why you'd bother with NHibernate then? If you really, truly want to avoid all objects and want a table result-set, why not just use ADO.NET (or other database) directly?

Why do we use NHibernate

1. Surprisingly as it is, OOP is middle solution between functional programming and aspect programming. I want to use objects in my database aware application too. Unfortunately, object oriented databases still are not as evolved as relational databases. And it happens so that we are not working on Zope platform and cannot take advantage of ZODB (Zope Object DB). I want objects, not some random bits of data.

2. My first acquiantance with ORM solution was Neo .Net. The project is no longer in development, but it served very well for learning what is ORM. Neo is decent tradeoff of power and flexibility to ease of use and very low footpring of learning it.

If you happen to be new to ORM world, don't try NHibernate first (even with Castle.ActiveRecord), take some simple ORM tool, and use it. Get to know what the pros and cons are for ORM solutions. Then take NHibernate, since, IMHO, that's the only really flexible and powerful solution.

Here you can find NEO with generics support

3. Our way of programming is strictly object oriented. And having to work with objects allows us to make very incredible functionality in very short amount of time.

4. NHibernate is database agnostic. To explain this: NHibernate is not tied to any datbase. You create your model and the use it ony practically any databse exists. NHibernate is not a 80% solutions. It is 100% solution, and you can have big confidence that NHibernate will support database you need.

In our project, everything runs on FireBird, but tests run on SQLite. Unfortunately there's one problem with SQLite - it does not support database roles. Too bad.

5. NHibernate ir powerful. You can execute your direct sql against the databse, or use powerful ICriteria API. You can both fetch a result set containing object propertiess, and the objects themself.

6. NHibernate is free software, thus we have the source code. The free software license does not deny the right to bundle NHibernate library with your proprietary code.

Performance upgrade month 2

We use hibernate projections for a performance gain, and a very big performance gain.

Projections work this way: instead of loading real objects, we can tell NHibernate to load a specific property instead. So what we did was instead of loading an object with necessary objects pre-fetched, we build an ICriteria query and a projection for each property we need to display in the grid.

The gains are these:
1. We bypass hydration process, so that we get an array of columns, which we can databind to specific datagrid. Databinding process does not involve reflection code to extract data from objects, that's a side effect :)
2. We still have ICriteria interface to specify whatever filters we might need.
3. If one look at NHibernate projections, one would see that projections contains some nice things such as AVG, MIN, MAX, group by, COUNT etc. Do you see the possibilities? Tree views, additional helpful info for various reports, etc.

After making projections, so that it instead of loading 15k object graphs, we select only the properties we display in the grid, the time went down to 3-5 seconds from 15 seconds. That is, the performance gain was 3 to 5 times. That's alot.

---

Disclaimer
We still have many problems with performance, but at the moment the UI responsiveness is satisfactory.

Here are the other problems we have and what we could improve:

1. We use Sourcegrid as our datagrid solution. It is much much better than System.Windows.Forms.DataGridView, since it provides a rather solid MVC separation and extensibility mechanism. However, it has it's own problems. Some of them might be solved easily, and some might be rooted deep in the sourcegrid itself.

Yes, you guessed it. One of the problems is performance :). SourceGrid has a problem of managing many cells (i.e. > 50k). Maybe that might be solved using VirtualGrid (a class in SourceGrid), have not delved alot into, so can not tell much, now.

P.s. for 15k objects default SourceGrid solution might add 3 to 5 seconds delay solely for constructing cells.

2. Every time we edit an object, we create and destroy a form. Forms could be cached and reused.

3. Object loading might be put on a separate thread, and instead of loading whole 15k rows, a thread could small chunk first, display it in grid, then load the other rows. This would increase UI responsiveness.

4. Sometimes some forms fail to open in 3 seconds. This happens when for example object Group has a collection of Students (count > 50), a collection of Companies (count > 50), a collection of other stuff, etc. All those collections we put into separate tabs in the form. We could make that each tab load the data incrementaly, and make the form responsive to user actions as soons as it has minimum amount of data. I.e. something like ajax thing, but for desktop. We might call it Desktop 2.0 :D