A simple and handy workflow framework

Workflow plays a critical role in business applications, but very few use it (me included). I never used WF in my dev even though I believe workflow is a critical tool for ensuring quality code. The reason is obvious.
In case of WF, usually pain is more than gain. For a simple workflow you’ve got to deal with lots of not so interesting fluff.
The solution is obvious: Cut the crap!
I want to implement a tiny state machine suitable for developing workflows. No crab. Just a little dll to reference with a flexible design. Here comes the story.

Everybody talks about the on-off switch to describe state machine, so do I. Assume the following state machine. A lamp can be in the off or on state and a click will change the state. If it is on will become off and vise versa.

Here comes the code that is required to be implemented with WF state machine. Well, I don’t write the code, its a pretty long story so check the msdn article yourself.

So much crap wouldn’t convince me to use the state machine at all. I also hate to define events and a bunch of codes and classes for such a simple task. This is what I want to do:

var onOffWf = WorkFlow.Start(ILamp)
.State("Off",
	l => l.IsOn, //Enterance condition
	l => l.MakeOff()) //What happens at the state
.Then("On")
.State("On",
	l => k.IsOff, //Enterance condition
	l => l.MakeOn()) //What happens at the state
.Then("Off")

I want all the code to be in one place and also easy to read and understand.

One more thing that I want is to be able to easily save the state of my lamp and its workflow in with my object. For example I want to add a column to my Lamp table (lampId, WorkFlowState) which holds the stat of my lamp. So each time I load my lamp, I continue from where I was before. This is the code that I want to go on to my button OnClick() event:


void OnClick(...)
{
	MyLamp.MoveNext();
}

I also would like to bind the status of my lamp to its state as:

<TextBox ItemSource="{Binding Path=StatusTitle}"/>

onOffWf has a property onOffWf.StatusTitle.

Is that possible? Not with WF. If you know a way let me know quick because I am going to implement this and some time would be spent.

Automatic Mid-tier Cache!

I have been thinking about this for a while. I started to work on a Silverlight project last year which was a nifty little business app. One of those apps that is just sitting down and working, you know, a web server, an average DB with a few million records, and a silverlight client with a bunch of forms, grids, and menus.

Everything was straightforward and brainless, like just follow a pattern and do the job. However, I was deeply dis satisfied from one aspect of the project. The Caching. What I could not accept was that in 21st century, when you have Entity Framework and IQueryable, you should still manually cache your data when it is appropriate and do all the pointless work of cache invalidation and loading, etc. Apart from the pain of working on something that should have been automated, I wouldn’t trust a programmer to decide which parts of the data should be cached and which part shouldn’t be. Not that I don’t believe they can do a good job on that, they don’t have enough information (at the time of dev work) to decide on it.

Caching strategy should be based on user behaviour and is subject to change by passage of time. For example at some stage lots of QLD pharmacies are queried, but next week NSW users decide to get ready for their conferences and start hammering the system for NSW pharmacies.

Le me be clear about my expectations of a caching system. It should have the following charachteristics:

  1. It should know what users are going to query a lot and cache that (and only that) part of the database.
  2. It should be able to re-use the caches. For example if I say 1.”Give me all QLD pharmacies”, and next one says 2.”Give me all QLD Chemists Warehouses”, the cache manager should be smart enough to run this new query 2., over the results of query 1. which has been retrieved a few minutes ago.
  3. It should optimize the indexes for performance based on the user queries.
  4. It should change the cache when user behaviour changes.
  5. It can call back the database only if there is absolutely no way of answering the query from cache.

Above requests seems to be a lot, but not really in 2011. All these methods are possible, in fact DBMSs do those kind of stuff for ages. We also have IQueryable, which makes it even easier to have a decent caching system.

So let me write a few examples:

Q1: Pharmacies.Join( … Address …).Join( … State …).Where( s => s.Sate = “QLD”).Select(…)

Q2: Pharmacies.Join( … Address …).Join( … State …).Where( a =>a.Sate = “QLD” && a.PostCode > 4000 && a.PostCode<4079).Select(…)

Q3: Pharmacies.Join( … Address …).Join( … State …).Where( s => s.Sate = “QLD”).GroupBy(…).Where( pg => pg.Count() > 4).Select(…)

Q4:  PharmacyStaff.Where(ps => ps.Position == “Manager”).Select(…)

Q5: Pharmacies.Join( … Address …).Join( … State …).Join(…PharmacyStaff…).Where( s => s.Sate=”QLD” && s.Position == “Manager” ).Select(…)

Users login to our system  and do stuff that will cause the above queries to be issued. Normally they will all be issued against the database, but it means that our caching strategy is stupid as a donkey. Really what I would expect is that only Q1 and Q4 are ran against the database. Q2, Q3, and Q4 are all subsets of Q1, hence if we already have those results from Q1, such a waste to run these new queries against the database. Why not look at the Expression Tree and figure out that Q2 is forms a query which is a subset of Q1. Then change the queries as below:

Q1: not changed…

Q2: Q1.Where(a => a.PostCode > 4000 && a. PostCode<4079).Select(…)

Q3: Q1.GroupBy(…).Where(…)

Q4: not changed…

Q5: Q1.Join(…Q4…).Select(…)

Check out the above queries. Aren’t they much better. We don’t expect user or programmer to waste his time on translating those queries. The caching system should do that. It should be an IQueryable that reads the ExpressionTree and translates it into a new ExpressionTree that uses existing data in the cache if there is no need to query database.

This specially make  sense in CLOUD, where you have to pay for querying your SQL Asure.

Enough talking about the dreams, lets become realistic! I did a bit of research and as I expected no such caching manager exists (if you know some tell me and save my hair). So I decided to do it myself. Check the Auto-mid-tier-cache project which I have already started. I haven’t gone far with it yet. It is just a proof on concepts and it implements no IQueryable. It uses a set of objects defined by myself for Relational Algebra operators. It does the very basic of view-matching to find what query is subset of what other, and it is able to translates queries to run against the database or cache alternatively.

I ran it and it worked fine and a bunch of benchmark analysis proved its effectiveness. What is left now is to complete the view-matching and write an IQueryable on top of it. Lot of work but it is worth it.

I forgot to say that you can limit the cache size by setting cost upper-bound. Next issue is that it does not keep itself up-to-date, but this is really another story.