Nullsafe dereference operator (?.) in C#

Nullsafe dereference operator (?.) as exists in some languages is a handy and nice pattern which saves a few bugs. Null references are serious problems and as people say nearly 70% of all bugs in the industry are directly or indirectly related to “Object reference not set to an instance of an object!”

However, there is noway to completely avoid null references, unless you write in a fully functional programming language like HASCUL. Some object oriented languages like groovy support a notion of nullsafe dereference operator which helps your null checks to be simpler. Here is the syntax in groovy:

bossName = Employee?.Supervisor?.Manager?.Boss?.Name

The above code returns the Name of the boss of the manager of the supervisor of an specific employee. It checks if any part of the right operand is null. If employee has no supervisor or supervisor has no manager, etc. bossName will become “null”.

Writing the above code in C# becomes like this:

bossName = Employee != null && Employee.Supervisor != null && Employee.Supervisor.Manager != null && Employee.Supervisor.Manager.Boss != null ? Employee.Supervisor.Manager.Boss.Name : null;

Pretty ugly, isn’t it? We can make a bunch of C# functions that takes us as closer to the groovy syntax of groovy. Note that nullsafe operator is going to be used in many places and inside the loops and it is not efficient to define any class for just a null check. Anyway a simple function won’t add much overhead.

//Groovy:
bossName = Employee?.Supervisor?.Manager?.Boss?.Name
 //C#:
 bossName = Nullify.Get(Employee, e => e.Supervisor, s => s.Manager, m => m.Boss, b => b.Name);

 

This one looks much more beautiful and more readable and this the the Nullify (static) class:

 public static class Nullify
 {
 public static TR Get<TF, TR>(TF t, Func<TF, TR> f) where TF : class
 {
 return t != null ? f(t) : default(TR);
 }

 public static TR Get<T1, T2, TR>(T1 p1, Func<T1, T2> p2, Func<T2, TR> p3)
 where T1 : class
 where T2 : class
 {
 return Get(Get(p1, p2), p3);
 }

 /// <summary>
 /// Simplifies null checking as for the pseudocode
 /// var r = Pharmacy?.GuildMembership?.State?.Name
 /// can be written as
 /// var r = Nullify( Pharmacy, p => p.GuildMembership, g => g.State, s => s.Name );
 /// </summary>
 public static TR Get<T2, T2, T3, TR>(T1 p1, Func<T1, T2> p2, Func<T2, T3> p3, Func<T3, TR> p4)
 where T1 : class
 where T2 : class
 where T3 : class
 {
 return Get(Get(Get(p1, p2), p3), p4);
 }
 }
 

UPDATE:

Below is the alternative way by using extension methods. It is really your preference to decide which way is better. Above approach might have slightly less overhead since it just adds a single function call in the IL code and is more concise for short expressions, but the code is shorter in the bottom approach and visually looks better for larger expressions.


 public static TOut NullSafe<TIn, TOut>(this TIn obj, Func<TIn, TOut> memberAction)
 {
 //Note we should not use obj != null because it can not test value types and also
 //compiler has to lift the type to a nullable type for doing the comparision with null.
 return (EqualityComparer<TIn>.Default.Equals(obj, default(TIn))) ? memberAction(obj) : default(TOut);

 }

 //Usage:..

 Employee.NullSafe( e => e.Supervisor ).NullSafe( s => s.Boss ).NullSafe( b => b.Name );

 
Posted in C#. 6 Comments »

A generic Database Project using MEF (Managed Extensibility Framework) – Part 1

A database project as of Visual Studio template is a series of .SQL scripts that run sequentially and create the schema of the project. My experience shows that in most real world scenarios, this is not enough. I list generic requirements of a database project here and take further steps of developing one:

  1. There should be a choice to initially create database from the schema script or load it from a basic backup file.
  2. Pre-requisits on the server should be checked.
  3. Should be cross platform. Basically anything that works with running scripts (not only SQL) should be runnable on this.
  4. Should be extensible. (As any other project should be)

Let’s create the solution. A console application should be good for this. We create a new solution called SDP (Simple Database Project).

For the 4th point, we use MEF (Managed Extensibility Framework). MEF is an opensource project supported by Microsoft and I think is a great framework. In this project, we additionally use MEF for IoC (Inversion of Control). In order to use MEF, we add System.ComponentModel.Composition to our solution. Note that a version of MEF is shipped with .Net Framework and Silverlight 4.0 and you don’t have to download anything.

The first step is to generate (create or restore) the initial database. For this purpose we define IDatabaseGenerator which has a method GenerateDatabase(). Any database generator need to do something with a database system which can be a server like SQL Server or can be an application like excel. Whatever the database system is, we do not introduce the requirement of database connection, etc. to our project. To keep the project extensible and simple we assume everything is possible with running a script using some tool or API. Hence, let’s define IScriptExecuter which executes a given script. It has a method ExecuteScript(). Any script executer needs a list of scripts to execute, so let’s define the generic interface IScriptProvider<T>. We keep this interface generic since we don’t know what type of script should we provide. It has a method GetScripts() that returns IEnumerable<T>.

So let’s implement our classes to run a sql script with a commandline tool and generate a database. We need a context to put essentials together, so let’s define SdpContext class. Before that we should have a script class defined to use with our context:

public class Script: IComparable<Script>
    {
        public virtual string Path { get; protected set; }
        public virtual string Title { get; protected set; }
        public virtual int Order { get; set; }

        public int CompareTo(Script other)
        {
            return Order.CompareTo(other.Order);
        }

        public Script(String path, String title, int Order)
        {
            Path = path;
            Title = title;
            Order = Order;
        }
    }
     public partial class SdpContext
    {
        [ImportMany]
        public IEnumerable<Lazy<IDatabaseGenerator>> DbGenerators {get; set;}

        [ImportMany]
        public IEnumerable<Lazy<IScriptProvider<Script>>> ScriptProviders { get; set; }

        [Import(AllowDefault=true, RequiredCreationPolicy = CreationPolicy.Shared)]
        public IVersionController VersionController { get; set; }

        public SdpContext(Assembly[] plugins)
        {
            ComposeFromAssemblies(plugins);
        }

        public void ComposeFromAssemblies(Assembly[] plugins)
        {
            var container = new CompositionContainer();
            var catalog = new AggregateCatalog();
            foreach (var plugin in plugins)
            {
                catalog.Catalogs.Add( new AssemblyCatalog(plugin) );
            }

            container.ComposeParts();
        }
    }

You have spotted some new stuff in the creation of SdpContext class. SdpContext is supposed to be self sufficient, meaning that you it should run by itself. Ofcourse you need to feed it with a bunch of plugins.
SdpContext is a partial class because I wanted to seperate the basic functionality of the context with other extended functionalities.
Import and ImportMany attributes are MEF attributes. They mean that our context needs a bunch of DbGenerators and ScriptProviders. MEF Framework then magically extracts any DbGenerator and ScriptProvider from the plugins and instantiates them for us. They are all defined as Lazy to avoid their instantiation before when we really need them. The interface VersionController is new here and we still don’t know how it should look like.

What has Brisbane floods got to do with Database? And why computer science can save lives!

Last few days Brisbane and Ipswich faced devastating flood in some areas up to 21 m. This means that some houses have been more than 17 meters under water! I have been isolated in a virtual island with no utilities for more than three days, and I have been extremely lucky that flood water did not reach my house. The disaster was devastating in the scale of the Queensland state (as large as England) . One-third of Ipswich city was under water. Unfortunately, the disaster claimed 12 lives so far, but in the absence of technology, this could be far more.

In the University of Queensland, we have been dealing with negotiations of a database project which if has been done, the destructive effect of the project could be far less. The problem ( basically a data quality related issue and very hard to solve ) is the old challenge of schema matching.

There are thousands of water monitoring systems in Australia  all over the land. They are all managed separately  and do not have much in common in terms of  IT. One might have an excel sheet filled manually, another may use a linux tool  to generate a bunch of txt files from measurements, many Access databases or ORACLE and SQL Server etc. All with different designs an schemas. It is an extremely sparse sensor network system. Obviously standardising all the schemas is possible, but is out of our hands and might be a very lengthy and expensive process. So what we could do here in UQ was to develop a smart system that looks at all different thousands of databases and matches their schemas and provides a holistic unified view over everything. This is a classic problem which have been worked on (in Computer Science community not industry) for three decades.

Apparently this project did not go anywhere, but having it won and finished gracefully, all the water movements could be monitored in the matter of hours and flash floods could have been predicted and lives could have been saved!

Dive into Expression Trees

I am in the process of developing a framework for sampling query results from LINQ queries. It is going to be hard work, you may ask WTF in this world someone need sampling query results from LINQ queries? Well, this can be particularly useful if you are developing a new database that purely works with LINQ or you are having heavy data services that use various data sources then you want to sample the query result.

Unfortunately, I am not currently working on any core database system or heavy duty data services (at work), but I have to do this as part of my study. This project which is going to be an open source project called [I don’t know yet] is supposed to do the job of evaluating my algorithms that I hopefully (fingers crossed) will publish at VLDB 2011.

So if you continue to visit this blog, you will see posts about these topics:

  • LINQ Expression Trees
  • Writing Provider for LINQ
  • Sampling Techniques
  • Data quality metrics
  • and, Rule based profiling

 

Quick guild to extend SAF (Simple Authorization Framework)

This post is a quick guide to extend SAF.

1. To create new authorizer (e.g. Authorization Attribute):

Easiest way to create new attributes is to inherit from Grand, Deny, or Custom attributes.

Authorizers are not limited to attributes. In general any authorizer should implement IAuthorizationContainer which contains an IEnumerable of IPrincipleAuthorizer.

All you need to extent the authorizers is to implement IPrincipleAuthorizer with your custom authorization logic.

2. Extend Authorization Rule Provider. If you don’t want to use the provided attribute based authorization rules, you can implement IAuthorizationRuleProvide with your own rules. For example, you may want to use database to get your authorization rules from it. In this case you implement the GetAuthorizers() and GetPropertyAuthorizers() methods of the authorization to generate your IPrincipalAuthorizers from database.

3. Extend AuthorizationContext. Authorization context depends to two providers: It needs 1) a rule provider, and 2) a principal provider. The principal provider. You can extend the context by writing these providers.


Authorization Rule Library in SAF and how to use it with RIA Services and Silverlight

A problem of applying rules by decorating objects using attributes is that there is no concept of re-usability. For example, we want NationalAdmins, StateAdmins, and Managers to be have all permissions over everything. Also permissions for my customer, customer_address, customer_emails, etc. are all similar. Normally you should apply all authorization attributes to every part of the model.

SAF provides an ImportAuthorizationAttribute that solves this problem.

public class AuthorizationMetadata
{
    [Grant...]
    [Grant...]
    public class AdminGroup { }

    [Deny...]
    [Deny...]
    [Deny...]
    public class RestrictedWriteGroup {}

    [Grant...]
    public class TitleViewersGroup {}
}

You can use AuthorizationMetadata class to centralize all your authorization rules. You can then use it in your model as follows:

[ImportAuthorization(SourceType=AuthorizationMetadata.AdminGroup)]
public partial class Model
{
}

What else do you want?

Refactoring SAF to be de coupled from the Authorization Attributes

SAF framework look at the metadata type to figure out the authorization rules. Although this is exactly what we want, but is a design flow. The reason I think it is a design flow and needs re-factoring is that it makes the rules to be defined in code.

Let’s imagine we want to seperate the role of Authorization rule definers from the programmers. Although, it is possible now, to implement IMetadataTypeProvider to load some external dll, but it is still bound to types which is not really necessary.

This means, the Grant and Deny attributes should not be limited to be attributes. It is possible with a small re factoring of the current code.

Let’s define a rule provider: IAuthorizationRuleProvider which retrieves rules for a given object. Grant and Deny are IPrincipalAuthorizer. We can give the IAuthorizationRuleProvider instead of to IMetadataProvier to the methods in PermissionHelper.

Then we need to define the AttributeAuthorizationRuleProvider which has a method GetAuthorizers and relies on an IMetadataProvider.

	public class AttributeAuthorizationRuleProvider<T> : IAuthorizationRuleProvider
	{
		private MetadataProvider _metadataProvider;

		public AttributeAuthorizatonRuleProvider(MetadataProvider meta)
		{
			_metadataProvider = meta;
		}

		public IEnumerable<IPrincipalAuthorizer<T>> GetAuthorizers(Type type)
		{
			var meta = metadataProvider.GetMetadataType(type);

            //Get all attributes for the type
            return meta.GetCustomAttributes(false).OfType<IPrincipalAuthorizer<Permission>>();
		}
	}

We later need an AuthorizationContext as a singleton like:

	public class AuthorizationContext
	{
		IMetadataProvider _metadataProvider;
		IAuthorizationRuleProvider _authorizationRuleProvider;
		public AuthorizationContext(IMetadataProvider meta, IAuthorizationRuleProvider rule)
		{
			_metadataProvider = meta;
			_authorizationRuleProvider = rule;
		}

		//Wrap all the methods in Permission Helper.
	}

Later, we can define a new authorizationRuleProvider called SqlAurhorizationRuleProvider that uses database for authorization rule and in database we can have a table like:
TABLE Authorizations (AuthID, Type, Property)
TABLE AuthorizationRules (RuleID, AuthID, RuleType (Grand/Deny/Custom), ConditionCode, ConditionQuery)
TABLE AuthorizationRuleAssignments (RuleID, Role)
In my providers I avoid using the traditional Provider model because the complexity is not required here and using IoC and a Context object, eliminates the role of Manager for every single provider.

Simple Authorization in RIA Services and Silverlight using SAF-Framework

SAF is designed with simplicity and extensibility in mind. All you have to do to use it in RIA-Services and Silverlight  to manage end-to-end security is not much. The following example shows how to use SAF to manage Authorization in a project with RIA services and Silverlight.

We have an Australia wide shop. We use ASP.Net membership provider. And three Roles: Customer, StateAdmin, and NationalAdmin.

1. In your data model:

Assign security attributes:

	[MetadataType(typeof(Metadata)]
	public partial class ShoppingItem
	{
		[Grant(Roles=new[]{"StateAdmin"}, Permissions = Permission.All)]
		[Grant(Roles=new[]{"NationalAdmin"}, Permissions = Permission.All)]
		[Deny(Roles=new[]{"StateAdmin"}, Permissions = Permission.All, ConditionMethod = "NotInState")]
                [Grant(Roles=new[]{"Customer"}, Permissions = Permission.View)]
		[Deny(Roles=new[]{"Customer"}, Permissions = Permission.View, ConditionMethod = "NotInState")]
		public class Metadata
		{
			public static bool NotInState(object instance, IPrincipal principal)
			{
				var user = principal as CustomUser;
				var shoppingItem = instance as ShoppingItem;
				//You should check for null in production code
				if (!user.ActivityStates.Contains(instance.State))
					return true;
				return false;
			}

			[Deny(Roles=new[]{"Customer"}, Permission.View)]
			public decimal PriceThatWeBought { get; set; };

			[Deny(Roles=new[]{"StateAdmin"}, Permission.Edit)]
			public decimal PriceThatWeSell {get; set; };
		}
	}

Above code means that you have a ShoppingItem in your model, and following security rules apply:

  1. Customers only can view items and can not change anything.
  2. State admins have all the rights over items in their states of activity. They have no access to anything in other states. (They can view other state items only if they have Customer role as well and this is because the grant view to customer if after deny to the StateAdmin)
  3. NationalAdmins can do anything
  4. Customers can not see the PriceThatWeBought
  5. SateAdmins can not change the PriceThatWeSell because CEO only trusts himself.

As you can see writing the above rules in code is as easy as describing them in English.

2. At your Service Level:

You have three methods. One to get a specific shopping item. Second to get all shopping items. And third to save a shopping item. Here is all the code you have to write:

	public partial class MyDomainService
	{
		public Tuple<ShoppingItem, AuthorizationToken> GetShoppingItemById(int id)
		{
			DomainContext.ShoppingItems
				.Where( s => s.Id = id)
				.Select(s => s)
				.FilterUnAtuthorizedWithToken()  //What user should not see will be scapped here.
				.First();
		}

		public IEnumerable<Tuple<ShoppingItem, AuthorizationToken>> GetShoppingItems()
		{
			DomainContext.ShoppingItems
				.Select(s => s)
				.FilterUnAtuthorizedWithToken()  //What user should not see will be scapped here.
				.First();
		}

		[RequiresPermission(Permissions.Write)]
		public SaveShoppingItem(ShoppingItem item)
		{
			DmoainContext.SaveChanges();
		}
	}

We have only written three lines of code here to do all the authorization:
First and Second are the FilterUnAuthorizedWithToken() extension method that filters everything based on the user’s privileges and scraps fields that user should not see like PriceThatWeBought.
Third is the RequresPremission attribute which infers only someone can run this method that has Write permission over ShoppingItem object. For example an StateAdmin of Queensland can not edit a shopping item from NSW.

You can see the return types are Tuple<ShoppingItem, AuthorizationToken>. This is because the FilterUnAuthorizedWithToken() will attach an authorization token to any item returned. You can use FilterUnAuthorized() method if you don’t want an authorization token.

3. At the View (and ViewModel)

In you silverlight, the data context for your view is your view model.

	public class ShoppingItemViewModel
	{
		private ShoppingItem _shoppingItem;
		private AuthorizationToken _authorizationToken;

		public void Load()
		{
			//Load the data and authorization tokens here.
		}

		public string ItemName {get{...}set{...}}
		public bool ItemNameVisible {get{return _authorizationToken.IsVisible("ItemName");}}
		public bool ItemNameEditable {get{return _authorizationToken.IsEditable("ItemName");}}

		//Do the same for all other properties.
	}

The only change here is the AuthorizationToken which we saw is return by the FilterUnAuthorizedWithToken() method in the previous section.
Now you have to specify the visibility and editability properties and bind them to the visibility and editability of you XAML controls.
If you want to be more smart in the client side you can write a converter for visibility and editability. And instead of using strings in your code for “ItemName” you can use the property.Name to be type safe.
The magic here is that you can change anything in your authorization rules and policies, but the only place you have to change is your model. You should not worry about the service or view, they get updated auto-magically.

How to shrink the log file in SQL Server 2008

All of us have wanted to get rid of the SQL Server log file at some stage. The natural way of shrinking the log file is to back it up and keep it in a safe place. Log file keeps transaction data and is essential for temporal disaster recovery.

However, it is a common request to get rid of the log file because it takes space and some people don’t need the log file at all. Beware that there is only ONE place and ONE situation that log file is a waste of space and that is Dev Environment.

SQL Server 2008 has removed the possibility of shrinking the log file using DBCC SHRINKFILE or BACKUP SHRINKONLY. But if you want to get rid of the log file in your dev environment, use the following script:

Note: Before running this script, in SQL Server Management studio, select Query -> SQL CMD mode.

:Setvar DbName **Your Database Here**

PRINT 'Droping log file for $(DbName)'
GO
EXEC sp_configure 'show advanced options', 1;
GO
RECONFIGURE;
GO
EXEC sp_configure 'xp_cmdshell', 1
GO
RECONFIGURE
GO
USE Master;
GO
DECLARE @logPath VARCHAR(400)
DECLARE @rowsPath VARCHAR(400)
DECLARE @logFile VARCHAR(400)
SELECT @rowsPath = physical_name FROM  [$(DbName)].sys.database_files AS df where type_desc='ROWS'
SELECT @logPath = physical_name FROM  [$(DbName)].sys.database_files AS df WHERE type_desc = 'LOG'
SELECT @logFile = name FROM [$(DbName)].sys.database_files AS  df WHERE type_desc = 'LOG'
PRINT 'Renaming the log file (' + @logPath + '). You can delete it later'
ALTER DATABASE [$(DbName)] SET  SINGLE_USER WITH ROLLBACK IMMEDIATE
EXEC master.dbo.sp_detach_db @dbname = '$(DbName)', @skipchecks = 'false'
DECLARE @CMD VARCHAR(400) = 'ren "' + @logPath + '" "' + @logFile +  REPLACE(CONVERT(VARCHAR(25), GETDATE(), 113), ':', '') + '"'
PRINT @CMD
EXEC master..xp_cmdshell @CMD
DECLARE @attachSql NVARCHAR(MAX) = N'CREATE DATABASE [$(DbName)] ON ( FILENAME = ''' + @rowsPath + ''' ) FOR ATTACH '
EXECUTE (@attachSql)
GO

Automatic End to End Authorization (Part 3)

I promised to show you how does the SAF framework really do the magic. I changed my test model from my last post a little bit so I post it back again here:

<pre> [Grant(Roles = new[] { "Everyone" }, Permission = Permission.View)]
[Grant(Roles = new[] { "QLDAdmin", "NSWAdmin" }, Permission = Permission.View | Permission.Edit | Permission.Create)]
[Grant(Roles = new[] { "Deleter" }, Permission = Permission.Delete)]
[Grant(Roles = new[] { "God" }, Permission = Permission.View | Permission.Own)]
[Deny(Roles = new[] { "NSWAdmin" }, Permission = Permission.Edit)]
[AuthorizationCustom(CustomType = typeof(TestObject), Method = "OnlyQld")]
public class TestObject
{
[Grant(Roles = new[] { "Everyone" }, Permission = Permission.View)]
[Deny(Roles = new[] { "God" }, Permission = Permission.View)]
public int YouCanSeeMe { get; set; }

[Grant(Roles = new[] { "Everyone" }, Permission = Permission.Edit)]
[Deny(Roles = new[] { "God" }, Permission = Permission.Edit | Permission.View)]
public string YouCanEditMe { get; set; }

[Deny(Roles = new[] { "Everyone" }, Permission = Permission.View)]
public string YouCanNotSeeMe { get; set; }

[Deny(Roles = new[] { "Everyone" }, Permission = Permission.Edit)]
public string YouCanSeeMeButNotEditMe { get; set; }

[Grant(Roles = new[] { "*" }, Permission = Permission.View)]
[Deny(Roles = new[] { "QLDMan" }, Permission = Permission.All)]
public string EverybodyCanSeeMe { get; set; }

public string[] States { get; set; }

public static Permission? OnlyQld(IPrincipal pri, object instance)
{
var user = pri as TestUser;
if (user.Roles.Any(r => r.Contains("QLD"))
&&
(((TestObject)instance).States != null && ((TestObject)instance).States.Contains("QLD"))
)
return Permission.All;
return null;
}
}</pre>

Note that the Permission for the custom authorization method is nullable this time. It actually won’t work if the return type is not nullable.

Now the magic comes:

  1. Filter what user should not see automatically using FilterAuthorized() extension method.
  2. Send authorization tokens to client. Client can use AuthorizationToken.Visible or AuthorizationToken.Editable methods to bind view parts to data.

Below is an example case for more clarity:

<pre>[TestMethod()]
public void FilterAuthorizedTest()
{
IMetadataClassProvider sm = new SelfMetadata();
var everyone = new TestUser() { Roles = new[] { "Everyone" } };
var qldMan = new TestUser() { Roles = new[] { "QLDMan" } };

var coll = new[]
{
new TestObject() {YouCanSeeMe = 0, States = new[] {"QLD", "NSW"}},
new TestObject() {YouCanSeeMe = 1},
new TestObject() {YouCanSeeMe = 2, YouCanNotSeeMe = "a", States = new[] {"QLD"}},
new TestObject() {YouCanSeeMe = 3},
new TestObject() {YouCanSeeMe = 4, YouCanNotSeeMe = "b"}
};
var filtered = coll.FilterAuthorized(sm, everyone).ToList();
Assert.AreEqual(5, filtered.Count());

filtered = coll.FilterAuthorized(sm, qldMan).ToList();
Assert.AreEqual(2, filtered.Count());
}</pre>

The framework auto-magically cares about what user can see and what he can’t see.

In case you were not following me, here is the link to SAF Framework.

The major hole here is that the framework does not scrap properties that are not available to user at this stage. This should be a quick fix once I get some more time. Feel free, if you like to contribute. The architecture is quiet extensible and easy to understand.