Monday, November 9, 2009

The Data Generation Framework in MbUnit v3 (Part 2)

The new data generation framework of Gallio v3.1 is not only about the generation of sequential numbers. It can generate constrainted pseudo-random numbers and strings for your tests as well. If not applied wisely, the use of pseudo-random input for a unit test may lead to inconsistent test results. Nevertheless, there are many cases where it is very useful. Today, I'm going to explain how to inject pseudo-random numbers in your MbUnit tests.

Like for the generation of sequential numbers, the generation of pseudo-random numbers is accessible in MbUnit through a dedicated attribute. You need to attach that attribute to one or several parameters of a test method. Please consider the following simple example.
public class Foo
{
private readonly value;

public int Value
{
get
{
return value;
}
}

public Foo(int value)
{
if (value < 0)
throw new ArgumentOutOfRangeException("value", "Must be greater than or equal to zero.");

if (value % 3 != 0)
throw new ArgumentException("value", "Must be divisible by 3.");

this.value = value;
}
}
First, we are going to verify that the construction of a Foo instance with a negative value really throws an exception.
[TestFixture]
public class FooTest
{
[Test]
[ExpectedArgumentOutOfRangeException]
public void Constructs_with_negative_value_should_throw_exception(
[RandomNumbers(Minimum = Int32.MinValue, Maximum = -1, Count = 50] int negativeValue)
{
new Foo(negativeValue);
}
}
The test will run 50 times, by generating each time a pseudo-random integer between Int32.MinValue and -1.

It is possible to intercept the generation of the numbers to filter them before they reach your test method. Let's use that feature to pass to the constructor, only values that are divisible by 3.
[TestFixture]
public class FooTest
{
[Test]
public void Constructs_with_value_divisible_by_3_ok(
[RandomNumbers(Minimum = 0, Maximum = Int32.MaxValue, Count = 50, Filter = "FilterDivisibleByThree"] int value)
{
var foo = new Foo(value);
Assert.AreEqual(value, foo.Value);
}

public static bool FilterDivisibleByThree(int value)
{
return value % 3 == 0;
}
}
That test will run 50 times as well. It generates each time a pseudo-random integer between 0 and Int32.MaxValue, divisible by 3.

Like for SequenceNumbersAttribute, the data generation framework works internally with System.Decimal numbers, which are converted later by the Gallio type conversion engine when they are bound to the parameters of your test method. It means that you can use RandomNumbersAttribute against any numeric primitive (double, short, byte, etc.) and not only System.Int32 like in the previous examples.

Remember that you can combine together several attributes related to data driven testing to create combinatorial tests.
[TestFixture]
public class CrazyTest
{
[Test]
public void SomeCrazyCombinatorialTest(
[EnumData(typeof(MyEnum))] MyEnum arg1,
[Column("Hello", "World")] string arg2,
[RandomNumbers(Minimum = 0.25, Maximum = 5.75, Count = 7)] double arg2,
[SequentialNumbers(Start = 0, End = 100, Count = 10)] ushort arg3)
{
// Assuming that MyEnum has 4 members,
// this test will run 4 * 2 * 7 * 10 = 560 times!

}
}
Next time, I will talk about pseudo-random strings.

Wednesday, October 28, 2009

The Data Generation Framework in MbUnit v3 (Part 1)

Gallio/MbUnit v3.1 has been released for a couple of weeks. But I just noticed that we did not speak too much about one of its nice additions: the Data Generation Framework. Jeff mentioned it in the release note, but I would like here to provide some more explanations.

The Data Generation Framework is a set of types present in the Gallio core component. The purpose is to ease the creation of predictable parameters for data driven tests. The feature can be used in MbUnit through a couple of new attributes. Today, we will have a look at [SequentialNumbers].

This attribute provides a column of sequential values. It must be applied against any parameter of a test method.
[TestFixture]
public class MyTestFixture
{
[Test]
public void MyTestMethod1([SequentialNumbers(Start = 0, End = 10, Count = 5)] decimal value)
{
// This test will run 5 times with the values 0, 2.5, 5, 7.5, and 10.
}

[Test]
public void MyTestMethod2([SequentialNumbers(Start = 1, Step = 1, Count = 4)] int value)
{
// This test will run 4 times with the values 1, 2, 3 and 4.
}

[Test]
public void MyTestMethod3([SequentialNumbers(Start = 0, End = 15, Step = 3)] double value)
{
// This test will run 6 times with the values 0, 3, 6, 9, 12, 15.
}
}
As you see, there are 3 possible way to configure the sequence, according the combination of properties you are going to initialized:
  • Start, Step, and Count.
  • Start, End, and Count.
  • Start, End, and Step

  • The attribute generates System.Decimal numbers, but you can use whatever primitive type with a lesser precision (double, int, byte, etc.)

    It is also possible to specify a hook method that is going to filter some specific numbers. The example below generates a sequence of numbers from 1 to 100, by excluding the values which are not divisible by 3 or by 10. The filter method should returns true to include the number in the sequence, or false to exclude it from the sequence.
    [TestFixture]
    public class MyTestFixture
    {
    [Test]
    public void Generate_filtered_sequence([SequentialNumbers(Start = 1, End = 100, Step = 1, Filter = "MyFilter")] int value)
    {
    // Code logic here...
    }

    public static bool MyFilter(int number)
    {
    return (n % 3 == 0) || (n % 10 == 0);
    }
    }

    Announcing Gallio and MbUnit v3.1 Update 1

    A bug fix release is available for Gallio/MbUnit v3.1. Please read Jeff's release note for more details. You can download the package at the usual location: http://www.gallio.org/Downloads.aspx.

    Have fun!

    Monday, September 14, 2009

    Announcing Gallio and MbUnit v3.1!

    Finally it's here!

    Tuesday, September 1, 2009

    Equality Assertions in MbUnit v3

    As .NET developers, we all know that the notion of object equality is not as simple as it looks first. Object equality is in fact probably as fundamental and difficult to understand as the famous concept of pointers in the good old C language. The fact is that writing unit tests, with Gallio/MbUnit or with any other existing framework is mostly about making equality assertions on the output of various code components. Thus chances are high you be stuck soon or later by an equality assertion which should obviously pass, but unexpectedly fails because your type does not implement any reliable equality mechanism. That's why I would like to review in that article, some features of MbUnit v3.1 which may help you using properly the powerful equality assertions.

    Let's start with a simple example.

    [TestFixture]
    public class SolarSystemTest
    {
    [Test]
    public void Number_of_planets_in_the_solar_system()
    {
    var repository = new StarSystemRepository();
    var solarSystem = repository.GetLocalSystem();
    int count = solarSystem.CountPlanets();
    Assert.AreEqual(8, count);
    }
    }
    We use here the well-known Assert.AreEqual to verify that the actual number of planets found in our solar system is 8, as expected. Of course, the equality assertion knows how to compare System.Int32 values. Basically, it knows how to compare any primitive like System.String or System.Double. But what happens while asserting on non-primitive types?

    By default, MbUnit relies on the result returned by the overridable Object.Equals method. And by default, that method simply consists of a referential equality. Thus it returns true only if the 2 objects compared represent the same instance (EDIT: OK, let's ignore the case where objects are null for the moment.). That's why, given our implementation of the class Planet, the following test miserably fails.

    public class Planet
    {
    public string Name
    {
    get;
    private set;
    }

    public Planet(string name)
    {
    Name = name;
    }
    }

    [TestFixture]
    public class SolarSystemTest
    {
    [Test]
    public void Mercury_is_the_closest_planet_to_the_sun()
    {
    var repository = new StarSystemRepository();
    var solarSystem = repository.GetLocalSystem();
    IPlanet actual = solarSystem.GetClosestPlanetToTheStar();
    Assert.AreEqual(new Planet("Mercury"), actual); // Fail!?
    }
    }
    So how should we write the assertion to get the expected result? There are several solutions.

    • Asserting the inner object properties.

      The most obvious solution is indeed to verify the inner properties of the actual object. We could just replace the failing assertion by:
      Assert.AreEqual("Mercury", actual.Name);
      For a simple scenario such as in our example, this is surely enough. But what if Planet is in fact a complex entity, with a dozen of properties, each hiding a complex tree of entities and value objects? At best, you would end up with a very large number of unmaintainable assertions. That's why you should avoid that solution if you get more than 3 or 4 assertions.
    • Supporting equality.

      The second possibility consists in implementing a dedicated equality mechanism for the evaluated type. The standard way of doing it is to implement the IEquatable<T> interface.
      public class Planet : IEquatable<Planet>
      {
      public string Name
      {
      get;
      private set;
      }

      public Planet(string name)
      {
      Name = name;
      }

      public bool Equals(Planet other)
      {
      return (other != null) && (Name == other.Name);
      }

      public override int GetHashCode()
      {
      return Name.GetHashCode();
      }

      public override bool Equals(object obj)
      {
      return Equals(obj as Planet);
      }
      }
      This may sound like a perfectly reasonable solution. In fact, if you are a lucky developer, chances are good that your class already implements such an equality mechanism. It is perhaps a needed feature of your code base. If yes, then look no further, and just use it. The initial assertion will pass. Congratulations! You made it.

      But if no, then Planet has no equality mechanism probably because it does not need of any. It means that you are about to add an unnecessary feature to your code base, just to make your tests easier to write. Remember YAGNI? You ain't gonna need it! If Planet does not need to be equatable, then why making it equatable? Your answer should never be: "To make it more testable". This is a short way to the dark side of the force, I assure you. Adding unnecessary code that will only eventually be used by your unit tests is certainly a bad practice.

    • Using the comparison delegate.

      Most of the MbUnit equality assertions take a third optional parameter of the type EqualityComparison<T>. The equality comparison delegate is a function which takes two instances of the same type, and returns true if they are equal. If your scenario is simple enough, you can provide such a function to specify to the assertion how to compare the objects. With the lambda syntax, it looks very elegant.
      Assert.AreEqual(new Planet("Mercury"), actual, (x, y) => x.Name == y.Name);
      And if the object has a reasonable number of properties, that's still a good solution.
      Assert.AreEqual(new Planet("Mercury"), actual, (x, y) => 
      x.Name == y.Name &&
      x.Weight = y.Weight &&
      x.DistanceToStar == y.DistanceToStar);
      But again, if the type has too many properties, or if these properties are not more equatable than their parent, you will get a complete mess.
    • Using the structural equality comparer.

      MbUnit v3.1 comes with a fantastic built-in feature which deserves to be better known (but that's the point of this post anyway). Basically, the structural equality comparer (StructuralEqualityComparer<T>) provides to the assertions a convenient way to determine whether two instances of a type are equal or not; while the type itself does not implement any relevant equality mechanism.
      Assert.AreEqual(new Planet("Mercury"), actual, 
      new StructuralEqualityComparer<Planet>
      {
      { x => x.Name }
      });
      Well, it does not look so impressive, does it? The comparer instance is populated with one comparison criterion that says to the assertion engine to use the property Name to compare two Planet instances. The syntax is not much more complicated when you have several properties.
      Assert.AreEqualnew Planet("Mercury"), actual, 
      new StructuralEqualityComparer<Planet>
      {
      { x => x.Name },
      { x => x.Weight },
      { x => x.DistanceToSun }
      });
      The true power appears when you know that each equality criterion is easily customizable, either with a comparison delegate, or with a new inner comparer.
      Assert.AreEqual(new Planet("Mercury"), actual, 
      new StructuralEqualityComparer<Planet>
      {
      { x => x.Name, (a, b) => a.Equals(b, StringComparison.OrdinalIgnoreCase) },
      { x => x.Weight },
      { x => x.DistanceToSun },
      { x => x.Revolution, (a, b) => a.Period == b.Period }
      });
      As you see, each criterion is able to define its own comparison rules. You can also nest structural equality comparers and define them as a comparison rule for an inner criterion.
      Assert.AreEqual(new Planet("Mercury"), actual, 
      new StructuralEqualityComparer<Planet>
      {
      { x => x.Name, (a, b) => a.Equals(b, StringComparison.OrdinalIgnoreCase) },
      { x => x.Weight },
      { x => x.DistanceToSun },
      { x => x.Revolution, new StructuralEqualityComparer<Revolution> { { x => x.Period } } }
      });
      The comparer works very well with enumerations too. It provides a similar result to what do Assert.AreElementsEqual and Assert.AreElementsEqualIgnoringOrder. Suppose that Planet has now a property named Satellites which returns an instance of the type IEnumerable<Satellite>. Adding them into the overall comparison structure is easy. The comparer also supports some options to ignore the order of the child elements.
      Assert.AreEqual(new Planet("Mercury"), actual, 
      new StructuralEqualityComparer<Planet>
      {
      { x => x.Name, (a, b) => a.Equals(b, StringComparison.OrdinalIgnoreCase) },
      { x => x.Weight },
      { x => x.DistanceToSun },
      { x => x.Revolution, (a, b) => a.Period == b.Period }
      { x => x.Satellites, new StructuralEqualityComparer<Satellite>
      {
      { x.Name },
      { x.DistanceToPlanet }
      }, StructuralEqualityComparerOptions.IgnoreEnumerableOrder
      }
      });
      As already explained, the structural equality comparer can be used with most of the equality assertions. It is particularly useful with the equality assertions for collections.
      Assert.AreElementsEqualIgnoringOrder(
      new[] { new Satellite("Deimos"), new Satellite("Deimos") },
      mars.Satellites,
      new StructuralEqualityComparer<Satellite> { { x => x.Name } });
    See you next time!

    Monday, August 3, 2009

    Assert.ForAll and Assert.Exists in MbUnit v3

    MbUnit v3.2 v3.1 is going to be released next week as part of the Gallio package. It contains two new very convenient assertions; So useful in fact, that they probably should have come earlier. But well... Better late than never!

    Anyway, Assert.ForAll and Assert.Exists are the counterparts of the LINQ extension methods IEnumerable<T>.All and IEnumerable<T>.Any respectively.

  • Assert.ForAll verifies that all the elements of the sequence meet the specified condition. The following example defines a test method which verifies that every integer of the sequence is an even number.
    [TestFixture]
    public class MyTestFixture
    {
    [Test]
    public void ForAllTest()
    {
    var data = new[] { 2, 8, 10, 6, 4, 20 };
    Assert.ForAll(data, x => x % 2 == 0); // pass!
    }
    }
  • Assert.Exists verifies that at least one element of the sequence meets the specified condition. The assertion evaluates each element until it finds a matching one, or until it reaches the end of the enumeration, which causes the assertion to fail. The next example shows a test method which verifies that at least one integer of the sequence is an odd number. Considering the sample array, the method should obviously fail.
    [TestFixture]
    public class MyTestFixture
    {
    [Test]
    public void ExistsTest()
    {
    var data = new[] { 2, 8, 10, 6, 4, 20 };
    Assert.Exists(data, x => x % 2 != 0); // fail!
    }
    }
  • Thursday, July 23, 2009

    A New Book : Using Gallio & MbUnit

    Dan Maharry has recently done a great effort to put the Gallio Book project back on the rails. As Dan explains in his post, volunteers are wanted to help to write and review the content of the book.

    If you are interested, rendezvous at the Gallio-Book discussion group.