Using export configuration files in Orchard Import / Export

In the more recent versions of Orchard (I think it came in in 1.9) the export options available at Import/Export became much more powerful. One of the underused features (thanks to a lack of documentation) is "Upload a configuration file with recipe steps to execute" option.

With this option selected, you can upload an xml file that defines which content types and other custom export features you wish to export with your site, allowing you a repeatable process, and avoiding manual mistakes.

Here is an example of such a file.



Note that "PagewithPromotion" is a custom content type that has been added to the site in this example. Unfortunately currently, there is only "inclusive" syntax it appears for listing which content types to export. I'm going to raise an issue for adding "exclude" syntax so that it is easier to say "all content types except bob".

I've constructed this template by looking through the source code. The example works, but there are probably also options I've not spotted, so feel free to explore the code further!

Serializing Enums to String in MVC6

Often you will want to change the default serialization of your API responses, to ensure that enums are reflected as strings, not integers. Why? Well, which is more self documenting?

{
  "Status": 1,
  "Message": "things are on fire"
}

or

{
  "Status": "Error",
  "Message": "things are on fire"
}

Fortunately this is really easy in MVC6. The default code uses the ever populate JSON.NET, and it's options are easily exposed. Note that this code example is against RC1, so should actually be correct going forward (the syntax has changed many times across vnext). I'm also assuming you have created your site using the template application, rather than creating an empty ASP.NET application.

Inside the "ConfigureServices" method find

services.AddMvc();


and replace it with

services.AddMvc().AddJsonOptions(options =>
{
    options.SerializerSettings.Converters.Add(new StringEnumConverter());
});

As you can see, you should be able to get at all the other json.net options from here too.

Lamda expressions in unit testing are hard

So, it turns out that Lamda expressions are not great for unit testing. A recent example of code that I wanted to unit test:

 public async Task RegisterIndexAsync<T>(IFoundocIndex<T> index, CancellationToken cancellationToken)  
     {  
       //.....  
       await _fdbStorageProvider.ReadWriteAsync(async transaction =>  
       {  
         var indexDefinitionState = await _indexProvider.GetIndexDefinitionStateFromStore(transaction, index).ConfigureAwait(false);  
         if (!indexDefinitionState.Exists || indexDefinitionState.Changed)  
         {  
           await _indexProvider.PersistIndexToStore(transaction, index).ConfigureAwait(false);  
           var documentCount = await _documentProvider.Count<T>(transaction).ConfigureAwait(false);  
           if (documentCount > 0)  
           {  
             Trace.WriteLine(documentCount + " items found in collection");  
             rebuildIndex = true;  
           }  
         }  
       }).ConfigureAwait(false);  
       if (rebuildIndex)  
       {  
         Trace.WriteLine("Rebuilding Index: " + index.Name);  
         await RebuildIndexAsync(index, cancellationToken).ConfigureAwait(false);  
       }  
       Trace.WriteLine(String.Format("Not Rebuilding Index: {0}.", index.Name));  
     }  
     public async Task RebuildIndexAsync<T>(IFoundocIndex<T> index, CancellationToken cancellationToken)  
     {  
       using (var queue = new BlockingCollection<IEnumerable<T>>(_settings.MaxBatchesInIndexQueue))  
       {  
         _batchEntityProvider.GetBatches(queue, _settings.MaxIndexBatchSize).ConfigureAwait(false);//deliberately not awaiting this  
         await _batchConsumer.Consume(queue, cancellationToken, async batch => await ConsumeWorkQueue(batch));  
       }  
     }  

Now of course, I'm unit testing an implementation call to RegisterIndexAsync<T>(IFoundocIndex<T> index, CancellationToken cancellationToken) but I also want to verify that in this test, that my index was not rebuilt. Normally you could do this by mocking (for example, using the amazing Moq and verifying the number of calls to_batchEntityProvider.GetBatches but here there is a complication.

In this example you would need to use a Setup operation on _fdbStorageProvider.ReadWriteAsync that would supply the entire of the lamda expression as its setup. Essentially - you would need to know and express the code for this function in your unit test setup. Your unit test becomes essentially "ensure that what the code does is what the code does" - and this is not right.

It is also extremely hard to do due to the way lamda expressions compile - the same resulting code will compile to a different object - so they are never going to the same object in your Moq setup call.

Looking more deeply into this example you could validly say that I shouldn't care about verifying whether an internal operation is called. All I should be worried about are external results right?

Quite possibly true in this case - if I cannot observe the impact through external interfaces its probably not worth knowing right? Except that I need to know if this method is rebuilding an index unnecessarily. If it does, there will be no observable difference - the index would be the same before and after, the only difference would be the time taken on larger indexes - something you can't identify in a unit test, and using time taken as a part of a test is a lousy idea anyway.

For this specific test, I think it is time to head back to integration tests, which leads to the future problem of - how do you do the equivalent of "Verify" when you are not mocking your classes. That's for another time.

How to Setup Async and Task Return methods with Moq 4.2

Moq 4.2 comes with a couple of nice changes that I hadn't noticed (and they are extension methods, so you might continue to miss them)

The main benefit is allowing you to change from writing

 _mock.Setup(m => m.GetStateAsync(It.IsAny<Profile>()))   
 .Returns(Task.FromResult(new IndexDefinitionState(true, true)));  

to writing

  _mock.Setup(m=>m.GetStateFromStore(It.IsAny<Profile>()))  
    .ReturnsAsync(new IndexDefinitionState(true, true));  

...which is just that little bit easier to manage (especially when it is a more complex return type than the example above), but it also allows methods with return type Task to work without further setup it seems. Both are extremely useful for the Async-first API I'm working on.

From the release notes for Moq 4.2

• Improved support for async APIs by making default value a completed task
• Added support for async Returns and Throws
• Improved mock invocation sequence testing

All great stuff. I really couldn't do without Moq - a long time back it was the thing that made me realise that unit testing was actually viable.

The battle for the internet is back on....

Battle for the Net | What is Net Neutrality

Thoughts on The Phoenix Project

I finally got round to reading The Phoenix Project last weekend. I know right? It's about time. I thought I'd share a few thoughts, as I think it's a great book and well worth a read for anyone in a business delivering products depending on IT (hint, nearly every business of any size these days depends on successful IT supporting the core business functions).

The Phoenix Project is told as a novel about the recovery story of a business who who used to be the best widget maker in the world, and are now being pounded on by a faster, more agile, hungrier up-and-comer. We follow Bill, who is promoted into the seventh circle of hell - VP of IT Operations, with only 90 days until the release of the-mother-of-all-projects which is vital to the companies survival. It's failing hard, VPs are falling left and right, every meeting looks like Game of Thrones and every piece of the puzzle seems to depend on one engineer, Brent, who's time is more oversubscribed than a year 2000 dotcom IPO.

Before I go any further - I cannot recommend enough the concept of getting your message through as a novel, rather than a text book. There are other books that take this similar approach, The Five Dysfunctions of a Team is another one I love, and for the same reason. It turns dry, bullet point material, and turns it into a "what happens next" adventure. You are with the protagonist on whether they solve their problems, and your brain is following the same steps as they do all the way through. A fiction is worth a thousand bullet points you might say. Consider them "text books with only extremely coherent examples". I read The Phoenix project in two days. I can't remember the last time I read a Terry Pratchett so fast.

There are some key messages to take from it. I want to avoid explaining the entire story of the book, because half of it's power is you working the problem yourself.

The first is actually not one really focused on by the book explicitly in its "lessons", but is worth learning: While development, operations, sales, marketing, "products" etc are all sniping at each other and seeing all other departments as "getting in the way of real work", you are in a pretty bad spot. Maybe a more positive assertion is: All teams need to work together with the vision that they are all responsible for delivering the companies core product

The second is more explicit: Understand the definition of work and from that understand what things your team actually works on and prioritise it. The book lists four, but you could easily make it two; planned work, and unplanned work. The book splits planned into business project, internal project (e.g. infrastructure upgrade) and changes (e.g. production db schema update). The obvious question is this - if you don't know all the work that your team are being asked to do, or where it comes from, how can you possibly ensure that you prioritise it correctly?

The next is a classic from lean, and is well developed in books such as The Toyota Way, and The Lean Startup. Work in progress kills productivity. In IT terms, anything that's started, but not working correctly in production is useless to the business. It is money spent and no return gained. When stock analysts look at companies they are interested in investing in, they look at how well they convert raised capital into further gains, and so should we as IT professionals. It ties in very closely to one of my own favourite mantras on "the definition of done", which I firmly believe can only be interpreted as working bug free as desired by the client or business in the production environment. Why such a stringent definition? Anything else can still come back as a task on your plate. You have to context switch (not multi-task, you actually can't do that you know), and internally prioritise. Putting something on hold either requires carefully "putting something back on the shelf", or even worse, simply dropping it without care or attention.

Possibly the most unique lesson in the book is: Any attempt to optimise a process that doesn't improve the time of that processes' bottleneck is false progress. This is gold dust, and if there is nothing else in this book for you, take that. When you read the book you see things in a new light (unless you've already read The Goal). In any process - look at the bottleneck and refine that and that only, until it is no longer the bottleneck. Then find the new bottleneck. The books example is the amazing "Brent" - at the heart of every production incident, architectural planning exercise and key software project deliverable. If he gets hit by a bus, the company could literally fold. However, after several chapters of removing Brent from every coal face so that he can actually improve the overall process of the organisation comes the big reveal. Brent is not a "work centre" - he's only a person at the work centre. Like any manufacturing plant many parts of what we do are automatable - particularly in the deployment life cycle and testing. Which leads to...

Identify your work centres: Identify all the parts of your software development cycle, they are your work centres. Now find the bottleneck among those, and improve that. Remember the golden rule above - if you are not improving the bottleneck of the process, then your effort will not reap the benefits you desire.

These last two are  the current topics of my own contemplation, as we try to reduce our continuous deployment cycle down to where we know others have already reached. It finally gives me a strategy for approaching the problem that doesn't involve simply "refine all the things". It may be obvious to say "improve your worst bits first", but what you think are your "worst bits" might not actually turn out to be your bottlenecks, so improving them would be false progress.

There are lots of other excellent snippets in there (and probably some major points I glossed over, but hey, you are going to read it anyway, right?). Why ten minutes work might take several days to be acted upon for example, and these just add more and more interesting food for thought.

I'll certainly be re-reading The Phoenix project - probably any time I get stuck on how we improve next, but first up I'll be reading The Goal

Happy reading :)

Could not copy the file manifest because it was not found

This is an old chestnut that has maddened me for quite some time. It all starts with the error message

• C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Microsoft.Common.targets(4453, 5): error MSB3030: Could not copy the file "bin\Release\myapp.exe.manifest" because it was not found. 
• Project src\mysolution\myapp\myapp.csproj failed. 
• Project src\mysolution\mysolution.sln failed. 

It almost certainly happens when you are doing the following:

• Using msbuild directly, rather than Visual Studio "right click" (e.g. you are using Team City)
• Have a solution with a web application and some other application types (e.g. a console app)
• Are using the Target "Publish" to publish the website

For example, here are the TC settings that cause the problem for me:

I'm also using a solution that has this structure

• MySolution
• MyConsoleApp
• MyWebApp

MyWebApp has a publish profile setup called "PublishToDisk". If I build or publish from Visual studio, everything is fine. If I build from Team City, I get the errors above.

I've known the cause of the problem for ages. The blanket "Publish" target is being applied to every project in the solution that might be publishable. That includes my console application. My console app is not setup for publishing however, nor do I want it to be. I do, however, want it to be built during the Team City process, as the app will get copied elsewhere via the packaging process in later stages.

However the possible solutions are not so obvious. running msbuild separately on myConsoleApp.csproj and MyWebApp.csproj seems ridiculously inefficient. Making my console application a click once app just to stop a build failure seems equally silly. By far the best solution I've found is to edit your myconsoleapp.csproj file and add the following section

<PropertyGroup>
    <GenerateManifests>true</GenerateManifests>
</PropertyGroup>

You can add this just before the ItemGroup for reference includes.

This should be enough to keep to allow your application to build without a load of very awkward feeling bodges.

Incidentally - this problem is one that occurs if you try to do this publishing process with Orchard via TC, so would solve that scenario too.