Having to parallelize almost every bit of code here at Delver, some common patterns emerged while we wrote a lot of back-end code.
I remember reading about a new framework from Microsoft Robotics Division called "Microsoft CCR" (Concurrency and Coordination Runtime) a few months ago in the "concurrent affairs" column at the MSDN Magazine but I didn't pay much attention to it at the time. Two weeks ago, it jumped back to my mind so I revisit the article and started diving a little deeper into it, thinking about what sort of problems it can solve in my world and if it does, where could I use it to our benefit. If you don't know anything about the CCR, there is great public content published already like the CCR User Guide but I'll try to give you a 2 minutes intro of the general architecture. The way CCR is built is very much like the SOA world, using messages to communicate between "services". The major components are the Dispatcher which is actually an array of OS Threads, the DispatcherQueue which holds a queue of delegates to run so the Dispatcher can "look" at the DispatcherQueue and when it has a free OS Thread available, it pulls one delegate out of the queue and run it. So far - we've got a classic ThreadPool. There are some differences but I'll let you read about it in the User Guide. The third component, which is the most important one is Port. Think about Port as a pipe that can receive messages (FIFO style - first in, first out) and hold them until someone will know what to do with them. The last component is the "manager", the Arbiter; Arbiter expose a set of methods that allows you to listen to a given pipe and if some conditions are met on the messages the pipe contains, we can take the message(s) and transform them into a runnable delegate placed in the DispatcherQueue.

One of the goals for this library is the make sure you've got much less places to go wrong, by exposing a set of common async patterns you can easily use to guarantee clean(er) code that is easier to read. Think about sending messages from one pipe to another, creating a flow-based code via messages rather than spaghetti code with a lot of messy indentation. This is a very powerful architecture.
Obviously, the entire CCR framework is thread-safe by design so no need to protect the library. A simple example:

Alright alright, so I didn't post anything for... a decade or so.
but I'm here (at the office that is) all day long, being a part of a great Team, building the greatest\coolest piece of software I've ever dream of.

I promised myself that I'll be short this time so here it goes, Oren's 60 seconds update:

1. We've changed our name from Semingo to Delver (delver: (n) deep thinker; one who investigates data).
   Hopefully (if God will hear his little buddy here), the phrase "To delve" will catch up with the scary "Google-it".
2. We're going to show our product to the world at the DEMO conference (28-30 January, yes, in 4 days!) in Palm Desert, CA.
   If you want to be one of our first beta users, please go to our site: www.delver.com and register (we'll send you an email once we'll release our beta).
3. We're looking for super talented folks to join our amazing Team, interested?

Imagine that you're living with one hell of a crazy wife. Every day she's giving you bunch of tasks. "Mow the grass", "water the plants", "take out the garbage", "replace the light in the kitchen", "build a fence" etc. For every task you complete, she goes bananas and create 10,000 more on the spot, all related to the task itself ("build a fence" leads to "paint the fence", "put a nice sign on the fence", "practice your wax-on, wax-off" … you get the drift). Reluctant to perform all of these tasks but smart enough to know that you'll lose at least 50% of your property (divorce are nasty), you start collecting these tasks (you write them on papers). You take one paper (single task), perform it and return to your wife with a big-fat smile of your face. She, in return, creates 10,000 new related tasks to the one you've just completed, write them on paper and put it in a BIG box. Every once in a while she's not waiting for you and adding "main" tasks to the box by herself. After performing one task, you pick another one from the box (FIFO), without knowing if it's a main task or not, you go to your way "eager" to perform that task as requested (as if you have a choice). This goes on and on and on…

The box is the tricky part here. Can one box hold billions of papers? hardly. So you start collecting boxes and now it's getting harder as you need to add new boxes when needed, find the "right" box to pull tasks from and making sure these boxes won't break (maintenance) with time.

Here are a few assumptions you can take as is:

1. Your wife is looking for perfection but only in the "main" tasks which means that if you were to build the fence, you have to do it perfectly so each mini task related to it is crucial.
2. Your wife tends to forget things so you can assume that occasionally, she'll add new "main" tasks that were already performed or exist in a different box.
3. You can't throw away tasks "just because" as you don't know if a "mini task" will be thrown by mistake (= your wife will be pissed off. 50% is gone.).
4. Be cool - you won't perform the same task twice. This one is on me.
5. Drinking RedBull (or XL or whatever energy drink you're familiar with) 24-7-365 you don't need to rest. You don't need to sleep. Think robot (funny combination, for 2007).

How to store billions of tasks?

I kinda like the "green feeling" of a forest. Oh right, we also need them in order to breath (El Gor is more convincing than I. Thank God). Most importantly, it costs a lot of money buying so many papers! And the boxes!  You'll need a lot of green ones ($) - not trees but we can't "breath" without them as well). Oh well… You're starting to build a "Boxing mechanism", hire a few guys to maintain them, getting a VC to give you some extra $$$ and after a few months\years you got it cover!

What do you do if you don't have the extra $$$ or more important the extra time to develop this kind of storage system?

How to store billions of tasks? You don't. You can't. 
In most scenarios, when things seem too difficult to accomplish (with the given limits) try a different angle: "If you don't like the answer, ask a different question".

We know that each task creates a lot of new related tasks right? We also know that keeping those new tasks is the tricky part (the BIG box problem) so what else can be done? Let's change the question. "How do I make my wife happy?" seems like a smarter question. If it's expensive to save those tasks why not doing these related tasks on the spot instead of storing them in the box(es)?  How is this going to help us? Now we can throw away tasks because these tasks will be added later on (assumption #2. Thank God your wife is not the robot). Assuming that we can save about 1,000,000 papers in one big box – we're all set. If the box is full, we'll simply throw away new "main" tasks, feeling good as we KNOW that we'll get to them later on (again, assumption #2). Now all we need is one simple box with a limited amount of papers. Less $$$ to waste and much simpler storage system to develop.

Crap, it just hit me. I'm doing some cool sh$% at Semingo! Join us!

Well this is mostly a good backup-post, but heck maybe a few other (VERY COOL) geeks will find it interesting.
Looking for some cool IDE colors\fonts, I came up with this:

[ Regular mode ... ]

[ Marking the for loop ... ]

[ Output window ]

I based my colors on ZenBurn.
The font I'm using is Consolas.
Read all about it in Jeff Atwood's post.

You can download MY version here: OrenEllenbogen_DarkSchema.rar (58 KB)

Note: if you're using ReSharper you'll have to disable the "Highlight current line" option. 

What is more important to you - having the brightest dude in the world in your team, doing his magic with God-like authority or real "together-will-conquer-the-world" Team work? Tricky question...

 

For those of you who don't know the TV series "House", this is your wake up call! Go see it. Now. Seriously.
Well, if you don't have the time or you're just too damn eager to read my post, we'll, "you're an idiot!", but that's your right so I'll give you a short summary: Dr. House, played by the genius actor Hugh Laurie, is the go-to-guy for all the rare cases where the rest of the doctors go bananas. With his extremely cynical point of view and shameless wittiness, combined with a very bright, analytic and (yet) creative thinking, he manage to solve all (we'll, almost) of these cases and still being a complete jerk to his "teammates" during the show. Just a few pearls from Wikiquote so you'll get the drift:

         Dr. Cuddy: You don't prescribe medicine based on guesses. At least we don't since Tuskeegee and Mengele. 
         Dr. House: You're comparing me to a Nazi? [admiringly] Nice ...

Moti and I have decided to form an invite-only Scrum Clan.
We would like to tag Pasha Bitz (snapshot below) as the 3rd clan member.

Pasha, choose carefully, you can only tag one Scrum-Lover like yourself to this distinguish clan ;)

p.s - If you want to be part of the Scrum Clan, please drop a comment and you *might* get an invitation...

                                 Scrum Rules !

Let me start with an out-loud recap of this post: Agile is not something you can put on a bread nor is it "a certain path to success".

It's about STATE OF MIND.

If I had to describe the meaning of "Agile" to a new teammate I would say: Agile is a constant thinking about how we, AS A TEAM, can produce working features to our users with high quality within a short time-frame.

Don't worry:
It's really OK to provide only a subset of feature(s) in one sprint.
It's really OK to leave SOME designing\architecture issues for later on as long as the high-level architecture is good enough (=you're comfortable with it) to answer the big questions.
It's really OK to implement only two REALLY-DONE-HIGH-QUALITY features in one sprint over four semi-working-not-demoable features.

The key here though is not really the practices, it's about the big bullets(again, state of mind):

1). Produce value for your customers and adjust\adopt early.
2). Build a Team (self leadership).

These ideas are hard to implement and require special kind of people. Putting the Team in front of yourself is not a very job-secure attitude.  The ability to help your teammates, shift tasks, taking ownership, critisize yourself and your teammates and getting better, produce high quality design, tests and code - all of it - requires versatile people with unique state of mind (and unique abilities, of course). It's worth it. When things glue, it's a real magic; Things start to get going by the Team, improvements and features starting to come from the developers\QA\Graphic Designer, adjustments are made on a regular basis, changes are welcome and productivity is celebrated.

You can feel something is going right.
That's Agile.

The way WCF proxies are designed is to live until shi* happens.

Let's assume that we have a CalcualtorService with one method named Divide(int a, int b). Sasha, a cool programmer-dude, trying to produce some usefull software writes:

public MyCalcualtorForm : Form {

   private CalculatorProxy _calc = new CalculatorProxy();

   Calc_Click(...) {
      _calc.Divide(firstNumber, secondNumber);
   }
}

What is the first error you can think of that could happen? Yep, DivideByZeroException.
Once the proxy gets an exception, it enters into a "Faulted" state which makes the proxy unusable(=you cannot use it again).
The quickest solution is to work "by the book" and create a new instance each and every time we need to call the service:

Calc_Click(...) {
   using (CalculatorProxy calc = new CalculatorProxy())
      calc.Divide(firstNumber, secondNumber);
}

But what's bad in this solution?

1. Performance - you pay (not a lot but neither little) for each creation of the proxy. Sure, it will probably not be your bottleneck, but heck, why is it useful? Most of the time the proxy will not throw an exception and yet we need to create it every time just to avoid the faulted state scenario. 
2. Design - If we declare this exception BY CONTRACT, I would expect that the proxy will still be usable afterwards. Do we really want to return Enum\int\string as status instead of throwing exception just because of poor design?
3. TDD - you know that I'm in love with it. Now imagine Dependency Injection. Component A recieve ICalculatorProxy and use it to... calculate something. Working "by the book" is no good as we want to recieve an instance of the proxy from the outside in order to mock it. Right, so we inject a proxy from the outside (got to love Windsor) and life is pretty sweeet. Darn! Wait! one poor (even by design) exception and our proxy goes dead. Very un-TDDish of Microsoft.

I had to come with a solution as no one will take TDD away from me. I present to you ProtectedProxy: this little IL-code-at-the-end-of-the-day will able you to recover from faulted state by creating a new proxy on each exception thus making your proxy... useable (couldn't think about a better word to describe it). Think about a situation where your proxy is trying to call the service but the service is down; In Semingo, we decided that we want to keep trying until the service is up. Via ProtectedProxy, you can determine how many times do you want to recover and when you should finally kill the proxy. Oh yea, ProtectedProxy uses Windsor in order to create new proxies if needed and logging messages to log4net. Good stuff.

In the example above, all Sasha had to do was to:
1). Initialize the _calc field by:
        ProtectedProxy<ICalculatorProxy> _calc = new ProtectedProxy<ICalculatorProxy>(new CalculatorProxy());
2). call _calc via:
        _calc.Instance.Divide(firstNumber, secondNumber);

But enough said, code please:

// Written by Oren Ellenbogen (07.08.07) - trying to protect our proxies so they could recover from:
// (A) The service is not up yet, but we want to try again later.
// (B) The service throws (ANY) exception, we still want our proxy to function (bubble the exception, but still keep on working).
// Microsoft intended to use a NEW proxy per call, but for TDD this is not ideal as we would like to inject proxies from outside as mocks
// in order to simulate multiple scenarios.

#region using

using System;
using System.Reflection;
using System.ServiceModel;
using Castle.Core.Resource;
using Castle.Windsor;
using Castle.Windsor.Configuration.Interpreters;
using log4net;

#endregion

namespace Semingo.Services.Proxies.Helpers
{
   
   public interface IProxy : ICommunicationObject { 
      bool Ping();
   }

Hours of joy...

Almost forgot, on the next post - "How to TDD WCF code" - stay tuned...

One of the most common question in moving towards Agile Development is "Where should I start from?". If you'll ask me, setup a Continuous Integration (aka "CC") would be the first thing you should start with.

Step 1 (Check for compilation bugs): Code Quality ~= 30%

The CC should be able to identify check-ins to your source control, get the latest source and compile it. The output should be either "green" (Everything compiles successfully with 0 warnings) or "red" (more than one warning or compilation errors). In addition to the fast feedback, the output should also include the files that were changed from the last build (and by whom, so people could know where to look).

The immediate value is priceless. the ability to SEE whether your source-code is stable enough to allow other programmers perform Get Latest and continue their work and the "fail-fast" attitude can save you a lot of time in the long run. It's important to realize though that even if the code compiles without warnings, it still doesn't mean that you could count on the quality of the code.

Step 2 (Check for component-based quality): Code Quality ~= 70%

If you can go one extra mile, write a few automated tests (via one of the available XUnit frameworks) for your components. This means that you are able to inject the component's dependencies from the outside and simulate mini use-cases on component's level. This step is crucial even if you write those after the code itself was written. Let the CC run them if Step 1 is OK. This should allow you to catch the majority of your bugs (I'll leave the "how to write good tests" to another post). If all is green, you know that the system behaves as expected, at least to some extent.

This step is not trivial as it requires you to design for testability and invest in proper testing. Don't let go of this step though, automated tests on this level will make your life much easier. It will take your code one (major) step ahead in "write code that could be changed". Agile is all about that state of mind.

Step 3 (Check for integration-based quality): Code Quality ~= 80%

Now that you're components are behaving as expected, you should try to write a few (automated, of course) tests that simulate the entire flow of 2 or more components (DB is a component as well) in the system. As the system grows and more uses cases are added, you should try to improve these tests as they give a solid proof that the SYSTEM works.

Step 4 (Make everything visible): Code Quality ~= 90%

The state & quality of your source control should be visible to the Team and Management as you want to insure IMMEDIATE response time in case someone check-ins a low quality code (on any level). Fixing a failing test three days after the change itself is a bad symptom of low visibility or low perception, by the Team, regarding the importance of the quality of the system.

Step 5 (Automated deployment): Code Quality ~= 95%

After successful build you would like to deploy the latest source on a dedicated environment which the developers could play with before deploying to another (testing?) environment. This won't be a stable environment, but at least it will give a quick look at the current state of the system - the way customers would see it.

Step 6 (Procedures checking): Code Quality > 95%:

You can add many more checks to the flow, such as Tests Coverage or FxCop. Leave those to the end. From my experience, Time .vs. Value in these features will vary from team to team. You'll gain much more from investing in Steps 2-3.

Semingo CC:

Each developer & manager on our team have a CCTray(the little red circle in the little picture above) which is either Red(source control is damaged), Yellow(Build in action) or Green(Life is sweet). We're using Cruise Control.Net, CCTray, MSBuild (and TFS plugin) and NUnit to perform all of the above. 

A few teasers:

* Image of Steve Urkel (from the famous Family Matters TV series) is shown for failing build.

* On the right you can find Pasha (with a V sign I've added), one of our finest hackers modeling a successful build. You can also notice the 582 green tests (including Integration tests) and 2 changes made by Sagie since the last build.

* Going to NUnit Details, you can get the full details:

I had to cut the pictures in order to keep a sane width for the post, but you can get the drift.

btw - Aviel, yet another Semingo hacker add a "Doh!" (Simpson) each time the CC is red on his computer. It can be quite funny (and scary, if fully concentrated on code).