I’ve spent around 2 hours to figure it out, so at least I’ll write it down.
I’m using dotTrace 3.1 to do some analysis for .net web application running on Windows Server 2008 R2 (IIS 7.5).

I got strange numbers and was missing some data (methods calls) running “profile web application” until I set the affinity of the w3wp.exe process to single Node (single core). This can be done easily by opening the Tasks Manager, right click on w3wp process –> Set Affinity –> pick just one core.

Only then you should record the required page(s) and take the snapshot.

During team’s growth, the practice of keeping the knowledge base sound, looks like mission impossible. There are simply too many concerns to write down and maintain: picking a proper (=”searchable”) title to the knowledge base item, keeping the history of changes in the design and why they were made, adding some drawing etc. Text notes (and pretty pictures), what can we do, are getting obsolete extremely fast.

“How can we keep the team’s knowledge base without running after our tail?”

My team came up with this suggestion:

“Let’s keep the titles and put owners next to it, the rest can be solved with good communication and tests!”

Examples for such “knowledge base items” are, from our real list:

• Supporting Contextual Sign-In scenarios (Avish)
• Common CSS Classes for our beautiful look-and-feel (Moran)
• Creating and using pretty-URL routes (Ken)
• …

We broke it into domains like “basic architecture”, “common features”, “cross-cutting concerns”. Now the list is easy to search at (wiki) and fun to maintain.

This work extremely well if:

• Code-reviews are done on a regular basis. This means that at least 2 developers can talk about each knowledge base item.
• Tests are written to built great confidence (to refactor easily) in the code. I talk about unit-testing, integration testing, UI testing etc.
  • Great tests explain how to use the API and what should be expected from it.
  • Tests (mostly integration tests) explain the latest architecture.
• Comments in code should include the “why” behind the logic. Team members can add to it even more if needed, by talking with each other.
• New developers are exposed to the knowledge base, talking with the relevant owners. This is a great reference to look at.

Simple, short, effective!

Great dishes. Great timing. Every single day. This is one amazing place to eat at!

In many ways, running a great team of developers is just like running top-quality kitchen.
You’re measured by your ability to serve great quality(CI? well tested?), super tasty(nice UI?) dishes (features?), to hungry people (clients?) in reasonable time (before they’ll leave to another site).

Without solid understanding of ETA, there is no way to predict quality and release in time.

Most of us are driven by what life has to throw on us, may it be some personal event, a new book you read that makes you want to refactor everything or simply a lot of context-switches between projects and bugs.
Add this to our constant urge to excel at what we do, and you get a “miraculous” skill of losing control over our time. We’re built this way. Life is pushing us around!

How does this effect us or the “wheel” I’m talking about? I want to introduce you to Joe. 
Joe is a nice guy in my imaginary world, working as a superstar developer for “TheBestOutThere” company. Joe was assigned to complete a feature and asked to estimate when it will be done. “This feature is quite simple”, he shoots fast, “a few changes here, a few there and then the UI to nail the all thing. Hmmm… I guess that no more than 2 days of work”. So our story begins…

The first day goes pretty well as Joe adding the needed fields to the database and even manage to complete some tasks regarding the logic needed. On the second day (reminder: the last day according to the ETA) things starting to bite Joe’s ass. The feature turns to be a bit more difficult than he anticipated, and he feels that it will take 2 extra hours to complete the work. Keeping it in mind, he already notifies the wife he’s going to be late today. “Damn, late again?”, he ponders, “oh well, at least I know what is left to do now”. Knowing that he needs to leave late today, he continues to work and suddenly he detects a really ugly piece of code in his path. “Refactoring time baby!”, he smiles to himself. “Well, I already have a few spare hours, why not?”, he convince himself once more. It works. The refactoring takes additional 4 hours. Joe, in a panic attack understands that it took more than he planned, hurries up missing a few critical tests and canceling the “code review” meeting claiming “this feature is simple enough!”. The code-review still takes place as this is simply a must for top-quality kitchens. Fixing the code review comments, including adding the missing tests (as needed to begin with) cost a few more hours and so days starting to fly by. After 5 days, Joe moving the feature to “code-ready”, now waiting for QA to test the feature. QA opens about 7 bugs, which take 6 more hours to fix (with solid unit tests to reproduce these bugs). On top of it, some more fixes were needed for easy deployment later on in production. The feature took total of about 7 days. Where is the missing 5 days went to? is Joe simply a poor estimator? Not quite, he just suffer from bad “context” we all do. He suffers from the constant battle between deliver fast and deliver with high quality. The problem is that it’s really hard to understand how estimation and quality work together.

If ETA is Jesus, Quality is God. Which is more tangible?

People are wasting time trying to convince themselves they can outperform on a daily basis. We can’t. Our best is probably not as great as we imagine. Instead, let’s not try to improve our ability to code faster, think faster or drink more coffee. Even if you manage to get better at it (or consume more coffee and urinate to a cup), it won’t last for long (urine tends to stink in the room).

Instead, think carefully about “where the hack my 5 more days went to? Could I see it coming?”
Here is the “context” Joe needed to have in order to produce a complete feature, with great quality, in time:

1. Did I spare enough time reading the spec and talking with the Product Manager about it? (yes, this is part of the feature!)
2. Did I make sure there are no open issues left?
3. Did I spare time thinking about how this feature will be tested? how we can automate these tests easily?
4. Did I spare time to design the solution and do the required review with other people? what about the hours needed to fix your solution based on the reviews?
5. Did I spare time to read QA test cases and give feedback on it?
6. Do I have everything you need to make the feature a huge success?

1. What about people helping you out where needed?

1. Again, the goal here is great confidence in quality, not 100% test coverage.

Estimating each one of these steps (use “buffers” if certainty is low) and you’re a bit closer to understand where the missing 5 days vanished. 
Joe was thinking only about the coding effort, not the entire picture! This, by nature, means inability to consistently predict when something will end. He was pricing the wrong thing!

What does it take to make your team great unit of brilliant minds producing great dishes, every time:

1. Hire the best. They simply worth it. (oh well, that was easy, right? :))
2. Trust them doing the best work one can do! Help them get them but don’t think they can’t get there by themselves with the right set of tools and context.
3. Start with explaining the meaning of great ETA - without it, prediction is impossible. Consistency is a lost cause. This is one bad kitchen.
4. Ask people to stand behind there ETA. They are responsible for it!
5. In the same breath, remind them that quality work is the right path to get a correct ETA. Quality work will lead to shorter cycle eventually!
6. But (here comes to tricky part!), they should also remember that over-refactoring leads to high quality of nothing important (as nothing really reach production).
7. So, define together what is “quality work”. Set it as “team context”.
8. Constantly hear what your guys has to say about “I wish we could fix it!”. Eliminate the big things (lack of tools, too many context-switches, not enough testing etc).
9. Try to inject a lot of honesty, communication and great motivation to the team. This is the basic engine oil no one can live without.
10. Measure delivery cycle length: spec –> design –> getting ETA –> feature is code-ready –> solving all bugs –> deployed in production.
11. Talk with your guys to understand waste again. Eliminate the big things now! seriously!
12. Keep hiring more people, only the best, you’ll need them soon enough.

Over time, the cycle of delivery should get much better while the quality will get even higher. The trick here is that people will feel more confident in the flow, “saving'” time to write quality piece of code, making features amazingly stable in the 1st attempt. This is the consistent rhythm you should dream of. The wheel spins…

Can you count the number of times you were asked with this innocent question?

I tried to think and analyze the "motivation" behind this question, considering why people are so intrigued to hear my answer when putting me on the spot. Is it really that important for others to know what Oren have to say about God? I can't ignore the simple fact that religion is a huge cornerstone in our life and finding more people with similar believes give us great comfort and sense of power. People are looking for acceptance, to become part of a group they can define and feel "right" in. This is a social aspect that is common to almost everything we do in life (love, friends, work, faith etc). I already know all of that and so do you. So, where am I going with this?

I'm trying to muse with the the notion of acceptance and where do I fit in. What is my "do you believe in God" question?

Personally, It's striking me as plain crazy that our life is so driven to into what I grasp as the "wrong" groups. Do you believe in God? Do you eat Kosher food? Do you fast at Yom Kipor? Are you rightist? And the list goes on and on. Are these the questions we should really ask ourselves? is these how we want to measure ourselves or find others to connect with? I'm not sure I know the answers, but I feel there are some deep feelings, clear ideas and emotional borders that I'm so passionate about, they must represent my scales. They must represent my inner truth. Writing allow one to shoot ideas and thoughts on paper, so I tried to explain myself my truths. Hopefully it will give me a common ground to fit myself into a world I truly believe in and measure myself about what I see as right. Trying to synthesize my list, I came up with the 3 questions I have definite feeling of, (but) these are only my big questions so read it with the appropriate suspicion:

1. Are you making someone else proud?
   I'm blessed with the best parents one can wish for and the greatest family one can be born into. The best thing they have done for me is giving me their complete trust, always stating how much faith they have in me making the right call. It was really hard for me to mess things up with that kind of love and support. Make sure that you've got someone in life that you can make them proud, may it be your parents, family, girlfriend, close friend or the girl from the cafeteria. "I'm proud of you" is huge empowerment and it's a constant motivator to drive your life beyond the easy and all-so-common mediocrity. This source should be your impregnable place for positive enforcement, keeping you down to earth and posses the right confident to make tough choices in life.
   
   Be careful not to confuse making one proud with making one satisfied. You must own your path thus there is no place for blind acceptance. You need someone that trust your best efforts and push you forward in your struggles.
   
   
2. Do you see yourself as a good person?
   Let's avoid the definition of good for a second as it doesn't really matter. Although some of us really cynical about the meaning (and with no doubt some are plain bad), we all know that Mother Teresa is a good soul. The pure notion of bad and good is embedded deep inside us. Try to be truthful to yourself, do you see yourself as a good person? how would you define good? Do you think you can measure "how good" are you? Do you think it's important? Are you doing what it takes to get "better"? does it get easier to look at the mirror?
   
   You can call me what ever you might feel, but there is a romantic notion behind "being good". The way I see it, my time here is limited so my constant question is "what am I going to leave behind?". I would like to think that I've touched some people during my short life, that I managed to teach something as my great family and friends have taught me. I'm not sure that I made a great change (or even a small one), but at least I feel I did the best I can do with my believe of sincere communication. I'll never be Mother Teresa, but at least I'll do the best I can with what I feel is right for me, trying to focus on my strength and talent to make others picking their brain and wonder about themselves. Being there for others will make you feel less alone, less "on a road to nowhere". What do you have to lose?
   
   
3. Is there a way for you to help others?
   Do you feel lucky? If you do, you played your cards right somehow. Is there a way for you to show your tricks to the world? to make others think of a different way to look at things? Helping others makes a great closure between making others proud and making you feel as a good person. It's a powerful tool you should use wisely. The purpose is not the "convert" someone into your believes or trying to bring him into your "group". One should reach his own right and wrong, you can only help him by asking the right questions and trying to share your thoughts and experience. This way you are part of the journey, part of a legacy in other's life. This is a great accomplishment to leave behind you.
   

The order of the questions is important. Without making others proud and have their support it's impossible to be highly motivated all the time. Without constant motivation you won't allow yourself to grow into the person you can become, making it easier to avoid helping others and seeking for the "wrong kind" of acceptance. Giving up is easy, fitting in just for the sake of acceptance will not make you happier (at least for long).

Don't allow yourself to give up just because you were born to the wrong environment, that you had troubles as a child or you feel just like the world is against you. The world don't owe you anything, you can blame your luck, your God (or lack of it) but bottom line it's really up to you. Don't hurt others to pay back to your awful sense of luck. Find your positive people around you and let them pick you up when needed, there is no shame of needing others help, they would love to do so in order to feel good about themselves and making others proud. You'll do the same for them one day, I promise you that.

Everyone who knows me are familiar with my reiterative phrase "I firmly believe in natural growth", this rule apply here as well. Find your own sources of strength and make sure that you keep your "good standards" before trying to influence others to ponder about their path in life. Once you have the confidence in your answers, you'll be ready to open your view to the world and see what it has to offer.

Do I believe in God?
We're probably ants in a giants world, but is it really matter? I would like to think that this God cares more about us as living being, trying to grow together than following strict rules without understanding the real truth behind them. Isn't this the all purpose of the bible? of the stories we are taught since we were born? I wish our discussions would address more of the "are we doing the best we can?" and "how can we make it better?" rather than such a superficial scratch of our real purpose.

I would love to hear what is your truth and which questions drive you in your path. Drop me a comment if you feel like sharing.

Ron challenged me to write my own implementation of reader/writers lock. He did a great job of doing his own implementation.

I came up with this:
(note: this is really naive implementation, it's far from ideal in terms of fairness and possible racing conditions. Just take it as brain-teaser)

                 1:
                public
                class StateReaderWriterLock
: IDisposable

                 2: {

                 3:
                private
                readonly AutoResetEvent
_changeStateAutoLock = new AutoResetEvent(false);

                 4:
                private
                readonly AutoResetEvent
_writerDone = new AutoResetEvent(false);

                 5:
                private
                readonly
                object _readersLock
= newobject();

                 6:
                private
                readonly
                object _writersLock
= newobject();

                 7:
                private
                int _readers;

                 8:
                private
                int _state; //
0 is "neutral", 1 is read, 2 is write

                 9:
                private
                int _writers; 

                 10:  

                 11:
                public
                void LockReader()

                 12: {

                 13:
                while (true)

                 14: {

                 15:
                //
try to bring the state from "neutral" or "read" to "read". If the current state is
"write", let's wait.
              

                 16:
                while (Interlocked.CompareExchange(ref _state,
1, 0) == 2)

                 17: _changeStateAutoLock.WaitOne(); 

                 18:  

                 19:
                //
an interesting case here where the last reader is now in ReleaseRead and we're trying
to read as well, we might be too late (the writer might have changed the _state already)
              

                 20:
                //
if that happens - we're back to square one, but we still want to avoid recursive locking!
              

                 21:
                bool loseInRace
= false;

                 22:
                lock (_readersLock)

                 23: {

                 24:
                if (Interlocked.CompareExchange(ref _state,
1, 0) == 2)

                 25: loseInRace
= true;

                 26:
                else
              

                 27: _readers++;

                 28: }

                 29:
                if (!loseInRace)

                 30:
                return; //
success!

                 31: }

                 32: } 

                 33:  

                 34:
                public
                void ReleaseReader()

                 35: {

                 36:
                lock (_readersLock)

                 37: {

                 38: _readers--; 

                 39:  

                 40:
                //
if I am the last reader, let's reset the state so any given reader/writer can take
it
              

                 41:
                if (_readers
== 0)

                 42: {

                 43: Thread.VolatileWrite(ref _state,
0);

                 44: _changeStateAutoLock.Set();

                 45: }

                 46: }

                 47: } 

                 48:  

                 49:
                public
                void LockWriter()

                 50: {

                 51:
                while (true)

                 52: {

                 53:
                //
try to bring the state from "neutral" or "write" to "write". If the current state
is "read", let's wait.
              

                 54:
                while (Interlocked.CompareExchange(ref _state,
2, 0) == 1)

                 55: _changeStateAutoLock.WaitOne(); 

                 56:  

                 57:
                bool loseInRace
= false;

                 58:
                lock (_writersLock)

                 59: {

                 60:
                if (Interlocked.CompareExchange(ref _state,
2, 0) == 1)

                 61: loseInRace
= true;

                 62:
                else
              

                 63: _writers++;

                 64: } 

                 65:  

                 66:
                if (!loseInRace)

                 67:
                break; //
great success!

                 68: } 

                 69:  

                 70:
                //
allow 1 writer only
              

                 71:
                if (Thread.VolatileRead(ref _writers)
> 1)

                 72: {

                 73: _writerDone.WaitOne();

                 74: }

                 75: } 

                 76:  

                 77:
                public
                void ReleaseWriter()

                 78: {

                 79:
                lock (_writersLock)

                 80: {

                 81: _writers--; 

                 82:  

                 83:
                //
if I am the last writer, let's reset the state so any given reader/writer can take
it
              

                 84:
                if (_writers
== 0)

                 85: {

                 86: Thread.VolatileWrite(ref _state,
0);

                 87: _changeStateAutoLock.Set();

                 88: }

                 89:
                else
              

                 90: {

                 91: _writerDone.Set();

                 92: }

                 93: }

                 94: } 

                 95:  

                 96:
                public
                void Dispose()

                 97: {

                 98: _changeStateAutoLock.Close();

                 99: _writerDone.Close();

                 100: }

                 101: }

                 102:  

Not sure it's the greatest job interview question, but it is indeed challenging and fun to play with.

Imagine you've got the following code, running in 2 separate threads T1 and T2:

private bool _go = true;

T1                                      T2
while (_go)                        // ... some code here ...
{                                         _go = false; // something happened, let's stop
   // ... do work
}

Even if T2 will set _go to false, this code might lead to endless loop in T1. Why is that?
Each CPU have a local memory called L1 Cache, that might cache the value of _go. Multiple threads running on multiple CPU's can (and will) cache data and re-order instructions to optimize code. So if for example, T1 is running on processorA and T2 is running on processorB, we might have an endless loop here. A simple solution here is to add the volatile keyword on the _go field definition to assure order and avoid caching on CPU level:

"The volatile modifier is usually used for a field that is accessed by multiple threads without using the lock statement to serialize access. Using the volatile modifier ensures that one thread retrieves the most up-to-date value written by another thread." (MSDN)

Now let's examine another example, again 2 threads T1 and T2:

private int _writers = 0;

T1                                                         T2
while (_writers > 1)                              ....
{                                                           Interlocked.Increment(ref _writers);
    _someAutoEvent.WaitOne();
}

You can see that we're using an atomic increment via Interlocked.Increment method, so we should be thread-safe here right? not quite.
We've got one thread that is reading (T1) and another thread that is writing (T2) to _writers. If T1 is running in processorA and T2 in processorB, the _writers value will be cached on the CPU level for some time which means T1 might see different value than T2. If T2 would have catch the returned value from Interlocked.Increment and it as the only reader of that field, then it was thread-safe. This is not the case here.
The solution here is to use Thread.VolatileRead(ref _writers) to make sure T1 gets the latest value. We could have used lock keyword as well of course to serialize access to _writers field.

Summary:

• Although setting a word size variable is atomic (like in our first example), it doesn't mean it is thread-safe! For "boolean flags" I would recommend using volatile as a clean and simple solution.
• For "counters scenarios", I would have used Interlocked with Thread.ReadVolatile. This should outperform lock usage and still keep your code neat and shiny.
• The lock keyword is probably the safest way to avoid dangerous race conditions, so unless you're sure about your solution, keep it simple and use lock.

It seems that someone has upload a 10 minutes video for some time now (thanks TristanIce!).
This is far from being a well prepared lecture (alt.net is all about "everyone can get up and talk"), I just tried to show the charm behind CCR after people started to share their pain developing multi-threaded applications. I'm talking crazy-fast and the material is not well organized so I had to think while talking/writing. The results are hard to watch. Multi-threading is a bitch eh?

All in all, the vibe was positive and people looked intrigued by it. Anyway, here it is:

Oren explains how to use CCR framework (MSN video, the movie is playing on the right)

Requirements:

1. Be able to read all the lines in a given file.
2. Be able to do so even if the file is HUGE ( == don't load it all at once).
3. Control the number of items I want to receive and whether or not the enumerator ignore empty lines. Always nice to have.
4. Thread-safe should be supported easily. Think about 50 threads, each reading the next line and processing it.
5. Nice performance is a plus.

Playing with the API with my teammate Ron gave the following (code written in notepad, stupidity won't compile):

"Common foreach" usage:

foreach (string line in FileStreamer.GetLines(@"c:\temp\myfile.txt", true, 1000)) { /* .. code */ } // read 1000 items from the file while ignoring empty lines.

Reading from multiple threads usage:

using (FileStreamer streamer = new FileStreamer(@"c:\temp\myfile.txt", true, -1)) // -1 means no limit, read all non-empty lines
{
    Thread[] threads = new Thread[10];
    for(int i=0; i<threads.Length; i++)
    {
        threads[i] = new Thread((ThreadStart)delegate {
            string line;
            if (!streamer.TryGetNextLine(out line)) // thread safe!
                return; // end of file, we can exit

            // do work ...
        });

        threads[i].Start();
    }

    // join the threads + whatever ...
}

After reading a few ideas in stackoverflow, I thought to share my solution:    

              //
written by bogen (30/12/2008)
            

              #region
              using
            

              using System;

              using System.Collections.Generic;

              using System.IO;

              #endregion
            

              namespace Semingo.Common.Utils

{

              ///
<summary>
            

              ///
Return a stream of lines for the specified file.
            

              ///
This class is thread safe by design!
            

              ///
Use the static method FileStreamer.GetLines for not thread safe usage (via foreach)
            

              ///
</summary>
            

              public
              class FileStreamer
: IDisposable

    {

              #region fields

              private
              readonly
              object _locker
= newobject();

              private
              readonly
              string _path;

              private
              readonly
              bool _ignoreEmptyLines;

              private
              readonly
              int _limit;

              private
              readonly IEnumerator<string>
_enumerator;

              private
              int _linesGiven;

              private
              bool _disposed;

              #endregion
            

              #region ctors

              ///
<summary>
            

              ///
Create a file streamer instance
            

              ///
</summary>
            

              ///
<param name="path">File path</param>
            

              public FileStreamer(string path)
: this(path, false,
-1)

        {

        }

              ///
<summary>
            

              ///
Create a file streamer instance
            

              ///
</summary>
            

              ///
<param name="path">File path</param>
            

              ///
<param name="ignoreEmptyLines">Should the streamer avoid empty lines</param>
            

              ///
<param name="limit">Number of maximum lines the streamer should return. Send
-1 for no limit</param>
            

              public FileStreamer(string path, bool ignoreEmptyLines, int limit)

        {

              if (!File.Exists(path))

              throw
              new ArgumentException("Cannot
find the file: " + path);

              if (limit
!= -1 && limit <=0 )

              throw
              new ArgumentException("Limit
must be bigger than 0 (or -1 for no limit) but was: " + limit + ".
File given was: " + path);

            _path = path;

            _ignoreEmptyLines = ignoreEmptyLines;

            _limit = limit;

            _enumerator = CreateStream().GetEnumerator();

        }

              #endregion
            

              #region
              public API

              public
              bool TryGetNextLine(outstring nextItem)

        {

              lock (_locker)

            {

              return TryGetNextLineAssumingInsideLock(out nextItem);

            }

        }

              public
              bool TryGetNextLines(out ICollection<string>
nextItems, int howMany)

        {

              if (howMany
<= 0)

              throw
              new ArgumentException("'howMany'
parameter must be > 0 but was " + howMany, "howMany");

            nextItems = new List<string>(howMany);

              lock (_locker)

            {

              string nextItem;

              for(int i=0;
i<howMany; i++)

                {

              if (!TryGetNextLineAssumingInsideLock(out nextItem))

              break; //
no more lines (EOF)

                    nextItems.Add(nextItem);

                }

            }

              return nextItems.Count
> 0;

        }

              public
              static IEnumerable<string>
GetLines(string path)

        {

              return GetLines(path, false,
-1);

        }

              ///
<summary>
            

              /// 
            

              ///
</summary>
            

              ///
<param name="path"></param>
            

              ///
<param name="ignoreEmptyLines"></param>
            

              ///
<param name="limit">send -1 for no limit</param>
            

              ///
<returns></returns>
            

              public
              static IEnumerable<string>
GetLines(string path, bool ignoreEmptyLines, int limit)

        {

              using (FileStreamer
streamer = new FileStreamer(path, ignoreEmptyLines,
limit))

            {

              string nextItem;

              while (streamer.TryGetNextLine(out nextItem))

              yield
              return nextItem;

              yield
              break; //
EOF

            }

        }

              ///<summary>
            

              ///Performs
application-defined tasks associated with freeing, releasing, or resetting unmanaged
resources.
            

              ///</summary>
            

              public
              void Dispose()

        {

            Dispose(true);

            GC.SuppressFinalize(this);

        }

              #endregion
            

              #region
              private API

              ///
<summary>
            

              ///
Get the next line in the file.
            

              ///
dev: assume that the lock is from the outside, by the caller (this is why it's a private
method)
            

              ///
</summary>
            

              private
              bool TryGetNextLineAssumingInsideLock(outstring nextItem)

        {

            nextItem = null;

              if (_linesGiven
== _limit)

              return
              false; //
we reached the limit, no more please.

              if (!_enumerator.MoveNext())

              return
              false; //
end of stream (EOF)

            nextItem = _enumerator.Current;

            _linesGiven++;

              return
              true;

        }

              private IEnumerable<string>
CreateStream()

        {

              using (FileStream
fs = new FileStream(_path, FileMode.Open, FileAccess.Read,
FileShare.Read, 1024, FileOptions.SequentialScan))

              using (StreamReader
reader = new StreamReader(fs))

            {

              string line;

              while ((line
= reader.ReadLine()) != null)

                {

              if (_ignoreEmptyLines
&& line == string.Empty)

              continue; //
skip empty lines if needed

              yield
              return line;

                }

              yield
              break;

            }

        }

              protected
              void Dispose(bool disposing)

        {

              if (_disposed)

              return;

              if (disposing)

            {

                _enumerator.Dispose();

            }

            _disposed = true;

        }

              #endregion
            

    }

}

"Why do you love working as X so much? so much that you're willing to spend that many hours of your life at?"

Pause for a second. Try to close your eyes and think what will be your answer for this question?

For me, the answer is obvious: It's the people and challenges that make my brain tick and my motivation SKY high. It's the feeling that I can really make things better by investing everything I got into it that makes me proud of my work. I get huge satisfaction installing TFS 2008, trying to make our Integration Tests work X6 faster, practicing some Agile principles I've read about or take any other "dirty yet important work" no one would like to touch. I'm not scared of  hard work and if I can feel, down there in my stomach, that it would make my teammates more productive - I'll do anything I can to make it happen. Oh, and I'm trying to build one of the most complex search engine the world has the offer with a bunch of b-r-i-l-l-i-a-n-t guys! Can you blame me for working so hard, enjoying every minute of it?

Sure, getting a few bucks more would be great, but that will not make me proud of what I'm doing. One of the main things I've learned in my 8 years of developing software, is that highly motivated teams will always make the best products. Leave aside for a moment the productivity boost these teams enjoy and imagine their daily work, their lunches together, their working environment, their joy of talking with one another about day to day stuff. Imagine how they dream about their goals together, discussing ways to making it better and more enjoyable. It's the buzz these companies have that drove the best guys to them, so "effortlessly". The commitment to one another will make sure you'll build quality systems, that you'll try your best to deliver on time, to make it better, smarter, BIGGER, everysingle day. It will allow you to grow like you could never anticipated. Trying to grow this culture in your team is one of the hardest things in the world, way harder than any logical puzzle thrown at you. Believe me.

It's just so damn hard to get it right.

This attitude will probably make you a winner, someone that others will enjoy working with, being with, taking inspiration from.
I know that these guys are the one I love working with or going to a bar close by, drinking some beer and talking about how to change the world.

Best people simply do for each other.

We're doing a lot of thinking these days about which features will give us the best ROI, trying to prioritize existing features and asking ourselves "did we miss something? is there a new feature out there we left behind?". It's not easy to think about great one-of-a-kind ideas. It is easy though to make it almost impossible.

Why? it's all because the "parallel" right hemisphere of our brain, imagine the following brain storming conversation:

me(Left brain): Alright I've got one! The user will enter the screen, do X and Y (we'll do some Z behind the scene) to receive ...
   me(Right brain kicks in): but, doing Z will take me two weeks to develop..
   me(R): gosh, we'll need to build a dictionary and hold it in memory if we want it to scale..
   me(R): reminder, use ReaderWriteLockSlim this time. It's much faster than ReaderWriterLock!
   me(R): I guess that this feature is not as important as feature F1, maybe I'm spending my time thinking about this feature??!
   me(R): I'm so hungry!
   me(R): Oh, we can use [some service name] to do Z. Cool, so now this feature is feasible.
   me(R): crap, [some service name] cost money, I think.

[To the surrounding, it looks like I'm saying one fluent sentence of course. During that time they have their right brain working on "why not" / "how" / "when"]
 
   joe(R): gosh, is he for real? this is the lamest idea I've heard!
   jack(R): hmm, maybe he have a point there. This feature reminds me something I've always wanted to do... what was it now??...
   jack(R): naa.. this guy is crazy. for sure.
   joe(R): oh wait! we can use something I wrote to implement this feature! might be cool to use this code finally. It is laying there for ages.
   sarah(R): wonderful idea! I wonder if I'll be assigned to work on it?
   jack(R): I'm listening to his bubbling for 20 minutes now. self reminder: talk about a bonus with the CEO.
   sarah(R): I need coffee! God, if you'll end this meeting now I promise the donate 10$ for charity! coffee... please...

me(L): .. a brilliant search result !!

Any wonder that most brain storming meetings are futile?

Brain storming is a process that should be mastered and I suggest that you'll jump to the nearest browser to find books at the topic, it's a skill worth investing time at.
Before you do so, here are some rules I use to silent my right brain while doing Left Brain Storming:

1. Never ever prioritize your ideas during brain storming. I can't stress enough how important is this rule. Don't worry about it now, you'll have time later. 
2. Listen to others.
3. Be patient = don't judge quality of ideas.
4. Write everything down. I really mean everything! There are no "stupid ideas" now.
5. You are not going to execute these ideas. At least that is what you should tell your right brain during that time.
6. Understand the meaning behind the feature, imagine how it will work, not how it will be executed!
7. Don't invest more than 2 hours in a single brain storming meeting. If you feel you've missed some ideas, rest a few hours (or even better - a few days) and then give it another shoot. "Burned out quickly, left brain does. Burned out leads to impatience. Impatience kicking the right brain in action. Right brain means trouble for your brain storming meeting" -- so does Master Yoda say (well, sort of)
8. 80/20: after you're done throwing out ideas (or the 2h gong), go over the features you've raised and mark features you think are interesting and feasible with 80 and features that are not with 20. This should take no longer than 2 minutes, so please use only 80 and 20 as numbers.
9. Set a separate meeting to prioritize features with the existing backlog you've got. Important: don't do it at the same day, you'll probably want to sleep things over.

Happy hunting!

[ Part 1, Part 2, this is Part 3 ]

In part 1 I've talked about the general notion behind SpawnEnumerator. In part 2 I've implemented it and talked a bit about CCR and how it works under the hood.
In the third and last part I'll try to show how we can actually execute billions of tasks very easily.

Iterating over billions of tasks:

In my previous post I've tried to show one of the greatest yet unused features in C# 2.0 - the yield statement. I've closed the post with:
"Prefer using the yield statement as long as calculating values doesn't require holding an expensive resource like a DB connection, or a FileHandler for long period of time."

Now, let's assume that we have 100,000,000 Tasks to pull from a repository (DB, files, doesn't really matter) and execute them all via SpawnEnumerator.
Should we define a list of Task, fill it with 100,000,000 items and then iterate over it? of course not! we can't even if we want to.
Assuming that reading 1000 (for example) Tasks is much faster than executing them (it usually is), and it's safe to hold them in memory, we can "bulk read & yield" the entire thing:

public delegate T Func<T>();

public static class Yield
{
    public static IEnumerable<T> Bulked<T>(Func<IEnumerable<T>> bulkYielder)
    {
        while (true)
        {
            IEnumerable<T> yielder = bulkYielder();
            if (yielder == null)
                throw new ArgumentException("bulkYielder cannot return a null enumerable", "bulkYielder");

            int itemsGiven = 0;
            foreach (T t in yielder)
            {
                itemsGiven++;
                yield return t;
            }

            if (itemsGiven == 0)
                break;
        }
    }
}

This can be used to return a stream of "tasks bulk" :

Yield.Bulked<Task>(delegate { return tasksRepository.Dequeue(1000); }) // return the next 1000 from the 100,000,000 items queue until no more tasks exists in the repository.

Important note:
In this scenario, we should return 1000 items from Dequeue method without using yield to avoid holding expensive resources for long period of time (as discussed).
Once we have those 1000 items, we can yield each one of them to the caller (very cheap, no resources are used).

Finally, we can execute billions of tasks in parallel via SpawnEnumerator:

IEnumerator<Task> tasksEnumerator = Yield.Bulked<Task>(delegate { return tasksRepository.Dequeue(1000); });
Action<Task> taskHandler = delegate(Task t) { /* execute single task */ };

// execute the tasks in parallel, using 50 threads and holding ~1000 items in memory.
_tasksExecutor = SpawnEnumerator<Task>.Start(50, "mypool", 1000, tasksEnumerator, taskHandler);

// note:don't forget to dispose _tasksExecutor when killing/stopping app!

Using Yield.Bulked guarantees that we won't hold expensive resources for too long while allowing us to generate a "stream" of Tasks to run in parallel.
The code is easy to read and follow (I hope) and we gain a simple method for executing billions of tasks very effectively (CPU & memory wise).

How many of you played with C# yield statement ? I guess that most of you did.
Anyway, like most of the MSDN examples out there, when used incorrectly, it could introduce very bad behavior to your code. Consider the following:

public IEnumerable<User> GetUsers(int count)
{
    using (MysqlConnection connection = new MysqlConnection("..."))
    {
        // MysqlDataReader reader = create an MysqlCommand and execute it
        while (reader.Read())
            yield return new User(/*... fill parameters from the reader ... */);
    }
}

Looks pretty harmless right? Not quite. The yield statement is actually transformed to a "state machine" which means that every time we yield a result back to the client (the caller of GetUsers in our example), we wait for the client to call to the next item (via IEnumerator<T>.MoveNext()). The code above will hold the connection open until the client done iterating all of the User items. This will lead into major scalability issues very quickly! You should always keep your DB connections open for short period to prevent connection exhaustion (threads waiting for available DB connection in the pool for long period, until timeout). Because yield returns the control to the caller, it might be that the caller will "take his time" thus leading to connection exhaustion.

On the other hand, used wisely and yield yields (lame joke, sorry) HUGE benefits:
[note: code written in notepad, stupidity won't compile]

• Avoid useless memory allocations

#1:
How many times you end up creating something like this:

    public List<T> Filter(List<T> input, Predicate<T> predicate)
    {
       List<T> output = new List<T>(input.Count /2);
       foreach (T item in input)
          if (predicate(item))
             output.Add(item);

       return output;
   }

We allocate much more memory than we need only to hold the output during the calculation. A better approach will be:

    public IEnumerator<T> Filter(IEnumerator<T> input, Predicate<T> predicate)
    {
       foreach (T item in input)
          if (predicate(item))
             yield return item;
   }

This way we allocate only one T at a time (will be saved in the generated state machine). In addition, the client could choose to send each item via yield as well, thus saving the need to create the "input" before calling our Filter method.

#2:
Another oh-(gosh-why)-so-common example is the following:

public void Save(T item)
{
    Save(new T[] { item });
}

public void Save(ICollection<T> items)
{
    // do your magic here to save items
}

Assuming you call Save with a single item quite a lot, you're allocating A LOT of memory to create one-item arrays. A better approach will be:

public void Save(T item)
{
    Save(Yield.One(item));
}

public void Save(IEnumerator<T> items)
{
    // do your magic here to save items
}

public static class Yield
{
    public static IEnumerator<T> One<T>(T item)
    {
        yield item;
    }
}

~Zero memory allocation here.

• Avoid "impossible" memory allocation

Let's say you want to read a 20G file with emails where every line holds a single email. Trying to declare a List<string> and filling it up will make your memory blow up obviously. You simply can't hold that much in memory. Instead, you can use Stream.ReadLine and yield back each row to your client, until all of the emails are taken care of. Yes, you can try to read the file in chunks (keeping a pointer), but this is exactly what yield does under the hood. Reminder: DRY principle is gold (or Don't Repeat .Net Framework, in our case).

• Execute synchronous code asynchronously (nicely achieved via CCR)

A bit advanced, but you can read all about it here. The great benefit is you can transform (almost) any "yield based" code to run async, if needed/wanted.

Recap:
Prefer using the yield statement as long as calculating values doesn't require holding an expensive resource like a DB connection, or a FileHandler for long period of time.

It seems that there are MANY ways to perform http web request poorly. This is a huge problem in today's world where web-services are more common than bankrupt banks. Here is a quick pattern of how to do it right:

public string Fetch(Uri requestUri)
{

HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(requestUri);

webRequest.Timeout = requestConnectTimeoutInMs; // take timeout from config
webRequest.ReadWriteTimeout = requestReadWriteTimeoutInMs; // take timeout from config

using (WebResponse webResponse = webRequest.GetResponse())
using (StreamReader streamReader = new StreamReader(new TimeoutStream(webResponse.GetResponseStream(), fetchTimeoutInMs)) // take timeout from config
     return streamReader.ReadToEnd();

}

Details:

1. Setting Timeout property: to make sure we don't wait the default 100 seconds for "ACK" from the server. WAY too much.
2. Setting ReadWriteTimeout: This is crucial to understand. StreamReader under the hood read data in chunks, this timeout determine how much time you should wait for reading a single chunk. 100 seconds, the default value, is again WAY too much.
3. Using TimeoutStream (you need to implement your own or let me know if you're interested and I'll send it to you): Alright, let's say you're willing to wait for 500ms for ACK (Timeout), up to 500ms for reading every chunk (ReadWriteTimeout) but not more than 5 seconds for the entire read to complete. There is no way to achieve it without TimeoutStream. It will start a timer internally and override Seek/Read/Write (etc) method by checking the timer before calling the internal stream method. TimeoutStream is a very simple wrapper around Stream. For example:

   public override int Read(byte[] buffer, int offset, int count)
      {
          CheckTimeout(); // throw TimeoutException if timeout was reached
          return _stream.Read(buffer, offset, count);
      }

Multiple HttpWebRequest limitation:
By default, you can't perform more than 2-3 async HttpWebRequest (depends on the OS). In order to override it (the easiest way, IMHO) don't forget to add this under <configuration> section in the application's config file:

<system.net>
  <connectionManagement>
     <add address="*" maxconnection="65000" />
  </connectionManagement>
</system.net>

Why should you follow these guidelines:

1. Never trust 3rd party components: avoid excuses like "my site is not responsive because 1000 threads are waiting for web-service-X to respond". By setting those parameters you're safe to make your own choices of how much time to wait. Log and monitor these things to adjust your application and alert your suppliers.
2. Be able to determine your own SLA for the world: again, if internally you need to call a web-service, make sure you're able to control the time you're willing to spend. You've got clients to serve and they want you to meet the SLA as you promised!

Important note about recycling HttpWebRequest.GetResposne()
Simply put, it's not working by design. That means that if you fail to get a response on time (due to 1,2 or 3), don't call the webRequest.GetResponse() again as it is cached internally (you'll get the same HttpWebResposne). What you should do is to re-create the HttpWebRequest and try again. I don't agree with the selected design by Microsoft for this method, but at least it's good to be aware of it.

   from MSDN:

   " Multiple calls to GetResponse return the same response object; the request is not reissued. "

Final note:
You should obviously consider writing a HttpWebRequestHelper class (or extension method) and use it instead of copy&paste this code all over your codebase.

[ Part 1, this is Part 2, Part 3 ]

In Part 1 I've talked about the general notions behind SpawnEnumerator and played with the API. If you're not familiar with Microsoft's CCR, this post might require a 2nd & 3rd read to understand completely. CCR changed the way you should think or address async code. It's a game worth playing and studying the rules is only for your advantage. Alright, enough chit chat, let's make it happen (complete code attached at the end).

Class definition:

  public sealed class SpawnEnumerator<T> : IDisposable
       private SpawnEnumerator(int threadsCount, string threadsPoolName, IEnumerator<T> filler) { /* initialize fields by parameters, nothing more */ }

Now, let's have a look at some of the fields:

  private const double DefaultLowThresholdPrcentage = 0.1; // When "items to process" queue reach 10%, we want to re-fill. Should be exposed of course.

  private event Action<DateTime> _enumeratorDepleted = delegate { }; // trigger when the enumerator is empty
  private event Action<DateTime> _allTasksAreCompleted = delegate { }; // trigger when all tasks are completed

  private readonly Dispatcher _dispatcher; // our "threadpool"
  private readonly DispatcherQueue _dispatcherQueue; // hold actual ITask, waiting for the dispatcher (aka worker threads) to handle them. more about it soon.

  private readonly Port<T> _itemsToProcessPort;  // hold wannabe tasks, currently a queue of items of T we want to process.
  private readonly Port<EmptyValue> _itemCompleteNotificationPort; // soon...
  private readonly Port<EmptyValue> _initializePort;  // soon...

  private readonly IEnumerator<T> _enumerator; // the enumerator we'll use to fill the _itemsToProcessPort

Deeper look on what we have so far:

• _itemsToProcessPort: will act as the queue of items we want to process. In CCR's world, Port<T> is actually a "smart queue" (more on it later).
• _itemCompleteNotificationPort: will be used to notify on every completed item. Assuming that we need to fill 100 items to the port, and our lower limit is 10%, we want to re-fill the _itemsToProcessPort every 90 completed items. Notice we're using EmptyValue as T. EmptyValue is a CCR type that holds EmptyValue.SharedInstance to avoid memory allocation.
• _initializePort: will be used to initialize the _itemsToProcessPort with the 1st bulk of items.

Step back, what's going on?!!? why so many Port ?
Well, the idea behind CCR is all about messaging. You can pass messages to ports (Port<T> is thread-safe of course) and by doing so, you can take advantage of the "smart queue" implementation behind Port<T>. When posting a message to a port, the CCR will try to apply some "predicates" on the port and if a "predicate" returns true, it will dequeue the item(s) from the Port matched that "predicate", create an ITask of it and push it into the DispatcherQueue as an "actual task".

Using Ports makes it easier for us to define complex async code. instead of putting locks all over the place, I can simply post a message to _itemCompleteNotificationPort and after X messages posted to this port, ask to re-fill the queue when a thread is available. This is much easier then counting each completed item and if ((counter % X) == 0), lock some object and re-fill. Both will work, but using the CCR world you don't have to think about technical async problems/solutions but rather on the logical async operations you want to perform. You'll write much less code, zero locks of your own and mostly think about "this could run concurrently", "this must run exclusively" and let the CCR schedule everything for you.

Start method: (as discussed in Part 1)
public static SpawnEnumerator<T> Start(int threadsCount, string threadsPoolName, int upperQueueSize, IEnumerable<T> filler, Action<T> handler)
{
  // .. validate parameters, nothing interesting...
  SpawnEnumerator<T> sw = new SpawnEnumerator<T>(threadsCount, threadsPoolName, filler.GetEnumerator());
  sw.Initialize(handler, upperQueueSize);
  return sw;
}

API Design: Why Start method with private constructor instead of public constructor alone:
The client of this method should understand that once she supply the arguments, things will start happening - we'll immediately start to process items from the enumerator. You'll soon find out that Start is non-blocking method. This "Start" method, so I feel, make it's it more explicit as it should be.

Initialize method:
private void Initialize(Action<T> handler, int upperQueueSize)
{
  RegisterRecievers(handler, upperQueueSize); // where CCR *magic* happens. soon...
  _initializePort.Post(EmptyValue.SharedInstance); // post a message to let "someone" know we want to fill the 1st bulk of items
}

RegisterRecievers method:
Before we look at the code, here is a remainder of the main things we want to accomplish:

1. We should fill the _itemsToProcessPort for the 1st bulk or once we reach the lower limit of the queue, by counting how many items were completed. Keeping in mind that _enumerator is not thread-safe and we don't want to start locking access to it on our own, we should make sure that re-filling is done exclusively from 1 thread only.
2. We want to handle each one of the items posted to _itemsToProcessPort with the supplied "handler" (given in Start method). Each item is independent so obviously we want to process each item concurrently, according to the amount of threads in the Dispatcher.

private void RegisterRecievers(Action<T> handler, int upperQueueSize)
{
  int numberOfItemsToDepleteBeforePushingNewBulk = (int)Math.Ceiling((1 - DefaultLowThresholdPrcentage) * upperQueueSize);

  Arbiter.Activate(_dispatcherQueue,

     Arbiter.Interleave(
         new TeardownReceiverGroup(),// nothing here
         new ExclusiveReceiverGroup(
                // 1st bulk:
                Arbiter.Receive(false, _initializePort, delegate { FillItemsToProcessQueueWithNextBulk(upperQueueSize); }), // only once, that's why the "false" is here.

                // enough items were completed which means "items to process" queue reached lower limit:
                Arbiter.MultipleItemReceive(true, _itemCompleteNotificationPort, numberOfItemsToDepleteBeforePushingNewBulk,
                                          delegate { FillItemsToProcessQueueWithNextBulk(upperQueueSize); })
            ),
         new ConcurrentReceiverGroup(
                // process items concurrently
                Arbiter.Receive(true, _itemsToProcessPort, // listen to every post, that's why the "true"
                     delegate(T item)
                     {
                         try
                         {
                             handler(item);
                         }
                         catch (Exception err) { /* log error */ }
                         finally
                         {
                             HandleCompletedItem();
                         }
                     })
            )
        )
    );
}

Alright, this is a bit harder to read but let's try to simplify it by reading it from inside-out.

• Under ExclusiveReceiverGroup you'll see 2 receivers, one that listen to _initalizePort and once a message is posted, it will create an ITask that internally (when a thread is available) call FillItemsToProcessQueueWithNextBulk method. The second receiver will listen to _itemCompleteNotificationPort and do the same as the first one for every numberOfItemsToDepleteBeforePushingNewBulk items posted to the Port.
• Under ConcurrentReceiverGroup you'll see a receiver listening to _itemsToProcessPort and for every message it will create an ITask that internally (again, one a thread is available) will run the given "item handler" with the item dequeued from the Port.
• We use Arbiter.Activate(_dispatcherQueue, ...) to register each created ITask from the receivers to the internal queue. Queue of actual tasks.
• This method is non-blocking, we only register receivers that know how to create and enqueue ITask from messages posted to some Port. That's it.

Recap:
We post messages to different Port<T> and write receivers that "listen" to the messages based on some rules (one receiver listen to every message, one listen to only one message, one listen to X messages etc). When the receiver's rules apply, the CCR will dequeue all the relevant (applied the rule) messages from the Port<T> and wrap them with the supplied delegated as ITask. This ITask instance will be enqueued to the DispatcherQueue until the Dispatcher have a free thread to handle it. The Dispatcher, according to the requested scheduling (some ITask must run exclusively, some can run concurrently, as we've seen), will execute those tasks.

You can download the complete code (with some extra features) here: SpawnEnumerator.txt (12KB) - you'll need CCR & log4net dll in order to compile it.

[ this is Part 1, Part 2, Part 3 ]

I promised to write more about CCR and how to use it in practice.
There is no better way to explain how infrastructure work then writing some code and play with it, so lets toy with a simple utility, based on CCR under the hood.

Scenario: you've got billions of *independent* tasks you need to execute as fast as possible.
Requirements:

1. Amount of "worker threads" should be easy to define. obviously.
2. Memory consumption - don't eat more than you can chew. We don't want to hold everything in memory, this should be easy to configure.
3. The process should *not* shutdown due to un-handled exception in one of the worker threads.
4. We want to know when the enumerator is depleted and/or all items were completed. This is crucial for testing and adjusting parameters.

The general notion behind SpawnEnumerator is something like this:

We're pulling a bulk of items from the enumerator, each one of them is actually a "wannabe" item. Combining the value of each T with a given "item handler" could be transformed into an actual task, waiting for its turn in the "threads pool". Once the port/queue of "items to execute" reach the lower limit, we'll pull another bulk of items from the enumerator until the enumerator is depleted.

API playground:

SpawnEnumerator<T>.Start(int numberOfThreads, string threadPoolName, int numberOfItemsToHoldInMemory, IEnumerator<T> enumerator, Action<T> singleItemHandler)

T = type of items we would like to run. if we have 10,000,000 items of type int, we'll supply "int" as T.
numberOfThreads = how many worker threads do we want to run?
threadPoolName = name the worker threads for easier debugging.
numberOfItemsToHoldInMemory = rough number of items we want to hold in memory, to make sure memory consumption won't blow up in our face.
enumerator = the enumerator of "items" to execute, assuming that it will hold a *huge* amount of tasks.
singleItemHandler = delegate that receives 1 item and handle it.

event Action<DateTime> EnumeratorDepleted

event Action<DateTime> AllTasksAreCompleted

Test playground:

(written in notepad, sorry for stupid mistakes)

[Test]
public void Execute_LargeEnumeratorOfNumbers_ExecuteAllItems
{

// arrange:
int numberOfThreads = 10;
string threadPoolName = "mypool";
int numberOfItemsToHoldInMemory = 100;

IEnumerator<int> items = Yield.For(1, 10000); // yield return numbers from 1-10000
Action<int> singleItemHandler = delegate(int item) {  /* sum the given number, via Interlocked.Add */ };

ManualResetEvent trigger = new ManualResetEvent(false);

// act:
_executor = SpawnEnumerator<int>.Start(threadPoolSize, threadPoolName, numberOfItemsToHoldInMemory, items, singleItemHandler))
_executor.AllTasksAreCompleted += delegate { trigger.Set(); };   

// assert:
bool signaled = trigger.WaitOne(TimeSpan.FromSeconds(1), false);
Assert.IsTrue(signaled, "timedout reached, that shouldn't happen!");
// assert that all of the items were called by matching sum of 1-N sequential numbers (simple formula) to what we collected in the handler

}

* In the "test teardown" we can check if _executor is not null and if so Dispose it to close the "worker threads".

Next post - Implementing SpawnEnumerator via CCR.

This is mostly a self-note but heck, maybe someone will reach this post via Delver (or Google :)), so I'm all about sharing.

Anyway, we're using (for now) Bugzilla and I tried to get all the "open" bugs for my team with status equal to X,Y,Z.

Sounds easy right? well, you are ... sadly mistaken.
After investing 15 minutes banging my head into the nearest wall with “Bugzilla Advanced Serach” (well, I'll be polite and say it's "advanced" alright), I gave in on that one.
Instead, I hacked the url a bit to understand the dark voodoo of Bugzilla and voila, 2 minutes later:

http://qabugz/cgi-bin/bugzilla/buglist.cgi
?bug_status=NEW
&bug_status=ASSIGNED
&bug_status=REOPENED
&assigned_to=Joe
&assigned_to=Joe2
&assigned_to=Joe3
&query_format=specific
&field0-0-0=bug_status
&field0-0-1=assigned_to
&type0-0-0=anyexact
&type0-0-1=anyexact
&order=bugs.bug_severity

Just remove the “break line” of course (this is easier to edit) and replace "Joe" with your favorite developer name.

If you want to make it a bit more complex, here is a nice site with the supported field names:
http://pkp.sfu.ca/bugzilla/page.cgi?id=quicksearchhack.html

Anyway, I hope it will help someone.

* geeky btw – notice the funny “binary index” of the fields! made me laugh quite a bit, being the geek that I am.

One of the biggest challenges in management is be able to track your own rhythm, making sure your plans are executed and things go smoothly.

In my previous post about Driven By Self Organization I stressed the importance of making things visible and plan for the future. How can one set the environment of the team to drive the entire team to success?

Thinking about it lately made me think that I have my own thoughts about what should be visible, how the team should react internally and how should one behave in such team as a Team Lead. Don't get me wrong, I don't have 30 years of experience leading small<->giant teams for small<->giant companies, these ideas are solely based on my gut feelings: making things SIMPLE (KISS) so the entire team will be driven by self-organization without the "burden" of self-management. If things are easy to do, it's easier to get better at them.

The trick is to allow all of the members in the team to be part of the organize->execute game. Some will play, some won't, but they will all be affected by the always-ready environment and notice how inner-interaction change their day. By making it visible, they will be motivated to take action (it will feel natural). I came up with this drawing to expose our sprint plans and progress:

This is a very SIMPLE presentation of the current sprint features and every-day progress - "cards" will move between columns as people work on them.
There are some magic "self organization" tricks integrated into this 1-Visible-view:

1. Achievement-based planning: Before arranging the features/cards, we try to set the "deliveries" for the week. This is a free-text (don't be tempted to make it a list of features) ideas to "set the mood" for the week. It will describe features we want to finish, some design we need to handle before the next week, some quality check we want to pass ("make sure no critical/major bugs are open") etc. They help to plan the week as they define goals that are measurable, driving the team to achievement-based planning rather than like-best-do-first development.
2. Visibility is key: Every member of the team knows exactly what's left for the entire team on a weekly level. No more "but I finished all my tasks two days earlier than we've talked! what can I do that Joe is new here and couldn't continue as you thought? oh, I didn't know we are dependent on his task before we can release the package..."
3. Help each other: remind yourselves that things need to be completed, help one another by helping out when someone is behind of schedule (alright Joe, I'll take Feature A, don't worry). The board will make it easier to know "where can I help?"
4. We want Quality: Nothing is DONE until it's tested and fixed. The idea of splitting "coded" tasks from "tested" tasks is to set the mood for "production ready" code.
5. Small goals are easier to achieve: Splitting the sprint into smaller chunks make it easier to win small battles. Each week defines small trophies - the "deliveries" we promised for that week.
6. Plan leftovers for tomorrow: at the end of every week\sprint, you could easily see what was left. Discuss why it failed and plan it for tomorrow (=next week or next sprint).

Team Lead in such a team will mostly act as a coach, helping the team members to split the features into tasks, remove obstacles, motivate cooperation and taking notes about "how can we get better?". Most importantly, it will allow her (or him) to be productive and feel he can help the team's effort instead of the constant-chaos feeling managers tend to have when things go poorly. The team members are aware of the plans and can balance the efforts to break loose of this chaos-like feeling.

Think about a young fellow, wanting to get into shape. Here are 2 scenarios of how one can tackle that wish:

1. Create a plan and "manage yourself" to keep it.

   20:00 - set the clock to 06:00
   06:00 - wake up, make sure you're not falling back to sleep! get dressed etc
   06:15 - make sure to eat something small and drink some water
   06:35 - run 5km
   08:00 - make sure your bag is ready for school
   08:15 - go to school
   ...
   20:00 - set the clock to 06:00

2. Organize things to drive your day:

   20:00 - put your shoes next to your bed, set your clock to 06:00 and set it FAR from the bed, prepare a little something to eat for tomorrow morning, prepare your bag for (tomorrow's) school.
   06:00 - wake up - go close the damn clock (you need to stand in order to do it), get dressed etc.
   06:15 - eat something small and drink some water
   06:35 - run 5km
   08:00 - relax for a few minutes and then go to school
   ...
   20:00 - put your shoes next to your bed, set your clock to 06:00 and set it FAR from the bed, prepare a little something to eat for tomorrow morning, prepare your bag for (tomorrow's) school.

Where most of us fail?

I might be wrong, but it seems that a lot of us (myself included) simply can't manage our time wisely during the day, for long period of time. It's too easy to forget something to do TODAY when you had to force yourself to plan it TODAY. It's too easy to fail. It's too easy to stop the rhythm.

This is why most of us can't lose weight, can't get into shape, can't read 5 books every month etc.

Driven by Self-organization

To me, it means that I want to set my environment to drive me into success. If I'll take the time to prepare my tomorrow, little chance I'll fail due to laziness: (1) I'm planning for TOMORROW, so what do I care to invest the time? I only plan things and set the environment / mood. I don't need to run the 5km now, I only want to make sure it will be easier to achieve tomorrow and (2) tomorrow morning, surprise surprise, everything is ready for me! I don't need to wake up and find out that I forgot a stupid thing like the fact that my shoes are in the washing machine. The rules are pretty easy: de-couple planning from performing and make sure everything you need for those actions are visible and available for the time you'll need them.

Although it seems like planning is more tedious than actually doing the task, planning for the future is quite relaxing. Try it: plan your tomorrow at the end of today, think about what should be set so your tomorrow will go smoothly. It will make it easier for you to come in the morning and simply perform, without the burden of planning fast to perform now:

"hhmmm... alright, it's Wednesday, what should I do today... gosh, so many things to complete! Maybe I'll start with sending those emails... naaa.. don't have power for that now.. maybe I'll finish that task I promised yesterday! naa.. she don't need it for today anyway... crap! I don't have the energy to deal with it! alright, first thing is to grab a cup of coffee".

This leads to "do what I like best first" syndrome.

At the end of the day, before going home, go over the things you've accomplished and try to see what is left for tomorrow. Relax, you don't need to do those chores now. Write everything you should accomplish for tomorrow down (with time estimations) and go home smiling, knowing that your tomorrow is best planned for achievements rather than for your personal whims.

I'm preparing a set of articles regarding Microsoft CCR and Parallel Extensions, trying to explain the multi threading libraries supplied by MS and how they are going to change our life in the multi-core world.

I thought about starting with some architecture point of view for each library, going over the data structures and then demonstrate usage for real life scenarios. I'll start with level 100 articles and dive deeper until complex scenarios and how to harness the framework's full power to your needs.

We had the pleasure here at Delver working with CCR so I'll share with you some of the code base we created on top of it, some tips for using it correctly and my take about using these libraries in mainstream applications.

Some nice articles to get you started if you didn't hear about these libraries so far:

Parallel Programming with .NET : Most Common Performance Issues in Parallel Programs

Concurrent Affairs: Concurrency and Coordination Runtime -- MSDN Magazine, September 2006

Channel9 Wiki: Concurrency Runtime

Parallel Programming with .NET : Coordination Data Structures Overview

Parallel Programming with .NET : Fork/Join parallelism with .NET CountdownEvent

Parallel Programming with .NET : Wrapping an APM implementation with Future<T>

Is there anything specific you're interested to read about regarding those topics ?

Having to parallelize almost every bit of code here at Delver, some common patterns emerged while we wrote a lot of back-end code.
I remember reading about a new framework from Microsoft Robotics Division called "Microsoft CCR" (Concurrency and Coordination Runtime) a few months ago in the "concurrent affairs" column at the MSDN Magazine but I didn't pay much attention to it at the time. Two weeks ago, it jumped back to my mind so I revisit the article and started diving a little deeper into it, thinking about what sort of problems it can solve in my world and if it does, where could I use it to our benefit. If you don't know anything about the CCR, there is great public content published already like the CCR User Guide but I'll try to give you a 2 minutes intro of the general architecture. The way CCR is built is very much like the SOA world, using messages to communicate between "services". The major components are the Dispatcher which is actually an array of OS Threads, the DispatcherQueue which holds a queue of delegates to run so the Dispatcher can "look" at the DispatcherQueue and when it has a free OS Thread available, it pulls one delegate out of the queue and run it. So far - we've got a classic ThreadPool. There are some differences but I'll let you read about it in the User Guide. The third component, which is the most important one is Port. Think about Port as a pipe that can receive messages (FIFO style - first in, first out) and hold them until someone will know what to do with them. The last component is the "manager", the Arbiter; Arbiter expose a set of methods that allows you to listen to a given pipe and if some conditions are met on the messages the pipe contains, we can take the message(s) and transform them into a runnable delegate placed in the DispatcherQueue.

One of the goals for this library is the make sure you've got much less places to go wrong, by exposing a set of common async patterns you can easily use to guarantee clean(er) code that is easier to read. Think about sending messages from one pipe to another, creating a flow-based code via messages rather than spaghetti code with a lot of messy indentation. This is a very powerful architecture.
Obviously, the entire CCR framework is thread-safe by design so no need to protect the library. A simple example:

There is no need to create an array of Threads and some sort of Queue<Uri> in order to pull the items. The "ThreadPool" is implemented as part of the CCR.
So far no big deal right? well, it turns out that you can easily write common patterns with much less complexity: almost no locks, less spaghetti code and much less code in general.

One of the patterns we (all of us) use a lot is the "execute-and-wait-till-finish" pattern where you've got a list of independent items you want to run in parallel, but you want your main thread to wait for them to finish before continue. The simplest way to achieve it is by creating an array of Thread, activating them with a delegate and then call Join() on each one of the Threads. Let's define a few more requirements for this pattern:

1. We want to be able to know about all the errors that occurred during the execution.
2. We want to be able to set a timeout so each operation(inside a Thread) won't take more than a sensible time.
3. We want to be able to know which items were timed out and when.
4. We want to be able to know which items were completed successfully.
5. BONUS: We want to avoid writing the obvious. 

Well, in order to implement these requirements from scratch, we need to use an outer timer with some sort of List (for example) so each thread will "register" to it when it begins and "unregister" when it's done. The timer should be able to interrupt the thread and be optimized to "wake up" as soon as possible (determined by the registered threads and which thread needs to wake up first(next in line)). In addition, we need some sort of List of exceptions to collect all the exceptions that occurred and make sure we lock shared objects. We'll need to use Thread[] and some sort of Queue to enqueue\dequeue items to\from it. A lot of code for a simple pattern.

With Microsoft CCR it's much easier.
Assuming that we want to handle a list of strings:

List<string> messagesToWorkOn =new List<string>();
for (int i = 0; i < 10; i++)
   messagesToWorkOn.Add("message "+ i);

Here is the final API I've implemented on top of the CCR:

using (SpawnAndWaitPattern<string> saw =new SpawnAndWaitPattern<string>(5, "mythreadspool"))
{
   AggregatedResult<string> result = saw.Execute(messagesToWorkOn, TimeSpan.FromMilliseconds(500),
                                             delegate(string msg)
                                             {
                                                if (msg.Contains("1"))
                                                   Thread.Sleep(2000); // simulate time-out
                                                elseif (msg.Contains("5"))
                                                   thrownew ArgumentException("5 is a very bad value..."); // simulate exception
   
                                                Console.WriteLine("The message: "+ msg +" processed successfully.");
                                             });

   Console.WriteLine("Done!");
   Console.WriteLine("Summarized Report:\n completed results: " 
      + result.SuccessfulItems.Count +"\n exceptions occurred: "+ result.FaultedItems.Count 
      +"\n timed-out results: "+ result.TimedoutItems.Count);
}

We've got 5 OS Threads, we're waiting for up to 0.5 second per item and we've got a full result object, holding all the requirements from above.

The code of SpawnAndWaitPattern class is quite neat and contains 0 locks (on my behalf, the CCR manage its own locks). The CCR schedule the tasks for me; combining it with thread-safe Port and we've got a very powerful yet simple framework at our hands. I decided to attach the entire solution (with A LOT of nice-green-comments) including the Ccr.Core.dll file so you could play with it:

CcrPatterns.rar (162.64 KB)

Have fun.

Alright alright, so I didn't post anything for... a decade or so.
but I'm here (at the office that is) all day long, being a part of a great Team, building the greatest\coolest piece of software I've ever dream of.

I promised myself that I'll be short this time so here it goes, Oren's 60 seconds update:

1. We've changed our name from Semingo to Delver (delver: (n) deep thinker; one who investigates data).
   Hopefully (if God will hear his little buddy here), the phrase "To delve" will catch up with the scary "Google-it".
2. We're going to show our product to the world at the DEMO conference (28-30 January, yes, in 4 days!) in Palm Desert, CA.
   If you want to be one of our first beta users, please go to our site: www.delver.com and register (we'll send you an email once we'll release our beta).
3. We're looking for super talented folks to join our amazing Team, interested?

Imagine that you're living with one hell of a crazy wife. Every day she's giving you bunch of tasks. "Mow the grass", "water the plants", "take out the garbage", "replace the light in the kitchen", "build a fence" etc. For every task you complete, she goes bananas and create 10,000 more on the spot, all related to the task itself ("build a fence" leads to "paint the fence", "put a nice sign on the fence", "practice your wax-on, wax-off" … you get the drift). Reluctant to perform all of these tasks but smart enough to know that you'll lose at least 50% of your property (divorce are nasty), you start collecting these tasks (you write them on papers). You take one paper (single task), perform it and return to your wife with a big-fat smile of your face. She, in return, creates 10,000 new related tasks to the one you've just completed, write them on paper and put it in a BIG box. Every once in a while she's not waiting for you and adding "main" tasks to the box by herself. After performing one task, you pick another one from the box (FIFO), without knowing if it's a main task or not, you go to your way "eager" to perform that task as requested (as if you have a choice). This goes on and on and on…

The box is the tricky part here. Can one box hold billions of papers? hardly. So you start collecting boxes and now it's getting harder as you need to add new boxes when needed, find the "right" box to pull tasks from and making sure these boxes won't break (maintenance) with time.

Here are a few assumptions you can take as is:

1. Your wife is looking for perfection but only in the "main" tasks which means that if you were to build the fence, you have to do it perfectly so each mini task related to it is crucial.
2. Your wife tends to forget things so you can assume that occasionally, she'll add new "main" tasks that were already performed or exist in a different box.
3. You can't throw away tasks "just because" as you don't know if a "mini task" will be thrown by mistake (= your wife will be pissed off. 50% is gone.).
4. Be cool - you won't perform the same task twice. This one is on me.
5. Drinking RedBull (or XL or whatever energy drink you're familiar with) 24-7-365 you don't need to rest. You don't need to sleep. Think robot (funny combination, for 2007).

How to store billions of tasks?

I kinda like the "green feeling" of a forest. Oh right, we also need them in order to breath (El Gor is more convincing than I. Thank God). Most importantly, it costs a lot of money buying so many papers! And the boxes!  You'll need a lot of green ones ($) - not trees but we can't "breath" without them as well). Oh well… You're starting to build a "Boxing mechanism", hire a few guys to maintain them, getting a VC to give you some extra $$$ and after a few months\years you got it cover!

What do you do if you don't have the extra $$$ or more important the extra time to develop this kind of storage system?

How to store billions of tasks? You don't. You can't. 
In most scenarios, when things seem too difficult to accomplish (with the given limits) try a different angle: "If you don't like the answer, ask a different question".

We know that each task creates a lot of new related tasks right? We also know that keeping those new tasks is the tricky part (the BIG box problem) so what else can be done? Let's change the question. "How do I make my wife happy?" seems like a smarter question. If it's expensive to save those tasks why not doing these related tasks on the spot instead of storing them in the box(es)?  How is this going to help us? Now we can throw away tasks because these tasks will be added later on (assumption #2. Thank God your wife is not the robot). Assuming that we can save about 1,000,000 papers in one big box – we're all set. If the box is full, we'll simply throw away new "main" tasks, feeling good as we KNOW that we'll get to them later on (again, assumption #2). Now all we need is one simple box with a limited amount of papers. Less $$$ to waste and much simpler storage system to develop.

Crap, it just hit me. I'm doing some cool sh$% at Semingo! Join us!

Well this is mostly a good backup-post, but heck maybe a few other (VERY COOL) geeks will find it interesting.
Looking for some cool IDE colors\fonts, I came up with this:

[ Regular mode ... ]

[ Marking the for loop ... ]

[ Output window ]

I based my colors on ZenBurn.
The font I'm using is Consolas.
Read all about it in Jeff Atwood's post.

You can download MY version here: OrenEllenbogen_DarkSchema.rar (58 KB)

Note: if you're using ReSharper you'll have to disable the "Highlight current line" option. 

What is more important to you - having the brightest dude in the world in your team, doing his magic with God-like authority or real "together-will-conquer-the-world" Team work? Tricky question...

 

For those of you who don't know the TV series "House", this is your wake up call! Go see it. Now. Seriously.
Well, if you don't have the time or you're just too damn eager to read my post, we'll, "you're an idiot!", but that's your right so I'll give you a short summary: Dr. House, played by the genius actor Hugh Laurie, is the go-to-guy for all the rare cases where the rest of the doctors go bananas. With his extremely cynical point of view and shameless wittiness, combined with a very bright, analytic and (yet) creative thinking, he manage to solve all (we'll, almost) of these cases and still being a complete jerk to his "teammates" during the show. Just a few pearls from Wikiquote so you'll get the drift:

         Dr. Cuddy: You don't prescribe medicine based on guesses. At least we don't since Tuskeegee and Mengele. 
         Dr. House: You're comparing me to a Nazi? [admiringly] Nice ...

Moti and I have decided to form an invite-only Scrum Clan.
We would like to tag Pasha Bitz (snapshot below) as the 3rd clan member.

Pasha, choose carefully, you can only tag one Scrum-Lover like yourself to this distinguish clan ;)

p.s - If you want to be part of the Scrum Clan, please drop a comment and you *might* get an invitation...

                                  Scrum Rules !

Let me start with an out-loud recap of this post: Agile is not something you can put on a bread nor is it "a certain path to success".

It's about STATE OF MIND.

If I had to describe the meaning of "Agile" to a new teammate I would say: Agile is a constant thinking about how we, AS A TEAM, can produce working features to our users with high quality within a short time-frame.

Don't worry:
It's really OK to provide only a subset of feature(s) in one sprint.
It's really OK to leave SOME designing\architecture issues for later on as long as the high-level architecture is good enough (=you're comfortable with it) to answer the big questions.
It's really OK to implement only two REALLY-DONE-HIGH-QUALITY features in one sprint over four semi-working-not-demoable features.

The key here though is not really the practices, it's about the big bullets(again, state of mind):

1). Produce value for your customers and adjust\adopt early.
2). Build a Team (self leadership).

These ideas are hard to implement and require special kind of people. Putting the Team in front of yourself is not a very job-secure attitude.  The ability to help your teammates, shift tasks, taking ownership, critisize yourself and your teammates and getting better, produce high quality design, tests and code - all of it - requires versatile people with unique state of mind (and unique abilities, of course). It's worth it. When things glue, it's a real magic; Things start to get going by the Team, improvements and features starting to come from the developers\QA\Graphic Designer, adjustments are made on a regular basis, changes are welcome and productivity is celebrated.

You can feel something is going right.
That's Agile.

The way WCF proxies are designed is to live until shi* happens.

Let's assume that we have a CalcualtorService with one method named Divide(int a, int b). Sasha, a cool programmer-dude, trying to produce some usefull software writes:

public MyCalcualtorForm : Form {

   private CalculatorProxy _calc =new CalculatorProxy();

   Calc_Click(...) {
      _calc.Divide(firstNumber, secondNumber);
   }
}

What is the first error you can think of that could happen? Yep, DivideByZeroException.
Once the proxy gets an exception, it enters into a "Faulted" state which makes the proxy unusable(=you cannot use it again).
The quickest solution is to work "by the book" and create a new instance each and every time we need to call the service:

Calc_Click(...) {
   using (CalculatorProxy calc =new CalculatorProxy())
      calc.Divide(firstNumber, secondNumber);
}

But what's bad in this solution?

1. Performance - you pay (not a lot but neither little) for each creation of the proxy. Sure, it will probably not be your bottleneck, but heck, why is it useful? Most of the time the proxy will not throw an exception and yet we need to create it every time just to avoid the faulted state scenario. 
2. Design - If we declare this exception BY CONTRACT, I would expect that the proxy will still be usable afterwards. Do we really want to return Enum\int\string as status instead of throwing exception just because of poor design?
3. TDD - you know that I'm in love with it. Now imagine Dependency Injection. Component A recieve ICalculatorProxy and use it to... calculate something. Working "by the book" is no good as we want to recieve an instance of the proxy from the outside in order to mock it. Right, so we inject a proxy from the outside (got to love Windsor) and life is pretty sweeet. Darn! Wait! one poor (even by design) exception and our proxy goes dead. Very un-TDDish of Microsoft.

I had to come with a solution as no one will take TDD away from me. I present to you ProtectedProxy: this little IL-code-at-the-end-of-the-day will able you to recover from faulted state by creating a new proxy on each exception thus making your proxy... useable (couldn't think about a better word to describe it). Think about a situation where your proxy is trying to call the service but the service is down; In Semingo, we decided that we want to keep trying until the service is up. Via ProtectedProxy, you can determine how many times do you want to recover and when you should finally kill the proxy. Oh yea, ProtectedProxy uses Windsor in order to create new proxies if needed and logging messages to log4net. Good stuff.

In the example above, all Sasha had to do was to:
1). Initialize the _calc field by:
        ProtectedProxy<ICalculatorProxy> _calc = new ProtectedProxy<ICalculatorProxy>(new CalculatorProxy());
2). call _calc via:
        _calc.Instance.Divide(firstNumber, secondNumber);

But enough said, code please:

// Written by Oren Ellenbogen (07.08.07) - trying to protect our proxies so they could recover from:
// (A) The service is not up yet, but we want to try again later.
// (B) The service throws (ANY) exception, we still want our proxy to function (bubble the exception, but still keep on working).
// Microsoft intended to use a NEW proxy per call, but for TDD this is not ideal as we would like to inject proxies from outside as mocks
// in order to simulate multiple scenarios.

#region using

using System;
using System.Reflection;
using System.ServiceModel;
using Castle.Core.Resource;
using Castle.Windsor;
using Castle.Windsor.Configuration.Interpreters;
using log4net;

#endregion

namespace Semingo.Services.Proxies.Helpers
{
   
   publicinterface IProxy : ICommunicationObject { 
      bool Ping();
   }

Hours of joy...

Almost forgot, on the next post - "How to TDD WCF code" - stay tuned...

One of the most common question in moving towards Agile Development is "Where should I start from?". If you'll ask me, setup a Continuous Integration (aka "CC") would be the first thing you should start with.

Step 1 (Check for compilation bugs): Code Quality ~= 30%

The CC should be able to identify check-ins to your source control, get the latest source and compile it. The output should be either "green" (Everything compiles successfully with 0 warnings) or "red" (more than one warning or compilation errors). In addition to the fast feedback, the output should also include the files that were changed from the last build (and by whom, so people could know where to look).

The immediate value is priceless. the ability to SEE whether your source-code is stable enough to allow other programmers perform Get Latest and continue their work and the "fail-fast" attitude can save you a lot of time in the long run. It's important to realize though that even if the code compiles without warnings, it still doesn't mean that you could count on the quality of the code.

Step 2 (Check for component-based quality): Code Quality ~= 70%

If you can go one extra mile, write a few automated tests (via one of the available XUnit frameworks) for your components. This means that you are able to inject the component's dependencies from the outside and simulate mini use-cases on component's level. This step is crucial even if you write those after the code itself was written. Let the CC run them if Step 1 is OK. This should allow you to catch the majority of your bugs (I'll leave the "how to write good tests" to another post). If all is green, you know that the system behaves as expected, at least to some extent.

This step is not trivial as it requires you to design for testability and invest in proper testing. Don't let go of this step though, automated tests on this level will make your life much easier. It will take your code one (major) step ahead in "write code that could be changed". Agile is all about that state of mind.

Step 3 (Check for integration-based quality): Code Quality ~= 80%

Now that you're components are behaving as expected, you should try to write a few (automated, of course) tests that simulate the entire flow of 2 or more components (DB is a component as well) in the system. As the system grows and more uses cases are added, you should try to improve these tests as they give a solid proof that the SYSTEM works.

Step 4 (Make everything visible): Code Quality ~= 90%

The state & quality of your source control should be visible to the Team and Management as you want to insure IMMEDIATE response time in case someone check-ins a low quality code (on any level). Fixing a failing test three days after the change itself is a bad symptom of low visibility or low perception, by the Team, regarding the importance of the quality of the system.

Step 5 (Automated deployment): Code Quality ~= 95%

After successful build you would like to deploy the latest source on a dedicated environment which the developers could play with before deploying to another (testing?) environment. This won't be a stable environment, but at least it will give a quick look at the current state of the system - the way customers would see it.

Step 6 (Procedures checking): Code Quality > 95%:

You can add many more checks to the flow, such as Tests Coverage or FxCop. Leave those to the end. From my experience, Time .vs. Value in these features will vary from team to team. You'll gain much more from investing in Steps 2-3.

Semingo CC:

Each developer & manager on our team have a CCTray(the little red circle in the little picture above) which is either Red(source control is damaged), Yellow(Build in action) or Green(Life is sweet). We're using Cruise Control.Net, CCTray, MSBuild (and TFS plugin) and NUnit to perform all of the above. 

A few teasers:

* Image of Steve Urkel (from the famous Family Matters TV series) is shown for failing build.

* On the right you can find Pasha (with a V sign I've added), one of our finest hackers modeling a successful build. You can also notice the 582 green tests (including Integration tests) and 2 changes made by Sagie since the last build.

* Going to NUnit Details, you can get the full details:

I had to cut the pictures in order to keep a sane width for the post, but you can get the drift.

btw - Aviel, yet another Semingo hacker add a "Doh!" (Simpson) each time the CC is red on his computer. It can be quite funny (and scary, if fully concentrated on code).

A few pearls from the office:

• Pasha: Damn, I like TDD so much. Writing a 5 minutes piece of code takes about a day (sarcastic tone, of course).
  
• Aviel: in my case, code does not come from my head but rather from my heart.
  Pasha: I saw your code. It comes from your ass.
  
• Me: Integration environment is the Tamagochi of an Agile Team
  
• (Phone ringing, Pasha answers)
  Robert: I'm Robert.
  Pasha: I'm Robert.
  Robert: I'm Robert.
  Pasha: I'm not Robert, I'm Pasha.
  Robert: Oh, My bad.
  Pasha: Bye then.
  (end of conversation).

After almost 1.5 months of Scrum at Semingo (a baby startup), I decided to expose the way we work at the moment and talk about the adjustments we've made in order to suit Scrum to our needs.

I'll start with our implementation to the Daily Stand-up Meeting. No doubt, our meetings are pretty funny (most of the time) and create the right vibe for the Team. The one thing I love most about our meetings is that by the end of the meeting, I know what is the general work-plan for each member in my Team.

Now it's easy to know what I'm planning to complete today (I spend 5-10 minutes planning my day before the meeting), when & if I need to finish something earlier (or at least decide about interfaces after the meeting) in order to integrate with others, help out or ask for someone's else help(code review for example), stay after the meeting in order to talk about something that pop up during the meeting, remove impediments (if I can) and most importantly - have a good laugh before the day begins.

Daily Stand-up Meeting (aka DSM) structure at Semingo:

When: 
Every day at 10:30.
Where:
Meetings room.
Time Box:
15 minutes.
Attendants: 
Pigs only (Chickens can (only) listen)
On the table: 
Each Pig answer these 4(!) questions:
(1) What have I done since the last DSM ?
(2) What am I planning to do until the next DSM ?
(3) Impediments - what bothers me to work ?
(4) Am I on track?
     If someone feels that one of his tasks won't be finished as planned, a flag is raised so the Team could assist. 
     The same goes if someone feels that he's going to finish before the expected time. 
     This means that he could help out someone else or take a few extra tasks we did not plan for the current iteration.

Notes:
Only one Pig talks at a time and he leads the conversation if reasonable questions comes up. He (alone) has the power to stop a conversation if he feels the conversation stray from the DSM path. Team members can decide to talk about an issue that was raised during the DSM just after the meeting is finished.

What next (DSM planned improvements)?
(1) You late, you get (punished, that is): each team member that late to the DSM must wear the "I was late to DSM, I will serve you coffee today" sign on his shirt.
(2) Red-Back on track: If the conversation is getting out of control (too many jokes, drill-down conversations, more than 1 Pig talk etc) - the red button is clicked and each Team member is being electrified with 120V. Well no, but it could have been a nice feature right? Clicking the red button will make a nice GONG!! so we could move on. The Team agree to listen to the GONG and get back on track, so we could finish in time and keeping the DSM productive.

Gosh, I did not know Raymond Lewallen was reading my blog (I guess I should start writing some meaningful stuff and stop playing around ;)) but I'm more than happy to raise up to the challenge and talk about what I am doing in order to go to the next level.

In one of my post, What it takes to become a great developer, I mentioned the notion of "Be Eager To Learn". I don't consider myself as a good developer due to my natural skills (I don't think that I'm mediocre, but certainly not Larry Page). Starting 8 years ago as a little teenager at 15, I had to work my ass off in order to keep up and show the rest of the people I was working with that I'm just as good as they are. Reaching this goal, I wanted to show myself that I can be the best guy at the company.

Eight years passed and a lot have changed, but I'm still very much eager to get better and more versatile. One thing I'll always keep with me, as it proved it self so far, is the no-fear attitude and the (sometimes) ridiculous optimism. I'm not afraid of doing new things or changing positions when an "offer you can't refuse" knocks on my door. Life is short and you most grow each and every day. I'm still the same team player guy, although I can get over confident (aka arrogant) or raise my voice here and there. I care about my teammates and know when to say "I'm sorry". I work with my heart and hopefully my current and future teammates will forgive me for my faults.

I think that in the last few years I've learned a lot about myself, about the things that really intrigued me, that push me to excel. I love coding, I love talking with people, mentoring, lecturing about technologies or Agile methodologies, but most of all - I enjoy taking ownership of projects I participate in and making them successful. I'm looking to surround myself with people smarter than me, those that have natural gifts in them, and making them better.

Things I should do

I should try to get more organized in planning my time. I read a lot of books about self management but I don't feel like I'm practicing them as much as I should. I should really invest more time in myself, trying to set goals and constantly reviewing them. I'm leading the Agile a la Scrum at Semingo so I hope to use this work & review notion more in my life.

I should learn more about Agile, Scrum and XP. I've read a few great books about Agile\Scrum\Management but I still have a lot of unanswered questions. I know that these methodologies only offer some solutions but I don't believe we should enforce them. I believe in making our own Agile process at Semingo. That said, I do want to read more books from people with different experience, different ideas and best practices I could learn from.

I should definitely write more posts! (particularly about Agile\Scrum)

Things I want to do

TDD: getting better in it and start lecturing about it more.
Multi-threading: This one is a new set of skills I'm developing at my current job. Looking at the near future, this skill is crucial as a developer.
WCF: I need to use it in my current job and I have a lot of catch up to do.
Lecturing: At least 4-5 lectures a year looks like a solid goal at the moment.

Most of all, I want to make Semingo the best place to work at, to bring more amazing guys&gals to work with us and making an application that will change the way millions of people work.

Things I won't do

I think that it's getting clear to me that I do not want to be an external coach. I don't see myself coaching a team for a 2-3 months and then shifting to another team. I enjoy working with people and I take pride and strength in making things complete.

I won't stop talking and writing about software, practices and people as long as I have keyboard and working set of 1-N fingers available. Count on it!

Tagging these folks

Pasha Bitz, Shani Raba, Doron Yaacoby, Eran Nachum, Ken Egozi

One of the downsides of using lock is obviously performance. While locking an object, any other thread trying to acquire the lock on that object will wait in line. This can open up a deep hole to performance hit. Rule of thumb while working with locks is to acquire it as late as possible and release it as soon as possible. To demonstrate the order of magnitude bad usage of locks can affect your performance, I decided to write a little demo. 
So let's assume we have a component that is responsible for executing tasks while getting new ones in the process (on different threads). I tried to make this example as simple as possible. Let's start with our "task" class:

public class Task
{
   privateint _id;
   privatestring _name;

   public Task(int id, string name) {
      _id = id;
      _name = name;
   }

   publicint Id { // getter, setter }
   publicstring Name { // getter, setter }
}

We have a TasksRunner that's responsible for getting new tasks and saving it to internal list and executing the current tasks every X milliseconds (via timer). In order to simulate a real-life process, I've made sure that executing a single task is expensive. Let's start with the non-optimized solution:

public class TasksRunner
{
   private List<Task> _tasks;
   private System.Timers.Timer _handleTasksTimer;

   public TasksRunner()
   {
      _tasks =new List<Task>();

      _handleTasksTimer =new Timer(200); 
      _handleTasksTimer.Elapsed += new System.Timers.ElapsedEventHandler(_handleTasksTimer_Elapsed);
      _handleTasksTimer.Start();
   }

   public void AddTask(Task t)
   {
      lock (_tasks)
      {
         _tasks.Add(t);
         Console.WriteLine("Task added, id: "+ t.Id +", name: "+ t.Name);
      }
   }

   //Execute the (delta) tasks in a thread from the ThreadPool
   privatevoid _handleTasksTimer_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
   {
      ExecuteCurrentTasks();
   }

   publicvoid ExecuteCurrentTasks()
   {
      lock (_tasks)
      {
         foreach (Task t in _tasks)
            ExecuteSingleTask(t);
         
         _tasks.Clear();
      }
   }

   privatevoid ExecuteSingleTask(Task t)
   {
      Console.WriteLine("Handling task, id: "+ t.Id +", name: "+ t.Name);
      Thread.Sleep(1000); //simulate long run
   }
}

AddTask will acquire the lock on _tasks and add the new task to the list while ExecuteCurrentTasks will acquire the lock (on _tasks) and simulate real execution on the task. Notice that during the execution, calling AddTask will wait until the current execution will be finished. Using Roy's ThreadTester, we can run the following in order to notice the behavior so far:

static void Main(string[] args)
{
   TasksRunner runner =new TasksRunner();
   ThreadTester threadTester =new ThreadTester();
   threadTester.RunBehavior = ThreadRunBehavior.RunUntilAllThreadsFinish;

   Stopwatch watch =new Stopwatch();
   watch.Start();

   int numberOfTasksToCreate = 100;
   threadTester.AddThreadAction(delegate
      {
         for (int j = 0; j < numberOfTasksToCreate; j++) 
         {
            runner.AddTask(new Task(j, "job "+ j));
            Thread.Sleep(100);
         }
      });

   threadTester.StartAllThreads(int.MaxValue); //wait, no matter how long

   Console.WriteLine("Total time so far (milliseconds): "+ watch.ElapsedMilliseconds);
   Console.WriteLine("Tasks added so far: "+ runner.TasksAdded);
   Console.WriteLine("Tasks executed so far: "+ runner.TasksExecuted);
   Console.WriteLine("Waiting for tasks to end...");

   while (runner.TasksExecuted < numberOfTasksToCreate)
      Thread.Sleep(1000);

   runner.Shutdown();

   Console.WriteLine("done!");
   Console.WriteLine("Total time so far (milliseconds): "+ watch.ElapsedMilliseconds);
   Console.WriteLine("Tasks added so far: "+ runner.TasksAdded);
   Console.WriteLine("Tasks executed so far: "+ runner.TasksExecuted);
}

Running this test will give us a very poor result for adding & executing 100 tasks takes around ~99 seconds.

No doubt, the lock on _tasks while executing each and every task in the list is too expensive as we're depend on ExecuteSingleTask (which is expensive by itself). This way, each new task we're trying to add must wait until the current execution is finished. An elegant solution to this problem, suggested by my teammate Tomer Gabel, is to use a temporal object to point to the current tasks thus freeing the lock much quicker. So here is an optimized version of ExecuteCurrentTasks:

public void ExecuteCurrentTasks()
{
   List<Task> copyOfTasks =null;
   lock (_tasks)
   {
      copyOfTasks = _tasks;
      _tasks =new List<Task>();
   }

   foreach (Task t in copyOfTasks)
      ExecuteSingleTask(t);
}

This little refactoring give us around ~11 seconds for adding & executing 100 tasks.

Smoking!

In my previous post, "How to mock static class or static member for testing", Eli replied that mocking static data can be easily achieved via TypeMock with no extra work around it. Although Eli raised a valid point while TypeMock is a very strong framework that you should check out, I still believe that designing for testability overcomes the basic need for testing your code. Trying to use some sort of powerful Profiler-based framework in order to mock your "wires" is a great thing, but I would use it only if refactoring the code will take too much time or effort that will beat the value of performing the change.

I would like to make my teammates think toward testability while designing the API.

Breaking your code so it will be easily tested will make things a little bit more complex in terms of dependency injection (object A will need to get IB in its constructor in order to work rather then creating new instance of B inside of it), but the allegedly weakness of breaking your code, making you stray from the pure OOP you've learned in "OOP for dummies" courses at the university, is the exact strength I see in TDD. Designing the code by writing use-cases, playing with the API, breaking classes for Single Responsibility and constant refactoring wins the purpose of testing solely or pure OO code. Don't get me wrong, pure OO is nice on the surface, but don't get fooled by it. You write code to create successful software, not successful practice of old paradigms.

TDD constantly demands to re-design the system and refactor it to fit the needs the system invocate, having the tests will back you up in the path. This way you can follow the guidelines of a sane development process:
1). Design the general architecture (the big picture)
2). Agree on interfaces between components
3). Unit-test user stories (use-cases) and drill down into the design by each test.
4). Build Integration tests to connect between components.

and so on...

Now, if you want to refactor existing code or add new behaviors, it's simple:
1). Refactor the code.
2). See that everything compiles.
3). Run unit & integrations tests.
4). If all is good - check-in your source.

Claiming that TDD-OO is not pure OO is fine by me. I would prefer TDD-OO based code on pure OO any day.

Pasha Bitz, friend, teammate, hacker and gifted(?) photographer took some photos on our last Daily Standup Meeting and a few more at our offices:

Daily Standup Meeting (I'm on the left, Sagie(aka Flash)  is on the right):

Our office door (notice: it's beta 0.1):

And finally, the new setup, although it's far from being ready & not updated (I've got 2 22' inch LCD at the moment):

More pictures we'll be posted soon...

You must admit, I'm a pretty cool (yellow) geek:

You can create your own avatar here.