The Job

Off late i have been tasked with creating a notification feature for the Cisco NetManager product we are building. This involves raising events, listening for internal and external events, and relaying them to the registered email addresses according to multiple rules defined in the system.

The Challenge

To implement this feature i have to use an eventing channel on which the events are relayed using a blocking mechanism. All listeners wait until everyone who listens for these messages successfully processes them.

The Right path

I feel that the correct way of going about this, is to queue the events and return as fast as possible from my listeners. This ensures that the system does not block on events even for possibly high event loads. Since we have been considering using this system in a more scaled manner supporting hundreds of more devices, it made sense to think of possible high event loads.

The Pain

Using a thread for the event queue, now brings with it the usual bag of synchronization and disconnected execution issues that, to be fair, are a pain in the ass. The whole situation begs the question – need i do this or not?

The Doubt

There is really nothing or no one holding me to implement events in this manner. A typical corporate practice, is to hit ‘code-complete’, by the committed deadline and then take up these individual bug-bears, IF NOTICED.

But What ?

Typically in 99% of the cases, the testing never thinks of raising the event loads on the system to test something like this, because they never imagine this as the underlying for the system being built. Only in cases where this behavior(not bug of-course) is noticed repeatedly and by important customers that too, would this issue then get investigated and closed. The full course of action would then take a loop involving customer support, technical assistance folks (tac), program managers, and a possibly new developer to fix the issue and test team for re-verification. Not to mention, the full amount of emails that needs to flow back and forth to initiate and communicate all the steps involved in the entire chain of events.

Business Angle

Does this flow make any sense in a business manner of speaking ? Only selling products, which are used by customers actually get money ploughed into it for support. So rightly speaking, any products that go out of a commercial enterprise should do only the bare minimum of the effort required to produce it. This will ensure that only money making products actually get supported by the organization.

Well? 2 cents for your thoughts…

If you ask me, something is just not right about this whole situation. My reasons ?

  1. Only when machines are involved, can small patches of time saved, be translated into equal amounts of work. Unless we are talking about measurable work, like assembly line jobs, human work rarely ever produces such transparent efficiencies of operation. This is especially so in IT line of work. The time saved in not doing this right, is probably going to be spend in un-noticable small increments of browsing and reading and emailing and what not.
  2. Finding the actual customer requirements AFTER field deployment sucks. It creates irritated customers and bad reputation in addition to expensive support trails.
  3. All said and done, it just makes sense to do the thing right in the first place and avoid all those trouble at least w.r.t implementing committed features.
  4. In short you avoid all technical debts

And your Incentive is ? ….

But what my dear reader is your incentive in creating the better code? Technical debts matter to the organization and not to the individuals / consultants who actually create the code, who also usually flit from one project to another or from one company to another. After all, for most managers i have encountered, “code is just implementation”.

Writing better / beautiful code is not going to bring the programmer direct money (unless you own part of the company) nor reputation unless you are into open source (2-5% of entire programmer population?) or unless your peers have the same mentality as you do.

Better still, in a typical corporate environment, the effort involved in resolving a hard to find bug could bring in more work and possible cheers later on in the game. So why indeed should any paid programmer write better code, especially in a corporate environment where the code is simply going to be piled along with millions of such others? Why indeed?

* Fortunately where I work, at Cisco, what needs to be done is clear enough. Quality is the top goal for the entire software group at Cisco.

*this post comes as an outcome of moral conundrums occasionally faced in the past by the author – to leave the perceived holier than thou aka “snobbish ” attitude or to hold on ones principles. Principles i tell you are a harder business than is normally imagined.

* update 6 months later – I ended up siding with my concience in the end and created a piece of system that far outperformed our currently selling enterprise scale tool AND matched the expectation of the next generation of monitoring tool intended for ISP’s. Howeer the question still remains – WHY DO IT? As to the ever important question of will i do it again…. well i have to admit i’m being tempted to move closer and closer towards the dark side….