The need

Whenever faced with a challenge to debug an installation of my current project (Cisco Netmanager), it  always helps to know the type and number of devices being debugged,  the kind-of monitoring actions associated, the program image version etc. Being a database driven architecture, all this information and much more is already available in the CNM central database, provided one can connect to it.

Repeated Tasks

But for a developer to obtain this information manually from a running system, it might take some time and effort, especially if the system is hard to get access to. Even after access is obtained, finding the install folder, navigating to it, running the right command, copying the output and such takes some time.

After i had done this a couple of times, i found it useful enough to write a detailed SQL script to automate this task. But no one was going to remember what commands to run, to get the output. I still had to spend time to go over and run the tool.

Tool

So to ease my pain and to satisfy my lazy ass, i ended up automating some more basic tasks like

1. FInd if netmanager is installed or not
2. Find if database has installed properly
3. FInd if netmanager folders are present
4. Issue the command to get the SQL output

Release 2

Once i knew the program install location , it made sense to copy the logs, because that is what we mostly require to debug the run time behaviour of the code. So i added code to do this.

Since there was more than one file in the output, i had to

1. Find temp folder location
2. Create a unique temp folder
3. Copy the output files to the new temp folder
4. find ways to zip files using VB script.
5. Get user preference for the ouput folder to save the zip file in.

Release 3

The program logs present in the output, got me thinking about the system event logs, which i found could be extracted using WMI. In the process i discovered that i could get the information about pretty much everything in the system using WMI like system configuration, memory, disk and swap spaces, processes etc. Needless to say the finished tool was good enough to give a somewhat complete snapshot of the system our program was installed in.

The only issue i faced was that running the script became progressively slower with more and more tasks i tried to do in the script. Since VB 6.0 was not an option anymore, i tried to find tools to compile the script, but could not find anything too useful. But my  needs where already satisfied, and i did not try to explore and spend more time on this option.

In short, the one time action of trying to automate a simple task, ended up creating a complete tool for debugging purposes, which was JUST right for our needs. All this was done in a space of 5-10 days totally over a period of few months, in small cycles of improvements and feed backs.

Efficient Task Management

If we had tried to scope the effort and create a design and then make a tool for debugging purposes, I’m sure we might have over estimated the effort and cut down the features, or over engineered this task in mistaken enthusiasm. The time we spend in investigations and discussions and going back and forth over the requested features with the different stake holders and all related communication overheads, was shaved straight off the chart.

Creating the tool, bit by bit as i required it and to the extend i required it, ensured that the task was completed in the most efficient manner as possible. That was one more time i experienced the benefits of not having to waste my time creating upfront designs and this is not the first time either. I have seen entire code bases and libraries which have high levels of quality created in the same iterative fashion.

Development Disconnect.

In software development, code is the final design. This aspect is recognized and enhanced in iterative / prototyping oriented development. Often due to the complexity of the technologies involved, the differences in quality of implementation and the unpredictability of the final system behavior, grand plans with upfront designs are most often than not doomed to fail. The industry knows this too but still follows this system.

Estimation Failures

Effort Estimation is hard and the little guidelines that exist in the industry in the form of best practices and useful advises are yet to be codified in a scientific manner at least as far as software is concerned. When faced with upfront design creation and set timeliness, guess-estimation failures become extremely common.

The end result is either a gross under estimation which results in cost and time overruns (more common for integration tasks for big systems) or an over estimation (more common for individual engineering tasks).

The combined effect results in a souped down product which is delayed by a huge margin. Classic failure signs no matter which industry you pick.

Signed contract does not ensure work happens.

Iterative development, has the challenge of closely following the progress of each iteration, measuring the progress and deciding the direction for the next iteration. This does presents a higher overhead for corporate managers, who prefer to follow the familiar design by contract method of management.

Signing a contract after tons of negotiations and then waiting for the results to get delivered is the worldwide approach for business contracts. But when this is single mindfully applied to software development, the result is a rather unpleasant experience. We all know how that has worked out.

Design Contract = Stress

When the management depends on individual designs to decide the time and features available in the final output, it is very common to try and include more features upfront and reduce the time commitment for these same features. Depending on the estimation tendencies of the engineers involved and their bargaining skills, this period becomes a very stressful part of the entire development.

Unequal effort Distribution

My experience has been that, the above vagrancy in design contract negotiation skills , often results in output and effort level variations between different engineers involved. Engineers with poor bargaining skills also end up taking more on their plate that result in the hectic and minuscule development time period that follows the upfront designs.

Even at Intel, we had as our manager a guy who developed web pages in the hey day of the dot com boom. Due to some lucky break involving another Dilbert style manager, he become the release manager for the wireless manager config project at Intel. It was impossible to argue with this person sensibly about the efforts involved since he could not speak intelligently about anything at all that we did. All that he was able to do was to trust certain folks and their estimates apart from relying on what his “gut” told him.

Admittedly this is the worse case, but not all managers can intelligently understand and gauge the efforts and the complexities of all the modules that they are responsible for. In the end its just horse trading when it comes to estimates. Teams where there is some level of trust things are better. However if an iterative / prototyping oriented form of development is followed, the output takes shape in front of everyone’s eyes and its easier to understand the challenges, see the requirements and move ahead.

Coding is easier than talking

I guess its just the way people are, but all this talking simply eats up time like crazy. Often the design time period is too long and folks just dron on and on depending on thier pateince for arguments and repelling bull-shit.

The larger the team is the longer it takes to get the idea across and to refute the arguments and concerns of each individual team member. The more energetic the team members are to establish their view points and create an impression of excellence, the more harder this process becomes.

Design discussions takes 3/5 time, and development 1/5 th.

The end result of all the above is that the time available for the real creation is ridiculously low.

Does that means its all light work then ? Engineers are kept busy by involving them in multiple projects and releases, in an effort to absorb all the free time that  they might have in-between, which results in more context switches and lost efficiency.

Overall

Overall, depending on the dynamics of the team, their argumentativeness, negotiation policies of the managers and engineers and the type of estimation failures that individuals make, decide the final outcome. More often that not, this is either totally a disaster or just about ok to allow the project to survive. But never does this combination create wildly successful or imaginative projects like in the startup world.

Following the iterative development model , It becomes much easier to steer the program direction and corect mistakes pretty early into the program. Unlike bridges and roads, software is easy to create and a management style that recognizes and takes advantage of this aspect creates the right zen and a more open and creative environment for everyone involved. It might be hard and daily work for management but when done right, it beats hands down the traditional fatty form of upfront design based contracts for software program management.

Advertisements