Design


The other day, while automating our load tests, (you heard that right – we are even automating load tests as part of agile efforts) i came across an interesting scenario.

Load runner, which we where using to automate the clustered web application, spread all its output across tons of small files, containing one/many similar lines. The 2-3 lines each file contained would look something like this

Report.c(199): Error: App Error: refresh report. DataSet took longer than the refresh rate to load.  This may indicate a slow database (e.g. HDS) [loadTime: 37.708480, refreshRate: 26].      [MsgId: MERR-17999]
Report.c(83): Error: App Error: report took longer than refresh rate: 26; iter: 32;  vuser: ?; total time 41.796206    [MsgId: MERR-17999]

But what we wanted to know to know at the end of the test was a summary of errors as in

  • Report.c(199): Error: App Error: refresh report. DataSet took longer than the refresh rate to load.  This may indicate a slow database (e.g. HDS) – 100 errors
  • Report.c(200) Error Null arguments – 50 errors

Approaching the solution

How would you solve this problem ? Think for a moment and not more ….

Well being the C++ junkie that i’m, i immediately saw an oppurtunity to apply my sword and utility belts of hash-tables / efficient string comparisons etc and whipped out Visual Studio 2010 (i havent used 2010 too much and wanted to use it) and then it occured to me

Why not use Java ? I had recently learned it and it might be easier.

The Final Solution

cat *.log | sed  ‘s/total time \([0-9]*\)\.\([0-9]*\)/total time/;s/loadTime: \([0-9]*\)\.\([0-9]*\)/loadTime: /;s/iter: \([0-9]*\)/iter: /;s/refresh rate: \([0-9]*\)/refresh rate:  /;s/refreshRate: \([0-9]*\)/refreshRate:  /’ | sort | uniq –c

I probably should optimize the sed list of commands such that i can pass in the replacement commands as input to sed from a file. But i left it at that, happy that my solution was ready to use / would work all the time / does not require maintenance / saved me a lot of time.

The end

 

Thinking about the episode, i was reminded of the hit Japanese samurai movie Zatoichi and its happy ending at the end.

  • The villain (problem) is dead
  • Samurai is happy with his effort
  • The subjects (files/compiler/system/software/hardware) are happy due to the efficiency created
  • Karma (company + job) is happy due to the time saved.

and to boot it nearly follows the Art of War zen which advices the clever warrior to win without pulling out his sword …

 

Happy coding !!!

 

 

We , (me and my kid sis) have been recently thinking of creating an expense manager application. Why an expense manager? Well, because an expense manager would be useful for us to keep expenses and since we could not really find any good ones (UI , usability wise) on the net, for free.

Once we decided to make one, the question was, what tools to use? Which platform /  language and related tools should we chose? Well after ruminating over these for a few days here is what we came up with.

Platform Choice – iPhone is cool but Apple has entry barriers for development.

Bad apple

I was initially in favor of developing the application for the iPhone but there seems to be so many available. We were also discouraged from developing for iPhone by the fact that it required a hacked up VM or a Mac for iPhone application development.  Rumors, of how Apple tight fistedly controls, who could develop for the platform, along with an entry fees for being listed, took out a lot of enthusiasm from Apple development.  I kept having this deja Vu on how Apple missed the boat on the PC revolution, due to its tight fisted policies of the past regarding Macs and now it seems to be doing the same, all over again.

I do though feel that a mobile phone based application would have more no of users, since the expenses can be entered into it without fail, and therefore very convenient. Its a shame that Apple is playing so hard to develop for. It

Platform Choice – Re-invent the desktop experience

Since iPhone seemed out of the question, the desktop screen area and its powerful graphics cards seemed the next best for developing a cool eye catching UI application and to utilize the bigger screens for useful reports.

So, its a desktop app that we would do.

What language should i use for desktop development?

Next up the question is the choice of programming language / platform on the desktop, which we are going to use for our apps. I definitely cant use C++ because that doesn’t seem to make any sense these days with all those .NET and Java or python type easy to use platforms available. These are simply too productive to ignore and we are not making a performance sensitive application anyhow. So that brings our choice down to a compiled Ui friendly language like

.NET or Java

I believe a slick UI (User Interface) is going to influence users more than anything else, with customer’s bar of expectations set high, by the iPhone and Vista user experiences.  .NET with its WPF User Interface and close Windows OS integration, supported by the excellent Visual Studio development environment would make for a great platform is what i felt.

WPF UI

 

I did not want to use Java remembering its old slower Spring based User experience and since .NET looked much more slicker (due to WPF) and Visual Studio would  be a more productive tool to develop with than eclipse can be, for the windows platform. JavaFX is still too unstable and the tool support is lacking to move into it. (We did run a small evaluation and it does seem another six months or one year would be gone before some support of JFX development was available)

Most of our probable customers (and the future ones who might be ready to pay for a better version) are probably running Windows, so cross platform is not a big consideration. NET also plays into my expertise with windows platform and win32 API experience. So we might not get stuck with any platform issues.

Choice of Database

The next tool that we wanted to freeze on was a database that i would want to use along side my application. It would have to be free since i plan to ship my application for free initially. The ones i have heard of that i think might be useful here are Microsft SQL Express, SQL Lite, FireBird, MySQL, PostGres, Ingres and HSQLDB. These are the ones i have heard the most about. The one thing i have learned from my years of experience is that a more widely used tool would have most of its common problems ironed out and the public forum based support might be better too.

So which of these can be my database?

Multi-platform support is not a big deal for me since i have already decided to use .NET and WPF. However it definitely needs to support .Net well and it should be small. I cant expect users to make 100MB downloads to check out if a free tool is good. 6-10MB is what i think the size ought to be for this app download for it to be viable.

That preempts PostGres and Ingres which are both heavy weight. I did not have to try these out to know they are heavy weight because a two minute search on Google reveals how PostGres is already a competitor for enterprise apps and Ingres used to be one. Additionally Ingres seems to have lost out all backing and is being sold so that rules it out too.

MySQL ? Well that’s not free for commercial use . Plus i have also heard from multiple sources (mainly on Sourceforge), that the sources are a real mess and i know for a fact that till recently it did not have good SQL features like foreign integrity. Not too impressive ,I’m afraid.

HSQLDB is java based and introduces additional platform considerations. Not interested in that set of problems too.

This leaves SQL server, SQL Lite and FireBird

SQL Server is also heavy though not as heavy as PostGres. It comes to about 53 Mb in size for deployment and once deployed SQL server express behaves like a proper DB consuming lots of memory etc on the client machine.  But no, i would not want to deploy a DB server on my personal laptop just to keep my accounts.

SQL Lite vs FireBird

So it boils down to SQL Lite and FireBird. Between the both it boils down to IF we require the extra functionality offered by FireBird. SQL Lite is very small (230K) but lacks many SQL features like row/table locking and integrity constraints or stored procedures and such. BUT it supports seamless movement to a mobile environment if we require it.

FireBird is great but might not  be suitable for mobile environments. So the question i guess is will i require a mobile environment support, given my infatuation towards mobile development?

Do we require Mobile Environment support?

After some thought i decided to not muddle my development by imposing some arbitrary requirement that might not ever be required. I should develop for Mobile if i want to go Mobile and not ruin my desktop experience for the sake of some doubt-able future compatibility or requirement. I’m sure a better user experience can be developed instead if i can devote  the time saved in getting around SQL Lite’s limitations to the development of my application.

Database = FireBird
So FireBird it is and hopefully it plays well with .NET (http://firebirdsql.org/dotnetfirebird/) else we shall have to move to to SQL Lite.

What does FireBird have?

Well it has .NET support, deployment size is 2 MB + has XCOPY deployment. It has row and table locking, integrity constraints, stored procedure support for faster data manipulation) and VStudio .NET Visual Design Support.  It also used to be developed by Borland and that speaks for an impressive past history however wrong i might be.

It has all the pluses is what i felt, not to mention a favorable review in the past from a friend who was doing the same kind of research. That was four or five years back. Things can only be better now.

So that’s it then i guess. The best tools for the job are decided and all it takes is the time to build one. I shall keep this space updated with how good or bad FireBird turns out to be, when we really start using it.

Cheers !!!

I had a chance to modify and extensively update an older post on multiple consumer, single producer problem after a reader request.

Cheers !!

Yesterday I happened to reverse tab into an old podcast from Scott HanselMan’s archives. In case you are wondering what i just said, the background is that i keep a lot of tabs opened in Firefox at all times.

FireFox has a cool feature which will re-open them for you, the next time it comes up after shutdown.

So all this un-read but un-opened tabs stick around as multiple book marks into different tasks, articles and applications i track. This lets me switch my tasks nilly-willy and save time in very minute increments, when my mood or tasks changes to flit from one topic to another.

But the down side is that at times i keep some tabs opened for so long, that i never get back to these readings until a month or so after i initially found them.

Anyway, coming back to Scott’s podcast –  This  podcast was a talk byRobert C Martin, the famous author and Object Mentor founder, on his advices on design practices. It is a nice talk and you should go over and listen to it even if you consider a master on object oriented design.

The other reason that i flipped out on this podcast was how the first two principles he espoused actually turned out to be lessons i had already learned, from multiple sources and from my own experiences .

I haven’t finished listening to the full podcast yet but here are first two of his (S-O-L-I-D) rules and my corresponding blog posts for the same

  1. Single Responsibility
  2. Closed to change but open to Extension

It feels nice to know when the best minds in the Industry gives some lessons and you find that not only did have you already learned it, but you have chosen to express your learnings in the same order too. 🙂

What is the yard stick by which you decide the trade offs while designing features? This post is an attempt to look at two radically different point of views regarding the trade offs in designs vis via an ongoing feature implementation and another example taken from Vista’s user interface changes.

The feature

Our network management tool, Cisco NetManager, can perform fault monitoring and notification for network devices. One such fault it can sense, are the link down states for the individual ports, which is basically when the particular network ports appear to be off-line or non operational.

event-list

Quite understandably Link Down is an important event to be sensed and this is done in three different ways by NetManager.

  1. By listening to ad-hoc Link-Down SNMP  traps sent from the device.
  2. By periodic SNMP based polling of all devices, which senses that a particular port is down
  3. By CDP based discovery process that runs every once in a while to discover the physical connectivity of the devices being monitored.

Issue – Flood of Correct but Un-Required Messages

As a result of these fast multiple port status detection methods, network port status changes are sensed almost in real time by NetManager. However real networks have switches and routers with plenty of un-used ports which appear to be down, simply because they are not in use. Users get inundated with notifications about the down ports of such devices, whenever notification sends down the initial list of failures it has detected initially after install. For big networks, this runs into many thousands of useless but correct alerts and needs to be controlled.

The New Requirement

We are planning to correct the above issue of un-necessary alerts in the upcoming version, which aims to cut down on this un-used ports traffic, by sensing un-used interfaces and not have any alerts reported on them. What follows is the list of facts we have considered in designing this feature and how we plan to accommodate them in the final design.

  1. Other Network Port Related Features –The next release of NetManager will also include a voice utilization feature for voice gateways, which can be used to properly size the available bandwidth using Erlang based calculations. For this reason, some of the ports discarded by the current code would be required to be processed.
  2. Only Specific Ports are Currently Alerted – Not all ports are considered for alerting. This is because many port types really do not have a strict up down status or they might translate to other status types frequently, depending on the nature of operations. For this reason, the current set of valid ports are hard coded into the source code, which is really a bad programming practice. One goal of the current set of modifications is to make this list of types looked up from some external source in an easily configurable manner.
  3. Future Requirements – There is a definite possibility that some customers might not want the same type of ports to be alerted upon by default, for reasons specific to their deployment.The challenge is to consider (not implement) this requirement along with the current feature so as to make the design flexible enough when the time comes for implementing it.
  4. Familiarity with Code – An additional advantage of expanding the design at this stage, rather than later is because this promotes familiarity with the relevant module and affords easier modifications than if the code base where to be freshly looked at during the next code cycle, after a gap of 6-7 months.

Designing for Configurable Network Port Types

The complete design, which makes the default list of network ports look up based would ideally

  1. Make the default list of port types configurable AND
  2. Make the alerting for the individual ports customizable, and based on user preference for the specific port.

Design Side Effects

Lets inspect what happens if the user changes the global alert-able setting on any port type.

  • What happens to devices with ports of the same type, that the user has already configured?
    • Leave the customized ports alone – in this case ports of some devices would show a different behaviour compared to other devices.
    • Change the alert settings irrespective of individual device settings – here the user would lose all the customizations he has made and the expectation of how a devices configured previously behaves would be violated.
  • What happens if the system does not make any changes for devices that are already present and instead affect only the new devices that will be added in the future – in this case some devices would behave totally differently from other devices and lead to confusions overall.

No matter which option we chose, we will be left with in-consistencies in the way system behaves wrt to the alert-able ports. Even though it is the user who changes these settings, in the end they would be left in need of explanations as to why the system behaves the way it does.

Rules Rules Rules

Features that exhibit this sort of behavior, forces users to learn how the system reacts, if they have to be able to use it effectively. When you have multiple pieces of system designed in this manner, with possible inter-relationships to moot, the overall complexity shoots up quite fast.

These sort of designs used to be the norm till recent times. But thoughts about quality in designs, that create simpler and more usable products have been around in the background for some time too, of the type famously pioneered by Apple perhaps, in their design of extremely user friendly and hence popular but at times restrictive products. Here is a sample

XP sound control

xp-sound-controlNotice how the sound control talks about SW Synth and Wave?

What are they and how is it different from the volume? This is the sort of design that’s typically promoted by feature rich craziness.

Designing for Usability – Vista

basic-vista-sound-control Contrast the above with new Vista Sound Control. This, is all that turns up whether you double / single click the sound icon on the Vista task bar. Notice how the other “features” have been cleaned up and the icon made more realistic so as to leave no more doubt regarding what is the intention.

Folks who are more detailed, would notice another click-able option below, termed the mixer, used to bring up the more advanced user interface. Basically Microsoft seems to be relying on the ubiquitous http style user interfacing which everyone already knows and favours, as a  basic program-user interfacing guideline.

Look below for what comes up, if you click on the mixer label.

The Real Feature – Mixer

extended-vista-sound-control

That dialog allows you to control the volumes emitted by the various applications you might have running inside your computer. Now, this is definitely what can be termed as a feature. Another rule, or complexity or menu interface is really not what makes a new feature, but something totally new, that makes life easier for the user, without adding more rules.

Apple’s, famous designs became favored in the same manner, with more simplification rather than complexity / features compared to other players. This unmistakably popular appeal of usability as opposed to pointless features is what i will use to guide our design on the network port configurability issue.

Design for Usability – Network Ports Decision

How do we factor this into the network port customization issue we discussed above.

  • When devices are added, each port will inherit a setting as to whether they are alert-able or not based on  the global defaults. BUT THE DEFAULTS WILL NOT BE CUSTOMIZABLE BY THE USER.
  • The user can override the alert-able setting for individual ports on a device

This solution i believe would be simple enough and give the user a feature that is required, without complicating usability anymore than is required.  It does give a feeling of lost features compared to the full design, as might be the case with the Vista sound control, which lost SW SYNTH, WAVE etc.

But ti feel that the usability improvements and reduction in complexity more than offsets the loss of these minor details more than anything else, for the simple reason that the product is more easily used and less complex and hence is more value for money.


ps : Customer types that require more control can very well tweak these settings themselves by using external scripts that can be provided to achieve this.

ps : Netmanager 1.1 has a notification filtering feature that can decide the type of alerts and the exact group of devices you want to be alerted for.

Social sciences can sometimes be the best predictors for success of a tool / product in many ways than one. The fact that human beings are the intended audience for any new innovation we make, usually ensures that good amount of forward thinking social science could perhaps avoid mistakes or plain disasters that sometimes get introduced in the name of innovation or product design.

StackOverFlow.Com

A case in point is the relatively new technical discussion site called StackOverFlow.Com.

Historically, there have been many avenues on the net to find answers. Newsgroups, mailing lists and chat rooms used to be the most prominent for this purpose.

The new entrant in this mix are social discussions sites, the best of which is slashdot. Digg is a similar catch up, but with subtle difference about the way how the top posts are identified, using votes or diggs that readers can attribute to an article.

Two of the best bloggers on the planet, decided to take up this idea and extend it further, to create a discussion site with a difference, stackoverflow.com those that up vote answers and give incentives for folks to answer more and more.

Expertsexchange.com does something similar but stackoverflow.com might, i think garner many more page views attributed to the blog readership and a cleaner page design. However, these sites or at least stackoverflow.com is not the place i would go to if i needed the best answers on the internet. Reasons?

Trolls

troll-web Vote garnering type of system, initiated by Digg and copied by StackOverFlow.com presents a target for one up-man-ship for geeks who have the time to indulge in this idle passion.

Trolls, though a problem on any and all discussion sites, cannot resist such juicy targets and this presents a real problem for legitimate users and administrators included.The quality of questions posted will suffer as a result and cause the good posters to stay away. StackOverflow uses its “top posters are also administrators” policy to take down trolls fast, but this still is an issue.

Wrong Boss

wrong-boss1 In StackOverflow.com, the correct answer is decided by the person least qualified to decide the correctness of the answer – namely the person who initially asked that question.

Many a time this causes the wrong answer or in-sufficient answer to be marked as the correct one. This means that the site gives all the wrong impressions to anyone looking for answers.

Volunteers have the correct motivation

volunteers-are-good Newsgroups work on knowledgeable volunteers who argue out the faintest bits of a question. However that model has been replaced in stackoverflow by geeks who for personal reasons want to gather the most number of votes by answering questions.  This means that no one looks at questions that have already been answered. Combine that with the problems arising with decision inversion issue above and you have a deadly combination where wrong answers remain marked as correct answers as long as these pages remain on the internet.

The real experts stay away – When you have an organization that rewards the wrong guys, soon the great guys figure out whats happening / gets frustrated and learn to stay away from it. From my brief time exploring stackoverflow i dont think i have ever encountered the great folks that you might tend to come across on newsgroups or even slashdot. This tremendously reduces the value of such a site aimed at technical answers .

For all these reasons i believe newsgroups to be still the best place on the internet to receive answers to the technical and otherwise questions you might have, simply because they are driven by volunteers competing to outdo correctness in others.

Niche Market

However due to all the reasons mentioned above, stackoverflow does seem to be the site to get real quick answers, that are easily verifiable like how to do x, or how to fix y. This i believe might be the niche that stackoverflow is going to settle into, muck like expertexchange, its main comptetitor.

Open source thought

Perhaps the same reasons as to why volunteers do a better job, might hold true as to why open source software trumps the best commercial softwares out there when it comes to quality. Of-course it might take time, but ultimately the quality will shine through.

Cisco netmanager product version 1.1 now features a new notification filtering feature, which will allow users to specify rules to filter the email notifications they chose to recieve. To implement this feature I had to implement a rule engine, which would filter the event notifications according to the rules specified. In this section we shall investigate how this can be done in C++.

Though the new capability is fairly straight forward, the implementation is not so …well simple. When events happen in the system, the only information that is at hand is the type of event that has occured and the device that caused the event. Based on this the system has to filter the event based on a combination of groups or devices, the event severity and the actual event type. A single rule can specify multiple groups and multiple events.

The Challenge

The primary difficulty in applying the rules, in the netmanager system, is that netmanager has the capability for creating DYNAMIC groups. ie You could write an SQL query that returns the devices correspoding to it. A device can also be part of multiple groups at the same time due to these dynamic queries. In order to match a device against a group, one would have to evaluate all these SQL queries at rune time, which is clearly not feasible. Therefore there has to be some system that determines in an efficient and fast manner the groups a device belongs to.

In other words given a relation like A contains {1,4,5} and B contains {1,4,6,7}, we should be able to say that the value 3 is contained in sets B and A. How do you create something like this using C++ ?


Solution – Reverse lookup sets

The cost / challenge that we face here is the computation cost for evaluating n queries each time an event occurs. Caching is a good solutiuon to store precomputed results and offset any computational overheads.

Therefore in this case

Set A Contains -> {1,4,5}
Set B Contains -> {1,4,6,7}

Pre-computing the reverse mappings would produce mappings that store the values against the set identifiers.

1 -> {A,B}
4 -> {A,B}
5 -> {A}
6 -> {B}
7 -> {B}

With this data structure in place, finding the presence of a value in any of the sets is now reduced to a single hash_find, of 0(1) cost.

C++ Implementation

Well, since we have sets, nothing beats the STL vector class in having fast access to a set of values. Since you need a fast map, STL hash_map is the best in this class. As you can see STL, written using templates gives some really useful libraries (usually containers and algorithms) you can use in any situation. But this barely represents the full use of STL.

How would you represent our solution in C++?

First off we need to represent a set of integers. Since we are using a vector, this can be represented as vector<int>. Next off we need a hash that will map the integer values to a vector containing the set idenitifers.

integer -> vector<int> mapping is required.

Therefore this would be represented as a hash

hash<int, std::vector<int>>

Not Generic

However as you might have noticed even though we decomposed the problem into a generic set mapping problem, we are now speaking in concrete terms like int and std::vector<int>. This will not help in generalization at all. If we do not generalize we cannot create a class that can be re-used in multiple situations. If a class is not generic enough, folks who want to reuse (for eg find certain names in multiple sets) will be faced with the uphill task of customization or modification of old code, and no one will take that up coz everyone knows how easy it is to write new code rather than modify old code. (a popular fallacy).

Therefore we need to implement the solution to this problem such that you can create a reverse index of ANY type of values. ie the sets can have numbers or strings or classes or any other type. The class we write should be able to reverse calculate the memberships. Only then would the code we write be more useful and achieve a broader reuse target.

Generic C++ Solution

This was solved by creating a templated class that takes as its argument two types, which represents the type of the set idenitifer (A,B,C) and the type of the values that are contained in these sets. That right. The class takes TYPES as its arguments.

It would then create the right type of hashes and vector inside it, of type int or string or whatever type you chose and then perform the mapping and reverse mapping on these types. Here is the actual code that does this in just 3 pages. (WARNING – this code will loook scary if you do not know hash_map declaration syntax)
Isnt it cool? What you have just done is specified an algorithm or a procedure independent of the types it contains. This allows your algorthm to be used against any data types and more importantly other folks with similar requirement can easily reuse your code. This class can now be used to create precomposed lookups for any kindof of data types.

-> STL Power

What you have seen above is why STL is so useful and powerful a tool. It allows you to encaspulate a concept (in this case reverse lookups on precomputed indexes) compared to a day to day code which encasulates an extrememly specific action. In fact templates allows you to generate code, that does the type specific task your program requires.

Once you start writing code using templates, the surrouding code also soon becomes templatized and soon you will found yourself writing entire templates of generic algorithms that can be resued as a body of code. In fact the whole of STL has been created in this manner. Someone thought of generalizing algorithms and then they required other artifacts (eg pointer = iterator concept) to support this and so on an so forth. It is a really pweorful concept and one helluva useful tool.

The downside

Unfortunately the readabaility is a bit of a pain but once you get used to this sort of code it becomes easy. The pay offs are huge and the work satisfaction is also a major boost.

Next Page »