Spot the bug

Code A

STDMETHODIMP CMIB2Iface::CollectData(LONG DeviceId, BSTR IpAddress) {
	list<CString>::iterator listIter;
	for(listIter = ccmList.begin(); listIter != ccmList.end(); listIter++) {
		CString ccmIP = *listIter;
		long tempCCMID = getDeviceID(ccmIP);

		// Handle it only if device ID is valid
		if (tempCCMID != -1) {
				CString component;
				long eventType = 0;
				if (endPointType == 0) {
					eventType = 47;
					component = "Gateway Link - ";
				} else if (endPointType == 1) {
					eventType = 48;
					component = "Unity Link - ";
				}

				component += endPointName;
				// Construct an event.
				LunaEvent ev;
				CString attributes = "linkDestination = " + endPointIP;
				ev.attributes = attributes.AllocSysString();
				ev.eventTypeId = eventType;
				ev.deviceId = theCCMID;
				ev.component = component.AllocSysString();
				ev.timestamp = (COleDateTime::GetCurrentTime()).operator DATE();

				try {
					if (endPointStatus == 0) {
						if (tempCCMID == theCCMID) {
							// Raise/Update an event whenever status is 0
							if (prevEndPointStatus == 0) {
								// Case 00
								unsigned long result;
								MY_LOG_DBG(functionName << ": Updating Event on Component : " << component);
								if (faultManager->updateEvent(ev, &result) == S_OK) {
									MY_LOG_DBG(functionName << ": Updating Event Secceeded");
								}
							} else {
								// Case 10
								unsigned long result;
								MY_LOG_DBG(functionName << ": Raising Event on Component : " << component);
								if (faultManager->raiseEvent(ev, &result) == S_OK) {
									MY_LOG_DBG(functionName << ": Raising Event Secceeded");
								}
							}
						}
					} else {
						// Case 11 and 01
						// Always clear the event, and for all CCMs in the cluster.
						unsigned long result;
						MY_LOG_DBG(functionName << ": Clearing Event on Component : " << component);
						if (faultManager->clearEvent(ev, &result) == S_OK) {
							MY_LOG_DBG(functionName << ": Clearing Event Succeeded");
						}
					}
				} catch(...) {

				}

				SysFreeString(ev.component);
				SysFreeString(ev.attributes);
		} // End of if (deviceID != 0)
	} // End of for loop
	return 0;
}

Code B

STDMETHODIMP CMIB2Iface::CollectData(LONG DeviceId, BSTR IpAddress)
	{
	this -> m_sLog.SetIpAddress(IpAddress);

	/*   ---------------------------------------------------------------------   */

	m_oInspector.AddCollatedRowInspectorFn_Post(FilterIfTypeInfoCollect,
						(void *) this);

	/*   ---------------------------------------------------------------------   */

	m_oInventory.ClearAllCollectors();

	if (BOMBED(DoTableGenericInit(m_oInventory, sIfaceTblColln, m_sLog)))
		{
		_log(L_ERROR, "Mib2 collect table init failed \n");
		return S_OK;
		}

	/*   ---------------------------------------------------------------------   */

	if (PASSED(m_oInventory.CollectInventory (DeviceId, IpAddress)))
		{
		GetIfFaultsExistForDevice(DeviceId, LINK_DOWN_NUM , m_iFaultExist, m_sLog);
		m_oInventory. InspectData(DeviceId, eForInventory);
		m_oInventory. SaveData   (DeviceId, eForInventory);
		}

	return S_OK;
	}

Code Review Department

If tasked at reviewing the 2 pieces of code listed above, chances are that, you might as a reviewer, want to get rid of the task of reviewing A, than spend any more time on an obviously bad effort at coding. Having badly styled code around, allows bugs to remain, and fixes to be nothing more than half hearted patches which neither the patcher nor the original coder can recognize after the act is done.

One bad turn soon begets another and the rate at which the bad code degenerates into sphagetti mess can only be matched by the speed at which engineers long for a total rewrite of this mess. (Failing which they secretly start yearning for a change of job or worse, stick to the same one in a lost/zombie sort of way or decide what they really want to be is a manager)

Humans are not compilers

Human Compiler

Unlike compilers, humans being are quite easily put off by bad code and no act of redemption can save a bad piece of code from degenerating further and further until it can only be discarded. This is frequently the reason why engineers advocate re-architecture, which is hugely more expensive than adapting existing code. The lost business oppurtunity that this represents can be huge and the very effort of doing so could run into millions of dollars per year for a major software company like CISCO.

Any act of review on such a bad piece of code can only seen as a sham – who after all in their right mind would want to spend so much time reviewing someone else’s code when it would so obviously take such a long time to do it.

So the first step in getting working reveiws is to not let the process to be viewed as sham – and the only way that this can be achieved is by respecting the reviewer’s time and patience.

How do you respect a reviewers time ?

By giving readable code (see above)

How do you make code readable ?

1. By avoiding coding smells
2. bi folwing sm cdng convntns(by following the same coding conventions aka coding style)

Twenty different people and their cousins would have an equal number of vastly differing coding styles. If all of these are to be heaped on same page or on the same reviewer, again its equivalent to implicitly admitting that you have no control and that the whole process is a sham. Therefore getting the same style of code to be used EVERYWHERE is what has to be ENFORCED for successful code reviews to happen.

My experience

In the only place i have personally seen code reviews to actually work, a body of code could be rejected simply because a character exceeded the max allowed no of column width of 80 or because in a single location in code, variable was not aligned. The rationale behind this was the assumption that that the coder did not know about this rule and therefore there would be many other places where this mistake would be made.

The act of rejecting the whole body of code due to single misfitting character sends the right signals and soon it becomes a matter of pride to get code accepted without major comments.

The other advantages in having readable code is that

1. Simple Bugs are easier to spot
2. Design issues are easier to spot
3. Review happens faster (that builds more support for review)
4. Bug fixes are easier
5. Module upgrades or re-architecture would be easier
6. SPotting reuse chances are easier – and hence aids faster development
7. tois which so often happens are much more easier

How can the organizations aid in creating readable code

1. Use tools that can perform static analysis based on accepted coding styles and flags issues, before reviewer even reads one line of code
2. Make code readability indexes, part of statistics of the project so that they can be easily tracked
3. Use tools that can reformat existing code base to the accepted style, no matter what that is, so that its easy to get off the block when the process starts

1 out of 2 is still only 50%

Fixing the code review-ability is only half the fight. Where would a reviewer be, without a programmer who wants the code to be reviewed ? In a team of colleagues, its only more than easy for everyone to scratch everyone else’s back and implicitly give easy reviews. Its also natural for colleagues to contest one others reviews. Therefore, there has to be an external entity and stimulus to engage in code reviews.

Of-course enforcing the code standards using tools part of the build process and enforced through management numbers would be a good start, though that alone might not make it stick.

My experience is that usually stimulus takes the form of bragging factor or nerd pride amongst peers. When an obviously respected person  / geek is to review your code, you would want it to be more shiny and that extra perfect.

Companies like Cisco can easily make this happen by having more tech leads to review the code, and having a system of recommendations where recommended parts of code can even get promoted to technical evangelists like Stroustrup or Gosling once a year for review, which can be such an honor. Tracking  programmers, coding bullshit index would also be a good measure for wanting a review.

The human factor

Coding and reviewing code, is ultimately a very human activity and any mechanical solutions to the problems in this domain are bound to fail. Ultimately a more humane and social solution would be required to transform this bit of ‘work’ into something more than statistic, and something of an advantage.

“Any fool can write code that a computer can understand. Good programmers write code that humans can understand.”

- Martin Fowler

We , (me and my kid sis) have been recently thinking of creating an expense manager application. Why an expense manager? Well, because an expense manager would be useful for us to keep expenses and since we could not really find any good ones (UI , usability wise) on the net, for free.

Once we decided to make one, the question was, what tools to use? Which platform /  language and related tools should we chose? Well after ruminating over these for a few days here is what we came up with.

Platform Choice – iPhone is cool but Apple has entry barriers for development.

I was initially in favor of developing the application for the iPhone but there seems to be so many available. We were also discouraged from developing for iPhone by the fact that it required a hacked up VM or a Mac for iPhone application development.  Rumors, of how Apple tight fistedly controls, who could develop for the platform, along with an entry fees for being listed, took out a lot of enthusiasm from Apple development.  I kept having this deja Vu on how Apple missed the boat on the PC revolution, due to its tight fisted policies of the past regarding Macs and now it seems to be doing the same, all over again.

I do though feel that a mobile phone based application would have more no of users, since the expenses can be entered into it without fail, and therefore very convenient. Its a shame that Apple is playing so hard to develop for. It

Platform Choice – Re-invent the desktop experience

Since iPhone seemed out of the question, the desktop screen area and its powerful graphics cards seemed the next best for developing a cool eye catching UI application and to utilize the bigger screens for useful reports.

So, its a desktop app that we would do.

What language should i use for desktop development?

Next up the question is the choice of programming language / platform on the desktop, which we are going to use for our apps. I definitely cant use C++ because that doesn’t seem to make any sense these days with all those .NET and Java or python type easy to use platforms available. These are simply too productive to ignore and we are not making a performance sensitive application anyhow. So that brings our choice down to a compiled Ui friendly language like

.NET or Java

I believe a slick UI (User Interface) is going to influence users more than anything else, with customer’s bar of expectations set high, by the iPhone and Vista user experiences.  .NET with its WPF User Interface and close Windows OS integration, supported by the excellent Visual Studio development environment would make for a great platform is what i felt.

I did not want to use Java remembering its old slower Spring based User experience and since .NET looked much more slicker (due to WPF) and Visual Studio would  be a more productive tool to develop with than eclipse can be, for the windows platform. JavaFX is still too unstable and the tool support is lacking to move into it. (We did run a small evaluation and it does seem another six months or one year would be gone before some support of JFX development was available)

Most of our probable customers (and the future ones who might be ready to pay for a better version) are probably running Windows, so cross platform is not a big consideration. NET also plays into my expertise with windows platform and win32 API experience. So we might not get stuck with any platform issues.

Choice of Database

The next tool that we wanted to freeze on was a database that i would want to use along side my application. It would have to be free since i plan to ship my application for free initially. The ones i have heard of that i think might be useful here are Microsft SQL Express, SQL Lite, FireBird, MySQL, PostGres, Ingres and HSQLDB. These are the ones i have heard the most about. The one thing i have learned from my years of experience is that a more widely used tool would have most of its common problems ironed out and the public forum based support might be better too.

So which of these can be my database?

Multi-platform support is not a big deal for me since i have already decided to use .NET and WPF. However it definitely needs to support .Net well and it should be small. I cant expect users to make 100MB downloads to check out if a free tool is good. 6-10MB is what i think the size ought to be for this app download for it to be viable.

That preempts PostGres and Ingres which are both heavy weight. I did not have to try these out to know they are heavy weight because a two minute search on Google reveals how PostGres is already a competitor for enterprise apps and Ingres used to be one. Additionally Ingres seems to have lost out all backing and is being sold so that rules it out too.

MySQL ? Well that’s not free for commercial use . Plus i have also heard from multiple sources (mainly on Sourceforge), that the sources are a real mess and i know for a fact that till recently it did not have good SQL features like foreign integrity. Not too impressive ,I’m afraid.

HSQLDB is java based and introduces additional platform considerations. Not interested in that set of problems too.

This leaves SQL server, SQL Lite and FireBird

SQL Server is also heavy though not as heavy as PostGres. It comes to about 53 Mb in size for deployment and once deployed SQL server express behaves like a proper DB consuming lots of memory etc on the client machine.  But no, i would not want to deploy a DB server on my personal laptop just to keep my accounts.

SQL Lite vs FireBird

So it boils down to SQL Lite and FireBird. Between the both it boils down to IF we require the extra functionality offered by FireBird. SQL Lite is very small (230K) but lacks many SQL features like row/table locking and integrity constraints or stored procedures and such. BUT it supports seamless movement to a mobile environment if we require it.

FireBird is great but might not  be suitable for mobile environments. So the question i guess is will i require a mobile environment support, given my infatuation towards mobile development?

Do we require Mobile Environment support?

After some thought i decided to not muddle my development by imposing some arbitrary requirement that might not ever be required. I should develop for Mobile if i want to go Mobile and not ruin my desktop experience for the sake of some doubt-able future compatibility or requirement. I’m sure a better user experience can be developed instead if i can devote  the time saved in getting around SQL Lite’s limitations to the development of my application.

Database = FireBird
So FireBird it is and hopefully it plays well with .NET (http://firebirdsql.org/dotnetfirebird/) else we shall have to move to to SQL Lite.

What does FireBird have?

Well it has .NET support, deployment size is 2 MB + has XCOPY deployment. It has row and table locking, integrity constraints, stored procedure support for faster data manipulation) and VStudio .NET Visual Design Support.  It also used to be developed by Borland and that speaks for an impressive past history however wrong i might be.

It has all the pluses is what i felt, not to mention a favorable review in the past from a friend who was doing the same kind of research. That was four or five years back. Things can only be better now.

So that’s it then i guess. The best tools for the job are decided and all it takes is the time to build one. I shall keep this space updated with how good or bad FireBird turns out to be, when we really start using it.

Cheers !!!

 My first job was to program Windows GUI objects using MFC architecture. So i do “get” objects and OO terminology.

However i was stumped to read the following explanation, of a .NET Data provider product (aka data source connectivity classes), that can talk to databases and provide integration with the new ORM (Object Relational Mapping) layer, that microsoft has come up with for .NET applications

“It equips Microsoft .NET based applications, development environments, and programming languages with conceptual entity-based access to native and heterogeneous data sources. “

Conceptual Entity

How soon can you say WTF?  I wonder why folks feel compelled to string so many heavy words together, when describing anything related to OO technology, in anything more than a  semi formal  setting. 

 I dont know if an architect / developer or a marketing personnel wrote that statement. In fact i did not find their site by searching on any of the weasel like words found in the description. In fact i searched for more practical stuff that i wanted my software to do, which thankfully was available on the page too.

This weaselization / complexfication of terminology seems to be a given fart to contend with, in the Java world and it makes me sick to see the cancer spreading to maybe the .NET / MSDN world too. Please, would anyone care to provider a linguistic inter mapping layer for conceptual to real word entities in the object oriented technology space?

I recently came across a requirement to dynamically generate a password on a Linux system.

Being a password, it implied that the text had to be truly random, which is a tough order of requirement for any home-brew algorithm, once you really get an idea of  on why its hard to generate randomness

Radom is hard

The solution = /dev/random .

In theory /dev/random is a file you can read infinitely. However in practise you would find that reading this file would block every now and then until it has contents in it, generated randomly from device driver noise bits. So you might have to move your mouse or spin the disk etc to get some random bits into the /dev/random file.

A More practical solution = /dev/urandom

urandom can get random iput from from /dev/random, else it generates fairly randomized bytes for you, using a Random Number Generator Cryptographic algorithm, provided by a kernel module of the same name. AND, its never guaranteed to block. Talk about best of both worlds !!!

My random Text Command

Bytes read from random or urandom are not likely to be all ASCII and therefore requires to be filtered for ASCII, for which this was the chosen solution -

tr -dc A-Za-z0-9 </dev/urandom |  head -c 14

The problem in the solution

The only doubt that this simple solution envisaged in my rustic Unix mind was if this command line would exit after head had received the 14 characters it required.

If you do not know the answer, it is contained in my previous post on Unix shell pipes.

Happy hacking !!!

Say you do this

ls / -R | head -2

The simple command line above, lists (ls) the entire directory structure starting from root in a recursive fashion. Its output is piped to head which will exit after it gets 2 lines of output.

The problem

Its clear that head will complete much before ls will. This poses some interesting questions like

1. Will ls run in the background after head exits?
2. Will the entire command keep running until ls is done or will you get the prompt as soon as head exits but ls will keep running to completion in the background?
3. Will head run only after ls has finishes giving its output, thus requiring ls to run to completion?

I recently ran into these same set of questions about assumptions being made in an install script that was being written for a new reporting tool we where building. So what is the deal here ? Think for a minute before you read the answers.

Pipe Basics

I knew that the way pipe worked was to redirect the output of one process (stdout) as the input (stdin) of the process that came next in line. But this did not quite give all the answers. Therefore googling the pipe process gave some more details which i shall describe below

The details

Poetry in design

1. The shell sets up the pipe. The running processes use stdin / stdout as they would normally have without knowing its a pipe rather than the terminal input / output.
2. A buffer exists  between the pipe descriptors, which is handled by the kernel. Writing process will block once this buffer is full and will be able to continue once the reading processes read off this buffer.
3. The process of exec and forking means that it is impossible to say which processes run first and is governed only by CPU scheduling.
4. All the processes are in the same hierarchical level and have the shell as their parent.
5. In Linux versions before 2.6.11, the capacity of a pipe was the same as the system page size (e.g., 4096 bytes on i386). Since Linux 2.6.11, the pipe capacity is 65536 bytes.

But what happens when one of the piped process exits?

Unfortunately, most of the documentation only explained how the pipe was setup by the shell and kernel.  But what we required was an explanation of what happened if one the process exits.

To test this i ran a few tests. Here are my observations.

ls / -R | head -2 | head -20000 (exits immediately)

ls / -R | head -20000 | head -2 (exits immediately)

cat small-file.txt | ls /-R (runs for a long time)

Obviously, it is the processes that read input, which makes a difference in how the piped command line behaves. So what causes this behaviour?

Enter Unix Signals

Signals are the IPC mechanisms akin to software interrupts used by Unix like operating systems to notify running processes of interesting events. This is possibly the only way an outside event can modify the behaviour of an otherwise normal process.

A process could recieve a signal for IO issues / child processes termination or plain Ctrl-C related interrupt signals.  Most of the default handlers (which you can change if requires) cause the process to exit.

My attention having turned to signals, i ended up walking through the wikipedia for the master list of UNIX signals, hoping to find anything related to pipe. This caused me to find an interesting entry …

SIGPIPE

The wikipedia entry for SIGPIPE says “When a pipe is broken, the process writing to it is sent the SIGPIPE signal. The default reaction to this signal for a process is to terminate”.

If attempt is made to read from a pipe whose write end has been closed, the reader receives an EOF, which causes the read program to close in an orderly fashion.

hmmmm – applying that knowledge to our pipe commands would show that this would cause the chain of processes to exit, if a listening process exits and thus breaks the pipe chain.

It would have been an extreme waste of resources had ls continued to run even after head exitted in the command line ls / -R | head -2 and shows how well the UNIX system has been thought out.

Even more importantly it is highly intuitive that ls / -R | head -2 to exit immediately after 2 lines have been output.  This is one of the cool things about Unix, that it behaves in the manner you expect it to, as if the commands understand the social niceties of human pysche or at least hacker psyche.

It is also highly interesting to note that the reader receives an EOF, because output finishing first is normal but the write receives a signal if the reader has already exit, since that is an abnormal situation. Very beautiful indeed.

It is no wonder that such clever hacks and excellence in design makes Unix the hacker platform of choice.

Happy hacking !!!!

ps : All this would have been clear had i chose to read the documentation of the pipe system call before the investigation was begun. It goes to show how much we ignore existing documentation at our own peril.

If you ever needed a reason to enforce clean code – here it is, right out of MIT and researched to the hilt. The video is extremely interesting from a behavioral standpoint and extremely educative. As highly recommended as the famous Steve Jobs video.

I can vouch for what is being said because i have seen this happening many times in development. The implied up shoot  ?

Clean code bases beget new code which is cleaner or clean enough. Dirty code bases attract more code which is at least just as dirty or usually even more so.

I had a chance to modify and extensively update an older post on multiple consumer, single producer problem after a reader request.

Cheers !!

Having been a full-time C++ programmer for all of the past ten years, I recently had the unique opportunity of seriously studying Java almost 10-15 years after it started becoming popular. In the meantime I’m also playing with Python on a personal note. This affords me a unique perspective, of having seen the past and at the same time looking at two new languages with a fresh eye.

C++ vs Java in 2009 = Assembly vs C++ during the 1980’s

The impressions or the vibes that i get in this new learning process is eerily similar to the same doubts and choices i felt / read about during the initial stages of my education, while playing around with assembly, C++ and Visual Basic.

Needs assembly or C

Assembly was THE choice if speed and memory choices was paramount and nothing else mattered but it was damn hard to use.  All the old timers claimed you had to know assembly to understand whats happening on the system and that C++ was only for productivity improvements. They lamented about how new minds never learned / worked seriously with assembly and hence where at a disadvantage when it came to figuring out what really happened inside the box.

However at the time (1980) C++ was fast catching up, even embedded systems where moving to C++(maybe without using the templates feature).  Java currently is in the same state that C++ used to be in 1980 wrt assembly and there are many who clamour for C or C++ to be taught in universities rather than Java for the sake of better understanding.

How can  Java be as fast as C++?

It is not. However the hardware that is around now is vastly superior to 1980 and 1990 hardware, which made Java seem slow THEN, in the same way C++ seemed slow in all those small devices compared to raw assembly or binary code. Our current crop of laptops are more powerful than the servers used during the time when Java was beginning to get popular. This improvement in hardware makes the difference in speed highly immaterial for all practical purposes, because human beings cannot make out the difference all that much between 500  and 50 microseconds.

Fast forward to 2010

Is as fast as 1990 computer and runs Java

C++ is what you require if you need pure speed like in the critical loops or core libraries. You could substitute the word Assembly instead of C++, in the previous line, had this post been written 15 years back.

But Java is more useful from a business point of view for application development due to increased productivity and more features.  It is therefore no coincidence then that embedded devices like mobile phones (and even dog collars, according to Java books) are now running Java. In fact the lost Nokia mobile phone that i bought was guided by the factor that it had a data port using which i could upload custom Java program onto it.

Java vs Python = C++ vs Visual Basic

When C++ used to be THE top language around, and assembly was gradually being kicked to the ground, there was something else in periphery, which was Visual Basic and all other RAD languages, which basically gave you forms based development and automatic memory management.

Internal IT or back office apps need not be cool

However they were nowhere as fast as C++ and hence where relegated to internal IT applications, which never needed to be as slick as professional, cool looking  applications and could be developed fast enough.

Fast development is very critical when it comes to small shops with less resources or IT departments on a shoe string budget. Many business applications certainly fit the bill and where developed on Visual Basic. It is exactly the same situation now with Python when comparing it to Java.

Python in 2009 = Visual Basic in 1990

Developing using Python does definitely seem like a breeze with not much hassles around object orientedness or strict rules around exception declaration or big infrastructures. Is Python as comprehensive as Java, with as many no of apis as Java has ? Probably not, although it is fast catching up in popularity, like Java before it, caught with C++ and  that is saying a lot. There is no denying the fact that Python and Ruby would soon have many programmers who are trained in it by choice.

Will run Python as fast as Java

I have not compared the performance of Java vs Python. But i assumed Python could be slower because of its more script like appearance compared to Java. Please do correct me if iam wrong on the above aspect.

If the economy keeps growing like it did in the last ten years, we would soon have quad core or even eight core desktops a common place and would not have to worry about Python’s performance compared to Java. I just hope folks don’t introduce messy co-orporate stuff and make it verbose and bulky as Java before it.

3  cheers for easy coding !!!

Read this instead for a longer and historic explanation.

Be mystified no longer. Web 2.0 is not a new version of the internet or a new technology or a new standard of the web.

It stands for changes in the way new internet applications are being build. These are

Interactivity.

Clock-Ad - animated gif

The current crop of web applications are now way more cooler and more interactive than it was possible using HTML programming. What powers this ? Faster computers, Faster Internet, Faster & Smarter Javascript Engines and Cool Libraries to do these tasks.

ps : Why does Javascript matter? B Cause that’s what runs inside the pages you download from the internet and makes stuff happen inside them. It powers the internet’s display side programming.

Unfortunately I cant add any JavaScript samples to demonstrate interactivity and hence that animated picture on top to convey the idea.

Sharing

You can now access data from e-bay and match it with Craiglist advertisement and build a new application out of this combination, which shows its results on Google Maps.  You could perform a similar mix and match (aka MASHUP) using data from MANY other web apps that choses to exposes its data or tech on the internet.

Google maps based app

Social Networking

Think about all the sites like Orkut,Facebook, Linked-In and of-course, of  all the blogs on the net.  This  represents an un-unprecedented level of personal and social information on the net. Most of these sites expose their API’s too. So that you could perhaps pull in your friends from Orkut into your Outlook, IP Phone or even vice-versa.

So what?

So its a possibility for a next wave of cool uses, like the original dot com which produced the ebay and amazon and all those useful sites which nearly everyone use. Like all waves this too will be hyped to be way more than what it really is – a new form of collaborative and more user friendly internet applications.

Heck, even my new project (that’s right – i got moved from my previous project to a new one) sports a web 2.0 tag and my manager is uttering the magic word every now and then to try and spot an oppurtunity to start a new initiative around it.

but too lazy to read all the white papers?

Read a small story of this in action, from none other than Joel Spolsky trying it out on their product.

This piece of code we are about to discuss was used in production software and i was tasked with reviewing the patch. Try to see if you can spot *any* issues with code. Later we will take a look at the exact observations, which are hugely interesting and instructive regarding system behavior.

The Query

This query retains top n records of a particular type and deletes the other rows in the table of the same type from a Sybase database.

for (int i = 0;i < ListOfTypes; i++)
{
ExecuteQuery “delete from table where PrimaryKey NOT IN (select top n PrimaryKey from table where Id = i) AND Id = i”
}

Observation – No Data to be Deleted in the database

1. CPU consumption during query execution is negligible when there are matching rows to be deleted. The fact that this code is issuing multiple dynamic queries which have to be parsed does not seem to be too much of an issue for Sybase.
2. 2000 queries took around 3 seconds overall at around 12% of CPU cost on a Xeon box.

Observation – delete finds rows for deletion

1. With data to be deleted, the query took 5 minutes when there was valid data in 30 loops, that is over 5 hours when deletes are extrapolated to be repeated 2000 times.
2. The CPU consumption figure remained the same.

Why are Database Deletes So Slow ?

Relational databases tracks everything using logs or otherwise called Transaction Logs because they record all the transactions aka changes that happen to the database. The ACID nature of transactions are guaranteed using these logs. So when you make a change to the database, a LOG is written to disk.

The actual change is only made in memory and not written to the disks immediately, for efficiency sake. But Logs are always written promptly and without fail, so that if the machine crashes or rollback is required, this log can be used to reconstruct the state of the database with minimal loss of data. This means that every time you make an individual delete you are going to make the disk spin and disks spinning are horridly slower that anything else you can think about.

How slow can the disks be ?

Numbers from a talk by Jeff Dean (works on both MapReduce and BigTable) of Google, as reported here -

Comparative slowness aka latency in terms of CPU cycles

Floating point calculations = 0.25 to 3 cycles
L1  cache access                       = 3 cycles
L2 cache access                       = 14 cycles
Dram                                            = 200 cycles
Flash                                            = 3 Million cycles
Disk                                               = 15 Million cycles

Comparative Slowness aka Latency in absolute time

L1 cache reference                                                        0.5 ns
Branch mispredict                                                             5 ns
L2 cache reference                                                             7 ns
Mutex lock/unlock                                                        100 ns
Main memory reference                                              100 ns
Compress 1K bytes with XXX                              10,000 ns
Send 2K bytes over 1 Gbps network                 20,000 ns
Read 1 MB sequentially from memory          250,000 ns
Round trip within same datacenter                500,000 ns
Disk seek                                                              10,000,000 ns
Read 1 MB sequentially from network    10,000,000 ns
Read 1 MB sequentially from disk            30,000,000 ns
Send packet CA->Netherlands->CA      150,000,000 ns

Can you speed up the disks?

SCSI disk array on a server

You bet !!. We tried to speed up this whole experiment by moving the tests from an IDE based system (with 7500 rpm drive) to a system with a single (the picture has six – no we used just one of them) 15k RPM SCSI disk + SCSI controller (no on board memory).

The results from moving to SCSI system -

1. The delete time improved by a factor of 4-5 times and the worse case average time period dropped to around 133 minutes from 5 hours.
2. However with the SCSI system in use, the CPU consumption during query execution also jumped by the same amount, by 4-5 times because there are fewer blocks due to disks and therefore more query execution (4-5 times worth) is done by the DB engine.

As you can see, it would be really easy for us to calculate how many disks we would need to speed up the deletes to a manageable time. I might have to go with a couple of SCSI disks in a RAID configuration and try to achieve the speeds that i want.  At the same time the database engine’s CPU consumption would also have to be separately addressed such that it does not have to do much work in finding the rows to be deleted.

Is there any way to improve this ?

1. Use Transactions -

   SQL server engine makes transactional records only as part of a single transaction. Not sure how this works vis via Java and sybase engine using which this query was tested. So if you ensure all your changes are made as part of a single transaction the disks would be less affected than if it was done otherwise.

2. Do not use a database for real time stuff
   An even more optimal solution will be to not rely on the database at all and instead do all this in a specialized in memory component in a thread safe hash or something equivalent. Use database only as a slow data backup for being used during failure recovery. This would give the incomparable throughputs compared to any RDBMS / disk based solution.
   

My suggestion to improve CPU (untested)

1. Move the entire loop into a stored procedure
2. Run a select to get the top 20 and extract the ID of the bottom one
3. Run a delete to delete all ID > ID extracted.What are the advantages – You remove  possible query compilation for the execution of 2000 queries and at the same time, remove usage of possibly highly inefficient NOT IN operator. This needs to be verified though.
4. Remove all foreign key references to the current table – but of-course that would be going against the grain of RDBMS concept itself. However for desperate situations, this might work ….

What was implemented in the end?

Well, the production software had this delete sequence running every 30 mins. The no of events this system claims support to is only upto 25 events per hour.  So at this rate, there would be no pile up and no big deletes happening. So we dont have a problem !!!!

If you have been following the sub-prime crisis and the following recession, you would have no doubt heard horror stories about how uncontrolled debt has managed to wipe out many an individual / corporate house.

Uncontrolled spending, reckless taking out of loans, made over a century has blown up. The key element however is not that debts where taken, but just how risky those debts where, and how CEOs and corporate boards never really discovered the reality of those debts until all that blew up from down under.

Good independent auditing and more controls on aggressive practices would have perhaps prevented this from happening or the very least reduced the impact by a huge factor. Last minute cleanup after the shit has hit the fan is never the time to do this, as it would hurt more to do it near the end than when there is time and expertise around to perform the required audit and cleanup.

Technical Debt (highly recommended that you read this)

I definitely KNOW that this is something IT projects can learn from. There are many projects, where the leadership and implementers never care to track or document the technical debts they have around. While it’s O.K to cut corners to come out with a feature in a short time, it’s imperative to know exactly what corners have been cut so that you can rebuild them or mend them before something else gets build on top of those shaky corners.

Risk Management + Audit +  Tracking = Good Management

And like in Finance a good amount of auditing and tests for worse case scenarios would bring home the point to everyone involved, as to how shaky everything really is. The risk factors once in the open and put on tracking would ensure that the technical debts do not get run over by callous implementations or over zealous marketing folks or program managers anxious to get the product out. This i believe should be the most important task for any IT manager, of tracking the technical debts and to exactly know which valves will burst when.

Happy Fixing !!

How do you know the type and version of the major hardware components you have in your PC / laptop? Thats easy, run PC Wizard and hope their database is up to date enough for your purposes. (Note that this tool does other cool stuff too like running Benchmarks, getting all sorts of other system related information)

I had the same dire necessity today, having junked the default Vista installation in a laptop, without checking the hardware, drivers and other requirements.  I had been told all the required CD’s are available and when i checked, (after re-install i have to admit), the CD that was referred to turned out to be a rescue CD.  The laptop manual did not care to list even the basic facts about which LAN / sound card it carried. (An acer 4620 if you are wondering)

Isnt Rescue CD enough ?

In the olden days, buying hardware gave you bits and pieces of drivers and other software in multiple CD’s and this was what i had in mind when i asked if the CD’s where available. But nowadays, machines come with a pre-installed OS partition, containing a valid image with everything pre-installed. So, all that you need in case things go wrong, is a single rescue CD to copy  this image onto your hard disk and get a brand new validated and optimized image running.

Whats wrong with Rescue CD’s ?

Rescue CD’s are an excellent solution, in most casual user cases. I remember originally encountering this concept in a Solaris server. I came away impressed with the solution. However in case of personal computers (or latpops) i feel someone ought to have thought of the possibility that a different OS or power user initiated custom installation might be required, in which case this partition idea becomes nothing but a waste of space. After all we are talking about hardware bought for personal use here and its a good idea not to restrict the owners conceptual freedom in trying to use it for whatever purposes they see fit.

The olden days

The olden days of individual driver CD’s, ensured that I had a valid driver for my hardware for all the OS’es i required. This was because the individual driver CD would come with drivers required for all the different version of OSes it supported (windows 3.1/95/98/Me/2k etc). This also meant that the different components of my machine was tested for the different OSes i required and it was easy to know which ones had the support by looking at the driver CD.  In fact i remember favouring devices that shipped with Linux drivers since i wanted to use Linux on my PC. However with the new fangled image idea, i’m not sure if my new box would run another OS or not.

I wish the hardware folks are listening and copied a folder of all required drivers and hardware documentation, into the image itself, which i can then copy out for my new OS, before trashing the old stuff. That way, i would have the best of both worlds, unless of-course i deleted the orignal rescue image partition too, which goes to show that a hard copy CD image of the rescue partition, with all the different OS driver folders in it of-course, would still be of some use.

Happy installations !!

I recently came across a cool replacement for Acrobat Reader, Foxit. Why do i want to use Foxit?

1. Less memory consumption
2. Feels more responsive
3. Tabbed reading

It goes to proves that building a better mouse trap still works, even against such a de-facto standard like using the Adobe reader for reading PDF files.

Collaboration might be a weasel word, but i suddenly found how powerful a tool it  can be, by using Google Docs.

Of course Google Docs has been around for some time, but i found out about them anew when i searched an alternative for the paid services of 37Signal’s BaseCamp software. I was not looking for BaseCamp either, but i stumbled into them through the blog posts they had made about building software and they do have many good blog posts for sure.

The blog was good, which made me interested in checking out what they build, which taught me all about how there are mature solutions that allow multiple distributed people to work on the same thing. It kind of proves how blogs can get you new customers.

If you are into new products, concepts or even existing tools to do the same, it proves how essential blogs and podcasts can be in getting your name across to the early adaptors !!!

Happy Blogging !!

Just listened to a podcast from Scoble, interviewing the so called king of tech shows Leo Laparte.  The talk does impart the impression on how content is most important than any glam and glitter and their wow’s about the new media, and some general other information about content sharing.

While they went on about the history of the show, they mentioned how they both have ran into one another a long time back and how Leo has touched based with many folks in the industry and even launched their careers.

This seems to be a theme that i have heard repeating every now and then with Scoble and many others. All these guys who have been in the Valley or manage to be in tech scenery in the States, seems to have run into the others “who matter”, at some time or other.

It really does seem to matter where you are from, for you to get picked up in the flow, noticed and perhaps to also feed you the enthusiasm to do more, and see more groovy stuff done and launch you into new circles that matter perhaps (VC’s anyone?)

Since all that is not here, i might as well keep myself fixated on the idea that the useful tool i will create some day will have to be at least 50% better than it might been necessary, had i been from the Valley.

I hope you realize that too.

Cheers !!

Yesterday I happened to reverse tab into an old podcast from Scott HanselMan’s archives. In case you are wondering what i just said, the background is that i keep a lot of tabs opened in Firefox at all times.

FireFox has a cool feature which will re-open them for you, the next time it comes up after shutdown.

So all this un-read but un-opened tabs stick around as multiple book marks into different tasks, articles and applications i track. This lets me switch my tasks nilly-willy and save time in very minute increments, when my mood or tasks changes to flit from one topic to another.

But the down side is that at times i keep some tabs opened for so long, that i never get back to these readings until a month or so after i initially found them.

Anyway, coming back to Scott’s podcast -  This  podcast was a talk byRobert C Martin, the famous author and Object Mentor founder, on his advices on design practices. It is a nice talk and you should go over and listen to it even if you consider a master on object oriented design.

The other reason that i flipped out on this podcast was how the first two principles he espoused actually turned out to be lessons i had already learned, from multiple sources and from my own experiences .

I haven’t finished listening to the full podcast yet but here are first two of his (S-O-L-I-D) rules and my corresponding blog posts for the same

1. Single Responsibility
2. Closed to change but open to Extension

It feels nice to know when the best minds in the Industry gives some lessons and you find that not only did have you already learned it, but you have chosen to express your learnings in the same order too.

Sometimes, folks are overwhelmed by the number of articles in the archives of those good bloggers (updated) i recommend to folks. So in the interest of their sanity and my reputation, I decided to post a quick and dirty list of good articles from Joel i could think of, after filtering through his archives.

If people find this useful i might follow up with a post on the Paul Graham articles too.  Note that i haven’t chosen to list the articles in the order of their quality but simply in the chronological order, copied from Joel’s site as is and simply filtered the ones i thought where not so important.  Your mileage might vary so please feel free to click around or read selectively or simply head over to the original site.

Architecture astronauts take over May 01 2008
Smart and Gets Things Done Jun 05 2007
Choices = Headaches Nov 21 2006
The Development Abstraction Layer Apr 11 2006
Set Your Priorities Oct 12 2005
Hitting the High Notes Jul 25 2005
Camels and Rubber Duckies Dec 15 2004
How Microsoft Lost the API War Jun 13 2004

Craftsmanship Dec 01 2003
Rick Chapman is In Search of Stupidity Aug 01 2003
Fixing Venture Capital Jun 03 2003
Lord Palmerston on Programming Dec 11 2002
The Law of Leaky Abstractions Nov 11 2002
Strategy Letter V Jun 12 2002
Product Vision May 09 2002
The Iceberg Secret, Revealed Feb 13 2002
Fire And Motion Jan 06 2002
In Defense of Not-Invented-Here Syndrome Oct 14 2001
Hard-assed Bug Fixin’ Jul 31 2001
Good Software Takes Ten Years. Get Used To it. Jul 21 2001
Don’t Let Architecture Astronauts Scare You Apr 21 2001
Strategy Letter IV: Bloatware and the 80/20 Myth Mar 23 2001
Human Task Switches Considered Harmful Feb 12 2001
Daily Builds Are Your Friend Jan 27 2001
Big Macs vs. The Naked Chef Jan 18 2001
Up the tata without a tutu Dec 02 2000
Feedback on Programmer Compensation Aug 28 2000
The Joel Test: 12 Steps to Better Code Aug 09 2000
Top Five (Wrong) Reasons You Don’t Have Testers Apr 30 2000
Things You Should Never Do, Part I Apr 06 2000
Incentive Pay Considered Harmful Apr 03 2000
Painless Software Schedules Mar 29 2000
The Guerrilla Guide to Interviewing Mar 23 2000
Command and Conquer and the Herd of Coconuts Mar 23 2000
Converting Capital Into Software That Works Mar 21 2000
Two Stories Mar 19 2000
USER INTERFACE DESIGN FOR PROGRAMMERS

A book about designing user interfaces, intended for software developers for whom the whole process is a bit of a mystery. Nine chapters are available on this site. A longer version is available in print from Apress.

Chapter: 1 2 3 4 5 6 7 8 9

I ‘m beginning this series, as extracts of notes from some of the best comp science classes i have ever witnessed, recorded by MIT.

However i will be emphasizing only the points which struck me, after 10 years of experience as worth remembering again, that I feel the industry might do well to never forget . I will try to leave out the details, like the data structures, recursive techniques and such, which most people are likely know. I would highly recommend you to access the full text or videos, if you feel  that you missed out any of your comp classes or have never listened to comparable masters explaining this art.

My aim is to capture the pearls of wisdom that i might pickup listening to this class all over again. This is part 1 of such notes and this is the reason why i felt i ought to do it.

What is Computer Science

Computer Science is not about computers in the same sense, that geometry is not really about surveying instruments. But when geometry was created in ancient Egypt, it really was about surveys and instruments, in the same way computer science started off with computers in the twentieth century.

What computer science is really about is

1. Formalizing intuitions about process ie “How to” knowledge

eg How to compute sqrt vs “what is sqrt”
eg How to draw triangle vs “what is a triangle”

So whats so hard about this? Real issues of computer science is not about telling people how to do stuff. That would be too easy. Real problems come when we try to build very very large systems that performs these processes. Real problems arise with these systems that have programs that are too big to fit in any one person’s head. Therefore,

2. Computer science, is also, all about techniques to control complexity of  large systems.

Abstract Characteristic (vs other engineering disciplines)

The components we use in computer science are asbtract, like an array or a hash. Unlike other disciplines that also deal with complexity the difference here is that the complexity comp science face is not physical in nature. They do not face the physical realities of real world like in the case of other engineering fields. Leakage, resisitace, heating, drag and such physical issues does not dog computer science. Rather what ails it, are the constraints of human mind in holding and keeping track of big programs.

In this sense computer science is an abstract form of engineering and therefore its most important tools are tools for abstraction, that aid human understanding and kill complexity. This is what comp science is mostly all about.

What are the techniques for controlling complexity

1. Black-Box Methods
2. Conventional Interfaces
3. Large Scale System Integration Metaphors
   1. OO
   2. Streams
4. New Languages

Black Box Techniques

Refers to making something and building a box about it such that it can be used without knowledge of its internals much like an IC chip. You want to suppress details so that you can build bigger boxes that use this.

You can also use black box techniques to express generality. eg A routine add might be able to add numbers, vectors, signals etc.

Conventional Interfaces

Are agreed upon ways of plugging things together.

These are operations that all kinds of “abstracts” might support eg +

Languages

Help you control complexity by dempahsizing certain aspects of the complexty and emphasizing others eg SQL emphasizes only the data modification part and hides everything else

Learning a language is like learning chess it just takes 2 minutes but learning the implications takes a lifetime.

Framework for evaluating and understanding Languages

If we view languages in the context of tools in reducing complexity, it immediately becomes clear that a new language is not all about how many lines of code to do the next cool thing. Rather they should be evaluated in terms of 

• What are the tools for abstractions it provides
• What are the primitives that it provides
• What are the ways of combining these primitives

My thoughts : The idea of computer science as only about tools to control complexity is not too well imbibed into the many minds (including my iniitial years)  i have come across in my decade of experience. Rather, computers are all about “coding” or Java or C++ and networking. If code is viewed and reviewed from this angle alone, much of the issues that plague the industry would have been non existent.

In these lectures everything that follows is a function of this characterisitc, of ways to reduce complexity, evident in the way languages are evalauted. I’m excited to take the new (rather old) take on all other topics in the following series and will be posting the follow up posts on the same.

This is the blog of a white computer guy who has married a black lady from South Africa.The culture differences he observes are funny to read and gives a perspective of life in South Africa.

The pod-casts are quite interesting too, giving more information, political and economical included, of the country. Kind-of makes me wish i had some ways to record more of what i heard and saw, when i travelled about in India myself.