The other day, while automating our load tests, (you heard that right – we are even automating load tests as part of agile efforts) i came across an interesting scenario.

Load runner, which we where using to automate the clustered web application, spread all its output across tons of small files, containing one/many similar lines. The 2-3 lines each file contained would look something like this

Report.c(199): Error: App Error: refresh report. DataSet took longer than the refresh rate to load.  This may indicate a slow database (e.g. HDS) [loadTime: 37.708480, refreshRate: 26].      [MsgId: MERR-17999]
Report.c(83): Error: App Error: report took longer than refresh rate: 26; iter: 32;  vuser: ?; total time 41.796206    [MsgId: MERR-17999]

But what we wanted to know to know at the end of the test was a summary of errors as in

• Report.c(199): Error: App Error: refresh report. DataSet took longer than the refresh rate to load.  This may indicate a slow database (e.g. HDS) – 100 errors
• Report.c(200) Error Null arguments – 50 errors

Approaching the solution

How would you solve this problem ? Think for a moment and not more ….

Well being the C++ junkie that i’m, i immediately saw an oppurtunity to apply my sword and utility belts of hash-tables / efficient string comparisons etc and whipped out Visual Studio 2010 (i havent used 2010 too much and wanted to use it) and then it occured to me

Why not use Java ? I had recently learned it and it might be easier.

The Final Solution

cat *.log | sed  ‘s/total time \([0-9]*\)\.\([0-9]*\)/total time/;s/loadTime: \([0-9]*\)\.\([0-9]*\)/loadTime: /;s/iter: \([0-9]*\)/iter: /;s/refresh rate: \([0-9]*\)/refresh rate:  /;s/refreshRate: \([0-9]*\)/refreshRate:  /’ | sort | uniq –c

I probably should optimize the sed list of commands such that i can pass in the replacement commands as input to sed from a file. But i left it at that, happy that my solution was ready to use / would work all the time / does not require maintenance / saved me a lot of time.

The end

Thinking about the episode, i was reminded of the hit Japanese samurai movie Zatoichi and its happy ending at the end.

• The villain (problem) is dead
• Samurai is happy with his effort
• The subjects (files/compiler/system/software/hardware) are happy due to the efficiency created
• Karma (company + job) is happy due to the time saved.

and to boot it nearly follows the Art of War zen which advices the clever warrior to win without pulling out his sword …

Happy coding !!!

How to find and keep crappy programmers

Why japan is doomed (and US and EU too)

How a small town went bust much like US and Greece

But wait a minute,  what is all this that you see when you expand the external dependencies ?

The drop down shows ALL the dependencies, even those that are indirectly pulled in. The feature is useful but i wish someone had put a thought into how it is used … If i had been shown the layered dependencies, that showed the directly referred h files and then the ones that are further pulled in and so on, it might have been more usable.

Wish someone from Microsoft does hear this and make amends !!

Even ghosting was not an option those days, since (Well we hadn’t heard of that back then) and the hard disk sizes was changing so often, we had to keep upgrading them every now and then.

Now finally, there is a PERFECT solution for this – create your own OS image or appliance (if you are into selling devices) using free software and cool web applications -

SUSE studio allows you to -

• choose the packages you want, (automatic dependency pull in)
• default customizations (locale, users, startup, installation scripts etc)
• copy your own files into the final system

You can finally create an installable image, of that perfect OS you had always dreamt of. Or at least the one we guys used to dream of.

Watch the web casts  on the home page and dedicated screen cast page. They are really cool. I wish this could be done with windows too and it would be god send for folks who sell computers or sell applications to dumb end customers. wow !!

I made 3 folders

• Check daily – contains max 10 blogs i will check and read new articles from
• Tech Stuff – Has blogs, tech news and lots of similar stuff
• Photography stuff

The point is, the only way i could catch up on my reading was if i was not trying to catch up with a lot of things. In short, i was coping with information overload by just ignoring all that information.

So my solution was to arrange stuff so that i could catch up with the ones that mattered. The rest, well i might flip through them on evenings or weekends IF i get around doing that and i would still be better than before, and not ignore it totally.

Where else do we face overload and how do you cope ?

• email accounts
• Electronic stuff that you work with/want to buy
• Health notes from forwards and emails
• List of things to do / read etc
• Things to keep track off, and manage at work
• Education (exams skipped during school coz you would not score anyway?)
• Weight loss – (So much to do, might as well not try)
• Social Networking updates ?
• Cooking ?
• Stocks and Financial info ?

I wonder if there is any other way to cope with overload, other than just filtering  … outsourcing  ??

The significance of this move is the implication that companies like CISCO are ready to provide internal support and has made resources available to support free apps like GIMP, or are paying someone else to do this, which basically means the same thing. This is really cool.

The loss of social stigma against uptake of free applications is the most important win here for the software development community as a whole. wow !! I guess we shall now see lots of small startups come up to support the free software and hopefully  this would mean more volunteers to add features and fix issues in free software.

FSF rocks !!!

Finally, change !!

The interesting question (one of many) from the survey, which prompted me into writing this post is as follows -

How much of your time do you spend on each of the following?

• Defining the problem
• Requirements gathering
• Designing solution
• Writing code
• Building code
• Refactoring code
• Debugging code
• Writing tests
• Testing
• Deployment Support

The options against each are, (Not at all, some, About 50% and A lot). There can be overlaps. So what would be your answer?

Here is mine -

1. About 50% time designing the solution
2. About 50% time writing code (Base code writing time)
3. About 50% time refactoring it  (I always keep improving the structure)
4. some time testing it    (I write a lot of functional tests to test my code)
5. some time debugging (I pride myself in lowest bug counts ever in most teams i ever worked)

What does your distribution look like?

ps : If you are in big corporation, your time spend overall for coding related activities above might be only 30-40% of the overall time you have.  This poll is meant to measure the activity spread within that little amount of time you really get to work with the code.

Personally i’m most thrilled and would be testing out these VC++ features in the same order -

• Call Hierarchy Window (great for learning existing code layout)
• VC++ Parallel programming Library (PPL)
• Return of the MFC class wizard.
• Ribbon Designer for MFC. Wonder how this control works and how to define a context.
• Code profiling and coverage (out with Rational for these tasks)  with emphasis on threading
• Generating code and architecture diagrams
• Recordings of manual tests and their playback – wow for QA teams i suppose ?

However since this was a pet project and done on my personal time, i could not get any QA time on this or afford to spend the normal time i would spend in creating the code normally. Not with the new Agile process and the accompanying rally tool used to measure all my tasks.  So  i  did a real quick job out it and ended making some parts sloppy and corrected a few not yet encountered bugs, which i had to rework, since after all i could not afford to have such a sloppy tool lying around with my name on it.

It was then that i thought i should perhaps list down the origins of the bug which i saw in the code, along with the other common ones i have seen over the years in different code bases. So here they are, without much ado.

1. An unfinished running list of “things to do” like cleanups / code smells, which introduces an unmanageable level of code debt.
2. Heavy refactoring, which moves code around, which changes post and preconditions for the code copied, which is not handled or ignored in its new area – this was the new one for me.
3. Lack of adequate error checks
4. Not performing required cleanup when you exit a function or block
5. Performing multiple things at a time WHILE coding (2 code-bases / support calls/ emails / calls / meetings …etc)

Bugs produced from this category of reasons are the most common and perhaps the most easiest to sense early, from a casual read of the code. For the same reason therefore, it is totally irresponsible of a team or its management if code which smells of the above manages to get released.

The next category of bug origins, would lie on a higher level and are more difficult to address and cannot be found from glancing the code and would typically require good code checking tools like lint or good reviews.

1. Incomplete information about the API being used (eg free has to be combined with malloc and not new)
2. Logical mistakes in the program flow

Seriously though, any programmer worth their salt, would have learned enough of their tools and would have passed enough theoretical exams, to make the above two points, extremely rare in the real world. It is here that the importance of the ‘frivolous’ reasons, increases in significance. In all my programming career, the most bugs i have seen are due to initial set of 5 reasons, which makes it all the more important to eliminate those.

It is also  true, that only if the first set is eliminated, will the second become more obvious in code reviews, which is another reason why the initial set MUST be caught and corrected.   The majority of the cost of a programming project,  which is incurred after the coding is done, (80/20) over its support cycle, is also perhaps a good reason why these frivolous yet frequent bugs must be rooted out of the code base.

The remaining 3 reasons which complete our top 10 would be

• Platform complexity (custom Operating Systems, rare databases / applications, messy legacy code etc)
• Architectural mess
• Missing or wrong or Incomplete Requirement

Of-course the last few are issues  that cannot be easily corrected once inherited and one can but put up with the mess and keep chugging along, managers and programmers included. In such conditions it is obvious that one need not add to the woes by creating more bugs and unwelcome noise on top of an already complex platform.

Therefore by all means and for gods sake create CLEAN code the first time !!!.

After many trials, of  SP3 file download / CAB download etc, i found this simple command, after running which the install succeeds

Jut run ” esentutl /p %windir%/security/database/secedit.sdb “

Yup, thats all.

Why does this happen – check these KB articles 1 , 2 and here is the page that has a small description of what secedit.sdb (Local security database) might be

Are you rationally Minded ?

LightPeak Port

Windows 7 makes SSD goes faster

Windows 7 comes with DirectX 11 and it allows apps to use the spare processing power in GPU

40+ great free tools

15 great chrome extensions

The best FireFox addons

Inspiring read about a truck driver who has changed the movie industry (AGAIN)

Microsoft Top coders prefer old school programming tools

Google says they resist hiring industry stars

Code A

STDMETHODIMP CMIB2Iface::CollectData(LONG DeviceId, BSTR IpAddress) {
	list<CString>::iterator listIter;
	for(listIter = ccmList.begin(); listIter != ccmList.end(); listIter++) {
		CString ccmIP = *listIter;
		long tempCCMID = getDeviceID(ccmIP);

		// Handle it only if device ID is valid
		if (tempCCMID != -1) {
				CString component;
				long eventType = 0;
				if (endPointType == 0) {
					eventType = 47;
					component = "Gateway Link - ";
				} else if (endPointType == 1) {
					eventType = 48;
					component = "Unity Link - ";
				}

				component += endPointName;
				// Construct an event.
				LunaEvent ev;
				CString attributes = "linkDestination = " + endPointIP;
				ev.attributes = attributes.AllocSysString();
				ev.eventTypeId = eventType;
				ev.deviceId = theCCMID;
				ev.component = component.AllocSysString();
				ev.timestamp = (COleDateTime::GetCurrentTime()).operator DATE();

				try {
					if (endPointStatus == 0) {
						if (tempCCMID == theCCMID) {
							// Raise/Update an event whenever status is 0
							if (prevEndPointStatus == 0) {
								// Case 00
								unsigned long result;
								MY_LOG_DBG(functionName << ": Updating Event on Component : " << component);
								if (faultManager->updateEvent(ev, &result) == S_OK) {
									MY_LOG_DBG(functionName << ": Updating Event Secceeded");
								}
							} else {
								// Case 10
								unsigned long result;
								MY_LOG_DBG(functionName << ": Raising Event on Component : " << component);
								if (faultManager->raiseEvent(ev, &result) == S_OK) {
									MY_LOG_DBG(functionName << ": Raising Event Secceeded");
								}
							}
						}
					} else {
						// Case 11 and 01
						// Always clear the event, and for all CCMs in the cluster.
						unsigned long result;
						MY_LOG_DBG(functionName << ": Clearing Event on Component : " << component);
						if (faultManager->clearEvent(ev, &result) == S_OK) {
							MY_LOG_DBG(functionName << ": Clearing Event Succeeded");
						}
					}
				} catch(...) {

				}

				SysFreeString(ev.component);
				SysFreeString(ev.attributes);
		} // End of if (deviceID != 0)
	} // End of for loop
	return 0;
}

Code B

STDMETHODIMP CMIB2Iface::CollectData(LONG DeviceId, BSTR IpAddress)
	{
	this -> m_sLog.SetIpAddress(IpAddress);

	/*   ---------------------------------------------------------------------   */

	m_oInspector.AddCollatedRowInspectorFn_Post(FilterIfTypeInfoCollect,
						(void *) this);

	/*   ---------------------------------------------------------------------   */

	m_oInventory.ClearAllCollectors();

	if (BOMBED(DoTableGenericInit(m_oInventory, sIfaceTblColln, m_sLog)))
		{
		_log(L_ERROR, "Mib2 collect table init failed \n");
		return S_OK;
		}

	/*   ---------------------------------------------------------------------   */

	if (PASSED(m_oInventory.CollectInventory (DeviceId, IpAddress)))
		{
		GetIfFaultsExistForDevice(DeviceId, LINK_DOWN_NUM , m_iFaultExist, m_sLog);
		m_oInventory. InspectData(DeviceId, eForInventory);
		m_oInventory. SaveData   (DeviceId, eForInventory);
		}

	return S_OK;
	}

Code Review Department

If tasked at reviewing the 2 pieces of code listed above, chances are that, you might as a reviewer, want to get rid of the task of reviewing A, than spend any more time on an obviously bad effort at coding. Having badly styled code around, allows bugs to remain, and fixes to be nothing more than half hearted patches which neither the patcher nor the original coder can recognize after the act is done.

One bad turn soon begets another and the rate at which the bad code degenerates into sphagetti mess can only be matched by the speed at which engineers long for a total rewrite of this mess. (Failing which they secretly start yearning for a change of job or worse, stick to the same one in a lost/zombie sort of way or decide what they really want to be is a manager)

Humans are not compilers

Human Compiler

Unlike compilers, humans being are quite easily put off by bad code and no act of redemption can save a bad piece of code from degenerating further and further until it can only be discarded. This is frequently the reason why engineers advocate re-architecture, which is hugely more expensive than adapting existing code. The lost business oppurtunity that this represents can be huge and the very effort of doing so could run into millions of dollars per year for a major software company like CISCO.

Any act of review on such a bad piece of code can only seen as a sham – who after all in their right mind would want to spend so much time reviewing someone else’s code when it would so obviously take such a long time to do it.

So the first step in getting working reveiws is to not let the process to be viewed as sham – and the only way that this can be achieved is by respecting the reviewer’s time and patience.

How do you respect a reviewers time ?

By giving readable code (see above)

How do you make code readable ?

1. By avoiding coding smells
2. bi folwing sm cdng convntns(by following the same coding conventions aka coding style)

Twenty different people and their cousins would have an equal number of vastly differing coding styles. If all of these are to be heaped on same page or on the same reviewer, again its equivalent to implicitly admitting that you have no control and that the whole process is a sham. Therefore getting the same style of code to be used EVERYWHERE is what has to be ENFORCED for successful code reviews to happen.

My experience

In the only place i have personally seen code reviews to actually work, a body of code could be rejected simply because a character exceeded the max allowed no of column width of 80 or because in a single location in code, variable was not aligned. The rationale behind this was the assumption that that the coder did not know about this rule and therefore there would be many other places where this mistake would be made.

The act of rejecting the whole body of code due to single misfitting character sends the right signals and soon it becomes a matter of pride to get code accepted without major comments.

The other advantages in having readable code is that

1. Simple Bugs are easier to spot
2. Design issues are easier to spot
3. Review happens faster (that builds more support for review)
4. Bug fixes are easier
5. Module upgrades or re-architecture would be easier
6. SPotting reuse chances are easier – and hence aids faster development
7. tois which so often happens are much more easier

How can the organizations aid in creating readable code

1. Use tools that can perform static analysis based on accepted coding styles and flags issues, before reviewer even reads one line of code
2. Make code readability indexes, part of statistics of the project so that they can be easily tracked
3. Use tools that can reformat existing code base to the accepted style, no matter what that is, so that its easy to get off the block when the process starts

1 out of 2 is still only 50%

Fixing the code review-ability is only half the fight. Where would a reviewer be, without a programmer who wants the code to be reviewed ? In a team of colleagues, its only more than easy for everyone to scratch everyone else’s back and implicitly give easy reviews. Its also natural for colleagues to contest one others reviews. Therefore, there has to be an external entity and stimulus to engage in code reviews.

Of-course enforcing the code standards using tools part of the build process and enforced through management numbers would be a good start, though that alone might not make it stick.

My experience is that usually stimulus takes the form of bragging factor or nerd pride amongst peers. When an obviously respected person  / geek is to review your code, you would want it to be more shiny and that extra perfect.

Companies like Cisco can easily make this happen by having more tech leads to review the code, and having a system of recommendations where recommended parts of code can even get promoted to technical evangelists like Stroustrup or Gosling once a year for review, which can be such an honor. Tracking  programmers, coding bullshit index would also be a good measure for wanting a review.

The human factor

Coding and reviewing code, is ultimately a very human activity and any mechanical solutions to the problems in this domain are bound to fail. Ultimately a more humane and social solution would be required to transform this bit of ‘work’ into something more than statistic, and something of an advantage.

“Any fool can write code that a computer can understand. Good programmers write code that humans can understand.”

- Martin Fowler

Once we decided to make one, the question was, what tools to use? Which platform /  language and related tools should we chose? Well after ruminating over these for a few days here is what we came up with.

Platform Choice – iPhone is cool but Apple has entry barriers for development.

I was initially in favor of developing the application for the iPhone but there seems to be so many available. We were also discouraged from developing for iPhone by the fact that it required a hacked up VM or a Mac for iPhone application development.  Rumors, of how Apple tight fistedly controls, who could develop for the platform, along with an entry fees for being listed, took out a lot of enthusiasm from Apple development.  I kept having this deja Vu on how Apple missed the boat on the PC revolution, due to its tight fisted policies of the past regarding Macs and now it seems to be doing the same, all over again.

I do though feel that a mobile phone based application would have more no of users, since the expenses can be entered into it without fail, and therefore very convenient. Its a shame that Apple is playing so hard to develop for. It

Platform Choice – Re-invent the desktop experience

Since iPhone seemed out of the question, the desktop screen area and its powerful graphics cards seemed the next best for developing a cool eye catching UI application and to utilize the bigger screens for useful reports.

So, its a desktop app that we would do.

What language should i use for desktop development?

Next up the question is the choice of programming language / platform on the desktop, which we are going to use for our apps. I definitely cant use C++ because that doesn’t seem to make any sense these days with all those .NET and Java or python type easy to use platforms available. These are simply too productive to ignore and we are not making a performance sensitive application anyhow. So that brings our choice down to a compiled Ui friendly language like

.NET or Java

I believe a slick UI (User Interface) is going to influence users more than anything else, with customer’s bar of expectations set high, by the iPhone and Vista user experiences.  .NET with its WPF User Interface and close Windows OS integration, supported by the excellent Visual Studio development environment would make for a great platform is what i felt.

I did not want to use Java remembering its old slower Spring based User experience and since .NET looked much more slicker (due to WPF) and Visual Studio would  be a more productive tool to develop with than eclipse can be, for the windows platform. JavaFX is still too unstable and the tool support is lacking to move into it. (We did run a small evaluation and it does seem another six months or one year would be gone before some support of JFX development was available)

Most of our probable customers (and the future ones who might be ready to pay for a better version) are probably running Windows, so cross platform is not a big consideration. NET also plays into my expertise with windows platform and win32 API experience. So we might not get stuck with any platform issues.

Choice of Database

The next tool that we wanted to freeze on was a database that i would want to use along side my application. It would have to be free since i plan to ship my application for free initially. The ones i have heard of that i think might be useful here are Microsft SQL Express, SQL Lite, FireBird, MySQL, PostGres, Ingres and HSQLDB. These are the ones i have heard the most about. The one thing i have learned from my years of experience is that a more widely used tool would have most of its common problems ironed out and the public forum based support might be better too.

So which of these can be my database?

Multi-platform support is not a big deal for me since i have already decided to use .NET and WPF. However it definitely needs to support .Net well and it should be small. I cant expect users to make 100MB downloads to check out if a free tool is good. 6-10MB is what i think the size ought to be for this app download for it to be viable.

That preempts PostGres and Ingres which are both heavy weight. I did not have to try these out to know they are heavy weight because a two minute search on Google reveals how PostGres is already a competitor for enterprise apps and Ingres used to be one. Additionally Ingres seems to have lost out all backing and is being sold so that rules it out too.

MySQL ? Well that’s not free for commercial use . Plus i have also heard from multiple sources (mainly on Sourceforge), that the sources are a real mess and i know for a fact that till recently it did not have good SQL features like foreign integrity. Not too impressive ,I’m afraid.

HSQLDB is java based and introduces additional platform considerations. Not interested in that set of problems too.

This leaves SQL server, SQL Lite and FireBird

SQL Server is also heavy though not as heavy as PostGres. It comes to about 53 Mb in size for deployment and once deployed SQL server express behaves like a proper DB consuming lots of memory etc on the client machine.  But no, i would not want to deploy a DB server on my personal laptop just to keep my accounts.

SQL Lite vs FireBird

So it boils down to SQL Lite and FireBird. Between the both it boils down to IF we require the extra functionality offered by FireBird. SQL Lite is very small (230K) but lacks many SQL features like row/table locking and integrity constraints or stored procedures and such. BUT it supports seamless movement to a mobile environment if we require it.

FireBird is great but might not  be suitable for mobile environments. So the question i guess is will i require a mobile environment support, given my infatuation towards mobile development?

Do we require Mobile Environment support?

After some thought i decided to not muddle my development by imposing some arbitrary requirement that might not ever be required. I should develop for Mobile if i want to go Mobile and not ruin my desktop experience for the sake of some doubt-able future compatibility or requirement. I’m sure a better user experience can be developed instead if i can devote  the time saved in getting around SQL Lite’s limitations to the development of my application.

Database = FireBird
So FireBird it is and hopefully it plays well with .NET (http://firebirdsql.org/dotnetfirebird/) else we shall have to move to to SQL Lite.

What does FireBird have?

Well it has .NET support, deployment size is 2 MB + has XCOPY deployment. It has row and table locking, integrity constraints, stored procedure support for faster data manipulation) and VStudio .NET Visual Design Support.  It also used to be developed by Borland and that speaks for an impressive past history however wrong i might be.

It has all the pluses is what i felt, not to mention a favorable review in the past from a friend who was doing the same kind of research. That was four or five years back. Things can only be better now.

So that’s it then i guess. The best tools for the job are decided and all it takes is the time to build one. I shall keep this space updated with how good or bad FireBird turns out to be, when we really start using it.

Cheers !!!

However i was stumped to read the following explanation, of a .NET Data provider product (aka data source connectivity classes), that can talk to databases and provide integration with the new ORM (Object Relational Mapping) layer, that microsoft has come up with for .NET applications

“It equips Microsoft .NET based applications, development environments, and programming languages with conceptual entity-based access to native and heterogeneous data sources. “

Conceptual Entity

How soon can you say WTF?  I wonder why folks feel compelled to string so many heavy words together, when describing anything related to OO technology, in anything more than a  semi formal  setting. 

 I dont know if an architect / developer or a marketing personnel wrote that statement. In fact i did not find their site by searching on any of the weasel like words found in the description. In fact i searched for more practical stuff that i wanted my software to do, which thankfully was available on the page too.

This weaselization / complexfication of terminology seems to be a given fart to contend with, in the Java world and it makes me sick to see the cancer spreading to maybe the .NET / MSDN world too. Please, would anyone care to provider a linguistic inter mapping layer for conceptual to real word entities in the object oriented technology space?

Being a password, it implied that the text had to be truly random, which is a tough order of requirement for any home-brew algorithm, once you really get an idea of  on why its hard to generate randomness

Radom is hard

The solution = /dev/random .

In theory /dev/random is a file you can read infinitely. However in practise you would find that reading this file would block every now and then until it has contents in it, generated randomly from device driver noise bits. So you might have to move your mouse or spin the disk etc to get some random bits into the /dev/random file.

A More practical solution = /dev/urandom

urandom can get random iput from from /dev/random, else it generates fairly randomized bytes for you, using a Random Number Generator Cryptographic algorithm, provided by a kernel module of the same name. AND, its never guaranteed to block. Talk about best of both worlds !!!

My random Text Command

Bytes read from random or urandom are not likely to be all ASCII and therefore requires to be filtered for ASCII, for which this was the chosen solution -

tr -dc A-Za-z0-9 </dev/urandom |  head -c 14

The problem in the solution

The only doubt that this simple solution envisaged in my rustic Unix mind was if this command line would exit after head had received the 14 characters it required.

If you do not know the answer, it is contained in my previous post on Unix shell pipes.

Happy hacking !!!

ls / -R | head -2

The simple command line above, lists (ls) the entire directory structure starting from root in a recursive fashion. Its output is piped to head which will exit after it gets 2 lines of output.

The problem

Its clear that head will complete much before ls will. This poses some interesting questions like

1. Will ls run in the background after head exits?
2. Will the entire command keep running until ls is done or will you get the prompt as soon as head exits but ls will keep running to completion in the background?
3. Will head run only after ls has finishes giving its output, thus requiring ls to run to completion?

I recently ran into these same set of questions about assumptions being made in an install script that was being written for a new reporting tool we where building. So what is the deal here ? Think for a minute before you read the answers.

Pipe Basics

I knew that the way pipe worked was to redirect the output of one process (stdout) as the input (stdin) of the process that came next in line. But this did not quite give all the answers. Therefore googling the pipe process gave some more details which i shall describe below

The details

Poetry in design

1. The shell sets up the pipe. The running processes use stdin / stdout as they would normally have without knowing its a pipe rather than the terminal input / output.
2. A buffer exists  between the pipe descriptors, which is handled by the kernel. Writing process will block once this buffer is full and will be able to continue once the reading processes read off this buffer.
3. The process of exec and forking means that it is impossible to say which processes run first and is governed only by CPU scheduling.
4. All the processes are in the same hierarchical level and have the shell as their parent.
5. In Linux versions before 2.6.11, the capacity of a pipe was the same as the system page size (e.g., 4096 bytes on i386). Since Linux 2.6.11, the pipe capacity is 65536 bytes.

But what happens when one of the piped process exits?

Unfortunately, most of the documentation only explained how the pipe was setup by the shell and kernel.  But what we required was an explanation of what happened if one the process exits.

To test this i ran a few tests. Here are my observations.

ls / -R | head -2 | head -20000 (exits immediately)

ls / -R | head -20000 | head -2 (exits immediately)

cat small-file.txt | ls /-R (runs for a long time)

Obviously, it is the processes that read input, which makes a difference in how the piped command line behaves. So what causes this behaviour?

Enter Unix Signals

Signals are the IPC mechanisms akin to software interrupts used by Unix like operating systems to notify running processes of interesting events. This is possibly the only way an outside event can modify the behaviour of an otherwise normal process.

A process could recieve a signal for IO issues / child processes termination or plain Ctrl-C related interrupt signals.  Most of the default handlers (which you can change if requires) cause the process to exit.

My attention having turned to signals, i ended up walking through the wikipedia for the master list of UNIX signals, hoping to find anything related to pipe. This caused me to find an interesting entry …

SIGPIPE

The wikipedia entry for SIGPIPE says “When a pipe is broken, the process writing to it is sent the SIGPIPE signal. The default reaction to this signal for a process is to terminate”.

If attempt is made to read from a pipe whose write end has been closed, the reader receives an EOF, which causes the read program to close in an orderly fashion.

hmmmm – applying that knowledge to our pipe commands would show that this would cause the chain of processes to exit, if a listening process exits and thus breaks the pipe chain.

It would have been an extreme waste of resources had ls continued to run even after head exitted in the command line ls / -R | head -2 and shows how well the UNIX system has been thought out.

Even more importantly it is highly intuitive that ls / -R | head -2 to exit immediately after 2 lines have been output.  This is one of the cool things about Unix, that it behaves in the manner you expect it to, as if the commands understand the social niceties of human pysche or at least hacker psyche.

It is also highly interesting to note that the reader receives an EOF, because output finishing first is normal but the write receives a signal if the reader has already exit, since that is an abnormal situation. Very beautiful indeed.

It is no wonder that such clever hacks and excellence in design makes Unix the hacker platform of choice.

Happy hacking !!!!

ps : All this would have been clear had i chose to read the documentation of the pipe system call before the investigation was begun. It goes to show how much we ignore existing documentation at our own peril.

I can vouch for what is being said because i have seen this happening many times in development. The implied up shoot  ?

Clean code bases beget new code which is cleaner or clean enough. Dirty code bases attract more code which is at least just as dirty or usually even more so.

Cheers !!