Moving to WordPress

WordPress has built in code formatting (SyntaxHighlighter) so no more manual formatting of code. My new blog is here

.

Random thoughts

Slack

Reading a book called Slack by Tom DeMarco.


Very large text files processing

If you are dealing with very large text files, i.e. more than 500MB in size, here are a few tips that might help

1. On windows, use Textpad for viewing/editing files. It handles large files very well. Alternatively you can use unix utilities or cygwin if you are working on windows.
2. Java doesn't handle large files very well. Consider using Perl or unix shell script. You will be amazed at the performance gains.
3. If you need to save this to a database, consider a direct bulk copy using your database's load utility e.g. sqlldr (oracle) or bcp(sybase, ms-sql).

Procrastinators logic: Cleaning your apartment is O(1) complexity

N being the number of days since you last cleaned your apartment, for small N, the time taken, t, to clean your apartment will not vary much over N.

This makes apartment cleaning an O(1) complexity algorithm.


.

Netbeans 6.7 is released. I am still not happy with "Go To File".

Netbeans 6.7 is released.

Despite "Improved search" as one of the features of the new release, the "Go To File" feature (Alt+Shift+O) is still as slow as the previous version. This is a bummer as I use this feature most often. In Netbeans, file search is either very slow or throws a <No Files Found > even when the file exists.

Compare this with "Open Resource" (Ctrl + R) feature in Eclipse. Works like a charm and gives you a filtered list of all matching resources even before you've finished typing.

--------

Are you testing your units ?

Read a brilliant and very apt article on Functional testing by Tim Sutherland. The article makes a case of why functional testing is more important than unit testing in some applications specially those that do not have complex algorithms or APIs in the code.

The application I work on at my workplace is a case in point. It's a highly data centric, legacy, ETL application written in Java. Most of the code does not have any complex business logic that requires testing at a unit level. In fact it is the integration of the tiny java components and how they collaborate during run time that contributes to the complexity of the application. In the last 2 years that I have worked on this code, I have seen very few cases where a bug could have been caught during unit testing. Typically, most defects occur due to unexpected or bad data.

In such cases, I strongly agree with the author of the above post that a small, carefully written set of functional tests is more useful than unit tests. We can run these tests nightly as part of continuous integration and also for smoke testing during every release.

I do think, however, that at the unit level, a test driven approach might still be useful. So when I am writing, let's say, a DAO, I can write a few integration tests first for testing the DAO. Even in such cases, hard core unit testing (with mocking etc.) does not yield much benefits. These tests could be reused later for low level integration testing of individual components. But they need not be run regularly as part of the continuous integration process to save time.



--------------

Why isn't my unix sort working?

Gaah.Today I ran into a strange problem while running the 'sort' command on Unix. On running this command with the following input,

AECS
@ADS
@AED

I was getting

@ADS
AECS
@AED

as the output. I was expecting the output to be

@ADS
@AED
AECS

It was as if the '@' character in my input data was completely being ignored. This caused a long running data load process to fail due to wrong data as I was using sort and merge logic to eliminate duplicates and merge data from multiple files.


On seraching the internet, I found that the 'sort' command depends on locale to decide the ordering of characters. you can check the default locale by using the 'locale' command.


The solution to fix the above sort is to set LC_ALL to "C" before calling sort. "C" stands for collation locale.

> export LC_ALL=C
> cat inputdata sort -s -T .


Turns out that there are some other comands that depend on locale. Read more on this subject here.



-----------

Ruby on Rails

I attended a 2 day course on RoR during the Good Friday weekend. It was a beginners course.



RoR has a lot of "magic" moments when you just click a few buttons and 'viola', it spews out a shiny new web application for you. Nothing hard core though as the "scaffolding" as it is called is only good for the very basic CRUD web apps.



On the other hand, I found very little information available on the net that could explain what was happening under the hoods. It is possible that I did not look in the right places, though.



I am now practicing the concepts by creating a simple effort tracking web application in my free time.





---

Beware of ls --color on Unix

I ran into a ver interesting problem today.



I was trying to redirect the output of the 'ls' command to a file.



$ ls
1.dat 2.dat 3.dat
$ ls > log



To my surprise the output file contained lots of 'special' characters'.

$ vi log

^[[00m^[[00m1.dat^[[00m^

[[00m2.dat^[[00m^

[[00m3.dat^[[00m^

[[00mlog^[[00m^[[m



This was giving several errors in some other process that was using this file.



After spending an hour on this problem, I figured out that the culprit was an 'alias' that had mapped 'ls' to 'ls --color'. This had caused the output to contain the escape sequences for colors.



The problem got resolved by unaliasing with 'unalias ls'.

Java is verbose

While I had read and heard this several times, I never really understood till today what it meant when people said Java was verbose.

Today, for the first time, I accept the veracity of this sad fact. Java is indeed very verbose; specially when compared to new languages. Suddenly I can see how most of the code I write in java can be avoided.

Job ad trap

Read this in a job ad recently.

You may occasionally need to be on calls....some weekend work on production releases may be required , but no actual shift work.

Take my advice. Don't fall into this trap. Not only will you work extra hours, you will not get compensated for it either. Negotiate for shift job.

Random thoughts

Epic Fail
Heard someone use the word 'Rightsizing' while referring to the lay offs. Yuck!!


Privacy

I wonder if it's possible to maintain privacy while trying to run a successful small web based business?





New language


I decided to learn Scala and JavaFx. Having struggled with learning a new programming language many times in the past, I realized that my problem is that for a second language, I need a top-down approach instead of the bottom-up approach most programming books offer. Before I learn the dirty details of the syntax, I must grasp the big picture. I want to know how a language compares to Java which is my primary programming language. It is important for me to understand how a new language "does things" before I start learning the syntax. So I have decided to skim through the Scala book first without writing even a single line of code. I'll then give the book another read, this time for the syntax and other granular details. I think it is a good way to learn a second/third language.

As for JavaFx, I am using the online course at www.javapassion.com.



Architects
The real reason why we need architects is that in many places (specially big organizations), programmers are not considered important or worthy enough to take any high level design decisions. Having an architect ensures that such decisions are at least coming from someone who has a technical background and experience required to make such decisions. Otherwise, such decisions are left in the hands of inept Project Managers. If there is an architect, project managers usually don’t poke their nose into technical discussions or decision making and leave it to the architect.

Programming Problem 1 - Train Times

The Problem

City transportation planners are developing a light rail transit system to carry commuters between the suburbs and the downtown area. Part of their task includes scheduling trains on different routes between the outermost stations and the metro center hub.

Part of the planning process consists of a simple simulation of train travel. A simulation consists of a series of scenarios in which two trains, one starting at the metro center and one starting at the outermost station of the same route, travel toward each other along the route. The transportation planners want to find out where and when the two trains meet. You are to write a program to determine those results.

This model of train travel is necessarily simplified. All scenarios are based on the following assumptions:

1. All trains spend a fixed amount of time at each station.
2. All trains accelerate and decelerate at the same constant rate. All trains have the same maximum possible velocity.
3. When a train leaves a station, it accelerates (at a constant rate) until it reaches its maximum velocity. It remains at that maximum velocity until it begins to decelerate (at the same constant rate) as it approaches the next station. Trains leave stations with an initial velocity of zero (0.0) and they arrive at stations with terminal velocity zero. Adjacent stations on each route are far enough apart to allow a train to accelerate to its maximum velocity before beginning to decelerate.
4. Both trains in each scenario make their initial departure at the same time.
5. There are at most 31 stations along any route.
6. The meeting time of both trains will never be at the departure of one of the trains from a station.

Input

All input values are real numbers. Data for each scenario are in the following format:



d1 d2...dn 0.0

For a single route, the list of distances (in miles - there are 5,280 feet in a mile) from each station to the metro centre hub, separated by one or more spaces. Stations are listed in ascending order, starting with the station closest to the metro centre hub (station 1) and continuing to the outermost station. All distances are greater than zero. The list is terminated by the sentinel value 0.0.


v

The maximum train velocity, in feet/minute.

s

The constant train acceleration rate in feet/minute2.

m

The number of minutes a train stays in a station.

The series of runs is terminated by a data set which begins with the number -1.0.


Output

For each scenario, the program should determine the following:

1. The number of the scenario (numbered consecutively, starting with Scenario #1).
2. The time when the two trains meet in terms of minutes from starting time. All times must be displayed to one decimal place. Also, if the trains meet in a station, the station number where they meet.
3. The distance in miles between the metro centre hub and the place where the two trains meet. Distances must be displayed to three decimal places.

Sample Input

15.0 0.0
5280.0
10560.0
5.0
3.5 7.0 0.0
5280.0
10560.0
2.0
3.4 7.0 0.0
5280.0
10560.0
2.0
-1.0



Sample output



Scenario #1:

Meeting time: 7.8 minutes

Meeting distance: 7.500 miles from metro centre hub



Scenario #2:

Meeting time: 4.0 minutes

Meeting distance: 3.500 miles from metro centre hub, in station 1



Scenario #3:

Meeting time: 4.1 minutes

Meeting distance: 3.400 miles from metro centre hub, in station 1

SpringSource acquires G2One

SpringSource acquires G2One - the company behind Groovy and Grails. I hope it augurs well for Groovy and helps in more shops adopting it for commercial development.

Language Remix

I stumbled across an interesting paper on polyglot programming here.

I am convinced more than ever that in the war of programming languages, it is impossible for one language to outshine others. So should we start looking at things differently? Should we move our focus away from trying to pick a winner?

A few trends are worth taking note of.

- Developers are not scared anymore to experiment with different programming languages and styles. While they still have favourites, they are increasingly using the most suitable language or tool for a given job. This is a welcome change from the days when we were 'shoehorning' every solution into one tool or technology.

- IDEs allow one to seamlessly integrate different tools and technologies into one platform.

- Virtual machines have brought the languages closer to each other in such a way that the language has become merely a tool for expression. Code written in one or mixed languages produces the same byte code so the run time environment has become language agnostic.

These trends suggest that the time is ripe to try out a polyglot approach to development and see if it has any real benefits or not. The paper above makes a convincing argument in favor of polyglot programming.

Fluent Interfaces

A Fluent Interface is a design construct that makes your interfaces read like natural language instructions.

I kinda like how a little refactoring of method signatures can make an interface much more "interesting", shall we say ? The idea is to make the interface read more like English. So, for ex., instead of

//<>
Aggregator a = new Aggregator();
List l = dao.getfeedFromEurope();
List l1 = dao.getfeedFromAPAC();
List l2 = dao.getfeedFromAmericas();
a.aggregate(l);
a.aggregate(l1);
a.aggregate(l2);
a.setFilterPolicy(FilterPolicy.NonPayingCountries);
a.filter();
List l3 = a.getFeeds();
...

you'll write

Aggregator a = new Aggregator();
...
List l3 = a.aggregate(l).and_aggregate(l1).and_aggregate(l2).filterWith(FilterPolicy.NonPayingCountries);


Neat, isn't it? The Aggregate class methods are self-referential. In the above example, and_aggregate() just calls aggregate() internally. But the goal here is to improve readability of the interface. Of course, like everything else, fluent interfaces can be overused or misused. You can, for ex. make a very verbose interface or worse make all the interfaces fluent.

I had a difficult time explaining this construct to the code reviewers last time. I guess it takes some time getting used to this style.

The first class

The snippet below prints the first class in the stacktrace. Might be useful for logging in some situations.

public vid test ()
{
Throwable th = new Throwable();
stackTraceElement[] ste = th.getStackTrace();
System.out.println(ste[ste.length -1].getClassName());
}

I live inside Eclipse

I have a fetish for Eclipse plugins. I use

1. Mylyn for tasks management. Super-cool productivity plugin.
2. Database Development Plugin. Replaces crappy DBArtisan on my desktop. Lets me perform basic database operations easily. Lightweight and super-fast.
3. Azzuri clay Database Modelling plugin to create ER diagrams and generate DDLs from them; a basic feature that Micro$oft decided not to provide in Visio Professional edition.
4. Beyond-CVS to integrate beyond-compare (my favourite comparison tool) with Eclipse.
5. QuickREx for those esoteric regular expressions. I don't miss RegEx buddy now.
6. Remote System Explorer for connecting to linux machines and running telnet/ssh/ftp shells.
7. Eclipse web browser. Limited in features but helps during basic debugging or investigation when I want to search for an inexplicable ibatis error or how to convert dates to varchar and vice versa in the database (i always keep forgetting that). Also for keeping an eye on the latest on reddit :-). Wish it offered tabbed browsing.
   

My wish List

1. A Mind map plugin to let me gather my thoughts.
2. More powerful text editor. I use block-editing heavily.
3. A plugin for MS Outlook that lets me attach emails to Mylyn tasks.
4. A plugin for MS Excel
5. A powerful desktop indexing and search plugin.
   

Amen.

Random Thoughts

Xtreme Programming is not a synonym for planning-less programming. It is not a euphemism that poor management can hide behind.

Xtreme is an ideological change and should be a conscious decision.

Ternary logic in SQL

A fellow colleague ran into a strange problem recently with a "not-in" SQL query. There were two tables, say A and B and we weretrying to get all values of a column from table A that did not exist in Table B.

Here's a result of the queries we ran.

select count(distinct column1) from A
-----17567 records

select count(distinct column1) from B
-----10234 records

So to get the values of column1 that are in A but not in B, we tried,

select distinct column1 from A where column1 not in (select distinct column1 from B)
-----0 records

Funnily, the query returned no records. Something was wrong.

After breaking our heads for several hours on this problem, the culprit turned out to be some null values in table B.

We changed the above query as follows to make it work.

select distinct column1 from A where column1 not in (select distinct column1 from B where column1 is not null)
-----~8000 records

SQL implements what is known as ternary logic for handling NULLs. I found a lucid explanation of this problem here.

My experience with unit testing

I am not new to programming but have had bad programming habits for too long. "Enough is enough", I said to myself one day and decided to use a test-first approach for the next project.

Here's what I experienced.

1. Although I created the test cases before writing the classes, I didn't keep up with the policy for long. I found it a little distracting to think of and write a test case before any change to my code. The result was that soon my test cases were totally out of sync with the code.

2. It is difficult to be disciplined enough to keep test cases up to date specially when you are working with tight deadlines. Effort estimation must take into account this fact.

3. I had to continuously refactor my code to make it more testable. Writing test cases turned out to be a great way to write well formed APIs and good code in general. I had to make several modifications to my initial design to allow dependency injection. Moral of the story - automated unit tests can't be implemented as an after thought.

4. I found it difficult to test private methods and had to change their visibility to test them.

5. I found it difficult to automate everything. There were times when I just wanted to print the values of a hashmap and inspect manually. I found it too time consuming to completely automate the testing in such cases. Perhaps it's a case of old habits. But I realized how much of our testing is based on manually inspecting the results. It is difficult to break this habit and to learn to rely on automated test results.

6. Mocks are confusing (jmock) but very very helpful.

7. Seeing the unit tests run successfully didn't give me much confidence except for the first time when I ran them with fresh code. I can think of several reasons for it. Firstly, as I mentioned before, manually verifying the results of a testcase gives one a confidence that's difficult to achieve by running automated tests. There is something reassuring about seeing the results with your own eyes. Secondly, my test cases were not comprehensive enough to test full functionality. So after some time, I didn't see much value being added by running these tests. And lastly, I was coding for a database intensive CRUD application; my code was heavily dependent on file i/o and database access, both of which needed extensive integration testing. After a while, the unit tests were rendered completely useless and I was hardly running them.

8. There is only so much that unit tests can achieve. Most of the bugs in my code were caught during integration testing. But I must admit that at least some of these could have been caught during unit testing if I had better unit tests.

10. There should be a set of integration tests that check essential functionality. These tests may not run during the continuous build but must run during the nightly build.

Java Code Review Tools

Today I gave a presentation to the team on java code review tools. I covered 2 tools in detail, PMD and Jupiter. while I was talking about them, I realized that both the tools can do with some improvements.

PMD

• Improved filtering to allow me to run the tool on selective files.

Jupiter

• Must let me view defects data in a more user friendly way. I want to be able to view all defects and then aply filters on them. The functionality is there but not in the way I want it.
• Must allow me to generate reports on defects metrics.
• The workflow is confusing (why should i choose a reviewer in the 'rework' phase ?). The process should be simplified.
• There should be a filtering icon in each view.

Note to self: Explore option of adding these features yourself.