03/02/2006
Have a look at the photo below. Where did I shoot it?
Seems somewhere lost in the middle of nothing.
Wrong.
This is the sunrise in the country, no more than 2 miles far from Milan. The photo was taken with my camera phone.
01/24/2006
Fog of War
Many of you will know the name of Von Clausewitz. In the middle of the 19th century, the prussian general Carl Von Clausewitz wrote an encompassing treaty on the art of war. It is still a fundamental book on the subject, a textbook in all military academies.
Among the others, a fundamental point Clausewitz made is about the Fog of War. He states that war is the kingdom of uncertainity. The details of war conduct tend to blurr like in the fog. This means that commanders must inherently take decisions based upon incomplete information.
If you have ever led an IT project of any complexity, most likely you have faced the fog of war.The analogy is not complete, as it should be applied in a competitive environment, where both parties actively fight each other. None the less the continous bargain between the client and the consultant may well be regarded as clash. Each one of us hopes to work in a fully cohoperative environment, but, often, this is not the case.
What is the fog source? There are many, on different layers.
The first layer is an incomplete/inaccurate requirements collection. As IT people, too often we tend to automate our reply to the user need. If we are asked by a customer about, let's say, better customer knowledge, we reply automatically with the acronym CRM. We are ready to provide a contact database full of glamorous fields like "contact first son birth date", a bunch of insightfull reports which tell wether the client is likely to churn or not etc. Maybe the client only wanted to know which are the clients which have been visited by salespeople in a quarter. (This is not a fake example, this happened to me, as the number of covered clients was by far the most important indicator of business wealth in that case).
The second layer derives from the customer lack of IT experience. They often divert you to interface details, cluttering the horizon of the process which is going to be automated. This does not mean that the user interface is not important; more often than not the customer experience is the key to success. The point is that polishing the user interface is the last task to be accomplished. This is very hard to achieve as an ugly-looking system will often be considered just ugly in spite of any other consideration.
The third layer is ours. A project involving more than one developer is inherently prone to insidious details. If there are no clear rules on coding and naming conventions, is sure that you will end up with collisions that will cause an unnecessary waste of time and efforts.
At the top of all this there is the project manager. He/She will inevitably lose a correct view of what's coming on and risk to take uninformed decision. This happens when, like me, you are managing 3 or 4 projects a time.
What is the solution? The obvious one is being carefull, or paranoid, and pay the highest attention on these aspects. Probably I do not have anything to teach you on this subject.
The second solution is, trust your istinct. Experience and thorough knowledge of the basic issues involved will often point you in the right direction. I listen to my istinct more often than I will, and it is often right.
Is this not scientific? Yes, but what do you expect from an hen-master?
01/17/2006
A Morning at Work
This morning, at work, I've found bad news.
A client of mine made a bad mistake.
Each 3 months they extract from the accounting system two files with vendor's commissions and load them in their corporate datawarehouse. It's an old , Informix based, system. We made it, but I, personally, inherited the system. I cannot blame my predecessor for what happened: he worked like everyone else used to work at that time.
My client, as I told before, made a mistake. They picked 4 months to be extracted instead of 3. This meant that there was one month of doubled data loaded up. Luckily I made a one-shot backup of the fact table, from where some other fact tables are rebuilt from scratch for reporting. After a short analysis it was clear that the only viable solution was cleaning up the table and reloading data.
At this point
· I made a copy of the backup file
· I counted the records in table and files to be sure that nothing strange, after backup, has occurred
· I extracted a short selection of records
· I connected to our local test system to test the load
· Ouch, the test system table has not the correct structure
· I connected to the production system
· I prepared to load the small extract of data
· Wait a bit, is that the correct table?
· I checked the sql script which loads data to be sure it was the correct table
· I prepared again to load the extract
· Am I targeting the right database?
· I checked the connection and reconnected
· I loaded the extract and it was ok
· I prepared to delete
· A coworker asked me for an urgent advice
· What was I doing?
· I prepared to delete
· Am I targeting the right db and table?
· I re checked the scripts
· I answered the phone
· I re checked again the scripts
· I prepared to delete
· I checked the SQL
· Is it the right table and the right db? Maybe the script does something strange
· Script rechecked
· Prepared the SQL
· Stared at it for ten minutes
· Run the delete (while a cold drop of sweat run through my spine)
· Done, prepared to load the backup file
· Is that the correct file?
· Check the file
· Prepared to load the file
· Stared at SQL loading statement for ten minutes
· Loading... loaded.
An entire morning spent on this and the other tasks accumulated. If I had to have the tape backups restored the mess would have exploded.
What does this teach me?
1) Never create an incremental datamart without a mechanism to rollback the loading. I had no chance to identify correctly the extra rows, so I had to reload everything. From this point of view, sap BW is perfect. It allows to get rid of wrong data loads in a click. The data load itself is an object which can be activated, deactivated, compressed, archived etc.
In a less structured environment, adding a load identifier is invaluable to delete all the records. Simply add a field to the fact table and fill it with a unique identifier, the same for the entire process. Probably the best way is to generate it from the date and the time of data load start, so it becomes human-readable. Tagged records may be easily and selectively deleted in case of mistakes.
2) Write down a procedure to do so, test it and put a copy within the safe. In this way you'll not be compelled to re-tell the entire story from start.
3) There is no way of cutting away a human being from the loop, so be prepared to correct even the most trivial mistakes. Better, let the user correct everything by themselves.
Even in a minimally complex dw, probably, you’ll have to provide a side application to manage those data which are not managed by the transactional system that feed it. You have to sit down and code anyway. At this point, the overhead to manage also the loadings is not so heavy not to be faced.
It will spare you a lot of time while charging the same 20% of the license fee for maintenance;-).
01/14/2006
Suburbs And Downtown
Oh boy, how cold these evenings are. Today has been one of those days when the temperature drops below zero (Celsius of course) at five pm. Coming home in the cold enhances the feeling of tiredness
At home, Poldina noticed my state of mind, and asked me why. The answer was simple. That day I planned to go on with a project of mine, but I was unable because we received a lot of support request from our customers. Many of them were urgent, and I had to leave everything behind to sort out the calls.
As usual, Poldina dropped a pearl of wisdom:
“Do not complain about that, it’s only your fault.”
“Why?” I replied.
“Because you, as a consultant, can not take charge of your customer’s daily job.”
“Maybe you’re right, but in this way they feel that we provide a good service.”
“They would be happier if they had not to call you.”
True.
Let’s start from the beginning.
Shrinkwrap software development is a difficult exercise to all aspects, but consultingware opens an entire new class of problems.
As usual a customer asks you to build a system (not a micro one, of course). With diligence, you go through the analysis. After that, you write down detailed specs. After that you plan carefully all your development and testing. Let us suppose you do your homework and start developing.
Of course, despite your best efforts, in this phase you will have to face unexpected issues and specs refinement or changes. This is normal.
Experienced project managers all agree with the immutable law that software is like wine, it requires an exact amount of time to be ready to be delivered.
The late shrinkwrap software developer has the choice to leave out some features from the next release and often the luxury to announce the release date when there is each reasonable guarantee to meet it.
The consultant who is developing a custom system must meet the deadlines, because the risk is not to be paid and start losing money while struggling to complete the project.
An experienced project manager does not fall in the trap of start coding like crazy throwing away the schedule or adding people to a late project, but scraps his head, browses the specs and the contract, and picks the tasks which can be left for last and can be delivered a bit late without spoiling the whole.
What are the candidates for such a choice?
Often, among the others, they are the tasks which pertain to the management of slowly changing data. They may be user security settings or some data quality constants. Maybe some translation tables are left unmanaged. Anyway, it is always possible to use a database utility or a file editor to change them, if required.
The final result is a system delivered in time, and this is good, but it is not under the customer’s complete control, and this is very bad.
Worst of all, this is bad for you, the consultant, not for the customer. The customer will simply call in and will ask you to do the job.
Often, once people start moving to the next project, these gaps are never filled and the customer calls remain an extra amount of maintenance for a long time. Worst of all, again, this activity often goes unpaid because what has been left should have been included in the system.
The net result is made of days like today, days of retard on your following projects. A small amount of applications like this may spoil your potential to complete more projects.
Then, what should be done to avoid this pitfall?
Well, the first answer is obvious: plan carefully enough not to risk being late.
This is easier said then done, of course, but you should try to do it anyway, is it?
A second option is to accomplish these tasks first. Often these parts pertain to interfaces and security, and setting them up early may be considered as a part of your overall risk reduction strategy. Your boss, also, will perceive them as a preliminary and will not press you to produce “measurable progress” as the true work has not begun yet. Going late will not produce a lame system as you complete the core.
We may generalize this approach. “Top-down” and “bottom-up” approaches are often discussed, but the “suburbs-downtown” approach is hardly mentioned in development and project management texts.
“The kernel is ready; we must only trim and had a bunch of interfaces”
“The core works ok, I must only throw in only few functionalities and the system will be ready.”
I’ve head this phraseology so many times that I can’t count them. This sounds also as a reassuring statement for non experienced managers, both on customer and consultant side. Well, this does not mean for sure that the job is on the other side of the hill.
In well-organized town, with an acceptable life quality, you’ll find a downtown where the productive life is concentrated (the town hall, public and private offices etc) and suburbs where people live quietly and confidently with the services they require located near them.
So is any reasonably large system.
Do not expect customers be comprehensive about the roughness of system management and interfaces; they will expect it to run smoothly most of the time. Sometimes, when in hurry, they’ll say to be ready to accept roughness but be sure that they will soon start complaining about shortly after deployment.
A not-so-happy customer, of course, is likely not to ask you for more consulting twice.
So, as usual, Poldina was right. It was up to me and my coworkers to reduce the constant flow of incoming calls. I’ll keep in mind.
01/09/2006
Back at work, today. As I expected a flood of mails come through and crowded my inbox. Most of them were service requests. The working year starts under the best auspices...
01/05/2006
Despite the cold and cloudy day, Poldina and I went to vist the Certosa di Pavia. It's an awesome lombard-gothic church located few miles south of Milan. It took 200 years to be built in white marble. It express the full power of the ancient rulers of Milan, Sforza and Visconti.
What has to do this with IT? Nothing but the fact that, whil we were there, I replied a couple of mails from some friends of mine and took the shots you can see here with my mobile phone.
01/01/2006
Happy new year, may God bless you.
12/25/2005
WOW! Santa Claus brought me a brand new Motorola RAZR V3!
I'm not a mobile gadget maniac but it's so slim an comfortable that, this time, I'll go in depth with the functionalities. I'll post an usability report in a month or so.
WOW! It syncronizes with Outlook! WOW!
12/14/2005
ARCHITECTURES
Once upon a time, a young programmer was hired by a subsidiary of a large multinational company.
The environment was new and stimulating, projects were interesting, coworkers and the boss were nice and polite and, last but not least, there where gorgeus girls everywhere.
The young programmer did his best efforts to pass through the test period and made a number of fairly interesting things. The first project worked out was amazing. The international headquarter released an application to monitor customer's sell-out; that is, to monitor how much goods sold to retailers was actually sold to the end users. He quite hacked the international db structure and created a more efficient data-load procedure.
He proudly released the application to few end-users, and the news regarding the new system adoption escalated toward the headquarter. On the other side, the system rollout required a huge amount of T-SQL coding, was hard to test and to maintain but, who cared? It worked very well.
Time passed and the young programmer moved from project to project, always quite successful and was awarded with more responsibility. Actually, he became a project manager; he coded less but still dealt with all the small systems he created over the years. He disseminated within the company a number of stand-alone applications which addressed a number of business needs.
He did not know that catastrophe was behind the corner.
A time come when the foreign headquarter proposed a new datawarehouse. It was really powerful and a real improvement if compared with the old and semi-amateurish dw that was in place at that time. Unluckily, the old dw was the source of all the internal stovepipe application built by the young programmer.
He spent countless hours to make them back to work, rewriting the interfaces directly from the central system.
Probably you heard this war story many times in your career. Probably you've heard a lot about breaking down the walls among applications.
I learned the hard way that this is a central issue within a business environment. Many of us, as developers, often forget this lesson.
In my experience, developing a specific solution to address a specific business need must be accomplished keeping an eye to the environment where the application lives. Almost each application requires a data feed from other systems and may require feeding data elsewhere.
The best integration level, normally, is achieved through a consistent choice of the platform.
Microsoft environment is designed to natively integrate. For example, it's very easy to connect to data sources in excel, use VBA to manipulate them, publish the data in html format to a SharePoint site etc.
SAP has tight integration level, and, with the coming Mendocino, Office integration will be further improved.
But the platform matters to a different level. Simply pick one and stick to it. Microsoft, Oracle, SAP, whatever, each one of them has pros and cons, but in the medium-long run sticking to one will spare money and headaches.
2005 buzzword was SOA (Service Oriented Architecture). Someone sold it as the panacea to all the integration issues. It is not. It’s a sensible way to improve the heterogeneous system integration but it’s quite hard to develop and maintain. It is a good choice when real-time integration is required but does not solve everything.
Think of this the next time someone asks you to develop the next siloed application.
10/12/2005
This is the beginning of my new blog. I'll try not to annoy you. Sometimes, I'll post long articles that I hope will be of interest.
Stay tuned