COMMENTS
I recently rewrote large portions of an existing application, and although I keep the "no re-write rule" in mind, I found there were two compelling reasons to do so:
1. To write out portions of code dependent upon 3rd components. In my case, I was using several components that caused 95% of my support issues. Getting rid of these was painful, but paramount.
2. The existing design doesn't lend itself to readily extending functionality. This type of refactoring is a judgment call based on (a) how much time it will take to recode and (b) how much time it will save you down the road. For startups, the time savings ROI must be very good because of the vagaries of time estimation and the short time window a startup has to succeed.
One possible situation where it might be smart to rewrite your software is when your underlying architecture is so screwed up that it makes the application slow and prone to failure -- e.g. when you have an Oracle expert write a PHP-based application server (a real-life example). A complete rewrite may also be a good moment to get rid of all the features that you have built in thinking that they will be usefuls but turned out not to be used by anyone.
That being said, I agree with your points there, and especially I agree with the refactoring as the ideal tool to make a messy code into a relatively good architecture.
You may be forced to rewrite when potential buyers refuse to buy your business(not your product) becausethe product is not written in Java/C#/Python/Ruby/PHP/Whatever(#3 from a different perspective).
"You may be forced to rewrite when potential buyers refuse to buy your business(not your product) becausethe product is not written in Java/C#/Python/Ruby/PHP/Whatever(#3 from a different perspective)"
How big a problem is this really? I'm sure that in some cases it does happen by I imagine the reasons behind this decision may be more to do with how they perceive (and to a lesser extent, use) the product.
E.g They want PHP instead of Java. Why? They may like the idea of being able to add/modify functionality and they perceive this to be easier in PHP than Java. But is there an opportunity to change how the client interacts with the product to fulfil at least some of their requirements? If that program was written in another language but identical in every other respect, it would be provide business benefit to the client?
It is not how the product is perceived, because the product has a well established and happy client base. The problem is that the language/platform is considered too esoteric and the buyer has no confidence of being able to maintain (if some unknown problem arises) or rewrite the product themselves. Also, the market might not justify the buyer to risk the rewrite themselves.
I am writing my comments for the purposes having of a mini-celebration! The Sox won! (Sorry Colorado. We like you but it was our time (again!) You’ll have your day in the sun.)
Dharmesh, you pose an interesting question. My old boss and mentor, Dale Reynolds says “its that one good idea” that starts you in one direction and you end up somewhere else because ultimately, you have a series of good ideas. To me, you never do a complete rewrite unless you aren’t getting traction whatsoever.
Due to be released Dec. 1, we wrote
www.epaperchase.com in .Net, C# using ActiveDirectory. Was it the right choice? We hope. Why? Growth, support, speed, infrastructure, object modeling, additional tool suites, etc. MS has the appropriate level of support. I considered open source. I love open source and what it stands for; however, I can’t spend time vetting the entire solution for exposures. It was a business decision. I’ll leave the rest of the exposure vetting to
www.hackersafe.com .
The first major rewrite of code I did was as a consultant. There were bugs that I should have just solved. Instead, out of sheer arrogance, I took too long and rewrote the code to make it run faster. It didn’t solve the business problem any faster. When my VP asked why I rewrote it since it was for the most part complete, I said something really stupid like “What I wrote is faster and more elegant.” That really didn’t go over well since it was his code in the first place! Let’s just say I have learned lots about myself since then.
On rewrites; I think you rewrite code that does not perform (speed, business logic, user logic, interface). Getting there in the first place? I believe the critical thing is the foundation. Did you get your data model right? Is your security in place? Did you write with any object hierarchy in mind? Is it extensible? Does it scale? Language selection then becomes driven, in many respects, by what is available based on the platform you have chosen and what is acceptable to the constituency. Users on the other hand, almost never care. They want it simple and to work. It’s usually only the geeks and technology wonks (geeks in suits), and hackers who care. One last note – time is the only thing we can’t rewrite. Unfortunately, it drives everything.
I have to disagree. I would say doing one or two rewrites is more the norm and I would even say it is recommended.
I am more maintenance and evolution oriented, but these days technology changes faster than ever. Best practices and available tools and techniques evolve rapidly.
Most importantly, you rarely understand the application fully the first time around. The first version of any system is inevitably off the mark in many ways.
Time to market is crucial and delaying development to 'get it right' is more costly than getting it close and getting it out in the real world where situations and users always reveal unexpected aspects and possible directions.
More often the impulse is to patch and evolve when you really should bite the bullet and rewrite. Or you should at least develop a plan for a methodical major overhaul done incrementally.
If you have good architects who have written major systems before, they know how to approach key aspects of the design such that they can evolve more readily when the time comes, and it *always* comes.
If you have mostly younger, less experienced developers, they will usually box themselves in and the overhaul process is more difficult and painful. This isn't good or bad, it's just part of the learning process. This is why it's usually valuable to have some more experienced system designers on the team.
The key is to be discerning and not to just take one person's word for it, or worse, to let the deveoplers operate unsupervised. The best approach is to have someone in management who understands all of these issues. Who can ask the right questions. And who can tell the difference and who can accurately and objectively assess the trade-offs.
This is why every senior management team needs a person who understands technology and has experience in these areas. Otherwise they are going by gut and often make the wrong decision or no decision at all.
There are no blanket rules that always apply. Each situation is different. You need people who can tell the difference.
If you have to do a full re-write, you're in trouble though. I agree with the evolutionary aspects of software design. I believe great object modeling helps in that regard.
I'm not an experienced developer, but I think that once you can list the business requirements for your product, and identify how technology choices will solve that requirement, you can stay focused at the start. Reducing the temptation later on to say, "well x technology/programming method has certain benefits, let's rewrite the code to implement it.
I'm currently in the design/implementation phase of a product I was to create a business out of. I had a lot of trouble deciding on the MVC platform and database(s). Whenever I think about the choice in my head, I start thinking about the various pros and cons and depending on my mood, I get different answer. But when ever I look at my list of business requirement for the product, the answer is always clear. Maybe not one that I'd like, but one that is best for the business.
In the future if I see another technology that is considered better, I'll know that this is simply the progress of technology and is unavoidable. I'd be less likely to consider a rewrite, if I believe I made the best decision at the time.
Having said that, it will be interesting to see how I feel 12-18 months after my product is released ;-)
One more thing. The key decision criteria should always be whether the system can do what it needs to do functionally, and whether you can enhance and upgrade the system effectively and efficiently.
It's not about technical esoterica. It's about real-world results.
If the developers consistently take too long to add new features, or say something can't be done, or if the system is consistently buggy and unreliable, you usually need to do something more drastic like a redesign.
I agree that if you have good experienced developers this is not a common occurance. Unfortunately those folks are not common. Furthermore, the current state of technology is fraught with pitfalls and unpassable terrain.
Sometimes you just have to admit you took the wrong path and start over. These days with so many choices that is a common occurance.
Microsoft would like you to believe that .NET has everything you need, all ready to go, but the reality is there are a lot of tricky issues like security models, and overall architecture which .NET does not address.
As a consultant I saw 70% of the .NET projects fail completely because of these issues. I was called in to fix it, only to realize there was no way out. That percentage should be lower today because those pitfalls are better known and understood, but most of them still exist.
On one project in particular, I brought in 2 highly respected technical people to plot a course out of the mess without a full restart, and neither one could do it.
I would agree - re-writing the code can take just as long, however, there are certain times where the original code is so dated, it cannot handle newer concerns.
A good example: Windows
Never written with security in mind, and now they are building it in. I can guarantee they restructured the code, and look at the result. Could a complete re-write improve things? I doubt it would hurt.
Perhaps using an agile methodology (or another iterative coding process) can help in this decision but lots of 1.0 software written by founders is virtually unusable, especially considering a universe of applications developed over the last 20 years or apps that may not be built in web standards
I agree that it is a BIG decision.
I agree that it is never easy, typically costs more than budgeted and can significantly impact the company morale, sales process and focus.
However, at the point in time, when the support costs exceed the rewrite of the code and significantly impact customer satisfaction, a complete rewrite needs to be seriously considered.
I've successfully rewritten a few internal apps for a couple of financial IT depts. There are just some things that should either be retired or rewritten. It depends on the project, benefits, people, time, and talent. One of my primary reasons for rewriting an old app recently was to to make it more refactorable: support unit testing from the get-go. Bug fixes and enhancements to the old system were taking too long to test and implement and were never stable.
Thanks to everyone for your comments so far.
Looks like we have a good discussion going.
Just one quick note: The article was written primarily from the perspective of a startup (which is usually dealing with more constrained resources and newer code).
Will let the conversation continue and respond to specific points later on.
We've been rewriting our application since January and we hope to finally be able to match our previous functionality by this January. We have migrated all our users from the old system (
www.thelinguist.com) to the new system (
www.lingq.com) but our 6 month rewrite will be over a year before we're back to where we're satisfied. Never mind any new functionality.
Sure, we've improved almost all facets of the site during the rewrite but the basic functions are still the same.
Having admittedly taken at least twice as long as we projected, we are still happy with our decision and it was the right one for us.
Our old system was limping along and we were spending more and more time holding it together. We also made a decision to switch from teaching only English, to teaching mulitiple languages. These and other significant changes would have been next to impossible for us to implement given the horribly restrictive framework we had in place.
We truly did make a horrible choice the first time in our framework. An untested automated code generation platform. We've learned our lesson there!
Having said we would have done this again knowing what we know now, we will never again rewrite our current system from scratch for all the reasons you gave!
I think a rewrite must be considered when the underlying architecture can't cope with new requirements anymore (scalability, usability, etc.).
I quit my last job because of such a beast. I was hired as a software architect to turn an over 10 year old 2-tier application into a scalable N-tier application that can be distributed world-wide. A terrible mixture of technologies was already in use and I desperately tried to introduce an object model, component-based development etc. After two years I gave up because the upper management didn't move an inch and always said that the new system must have the same functionality as the old one, even when this functionality was too complex, too slow etc.
Regarding Dharmesh's comments about constrained resources, perhaps this is the key point. A startup is going to be constrained by time and skills (unless they have a generous VC funding them). And given these constraints some aspects will "de-prioritised". If I'm trying to build a web app I may not spend much time designing for security or scalability (and it's quite obvious that some companies do work this way - I've worked in them) because that may not matter much to initial customers.
Some may say that if you don't design for such things at the beginning, you're creating your own mess. But the truth is that you don't have time do everything well, because you're limited by time and skills.
A few people with extensive experience in business or development have said to me, "Don't worry about getting it (perfectly) right the first time, just get it out there. Get some customers then go back and make it pretty". But it can be hard to tell what decisions will come back to bite you in a major way in the future, forcing you to consider a rewrite.
Are there key principles to adhere to which would reduce the risk of a rewrite, or does the list of do's and don'ts change from project to project?
With my one data point I can say its a bad idea. We had 6 months budgeted for it and it took over a year.
Did we reduce our bug load? No, we actually introduced a huge heap of them in the conversions.
Did we create a better working environment? Several people left the team during the rewrite out of complete frustration.
Did we reduce the long term speed of deliveries for new features? I hope so, but we have a debt of over a year and the lose of some key team members. I highly doubt we will make up that debt in 1 year or 2 years or 5 years. To make up that debt in 5 years would mean the rewrite improves productivity by over 20%.
From now on, my plan is to identify specific problems and focus on them for a shorter period of time. It is far less risky.
you guys forget about the bugs, not every bug is easy to see and easy to fix, sometimes you find bug when program is used by clients and if you do rewrite the whole thing you got a good change of using the clients to find that bug again (i dont believe that you will rememeber every little thing about the project, plus if you rewriting it doesnt that mean you dont like the code so you havent written it).
Rewrites are good if project is small or you do it for fun in another language/technologii.
My approach is to refactor extensively as I develop. I will slip the schedule to do it right the first time. Much of my core functionality is hard as h*ll to get right and I have gone over the tough parts 3-5 times. I have a very complex but pretty clean and robust beta release product. I am satisfied with the result.
I believe there is a 'rewrite / refactor' threshold which was nicely illustrated in the first article.
You should never rewrite code that you've already sold.
The very fact that you sold it and that someone is using it is proof enough that it works and is valuable. Therefore you should refactor so that you remain both agile and responsive to your client base. After all, they are the reason you have the funds / resources to consider rewriting your entire app in the first place!
However, as long as your code hasn't hit the market and no one is using it or building up on it then feel free to rewrite as often and as much as you care, because frankly, aside from you, nobody cares !
Joel's original story is looking decidedly old - he cites Netscape Navigator as a prime example of what happens when a re-write occurs. But a big part of Firefox's success today is that it was NOT based on netscape and was completely re-written, so there's no hard-and-fast rule. Still, I'd prefer to refactor chunks of the application if possible.
Hello,
Although you've got some smart ideas here Dharmesh, I don't think you must abhor re-writing. It can work very well sometimes when the reasons are right. I can say that from experience. I re-wrote an app that I made in 1999 again in 2003 and it sales jumped way up! No it didn't take me longer to write the new app than the older app, and no, I wasn't dissatisfied.
In fact the experience was so enriching that I plan to re-write than app in a newer more advanced platform again early next year. I've already started laying the foundation.
Let me tell you a few reasons why it might be a good idea to re-write.
1. New competitors will have access to newer technology -- If you made your app in VB6, or Powerbuilder, or whatever... You can bet your every sweet dollar that market's going to be stolen when a competitor uses a newer technology to make a sweeter looking, better working application.
To stay competitive, it's better to re-write then to be obsoleted.
Yeah, you've given me an idea for an article there...
Cheers!
Cyril Gupta
@ Cyril
Does really a new platform or the technology drive a product ? Or Some really good innovation ?
In my opinion It good to rewrite when you have a stable customer base and Your developers are free. I believe even worst written applications can be easily re-factored if planned properly. (Yes...even VB 6 app can be migrated to j2ee with some small bridges). As Joel has said, Earlier app is tested and fixed for a longer time so it more safer for a startup.
I have seen projects (In a startup companies) in which they started with redesign.The design was really promising. If successful it can Push up performance even 30 times faster. (And it was needed). But as joel said, When you start from scratch you have no guaranty that it will be better than existing ones. And even if it is; If it takes little more time; Your product has already lost its luster.
Especially for startups; Products are experiments. If they are successful they make business. If failed then however good they are they are of no use however best its written its useless.
Great article and comments. A few comments on our situation. Feedback is very welcome.
We have an internal application, over 10 years old, written in PowerBuilder v5 with a Sybase v12 database running on Unix. It is a reasonably well-built, well-designed application and there are few complaints from any perspective. A few circumstances, however, have us considering the possibility of a re-write:
1. Scalability. At some point in the near future this application may need to support twice the number of concurrent users. Our current infrastructure likely will not be able to support this growth and hardware upgrades will be necessary. At this point we could either upgrade the Unix server (high $$$) or migrate the database server to a Microsoft/SQL Server environment (lower $$$).
2. Disaster recovery. There currently is no disaster recovery in place for this application and there needs to be. The issues here are similar to #1 above. Do we implement disaster recovery for the Sybase/Unix platform or move to Windows?
3. Skills availability. Code and database support are provided by a third-party consulting company. The perception (not confirmed) is that local availability of PowerBuilder/Sybase technical resources will not be sufficient, and that we might improve this situation with more "modern" technologies.
I realize that there are many more things to consider and I'm sure my train of though may already be flawed, but I thought I would throw this out there and see if anyone had any thoughts or comments.
Thanks in advance.
Rich: You bring up some very good points.
In my opinion, you're in a situation where the cost/risk of not doing a major overhaul may be higher than doing a rewrite.
I think you're looking at it in the right way. The one thing I would do (if I were you) is to break it up into manageable pieces so you're not changing too much at once. For example, one could make the case to change the database platform first -- and worry about the app later. Or, the other way around. In my experience, by tackling smaller pieces at a time, you dramatically reduce the risk -- even if the net desired outcome is the same.