Day: November 30, 2016

Software development teams often make decisions about software architecture or technological stack based on inaccurate opinions, social media, and in general on what is considered to be “hot”, rather than solid research and any serious consideration of expected impact on their projects. I call this trend Hype Driven Development, perceive it harmful and advocate for a more professional approach I call “Solid Software Engineering”.

New technology — new hope

Have you seen it? A team picking newest, hottest technology to apply in the project. Someone reads a blog post, and we just came back from a conference where there was a great talk about it. Soon after, the team starts using this new shiny technology (or software architecture design paradigm), but instead of going faster (as promised) and building a better product they get into trouble. They slow down, get demotivated, have problems delivering next working version to production. Some teams even keep fixing bugs instead of delivering new features.

HDD is how teams bring doom on themselves

The problem with hype is that it easily leads to bad decisions. Both bad architectural decisions and technological stack decisions often hunt a team months or even years later. In worst case they may lead to another very problematic situation in software engineering: The Big Rewrite. Which almost never works out. The root of all evil seems to be social media — where new ideas spread much faster than they get tested. Much faster than people are able to understand their pros and cons.

The Anatomy of Hype

Step 1: Real problem and solution
They start in some company with a problem. A team within some company decides that solution to the problem is beyond the current technological stack, process or architecture. The company creates a new framework, library or paradigm and soon the problem is solved.

Step 2: Announcement, buzz and keywords
The team is excited to show their work to the rest of the world and soon they write blog posts and do talks on conferences. The problem oftentimes is non-trivial, so they are proud to present the impressive results of a non-trivial solution. People get excited about the new technology. The only problem is not everybody who gets excited is able to fully understand what the exact problem was and all the details of the solution. It was a non-trivial problem with a non-trivial solution after all. With communication tools like social media, blog posts and conference lightning talks the message gets blurred along the way.

Step 3: Mania starts
All shades of hype driven developers read blog posts and attend conferences. Soon the teams all over the world start using the new technology. Due to the blurred message — some of them make hasty decision to use framework even though it does not solve any of their actual problems. Yet the team does have expectation that this new technology will help.

Step 4: Disappointment
As the sprints go by, the technology does not improve the team’s life as much as people hoped, but brings a lot of extra work. There’s a lot of rewriting the code and extra learning for the team. Teams slow down, management gets pissed off. People feel cheated.

Step 5: Realisation!
Finally the team does retrospection and realizes what are the tradeoffs of the new technology and for what purpose it would be more relevant. They get wiser… till the next hype shows up.

Examples of Hype:

Example: Microservices
Step 1: Big monolith application scales hard. There is a point when we can break them down into services. It will be easier to scale in terms of req/sec and easier to scale across multiple teams.
Step 2: Hype keywords: scalability, loose coupling, monolith.
Step 3: Let’s rewrite all to services! We have a ‘spaghetti code’ because we have a monolith architecture! We need to rewrite everything to microservices!
Step 4: Shit! It is now way slower to develop the app, difficult to deploy and we spend a lot of time tracking bugs across multiple systems.
Step 5: Microservices require a lot of devops skills in the team and with right investment might pay off as a way to scale the system and team. Before you reach serious scale issues it’s an overinvestment. Microservices are extracted not written. You must be this tall to use microservices.

Example: NoSQL
Step 1: SQL databases have problems with high loads and unstructured data. Teams around the world start developing new generation of databases.
Step 2: Hype keywords: Scalability, BigData, High Performance.
Step 3: Our database is too slow and not big enough! We need NoSql!
Step 4: We need to join tables? That is a no go. Simple SQL operations are becoming increasingly challenging. Development is slow and our core problems are not solved.
Step 5: NoSql are tools to solve very specific problems (either extremely high volumes of data, unstructured data or very high load). SQL is actually a great tool and handles high load and huge data volumes well if used skillfully.

Good practices

Test and research before you decide:

Spikes — learn about your technology not from blogs, but from experience. Take 1 or 2 days to build a prototype of a new functionality in the new technology, before you make the decision. Let the team analyse pros and cons. You might take a couple of competitive technologies and let different parts of the team builds prototypes with different technology.

Hackathon is a great way to build awareness of a team on tradeoffs of different technologies. Take 1 or 2 days for the whole team to hack all the technologies that are on the rise and look tempting to use. That will allow the team to make clever decisions on it’s own and take decisions based on their own experience.

When to go?

In principle when return on investment is huge. Most technologies are created to solve a particular problem. Do you have that problem? Is it a big problem Will it save a lot of time? Will the use of the technology pay back the cost of the entrance curve and rewriting? What if we initially slow down development by the factor of two? Or four? Is it still worth it? Great teams are allowed more — some teams are just faster than others at delivering the value. They get bored with what they do more easily. These teams can introduce new technologies more often. That’s no excuse not to use spikes and do hackathons. On the other hand, if the team has troubles delivering — proceed with caution.

Hire the right people

Strong technical background is your friend — people who know different paradigms, understand theory of programming (e.g. algorithms, concurrency) and have good engineering culture tend to hype less. Experience — hype is stronger with young developers. With years people who have seen many technologies and been in troubles many times tend to get more balanced views of picking technology.

Adapted from: DaftCode