If your company has contracted with a software development team within the last 20 years, you are familiar with Agile development. If your company has been in business for at least 50 years, you probably also have a general understanding of how Agile differs from the old “waterfall” method, and why Agile methodology is now preferred by almost everyone.
What you may not be familiar with, however, is how Agile applies to the world of artificial intelligence (AI) and machine learning (ML). As it turns out, the speed and efficiency of Agile is perfectly suited to the rapidly advancing fields of AI and ML. Flexible businesses can jump on emerging trends and pivot quickly in new directions much faster than businesses that rely on old and inflexible development practices.
After all, it’s hard to capitalize on a new AI or ML advancement if your business is still halfway through a years-long project that has been built around a previous-algorithm version. By the time your final product is released, it will already be obsolete.
Before we jump into the specifics of AI, ML, and Agile, let’s take a step back and look at a brief history of software development processes for business leaders who aren’t familiar with Agile.
What is the Agile software development methodology?
Perhaps the best way to explain Agile is to first explore how software was developed before the Agile methodology became commonplace.
In the “bad old days,” software developers had to adhere to lengthy development schedules (lasting months, if not years) that had little flexibility. A product was built according to the specifications given by the client, and that was that. Months or years down the line, the client would receive their finished application, and only then could they determine whether it actually met their needs and solved the problem they wanted to solve in the first place.
Usually, the final product needed improvement. Some features were underutilized, others needed to be scaled up, and still others didn’t function exactly as the client had hoped. However, the process of dramatically altering the software after the finished product had been built was so cumbersome, lengthy, and expensive, that clients often had to decide whether their requested changes were really worth it.
This was the state of software development until new internet applications forced application developers to devise more efficient, client-focused ways of building systems.
Enter the Agile methodology. Agile was born out of the need to develop software faster, and to make sure that the customer’s needs were designed into it at every step. Instead of waiting years to receive a product that only partially fulfilled their desires, software clients are now invited to the table for frequent collaboration. Developers put together barebones systems at first, to make sure the overall idea works as intended. After the client signs off, the developers can then add features piece by piece, with regular check-ins with the client to make sure everything performs as intended.
The process still takes time—software development is unlikely to become a speedy endeavor anytime soon—but so much time, money, and effort is saved down the line from not having to gut and rebuild a product that didn’t perform as the client wanted.
How does Agile apply to artificial intelligence and machine learning?
The development of AI and ML algorithms often require massive amounts of research and testing. Developers often find that, even if the algorithm responds beautifully for certain use cases, other areas fall short, and the algorithm must be tweaked repeatedly until results are optimal across the board.
Such an extensive process would be a nightmare to complete without Agile principles. Imagine spending months putting together a complex AI algorithm, only to test the final product and realize that it does not perform at all as you expected.
Interestingly, the beneficial relationship between AI and Agile goes in both directions. Though it’s clear to see why Agile helps in the development of AI, AI can also help businesses become more agile.
The automation made possible by modern AI can dramatically boost the speed and efficiency of testing, monitoring, and updating software. This frees up human developers to continue with coding and building the infrastructure of applications, greatly cutting the total time required to complete a project. As we mentioned above, software development is unlikely to be described as “quick” anytime soon. But AI is certainly making a positive impact.
Additionally, AI algorithms can gather vital data from market analysis or other research that could alter the company’s priorities. When a business remains flexible, it can shift to address these new priorities long before competitors even see the trend emerging. Deploying enough AI projects in this manner could help businesses get—and stay—on the cutting edge in their fields.
You can take these benefits as far as you want, if you’re willing to develop the necessary algorithms (or pay to have them developed). ML can, for example, learn the production techniques and usual development times for certain applications, and AI can apply those data to a model that more accurately predicts the time and resources necessary to complete future projects. Or, intelligence algorithms could be trained to recognize outliers and anomalies that result in greater or lesser profitability, allowing businesses to cut out the fluff that isn’t helping and focus on the most efficient processes.
However, it isn’t all smooth sailing when you develop AI using Agile methodologies.
Agile, as most developers understand it, provides a certain structure by which to work and make progress. While this approach is excellent for traditional software development, AI and ML differ from traditional software in several ways.
Algorithms are only as good as the data they use, so both the quality and quantity of data matter as much, if not more, than the coding methodology and project timelines usually followed by Agile developers. Additionally, the model underlying the entire project may need to be tweaked several times, or even completely rebuilt, before the project functions optimally.
Agile was developed with the goal of breaking extremely complex and confusing situations down into easily understandable and manageable parts. What better way to handle the unpredictable and often extremely complex projects in the world of AI and ML? When flexibility and rapid adaptation are the name of the game, don’t let your business fall behind because you’ve put too much emphasis on sticking to the exact Agile methodology as it was originally devised.
Here’s how to use the Agile methodology to develop intelligent algorithms.
The key is to stay agile with Agile, in a manner of speaking. If a certain project would benefit from finding new data sources to build a more robust algorithm, rather than the next steps outlined by the Agile method, then by all means do so.
In fact, you don’t have to reinvent the wheel. The concept of using Agile to develop AI and ML algorithms is one that has been tossed around at the academic and developer levels for the past few years, since intelligent algorithms started gaining so much popularity.
Georgetown University recently published an article on the topic outlining five project phases that work well for Agile development of machine learning projects. They are as follows:
- Devise a plan for the project that assigns the project scope, the requirements, and the responsibilities of each team member for the duration of the project. Include metrics for measuring progress toward the final goal. It’s also beneficial to come up with checkpoint milestones and their associated deadlines.
- Do a deep dive into the data that will form the basis for the project. Sometimes, this step can be lengthy and complex, and it could lead to changing your project scope and timelines if you run into anything unexpected. Make sure you have planned enough time to handle all of these setbacks without throwing your entire project off course.
- Build a few different models using the data you curated. If you prefer, you can stick to Agile methods to develop these models.
- Evaluate your metrics and see how the models perform. While you’re analyzing the project, take a look at the resources, including time and manpower, you’ve used so far. Are all of your milestones on track for completion? If anything seems out of balance, make adjustments before you continue.
- As the algorithm becomes more functional, check for biases that emerge. Sometimes the algorithm is too heavily weighted one way or another, and other times the data is not an accurate representation of reality. It’s important to iron out these wrinkles before bringing your final project to market, because any biases or inaccuracies could have real-world consequences wherever your algorithm is used.
Following a project plan such as this one can help your team use Agile principles more effectively in the realm of intelligent algorithm development. Remember, when possible, to also use AI and ML to facilitate your development and other business processes.
