It's hard to believe that it's been three decades since I started my journey in the software industry, beginning with writing machine code on a piece of paper and then transferring it into a PC in my college computer lab—a PC you had to reserve days in advance. That was my software development lifecycle. But here we are in 2023, standing on the precipice of another profound shift in the software development lifecycle; generating entire software products, and I’m not just talking about code-LLMs.
It's an interesting time in our industry. We have an estimated 30 million developers globally [1], and yet, we find ourselves in the midst of a massive talent shortfall - to the tune of 40 million developers [2]! This shortage is projected to skyrocket to 85 million by 20301. It's not merely a matter of numbers. The gap between the available talent pool and the specific skills demanded for certain jobs is rapidly expanding, accentuated by the swift evolution and growing specialization of software engineering.
In the past, software development followed a linear, sequential process—starting with requirement analysis, then design, followed by implementation, testing, deployment, and finally maintenance. However, with the emergence of Agile, DevOps, continuous integration, delivery, and now, the advent of MLOps—short for Machine Learning Operations—we're witnessing a transformative shift towards more iterative, flexible, and data-driven approaches.
MLOps brought the sophistication of machine learning into the development pipeline, facilitating seamless collaboration between data scientists and operations or DevOps teams. This led to more efficient development, testing, and deployment of machine learning models, ensuring that the promise of AI can be fully realized in practical applications.
But, it didn’t stop at MLOps. Back in 2015, I read an article by Andrey Karpathy, titled "The Unreasonable Effectiveness of Recurrent Neural Networks," [3] and I've been waiting for the moment Andrey hinted at ever since. Today, we are seeing the birth of an entirely new paradigm in software development - Programming by Natural Language [4], enabled by the rise of Language Learning Models (LLMs). Another name for these tools is code-generators or code-LLMs.
Commercial code-LLMs like GitHub's Copilot [5], and AWS's CodeWhisperer [6] are equipping developers with the ability to generate code using natural language instructions, enhancing efficiency and inclusivity in software engineering. In parallel, research initiatives such as StarCoder [7], WizardCoder [8], Salesforce CodeT5 [9], Google PaLM-Coder [10], and OpenAI Codex [11] and Code-Davinci highlight the active exploration within the research community to utilize LLMs for code development. These Code-LLMs are not just handy tools; they signify the dawn of a transformative shift in the software development lifecycle.
While code-LLMs are making their entrance the cost, time, and resources limitations of the Iron Triangle in project management [12] stay intact. Research showed [13] that code-LLMs didn't necessarily speed up task completion or success rate and participants only preferred its use in daily programming tasks due to its utility as a starting point and saving online search effort, despite difficulties understanding, editing, and debugging the generated code snippets, which impacted task-solving effectiveness.
Why do you even need code-LLMs if the developers are not going to use it? This is a question Google researchers also asked themselves and created DIDACT [14] as a first step towards a general-purpose developer-assistance agent that tries to understand where is the best use of intelligent tools in the software development process. With 27 K+ software engineers [15] Google knows that software creation is a gradual process involving incremental enhancements, edits, error-fixing, code reviews, and tool interaction. It's a dialogue among developers, reviewers, bug reporters, architects, and tools. Therefore rather than using finished code, DIDACT learns from the entire software development process, thereby training ML models for more dynamic and integrated software development.
Regardless of figuring out the best use of these tools, and even if it improves the productivity by 100%, the world will always have a shortage of software engineers because humans can imagine new software ideas faster than one can go through a whole software development process. This is where the era of end-to-end software product generators is starting. I call it “Application Generators” but perhaps the industry will find a better name for it. Although we are at the beginning of this era, and the challenge of reducing the existing software backlog is immense, there's no denying that we are at the dawn of a breakthrough.
Application Generators, through AI and Generative AI, streamline the entire software development lifecycle, thereby fostering unimpeded innovation and infinite scalability. Businesses can now actualize their ideas faster and more economically, a core mission at my new startup, SoftwareAgent.ai (https://SoftwareAgent.ai).
I expect that progress won't pause here. Following Application Generators, possibly within a decade or less, I predict the emergence of general-purpose backend systems. Picture a user interface interfacing with AI services that oversee everything, obviating the need for explicit backend software code. Inevitably, much like the shift from narrow AI to general-purpose AI, a similar transition will likely take place. While this notion may seem like a distant future, the rudiments of this concept already exist in specialized applications like ChatGPT.
Ultimately, the final stage will obliterate the traditional software development lifecycle as we know it. I foresee a service integrating both frontend and backend systems, eliminating any need for code generation. At this stage, systems will exhibit self-adaptation and offer a degree of personalization currently unimaginable. Reflecting upon my experience with personalization platforms at Yahoo and eBay, this advancement appears to be a logical progression, poised to elevate recommendation engines to unprecedented heights.
With these developments, the software development lifecycle is poised to evolve in ways that were unimaginable a few years ago. We're not just filling vacancies or bridging the talent gap, but reshaping the very foundations of software development. As we continue on this path, I am excited to witness how these changes will transform our industry and the world at large.
Stay tuned, the future of software development promises to be an exciting ride!
If you are interested in these topics feel free to sign-up to my news at https://aiproductinstitute.com, https://softwareagent.ai or https://www.linkedin.com/in/adnanboz . You can also join one of my classes at AI Product Institute or Stanford Continuing Studies.
[3] The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy: http://karpathy.github.io/2015/05/21/rnn-effectiveness
[4] AI for Low-Code for AI: https://arxiv.org/pdf/2305.20015.pdf
[5] Github Copilot: https://github.com/features/copilot
[6] Amazon CodeWhisperer: https://aws.amazon.com/codewhisperer/
[7] StarCoder: May the source be with you: https://arxiv.org/pdf/2305.06161.pdf
[8] WizardCoder: Empowering Code Large LanguageModels with Evol-Instruct: https://arxiv.org/pdf/2306.08568.pdf
[9] Salesforce CodeT5: https://github.com/salesforce/CodeT5
[10] Google Palm-Coder: https://arxiv.org/abs/2204.02311
[11] OpenAI Codex: https://openai.com/blog/openai-codex
[13] Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models: https://dl.acm.org/doi/pdf/10.1145/3491101.3519665
[14] Large sequence models for software development activities: https://ai.googleblog.com/2023/05/large-sequence-models-for-software.html