Trends and Trajectories in the Software Industry: implications for the future of work | SpringerLink

Four theory-guided aggregate dimensions emerged as a result of the analysis (see Fig. 1): (1) shift from manual tasks to scalable solutions, (2) increased emphasis on data, (3) convergence of IT and non-IT industries, and (4) cloud as the dominant computing paradigm. These themes are described in the following subsections. First, we elaborate on the technology trends, and then we connect them to the identified aggregate dimensions representing change in nonmaterial culture.

4.1

Shift from manual tasks to scalable solutions

The informants noted that the abstraction level of development tools across domains within the software industry has constantly increased, and this trend can be expected to continue. This means that while there is (and will be) a need for developers throughout the software stack, from the operating system kernel to the highest abstraction-level user space applications, the proportion of development work that takes place on the high levels of the stack increases. One of the developments that may boost this is the proliferation, advancement, and broad application of ML (deep learning in particular). For example, this was noted by informant P5, who stated the following regarding the future of software development:

“There is this idea of Software Development 2.0 and connected to languages like Swift, and the idea here is that you use ML to approximate any function. Because ML is essentially just approximating some function.” (P5).

The growing abstraction level is also visible in the popularity of programming languages, where there has been a shift from low-level languages (e.g., C/ C++) to virtual machine-based languages (e.g., Java), and further towards scripting languages primarily intended for web environments (e.g., JavaScript). The informants also mentioned and further speculated about the role of visual programming tools, which appear to be fundamentally present in various development platforms. On this topic, informant P6 provided the following explanation:

“We use SAS enterprise guide that has a graphical user interface, and we just press buttons and it does the SQL queries and all that automatically. This helps people who have no prior experience in SAS to be able to contribute faster than with old code-based SAS-versions.” (P6).

In addition to programming languages and tools (such as SAS), the increasing abstraction level in development tools can be seen in the emergence of various development platforms that manage a multitude of aspects for developers. The informants discussed low-code and no-code environments in robotics (e.g., RPA), game development (e.g., Unity, Unreal), and web development (e.g., Drupal and WordPress) as examples of how the abstraction level in developer tools has increased. As an example, informant P7 stated the following:

“Companies doing robot process automation (RPA) have largely transitioned into low code/no code. So there, the skill requirement for getting certain things done is lower. I actually see the development as a camel with two humps. On the first is the developer tool developers, who are highly skilled and specialized, and then on the other we will have low code/no code developers” (P7).

In addition to developer tools, the proliferation of the DevOps/MLOps paradigm and related technologies is another major technological driver of the reduction of manual labor within software development, as indicated by the informants. For example, popular online repositories (such as GitHub, GitLab, and BitBucket) provide support for using DevOps with CI/CD pipelines, cloud service providers have online lectures on how to build MLOps pipelines on their services, and tools such as Docker have become a standard within most software development projects. Informant P3 stated the following:

“There are essentially… or I mean in essence, we have two steps in virtualization. First, there was the shift from physical servers to cloud, and this has already happened. Then there is now the container world that you run all software in containers, and this will likely stay to some degree, but I don’t know if containers are suitable for all corners of larger software systems” (P3).

The informants also discussed the occurrence of shifts within project management to accommodate DevOps and enable it to be employed efficiently. In this transition, software developers seem more eager to start using DevOps in full, whereas customers appear to be more skeptical regarding the potential benefits of DevOps, according to our informants. This issue is articulated in the following quotes:

“We are fully using DevOps, but some of our customers are skeptical and do not understand why we need to update [our product] constantly. [They ask] can’t we just make a solid product and that’s that?” (P2).

“Even if software is developed in fast cycles, the customers may not appreciate a new update every day. Not even every month, and not even every year.” (P17).

One final concept related to this theme was that of microservices and the idea of utilizing premade components for building complex systems rapidly. This trend was fueled by the availability of free software blocks, the architectural trend of creating “mosaic software,” and the increasing abstraction level of development work. Informant P4 discussed this process as follows:

“When Unix command line tools do that one thing well, you can chain the commands together or use the outcome of one command in the next command. Similarly, in microservices, if the responsibility limits are well set, then in the best cases you can build bigger working systems by using smaller blocks.” (P4).

Overall, these technology trends were driven by (and connected to) the nonmaterial cultural trend of a shift from manual tasks to scalable solutions. The informants argued that while the increased emphasis on scalability and automation has fueled and directed the formation of these technologies and practices simultaneously, the technologies fuel automation and emphasize scalability in software business. Regardless of the drivers of scalability, the informants perceived that software development as a whole was transforming in such a way that an increasing amount of manual labor was being replaced with automated systems. The main barriers to this change were currently seen to be nonmaterial cultural aspects, such as company culture and developer skills, which connect all the way to IT education. Moreover, this shift toward scalable and automated solutions has implications for developer roles, with increased development time being spent in writing automated tests and making use of available tools and components as much as possible. However, there are also limits imposed by technology and nonmaterial culture pertaining to what can be automated as illustrated by the following quote:

“There is unavoidably a limit in what you can fully automate, even in our case, and we are not at the limit yet, but what we are doing is trying to use existing technologies and AI (–) to automate our product as far as reliably possible.” (P1).

4.2

Increased emphasis on data

According to our informants, there is an on-going process in which a once very specialized form of software, ML, is becoming mundane. They argued that creating specific ML systems, such as machine vision tools, no longer requires the expertise of a data scientist, as these systems can be built through relying on pre-built application programming interfaces (APIs). Overall, the informants discussed the following reasons to explain this proliferation of ML: (1) solutions that have made the handling of data and models easier, (2) the availability of processing power, and (3) the use of existing APIs for building ML systems. However, all agreed that ML technology was not even close to its peak, and that these technologies still had significant momentum in academia, industry, and public debate. Many of the contemporary solutions built to support the development of ML systems remove two essential barriers for training ML models: (1) the high technical skill requirement associated with understanding the mathematics behind the training routines, and (2) having access to sufficiently powerful hardware for executing the required computations. Moreover, the proliferation of ML techniques has been rapid, as illustrated by the following quote:

“10 years ago, when ML was largely an academic field and we studied random forests and support-vector machines, nobody was interested. Then, suddenly, deep learning became prominent and immediately a narrative surfaced that the age of man is over and Skynet is coming” (P18).

Despite ML tools and systems becoming more common, the models themselves have increased in complexity. Solutions are also developed by both the industry and the academia for explaining inscrutable ML models, as explained by informant P9.

“There’s LIME, made by a guy called Marco Tulio Ribeiro and that’s used [for explaining ML models]” (P9).

The informants highlighted that (most) ML algorithms are being published as open source and shared openly via academic repositories (e.g., ArXiv), code repositories (e.g., GitLab, GitHub, and BitBucket), and software forums (e.g., StackOverflow). Premade APIs and frameworks bring complex algorithms to the disposal of programmers with relative ease. For example, the PyTorch and TensorFlow APIs were mentioned by several informants as highly important for the software business in general, as they enable nonspecialized software engineers to implement ML systems. Similarly, the role of monitoring tools, tools for data versioning (e.g., DVC and Delta Lake), and various other open source pre-made components have become a standard in software development in recent years. The informants also mentioned an ongoing convergence process between DevOps tools and ML development, discussed as MLOps, where tools such as Azure Machine Learning, Amazon SageMaker, and ML Flow Databricks are used to automatically track every trained model version and the parameters and data employed.

As software engineers globally have access to mostly the same tools for making use of data, high-quality data is becoming an asset that provides companies with a competitive advantage. This means that companies are placing greater emphasis on data collection and curation. Consequently, this could lead to data collection practices that are harmful to consumers, although counter developments have already emerged. For example, legislation such as GDPR has been introduced to protect consumers from rampant data collection practices and any subsequent negative outcomes of data collection, such as privacy violations and personalized cyberattacks that build on information leaks and personal information. Data assets that have been accumulated for years also reinforce the position of leading players, increasing the costs of entering a market for new businesses. On this topic, informant P6 explained the following:

“Our company has our own data, and then there is publicly available data. If you are a new company entering the market, then you only have the public data, and that puts you at a disadvantage.” (P6).

While the changes in data-driven development could lead to more data scientists being hired, some informants disagreed. Instead, they felt that the skills of data scientists would simply become part of the skill toolbox of all developers. Furthermore, the experts suggested that the boundaries of specialized developer roles are becoming looser and that individual developers may need to step outside clearly defined boxes (i.e., “UI designer” or “data scientist”) to support development work more effectively. Informant P11 explained this topic as follows:

“At least all developers should work closely together. (…) Too clearly defined roles in a development team lead to problems sooner or later. Of course, sharing [responsibility] is not always easy either. (…) At some point there might be a situation where you need to call a friend if you’re doing something where your own expertise is insufficient.“ (P11).

The informants also discussed how ML and deep learning are utilized in increasingly many solutions and systems. Drawing from cultural lag theory (Ogburn, 1957; Ogburn, 1966), despite continual advances in ML and deep learning technologies, they can be viewed as an existing technology that is now being adopted into practice. As the tools and platforms for creating ML systems become readily available, the adoption of ML has shifted focus toward the acquisition and curation of training data. This increased emphasis on data has had various implications for the field of software development. These include the need to recruit personnel responsible for curating data, increasing the maturity level with regards to data collection, resolving legal issues related to data storing, and validating and ensuring the quality of ML system training data. Altogether, the informants suggested that there has been a holistic shift in the nonmaterial culture of software development toward more data-intensive development practices.

4.3

Convergence of non-IT and IT industries

The convergence of non-IT and IT industries was discussed during the interviews in a variety of ways. A recurring theme was the blurring of boundaries between digital and physical products and services toward cyber-physical and increasingly systemic offerings. The following quote by informant P7 illustrates this trajectory:

“Digital and physical components (in products and services) are being intertwined, often indistinguishable from one another…so in various so-called ‘traditional’ industries the offering…I mean the product or service or a combination of them…is in its essence cyber-physical.” (P7).

This convergence (of non-IT and IT industries) was also connected to automation in the way that IT is currently being applied in previously non-IT industries to automate labor that was previously manual. Hence, the informants differentiated between automation in software development (theme 1) and automating manual labor, with the latter being related to non-IT businesses becoming more digital. The participants provided examples, such as machine vision and anomaly detection, which are increasingly used to solve various business problems in non-IT fields. Informant P2 discussed how this process creates competition between novel IT startups and incumbent companies in traditionally non-IT fields:

“There is competition between incumbent companies (…) new startups are continuously looking to hog a share of the market. Sometimes, if incumbent companies are slow to adapt to new possibilities, clinging to their old ways, the new companies who have made scalable models from the get go win over, quickly outpacing the incumbents” (P2).

Cultural lag arose in the interviews when discussing disruptive technologies and the use of IT in traditional industries. According to the informants, companies need to constantly observe and follow IT developments as the advancement of technology is rapid. Moreover, several small, rapid, consecutive improvements to systems can quickly amount to bigger leaps. Informant P18 explained this topic on ML system development as follows:

“It’s funny to look at how in 2018 (…) we used convolutional networks for biotext mining and got quite good results, and it was quite timely then. But at that time, these new long-short term memory networks became prominent. They were the hottest thing for about a year, but then came these attention models like BERT. So, the advancement cycle of these technologies is really fast.” (P18).

The increasing role of digital elements in various products and services is considered one of the clearest signs of the convergence of IT and non-IT industries. In turn, it is viewed that this increases the demand for developers and other IT staff and constitutes a change in the role of IT functions in organizations. For example, the informants mentioned that several incumbent retail companies are now aggressively hiring developers as more of their business moves online. Similarly, as a result of banks opening more online services, they have less need for customer service personnel and a constantly growing need for IT staff. Moreover, logistics companies are hiring data scientists and investing in IT companies to stay on the edge of self-driving vehicle development. The following quotes from the interviews reflect these changes:

“I think they [enterprises who increasingly use and offer IT products] really should employ their own IT people, but when we look at the company landscape today, we do not see this happening in practice” (P4).

“If you look at our products…and the same applies to our direct competitors but also to the whole ecosystem… the digital things become more and more important. (…) Various tools that help operate the products better and more efficiently are being infused to the products themselves (…) and making the digital play together with the non-digital becomes what the customers expect.” (P7).

“Previously IT has been some kind of a support service, but today when we look at, for example banks, software is in fact their core service. In this case, it is almost impossible to outsource the programming.“ (P8).

While this trend of non-IT businesses transforming into software businesses was pertinent, the interview data indicated that there is a great deal of nuance and complexity involved. First, the trend may not apply to all industry sectors. For example, according to the informants, most service professions are unlikely to be replaced by robots. Second, there is a countertrend emanating from the direction of software consulting businesses, where they wish to sell complete solutions to customers and obtain larger profit margins, instead of renting workers. As companies have an increasing number of IT systems as part of their portfolio, it may be feasible to outsource some development work. Informant P9 gave the perspective of a software consultant company, arguing that it is in their business interest to provide software as a service (SaaS) to customers instead of lending workers:

“[Our company] wants to move towards providing entire software and platform products as a service. (…) But for this, we would need to increase the level of our competence to extend beyond mere programming, more towards business transformation and life cycle support.“ (P9).

Building on cultural lag theory, the opportunities provided by technology to automate business operations increase pressure on nonmaterial culture to automate manual tasks. However, there are resisting forces, such as the level of maturity within companies to automate tasks and the needs and demands of the workforce. Moreover, the invasion of IT into non-IT industries has created room for various startups to challenge incumbent companies, as fairly stable industry sectors have suddenly been dragged under the influence of rapidly advancing IT systems. The informants provided various examples of brick-and-mortar retailers (e.g., H&M) being challenged by new competitors who have scalable business models designed for the web from their inception (e.g., Zalando and ASOS). These examples suggest that the nonmaterial culture of a company can influence the pace of digitalization within a company by hindering a company’s ability to make optimal use of the latest technological affordances.

4.4

Cloud as the dominant computing paradigm

Cloud services and their role in software development were mentioned by almost all informants (often spontaneously and in conjunction with other topics) on multiple occasions during the interviews. Cloud platforms provide a wide range of benefits for developers, ranging from reducing development costs to guiding developers to use well-tested and efficient development practices. The informants maintained that knowledge related to leading cloud platforms has become essential for software engineers. Furthermore, they suggested that other stakeholders, such as company leadership and potential clients of the software or software projects, should also have a general level of knowledge about them. The informants almost unanimously considered cloud services to be an essential part of the software stack of most development projects, and that it was no longer an option for most businesses to not utilize them. Informant P1 elaborated on this as follows:

“Various services and frameworks are like a stack, where you have the hardware and infrastructure at the bottom, and in principle the uppermost layers are ready SaaS applications. And the higher up in the stack you operate, the more stuff you have there at the bottom that is made for you. (…) Of course, there is some cost in changing your stack to another, but the alternative of building everything from scratch costs too much. Being free from vendor locks is no longer financially feasible. “(P1).

These thoughts were echoed by the other informants. Another key trend that was discussed related to the growing role of cloud services in providing guidance to the software development process. Although cloud services initially and predominantly handled the hardware side and nothing else (see e.g., infrastructure as a service), they are currently managing an increasing proportion of the entire software stack. In other words, cloud services are already at the level of providing a platform and software as a service, but their role in the software development business is only expected to increase. For example, informant P11 stated the following:

“The cloud services provide premade tools that enable the building of alarms and monitoring [into the software], and we of course use and rely on them heavily.” (P11).

Consequently, knowledge of cloud services has become an important skill to teach at universities as part of software engineering curricula, and a requirement in several job openings in the field of software. The informants also raised concerns that the proliferation of a few cloud platforms (such as AWS, Google Cloud, and Microsoft Azure) has contributed to the materialization of an oligopolistic situation where only a few dominant platforms remain, and where it is difficult for new alternatives to enter the market. Although this trajectory was viewed as somewhat problematic, the informants underscored the importance of the platforms. For example, the dominant role of cloud computing platforms was described as follows:

“It would be a waste [to not utilize the big cloud platforms]. They are big products, widely tested, and not easy to do ourselves. (…) I pay for electricity as well, don’t I?“ (P13).

Edge and fog computing approaches were perceived as a potential counter trend to the proliferation of cloud computing. The informants viewed privacy and security as the major drivers of these approaches, in addition to being less prone to issues arising from poor or a complete lack of internet connectivity. While there were drivers toward (and away from) cloud computing, the informants were skeptical about a future where the overwhelming majority of computation was not carried in the cloud. For example, informant P5 stated the following:

“For quite some time we’ve discussed edge computing and that edge computing is coming, but so far that trend has not become reality (…) Instead, we seem to continue to move towards cloud computing.” (P5).

Looking at the trend of cloud computing as the dominant paradigm from the cultural lag perspective (Ogburn, 1957), we have already seen clear evidence of businesses reacting to this trend by adjusting their nonmaterial culture. For example, there are observable shifts in the hiring and development practices of software consulting businesses, where increasing emphasis is given to experience with prominent contemporary cloud vendors. Furthermore, a few informants presented evidence regarding the convergence of software development and the development culture promoted by major cloud service providers. More precisely, and as already mentioned, cloud services are taking increased responsibility for how software is made and are producing many instructional videos and offering guidance and documentation, allowing users to make the best use of their systems. Such developments can be seen to further bolster the role of cloud services in the software industry.