As Data And Cloud Rise, Lines Are Blurring Between Developers And Data Scientists

With the rise of the cloud comes an explosion in the value of data. Because of the increasing prevalence of cloud infrastructure in nearly every industry, data is easier to collect, and the tools needed to glean intelligence from it are growing in accessibility. The potential that data offers has catapulted it to the forefront of an organization’s priorities. In fact, the demand for both data scientists and data engineers is projected to grow by 39% by 2020, according to a recent IBM report.

Aside from the spiking demand for data professionals, what’s interesting is how the rise of data is reshaping countless roles within companies, from marketers to operations management. Almost any job critical to a business can be improved with data. Content can be strategically marketed to consumers based on their preferences and location; doctors can diagnose patients more accurately with real-time disease studies; even sports stadiums can tap user data to bring fans an interactive experience from the second they set foot in the arena.

To create data-intensive apps, organizations are looking to their developers and, in turn, their data scientists. As we shift toward a data-first approach, these two teams -- which have historically operated in close but separate silos -- are finding themselves thrown together as they identify, design and create technologies that harness data.


When joined together with the right cloud tools and open technologies, developers and data scientists can help each other to analyze and pick out key data trends and patterns, build new AI and machine learning models, and make sure that the right data gets to the right business stakeholders at the right time -- whether they are customers, employees or decision makers.


Bridging The Gap

The distinctions between data scientists and developers have always been somewhat ambiguous, and it seems they continue to become more unclear every day. For example, if you examine almost any programming model, what you will find is that core analytics functions that data scientists have traditionally managed (data handling, statistical analyses and calculations) have always been at the heart of app development.

Data scientists have always played a role in building these models by creating the tools developers need to power their apps. However, this has historically been done in a process akin to a relay handoff, with data scientists translating incoming raw data into analytics-friendly languages such as Python, analyzing it, building models with the resulting insights and then handing it back over to the development team, which then typically translates the data again into the best programming language for its app. After building in these data models, developers then provide feedback to the data science team on what needs improvement and the process starts again.

In today’s world of constant innovation, this pace is not fast enough. To build with data at a competitive pace, it’s imperative for these two roles to take advantage of new tools and approaches that allow them to work together. A few ways include:

Turning to the cloud: Data platforms and notebooks, powered by the cloud, allow teams such as data scientists, developers and business analysts to work together across different languages and data models.

Implementing a collaborative, trusted environment to share data: Data owners are often hesitant to share intelligence across teams, as they can be unsure of how to share data as well as what is safe to share broadly. Creating a data governance strategy, as well as implementing tools that simplify the cleansing and sharing of data with trusted parties, helps break down barriers and facilitate collaboration.

Eliminating data analysis silos: Data analysis has typically been done in a conveyor belt-style approach, with each data scientist performing a separate step. While organized, this means a single mistake has the power to ruin an entire workload. Automation tools can help to eliminate these silos by simplifying and accelerating data discovery, cleansing and training, thus giving data scientists the capacity to focus on analysis and collaboration with others.

As an example, let’s take an airline that wants to decrease the amount of turbulence each flight experiences. Working together on a unified cloud platform geared toward both data science and app development, the airline’s developer and data science teams can work in a centralized environment to explore data patterns and identify ideas for a solution. Previously, these teams would have had to spend a tremendous amount of time passing data back and forth and iterating potential ideas and machine learning models to build out instead of collaborating on them together in one space.

This new level of communication and iteration would yield a product ready to be put into production much more rapidly. For example, imagine a mobile app for pilots that would place the most relevant weather data, previous flight reports and recommended route changes into their hands, helping them to achieve the smoothest flight possible. In this case, the combined team could bring in multiple data streams from flight routes, weather patterns and previous flights.


Working side by side with developers, data scientists can analyze trends across these sources, communicating with developers to identify and visualize the most relevant findings. As data scientists create machine learning models, developers can simultaneously build out features around these models, giving immediate feedback about what’s working and what’s not.

The Power Of Collaboration

Technology is enabling developers and data scientists to do more and share more while using fewer resources. To streamline processes and effectively share knowledge, both teams need to be equipped with a cloud platform and embrace a collaborative environment instead of being restricted to working in separate silos with different tools and programming languages.

But this shift is not just about adopting new technologies -- a culture shift is also required. Business leaders must also open a line of direct communication with their data science and development teams to be sure a collaborative mindset is implemented and adopted.

So, what can your organization do to best serve the people who handle such a valuable asset? More on that to come.

This Article Source is from : https://www.forbes.com/sites/forbestechcouncil/2017/09/14/as-data-and-cloud-rise-lines-are-blurring-between-developers-and-data-scientists/#1dfaecc75036