Big Data: professionals involved and the challenges posed by scarcity

We will live in Brazil a situation in which the supply of professionals trained to work with data analysis will fall far short of the level of demand in the coming years.
Publicado em:

As in other countries, Brazil faces a shortage of resources for Big Data projects. Brazilian universities invest little in this area due to the costs of infrastructure, lack of funding (in the case of public universities) and even in the face of the current theme. Only in the last two years courses have developed, mainly at the level of specialization, directed to the theme Big Data.

The shortage of professionals covers other areas of data analysis such as analytical applications and Business Intelligence. We will live a situation in these areas where the supply of professionals will fall far short of the level of demand in the coming years. The current economic crisis has significantly slowed the growth of this demand, but the investments of companies in these areas continue to occur, since we are addressing priority areas for growth and competitiveness. At a given moment, when the economic cycle is reversed, we will experience this scarcity more sharply, which could further compromise the level of national competitiveness.

Big Data will be a huge opportunity for IT professionals and those with capabilities to analyze, model, cross-reference, interpret and generate business insights relevant to their organizations. The profile of these professionals includes the ability to work with sophisticated quantitative techniques by identifying patterns of behavior and correlations to produce insights and predictive analytics.

Many Big Data professionals usually have training in areas such as statistics, applied mathematics, and operational research. Being generally able to work with advanced tools and methodologies like modeling, data mining and programming in SAS, Python, R, Hadoop and SQL. Handled data volumes are typically measured in giga or terabytes and include structured or unstructured data analyzes. The areas of responsibility of these professionals are generally as follows:

  • Database marketing: analyzes that involve customer segmentation, marketing campaign effectiveness, buying propensity models (or evasion, withdrawal, etc.), customer lifetime value, among other aspects.
  • Credit risk analysis: construction of risk models for customers and companies. The results of this type of analysis are used for predicting losses, establishing the levels of credit rates, for example.
  • Data mining: mining unstructured data with the use of scripts and code lines in languages ​​such as Python and R, for example, in social media marketing campaigns (based on the processing of text information provided by followers as comments , testimonies and criticism) and feeling (with the objective of evaluating how the campaign impacted the perception of the brand and how the messages were disseminated in the different social media).
  • Marketing data modeling: construction of statistical models of customer behavior analysis with the use of transaction data at different online and offline contact points. This type of analysis can be used to feed pricing models, replenishment of gondola products, prediction of repurchases or renewals of subscriptions, analysis of purchases made, among others.

In terms of managerial functions, Big Data projects require sophisticated skills from skilled and experienced professionals. Our research points to some characteristics that are independent of the sector of performance or size of the company:

  • Experience in the management of projects involving different business areas: in this sense, aspects such as capacity for conciliation of interests, negotiation, understanding of divergent (and often opposing) needs and broad perspective of the results and objectives of the project are essential;
  • Experience in software development projects: this is not a prerequisite for success in Big Data, but a very desirable point, given the similarities between the two types of project;
  • Knowledge about data collection, preparation, analysis and distribution: management experiences and / or project execution involving these activities are crucial, since Big Data deals exactly with data, in this case, considering all perspectives of the 3Vs model;
  • Soft Skills: Collaborative skills, leadership, curiosity (minds oriented to the discovery of new patterns and perspectives with foundation and logic);
  • Focus on problem solving: The recommended approach for Big Data projects is that they focus on defined, precisely identified and recognized problems. Thus, organizations would begin with smaller projects with clear objectives and questions to be answered. It seeks to identify in which points the analysis of data will enable significant leaps in performance. It is critical to know which questions you want to answer and the decisions to be made. The most appropriate is to start initially for a pilot project;
  • Knowledge about the information that is available: this is the capacity to draw up a mapping of the data available at the bases of the organization. Only with this knowledge will it be possible to know exactly what information and crosses will be needed, as well as the expected results with a Big Data project;
  • Fostering an Information Culture: Big Data project managers often take on the role of broadcasters of cultural changes related to how people deal with information and recognize its value. The success of the initiatives is linked to the development of an organizational culture that values ​​information and discovery, encouraging the contribution to the best possible use of this asset and the constant thinking about the value that can be extracted from the critical and grounded analysis.

Finally, coming to the end of this series of posts about Big Data, we highlight some of the best practices adopted of projects adopted in Brazil and abroad. As commented in a previous post, Big Data will be one of the key competitive differentiators for companies from virtually every industry in the next few years, and will be used in all business areas especially marketing, sales, HR and finance.

  • Usability: Remembering that the interfaces and logic of Big Data applications should ultimately serve end users. Many projects of this type have failed because the adherence of practitioners who effectively use the applications and manipulate the data for insights has been low because of interfaces and applications that are too complicated to actually use. Thinking about the end user of the data is fundamental in this sense;
  • Data quality: regardless of the sophistication of the technological arsenal and the body of professionals involved, no project will be saved if there is no quality of data analyzed (trash in, trash out). One of the biggest challenges companies face for Big Data’s design success is the trust that information users have in their quality;
  • Knowledge of the tools available: a Big Data initiative can be supported by different technologies, which can be more effective and appropriate to the context of each organization according to its specific sector and processes. It is important to carefully research which set of technologies best suit each reality;
  • Leadership Commitment: A business strategy that includes the use of Big Data technologies will not be sustained over time without the commitment of organizational leaders. As commented earlier, it is well indicated that an information culture is cultivated. Cultural changes often involve the participation of leaders to make their results effective.

ASM will continue to monitor the development of Big Data in Brazil in order to contribute to the development of the national and multinational organizations present. We know that the increased competitiveness of these organizations will inexorably go through the incorporation of processes and technologies aimed at tracking trends such as Internet of Things, Sharing Economy, Deep Learning and personalization of consumer experiences and self-service.