Towards the use of Data Engineering, Advanced Visualization techniques and Association Rules to support knowledge discovery for public policies

Abstract

Education and employment are key aspects of a country's well-being. Governments expend valuable resources on designing education plans and employment programs. These two aspects are usually analysed separately, although, as they are closely related, considering them together might improve their efficacy. The problem lies, at least in part, in the fact that different public entities manage their own data with their own isolated systems, and do not develop joint educational and employment policies. In order to facilitate working towards this goal, in this manuscript, we make use of Data Engineering, Data Visualization, and Intelligent Data Analytics methods to create a decision support system for the Government of Extremadura. Extremadura is a European Union Objective 1 region in Spain with high rates of unemployment and secondary school drop-out. Data Engineering is used to create a Data Warehouse that unifies the different data sources into a central repository for quick access and control. This allows dealing with the challenge of transforming, processing, storing and accessing the data. Data Visualization techniques are applied to create an interactive dashboard that assists users in analysing and interpreting the data in the Data Warehouse repository. Thus, charts, diagrams, and maps are created specifically to help technical or political decision-makers. Finally, Intelligent Data Analytics techniques are used to incorporate Association Rules into the visualization dashboard. Its goal is to identify associations, relationships, and patterns in data that, at least in plain sight, are not readable or interpretable by humans. It does this by inferring knowledge that humans cannot pick out by themselves. As a result, a complete system was defined and implemented to support public administrations in their decision-making and definition of precise evidence-based policies in the areas of education and employment. In particular, it allows the definition of unified strategies to reduce the unemployment rate.