#fridaypost

The right data visualization for an efficient workflow

It’s possible to get a complete picture of current business situation using data visualization. This is especially useful when there are complex datasets and unrelated information. At the moment, there are many types of data visualization. A large number of data visualization options (arc, tagged, waterfall, violin, etc.) provide many ways to analyze data, share information, and discover new ideas. However, each information requires a certain way of visualization in order to effectively present data and meet information needs. For example,

Slope Chart

This chart shows the change between 2 points. It is effective when there are 2 time periods or comparison points and it is necessary to show an increase or decrease in different categories between 2 data points. This type of chart is suitable for visualizing changes in sales, costs, profits in order to obtain information about which indicators increased, which decreased, and how quickly this happened.

Calendar Heat Map

Heatmaps show the changes in a data set over specific periods (months, years). The data is superimposed on the calendar, relative values are displayed in color over time. This option is suitable for visualizing quantity changes depending on the day of the week, how it changes over time (retail purchases, network activity, etc.).

Marimekko Chart

A diagram is used to show the relationship of parts to a whole. It compares groups and measures the influence of categories within each group. It is commonly used in finance, sales and marketing.

With Qlik, it’s possible to create any visualization that will be most effective for achieving a goal. Interactive charts, tables, and objects give an ability to explore and analyze data in depth, that helps to generate new ideas and make the right decisions.

DataLabs is a Qlik Certified Partner. A high level of team competence and an individual approach allows to find a solution in any situation. You can get additional information by filling out the form at the link

Data Governance to improve data quality and security

The key to effective work and data analytics is data quality and security. Decisions quality and actions efficiency directly depends on data quality used to make them. This, in turn, affects the efficiency of the business as a whole. Poor quality, incomplete and inaccurate data undermine the entire business chain and prevent from achieving the desired results. In this case, user doesn’t have a complete understanding of a current business state, makes wrong decisions and develops a strategy that will not only be ineffective, but may also lead to losses. If there is no trust in the data, nothing else matters, even with a good information system.

It’s possible to provide full control over data assets using Data Governance and Data Integration. These are processes that include tracking, maintaining and protecting data at every stage of the data life cycle.

Data Governance is processes, policies, and tools implementation to manage data security, quality, usability, and availability throughout its lifecycle.

All data management processes should be automated to prevent errors and inaccuracies that occur during manual processing. With automation, it is possible to implement rules and policies to manage data discovery and operational quality improvement. The managed data catalog allows documentation and control of each data asset, definition and control of each user rights. Through profiling, cataloging and access control, user gets the access they need to well-structured datasets and accurate information at the right time.

Data Integration is a platform that automates the entire data pipeline, from ingesting raw data to publishing analytics-ready datasets. Deduplication, standardization, filtering, validation etc provide clean data delivery. The platform includes a data catalog with robust content for data analysis and exploration.

Qlik vs Power BI

Let’s continue the comparison of leaders in BI and data integration. Below is a comparison of Qlik and Power BI 12 key features.

Interactive dashboard

Qlik: the ability to explore data from any angle directly from the dashboard, generating new ideas for research;
Power BI: data mining is limited to the SQL engine. For each request, only a part of the data is analyzed, which makes it difficult to identify patterns and relationships;

Data visualization

Qlik: a complete set of interactive visualizations for free data exploration;
Power BI: has a full set of interactive visualizations. But a query-based approach limits the user’s filtering and exploration;

Deployment flexibility

Qlik: offers users complete freedom and control over data, provides a complete SaaS environment, on-premise and private cloud options;
Power BI: users need to use Azure (a cloud platform that is designed for developers of cloud computing applications, data storage);

Total cost of ownership (TCO)

Qlik: lower cost with no hidden extras;
Power BI: relatively low cost at the initial stage. But scaling requires additional upgrade costs;

Scalability

Qlik: providing high performance computing when working with large datasets and real-time data;
Power BI: using a dynamic query slows things down considerably. Additional cost need to upgrade to a Premium package for scaling;

Self-service

Qlik: provides centralization and consolidation of data in the cloud, the creation of managed data models with reliable protection. All content is created, managed and controlled in the cloud;
Power BI: the use of a decentralized approach, the distribution of data between user desktops and the cloud, the inability of users to create their own visualizations and make changes to existing ones that makes data management expensive and time consuming;

Data integration

Qlik: the best built-in capabilities for data integration, merging and transformation;
Power BI: data integration requires the purchase of additional Microsoft products;

AI-based analytics

Qlik: a full range of AI capabilities built into the platform;
Power BI: additional Microsoft AI features (Quick Insights and Q&A) on the toolbar must be added each time;

Advanced analytics

Qlik: built on the basis of the API, the platform allows to embed a dashboard, individual numbers, values, indicators into the latest web technologies and applications;
Power BI: the ability to embed dashboards in other applications, however, many features are not available in the SDK;

Use cases

Qlik: all users have the ability to perform all actions on one platform, use application automation, automatic machine learning, receive alerts, generate reports, conduct analytics in real time;
Power BI: many use cases, but will require additional costs to purchase the full set of Microsoft products;

Mobile business intelligence

Qlik: native mobile application with the possibility of interactive online and offline research;
Power BI: View-only mobile application;

Information literacy support

Qlik: a user of any knowledge and skills level can view the data, also there is a possibility of learning;
Power BI: Only the author has access to self-service.

Qlik vs Tableau

Qlik, Power BI and Tableau are leaders in BI and data integration according to Gartner report. Each tool has many benefits. However, in order to make the right choice, it is necessary to clearly understand business needs, its tasks and goals, as well as a potential value of BI introducing into different departments workflows, etc. By understanding the business needs and knowing the capabilities of each tool, it is easier to make the right choice.

Comparison of 12 key factors of Qlik and Tableau

Data visualization – data visualization using interactive charts, graphs and maps. This allows to study data in detail in any direction, identify relationships, etc.

Qlik: has a complete set of interactive visualizations. The associative mechanism allows the user to explore data from all directions while machine learning helps to identify new ideas for data exploration and offers new visual effects;
Tableau: also provides a full range of visualizations but using a query-based approach. This approach limits the user to linear possibilities within a specific data piece, that makes it impossible to explore data from different angles;

Interactive dashboard – the ability to create dashboards for more convenient and free data study.

Qlik: offers the ability to work freely with data using a dashboard. Machine learning makes dashboards more powerful by compressing large datasets and revealing data shape;
Tableau: allows to connect charts to create a dashboard. The SQL-based approach limits filtering and exploration to the queries and dataset that were chosen by the original author;

Total cost of ownership (TCO) – accounting for all costs associated with BI solutions usage for 3-5 years (infrastructure, system configuration, application development, system administration and support).

Qlik: lower cost and no hidden costs as you scale;
Tableau: the need for investments as you scale;

AI-driven analytics – new insights and connections discovering, quickly data analyzing, team productivity increasing, informed decisions based on data.

Qlik: has a full range of AI capabilities built into the platform. AI is a constantly included mechanism in the context of the analysis performed and offers new ideas and connections;
Tableau: Ask Data (an AI feature) is not a core platform and was added to the tool later. There is no possibility to work with business logic built into workbooks; connection only to data sources, that makes it impossible to use the knowledge gained by analysts;

Different use cases (on the same platform) – many use cases for BI, working with the same data and platform.

Qlik: Users have the ability to do everything, including real-time analytics, application automation, Auto ML, alerts and reports, on one platform;
Tableau: mainly it’s a development environment for analysts to visualize data;

Managed self-service – data and content control with centralized rule-based management and unlimited user power.

Qlik: data centralization and consolidation in the cloud, creation of managed data models with reliable protection. Content creation, management and control takes place in the cloud. Managed libraries enable reuse and standardization of analytics;
Tableau: The user has the ability to create copies of the content. However, managing such iterations takes a lot of time and effort;

Mobile business intelligence – the ability to explore and analyze data from any location.

Qlik: native mobile application offers interactive exploration (online and offline) and built-in alerts with push notifications;
Tableau: the ability to download only 1 sheet from the workbook to a mobile device, there is no filtering option.

Scalability – complete and up-to-date presentation of data, processing it at any scale without affecting performance and increasing costs, data integrating and combining from different sources.

Qlik: instant computing performance, including when working with large datasets, real-time data and a large number of users;
Tableau: slow processing of large data volumes and disparate data, inability to maintain data relevance;

Embedded analytics – the presence of full analytical capabilities in other processes, applications and portals in the company for effective decision-making by employees, partners, customers, suppliers etc.

Qlik: the ability to embed a dashboard and individual numbers, values and metrics;
Tableau: the ability to embed only the full dashboard, but not individual values;

Data integration – combining and transforming raw data into data ready for analysis. Modern tools allow to make data available to the entire company using real-time integration technologies (data capture, streaming data pipeline).

Qlik: better data integration capabilities to combine, transform and catalog data;
Tableau: 2 tools (Tableau Prep and Tableau Desktop) provide data preparation. Each product solves a specific problem, however, none of the 2 solves all problems;

Flexible deployment – an independent multi-cloud architecture that will allow deployment in any environment.

Qlik: complete data freedom and control, full SaaS enterprise environment, on-premises and private cloud options;
Tableau: no hybrid option;

Data literacy – improving the information literacy of employees at all levels, the ability to work with data and make decisions based on them.

Qlik: the ability to explore data for a user with any skill level, as well as data literacy training for any user;
Tableau: the user must be an author to work with data and benefit from initiatives.

Efficient Data Management with Data Fabric

Modern companies often deal with large and complex data sets from different and possibly unrelated data sources (CRM, IoT, streaming data, marketing automation, finance, etc.). Large companies often have branches in different geographic locations. This can complicate the process of data using or storing (in the cloud, hybrid multicloud, on-premises, etc.). Data Fabric will help to combine data from different sources and repositories, transform and process it for further work. As a result, users get a holistic picture of the current situation, that allows them to explore and analyze data to conduct effective business activities.

Data Fabric is a data integration architecture using metadata assets to unify, integrate, and manage disparate data environments. The main task of Data Fabric is to structure the data environment, and it doesn’t require replacement of existing infrastructure. Metadata and data access are managed by adding an additional technology layer over the existing infrastructure. Standardizing, connecting, and automating of Data Fabric data management practices improves data security and availability, enables end-to-end integration of data pipelines and on-premises cloud, hybrid multicloud, and edge device platforms.

Benefits of Data Fabric using:

consistency across integrated environments through metadata management, knowledge graphs and machine learning;
a holistic view of the current business situation, allowing users, analysts and data scientists to find relationships;
maximize hybrid cloud capabilities and reduce development and management time for integration, deployment, and maintenance by simplifying infrastructure configuration;
a simple process of studying and analyzing data for users without the participation of IT-specialists;
availability of all approaches to data delivery through support for ETL, data virtualization, change data capture, streaming and APIs;
automating routine tasks (for example, schema matching with new data sources and dataset profiling).

Data Fabric simplifies a distributed data environment where they it can be received, transformed, managed, stored. It also defines access for multiple repositories and use cases (BI tools, operational applications. This is made possible by continuous metadata analytics to build the web layer. It integrates data processing processes and many sources, types, and locations of data.

Differences Data Fabric from the standard data integration ecosystem:

Enhanced data catalog that includes and analyzes all types of metadata;
Knowledge graphs showing relationships between data elements (concepts, objects, events, etc.);
Metadata activation – using machine learning active metadata management allows to create and process metadata on a large scale;
Data preparation and ingestion – support for all approaches to data preparation and delivery including ETL, ELT, data streaming, application integration, data virtualization;
Recommendation engine – constant analysis, study and formation of recommendations and forecasts in relation to integration and data management;
DataOps – bringing the DevOps team together with engineers and data scientists to meet needs.

The Data Fabric architecture depends on individual data needs and queries of business. However, there are 6 main levels:

Data management (ensuring management and security processes);
Receiving data (determining the relationship between structured and unstructured data);
Data processing (only relevant data extraction);
Data orchestration (data cleansing, transformation and integration);
Data discovery (identifying new ways to integrate different data sources);
Access to data (the ability of users to explore data using BI tools).

When implementing Data Fabric, you need to consider:

collection and analysis of all metadata types;
transformation of passive metadata into active ones;
creation and management of knowledge graphs;
providing a solid basis for data integration.

QOps work principles

The main problem that QOps solves is the difficulty of applying the GIT version control system to the final development file, that contains data, data model, data loading code, codes of all charts and objects for visualization.

What is GIT?

GIT is a version control system that helps track changes made to a codebase, identify the user who made them, and recover deleted or modified code. GIT is one of the most popular open-source version control systems at the moment and is very easy to use. GIT is also completely free software and supports different operating systems (Vac, Windows, Linux, Solaris).

The benefits of GIT are:

distributed version control system;
the ability to have many independent code branches;
atomicity of operations;
storing data in .git catalog;
using a data model that ensures the cryptographic integrity of the containing repository;
the ability to format commits and preview them before applying them within the index.

How to use GIT in the development of Qlik applications?

The problems of applications parallel development and tracking changes in the source code, development process involving more than 1 developer, manual deletion of code and charts are solved by GitLab and GitHub. However, if you put the Qlik Sense/QlikView file on GitLab, this will not work. This file must be divided into separate parts: code, chart settings, variables, extensions etc. Then these parts should be put back into a file. QOps helps automate the process of application disassembling and reassembling.

How does QOps work?

QOps extracts absolutely all source codes from the files of developed applications. Thus, the extracted codes can be placed in the repositories of any version control systems. Modern version control systems have advanced tools for branches, commits, and merges. This allows multiple developers to work on a project in parallel.

QOps allows to automatically build an application from a local or remote repository. Source codes are stored separately from data, that provides flexibility in the development of complex projects. This makes it possible to set up automatic testing and deployment of the application. This, in turn, allows to avoid accidental errors during deploying applications to production, as well as reverting to any previous version of the application, if necessary.

QOps allows to:

keep application source code separate from data;
compare and analyze changes in the source code of all application objects;
collect and implement the developed functionality into the application in parts (they can be created by different developers);
set up a pipeline of automatic checks and automatic deployment of the application;
quickly find errors and undo incorrect changes.

More information about QOps here

What is QOps and what problems does it solve?

Qlik is a provider of business intelligence and data visualization applications. Its main products are QlikView and Qlik Sense, which have the same logic and analysis algorithm, but a different user interface. QlikView can run in its own environment with custom charts and visualizations and through a web interface. Qlik Sense is focused on web application development.

The Qlik data-driven business intelligence platform features associative connectivity and available graphical capabilities. It allows users to make calculations very quickly by storing compressed information in RAM and quickly accessing it. Qlik allows to connect various data sources from simple files to modern databases, and also provides for the development of individual connectors to the data source.

Qlik Sense is both a development environment and an off-the-shelf application. The result of the development is a large file that includes data, a data model, data loading code, codes for all charts and objects for visualization. Qlik supports built-in SVN and TFS version control. However, there is no way to apply the GIT version control system. This problem is solved by QOps.

QOps is a console tool that removes the complexities of version control and automates Qlik applications development and deployment. The product works in PowerShell, integrates with Qlik, GIT, GITLab/GitHub.

The main problems that QOps helps to solve:

Impossibility of parallel development of applications and tracking changes in the source code;
Slow development of large applications due to the inability to use the usual version control systems;
Complexity of development process involving more than 1 developer: each developer makes all changes to their own copy of the file. In order to add into production changes made by both developers, one of the developers will have to duplicate their changes in a common file;
The need to manually remove the code and charts or find a backup file if there is a need to return to a specific version;

You can get additional information by filling out the form at the link

Visual analytics – definition and benefits

Currently, one of the most promising and rapidly developing areas is visual analytics. Its advantage lies in the ability to work with large datasets, combining graphical visualization and powerful analytical calculations.

Visual analytics is the process of using sophisticated tools and methods to analyze data using visual data representations in the form of graphs, charts, and maps. This allows users identify patterns and insights that help them make better data-driven decisions.

Visual analytics is not just a graphic data representation and should not be confused with data visualization. State-of-the-art interactive visual analytics makes it easy to combine data from multiple sources and in-depth data analysis right in the visualization. The use of artificial intelligence and machine learning algorithms generate recommendations for a more detailed data study. The main task of this tool is to turn large data amounts into successful business ideas.

Visual analytics advantages:

Ability to systematize, share information on KPI and track progress using interactive reports and dashboards;
Ability to make decisions faster by working and understanding data and its meaning;
Simplify the process of data exploration with self-service tools that allow users to interact with data, identify hidden relationships and patterns without the intervention of IT specialists;
Increasing information literacy by making it easier to work with data.

Key recommendations for the qualitative use of visual analytics:

Definition of a specific goal and questions that need to be answered using visual analytics;
Integration and data management (combining and replicating data from different sources, converting to standardized formats);
Choosing the right visual technique;
Explore data visualization practices.

The large number of vendors that offer visualization features as part of the software makes it difficult to choose the right tool.

Modern data analysis tools include the following features:

Integration and data management;
Free research data in any direction;
Visual effects that allow even novice users to create their own analytical views;
Ability to analyze data from any sites, portals, applications and business processes.

More possibilities with Qlik AutoML

Machine learning applications have reached ubiquity (from solving health problems to choosing music or products). Business is also an active user of machine learning tools.

Today, companies, including, which are interested in expanding the capabilities of their teams and specialists who know how to work with data. For example, a BI engineer who participates in the analytics process could develop features, train, automatically select a reliable model and help deploy it without the involvement of data scientists and machine learning. Qlik AutoML helps companies maximize their data and analytics strategy.

Qlik AutoML is an automated machine learning platform designed for analysts to create models, forecasts, and conduct business scenario testing. A simple, no-code user interface makes it easy to work with data by identifying its key drivers, make predictions with a full data understanding, publish and integrate models into Qlik Sense dashboards for interactive analysis.

This tool identifies key drivers of historical data and builds machine learning models using the best algorithms. Qlik AutoML allows to:

Evaluate and rank machine learning models;
Determine the most efficient model for a particular dataset;
Determine the importance of the feature to identify key business drivers;
Easy to customize and improve the model;
Deploy the model to create predictions;
Publish forecast data;
Integrate the model with Qlik Sense for the purpose of detailed analysis;
Test different «what if» scenarios;
Plan decisions and actions.

Using Qlik AutoML:

Sales (sales funnel, predicting wins and losses, customer churn and retention, customer search, targeting);
Marketing (demand forecasting, customer value indicator, selection of the optimal offer);
Finance (investment optimization, cost management, risk management);
HR (employee retention, employee satisfaction, recruiting, candidate profiling);
IT (software usage, software licensing, infrastructure performance forecast);
Supply chain (inventory management, inventory shortage forecast, supply chain performance, transport optimization);
Support (forecasting calls to the helpdesk, preventive maintenance).

What is ESG?

Nowadays, it is important to create a sustainable business in accordance with the rules of sustainability and sociality. Until 2030, the UN has identified 17 goals and 169 targets for sustainable development, which are a benchmark for modern business. Modern business strategy must include ESG goals.

ESG (environment, social, governance) – is a management principle that helps to achieve the involvement of the company in solving environmental, social and management problems. Such non-financial indicators allow to explore business activities through social and ethical relationships with the environment.

The approach to implementing sustainability principles may differ depending on the business area, its profitability and scale, the number of employees, etc. However, the basis and principles of ESG are common. Small and medium-sized businesses often postpone the transition to sustainable development, explaining this by the fact that they don’t have enough funds for implementation. Also, the non-obviousness and unpredictability of the transition effect stops the leadership on the transition path. However, experts argue that sustainable development will bring benefits.

ESG – key factors that determine the level of sustainability and ethical impact of an investment in a company. Investors use ESG standards to monitor the company for: nature conservation (environmental), management of relationships with employees, partners, customers, society (social), remuneration, audit, internal control (corporate). There are financial institutions and funds that monitor ethically sound companies and check them before investing.

ESG includes

Environmental factors:

Waste products
Pollution
Carbon footprint
Energy use
Water use
Use of natural resources

Social factors:

Prejudgment and discrimination
Diversity, equity and inclusion
Employee health (mental and physical)
Charity
Working conditions
Corporate culture

Corporate Governance:

Fight against corruption
Salary
Transparency
Diversity in management
Tax strategy

ESG Benefits:

Attracting and retaining effective employees. Modern specialists pay attention to working conditions, indifference to nature and the environment, social values, corporate culture, etc.;
Increasing brand value, reputation and customer loyalty;
Improving the financial performance of the business, its profitability, operational efficiency, introduction of innovations, risk management;
Increase attractiveness for investors.

ESG data is data about the environmental, social and governance company performance. These include: the amount of water consumed, the amount of CO2 emissions, the number of employees, the gender ratio, etc.

Accurate, transparent and actionable data are essential to create and effectively monitor a sustainable development strategy. Many companies store their environmental impact information in spreadsheets. This complicates and extends the process of extracting reports and analysis. However, Qlik simplifies data analytics and connects the sources of sustainable development into a single dashboard. This makes impact as easy to measure as sales, profits, and other KPIs.