Quản lý và Trực quan hóa Dữ liệu: Chương 1 - Giới thiệu về quản lý và trực quan hóa dữ liệu

Chapter 1 Introduction to data management and visualization

How big is big data?

Data science: The 4th paradigm for scientific discovery

Big data in 2008

Big data sources

• E-commerce

• Social networks

• Internet of things

• Data-intensive experiments (bioinformatics, quantum

physics, etc)

Data is the new oil

Big data 5'V

Big data is a term for data sets that are so large or complex that traditional data processing application software is inadequate to deal with them (wikipedia)

Big data – big value

source: wipro.com10

Introduction to data management

What is Data Management

• Data management is the development and execution of architectures, policies, practices and procedures in order to manage the information lifecycle needs of an enterprise in an effective manner

Poor Data Management

• 94% of companies suffering from a catastrophic data

loss do not survive – 43% never reopen and 51% close within two years. (University of Texas)

• 7 out of 10 small firms that experience a major data loss go out of business within a year. (DTI/Price Waterhouse Coopers)

• 50% of all tape backups fail to restore. (Gartner)

• 25% of all PC users suffer from data loss each year

(Gartner)

Why Data Management: Foundation to Advance Science • Data is a valuable asset – it is expensive and time

consuming to collect

o maximize the effective use and value of data and information

assets

o continually improve the quality including data accuracy, integrity,

integration, timeliness of data capture and presentation, relevance and usefulness

o ensure appropriate use of data and information o facilitate data sharing o ensure sustainability and accessibility in long term for re-use in

science

• Data should be managed to:

A new image processing technique reveals something not before seen in this Hubble Space Telescope image taken 11 years ago: A faint planet (arrows), the outermost of three discovered with ground-based telescopes last year around the young star HR 8799.D. Lafrenière et al., Astrophysical Journal Letters

s r e t t e L

J p A

“Planet hidden in Hubble archives” Science News (Feb. 27, 2009)

, . l a t e e r è n e r f a L .

“The first thing it tells you is how valuable maintaining long-term archives can be. Here is a major discovery that’s been lurking in the data for about 10 years!” comments Matt Mountain, director of the Space Telescope Science Institute in Baltimore, which operates Hubble.

“The second thing its tells you is having a well calibrated archive is necessary but not sufficient to make breakthroughs — it also takes a very innovative group of people to develop very smart extraction routines that can get rid of all the artifacts to reveal the planet hidden under all that telescope and detector structure.”

Data Management Facilitates Sharing and Re-use…

Where a majority of data end up now…

Imagine if data were more accessible….

Data Life Cycle

Plan

Analyze

Collect

Integrate

Assure

Discover

Describe

Preserve

Planning

• What kind of data will be collected? • Which methods will be used (sensors, samples, etc.)? • What data formats/standards are appropriate? • How will the data be used? • How will you share the data? • Will your methods satisfy

• Funding requirements • Policies for access, sharing, reuse

• Budget – most of the time tihis is overlooked!

• Consider data management before you collect data

• Formal document

• Output

Information about data and data format

• Types of data that will be produced (e.g. experimental,

observational, raw or derived, physical collections, models, images, etc.) • Volume of data • When, where and how the data will be acquired (e.g.

methods, instruments)

• How the data will be processed (e.g. software, algorithms

and workflows)

• File formats (e.g. csv, tab-delimited or naming conventions) • Quality assurance and control procedures used • Other sources of data (e.g. origins, relationship to one’s

data and data integration plans)

• Approaches for managing data in the near-term (e.g.

version control, backing up, security and protection, and responsible party)

Metadata content and format

• Metadata that are needed

• How metadata will be created or captured (e.g. lab

notebooks, autogenerated by instruments, or manually created)

• Format or standard that will be used for the metadata

Policies for access, sharing and re-use

• Requirements for sharing (e.g. by research sponsor or

host institution)

• Details of data sharing (e.g. when and how one can

gain access to the data)

• Ethical and privacy issues associated with data sharing

(e.g. human subject confidentiality or endangered species locations)

• Intellectual property and copyright issues Intended

future uses for data

• Recommendations for how the data can be cited

Long-term storage and technical data management • Identification of data that will be preserved

• Repository or data centre where the data will be

preserved

• Data transformations and formats needed (e.g. data

centre requirements and community standards) Identification of responsible parties

Budget

• Anticipated costs (e.g. data preparation and

documentation, hardware and software costs, personnel costs and archive costs)

• How costs will be paid (e.g. institutional support or

budget line items)

Responsibilities

• Who is responsible for what?

Collect

• What are some ways that we produce data?

• Experiments, observations, samples,

• Varying frequency, temporal and spatial coverage

• Transcribing notebooks into digital forms • Automated processing of data into a database

• Data collection includes data entry

Assure

• Standard data entry forms • Pre-specification of formats, units, etc.

• Strategies for preventing errors from entering datasets

• Activities to ensure quality during collection • Standard field and laboratory procedures • Automated rannge checks for sensor data

• Activities to clean collected data • Common to sensor data streams • Dependent upon variable and sensor • Graphical and statistical summaries

Describe

• What metadata are needed? • What format for the metadata?

• Metadata

• Documentation and reporting of data

• What is it critical to know about the data?

• Contextual details

• Description of temporal and spatial details,

instruments/sensors, methods, units, files, etc.

Preserve

• What will be preserved • Where will it be preserved • Backup, version control?

• How are you preserving your data?

• Policies for access, sharing, and reuse

Discover

• Most data are not easily discoverable • Encapsulated in databases or files • Formats not compatible with web indexing technologies

• Highly curated data, well described via structured metadata • Standards for data and metadata formats

• Conditions for effective data discovery

Integrate & Analyze

• Combining data from different sources • Creating a unifying view of the data • Ovecoming heterogeneity

• Integration

• To find out insightful values from data

• Analysis

Data Management Plan

• Information about data and data format

• Metadata content and format

• Policies for access, sharing and re-use

• Long-term storage and technical data management

• Budget

• Responsibilities

Summary

• The data deluge has created a surge of information

that needs to be well-managed and made accessible.

• The cost of not doing data management can be very

high.

• Knowknow of best practices and tools associated with

the data lifecycle to manage your data well.

• Many benefits are associated with the act of managing data, including the ability to find, access, understand, integrate and re-use data.

Summary, con’t

• Well-organized • Documented • Preserved • Accessible • Verified as to Accuracy and validity

• If data are:

• High quality data • Easy to share and re-use in science • Citation and credibility to the researcher • Cost-savings to science

• Result is:

Data Visualization

• Data visualization is a general term that describes any effort to help people understand the significance of data by placing it in a visual context. Patterns, trends and correlations that might go undetected in text-based data can be exposed and recognized easier with data visualization software.