Finance big data center:Catalog of Financial Data Information Resources, Data Standard Storage, and

Finance big data center:
Catalog of Financial Data Information Resources, Data Standard Storage, and Big Data Asset Planning Scheme
catalogue
1 Overview 1
2 Planning ideas 1
2.1 Overall logical relationship 1
2.1.1 Overall structure of data resource directory 2
2.1.2 Technical Framework for Data Resource Catalog 4
2.2 Overall Technical Roadmap 5
3 Functional Design 5
3.1 Resource Catalog Management 6
3.1.1 Functionality for Resource Providers 8
3.1.2 Functions for Resource Managers 9
3.1.3 Functions for Resource Users 10
3.1.4 Multidimensional Resource Catalog Display 11
3.2 ETL Function 13
3.2.1 metadata management 13
3.2.2 Data Collection 15
3.2.3 Data quality inspection 16
3.2.4 Data Conversion 17
3.2.5 ETL Task 18
3.2.6 Task Scheduling 19
3.2.7 ETL Log20
3.2.8 ETL monitoring 20
3.3 Data collection and ETL technical support functions 21
3.3.1 Data Collection Configuration 21
3.3.2 Data Collection 22
3.3.3 Data Confirmation 23
3.3.4 Data Review 23
3.3.5 Data Conversion 23
3.4 Data Query, Statistics, and Comparison 25
3.4.1 Classification Data Query 25
3.4.2 Comprehensive Query of Unit Data 30
3.4.3 Revenue and expenditure statistics query 31
3.4.4 Regional Ranking Query 34
3.4.5 Years of Comparison Query 36
4. Outlook on Data Analysis and Mining Themes 40
5 Big data assetization 42
5.1 Value scale measurement standards for big data assets 42
5.2 Data Asset Management in the Era of Big Data 43
5.2.1 Making Data an Asset 44
5.2.2 Preservation and appreciation of data assets 46
5.2.3 Promoting the construction of data trading markets and accelerating the process of data assetization 48
1 Overview
Based on the current application status of data analysis products in various regions, in order to effectively promote the application of big data and implement data analysis, it is urgent to build a financial data resource catalog based on the "Golden Finance Project" application support platform. On the one hand, it can truly manage various types of financial data resources in an orderly manner, achieving "clear resources and convenient access", and on the other hand, it can truly promote the implementation of data application planning schemes.
This plan aims to clarify the position, overall structure, and functional design of the financial data resource catalog in the entire data analysis product system.
2 Planning ideas
2.1 Overall logical relationship
Firstly, clarify the logical relationship between the data resource directory and the overall data analysis application:
Figure 2-1 Logic structure diagram of data resource directory and standard storage in overall data analysis application
Figure 2-1 Data Resource Catalog and Standard Storage (Distributed Scheme)
The construction of a data resource directory needs to be based on the "Financial Basic Data Specification" and "Financial Standard Storage Specification". After obtaining data from the production system according to certain cleaning and conversion rules, it should form standard storage that meets the specifications for further thematic analysis resource library use, ultimately achieving deep level data analysis and data mining.
The production system data source can be an established disaster recovery backup pool, the original production library, or even non platform based third-party systems or external files and other resources. After the data resource directory is established based on standard storage, it can also provide preliminary data query and comparison topics such as multi region and multi year revenue and expenditure statistics, regional rankings, and multi-year comparisons. Of course, further topic analysis and deep mining require further abstract classification and extraction based on the formed standard storage data warehouse.
2.1.1 Overall structure of data resource directory
The data resource directory system mainly consists of a data resource directory service system, which also has a supporting environment of software, hardware, network, as well as the construction of standards and management norms and security guarantees.
The construction of the entire data resource directory is an integrated and unified management architecture that follows the financial basic data specifications and standard storage specifications, as well as unified front-end code management. The entire data resource directory system structure is reflected in a three-level system architecture of provinces, cities, and counties (districts).
Figure 2-2 Hierarchical Architecture of Data Resource Catalog
This three-layer system architecture establishes a data resource directory management center from top to bottom, and each level of directory management center constructs and manages the data resource directory of the local and lower level finance. During specific implementation, the system hierarchy will vary depending on the financial level. Overall, the main responsibilities of the provincial data resource management center are to be responsible for the overall plan for the development and utilization of data resources throughout the province, formulate rules and scope for information provision, exchange, and sharing, and unify the management of the data resource directory of all levels of finance in the province. The construction plan for other financial level data resource catalogs shall refer to the provincial implementation.
In addition to building the local financial data resource directory, each level of directory management center also needs to fully consider upward compatibility, especially the compatibility of some key contents such as directories and core metadatabases for exchange. After the construction of the local data resource directory is completed, as a node of the superior financial resource directory system, metadata registration and other work should be carried out to enable the local resource directory to serve as a branch of the superior financial data resource directory tree.
The financial data resource directory system is an independent and complete directory system, and its overall structure can be summarized into three parts: data resources, data resource directory system, and data resource directory exchange system.
Data resources are the foundation of the construction of the financial data resource directory system. Its meaning refers to the information resources collected, processed, and used in the financial business process, including the information resources generated and generated within the application support platform in the financial business process, as well as other business system databases and external file resources controlled by the finance department.
The data resource directory system and the data resource directory exchange system are closely inseparable. The definition of a directory is to use "name address mapping" to allow dynamic connections between objects and their locations. The concept of financial data resource catalog is more similar to the classification catalog concept used in libraries. Using core metadata as the main description method, arrange the core metadata of financial information resources in an orderly manner according to the information resource classification system or other methods. By using a directory, one can accurately understand and grasp the basic overview of information resources, discover and locate the necessary financial information resources.
The data resource directory exchange system provides a channel to organically combine related superior and subordinate financial information resources, which is the process of information resources moving from one physical location to another through a network system. The exchange methods are relatively broad, and methods such as email transmission, file transmission, and message transmission can all achieve resource exchange. Its focus is on the integration and unified management of information between upper and lower levels of finance in the process of data resource directory construction.
The directory system and exchange system are both relatively independent and independent platforms, as well as two important components in the development and utilization of financial information resources. After accurately locating the location of resources and services through the data resource directory system and accurately understanding the overview of information resources, information resources can be obtained through the exchange system within the authorized scope. On the other hand, the formation of a data resource directory system requires the exchange of resource description information through an exchange system. The two systems complement each other, the directory system is a prerequisite for the exchange system, and the exchange system is the implementation carrier of the directory system.
2.1.2 Technical framework for data resource directory
The technical framework of the data resource directory system mainly consists of two types of directories, namely the public resource directory and the exchange service directory. The public resource directory mainly targets data and provides information on the data resource directory for discovering and locating information resources; The exchange service directory mainly targets data exchange service resources, providing directory information such as exchange service addresses for discovering and locating exchange services.
Figure 2-3 Technical Framework for Data Resource Catalog
There are three types of roles in the construction and use of directory systems: information providers, information users, and directory information managers. The information provider is responsible for cataloging and registering information, ensuring the accuracy and real-time nature of the cataloging information. Information users can query the public resource directory and exchange service directory through the application system to discover the required information. The directory information manager is responsible for establishing and managing resource directories, and ensuring the security and maintenance of directory information.
2.2 Overall technical roadmap
Overall, it utilizes the data dictionary, standard storage, file services, and other components of the multi year and multi finance platform 2.0 to support the storage and management of structured and unstructured data, making the platform truly the manager of financial data centers.
When utilizing data, it is also necessary to use distributed, search engine, data mining and other related technologies according to the requirements of the analysis topic to achieve data analysis and utilization.
3 Functional Design
The definition of the financial data resource directory system is based on metadata as the core, with the financial data classification table and thematic vocabulary as the control vocabulary. It organizes financial data resources in a network, meeting the requirements of managing, identifying, locating, discovering, evaluating, and selecting financial information resources from multiple perspectives such as classification, theme, and application.
Establish a data resource management mechanism by defining a data resource directory system. The data resource directory system should have the following functions:
Establish core metadata to describe resources, identify and describe all data resources, including their names, descriptions, and other information;
Establish a classification system for describing data resources, and based on this, classify all data resources;
Associated with resource file storage, describe the access information of resources, including permissions, access methods, addresses, etc. Based on this, obtain resource entities through the information recorded in the resource directory.
Based on the overall data analysis application logic, the establishment of the data resource catalog system can be roughly divided into three parts: data collection and ETL technology support function, data resource catalog management function, and comparison theme display function. The overall functional structure diagram is as follows:
Figure 2-7 Functional structure diagram of data resource directory
3.1 Resource Catalog Management
The management architecture of the data resource directory system includes three roles and six activities for the use and management of the data resource directory system. The three roles are the provider, user, and manager of the data resource directory. The six activities include planning, cataloging, registering, publishing, maintaining, and querying.
The information provider is responsible for planning and cataloging the content of the financial data resource directory, registering the directory content with the manager and updating it, setting usage permissions for the content of this financial data resource directory, and providing information resources related to the directory content.
The information manager is responsible for the registration, publication, and system maintenance of the content of the data resource directory. They allocate, manage, and use resource identifiers in accordance with national standards and relevant management methods, and provide query services for the content of the data resource directory.
Information users can query the content of the data resource directory and use the obtained directory content within the authorized scope.
Figure 2-4 Data Resource Catalog Management Activity Diagram
The business process of the data resource directory system includes resource cataloging, resource registration, resource publishing, and provides functions such as resource access and resource maintenance. Cataloging provides editing functions for core metadata of public resources and exchange service resources; Registration refers to the provision of public resource core metadata and exchange service information resource core metadata registration by data resource directory providers to data resource directory managers; Publishing refers to managers publishing the content of the core metadata database of financial information resources into the system through a directory server; Query refers to providing a standard calling interface for application systems, supporting the query of common resource core metadata and exchange service resource core metadata.
Figure 2-8 Business Process of Data Resource Catalog System
3.1.1 Functionality for resource providers
3.1.1.1 Resource Cataloging
Business personnel of various financial organizations conduct business and resource sorting, and extract features from the information resource library based on unified standards and specifications to form metadata for data and service resources.
(1) Extract relevant feature information from information resources according to the core metadata standards of information resources, and on this basis, appropriately add the required metadata in conjunction with specific businesses to form information resource metadata.
(2) According to the information resource identifier coding standard, apply for the identifier coding of information resources from the directory management agency and assign values to the identifier information in the metadata.
(3) Assign values to the classification information in metadata according to the information resource classification standards.
3.1.1.2 Resource registration application
The resource provider applies for registration of the metadata formed by cataloging to the management agency of the directory center through the metadata registration system. The local resource provider submits data resource metadata, and the subordinate directory center registers the financial directory service address and core metadata of that level with the superior directory center.
3.1.2 Functions for Resource Managers
3.1.2.1 Resource registration review
The management agency confirms whether the format and content of the information resource metadata submitted by the provider meet the standard requirements, and returns the metadata that has not passed the review to the provider for modification. For the approved metadata, generate the core metadata of data resources and service resources, and form a formal directory.
3.1.2.2 Resource Release
The audited metadata is entered into the metabase, and the directory center management agencies at all levels automatically or manually extract core metadata according to the prescribed core metadata standards and place it in the standard storage of the local directory center as the basis for directory display. The lower level directory center extracts the core metadata of the current level center and publishes it to the core metadata database of the higher-level directory center, and publishes the address information of the directory center, including the name and network location identifier of the directory center. The directory center management agencies at all levels generate, publish, and maintain directory content based on the registered metadata.
Publishing is the process in which the manager categorizes the incoming resources based on the cataloging rules of each category, publishes the latest metadata, and exports the successfully registered financial data resource directory information from the financial information resource metadatabase through the directory service system according to the requirements of the directory service. This generates resource directory entries and imports them into the data resource directory database for users to access.
3.1.2.3 Resource permission management
Assign different access permissions to different users to ensure that they can correctly access resource directory information and protect the security of directory system directory information. According to the classification of users and the security management requirements of the directory service system, users are divided into three categories: the first category is public users who can access public data resource directory information; The second type is financial users who can access public data resource directory information and limited exchange resource directory information; The third category is directory service system managers, responsible for managing the directory service system, updating and maintaining directory information, and other related work. Set several types of permission levels in these three roles, which correspond to the six security levels of resources (whole society, local unit, restriction, secret, confidential, top secret). When users use the directory service system, the system will determine the operations and filters that users can perform based on their permissions, prevent illegal operations, and filter the results of directory queries performed by users based on their permissions and resource confidentiality attributes in directory entries.
3.1.2.4 Resource maintenance
Establish an information resource metabase, core metabase, and directory, and perform regular updates, backups, and restores, as well as inbound and outbound operations; Monitor the operation of the directory server; According to the query log, count the number of times the system is accessed, and analyze the number of times different information resources are queried.
maintenance