MetaData Based MetaProgramming System (MDBMPS)

最新推荐文章于 2022-07-25 11:36:16 发布
最新推荐文章于 2022-07-25 11:36:16 发布
阅读量1w 收藏 1
点赞数 1
分类专栏： 架构 c/c++ 文章标签： system database templates network attributes table
c/c++ 同时被 2 个专栏收录
76 篇文章 0 订阅
订阅专栏
架构
28 篇文章 0 订阅
订阅专栏

MetaData Based MetaProgramming System (MDBMPS)

Author(s):The Add 1 to COBOL Team
Date:2009-01-26 15:46
Revision:318
Status:Experimental - Not for public release
Description:Specification for a metadata based metaprogramming system and the follow throughs.

Table of Contents

1   Abstract

This specification defines MetaData Based MetaProgramming System (MDBMPS), a concept for all coding languages.

2   Introduction

Computer software is [usually] really nothing more than a series ofinstructions in a machine readable format generally called"machine language" (A great deal of this infomation will be familiar toMIS/IT personnel. It is included only for completness, though in greatlyoversimplified format.). Machine language can be thought of as hundreds,thousands, millions, billions and even more of a series of 1's and 0's.Originally programmers created applications directly in machine language,a tedious approach for sure. Further complicating matters is that variouscomputer system have their own machine language.

As time went on, programmers began to notice repeating patterns in the codethat they created. So they would save and then reuse these patterns.Eventually from this process, the concept of the compiler was born.A compiler takes instructions written in a human readable language andconverts it or compiles it, into machine language, then the program canbe run many times thereafter. An interpreter functions similarly, exceptthat it translates the program into machine language each time it is run.A further advantage of compilers and interpreters is that the humanreadable version of a program is often transferable from one kind ofcomputer system to another. Programs became portable.

Compilers and interpreters dramatically increased the efficiency ofcreating software programs. Still, Systems Analysts, Programmers, andCoders are always looking for tools which increase their efficiency evenmore, while also reducing errors. As occured when programming inmachine language, we have noticed that we'll write repeating patterns inhuman readable programming languages. So often we will save these assnippets for reuse throughout the many programs that we will design.Alternatively, a template can provide a skeleton for an entire program,so that the programmer only has to fill in certain parts of the templatewith code unique to the application he is working on (perhaps withsnippets as well).

Two of the most efficient tools at a programmer's disposal are programsdesigned for metaprogramming and for managing metadata.Metaprogramming is simply writing software, which writes software.Metadata is data, or information, about data, such as the layout ofdata structures. While there have been efforts to combine these twoconcepts, often the resulting systems stop just short of a complete,cohesive unit.

The MetaData Based MetaProgramming System (MDBMPS) proposes to take the extra steps necessary to create afully workable and complete unification of these two processes. TheMDBMPS will be a system of tools and procedures which will enablea programmer to create his own templates and snippets, use those of othersand combine these in ways which will speed and ease his development cycle.

This paper will explain MetaData, MetaProgramming, CASE Tools, Templates,Snippets, Embedded Code and other factors needed to begin coding theMDBMPS. At the paper's end, a skilled MIS/IT individual or teamshould have enough information to begin formally planning the system'sconstruction.

3   Meta Data

Metadata (meta data, or sometimes metainformation) is "data about other data",of any sort in any media. An item of metadata may describe an individualdatum, or content item, or a collection of data including multiple contentitems and hierarchical levels, for example a database schema. In dataprocessing, metadata is definitional data that provides information aboutor documentation of other data managed within an application or environment.The term should be used with caution as all data is about something, andis therefore metadata.

For example, metadata would document data about data elements or attributes,(name, size, data type, etc) and data about records or data structures(length, fields, columns, etc) and data about data (where it is located,how it is associated, ownership, etc.). Metadata may include descriptiveinformation about the context, quality and condition, or characteristicsof the data. It may be recorded with high or low granularity

3.1   Purpose

Metadata provides context for data.

Metadata is used to facilitate the understanding, characteristics, andmanagement usage of data. The metadata required for effective datamanagement varies with the type of data and context of use. In a library,where the data is the content of the titles stocked, metadata about a titlewould typically include a description of the content, the author, thepublication date and the physical location.

3.2   Examples of metadata

3.2.1   Book

Examples of metadata regarding a book would be the title, author, date ofpublication, subject, a unique identifier (such an International StandardBook Number), its dimensions, number of pages, and the language of thetext.

3.2.2   Photograph

Metadata for a photograph would typically include the date and time at whichit was taken and details of the camera settings (such as focal length,aperture, exposure). Many digital cameras record metadata in exchangeableimage file format (EXIF).

3.2.3   Audio

A digital audio file, such as an MP3 of a song, might include the albumname, song title, genre, year, composer, contributing artist, track numberand album art.

3.2.4   Web page

The HTML used to mark-up web pages allows for the inclusion of a varietyof types of meta data, from simple descriptive text, dates and keywords tohighly-granular information such as the Dublin Core and e-GMS standards.Pages can be geotagged with coordinates. Metadata may be included in thepage's header or in a separate file. Microformats allow on-page data to bemarked up as meta data. The Hypertext Transfer Protocol used to link webpages also includes metadata.

3.3   Levels

The hierarchy of metadata descriptions can go on forever, but usuallycontext or semantic understanding makes extensively detailed explanationsunnecessary.

The role played by any particular datum depends on the context. For example,when considering the geography of London, "E8 3BJ" would be a datum and"Post Code" would be metadatum. But, when considering the data management ofan automated system that manages geographical data, "Post Code" might be adatum and then "data item name" and "6 characters, starting with A–Z" wouldbe metadata.

In any particular context, metadata characterizes the data it describes, notthe entity described by that data. So, in relation to "E8 3BJ", the datum"is in London" is a further description of the place in the real world whichhas the post code "E8 3BJ", not of the code itself. Therefore, although itis providing information connected to "E8 3BJ" (telling us that this is thepost code of a place in London), this would not normally be consideredmetadata, as it is describing "E8 3BJ" as a place in the real world and notas data.

3.4   Definitions

3.4.1   Etymology

Meta is a classical Greek preposition and prefix conveying the followingsenses in English, depending upon the case of the associated noun: among;along with; with; by means of; in the midst of; after; behind. Inepistemology, the word means "about (its own category)"; thus metadata is"data about the data".

3.4.2   Varying definitions

The term was introduced intuitively, without a formal definition. Becauseof that, today there are various definitions. The most common one is theliteral translation:

  • "Data about data are referred to as metadata."

Example: "12345" is data, and with no additional context is meaningless.When "12345" is given a meaningful name (metadata) of "ZIP code", one canunderstand (at least in the United States, and further placing "ZIP code"within the context of a postal address) that "12345" refers to the GeneralElectric plant in Schenectady, New York.

As for most people the difference between data and information is merely aphilosophical one of no relevance in practical use, other definitions are:

  • Metadata is information about data.
  • Metadata is information about information.
  • Metadata contains information about that data or other data

There are more sophisticated definitions, such as:

"Metadata is structured, encoded data that describe characteristics ofinformation-bearing entities to aid in the identification, discovery,assessment, and management of the described entities."

And,

"Metadata is a set of optional structured descriptions that are publiclyavailable to explicitly assist in locating objects."

These are used more rarely because they tend to concentrate on one purposeof metadata - to find "objects", "entities" or "resources" - and ignoreothers, such as using metadata to optimize compression algorithms, or toperform additional computations using the data.

The metadata concept has been extended into the world of systems to includeany "data about data": the names of tables, columns, programs, and the like.Different views of this "system metadata" are detailed below, but beyondthat is the recognition that metadata can describe all aspects of systems:data, activities, people and organizations involved, locations of data andprocesses, access methods, limitations, timing and events, as well asmotivation and rules.

Fundamentally, then, metadata is "the data that describe the structure andworkings of an organization's use of information, and which describe thesystems it uses to manage that information". To do a model of metadata is todo an "Enterprise model" of the information technology industry itself.

3.5   Metadata and Markup

In the context of the web and the work of the W3C in providing markuptechnologies of HTML, XML and SGML the concept of metadata has specificcontext that is perhaps clearer than in other information domains. Withmarkup technologies there is metadata, markup and data content. The metadatadescribes characteristics about the data, while the markup identifies thespecific type of data content and acts as a container for that documentinstance. This page in Wikipedia is itself an example of such usage, wherethe textual information is data, how it is packaged, linked, referenced,styled and displayed is markup and aspects and characteristics of thatmarkup are metadata set globally across Wikipedia.

In the context of markup the metadata is architected to allow optimizationof document instances to contain only a minimum amount of metadata, whilethe metadata itself is likely referenced externally such as in a schemadefinition (XSD) instance. Also it should be noted that markup providesspecialised mechanisms that handle referential data, again avoidingconfusion over what is metadata or data, and allowing optimizations. Thereference and ID mechanisms in markup allowing reference links betweenrelated data items, and links to data items that can then be repeated abouta data item, such as an address or product details. These are then allthemselves simply more data items and markup instances rather than metadata.

Similarly there are concepts such as classifications, ontologies andassociations for which markup mechanisms are provided. A data item can thenbe linked to such categories via markup and hence providing a cleandelineation between what is metadata, and actual data instances. Thereforethe concepts and descriptions in a classification would be metadata, but theactual classification entry for a data item is simply another data instance.

Some examples can illustrate the points here. Items in bold are datacontent, in italic are metadata, normal text items are all markup.

The two examples show in-line use of metadata within markup relating to adata instance (XML) compared to simple markup (HTML).

A simple HTML instance example
<span style="normalText">Example</span>
An XML instance example with metadata
<PersonMiddleName nillable="true">John</PersonMiddleName>

Where the inline assertion that a person's middle name may be an empty dataitem is metadata about the data item. Such definitions however are usuallynot placed inline in XML. Instead these definitions are moved away into theschema definition that contains the metadata for the entire documentinstance. This again illustrates another important aspect of metadata in thecontext of markup. The metadata is optimally defined only once for acollection of data instances. Hence repeated items of markup are rarelymetadata, but rather more markup data instances themselves.

3.6   Hierarchies of metadata

When structured into a hierarchical arrangement, metadata is more properlycalled an ontology or schema. Both terms describe "what exists" for somepurpose or to enable some action. For instance, the arrangement of subjectheadings in a library catalog serves not only as a guide to finding books ona particular subject in the stacks, but also as a guide to what subjects"exist" in the library's own ontology and how more specialized topics arerelated to or derived from the more general subject headings.

Metadata is frequently stored in a central location and used to helporganizations standardize their data. This information is typically storedin a metadata registry.

3.7   Difference between data and metadata

Usually it is not possible to distinguish between (plain) data and metadatabecause something can be data and metadata at the same time. The headline ofan article is both its title (metadata) and part of its text (data). Also,Data and metadata can change their roles. A poem, as such, would be regardedas data, but if there is a song that uses it as lyrics, the whole poem couldbe attached to an audio file of the song as metadata. Thus, the labelingdepends on the point of view.

These considerations apply no matter which of the above definitions isconsidered, except where explicit markup is used to denote what is data andwhat is metadata.

3.8   Use

Metadata has many different applications; this section lists some of themost common.

Metadata is used to speed up and enrich searching for resources. In general,search queries using metadata can save users from performing more complexfilter operations manually. It is now common for web browsers (with thenotable exception of Mozilla Firefox), P2P applications and media managementsoftware to automatically download and locally cache metadata, to improvethe speed at which files can be accessed and searched.

Metadata may also be associated to files manually. This is often the casewith documents which are scanned into a document storage repository such asFileNet or Documentum. Once the documents have been converted into anelectronic format a user brings the image up in a viewer application,manually reads the document and keys values into an online application to bestored in a metadata repository.

Metadata provide additional information to users of the data it describes.This information may be descriptive ("These pictures were taken by childrenin the school's third grade class.") or algorithmic ("Checksum=139F").

Metadata helps to bridge the semantic gap. By telling a computer how dataitems are related and how these relations can be evaluated automatically, itbecomes possible to process even more complex filter and search operations.For example, if a search engine understands that "Van Gogh" was a "Dutchpainter", it can answer a search query on "Dutch painters" with a link to aweb page about Vincent Van Gogh, although the exact words "Dutch painters"never occur on that page. This approach, called knowledge representation, isof special interest to the semantic web and artificial intelligence.

Certain metadata is designed to optimize lossy compression. For example, ifa video has metadata that allows a computer to tell foreground frombackground, the latter can be compressed more aggressively to achieve ahigher compression rate.

Some metadata is intended to enable variable content presentation. Forexample, if a picture has metadata that indicates the most importantregion - the one where there is a person - an image viewer on a smallscreen, such as on a mobile phone's, can narrow the picture to that regionand thus show the user the most interesting details. A similar kind ofmetadata is intended to allow blind people to access diagrams and pictures,by converting them for special output devices or reading their descriptionusing text-to-speech software.

Other descriptive metadata can be used to automate workflows. For example,if a "smart" software tool knows content and structure of data, it canconvert it automatically and pass it to another "smart" tool as input. As aresult, users save the many copy-and-paste operations required whenanalyzing data with "dumb" tools.

Metadata is becoming an increasingly important part of electronic discovery.Application and file system metadata derived from electronic documents andfiles can be important evidence. Recent changes to the Federal Rules ofCivil Procedure make metadata routinely discoverable as part of civillitigation. Parties to litigation are required to maintain and producemetadata as part of discovery, and spoliation of metadata can lead tosanctions.

Metadata has become important on the World Wide Web because of the need tofind useful information from the mass of information available.Manually-created metadata adds value because it ensures consistency. If aweb page about a certain topic contains a word or phrase, then all web pagesabout that topic should contain that same word or phrase. Metadata alsoensures variety, so that if a topic goes by two names each will be used. Forexample, an article about "sport utility vehicles" would also be tagged "4wheel drives", "4WDs" and "four wheel drives", as this is how SUVs are knownin some countries.

Examples of metadata for an audio CD include the MusicBrainz project and AllMedia Guide's Allmusic. Similarly, MP3 files have metadata tags in a formatcalled ID3.

3.9   Types of metadata

Metadata can be classified by:

Content. Metadata can either describe the resource itself (for example,name and size of a file) or the content of the resource (for example, "Thisvideo shows a boy playing football").

Mutability. With respect to the whole resource, metadata can be eitherimmutable (for example, the "Title" of a video does not change as the videoitself is being played) or mutable (the "Scene description" does change).

Logical function. There are three layers of logical function: at thebottom the subsymbolic layer that contains the raw data itself, then thesymbolic layer with metadata describing the raw data, and on the top thelogical layer containing metadata that allows logical reasoning using thesymbolic layer

types of metadata are;

  1. descriptive metadata.
  2. administrative metadata.
  3. structural metadata.
  4. technical metadata.
  5. use metadata

To successfully develop and use metadata, several important issues should betreated with care:

3.10   Metadata risks

Microsoft Office files include metadata beyond their printable content, suchas the original author's name, the creation date of the document, and theamount of time spent editing it. Unintentional disclosure can be awkward oreven, in professional practices requiring confidentiality, raise malpracticeconcerns. Some of Microsoft Office document's metadata can be seen byclicking File then Properties from the program's menu. Other metadata is notvisible except through external analysis of a file, such as is done inforensics. The author of the Microsoft Word-based Melissa computer virus in1999 was caught due to Word metadata that uniquely identified the computerused to create the original infected document.

3.11   Metadata lifecycle

Even in the early phases of planning and designing it is necessary to keeptrack of all metadata created. It is not economical to start attachingmetadata only after the production process has been completed. For example,if metadata created by a digital camera at recording time is not storedimmediately, it may have to be restored afterwards manually with greateffort. Therefore, it is necessary for different groups of resourceproducers to cooperate using compatible methods and standards.

Manipulation. Metadata must adapt if the resource it describes changes.It should be merged when two resources are merged. These operations areseldom performed by today's software; for example, image editing programsusually do not keep track of the Exif metadata created by digital cameras.

Destruction. It can be useful to keep metadata even after the resourceit describes has been destroyed, for example in change histories within atext document or to archive file deletions due to digital rights management.None of today's metadata standards consider this phase.

3.12   Storage

Metadata can be stored either internally, in the same file as the data, orexternally, in a separate file. Metadata that are embedded with content iscalled embedded metadata. A data repository typically stores the metadatadetached from the data. Both ways have advantages and disadvantages:

Internal storage allows transferring metadata together with the data itdescribes; thus, metadata is always at hand and can be manipulated easily.This method creates high redundancy and does not allow holding metadatatogether.

External storage allows bundling metadata, for example in a database,for more efficient searching. There is no redundancy and metadata can betransferred simultaneously when using streaming. However, as most formatsuse URIs for that purpose, the method of how the metadata is linked to itsdata should be treated with care. What if a resource does not have a URI(resources on a local hard disk or web pages that are created on-the-flyusing a content management system)? What if metadata can only be evaluatedif there is a connection to the Web, especially when using RDF? How torealize that a resource is replaced by another with the same name butdifferent content?

Moreover, there is the question of data format: storing metadata in ahuman-readable format such as XML can be useful because users can understandand edit it without specialized tools. On the other hand, these formats arenot optimized for storage capacity; it may be useful to store metadata in abinary, non-human-readable format instead to speed up transfer and savememory.

3.13   Criticisms

Some critics argue the Metadata is too expensive, time-consuming,subjective, requires both domain and metadata creation expertise and dependsheavily on context. Doctorow and Shirkey outline some of these criticisms.

The opposers of metadata sometimes use the term metacrap to refer to theunsolved problems of metadata in some scenarios. These people are alsoreferred to as "Meta Haters."

Critics may also observe that perceived ineffectiveness of metadata is notdue to lack of maturity in the industry. Definitive theoretical knowledgeand over a dozen industry tools to handle metadata can be referenced as farback as 1982. This makes metadata theory and tooling far older, yetobservably less successful, than other technology areas such asobject-oriented programming, data warehousing, and the commercializedInternet.

3.14   Types

In general, there are two distinct classes of metadata: structural orcontrol metadata and guide metadata. Structural metadata is used to describethe structure of computer systems such as tables, columns and indexes. Guidemetadata is used to help humans find specific items and is usually expressedas a set of keywords in a natural language.

Metatadata can be divided into 3 distinct categories:

  • Descriptive
  • Administrative
  • Structural

3.15   Information Technology and Software Engineering metadata

3.15.1   General IT metadata

In contrast, David Marco, another metadata theorist, defines metadata as"all physical data and knowledge from inside and outside an organization,including information about the physical data, technical and businessprocesses, rules and constraints of the data, and structures of the dataused by a corporation." Others have included web services, systems andinterfaces. In fact, the entire Zachman framework (see EnterpriseArchitecture) can be represented as metadata.

Notice that such definitions expand metadata's scope considerably, toencompass most or all of the data required by the Management InformationSystems capability. In this sense, the concept of metadata has significantoverlaps with the ITIL concept of a Configuration Management Database(CMDB), and also with disciplines such as Enterprise Architecture and ITportfolio management.

This broader definition of metadata has precedent. Third generationcorporate repository products (such as those eventually merged into the CAAdvantage line) not only store information about data definitions (COBOLcopybooks, DBMS schema), but also about the programs accessing those datastructures, and the Job Control Language and batch job infrastructuredependencies as well. These products (some of which are still in production)can provide a very complete picture of a mainframe computing environment,supporting exactly the kinds of impact analysis required for ITIL-basedprocesses such as Incident and Change Management. The ITIL Back Catalogueincludes the Data Management volume which recognizes the role of thesemetadata products on the mainframe, posing the CMDB as the distributedcomputing equivalent. CMDB vendors however have generally not expanded theirscope to include data definitions, and metadata solutions are also availablein the distributed world. Determining the appropriate role and scope foreach is thus a challenge for large IT organizations requiring the servicesof both.

Since metadata is pervasive, centralized attempts at tracking it need tofocus on the most highly leveraged assets. Enterprise Assets may onlyconstitute a small percentage of the entire IT portfolio.

Some practitioners have successfully managed IT metadata using the DublinCore metamodel.

3.15.2   IT metadata management products

First generation data dictionary/metadata repository tools would be thoseonly supporting a specific DBMS, such as IDMS's IDD (integrated datadictionary), the IMS Data Dictionary, and ADABAS's Predict.

Second generation would be ASG's DATAMANAGER product which could supportmany different file and DBMS types.

Third generation repository products became briefly popular in the early1990s along with the rise of widespread use of RDBMS engines such as IBM'sDB2.

Fourth generation products link the repository with more Extract, transform,load tools and can be connected with architectural modeling tools.

Fifth generation products are taking things to a new level by integratingdistributed computing, specialized hardware, extreme visualization, andanalytics, in a sense that now allows vertical uses of metadata in all sortsof things such as applications, messaging buses etc.

3.15.3   Relational database metadata

Each relational database system has its own mechanisms for storing metadata.Examples of relational-database metadata include:

  • Tables of all tables in a database, their names, sizes and number of rows in each table.
  • Tables of columns in each database, what tables they are used in, and the type of data stored in each column.

In database terminology, this set of metadata is referred to as the catalog.The SQL standard specifies a uniform means to access the catalog, called theINFORMATION_SCHEMA, but not all databases implement it, even if theyimplement other aspects of the SQL standard. For an example ofdatabase-specific metadata access methods, see Oracle metadata.

3.15.4   Data warehouse metadata

Data warehouse metadata systems are sometimes separated into two sections:

  1. back room metadata that are used for Extract, transform, load functions to get OLTP data into a data warehouse
  2. front room metadata that are used to label screens and create reports

Kimball lists the following types of metadata in a data warehouse:

  • source system metadata- source specifications, such as repositories, and source logical schemas- source descriptive information, such as ownership descriptions, update frequencies and access methods- process information, such as job schedules and extraction code
  • data staging metadata- data acquisition information, such as data transmission scheduling and results, and file usage- dimension table management, such as definitions of dimensions, and surrogate key assignments- transformation and aggregation, such as data enhancement and mapping, DBMS load scripts, and aggregate definitions- audit, job logs and documentation, such as data lineage records, data transform logs
  • DBMS metadata, such as:- DBMS system table contents- processing hint

Michael Bracket defines metadata (what he calls "Data resource data") as"any data about the organization's data resource". Adrienne Tannenbaumdefines metadata as "the detailed description of instance data. The formatand characteristics of populated instance data: instances and values,dependent on the role of the metadata recipient". These definitions arecharacteristic of the "data about data" definition.

3.15.5   Business Intelligence metadata

Business Intelligence is the process of analyzing large amounts of corporatedata, usually stored in large databases such as a Data Warehouse, trackingbusiness performance, detecting patterns and trends, and helping enterprisebusiness users make better decisions. Business Intelligence metadatadescribes how data is queried, filtered, analyzed, and displayed in BusinessIntelligence software tools, such as Reporting tools, OLAP tools, DataMining tools.

Examples:

  • OLAP metadata: The descriptions and structures of Dimensions, Cubes, Measures (Metrics), Hierarchies, Levels, Drill Paths
  • Reporting metadata: The descriptions and structures of Reports, Charts, Queries, DataSets, Filters, Variables, Expressions
  • Data Mining metadata: The descriptions and structures of DataSets, Algorithms, Queries

Business Intelligence metadata can be used to understand how corporatefinancial reports reported to Wall Street are calculated, how the revenue,expense and profit are aggregated from individual sales transactions storedin the data warehouse. A good understanding of Business Intelligencemetadata is required to solve complex problems such as compliance withcorporate governance standards, such as Sarbanes Oxley (SOX) or Basel II.

3.15.6   File system metadata

Nearly all file systems keep metadata about files out-of-band. Some systemskeep metadata in directory entries; others in specialized structure likeinodes or even in the name of a file. Metadata can range from simpletimestamps, mode bits, and other special-purpose information used by theimplementation itself, to icons and free-text comments, to arbitraryattribute-value pairs.

With more complex and open-ended metadata, it becomes useful to search forfiles based on the metadata contents. The Unix find utility was an earlyexample, although inefficient when scanning hundreds of thousands of fileson a modern computer system. Apple Computer's Mac OS X operating systemsupports cataloguing and searching for file metadata through a feature knownas Spotlight, as of version 10.4. Microsoft worked in the development ofsimilar functionality with the Instant Search system in Windows Vista, aswell as being present in SharePoint Server. Linux implements file metadatausing extended file attributes.

3.15.7   Program metadata

Metadata is casually used to describe the controlling data used in softwarearchitectures that are more abstract or configurable. Most executable fileformats include what may be termed "metadata" that specifies certain,usually configurable, behavioral runtime characteristics. However, it isdifficult if not impossible to precisely distinguish program "metadata" fromgeneral aspects of stored-program computing architecture; if the machinereads it and acts upon it, it is a computational instruction, and the prefix"meta" has little significance.

In Java, the class file format contains metadata used by the Java compilerand the Java virtual machine to dynamically link classes and to supportreflection. The Java Platform, Standard Edition since J2SE 5.0 has includeda metadata facility to allow additional annotations that are used bydevelopment tools.

In MS-DOS, the COM file format does not include metadata, while the EXE fileand Windows PE formats do. These metadata can include the company thatpublished the program, the date the program was created, the version numberand more.

In the Microsoft .NET executable format, extra metadata is included to allowreflection at runtime.

3.15.8   Existing software metadata

Object Management Group (OMG) has defined metadata format for representingentire existing applications for the purposes of software mining, softwaremodernization and software assurance. This specification, called the OMGKnowledge Discovery Metamodel (KDM) is the OMG's foundation for "modeling inreverse". KDM is a common language-independent intermediate representationthat provides an integrated view of an entire enterprise application,including its behavior (program flow), data, and structure. One of theapplications of KDM is Business Rules Mining.

Knowledge Discovery Metamodel includes a fine grained low-levelrepresentation (called "micro KDM"), suitable for performing static analysisof programs.

3.15.9   Document metadata

Most programs that create documents, including Microsoft SharePoint,Microsoft Word and other Microsoft Office products, save metadata with thedocument files. These metadata can contain the name of the person whocreated the file (obtained from the operating system), the name of theperson who last edited the file, how many times the file has been printed,and even how many revisions have been made on the file. Other savedmaterial, such as deleted text (saved in case of an undelete command),document comments and the like, is also commonly referred to as "metadata",and the inadvertent inclusion of this material in distributed files hassometimes led to undesirable disclosures.

Document Metadata is particularly important in legal environments wherelitigation can request this sensitive information (metadata) which caninclude many elements of private detrimental data. This data has been linkedto multiple lawsuits that have got corporations into legal complications.

Many legal firms today use "Metadata Management Software", also known as"Metadata Removal Tools". This software can be used to clean documentsbefore they are sent outside of their firm. This process, known as metadatamanagement, protects lawfirms from potentially unsafe leaking of sensitivedata through Electronic Discovery.

For a list of executable formats, see object file.

3.15.10   Digital library metadata

There are three categories of metadata that are frequently used to describeobjects in a digital library:

  1. descriptive - Information describing the intellectual content of the object, such as MARC cataloguing records, finding aids or similar schemes. It is typically used for bibliographic purposes and for search and retrieval.
  2. structural - Information that ties each object to others to make up logical units (e.g.. information that relates individual images of pages from a book to the others that make up the book).
  3. administrative - Information used to manage the object or control access to it. This may include information on how it was scanned, its storage format, copyright and licensing information, and information necessary for the long-term preservation of the digital objects.

Standards for metadata in digital libraries include Dublin Core, METS,PREMIS schema, and OAI-PMH.

3.15.11   Image metadata

Examples of image files containing metadata include Exchangeable image fileformat (EXIF) and Tagged Image File Format (TIFF).

Having metadata about images embedded in TIFF or EXIF files is one way ofacquiring additional data about an image. Tagging pictures with subjects,related emotions, and other descriptive phrases helps Internet users findpictures easily rather than having to search through entire imagecollections. A prime example of an image tagging service is Flickr, whereusers upload images and then describe the contents. Other patrons of thesite can then search for those tags. Flickr uses a folksonomy: a free-textkeyword system in which the community defines the vocabulary through userather than through a controlled vocabulary.

Users can also tag photos for organization purposes using Adobe's ExtensibleMetadata Platform (XMP) language, for example.

Digital photography is increasingly making use of technical metadata tagsdescribing the conditions of exposure. Photographers shooting Camera RAWfile formats can use applications such as Adobe Bridge or Apple Computer'sAperture to work with camera metadata for post-processing.

3.15.12   Geospatial metadata

Metadata that describe geographic objects (such as datasets, maps, features,or simply documents with a geospatial component) have a history going backto at least 1994 (refer MIT Library page on FGDC Metadata). This class ofmetadata is described more fully on the Geospatial metadata page.

3.15.13   Meta-metadata

Since metadata are also data, it is possible to have metadata ofmetadata–"meta-metadata." what is Machine-generated meta-metadata, such asthe reversed index created by a free-text search engine, is generally notconsidered metadata, though

3.16   Further Explanations

We begin with MetaData because it will be the central focus of allactions done within the MDBMPS. For the purposes of MDBMPSwe define metadata simply as data about our data. Wikipedia's articleabout metadata can enlighten the reader to the many theories aboutmetadata. But this definition will suffice for our needs.

So then let's expand this concept a bit. What kind of "data about data" dowe mean? In a typical DBMS or ISAM based system we would be talking aboutData Structures. So some of the types of information we would store in ourprograms and later need to reference would include:

  • Schema (or Database) Name
  • File (or Table) Name
  • Field (or Column) Name
  • File Type
  • Field Type
  • Field Length
  • Keys and Indexes

At this point an example would be useful. We will look at the filedefinition statements in two real world examples (COBOL and SQL), both withpseudocode examples.

First COBOL (with pseudocode):

Pseudocode

Use file called customer.datwith an index and create record key called customer-id

Allow file to set variable customer-record

Set variables for customer-record:

  1. customer-id
  2. customer-firstname
  3. customer-middlename
  4. customer-lastname
  5. customer-address
  6. customer-city
  7. customer-state
  8. customer-zip
  9. customer-zip4
Real World (COBOL)
ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.
FILE-CONTROL.

    SELECT CUSTOMER-FILE
        ASSIGN TO "customer.dat"
        ORGANIZATION IS INDEXED
        RECORD KEY IS CUSTOMER-ID
        ACCESS MODE IS DYNAMIC.

DATA DIVISION.
FILE SECTION.

FD  CUSTOMER-FILE
    LABEL RECORDS ARE STANDARD.

01  CUSTOMER-RECORD.
    05  CUSTOMER-ID                        PIC 9(10).
    05  CUSTOMER-FIRSTNAME                 PIC X(50).
    05  CUSTOMER-MIDDLENAME                PIC X(50).
    05  CUSTOMER-LASTNAME                  PIC X(50).
    05  CUSTOMER-ADDRESS                   PIC X(50).
    05  CUSTOMER-CITY                      PIC X(50).
    05  CUSTOMER-STATE                     PIC X(2).
    05  CUSTOMER-ZIP                       PIC 9(5).
    05  CUSTOMER-ZIP4                      PIC 9(4).

And now the same example in SQL (specifically MySQL) with accompanyingpseudocode:

Pseudocode

Create table called customer and refer to it as custom-file

Set in the table customer following variables:

  1. set id with: 10 integers of space and incrementing automatically (1, 2, 3, and so on)
  2. set firstname with: 50 variable characters
  3. set middlename with: 50 variable characters
  4. set lastname with: 50 variable characters
  5. set address with: 50 variable characters
  6. set city with: 50 variable characters
  7. set zip with: 5 integers
  8. set zip4 with: 4 integers
  9. use id for primary key
Real World (SQL)
CREATE TABLE `customer`(
    `id`         int(10)     unsigned NOT NULL auto_increment,
    `firstname`  varchar(50) NOT NULL default '',
    `middlename` varchar(50) NOT NULL default '',
    `lastname`   varchar(50) NOT NULL default '',
    `address`    varchar(50) NOT NULL default '',
    `city`       varchar(50) NOT NULL default '',
    `zip`        int(5)      unsigned NOT NULL default 0,
    `zip4`       int(4)      unsigned NOT NULL default 0,
    PRIMARY KEY  (`id`)
);

These two examples produce nearly identical layouts. By reviewing theselayouts, we're provided with "data about data." We can see for instance thatcertain fields may contain up to fifty characters (first name, middle name,last name, adddress and city). Other fields only allow numbers to be placedin them (id, zip and zip plus four). We can search* the data because wehave an Index or a Primary Key on the field labeled ID. MySQL provides usslightly more information in this case than COBOL, because it declares thefield is to be Automatically Incremented, each time a new record is added.While the COBOL definition does not explicitely state this, most programmerswill understand that an ID field will be incremented, whether by a DBMSautomatically, or at the behest of the programmer through his own code.

* More accurately, SQL allows searching and sorting on any column.However, columns that have indexes or keys will sort significantlyfaster than those lacking them.

So these examples provide for us a glimpse of what we mean when we saymetadata. There are of course many other options extant in programmingsystems. Fields generally have a number of options for their type. Suchwould include strings (PIC X/varchar), integers (PIC 9/int), real numbers,dates, times and other commonly used types from the real world. Systemdesigners chose types that we could consider "base types" since nearly allinformation can be broken down into one sort or another of these. Thesetypes provide the programming language or data storage system with certainrules about the field. These rules are fairly limited though generallyincluding simply what kind of characters can be stored in the field and howmany characters. In the case of dates or times, the rules are slightlyenhanced to allow for date calculations. There is however, for instance no"Zip Code" or "First Name" type, and thus no rules to go along with these.

The MDBMPS would make use of these existent types, and it wouldthoroughly expand upon them. In the MDBMPS there will in fact be aZip Code type, a Name type and many more types. As many as can be imagined,can be created. Some systems such as PostgreSQL call these, user definedtypes. Programmers would create information and rules about fields andsave these into their own database, to be called forth when needed in anyproject. Thenceforth, when a programmer needs a zip code in any of hisapplications, he can simply 'drop' the zipcode into his application. Therules for the zipcode type will already have been defined and theapplication will follow those rules.

Thus, in the sense that we are speaking here, metadata is literally alldata about our data. It includes as much information as we can fathom abouteverything from an individual field, a screen, a report, a table, a databaseand so on. Because the system will be flexibily built, as new metadata isdesired, the system can be modified to include these additional options.

4   Metaprogramming

Metaprogramming in it's most simple definition is creating software, thatcreates software. It is the writing of computer programs that write ormanipulate other programs (or themselves) as their data. Indeed, thecompilers we mentioned in the introduction are one kind of metaprogram,for they take input, and convert it into a program, thus they createsoftware.

"'Yields falsehood when preceded by its quotation' yields falsehood whenpreceded by its quotation." - Willard Van Orman Quine

Specifying about a metaprogramming code generator requires first a referenceabout a quine. A quine is a computer program which produces a copy of itsown source code as its ownly output. Similar to a quine, a metaprogram willgenerate code from either a set of rules or by convention. But differentfrom a quine, a metaprogram generates other programs and subprograms. Ofcourse it will be a goal of the MDBMPS team to have theMDBMPS generate itself.

Metaprogramming varies significantly in form from one type to another.Some examples include those listed in these subtopics. But there are otheroptions, as metaprogramming is a theory unto itself.

4.1   What Is Metaprogramming

Metaprogramming is the writing of computer programs that write or manipulateother programs (or themselves) as their data, or that do part of the workat runtime that would otherwise be done at compile time. In many cases, thisallows programmers to get more done in the same amount of time as theywould take to write all the code manually, or it gives programs greaterflexibility to efficiently handle new situations without recompilation.

The language in which the metaprogram is written is called themetalanguage. The language of the programs that are manipulated is calledthe object language. The ability of a programming language to be its ownmetalanguage is called reflection or reflexivity.

Reflection is a valuable language feature to facilitate metaprogramming.Having the programming language itself as a first-class data type (as inLisp or REBOL) is also very useful. Generic programming invokes ametaprogramming facility within a language, in those languages supportingit.

Metaprogramming usually works through one of two ways. The first way is toexpose the internals of the run-time engine to the programming code throughapplication programming interfaces (APIs). The second approach is dynamicexecution of string expressions that contain programming commands. Thus,"programs can write programs". Although both approaches can be used in thesame language, most languages tend to lean toward one or the other.

4.1.1   Examples

A simple example of a metaprogram is this bash script, which is an exampleof generative programming:

Metaprogramming Example
#!/bin/bash
# metaprogram
echo '#!/bin/bash' >program
for ((I=1; I<=992; I++)) do
    echo "echo $I" >>program
done
chmod +x program

This script (or program) generates a new 993-line program which prints outthe numbers 1–992. This is only an illustration of how to use code to writemore code; it is not the most efficient way to print out a list of numbers.Nonetheless, a programmer can write and execute this metaprogram in just acouple of minutes, and will have generated exactly 1000 lines of code inthat amount of time.

Not all metaprogramming involves generative programming. If programs aremodifiable at runtime or if an incremental compilation is available (suchas in Lisp, Python, REBOL, Smalltalk, Ruby, PHP, Perl, Tcl, Lua, Groovy,and JavaScript), then techniques can be used to perform metaprogrammingwithout actually generating source code.

The most common metaprogramming tool is a compiler which allows a programmerto write a relatively short program in a high-level language and uses it towrite an equivalent assembly language or machine language program. This is afundamental tool for programming since, in most cases, it is vastlyimpractical to write the machine language program directly.

A more flexible method is to incorporate an interpreter in the program,which works directly with the program’s data. There are implementations ofthis kind for some common high-level languages, such as RemObject’s PascalScript for Object Pascal.

Another still fairly common example of metaprogramming is the use of lexand yacc, two tools used to generate lexical analyzers and parsers. Yacc isoften used as a compiler compiler, a tool to generate a tool for translatinghigh-level language programs to machine language.

A quine is a special kind of metaprogram that produces its own source codeas its output.

One style of programming which focuses heavily on metaprogramming islanguage-oriented programming, which is done via domain-specific programminglanguages.

4.2   Metalanguage

In logic and linguistics, a metalanguage is a language used to makestatements about statements in another language which is called the objectlanguage. Formal syntactic models for the description of grammar, e.g.generative grammar, are a type of metalanguage. More broadly, it can referto any terminology or language used to discuss language itself—a writtengrammar, for example, or a discussion about language use.

4.2.1   Kinds

There are a variety of recognized metalanguages, including embedded,ordered, and nested (hierarchical).

An embedded metalanguage, as its name suggests, is a language embedded in anobject language. It occurs both formally and naturally. This idea is foundin Douglas Hofstadter's book, Gödel, Escher, Bach, in his discussion of therelationship between formal languages and number theory: ". . .it is in thenature of any formalization of number theory that its metalanguage isembedded within it." (pg.270). It occurs in natural, or informal, languages,as well—such as in English, where adjectives, adverbs, and possessivepronouns constitute an embedded metalanguage; and where nouns, verbs, and,in some instances, adjectives and adverbs, constitute an object language.Thus, the adjective 'red' in the phrase 'red barn' is part of the embeddedmetalanguage of English, and the noun 'barn' is part of the object language.In the example, 'slowly running', the verb 'running' is part of the objectlanguage, and the adverb 'slowly' is part of the metalanguage.

An ordered metalanguage is analogous to ordered logic. An example of anordered metalanguage would be the construction of one metalanguage to talkabout an object language, followed by the creation of another metalanguageto talk about the first metalanguage, and so on.

A nested, or hierarchical, metalanguage is similar to an orderedmetalanguage in that each level represents a greater degree of abstraction.However, a nested metalanguage differs from an ordered one, in that eachlevel includes the one below. The paradigmatic example of a nestedmetalanguage comes from the Linnean taxonomic system in biology. Each levelin the system incorporates the one below it. The language used to talk aboutgenus is also used to talk about species; the language that is used to talkabout orders is also used to talk about geni; and, so on, up to kingdoms.

4.2.2   Role in metaphor

Michael Reddy (1979) has demonstrated that much of the language we use totalk about language is conceptualized and structured by what he refers to asthe conduit metaphor, which holds that ideas can be expressed andinterpreted through language. It incorporates these three interconnectedmetaphors:

  • Concepts, thoughts, feelings, meanings, sense and ideas are objects.
  • Words, sentences, and so on are containers (with an inside and an outside) for these objects.
  • Finally, communication is the act of sending and receiving these containers (through a conduit).

Those who speak and expect their meaning to be understood as it was intendedare thinking of language as a conduit, perhaps a more faithful one than ittruly is: miscommunications and misunderstandings can be attributed to thisassumption.

Reddy offers sentences similar to the following as evidence of theprevalence of the conduit metaphor in society:

  1. What is the meaning in his words?
  2. Try to get your thoughts into words.
  3. I couldn't get any meaning out of his words.
  4. I couldn't find any sense in his words.
  5. His words were empty and 'devoid' of feeling.
  6. His promises were hollow.
  7. His ideas were hidden in a dense thicket of sentences.
  8. Like a maggot in a turd he hid within the word.
  9. How do I convey my love in mere words.
  10. How do I get it across to you that I don't want to see you again.
  11. I gave her a call.
  12. I received your call.
  13. I got the message.
4.2.3   Computing

Computers follow programs which are sets of instructions in a clear andsimple language. The development of a programming language involves the useof a metalanguage. Backus–Naur form is one of the earliest metalanguagesused in computing and was developed in the 1960s by John Backus and PeterNaur.

4.3   Object language

Object language has meaning in contexts of computer programming andoperation, and in linguistics and logic.

4.3.1   Programming

At their basic level, computers act on what is given to them through alimited set of instructions which are understood by their CPUs. In theearliest computers, that meant programmers sometimes typed 1's and 0's toprogram. Since this requires considerable programmer training (and patience)to create instructions, later computer languages have gone to great lengthsto simplify the programmer's task. (For example, now it's common for peoplewith little training to drag-and-drop icons to create a Web page; all thesteps to create the actual instructions which are run by computers areautomatically performed, and not visible.)

One common practice for decades is to allow a programmer to use sourcelanguage (whose use may still require extensive training), and have thatlanguage translated into object code which the computer can immediately use.The compiling of one into the other varies depending on what CPU is beinggiven the instructions.

Object language in this context means something akin to "the object of whatthe programmer is trying to achieve". It should not be confused withobject-oriented language, which is a type of computer programming languagewhich changes the programmer's environment into convenient objects which canbe used in something similar to a drag-and-drop fashion.

Object language in this context is synonymous with target language. Theobject language of a translation most often is a machine language, but canbe some other kind of language, such as assembly language.

Because the object language of compilation has usually been machinelanguage, the term object file has come to mean a file containing machineinstructions, and sometimes the translated program itself is simply calledan object.

4.3.2   Formal languages

Mathematical logic and linguistics make use of metalanguages, which arelanguages for describing the nature of other languages. In mathematicallogic, the object language is usually a formal language. The language whicha metalanguage is used to describe is the object language. It is called thatbecause that language is the object under discussion using the metalanguage.

An example may be:someone who says "In French, you say Bonjour to greet someone" uses Englishas a metalanguage to describe the object language French.

4.4   Template Metaprogramming

Template metaprogramming is a metaprogramming technique in which templatesare used by a compiler to generate temporary source code, which is merged bythe compiler with the rest of the source code and then compiled. The outputof these templates include compile-time constants, data structures, andcomplete functions. The use of templates can be thought of as compile-timeexecution. The technique is used by a number of languages, the mostwell-known being C++, but also Curl, D, Eiffel, Haskell, ML and XL.

4.4.1   Components of template metaprogramming

The use of templates as a metaprogramming technique requires two distinctoperations: a template must be defined, and a defined template must beinstantiated. The template definition describes the generic form of thegenerated source code, and the instantiation causes a specific set of sourcecode to be generated from the generic form in the template.

Template metaprogramming is generally Turing-complete, meaning that anycomputation expressible by a computer program can be computed, in some form,by a template metaprogram.

Templates are different from macros. A macro, which is also a compile-timelanguage feature, generates code in-line using text manipulation andsubstitution. Macro systems often have limited compile-time process flowabilities and usually lack awareness of the semantics and type system oftheir companion language (an exception should be made with Lisp's macros,which are written in Lisp itself, and is not a simple text manipulation andsubstitution).

Template metaprograms have no mutable variables— that is, no variable canchange value once it has been initialized, therefore templatemetaprogramming can be seen as a form of functional programming. In factmany template implementations only implement flow control through recursion,as seen in the example below.

4.4.2   Using template metaprogramming

Though the syntax of template metaprogramming is usually very different fromthe programming language it is used with, it has practical uses. Some commonreasons to use templates are to implement generic programming (avoidingsections of code which are similar except for some minor variations) or toperform automatic compile-time optimization such as doing something once atcompile time rather than every time the program is run — for instance, byhaving the compiler unroll loops to eliminate jumps and loop countdecrements whenever the program is executed.

4.4.3   Compile-time class generation

What exactly "programming at compile-time" means can be illustrated with anexample of a factorial function, which in non-templated C++ can be writtenusing recursion as follows:

Example
int factorial(int n)
{
    if (n == 0)
       return 1;
    return n * factorial(n - 1);
}

void foo()
{
    int x = factorial(4); // == (4 * 3 * 2 * 1 * 1) == 24
    int y = factorial(0); // == 0! == 1
}

The code above will execute when the program is run to determine thefactorial value of the literals 4 and 0.

Instead, by using template metaprogramming and template specialization toprovide the ending condition for the recursion, the factorials used in theprogram, ignoring any factorial not used, can be calculated at compile-timeby

Example
template <int N>
struct Factorial
{
    enum { value = N * Factorial<N - 1>::value };
};

template <>
struct Factorial<0>
{
    enum { value = 1 };
};

// Factorial<4>::value == 24
// Factorial<0>::value == 1
void foo()
{
    int x = Factorial<4>::value; // == 24
    int y = Factorial<0>::value; // == 1
}

The code above calculates the factorial value of the literals 4 and 0 atcompile time and uses the result as if they were precalculated constants.

While the two versions are similar from the point of view of the program'sfunctionality, the first example calculates the factorials at run time,while the second calculates them at compile time. However, to be able touse templates in this manner, the compiler must know the value of itsparameters at compile time, which has the natural precondition thatFactorial<X>::value can only be used if X is known at compile time. In otherwords, X must be a constant literal or a constant expression, such as usingsizeof operator.

4.4.4   Compile-time code optimization

The factorial example above is one example of compile-time code optimizationin that all factorials used by the program are pre-compiled and injected asnumeric constants at compilation, saving both run-time overhead and memoryfootprint. It is, however, a relatively minor optimisation.

As another, more significant, example of compile-time loop-unrolling,templated metaprogramming can be used to create n-dimensional vector classes(where n is known at compile time). The benefit over a more traditionaln-dimensional vector is that the loops can be unrolled, resulting in veryoptimized code. As an example, consider the addition operator. Ann-dimensional vector addition might be written as

Example
template <int dimension>
Vector<dimension>& Vector<dimension>::operator+=(const Vector<dimension>& rhs)
{
    for (int i = 0; i < dimension; ++i)
        value[i] += rhs.value[i];
    return *this;
}

When the compiler instantiates the templated function defined above,the following code may be produced:

Example
template <>
Vector<2>& Vector<2>::operator+=(const Vector<2>& rhs)
{
    value[0] += rhs.value[0];
    value[1] += rhs.value[1];
    return *this;
}

The compiler's optimizer should be able to unroll the for loop because thetemplate parameter dimension is a constant at compile time.

4.4.5   Static polymorphism

Polymorphism is a common standard programming facility where derived objectscan be used as instances of their base object but where the derived objects'methods will be invoked, as in this code

Example
class Base
{
    public:
    virtual void method() { std::cout << "Base"; }
};

class Derived : public Base
{
    public:
    virtual void method() { std::cout << "Derived"; }
};

int main()
{
    Base *pBase = new Derived;
    pBase->method(); //outputs "Derived"
    delete pBase
    return 0;
}

where all invocations of virtual methods will be those of the most-derivedclass. This dynamically polymorphic behaviour is obtained by the creation ofvirtual look-up tables for classes with virtual methods, tables that aretraversed at run time to identify the method to be invoked. Thus, run-timepolymorphism necessarily entails execution overhead.

However, in many cases the polymorphic behaviour needed is invariant and canbe determined at compile time. Then the Curiously Recurring TemplatePattern (CRTP) can be used to achieve static polymorphism, which is animitation of polymorphism in programming code but which is resolved atcompile time and thus does away with run-time virtual-table lookups. Forexample:

Example
template <class Derived>
struct base
{
    void interface()
    {
         // ...
         static_cast<Derived*>(this)->implementation();
         // ...
    }
};

struct derived : base<derived>
{
     void implementation();
};

Here the base class template will take advantage of the fact that memberfunction bodies are not instantiated until after their declarations, and itwill use members of the derived class within its own member functions, viathe use of a static_cast, thus at compilation generating an objectcomposition with polymorphic characteristics. As an example of real-worldusage, the CRTP is used in the Boost iterator library.

Another similar use is the "Barton-Nackman trick", sometimes referred to as"restricted template expansion", where common functionality can be placedin a base class that is used not as a contract but as a necessary componentto enforce conformant behaviour while minimising code redundancy.

4.4.6   Benefits and drawbacks of template metaprogramming
  • Compile-time versus execution-time tradeoff: Since all templated code is processed, evaluated and expanded at compile-time, compilation will take longer while the executable code may be more efficient. This overhead is generally small, but for large projects, or projects relying pervasively on templates, it may be significant.
  • Generic programming: Template metaprogramming allows the programmer to focus on architecture and delegate to the compiler the generation of any implementation required by client code. Thus, template metaprogramming can accomplish truly generic code, facilitating code minimization and better maintainability.
  • Readability: With respect to C++, the syntax and idioms of template metaprogramming are esoteric compared to conventional C++ programming, and advanced, or even most non-trivial, template metaprogramming can be very difficult to understand. Metaprograms can thus be difficult to maintain by programmers inexperienced in template metaprogramming (though this may vary with the language's implementation of template metaprogramming syntax).
  • Portability: With respect to C++, due to differences in compilers, code relying heavily on template metaprogramming (especially the newest forms of metaprogramming) might have portability issues

4.5   Automatic programming

With a subset of metaprogramming called automatic programming, thegenerated code is created automatically, based on templates or data-driven,when the metaprogram is executed.

In computer science, the term automatic programming identifies a typeof computer programming in which some mechanism generates a computerprogram rather than have human programmers write the code.

There has been little agreement on the precise definition of automaticprogramming, mostly because its meaning has changed over time. David Parnas,tracing the history of "automatic programming" in published research, notedthat in the 1940s it described automation of the manual process of punchingpaper tape. Later it referred to translation of high-level programminglanguages like Fortran and ALGOL. In fact, one of the earliest programsidentifiable as a compiler was called Autocode. Parnas concluded that"automatic programming has always been a euphemism for programming in ahigher-level language than was then available to the programmer."

4.5.1   Generative programming

Generative programming is a style of computer programming that usesautomated source code creation through generic frames, classes, prototypes,templates, aspects, and code generators to improve programmer productivity.It is often related to code-reuse topics such as component-based softwareengineering and product family engineering.

4.5.2   Source code generation

Source code generation is the act of generating source code basing on anontological model such as a template and is accomplished with a programmingtool such as a template processor or an IDE. These tools allow the generationof source code through any of various means. A macro processor, such asthe C preprocessor, which replaces patterns in source code according torelatively simple rules, is a simple form of source code generator.

4.5.3   Implementations

IDEs such as Microsoft Visual Studio have more advanced forms of sourcecode generation, with which the programmer can interactively select andcustomize "snippets" of source code. Program "wizards", which allow theprogrammer to design graphical user interfaces interactively while thecompiler invisibly generates the corresponding source code, are anothercommon form of source code generation.

Examples:

NConstruct is a Windows and Web rapid enterprise application developmenttool and environment for .NET Framework environment. It generates .NETC# source code for all tiers of the application through simple wizardprocedures.

MyGeneration is a template-based code generator for the Microsoft.NET Framework.

Ebenezer Enterprises provides an on line service that writes efficientC++ marshalling code.

LatticeFramework Studio is a model-driven and template-based rapidenterprise application development tool for .NET Framework environment.It generates C#, VB.NET, Java, Php, diagrams and documentations throughsimple wizard procedures.

MyProLang is a template-driven automatic natural imperative programminglanguage that utilizes GUI templates to generate proprietary naturallanguage source-code, that can be eventually compiled into an executableprogram.

4.5.4   Generic programming

Generic programming isa style of computer programming in which algorithms are written in terms ofto-be-specified-later types that are then instantiated when needed forspecific types provided as parameter.

Generic programming is a style of computer programming in which algorithmsare written in terms of to-be-specified-later types that are then instantiatedwhen needed for specific types provided as parameters and was pioneered by Adawhich appeared in 1983. This approach permits writing common functions ortypes that differ only in the set of types on which they operate when used,thus reducing duplication. Software entities created using generic programmingare known as generics in Ada, Eiffel, Java, C#, Visual Basic .NET and Haskell;templates in C++; and parameterized types in the influential 1994 book DesignPatterns. The authors of Design Patterns note that this technique, especiallywhen combined with delegation, is very powerful but that "Dynamic, highlyparameterized software is harder to understand than more static software."(Gang of Four 1995:21)

Generic programming refers to features of certain statically typedprogramming languages that allow some code to effectively circumvent thestatic typing requirements. For instance in C++, a template is a routinein which some parameters are qualified by a type variable. Since codegeneration in C++ depends on concrete types, the template is specializedfor each combination of argument types that occur in practice. Genericprogramming is commonly used to implement containers such as lists andhash tables and functions such as a particular sorting algorithm forobjects specified in terms more general than a concrete type.

4.5.4.1   Programming language support for generic programming

Generic programming facilities first appeared in the 1970s in languageslike CLU and Ada, and were subsequently adopted by many object-based andobject-oriented languages, including BETA, C++, D, Eiffel, Java, and DEC'snow defunct Trellis-Owl language. Implementations of generics in languagessuch as Java and C# are formally based on the notion of parametricity, dueto John C. Reynolds. Software entities created using generic programmingare known as generics in Ada, Eiffel, Java, C#, VB .NET and Haskell;templates in C++; and parameterized types in Design Patterns.

Generic programming is implemented and supported differently within eachlanguage. The term "generic" has also been used differently in programmingcontexts. For example, in Forth the compiler can execute code whilecompiling and one can create new compiler keywords and new implementationsfor those words on the fly. It has few words that expose the compilerbehaviour and therefore naturally offers generic programming capacitieswhich, however, are not referred to as such in most Forth texts. The termhas been used in functional programming, specifically in Haskell-likelanguages, which use a structural type system where types are alwaysparametric and the actual code on those types is generic. In these usesgeneric programming still serves the similar purpose of code-saving andthe rendering of an abstraction.

4.5.4.2   Generic programming in object-oriented languages

When creating containers of objects it is possible to write specificimplementations for each datatype contained, even if the code is virtuallyidentical except for different datatypes. In C++, this duplication of codecan be circumvented by defining a template class:

Example
template<typename T>
class List
{
   /* class contents */
};

List<Animal> list_of_animals;
List<Car> list_of_cars;

Above, T is a placeholder for whatever type is specified when the list iscreated. These "containers-of-type-T", commonly called generics, are ageneric programming technique allowing a class to be reused with differentdatatypes as long as certain contracts such as subtypes and signature arekept. Generic programming should not to be confused with inclusion polymorphism,which is the algorithmic usage of exchangeable sub-classes: for instance,a list of objects of type Moving_Object containing objects of type Animaland Car. Generics can also be used for type-independent functions as inthe Swap example below:

Example
template<typename T>
void Swap(T & a, T & b) //"&" passes parameters by reference
{
   T temp = b;
   b = a;
   a = temp;
}

string hello = "world!", world = "Hello, ";
Swap( world, hello );
cout << hello << world << endl; //Output is "Hello, world!"

The C++ template construct used above is widely cited as the genericprogramming construct that popularized the notion among programmers andlanguage designers and supports many generic programming idioms. The Dprogramming language also offers fully generic-capable templates basedon the C++ precedent but with a simplified syntax. The Java programminglanguage has provided generic programming facilities syntactically basedon C++'s since the introduction of J2SE 5.0 and implements the genericssubset of generic programming.

Although the examples above are a common use of generic programming, andsome languages implement only this aspect of it, the concept of genericprogramming is not limited to "generics" as a programming language construct.In Python, a language with strong, dynamic typing, generic programmingis both transparent and also the simplest way to write routines. Forexample, the preceding example can be translated into Python if a boxclass is created to simulate the C++ notion of call-by-reference.

Example
class Box:
  def __init__(self,ob):
    self.contents = ob

def swap(box1,box2):
  box2.contents, box1.contents = box1.contents, box2.contents

C# 2.0, Chrome 1.5 and Visual Basic .NET 2005 have constructs that takeadvantage of the support for generics present in the Microsoft .NETFramework since version 2.0. The ML family of programming languages encouragegeneric programming through parametric polymorphism and generic modules calledfunctors. The type class mechanism of Haskell supports generic programming.

Dynamic typing, such as is featured in Objective-C, and, if necessary,judicious use of protocols circumvent the need for use of generic programmingtechniques, since there exists a general type to contain any object. WhileJava does so also, the casting that needs to be done breaks the disciplineof static typing, and generics are one way of achieving some of the benefitsof dynamic typing with the advantages of having static typing.

Generics in Ada

Ada has had generics since it was first designed in 1977-1980. The standardlibrary uses generics to provide many services. Ada 2005 adds a comprehensivegeneric container library to the standard library, which was inspired by C++'sstandard template library.

A generic unit is a package or a subprogram that takes one or more genericformal parameters.

A generic formal parameter is a value, a variable, a constant, a type,a subprogram, or even an instance of another, designated, generic unit. Forgeneric formal types, the syntax distinguishes between discrete, floating-point,fixed-point, access (pointer) types, etc. Some formal parameters can havedefault values.

To instantiate a generic unit, the programmer passes actual parametersfor each formal. The generic instance then behaves just like any otherunit. It is possible to instantiate generic units at run-time, for exampleinside a loop.

Ada example

The specification of a generic package:

Example
generic
   Max_Size : Natural; -- a generic formal value
   type Element_Type is private; -- a generic formal type; accepts any
   nonlimited type
package Stacks is
   type Size_Type is range 0 .. Max_Size;
   type Stack is limited private;
   procedure Create (S : out Stack;
                     Initial_Size : in Size_Type := Max_Size);
   procedure Push (Into : in out Stack; Element : in Element_Type);
   procedure Pop (From : in out Stack; Element : out Element_Type);
   Overflow : exception;
   Underflow : exception;
private
   subtype Index_Type is Size_Type range 1 .. Max_Size;
   type Vector is array (Index_Type range <>) of Element_Type;
   type Stack (Allocated_Size : Size_Type := 0) is record
      Top : Index_Type;
      Storage : Vector (1 .. Allocated_Size);
   end record;
end Stacks;

Instantiating the generic package:

Example
type Bookmark_Type is new Natural;
-- records a location in the text document we are editing

package Bookmark_Stacks is new Stacks (Max_Size => 20,
                                       Element_Type => Bookmark_Type);
-- Allows the user to jump between recorded locations in a document

Using an instance of a generic package:

Example
type Document_Type is record
   Contents : Ada.Strings.Unbounded.Unbounded_String;
   Bookmarks : Bookmark_Stacks.Stack;
end record;

procedure Edit (Document_Name : in String) is
  Document : Document_Type;
begin
  -- Initialise the stack of bookmarks:
  Bookmark_Stacks.Create (S => Document.Bookmarks, Initial_Size => 10);
  -- Now, open the file Document_Name and read it in...
end Edit;

Advantages and limitations

The language syntax allows precise specification of constraints on genericformal parameters. For example, it is possible to specify that a genericformal type will only accept a modular type as the actual. It is alsopossible to express constraints between generic formal parameters; forexample:

Example
generic
   type Index_Type is (<>); -- must be a discrete type
   type Element_Type is private; -- can be any nonlimited type
   type Array_Type is array (Index_Type range <>) of Element_Type;

In this example, Array_Type is constrained by both Index_Type and Element_Type.When instantiating the unit, the programmer must pass an actual array type thatsatisfies these constraints.

The disadvantage of this fine-grained control is a complicated syntax, but,because all generic formal parameters are completely defined in the specification,the compiler can instantiate generics without looking at the body of the generic.

Unlike C++, Ada does not allow specialised generic instances, and requiresthat all generics be instantiated explicitly. These rules have several consequences:

  • the compiler can implement shared generics: the object code for ageneric unit can be shared between all instances (unless the programmerrequests inlining of subprograms, of course).As further consequences:

    • there is no possibility of code bloat (code bloat is common inC++ and requires special care, as explained below).
    • it is possible to instantiate generics at run time, as well asat compile time, since no new object code is required for a new instance.
    • actual objects corresponding to a generic formal object arealways considered to be nonstatic inside the generic; seeGeneric formal objects in the Wikibook for details and consequences.

  • all instances of a generic being exactly the same, it is easier toreview and understand programs written by others; there are no"special cases" to take into account.

  • all instantiations being explicit, there are no hidden instantiationsthat might make it difficult to understand the program.

  • Ada does not permit "template metaprogramming", because it does notallow specialisations.

Templates in C++

Templates are of great utility to programmers in C++, especially when combinedwith multiple inheritance and operator overloading. The C++ Standard TemplateLibrary (STL) provides a framework of connected templates. Templates in C++may also be used for template metaprogramming, which is a way of pre-evaluatingsome of the code at compile-time rather than run-time. Due to Templatespecialization, C++ Templates are considered turing complete.

Technical overview

There are two kinds of templates: function templates and class templates.A function template is a pattern for creating ordinary functions basedupon the parameterizing types supplied when instantiated. For example,the C++ Standard Template Library contains the function template max(x, y)which creates functions that return either x or y, whichever is larger.max() could be defined like this:

Example
template <typename T>
T max(T x, T y)
{
    if (x < y)
        return y;
    else
        return x;
}

Specializations of this function template, instantiations with specifictypes, can be called just like an ordinary function:

Example
cout << max(3, 7);   // outputs 7

The compiler examines the arguments used to call max and determines thatthis is a call to max(int, int). It then instantiates a version of thefunction where the parameterizing type T is int, making the equivalentof the following function:

Example
int max(int x, int y)
{
    if (x < y)
        return y;
    else
        return x;
}

This works whether the arguments x and y are integers, strings, or anyother type for which the expression x < y is sensible, or more specifically,for any type for which operator< is defined. Common inheritance is notneeded for the set of types that can be used, and so it is very similarto duck typing. A program defining a custom data type can use operatoroverloading to define the meaning of < for that type, thus allowing itsuse with the max() function template. While this may seem a minor benefitin this isolated example, in the context of a comprehensive library likethe STL it allows the programmer to get extensive functionality for anew data type, just by defining a few operators for it. Merelydefining < allows a type to be used with the standard sort(), stable_sort(),and binary_search() algorithms or to be put inside data structures suchas sets, heaps, and associative arrays.

C++ templates are completely type safe at compile time. As a demonstration,the standard type complex does not define the < operator, because thereis no strict order on complex numbers. Therefore max(x, y) will fail witha compile error if x and y are complex values. Likewise, other templatesthat rely on < cannot be applied to complex data. Unfortunately,compilers historically generate somewhat esoteric, long, and unhelpfulerror messages for this sort of error. Ensuring that a certain objectadheres to a method protocol can alleviate this issue.

The second kind of template, a class template, extends the same conceptto classes. A class template specialization is a class. Class templatesare often used to make generic containers. For example, the STL has alinked list container. To make a linked list of integers, one writeslist<int>. A list of strings is denoted list<string>. A list has a setof standard functions associated with it, which work for any compatibleparameterizing types.

Template specialization

A powerful feature of C++'s templates is template specialization. Thisallows alternative implementations to be provided based on certaincharacteristics of the parameterized type that is being instantiated.Template specialization has two purposes: to allow certain forms ofoptimization, and to reduce code bloat.

For example, consider a sort() template function. One of the primaryactivities that such a function does is to swap or exchange the valuesin two of the container's positions. If the values are large (in termsof the number of bytes it takes to store each of them), then it is oftenquicker to first build a separate list of pointers to the objects, sortthose pointers, and then build the final sorted sequence. If the valuesare quite small however it is usually fastest to just swap the valuesin-place as needed. Furthermore if the parameterized type is alreadyof some pointer-type, then there is no need to build a separate pointerarray. Template specialization allows the template creator to writedifferent implementations and to specify the characteristics that theparameterized type(s) must have for each implementation to be used.

Unlike function templates, class templates can be partially specialized.That means that an alternate version of the class template code can beprovided when some of the template parameters are known, while leavingother template parameters generic. This can be used, for example, tocreate a default implementation (the primary specialization) that assumesthat copying a parameterizing type is expensive and then create partialspecializations for types that are cheap to copy, thus increasing overallefficiency. Clients of such a class template just use specializations ofit without needing to know whether the compiler used the primaryspecialization or some partial specialization in each case. Classtemplates can also be fully specialized, which means that an alternateimplementation can be provided when all of the parameterizing types are known.

Advantages and disadvantages

Some uses of templates, such as the max() function, were previously filledby function-like preprocessor macros (a legacy of the C programming language).For example, here is a possible max() macro:

Example
#define max(a,b) ((a) < (b) ? (b) : (a))

Both macros and templates are expanded at compile time. Macros are alwaysexpanded inline; templates can also be expanded as inline functions whenthe compiler deems it appropriate. Thus both function-like macros andfunction templates have no run-time overhead.

However, templates are generally considered an improvement over macrosfor these purposes. Templates are type-safe. Templates avoid some of thecommon errors found in code that makes heavy use of function-like macros.Perhaps most importantly, templates were designed to be applicable tomuch larger problems than macros.

There are three primary drawbacks to the use of templates: compiler support,poor error messages, and code bloat. Many compilers historically have poorsupport for templates, thus the use of templates can make code somewhatless portable. Support may also be poor when a C++ compiler is being usedwith a linker which is not C++-aware, or when attempting to use templatesacross shared library boundaries. Most modern compilers however now havefairly robust and standard template support, and the new C++ standard,C++0x, is expected to further address these issues.

Almost all compilers produce confusing, long, or sometimes unhelpful errormessages when errors are detected in code that uses templates. This canmake templates difficult to develop.

Finally, the use of templates requires the compiler to generate a separateinstance of the templated class or function for every permutation of typeparameters used with it. (This is necessary because types in C++ are notall the same size, and the sizes of data fields are important to howclasses work.) So the indiscriminate use of templates can lead to codebloat, resulting in excessively large executables. However, judicioususe of template specialization can dramatically reduce such code bloatin some cases. The extra instantiations generated by templates can alsocause debuggers to have difficulty working gracefully with templates.For example, setting a debug breakpoint within a template from a sourcefile may either miss setting the breakpoint in the actual instantiationdesired or may set a breakpoint in every place the template is instantiated.

Also, because the compiler needs to perform macro-like expansions of templatesand generate different instances of them at compile time, the implementationsource code for the templated class or function must be available (e.g.included in a header) to the code using it. Templated classes or functions,including much of the Standard Template Library (STL), cannot be compiled.(This is in contrast to non-templated code, which may be compiled to binary,providing only a declarations header file for code using it.) This may bea disadvantage by exposing the implementing code, which removes someabstractions, and could restrict its use in closed-source projects.

Templates in D

The D programming language supports templates that are evolved from thosein C++. Most C++ template idioms will carry over to D without alteration,but D adds functionality that streamlines some common cases.

The most obvious differences are syntax changes. D uses parentheses ( )instead of angle-brackets < > in a template definition. It also usesthe !( ) construct (that is, parentheses preceded by an exclamation point)instead of angle-brackets in template instantiation. Therefore, a!(b) inD is the equivalent of a<b> in C++. These changes were made in the interestof making templates easier to parse, as using angle-brackets can lead toambiguous syntax.

Static-if

D provides a static if construct that checks conditions at compile-time.This is vaguely analogous to C++'s #if and #endif preprocessor macros.The major difference is that static if has access to all compile-timevalues, including template arguments. Therefore, many situations whichrequire template specialization in C++ may be written inline in D.Recursive templates in D can look almost identical to their runtimeequivalents. This is shown in the classic compile-time factorial template.

Example
template factorial(uint n) {
    static if (n < 2)
        const uint factorial = 1;
    else
        const uint factorial = n * factorial!(n-1);
}

Alias parameters

Templates in D may also accept alias parameters. Alias parameters aresimilar to C++'s typedef but can also substitute templates parameters.This is a superset of the functionality of template arguments in C++,and will be added in the forthcoming C++0x specification. Alias parametersmay be templates, functions, types, or any other compile-time symbol.This allows a coder to, for example, "inject" a function into the middleof a template function.

Example
template wrapper(alias Fn) {
    // Wraps a D function with an "extern(C)" interface.
    extern(C) void wrapper() {
        Fn();
    }
}

This sort of template might be useful when interfacing D code with a C API.If a hypothetical C API wants a function pointer, you might use thetemplate like this:

Example
void foo() {
    // ...
}

some_c_function(&wrapper!(foo));

Generics in Java

Support for the generics, or "containers-of-type-T", subset of genericprogramming were added to the Java programming language in 2004 as partof J2SE 5.0. In Java, generics are checked at compile time for typecorrectness. The generic type information is then removed via a processcalled type erasure, and is unavailable at runtime. For example, aList<String> is converted to the raw type List. The compiler insertstype casts to convert the elements to the String type when they areretrieved from the list.

Generic programming in C# and .NET

Generics in C# (and other .NET languages) were added as part of .NETFramework 2.0 in November 2005. Although similar to generics in Java,.NET generics do not apply type erasure, but implement generics as a firstclass object in the runtime using reification. This design choice isleveraged to provide additional functionality, such as allowing reflectionwith preservation of generic types, as well as alleviating some of thelimitations of erasure (such as being unable to create generic arrays).This also means that there is no performance hit from runtime casts andnormally expensive boxing conversions. When primitive and value types areused as generic arguments, they get specialized implementations, allowingfor efficient generic collections and methods. As in C++ and Java, nestedgeneric types such as Dictionary<string, List<int>> are valid types.

C# (and .NET in general) allows six varieties of generic type constraintsusing the where keyword including restricting generic types to be valuetypes, to be classes, to have constructors, and to inherit from interfaces.Below is an example with an interface constraint:

Example
class Sample
{
  public static void Main()
  {
    int[] list = { 0, 1, 2, 3 };
    MakeAtLeast<int>(list, 2); // change array to { 2, 2, 2, 3 }
  }

  public static void MakeAtLeast<T>(IList<T> list, T lowest) where T : IComparable<T>
  {
    for (int i = 0; i < list.Count; i++)
      if (list[i].CompareTo(lowest) < 0) list[i] = lowest;
  }
}

Since all arrays implement the IList<T> interface (commonly associatedwith list classes), the MakeAtLeast() method allows operation on arrays,with elements of type T. The method's type constraint indicates that themethod is applicable to any type T that implements the generic IComparable<T>interface. This ensures a compile time error if the method is called witha list of another type. The interface provides the generic method CompareTo(T).

The above method could also be written without generic types, simply usingthe non-generic IList type. However, this would make the method less typesafe, as it would allow the method to be applied to a list of incomparableitems, resulting in a run time error. The method would need to access thelist items as objects instead, and would require casting to compare twoelements. (For value types like types such as int this requires a boxingconversion, although this can be worked around using the Comparer<T> class,as is done in the standard collection classes.)

Generic programming in Delphi

Generic support was added to Delphi language in October 2008. The firstgeneric support to Generics was Delphi 2007 .NET in 2006, but it was onlyfor .NET platform, and real support for Generics in Delphi was add in Delphi 2009.

4.5.4.3   Generic programming in Functional languages

Generic programming in Haskell

Six of the predefined type classes in Haskell (including Eq, the types thatcan be compared for equality, and Show, the types whose values can berendered as strings) have the special property of supporting derived instances.This means that a programmer defining a new type can state that this type is tobe an instance of one of these special type classes, without providingimplementations of the class methods as is usually necessary when declaringclass instances. All the necessary methods will be "derived" -- that is,constructed automatically -- based on the structure of the type. For instance,the following declaration of a type of binary trees states that it is to bean instance of the classes Eq and Show:

Example
data BinTree a = Leaf a | Node (BinTree a) a (BinTree a)
      deriving (Eq, Show)

This results in an equality function (==) and a string representationfunction (show) being automatically defined for any type of the form BinTreeT provided that T itself supports those operations.

The support for derived instances of Eq and Show makes their methods == andshow generic in a qualitatively different way from parametrically polymorphicfunctions: these "functions" (more accurately, type-indexed families of functions)can be applied to values of various types, and although they behave differentlyfor every argument type, little work is needed to add support for a new type.Ralf Hinze (2004) has shown that a similar effect can be achieved foruser-defined type classes by certain programming techniques. Otherresearchers have proposed approaches to this and other kinds of genericityin the context of Haskell and extensions to Haskell (discussed below).

PolyP

PolyP was the first generic programming language extension to Haskell.In PolyP, generic functions are called polytypic. The language introducesa special construct in which such polytypic functions can be defined viastructural induction over the structure of the pattern functor of a regulardatatype. Regular datatypes in PolyP are a subset of Haskell datatypes. Aregular datatype t must be of kind * -> *, and if a is the formal typeargument in the definition, then all recursive calls to t must have theform t a. These restrictions rule out higher kinded datatypes as well asnested datatypes, where the recursive calls are of a different form. Theflatten function in PolyP is here provided as an example:

Example
flatten :: Regular d => d a -> [a]
flatten = cata fl

polytypic fl :: f a [a] -> [a]
  case f of
    g+h -> either fl fl
    g*h -> \(x,y) -> fl x ++ fl y
    () -> \x -> []
    Par -> \x -> [x]
    Rec -> \x -> x
    d@g -> concat . flatten . pmap fl
    Con t -> \x -> []

cata :: Regular d => (FunctorOf d a b -> b) -> d a -> b

Generic Haskell

Generic Haskell is another extension to Haskell, developed at UtrechtUniversity in The Netherlands. The extensions it provides are:

Type-indexed values are defined as a value indexed over the variousHaskell type constructors (unit, primitive types, sums, products, anduser-defined type constructors). In addition, we can also specify thebehaviour of a type-indexed values for a specific constructor usingconstructor cases, and reuse one generic definition in anotherusing default cases.

The resulting type-indexed value can be specialised to any type.

Kind-indexed types are types indexed over kinds, defined by givinga case for both * and k -> k'. Instances are obtained by applying thekind-indexed type to a kind.

Generic definitions can be used by applying them to a type or kind.This is called generic application. The result is a type or value,depending on which sort of generic definition is applied.

Generic abstraction enables generic definitions be defined byabstracting a type parameter (of a given kind).

Type-indexed types are types which are indexed over the typeconstructors. These can be used to give types to more involvedgeneric values. The resulting type-indexed types can be specialisedto any type.

As an example, the equality function in Generic Haskell:

Example
type Eq {[ * ]} t1 t2 = t1 -> t2 -> Bool
type Eq {[ k -> l ]} t1 t2 = forall u1 u2. Eq {[ k ]} u1 u2 -> Eq {[ l ]} (t1 u1) (t2 u2)

eq {| t :: k |} :: Eq {[ k ]} t t
eq {| Unit |} _ _ = True
eq {| :+: |} eqA eqB (Inl a1) (Inl a2) = eqA a1 a2
eq {| :+: |} eqA eqB (Inr b1) (Inr b2) = eqB b1 b2
eq {| :+: |} eqA eqB _ _ = False
eq {| :*: |} eqA eqB (a1 :*: b1) (a2 :*: b2) = eqA a1 a2 && eqB b1 b2
eq {| Int |} = (==)
eq {| Char |} = (==)
eq {| Bool |} = (==)

The "Scrap your boilerplate" approach

The Scrap your boilerplate approach is a lightweight generic programmingapproach for Haskell (Lämmel and Peyton Jones, 2003). A web site for thisapproach explains which components of it are currently implemented in GHC.Uniplate is an even simpler package with a similar basic approach.

Clean

Clean offers generic programming based PolyP and the generic Haskell assupported by the GHC>=6.0. It parametrizes by kind as those but offersoverloading.

4.5.4.4   Generic programming features in other languages

Standard ML and OCaml provide functors, which are similar to class templatesand to Ada's generic packages. Scheme syntactic abstractions also haveconnection to generic programming -- these are in fact a superset oftemplating a la C++.

4.5.5   Language-oriented programming

Language-oriented programming is a style ofcomputer programming, rather than solvingproblems in general-purpose programming languages, the programmer createsone or more domain-specific programming languages for the problem first, andsolves the problem in those languages.

Language oriented programming is a style of computer programming, viametaprogramming in which, rather than solving problems in general-purposeprogramming languages, the programmer creates one or more domain-specificprogramming languages for the problem first, and solves the problem in thoselanguages. This concept is described in detail in the paper by Martin Wardentitled Language Oriented Programming published in Software - Conceptsand Tools, Vol.15, No.4, pp 147-161, 1994 and in the article by Sergey Dmitrieventitled Language Oriented Programming: The Next Programming Paradigm.

Existing implementations of this concept include:

  • Meta Programming System by JetBrains.
  • XMF by Ceteva.
  • openArchitectureWare.
  • Whole Platform.
  • Microsoft Software Factories.
  • Intentional Software.
  • LayerD.
  • The XL programming language
  • REBOL

A book that describes this concept, with examples, is Building BetterApplications: a Theory of Efficient Software Development. It takes theapproach to capture requirements in the user's terms, and then to try tocreate an implementation language as isomorphic as possible to the user'sdescriptions, so that the mapping between requirements and implementationis as direct as possible. A measure of the closeness of this isomorphismis the "redundancy" of the language, defined as the number of editingoperations needed to implement a stand-alone change in requirements. Itis not assumed a-priori what is the best language for implementing thenew language. Rather, the developer can choose among options created byanalysis of the information flows — what information is acquired, whatits structure is, when it is acquired, from whom, and what is done withit. See Linguistic Method.

4.6   Attribute-Oriented Programming

With a subset of metaprogramming called attribute-oriented programming,developers can designate program elements (e.g. classes and methods) toindicate that they maintain application-specific or domain-specificsemantics.

Attribute-oriented programming (@OP) is a program-level marking technique.Programmers can mark program elements (e.g. classes and methods) to indicatethat they maintain application-specific or domain-specific semantics. Forexample, some programmers may define a "logging" attribute and associate itwith a method to indicate the method should implement a logging function,while other programmers may define a "web service" attribute and associateit with a class to indicate the class should be implemented as a web service.Attributes separate application's core logic (or business logic) fromapplication-specific or domain-specific semantics (e.g. logging and webservice functions). By hiding the implementation details of those semanticsfrom program code, attributes increase the level of programming abstractionand reduce programming complexity, resulting in simpler and more readableprograms. The program elements associated with attributes are transformedto more detailed programs by a supporting tool (e.g. preprocessor). Forexample, a preprocessor may insert a logging program into the methodsassociated with a "logging" attribute.

4.6.1   Attribute-oriented programming in various languages
4.6.1.1   Java

With the inclusion of The Metadata Facility for the Java ProgrammingLanguage (JSR-175) into the J2SE 5.0 release it's possible to utilizeattribute-oriented programming right out of the box. XDoclet librarymakes it possible to use attribute-oriented programming approach inearlier versions of Java.

4.6.1.2   C#

C# language supports attributes from the very first release. Howeverthese attributes are used to give run-time information and are not usedby a pre-processor (there isn't one in C#'s reference implementation).

4.6.1.3   UML

The Unified Modeling Language (UML) supports a kind of attributes namedStereotypes.

4.7   Reflective Programming

In computer science, reflection is the process by which a computer programcan observe and modify its own structure and behaviour. The programmingparadigm driven by reflection is called reflective programming. It is aparticular kind of metaprogramming.

In most modern computer architectures, program instructions are storedas data - hence the distinction between instruction and data is merelya matter of how the information is treated by the computer and programminglanguage. Normally, 'instructions' are 'executed' and 'data' is 'processed';however, in some languages, programs can also treat instructions as dataand therefore make reflective modifications. Reflection is most commonlyused in high-level virtual machine programming languages like Smalltalkand scripting languages, and less commonly used in manifestly typed and/orstatically typed programming languages such as Java and C.

4.7.1   Reflection-oriented programming

Reflection-oriented programming, or reflective programming, is a functionalextension to the object-oriented programming paradigm. Reflection-orientedprogramming includes self-examination, self-modification, and self-replication.However, the emphasis of the reflection-oriented paradigm is dynamic programmodification, which can be determined and executed at runtime. Someimperative approaches, such as procedural and object-oriented programmingparadigms, specify that there is an exact predetermined sequence ofoperations with which to process data. The reflection-oriented programmingparadigm, however, adds that program instructions can be modified dynamicallyat runtime and invoked in their modified state. That is, the programarchitecture itself can be decided at runtime based upon the data, services,and specific operations that are applicable at runtime.

Programming sequences can be classified in one of two ways, atomic orcompound. Atomic operations are those that can be viewed as completingin a single, logical step, such as the addition of two numbers. Compoundoperations are those that require a series of multiple atomic operations.

A compound statement, in classic procedural or object-oriented programming,can lose its structure once it is compiled. The reflective programmingparadigm introduces the concept of meta-information, which keeps knowledgeof program structure. Meta-information stores information such as the nameof the contained methods, the name of the class, the name of parent classes,and/or what the compound statement is supposed to do. Using this storedinformation, as an object is consumed (processed), it can be reflectedupon to find out the operations that it supports. The operation that issuesin the required state via the desired state transition can be chosen atrun-time without hard-coding it.

4.7.2   Uses

Reflection can be used for observing and/or modifying program executionat runtime. A reflection-oriented program component can monitor theexecution of an enclosure of code and can modify itself according to adesired goal related to that enclosure. This is typically accomplished bydynamically assigning program code at runtime.

Reflection can also be used to adapt a given program to different situationsdynamically. For example, consider an application that uses two differentclasses X and Y interchangeably to perform similar operations. Withoutreflection-oriented programming, the application might be hard-coded tocall method names of class X and class Y. However, using the reflection-orientedprogramming paradigm, the application could be designed and written to utilizereflection in order to invoke methods in classes X and Y without hard-codingmethod names. Reflection-oriented programming almost always requires additionalknowledge, framework, relational mapping, and object relevance in order totake advantage of more generic code execution. Hard-coding can be avoided tothe extent that reflection-oriented programming is used.

Reflection is also a key strategy for metaprogramming.

[edit] Implementation

A language supporting reflection provides a number of features availableat runtime that would otherwise be very obscure or impossible to accomplishin a lower-level language. Some of these features are the abilities to:

  • Discover and modify source code constructions (such as code blocks,classes, methods, protocols, etc.) as a first-class object at runtime.
  • Convert a string matching the symbolic name of a class or functioninto a reference to or invocation of that class or function.
  • Evaluate a string as if it were a source code statement at runtime.
  • Create a new interpreter for the language's bytecode to give a newmeaning or purpose for a programming construct.

These features can be implemented in different ways. In MOO, reflectionforms a natural part of everyday programming idiom. When verbs (methods)are called, various variables such as verb (the name of the verb being called)and this (the object on which the verb is called) are populated to givethe context of the call. Security is typically managed by accessing thecaller stack programmatically: Since callers() is a list of the methodsby which the current verb was eventually called, performing tests on callers()(the command invoked by the original user) allows the verb to protect itselfagainst unauthorised use.

Compiled languages rely on their runtime system to provide informationabout the source code. A compiled Objective-C executable, for example,records the names of all methods in a block of the executable, providinga table to correspond these with the underlying methods (or selectors forthese methods) compiled into the program. In a compiled language thatsupports runtime creation of functions, such as Common Lisp, the runtimeenvironment must include a compiler or an interpreter.

Reflection can be implemented for languages not having built-in reflectionfacilities by using a program transformation system to define automatedsource code changes.

4.7.3   Examples
4.7.3.1   C#
Here is an equivalent example in C#
//Without reflection
Foo foo = new Foo();
foo.Hello();

//With reflection
Type t = Assembly.GetCallingAssembly().GetType("FooNamespace.Foo");
t.InvokeMember("Hello", BindingFlags.InvokeMethod, null, Activator.CreateInstance(t), null);

This next example demonstrates the use of advanced features of reflection.It loads an assembly (which can be thought of as a class library) dynamicallyand uses reflection to find the methods that take no parameters and figureout whether it was recently modified or not. To decide whether a method wasrecently modified or not, it uses a custom attribute RecentlyModified whichthe developer tags the methods with. Presence of the attribute indicatesit was recently modified. An attribute is itself implemented as a class,that derives from the Attribute class.

The program loads the assembly dynamically at runtime, and checks that theassembly supports the RecentlyModified attribute. This check is facilitatedby using another custom attribute on the entire assembly: SupportsRecentlyModified.If it is supported, names of all the methods are then retrieved. For each method,objects representing all its parameters as well as the objects representing theRecentlyAttribute are retrieved. If the attribute retrieval succeeds and theparameter retrieval fails (indicating presence of the attribute but absenceof any parameters), a match has been found.

Definition of the custom attributes
//RecentlyModified: Applicable only to methods.
//SupportsRecentlyModified: Applicable to the entire assembly.

[AttributeUsage(AttributeTargets.Method, AllowMultiple=false)]
class RecentlyModifiedAttribute : Attribute
{
    public RecentlyModifiedAttribute() { }
}

[AttributeUsage(AttributeTargets.Assembly, AllowMultiple=false)]
class SupportsRecentlyModifiedAttribute : Attribute
{
    public SupportsRecentlyModifiedAttribute() { }
}
Implementation of the method filter
static void Main(string[] args)
{
  //Load the assembly dynamically and make sure it supports the Recently Modified Attribute
    Assembly loadedAssembly = Assembly.LoadAssembly(Console.ReadLine());
    if (Attribute.GetCustomAttribute(
           loadedAssembly, typeof(SupportsRecentlyModifiedAttribute)) != null)
        //For each class in the assembly, get all the methods. Iterate through the methods
        //and find the ones that have both the RecentlyModified attribute set as well as do
        //not require any arguments.
        foreach (Type t in loadedAssembly.GetTypes())
        {
            if (t.IsClass)
                foreach (MethodInfo method in t.GetMethods())
                {
                    //Try to retrieve the RecentlyModified attribute
                    object[] rmAttribute = method.GetCustomAttributes(
                        typeof(RecentlyModifiedAttribute), false);

                    //Try to retrieve the parameters
                    ParameterInfo[] parames = method.GetParameters();
                    if (rmAttribute.Length == 1 && parames.Length == 0)
                        Console.WriteLine("{0}", method.Name);
                }
        }
}
4.7.3.2   C++

Although the language itself does not provide any support for reflection,there are some attempts based on templates, RTTI information, using debuginformation provided by the compiler, or even patching the GNU compilerto provide extra information.

4.7.3.3   Common Lisp
Here is an equivalent example in Common Lisp
;;Without reflection
(hello)

;;With reflection
(funcall (read-from-string "HELLO"))

;;or
(funcall (symbol-function (intern "HELLO")))

However, this works only for symbols in topmost lexical environment(or dynamic one). A better example, using CLOS, is:

Here is another equivalent example in Common Lisp
;;Method hello is called on instance of class foo.
(hello (make-instance 'foo))

;;or
(hello (make-instance (find-class 'foo))

This code is equivalent to Java's one. Common Lisp, like Scheme, is ableto transform lists into functions or procedures. This is the source ofthe Lisp macro ability to change the code behavior (including from othermacros) during (and after) compilation.

4.7.3.4   ECMAScript (JavaScript)
Here is an equivalent example in ECMAScript
// Without reflection
new Foo().hello()

// With reflection

// assuming that Foo resides in this
new this['Foo']()['hello']()

// or without assumption
new (eval('Foo'))()['hello']()
4.7.3.5   Java

The following is an example in Java using the Java package java.lang.reflect.Consider two pieces of code

Java Example 1
// Without reflection
Foo foo = new Foo();
foo.hello();
Java Example 2
// With reflection
Class cls = Class.forName("Foo");
Object foo = cls.newInstance();
Method method = cls.getMethod("hello", null);
method.invoke(foo, null);

Both code fragments create an instance of a class Foo and call its hello()method. The difference is that, in the first fragment, the names of theclass and method are hard-coded; it is not possible to use a class ofanother name. In the second fragment, the names of the class and methodcan easily be made to vary at runtime. The downside is that the secondversion is harder to read, and is not protected by compile-time syntaxand semantic checking. For example, if no class Foo exists, an errorwill be generated at compile time for the first version. The equivalenterror will only be generated at run time for the second version.

4.7.3.6   Perl
Here is an equivalent example in Perl
# without reflection
my $foo = Foo->new();
$foo->hello();

# with reflection
my $class  = "Foo";
my $method = "hello";
my $object = $class->new();
$object->$method();
4.7.3.7   PHP

Here is an equivalent example in PHP (check the PHP reflection documentation).

This is the non-reflective way to invoke Foo::hello
$Foo = new Foo();
$Foo->hello();

Using reflection the class and method are retrieved as reflection objectsand then used to create a new instance and invoke the method.

This is the reflective way
$f = new ReflectionClass("Foo");
$m = $f->GetMethod("hello");
$m->invoke( $f->newInstance() );
4.7.3.8   Python

Here is an equivalent example from the Python shell. Reflection is animportant part of Python and there are several ways it can be achieved,many of which do not include the use of the eval() function and itsattendant security risks:

Here is an equivalent example in Python
>>> # Class definition
>>> class Foo(object):
...     def hello(self):
...         print "Hi"
...
>>> # Instantiation
>>> foo = Foo()
>>> # Normal call
>>> foo.hello()
Hi
>>> # Evaluate a string in the context of the global namespace
>>> eval("foo.hello()", globals())
Hi
>>> # Interpret distinct instance & method names from strings
>>> instancename = 'foo'
>>> methodname = 'hello'
>>> instance = globals()[instancename]
>>> method = getattr(instance , methodname)
>>> method()
Hi
>>> # or get a method from class not from object
>>> classfoo = type(instance)
>>> unboundmethod= getattr(classfoo , methodname)
>>> unboundmethod(instance)
Hi
4.7.3.9   Ruby
Here is an equivalent example in Ruby
# without reflection
Foo.new.hello

# with reflection
Object.const_get(:Foo).new.send(:hello)
4.7.3.10   Smalltalk
Here is an equivalent example in Smalltalk
"Without reflection"
Foo new hello

"With reflection"
(Compiler evaluate: 'Foo') new perform: #hello

The class name and the method name will often be stored in variables andin practice runtime checks need to be made to ensure that it is safe toperform the operations:

With Reflection
| x y className methodName |

className := 'Foo'.
methodName := 'hello'.

x := (Compiler evaluate: className).

(x isKindOf: Class) ifTrue: [
    y := x new.

    (y respondsTo: methodName asSymbol) ifTrue: [
        y perform: methodName asSymbol
    ]
]

Smalltalk also makes use of blocks of compiled code that are passed aroundas objects. This means that generalised frameworks can be given variationsin behavior for them to execute. Blocks allow delayed and conditionalexecution. They can be parameterised. (Blocks are implemented by theclass BlockClosure).

Here is an equivalent example in C#:
X := [ :op | 99 perform: op with: 1].
   ".....then later we can execute either of:"
X value: #+   "which gives 100, or"
X value: #-   "which gives 98."

4.8   Case Tools

Computer-aided Software Engineering (CASE), in the field software engineeringis the scientific application of a set of tools and methods to a softwarewhich results in high-quality, defect-free, and maintainable software products.It also refers to methods for the development of information systems togetherwith automated tools that can be used in the software development process.

4.8.1   Overview

The term "Computer-aided software engineering" (CASE) can refer to the softwareused for the automated development of systems software, i.e., computer code.The CASE functions include analysis, design, and programming. CASE tools automatemethods for designing, documenting, and producing structured computer code in thedesired programming language.

Two key ideas of Computer-aided Software System Engineering (CASE) are:

  • the harboring of computer assistance in software development and or software maintenance processes, and
  • An engineering approach to the software development and or maintenance.

Some typical CASE tools are:

  • Configuration management tools
  • Data modeling tools
  • Model transformation tools
  • Refactoring tools
  • Source code generation tools, and
  • Unified Modeling Language

Many CASE tools not only output code but also generate other output typicalof various systems analysis and design methodologies such as

  • data flow diagram
  • entity relationship diagram
  • logical schema
  • Program specification
  • SSADM.
  • User documentation
4.8.2   History of CASE

The term CASE was originally coined by software company, Nastec Corporationof Southfield, Michigan in 1982 with their original integrated graphics andtext editor GraphiText, which also was the first microcomputer-based systemto use hyperlinks to cross-reference text strings in documents — an earlyforerunner of today's web page link. GraphiText's successor product, DesignAidwas the first microprocessor-based tool to logically and semantically evaluatesoftware and system design diagrams and build a data dictionary.

Under the direction of Albert F. Case, Jr. vice president for product managementand consulting, and Vaughn Frick, director of product management, the DesignAidproduct suite was expanded to support analysis of a wide range of structuredanalysis and design methodologies, notable Ed Yourdon and Tom DeMarco,Chris Gane & Trish Sarson, Ward-Mellor (real-time) SA/SD andWarnier-Orr (data driven).

The next entrant into the market was Excelerator from Index Technology inCambridge, Mass. While DesignAid ran on Convergent Technologies and laterBurroughs Ngen networked microcomputers, Index launched Excelerator on theIBM PC/AT platform. While, at the time of launch, and for several years,the IBM platform did not support networking or a centralized database as didthe Convergent Technologies or Burroughs machines, the allure of IBM was strong,and Excelerator came to prominence. Hot on the heels of Excelerator were a rashof offerings from companies such as Knowledgeware (James Martin, Fran Tarkentonand Don Addington), Texas Instrument's IEF and Accenture's FOUNDATION toolset(METHOD/1, DESIGN/1, INSTALL/1, FCP).

CASE tools were at their peak in the early 1990s. At the time IBM had proposedAD/Cycle which was an alliance of software vendors centered around IBM's mainframe:

The application development tools can be from several sources: from IBM,from vendors, and from the customers themselves. IBM has entered intorelationships with Bachman Information Systems, Index Technology Corporation,and Knowledgeware, Inc. wherein selected products from these vendors willbe marketed through an IBM complementary marketing program to provide offeringsthat will help to achieve complete life-cycle coverage.

With the decline of the mainframe, AD/Cycle and the Big CASE tools died off,opening the market for the mainstream CASE tools of today. Interestingly, nearlyall of the leaders of the CASE market of the early 1990s ended up being purchasedby Computer Associates, including IEW, IEF, ADW, Cayenne, and Learmonth & BurchettManagement Systems (LBMS).

There are two generations of CASE tools. The first generation of CASE toolscan be broadly classified into three groups: information generations or 4GLs,front end design/analysis tools. CASE tools are a class of software thatautomates many of the activities involved in various life cycle phases. Forexample, when establishing the functional requirements of a proposed application,prototyping tools can be used to develop graphic models of application screensto assist end users to visualize how an application will look after development.Subsequently, system designers can use automated design tools to transform theprototyped functional requirements into detailed design documents. Programmerscan then use automated code generators to convert the design documents into code.Automated tools can be used collectively, as mentioned, or individually. Forexample, prototyping tools could be used to define application requirementsthat get passed to design technicians who convert the requirements into detaileddesigns in a traditional manner using flowcharts and narrative documents,without the assistance of automated design software.

4.8.3   Classification of CASE Tools

Existing CASE Environments can be classified along 4 different dimensions :

  1. Life-Cycle Support
  2. Integration Dimension
  3. Construction Dimension
  4. Knowledge Based CASE dimension

Let us take the meaning of these dimensions along with their examples one by one :

4.8.3.1   Life-Cycle Based CASE Tools

This dimension classifies CASE Tools on the basis of the activities theysupport in the information systems life cycle. They can be classified asUpper or Lower CASE tools.

4.8.3.1.1   Upper CASE Tools

Support strategic, planning and construction of conceptual level product andignore the design aspect. They support traditional diagrammatic languagessuch as ER diagrams, Data flow diagram, Structure charts etc.

4.8.3.1.2   Lower CASE Tools

Concentrate on the back end activities of the softwarelife cycle and hence support activities like physical design, debugging,construction, testing, integration of software components, maintenance,reengineering and reverse engineering activities.

4.8.3.2   Integration Dimension

Three main CASE Integration dimension have been proposed :

  1. CASE Framework
  2. ICASE Tools
  3. Integrated Project Support Environment(IPSE)
4.8.3.3   Applications

All aspects of the software development life cycle can be supported bysoftware tools, and so the use of tools from across the spectrum can,arguably, be described as CASE; from project management software throughtools for business and functional analysis, system design, code storage,compilers, translation tools, test software, and so on.

However, it is the tools that are concerned with analysis and design, andwith using design information to create parts (or all) of the softwareproduct, that are most frequently thought of as CASE tools. CASE applied,for instance, to a database software product, might normally involve:

  • Modelling business / real world processes and data flow
  • Development of data models in the form of entity-relationship diagrams
  • Development of process and function descriptions
  • Production of database creation SQL and stored procedures
4.8.3.4   Risks and associated controls

Common CASE risks and associated controls include:

4.8.3.4.1   Inadequate Standardization

Linking CASE tools from different vendors(design tool from Company X, programming tool from Company Y) may bedifficult if the products do not use standardized code structures anddata classifications. File formats can be converted, but usually noteconomically. Controls include using tools from the same vendor, orusing tools based on standard protocols and insisting on demonstratedcompatibility. Additionally, if organizations obtain tools for only aportion of the development process, they should consider acquiring themfrom a vendor that has a full line of products to ensure futurecompatibility if they add more tools.

4.8.3.4.2   Unrealistic Expectations

Organizations often implement CASEtechnologies to reduce development costs. Implementing CASE strategiesusually involves high start-up costs. Generally, management must bewilling to accept a long-term payback period. Controls include requiringsenior managers to define their purpose and strategies for implementingCASE technologies.

4.8.3.4.3   Quick Implementation

Implementing CASE technologies can involve asignificant change from traditional development environments. Typically,organizations should not use CASE tools the first time on criticalprojects or projects with short deadlines because of the lengthy trainingprocess. Additionally, organizations should consider using the tools on smaller,less complex projects and gradually implementing the tools to allow moretraining time.

4.8.3.4.4   Weak Repository Controls

Failure to adequately control access toCASE repositories may result in security breaches or damage to the workdocuments, system designs, or code modules stored in the repository.Controls include protecting the repositories with appropriate access,version, and backup controls.

4.8.4   Further Considerations

So, Case tools (Computer Assisted Software Engineering) and ProgrammingFrameworks are software programs designed to assist a programmer writesoftware. Some tools include those for code formatting, code generation,data modelling, translation to another programming language, screengeneration, report writing and so on. MDBMPS is proposed to be acase tool meant in part for some of these options and more.

Though there are many good case tools, each has it's weaknesses. The onemost important to us though, is that of the need to modify generated code.Very often, the code generators provide the programmer with no control overthe code output, other than to fill in some blanks. More specifically, themanner in which the code is generated is beyond the input of theprogrammer.

Let's demonstrate with a fabricated example in COBOL (with pseudocodeexample). First, the case tool would likely start with our examplecustomer file:

Pseudocode

Use file called customer.datwith an index and create record key called customer-id

Allow file to set variable customer-record

Set variables for customer-record:

  1. customer-id
  2. customer-firstname
  3. customer-middlename
  4. customer-lastname
  5. customer-address
  6. customer-city
  7. customer-state
  8. customer-zip
  9. customer-zip4
Real World (COBOL)
ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.
FILE-CONTROL.

    SELECT CUSTOMER-FILE
        ASSIGN TO "customer.dat"
        ORGANIZATION IS INDEXED
        RECORD KEY IS CUSTOMER-ID
        ACCESS MODE IS DYNAMIC.

DATA DIVISION.
FILE SECTION.

FD  CUSTOMER-FILE
    LABEL RECORDS ARE STANDARD.
01  CUSTOMER-RECORD.
    05  CUSTOMER-ID                        PIC 9(10).
    05  CUSTOMER-FIRSTNAME                 PIC X(50).
    05  CUSTOMER-MIDDLENAME                PIC X(50).
    05  CUSTOMER-LASTNAME                  PIC X(50).
    05  CUSTOMER-ADDRESS                   PIC X(50).
    05  CUSTOMER-CITY                      PIC X(50).
    05  CUSTOMER-STATE                     PIC X(2).
    05  CUSTOMER-ZIP                       PIC 9(5).
    05  CUSTOMER-ZIP4                      PIC 9(4).

Now, to that the case tool would add a place for the user intput variables:

Pseudocode

Set variables for input-record:

  1. customer-id
  2. customer-firstname
  3. customer-middlename
  4. customer-lastname
  5. customer-address
  6. customer-city
  7. customer-state
  8. customer-zip
  9. customer-zip4
Real World (COBOL)
WORKING-STORAGE SECTION.

01  INPUT-RECORD.
    05  INPUT-ID                        PIC 9(10).
    05  INPUT-FIRSTNAME                 PIC X(50).
    05  INPUT-MIDDLENAME                PIC X(50).
    05  INPUT-LASTNAME                  PIC X(50).
    05  INPUT-ADDRESS                   PIC X(50).
    05  INPUT-CITY                      PIC X(50).
    05  INPUT-STATE                     PIC X(2).
    05  INPUT-ZIP                       PIC 9(5).
    05  INPUT-ZIP4                      PIC 9(4).

After the user input from a screen the case tool would transfer thevariables from the input variables to the file using one of these twomethods:

Pseudocode

Move variable input-id to customer-id

Move variable input-firstname to customer-firstname

Move variable input-middlename to customer-middlename

Move variable input-lastname to customer-lastname

Move variable input-address to customer-address

Move variable input-city to customer-city

Move variable input-state to customer-state

Move variable input-zip to customer-zip

Move variable input-zip4 to customer-zip4

Or,

Move all variables for input-record to customer-record

Real World (COBOL)
MOVE INPUT-ID         TO CUSTOMER-ID
MOVE INPUT-FIRSTNAME  TO CUSTOMER-FIRSTNAME
MOVE INPUT-MIDDLENAME TO CUSTOMER-MIDDLENAME
MOVE INPUT-LASTNAME   TO CUSTOMER-LASTNAME
MOVE INPUT-ADDRESS    TO CUSTOMER-ADDRESS
MOVE INPUT-CITY       TO CUSTOMER-CITY
MOVE INPUT-STATE      TO CUSTOMER-STATE
MOVE INPUT-ZIP        TO CUSTOMER-ZIP
MOVE INPUT-ZIP4       TO CUSTOMER-ZIP4

OR

MOVE INPUT-RECORD     TO CUSTOMER-RECORD

Based on our layout provided earlier this is perfectly legal COBOL. However,suppose that the program requirements included (insanely?) that any missingmiddle name contain the value "NOT ENTERED". The programmer would have togenerate the code. Then edit the section after the last MOVE command(regardless of whether the multiple or single move method was used). Hewould add the following code:

Pseudocode
If variable input-middlename is all spaces (no input) thenMove the text 'not entered' into variable customer-middlenameend the if
Real World (COBOL)
IF INPUT-MIDDLENAME EQUALS SPACES THEN
  MOVE 'NOT ENTERED' TO CUSTOMER-MIDDLENAME
END-IF

As we can see this certainly solves the problem. However, a major flaw hereis revealed. Should changes be made in the case tool, and the code beregenerated, we are right back to the original output demonstrated above.The obvious result is that the programmer has to once again make the verysame change. So while saving time in one instance, a great waste of time isgenerated in another. MDBMPS accounts for this through the use ofEmbedded Code.

This is only one flaw of most case tools. A more common complaint is thatthe code generated in the first place is unappealing to the programmer.Since programming styles vary between programmers and even organizations,one can expect a certain dissatisfaction when having to work with the codingstyle of another. In some cases the generated code is simply unacceptable,following the worst of coding standards. MDBMPS proposes to solvethis problem through the use of Templates and Snippets.

5   Database

A database is a structured collection of records or data that is stored in acomputer system. The structure is achieved by organizing the data accordingto a database model. The model in most common use today is the relationalmodel. Other models such as the hierarchical model and the network modeluse a more explicit representation of relationships.

5.1   Architecture

Depending on the intended use, there are a number of database architecturesin use. Many databases use a combination of strategies. On-line TransactionProcessing systems (OLTP) often use a row-oriented datastore architecture,while data-warehouse and other retrieval-focused applications like Google'sBigTable, or bibliographic database (library catalogue) systems may use aColumn-oriented DBMS architecture.

Document-Oriented, XML, knowledgebases, as well as frame databases andRDF-stores (aka triple-stores), may also use a combination of thesearchitectures in their implementation.

Finally, it should be noted that not all database have or need a database'schema' (so called schema-less databases).

Over many years the database industry has been dominated by General Purposedatabase systems, which offer a wide range of functions that are applicableto many, if not most circumstances in modern data processing. These havebeen enhanced with extensible datatypes, pioneered in the PostgreSQLproject, to allow a very wide range of applications to be developed.

There are also other types of database which cannot be classified asrelational databases.

5.2   Database management systems

A computer database relies upon software to organize the storage of data.This software is known as a database management system (DBMS). Databasemanagement systems are categorized according to the database model that theysupport. The model tends to determine the query languages that are availableto access the database. A great deal of the internal engineering of a DBMS,however, is independent of the data model, and is concerned with managingfactors such as performance, concurrency, integrity, and recovery fromhardware failures. In these areas there are large differences betweenproducts.

A Relational Database Management System (RDBMS) implements the features ofthe relational model outlined above. In this context, Date's "InformationPrinciple" states: "the entire information content of the database isrepresented in one and only one way. Namely as explicit values in columnpositions (attributes) and rows in relations (tuples). Therefore, there areno explicit pointers between related tables."

5.3   Database models

A database model or database schema is the structure or format of adatabase, described in a formal language supported by the databasemanagement system. Schemas are generally stored in a data dictionary.

Although a schema is defined in text database language, the term is oftenused to refer to a graphical depiction of the database structure.

A database model is a specific type of data model.

Database_Models

5.3.1   Overview

A database model is a theory or specification describing how a database isstructured and used. Several such models have been suggested.

Common models include:

  • Hierarchical model
  • Network model
  • Relational model
  • Entity-relationship
  • Object-relational model
  • Object model

A data model is not just a way of structuring data: it also defines a setof operations that can be performed on the data. The relational model, forexample, defines operations such as select, project, and join. Althoughthese operations may not be explicit in a particular query language, theyprovide the foundation on which a query language is built. Examples ofdatabases include word processors, dictionary, mail merges, userpreferences and information management systems.

Models

Various techniques are used to model data structure. Most database systemsare built around one particular data model, although it is increasinglycommon for products to offer support for more than one model. For any onelogical model various physical implementations may be possible, and mostproducts will offer the user some level of control in tuning the physicalimplementation, since the choices that are made have a significant effecton performance. An example of this is the relational model: all seriousimplementations of the relational model allow the creation of indexes whichprovide fast access to rows in a table if the values of certain columns areknown.

5.3.2   Flat model/Flat File Model.

This may not strictly qualify as a data model, as defined above.

The flat (or table) model consists of a single, two-dimensional array ofdata elements, where all members of a given column are assumed to besimilar values, and all members of a row are assumed to be related to oneanother. For instance, columns for name and password that might be used asa part of a system security database. Each row would have the specificpassword associated with an individual user. Columns of the table often havea type associated with them, defining them as character data, date or timeinformation, integers, or floating point numbers.

Flat_File_Model

5.3.3   Hierarchical Model.

A hierarchical data model is a data model in which the data is organizedinto a tree-like structure. The structure allows repeating informationusing parent/child relationships: each parent can have many children buteach child only has one parent. All attributes of a specific record arelisted under an entity type.

In a database, an entity type is the equivalent of a table; each individualrecord is represented as a row and an attribute as a column. Entity typesare related to each other using 1: N mapping, also known as one-to-manyrelationships. The most recognized example of hierarchical model databaseis an IMS designed by IBM.

In a hierarchical model, data is organized into a tree-like structure,implying a single upward link in each record to describe the nesting, anda sort field to keep the records in a particular order in each same-levellist. Hierarchical structures were widely used in the early mainframedatabase management systems, such as the Information Management System(IMS) by IBM, and now describe the structure of XML documents. Thisstructure allows one 1:N relationship between two types of data. Thisstructure is very efficient to describe many relationships in the realworld; recipes, table of contents, ordering of paragraphs/verses, anynested and sorted information. However, the hierarchical structure isinefficient for certain database operations when a full path (as opposedto upward link and sort field) is not also included for each record.

One limitation of the hierarchical model is its inability to efficientlyrepresent redundancy in data. Entity-Attribute-Value database models likeCaboodle by Swink are based on this structure.

Parent–child relationship: Child may only have one parent but a parent canhave multiple children. Parents and children are tied together by linkscalled "pointers“. A parent will have a list of pointers to each of theirchildren.

History

Prior to the development of the first database management system (DBMS),access to data was provided by application programs that accessed flatfiles. Data integrity problems and the inability of such file processingsystems to represent logical data relationships lead to the first datamodel: the hierarchical data model. This model, which was implementedprimarily by IBM's Information Management System (IMS) only allowsone-to-one or one-to-many relationships between entities. Any entity at themany end of the relationship can be related only to one entity at the oneend.

Example

An example of a hierarchical data model would be if an organization hadrecords of employees in a table (entity type) called "Employees". In thetable there would be attributes/columns such as First Name, Last Name,Job Name and Wage. The company also has data about the employee’s childrenin a separate table called "Children" with attributes such as First Name,Last Name, and date of birth. The Employee table represents a parentsegment and the Children table represents a Child segment. These twosegments form a hierarchy where an employee may have many children, buteach child may only have one parent.

Consider the following structure:

EmpNoDesignationReportsTo
10Director 
20Senior Manager10
30Typist20
40Programmer20

In this, the "child" is the same type as the "parent". The hierarchystating EmpNo 10 is boss of 20, and 30 and 40 each report to 20 isrepresented by the "ReportsTo" column. In Relational database terms, theReportsTo column is a foreign key referencing the EmpNo column. If the"child" data type were different, it would be in a different table, butthere would still be a foreign key referencing the EmpNo column of theemployees table.

This simple model is commonly known as the adjacency list model, and wasintroduced by Dr. Edgar F. Codd after initial criticisms surfaced thatthe relational model could not model hierarchical data.

Hierarchical_Model

5.3.4   Network Model.

The network model is a database model conceived as a flexible way ofrepresenting objects and their relationships.

The network model original inventor was Charles Bachman, and it wasdeveloped into a standard specification published in 1969 by the CODASYLConsortium.

The network model (defined by the CODASYL specification) organizes datausing two fundamental constructs, called records and sets. Records containfields (which may be organized hierarchically, as in the programminglanguage COBOL). Sets (not to be confused with mathematical sets) defineone-to-many relationships between records: one owner, many members. Arecord may be an owner in any number of sets, and a member in any numberof sets.

The network model is a variation on the hierarchical model, to the extentthat it is built on the concept of multiple branches (lower-levelstructures) emanating from one or more nodes (higher-level structures),while the model differs from the hierarchical model in that branches canbe connected to multiple nodes. The network model is able to representredundancy in data more efficiently than in the hierarchical model.

The operations of the network model are navigational in style: a programmaintains a current position, and navigates from one record to another byfollowing the relationships in which the record participates. Records canalso be located by supplying key values.

Although it is not an essential feature of the model, network databasesgenerally implement the set relationships by means of pointers that directlyaddress the location of a record on disk. This gives excellent retrievalperformance, at the expense of operations such as database loading andreorganization.

Most object databases use the navigational concept to provide fastnavigation across networks of objects, generally using object identifiersas "smart" pointers to related objects. Objectivity/DB, for instance,implements named 1:1, 1:many, many:1 and many:many named relationships thatcan cross databases. Many object databases also support SQL, combining thestrengths of both models.

Overview

Where the hierarchical model structures data as a tree of records, witheach record having one parent record and many children, the network modelallows each record to have multiple parent and child records, forming alattice structure.

The chief argument in favour of the network model, in comparison to thehierarchic model, was that it allowed a more natural modeling ofrelationships between entities. Although the model was widely implementedand used, it failed to become dominant for two main reasons. Firstly, IBMchose to stick to the hierarchical model with semi-network extensions intheir established products such as IMS and DL/I. Secondly, it was eventuallydisplaced by the relational model, which offered a higher-level, moredeclarative interface. Until the early 1980s the performance benefits of thelow-level navigational interfaces offered by hierarchical and networkdatabases were persuasive for many large-scale applications, but as hardwarebecame faster, the extra productivity and flexibility of the relationalmodel led to the gradual obsolescence of the network model in corporateenterprise usage.

Some Well-known Network Databases

  • TurboIMAGE
  • IDMS
  • RDM Embedded

History

In 1969, the Conference on Data Systems Languages (CODASYL) establishedthe first specification of the network database model. This was followedby a second publication in 1971, which became the basis for mostimplementations. Subsequent work continued into the early 1980s,culminating in an ISO specification, but this had little influence onproducts.

Network_Model

5.3.5   Relational model

The relational model was introduced by E. F. Codd in 1970 as a way to makedatabase management systems more independent of any particular application.It is a mathematical model defined in terms of predicate logic and settheory.

The products that are generally referred to as relational databases infact implement a model that is only an approximation to the mathematicalmodel defined by Codd. Three key terms are used extensively in relationaldatabase models: relations, attributes, and domains. A relation is a tablewith columns and rows. The named columns of the relation are calledattributes, and the domain is the set of values the attributes are allowedto take.

The basic data structure of the relational model is the table, whereinformation about a particular entity (say, an employee) is representedin columns and rows (also called tuples). Thus, the "relation" in"relational database" refers to the various tables in the database; arelation is a set of tuples. The columns enumerate the various attributesof the entity (the employee's name, address or phone number, for example),and a row is an actual instance of the entity (a specific employee) that isrepresented by the relation. As a result, each tuple of the employee tablerepresents various attributes of a single employee.

All relations (and, thus, tables) in a relational database have to adhereto some basic rules to qualify as relations. First, the ordering of columnsis immaterial in a table. Second, there can't be identical tuples or rowsin a table. And third, each tuple will contain a single value for each ofits attributes.

A relational database contains multiple tables, each similar to the one inthe "flat" database model. One of the strengths of the relational model isthat, in principle, any value occurring in two different records (belongingto the same table or to different tables), implies a relationship amongthose two records. Yet, in order to enforce explicit integrity constraints,relationships between records in tables can also be defined explicitly, byidentifying or non-identifying parent-child relationships characterized byassigning cardinality (1:1, (0)1:M, M:M). Tables can also have a designatedsingle attribute or a set of attributes that can act as a "key", which canbe used to uniquely identify each tuple in the table.

A key that can be used to uniquely identify a row in a table is called aprimary key. Keys are commonly used to join or combine data from two or moretables. For example, an Employee table may contain a column named Locationwhich contains a value that matches the key of a Location table. Keys arealso critical in the creation of indexes, which facilitate fast retrievalof data from large tables. Any column can be a key, or multiple columns canbe grouped together into a compound key. It is not necessary to define allthe keys in advance; a column can be used as a key even if it was notoriginally intended to be one.

A key that has an external, real-world meaning (such as a person's name, abook's ISBN, or a car's serial number) is sometimes called a "natural" key.If no natural key is suitable (think of the many people named Brown), anarbitrary or surrogate key can be assigned (such as by giving employees IDnumbers). In practice, most databases have both generated and natural keys,because generated keys can be used internally to create links between rowsthat cannot break, while natural keys can be used, less reliably, forsearches and for integration with other databases. (For example, records intwo independently developed databases could be matched up by social securitynumber, except when the social security numbers are incorrect, missing, orhave changed.)

Relational_Model_2

5.3.6   Dimensional model

The dimensional model is a specialized adaptation of the relational modelused to represent data in data warehouses in a way that data can be easilysummarized using OLAP queries. In the dimensional model, a database consistsof a single large table of facts that are described using dimensions andmeasures. A dimension provides the context of a fact (such as whoparticipated, when and where it happened, and its type) and is used inqueries to group related facts together. Dimensions tend to be discrete andare often hierarchical; for example, the location might include thebuilding, state, and country. A measure is a quantity describing the fact,such as revenue. It's important that measures can be meaningfullyaggregated - for example, the revenue from different locations can beadded together.

In an OLAP query, dimensions are chosen and the facts are grouped and addedtogether to create a summary.

The dimensional model is often implemented on top of the relational modelusing a star schema, consisting of one table containing the facts andsurrounding tables containing the dimensions. Particularly complicateddimensions might be represented using multiple tables, resulting in asnowflake schema.

A data warehouse can contain multiple star schemas that share dimensiontables, allowing them to be used together. Coming up with a standard setof dimensions is an important part of dimensional modeling.

5.3.7   Object database models

In recent years, the object-oriented paradigm has been applied to databasetechnology, creating a new programming model known as object databases.These databases attempt to bring the database world and the applicationprogramming world closer together, in particular by ensuring that thedatabase uses the same type system as the application program. This aims toavoid the overhead (sometimes referred to as the impedance mismatch) ofconverting information between its representation in the database (forexample as rows in tables) and its representation in the application program(typically as objects). At the same time, object databases attempt tointroduce the key ideas of object programming, such as encapsulation andpolymorphism, into the world of databases.

A variety of these ways have been tried for storing objects in a database.Some products have approached the problem from the application programmingend, by making the objects manipulated by the program persistent. This alsotypically requires the addition of some kind of query language, sinceconventional programming languages do not have the ability to find objectsbased on their information content. Others have attacked the problem fromthe database end, by defining an object-oriented data model for thedatabase, and defining a database programming language that allows fullprogramming capabilities as well as traditional query facilities.

Object databases suffered because of a lack of standardization: althoughstandards were defined by ODMG, they were never implemented well enough toensure interoperability between products. Nevertheless, object databaseshave been used successfully in many applications: usually specializedapplications such as engineering databases or molecular biology databasesrather than mainstream commercial data processing. However, object databaseideas were picked up by the relational vendors and influenced extensionsmade to these products and indeed to the SQL language.

Object_Oriented_Model

5.3.8   Object-Relational Models

An object-relational database (ORD) or object-relational database managementsystem (ORDBMS) is a database management system (DBMS) similar to arelational database, but with an object-oriented database model: objects,classes and inheritance are directly supported in database schemas and inthe query language. In addition, it supports extension of the data modelwith custom data-types and methods.

An object-relational database can be said to provide a middle ground betweenrelational databases and object-oriented databases (OODBMS). Inobject-relational databases, the approach is essentially that of relationaldatabases: the data resides in the database and is manipulated collectivelywith queries in a query language; at the other extreme are OODBMSes in whichthe database is essentially a persistent object store for software writtenin an object-oriented programming language, with a programming API forstoring and retrieving objects, and little or no specific support forquerying.

Overview

One aim for the Object-relational database is to bridge the gap betweenconceptual data modeling techniques such as Entity-relationship diagram(ERD) and object-relational mapping (ORM), which often use classes andinheritance, and relational databases, which do not directly supportthem.

Another, related, aim is to bridge the gap between relational databases andthe object-oriented modeling techniques used in programming languages suchas Java, C++ or C#. However, a more popular alternative for achieving sucha bridge is to use a standard relational database systems with some form ofORM software.

Whereas traditional RDBMS or SQL-DBMS products focused on the efficientmanagement of data drawn from a limited set of data-types (defined by therelevant language standards), an object-relational DBMS allowssoftware-developers to integrate their own types and the methods that applyto them into the DBMS. ORDBMS technology aims to allow developers to raisethe level of abstraction at which they view the problem domain. This goalis not universally shared; proponents of relational databases often arguethat object-oriented specification lowers the abstraction level.

Many SQL ORDBMSs on the market today are extensible with user-defined types(UDT) and custom-written functions (e.g. stored procedures). Some (e.g.Microsoft SQL Server) allow such functions to be written in object-orientedprogramming languages, but this by itself doesn't make them object-orienteddatabases; in an object-oriented database, object orientation is a featureof the data model.

History

Object-relational database management systems grew out of research thatoccurred in the early 1990s. That research extended existing relationaldatabase concepts by adding object concepts. The researchers aimed to retaina declarative query-language based on predicate calculus as a centralcomponent of the architecture. Probably the most notable research project,Postgres (UC Berkeley), spawned two products tracing their lineage to thatresearch: Illustra and PostgreSQL.

In the mid-1990s, early commercial products appeared. These includedIllustra (Illustra Information Systems, acquired by Informix Software whichwas in turn acquired by IBM), Omniscience (Omniscience Corporation, acquiredby Oracle Corporation and became the original Oracle Lite), and UniSQL(UniSQL, Inc., acquired by KCOMS). Ukrainian developer Ruslan Zasukhin,founder of Paradigma Software, Inc., developed and shipped the first versionof Valentina database in the mid-1990s as a C++ SDK. By the next decade,PostgreSQL had become a commercially viable database and is the basis forseveral products today which maintain its ORDBMS features.

Computer scientists came to refer to these products as "object-relationaldatabase management systems" or ORDBMSs.

Many of the ideas of early object-relational database efforts have largelybecome incorporated into SQL:1999. In fact, any product that adheres to theobject-oriented aspects of SQL:1999 could be described as anobject-relational database management product. For example, IBM's DB2,Oracle database, and Microsoft SQL Server, make claims to support thistechnology and do so with varying degrees of success.

Comparison to RDBMS

An RDBMS might commonly involve SQL statements such as these
CREATE TABLE Customers  (
    Id          CHAR(12)    NOT NULL PRIMARY KEY,
    Surname     VARCHAR(32) NOT NULL,
    FirstName   VARCHAR(32) NOT NULL,
    DOB         DATE        NOT NULL
 );
 SELECT InitCap(Surname) \|\| ', ' \|\| InitCap(FirstName)
   FROM Customers
  WHERE Month(DOB) = Month(getdate())
    AND Day(DOB) = Day(getdate())
Most current SQL databases allow the crafting of custom functions,which would allow the query to appear as
SELECT Formal(Id)
  FROM Customers
 WHERE Birthday(Id) = Today()
In an object-relational database, one might see something like this,with user-defined data-types and expressions such as BirthDay()
CREATE TABLE Customers (
  Id           Cust_Id     NOT NULL  PRIMARY KEY,
  Name         PersonName  NOT NULL,
  DOB          DATE        NOT NULL
);
SELECT Formal( C.Id )
  FROM Customers C
 WHERE BirthDay ( C.DOB ) TODAY;

The object-relational model can offer another advantage in that the databasecan make use of the relationships between data to easily collect relatedrecords. In an address book application, an additional table would be addedto the ones above to hold zero or more addresses for each user.

Using a traditional RDBMS, collecting information for boththe user and their address requires a "join"
SELECT InitCap(C.Surname) \|\| ', ' \|\| InitCap(C.FirstName), A.city
  FROM Customers C JOIN Addresses A ON A.Cust_Id=C.Id -- the join
 WHERE A.city="New York"
The same query in an object-relational database appears more simply
SELECT Formal( C.Name )
  FROM Customers C
 WHERE C.address.city="New York" -- the linkage is 'understood' by the ORDB
5.3.9   Post-relational database models

Products offering a more general data model than the relational model aresometimes classified as post-relational. The data model in such productsincorporates relations but is not constrained by the Information Principle,which requires that all information is represented by data values inrelations.

Some of these extensions to the relational model actually integrate conceptsfrom technologies that pre-date the relational model. For example, theyallow representation of a directed graph with trees on the nodes.

Some products implementing such models have been built by extendingrelational database systems with non-relational features. Others, however,have arrived in much the same place by adding relational features topre-relational systems. Paradoxically, this allows products that arehistorically pre-relational, such as PICK and MUMPS, to make a plausibleclaim to be post-relational in their current architecture.

5.4   Database storage structures

Relational database tables/indexes are typically stored in memory or onhard disk in one of many forms, ordered/unordered flat files, ISAM, heaps,hash buckets or B+ trees. These have various advantages and disadvantagesdiscussed further in the main article on this topic. The most commonly usedare B+ trees and ISAM.

Object databases use a range of storage mechanisms. Some use virtual memorymapped files to make the native language (C++, Java etc.) objectspersistent. This can be highly efficient but it can make multi-languageaccess more difficult. Others break the objects down into fixed and varyinglength components that are then clustered tightly together in fixed sizedblocks on disk and reassembled into the appropriate format either for theclient or in the client address space. Another popular technique is to storethe objects in tuples, much like a relational database, which the databaseserver then reassembles for the clients.

Other important design choices relate to the clustering of data by category(such as grouping data by month, or location), creating pre-computed viewsknown as materialized views, partitioning data by range or hash. Memorymanagement and storage topology can be important design choices for databasedesigners as well. Just as normalization is used to reduce storagerequirements and improve the extensibility of the database, converselydenormalization is often used to reduce join complexity and reduce executiontime for queries.

5.5   Indexing

All of these databases can take advantage of indexing to increase theirspeed. This technology has advanced tremendously since its early uses in the1960s and 1970s. The most common kind of index is a sorted list of thecontents of some particular table column, with pointers to the rowassociated with the value. An index allows a set of table rows matching somecriterion to be located quickly. Typically, indexes are also stored in thevarious forms of data-structure mentioned above (such as B-trees, hashes,and linked lists). Usually, a specific technique is chosen by the databasedesigner to increase efficiency in the particular case of the type of indexrequired.

Most relational DBMS's and some object DBMSs have the advantage thatindexes can be created or dropped without changing existing applicationsmaking use of it. The database chooses between many different strategiesbased on which one it estimates will run the fastest. In other words,indexes are transparent to the application or end-user querying thedatabase; while they affect performance, any SQL command will run with orwithout index to compute the result of an SQL statement. The RDBMS willproduce a plan of how to execute the query, which is generated by analyzingthe run times of the different algorithms and selecting the quickest. Someof the key algorithms that deal with joins are nested loop join, sort-mergejoin and hash join. Which of these is chosen depends on whether an indexexists, what type it is, and its cardinality.

An index speeds up access to data, but it has disadvantages as well. First,every index increases the amount of storage on the hard drive necessary forthe database file, and second, the index must be updated each time the dataare altered, and this costs time. (Thus an index saves time in the readingof data, but it costs time in entering and altering data. It thus depends onthe use to which the data are to be put whether an index is on the whole anet plus or minus in the quest for efficiency.)

A special case of an index is a primary index, or primary key, which isdistinguished in that the primary index must ensure a unique reference to arecord. Often, for this purpose one simply uses a running index number (IDnumber). Primary indexes play a significant role in relational databases,and they can speed up access to data considerably.

5.6   Transactions and concurrency

In addition to their data model, most practical databases ("transactionaldatabases") attempt to enforce a database transaction. Ideally, the databasesoftware should enforce the ACID rules, summarized here:

Atomicity: Either all the tasks in a transaction must be done, or noneof them. The transaction must be completed, or else it must be undone(rolled back).

Consistency: Every transaction must preserve the integrityconstraints - the declared consistency rules - of the database. It cannotplace the data in a contradictory state.

Isolation: Two simultaneous transactions cannot interfere with oneanother. Intermediate results within a transaction are not visible toother transactions.

Durability: Completed transactions cannot be aborted later or theirresults discarded. They must persist through (for instance) restarts of theDBMS after crashes

In practice, many DBMSs allow most of these rules to be selectivelyrelaxed for better performance.

Concurrency control is a method used to ensure that transactions areexecuted in a safe manner and follow the ACID rules. The DBMS must beable to ensure that only serializable, recoverable schedules are allowed,and that no actions of committed transactions are lost while undoing abortedtransactions.

5.7   Replication

Replication of databases is closely related to transactions. If a databasecan log its individual actions, it is possible to create a duplicate of thedata in real time. The duplicate can be used to improve performance oravailability of the whole database system. Common replication conceptsinclude:

  • Master/Slave Replication: All write requests are performed on the master and then replicated to the slaves
  • Quorum: The result of Read and Write requests are calculated by querying a "majority" of replicas.
  • Multimaster: Two or more replicas sync each other via a transaction identifier.

Parallel synchronous replication of databases enables transactions to bereplicated on multiple servers simultaneously, which provides a method forbackup and security as well as data availability.

5.8   Security

Database security denotes the system, processes, and procedures thatprotect a database from unintended activity.

Security is usually enforced through access control, auditing, andencryption.

  • Access control ensures and restricts who can connect and what can be done to the database.
  • Auditing logs what action or change has been performed, when and by whom.
  • Encryption: Since security has become a major issue in recent years, many commercial database vendors provide built-in encryption mechanisms. Data is encoded natively into the tables and deciphered "on the fly" when a query comes in. Connections can also be secured and encrypted if required using DSA, MD5, SSL or legacy encryption standard.

Enforcing security is one of the major tasks of the DBA.

In the United Kingdom, legislation protecting the public from unauthorizeddisclosure of personal information held on databases falls under the Officeof the Information Commissioner. United Kingdom based organizations holdingpersonal data in electronic format (databases for example) are required toregister with the Data Commissioner.

5.9   Locking

Locking is how the database handles multiple concurrent operations. This ishow concurrency and some form of basic integrity is managed within thedatabase system. Such locks can be applied on a row level, or on otherlevels like page (a basic data block), extent (multiple array of pages)or even an entire table. This helps maintain the integrity of the data byensuring that only one process at a time can modify the same data.

In basic filesystem files or folders, only one lock at a time can be set,restricting the usage to one process only. Databases, on the other hand,can set and hold mutiple locks at the same time on the different level ofthe physical data structure. How locks are set, last is determined by thedatabase engine locking scheme based on the submitted SQL or transactionsby the users. Generally speaking, no activity on the database should betranslated by no or very light locking.

For most DBMS systems existing on the market, locks are generally shared orexclusive. Exclusive locks mean that no other lock can acquire the currentdata object as long as the exclusive lock lasts. Exclusive locks are usuallyset while the database needs to change data, like during an UPDATE orDELETE operation.

Shared locks can take ownership one from the other of the current datastructure. Shared locks are usually used while the database is reading data,during a SELECT operation. The number, nature of locks and time the lockholds a data block can have a huge impact on the database performances. Badlocking can lead to disastrous performance response (usually the result ofpoor SQL requests, or inadequate database physical structure)

Default locking behavior is enforced by the isolation level of thedataserver. Changing the isolation level will affect how shared or exclusivelocks must be set on the data for the entire database system. Defaultisolation is generally 1, where data can not be read while it is modified,forbidding to return "ghost data" to end user.

At some point intensive or inappropriate exclusive locking, can lead to the"dead lock" situation between two locks. Where none of the locks can bereleased because they try to acquire resources mutually from each other. TheDatabase has a fail safe mechanism and will automatically "sacrifice" one ofthe locks releasing the resource. Doing so processes or transactionsinvolved in the "dead lock" will be rolled back.

Databases can also be locked for other reasons, like access restrictions forgiven levels of user. Some databases are also locked for routine databasemaintenance, which prevents changes being made during the maintenance. See"Locking tables and databases" (section in some documentation / explanationfrom IBM) for more detail.) However, many modern databases don't lock thedatabase during routine maintenance. e.g. "Routine Database Maintenance" forPostgreSQL.

5.10   Applications of databases

Databases are used in many applications, spanning virtually the entire rangeof computer software. Databases are the preferred method of storage forlarge multiuser applications, where coordination between many users isneeded. Even individual users find them convenient, and many electronic mailprograms and personal organizers are based on standard database technology.Software database drivers are available for most database platforms so thatapplication software can use a common Application Programming Interface toretrieve the information stored in a database. Two commonly used databaseAPIs are JDBC and ODBC.

5.11   Relational Databases

Relational databases deserve special attention, because the project will bebased upon them.

Overview

The relational model for database management is a database model based onfirst-order predicate logic, first formulated and proposed in 1969 by EdgarCodd.

Its core idea is to describe a database as a collection of predicates overa finite set of predicate variables, describing constraints on the possiblevalues and combinations of values. The content of the database at any giventime is a finite (logical) model of the database, i.e. a set of relations,one per predicate variable, such that all predicates are satisfied. Arequest for information from the database (a database query) is also apredicate.

The purpose of the relational model is to provide a declarative method forspecifying data and queries: we directly state what information the databasecontains and what information we want from it, and let the databasemanagement system software take care of describing data structures forstoring the data and retrieval procedures for getting queries answered.

IBM implemented Codd's ideas with the DB2 database management system; itintroduced the SQL data definition and query language. Other relationaldatabase management systems followed, most of them using SQL as well. Atable in an SQL database schema corresponds to a predicate variable; thecontents of a table to a relation; key constraints, other constraints, andSQL queries correspond to predicates. However, it must be noted that SQLdatabases, including DB2, deviate from the relational model in many details;Codd fiercely argued against deviations that compromise the originalprinciples.

Relational_model_concepts

Relational_key

Alternatives to the relational model

Other models are the hierarchical model and network model. Some systemsusing these older architectures are still in use today in data centers withhigh data volume needs or where existing systems are so complex andabstract it would be cost prohibitive to migrate to systems employing therelational model; also of note are newer object-oriented databases.A recent development is the Object-Relation type-Object model, which isbased on the assumption that any fact can be expressed in the form of oneor more binary relationships. The model is used in Object Role Modeling(ORM), RDF/Notation 3 (N3) and in Gellish English.

The relational model was the first formal database model. After it wasdefined, informal models were made to describe hierarchical databases (thehierarchical model) and network databases (the network model). Hierarchicaland network databases existed before relational databases, but were onlydescribed as models after the relational model was defined, in order toestablish a basis for comparison.

Implementation

There have been several attempts to produce a true implementation of therelational database model as originally defined by Codd and explained byDate, Darwen and others, but none have been popular successes so far.Rel is one of the more recent attempts to do this.

History

The relational model was invented by E.F. (Ted) Codd as a general model ofdata, and subsequently maintained and developed by Chris Date and HughDarwen among others. In The Third Manifesto (first published in 1995) Dateand Darwen show how the relational model can accommodate certain desiredobject-oriented features.

Controversies

Codd himself, some years after publication of his 1970 model, proposed athree-valued logic (True, False, Missing or NULL) version of it in order todeal with missing information, and in his The Relational Model for DatabaseManagement Version 2 (1990) he went a step further with a four-valued logic(True, False, Missing but Applicable, Missing but Inapplicable) version.But these have never been implemented, presumably because of attendingcomplexity. SQL's NULL construct was intended to be part of a three-valuedlogic system, but fell short of that due to logical errors in the standardand in its implementations.

5.11.1   The Model

The fundamental assumption of the relational model is that all data isrepresented as mathematical n-ary relations, an n-ary relation being asubset of the Cartesian product of n domains. In the mathematical model,reasoning about such data is done in two-valued predicate logic, meaningthere are two possible evaluations for each proposition: either true orfalse (and in particular no third value such as unknown, or not applicable,either of which are often associated with the concept of NULL). Some thinktwo-valued logic is an important part of the relational model, where othersthink a system that uses a form of three-valued logic can still beconsidered relational.

Data are operated upon by means of a relational calculus or relationalalgebra, these being equivalent in expressive power.

The relational model of data permits the database designer to create aconsistent, logical representation of information. Consistency is achievedby including declared constraints in the database design, which is usuallyreferred to as the logical schema. The theory includes a process of databasenormalization whereby a design with certain desirable properties can beselected from a set of logically equivalent alternatives. The access plansand other implementation and operation details are handled by the DBMSengine, and are not reflected in the logical model. This contrasts withcommon practice for SQL DBMSs in which performance tuning often requireschanges to the logical model.

The basic relational building block is the domain or data type, usuallyabbreviated nowadays to type. A tuple is an unordered set of attributevalues. An attribute is an ordered pair of attribute name and type name.An attribute value is a specific valid value for the type of the attribute.This can be either a scalar value or a more complex type.

A relation consists of a heading and a body. A heading is a set ofattributes. A body (of an n-ary relation) is a set of n-tuples. The headingof the relation is also the heading of each of its tuples.

A relation is defined as a set of n-tuples. In both mathematics and therelational database model, a set is an unordered collection of items,although some DBMSs impose an order to their data. In mathematics, a tuplehas an order, and allows for duplication. E.F. Codd originally definedtuples using this mathematical definition. Later, it was one of E.F. Codd'sgreat insights that using attribute names instead of an ordering would be somuch more convenient (in general) in a computer language based on relations.This insight is still being used today. Though the concept has changed, thename "tuple" has not. An immediate and important consequence of thisdistinguishing feature is that in the relational model the Cartesian productbecomes commutative.

A table is an accepted visual representation of a relation; a tuple issimilar to the concept of row, but note that in the database language SQLthe columns and the rows of a table are ordered.

A relvar is a named variable of some specific relation type, to which at alltimes some relation of that type is assigned, though the relation maycontain zero tuples.

The basic principle of the relational model is the Information Principle:all information is represented by data values in relations. In accordancewith this Principle, a relational database is a set of relvars and theresult of every query is presented as a relation.

The consistency of a relational database is enforced, not by rules builtinto the applications that use it, but rather by constraints, declared aspart of the logical schema and enforced by the DBMS for all applications.In general, constraints are expressed using relational comparison operators,of which just one, "is subset of" (⊆), is theoretically sufficient. Inpractice, several useful shorthands are expected to be available, of whichthe most important are candidate key (really, superkey) and foreign keyconstraints.

5.11.2   Interpretation

To fully appreciate the relational model of data it is essential tounderstand the intended interpretation of a relation.

The body of a relation is sometimes called its extension. This is because itis to be interpreted as a representation of the extension of some predicate,this being the set of true propositions that can be formed by replacing eachfree variable in that predicate by a name (a term that designatessomething).

There is a one-to-one correspondence between the free variables of thepredicate and the attribute names of the relation heading. Each tuple of therelation body provides attribute values to instantiate the predicate bysubstituting each of its free variables. The result is a proposition that isdeemed, on account of the appearance of the tuple in the relation body, tobe true. Contrariwise, every tuple whose heading conforms to that of therelation but which does not appear in the body is deemed to be false. Thisassumption is known as the closed world assumption.

For a formal exposition of these ideas, see the section Set TheoryFormulation, below.

5.11.3   Application to databases

A type as used in a typical relational database might be the set ofintegers, the set of character strings, the set of dates, or the twoboolean values true and false, and so on. The corresponding type namesfor these types might be the strings "int", "char", "date", "boolean",etc. It is important to understand, though, that relational theory doesnot dictate what types are to be supported; indeed, nowadays provisionsare expected to be available for user-defined types in addition to thebuilt-in ones provided by the system.

Attribute is the term used in the theory for what is commonly referred toas a column. Similarly, table is commonly used in place of the theoreticalterm relation (though in SQL the term is by no means synonymous withrelation). A table data structure is specified as a list of columndefinitions, each of which specifies a unique column name and the type ofthe values that are permitted for that column. An attribute value is theentry in a specific column and row, such as "John Doe" or "35".

A tuple is basically the same thing as a row, except in an SQL DBMS, wherethe column values in a row are ordered. (Tuples are not ordered; instead,each attribute value is identified solely by the attribute name and neverby its ordinal position within the tuple.) An attribute name might be"name" or "age".

A relation is a table structure definition (a set of column definitions)along with the data appearing in that structure. The structure definitionis the heading and the data appearing in it is the body, a set of rows. Adatabase relvar (relation variable) is commonly known as a base table. Theheading of its assigned value at any time is as specified in the tabledeclaration and its body is that most recently assigned to it by invokingsome update operator (typically, INSERT, UPDATE, or DELETE). The headingand body of the table resulting from evaluation of some query are determinedby the definitions of the operators used in the expression of that query.(Note that in SQL the heading is not always a set of column definitions asdescribed above, because it is possible for a column to have no name andalso for two or more columns to have the same name. Also, the body is notalways a set of rows because in SQL it is possible for the same row toappear more than once in the same body.)

5.11.4   SQL and the relational model

SQL, initially pushed as the standard language for relational databases,deviates from the relational model in several places. The current ISO SQLstandard doesn't mention the relational model or use relational terms orconcepts. However, it is possible to create a database conforming to therelational model using SQL if one does not use certain SQL features.

The following deviations from the relational model have been noted in SQL.Note that few database servers implement the entire SQL standard and inparticular do not allow some of these deviations. Whereas NULL is nearlyubiquitous, for example, allowing duplicate column names within a table oranonymous columns is uncommon.

Duplicate rows

The same row can appear more than once in an SQL table. The same tuplecannot appear more than once in a relation.

Anonymous columns

A column in an SQL table can be unnamed and thus unable to be referenced inexpressions. The relational model requires every attribute to be named andreferenceable.

Duplicate column names

Two or more columns of the same SQL table can have the same name andtherefore cannot be referenced, on account of the obvious ambiguity. Therelational model requires every attribute to be referenceable.

Column order significance

The order of columns in an SQL table is defined and significant, oneconsequence being that SQL's implementations of Cartesian product and unionare both noncommutative. The relational model requires there to be nosignificance to any ordering of the attributes of a relation.

Views without CHECK OPTION

Updates to a view defined without CHECK OPTION can be accepted but theresulting update to the database does not necessarily have the expressedeffect on its target. For example, an invocation of INSERT can be acceptedbut the inserted rows might not all appear in the view, or an invocation ofUPDATE can result in rows disappearing from the view. The relational modelrequires updates to a view to have the same effect as if the view were abase relvar.

Columnless tables unrecognized

SQL requires every table to have at least one column, but there are tworelations of degree zero (of cardinality one and zero) and they are neededto represent extensions of predicates that contain no free variables.

NULL

This special mark can appear instead of a value wherever a value can appearin SQL, in particular in place of a column value in some row. The deviationfrom the relational model arises from the fact that the implementation ofthis ad hoc concept in SQL involves the use of three-valued logic, underwhich the comparison of NULL with itself does not yield true but insteadyields the third truth value, unknown; similarly the comparison NULL withsomething other than itself does not yield false but instead yields unknown.It is because of this behaviour in comparisons that NULL is described as amark rather than a value. The relational model depends on the law ofexcluded middle under which anything that is not true is false and anythingthat is not false is true; it also requires every tuple in a relation bodyto have a value for every attribute of that relation. This particulardeviation is disputed by some if only because E.F. Codd himself eventuallyadvocated the use of special marks and a 4-valued logic, but this was basedon his observation that there are two distinct reasons why one might want touse a special mark in place of a value, which led opponents of the use ofsuch logics to discover more distinct reasons and at least as many as 19have been noted, which would require a 21-valued logic. SQL itself uses NULLfor several purposes other than to represent "value unknown". For example,the sum of the empty set is NULL, meaning zero, the average of the emptyset is NULL, meaning undefined, and NULL appearing in the result of aLEFT JOIN can mean "no value because there is no matching row in theright-hand operand".

Concepts

SQL uses concepts "table", "column", "row" instead of "relvar", "attribute","tuple". These are not merely differences in terminology. For example, a"table" may contain duplicate rows, whereas the same tuple cannot appearmore than once in a relation.

5.11.5   Relational operations

Users (or programs) request data from a relational database by sending it aquery that is written in a special language, usually a dialect of SQL.Although SQL was originally intended for end-users, it is much more commonfor SQL queries to be embedded into software that provides an easier userinterface. Many web sites, such as Wikipedia, perform SQL queries whengenerating pages.

In response to a query, the database returns a result set, which is just alist of rows containing the answers. The simplest query is just to returnall the rows from a table, but more often, the rows are filtered in someway to return just the answer wanted.

Often, data from multiple tables are combined into one, by doing a join.Conceptually, this is done by taking all possible combinations of rows(the Cartesian product), and then filtering out everything except theanswer. In practice, relational database management systems rewrite("optimize") queries to perform faster, using a variety of techniques.

There are a number of relational operations in addition to join. Theseinclude project (the process of eliminating some of the columns),restrict (the process of eliminating some of the rows), union (a way ofcombining two tables with similar structures), difference (which liststhe rows in one table that are not found in the other), intersect (whichlists the rows found in both tables), and product (mentioned above, whichcombines each row of one table with each row of the other). Depending onwhich other sources you consult, there are a number of other operators -many of which can be defined in terms of those listed above. These includesemi-join, outer operators such as outer join and outer union, and variousforms of division. Then there are operators to rename columns, andsummarizing or aggregating operators, and if you permit relation values asattributes (RVA - relation-valued attribute), then operators such as groupand ungroup. The SELECT statement in SQL serves to handle all of theseexcept for the group and ungroup operators.

The flexibility of relational databases allows programmers to writequeries that were not anticipated by the database designers. As a result,relational databases can be used by multiple applications in ways theoriginal designers did not foresee, which is especially important fordatabases that might be used for a long time (perhaps several decades).This has made the idea and implementation of relational databases verypopular with businesses.

5.11.6   Database normalization

Relations are classified based upon the types of anomalies to whichthey're vulnerable. A database that's in the first normal form isvulnerable to all types of anomalies, while a database that's in thedomain/key normal form has no modification anomalies. Normal forms arehierarchical in nature. That is, the lowest level is the first normalform, and the database cannot meet the requirements for higher levelnormal forms without first having met all the requirements of the lessernormal forms.

5.11.7   Examples

An idealized, very simple example of a description of some relvars andtheir attributes:

  • Customer(Customer ID, Tax ID, Name, Address, City, State, Zip, Phone)
  • Order(Order No, Customer ID, Invoice No, Date Placed, Date Promised, Terms, Status)
  • Order Line(Order No, Order Line No, Product Code, Qty)
  • Invoice(Invoice No, Customer ID, Order No, Date, Status)
  • Invoice Line(Invoice No, Line No, Product Code, Qty Shipped)
  • Product(Product Code, Product Description)

In this design we have six relvars: Customer, Order, Order Line, Invoice,Invoice Line and Product. The bold, underlined attributes are candidatekeys. The non-bold, underlined attributes are foreign keys.

Usually one candidate key is arbitrarily chosen to be called the primarykey and used in preference over the other candidate keys, which are thencalled alternate keys.

A candidate key is a unique identifier enforcing that no tuple will beduplicated; this would make the relation into something else, namely abag, by violating the basic definition of a set. Both foreign keys andsuperkeys (which includes candidate keys) can be composite, that is, canbe composed of several attributes. Below is a tabular depiction of arelation of our example Customer relvar; a relation can be thought of asa value that can be attributed to a relvar.

Customer relation
Customer ID     Tax ID              Name                 Address                 [More fields....]
==================================================================================================
1234567890      555-5512222         Munmun               323 Broadway            ...
2223344556      555-5523232         SS4 Vegeta           1200 Main Street        ...
3334445563      555-5533323         Ekta                 871 1st Street          ...
4232342432      555-5325523         E. F. Codd           123 It Way              ...
1234567891      111-1231231         Mudit Khetan         Malviya Nagar, Jaipur   ...

If we attempted to insert a new customer with the ID 1234567890, thiswould violate the design of the relvar since Customer ID is a primarykey and we already have a customer 1234567890. The DBMS must reject atransaction such as this that would render the database inconsistent bya violation of an integrity constraint.

Foreign keys are integrity constraints enforcing that the value of theattribute set is drawn from a candidate key in another relation. Forexample in the Order relation the attribute Customer ID is a foreign key.A join is the operation that draws on information from several relationsat once. By joining relvars from the example above we could query thedatabase for all of the Customers, Orders, and Invoices. If we onlywanted the tuples for a specific customer, we would specify this using arestriction condition.

If we wanted to retrieve all of the Orders for Customer 1234567890, wecould query the database to return every row in the Order table withCustomer ID 1234567890 and join the Order table to the Order Line tablebased on Order No.

There is a flaw in our database design above. The Invoice relvar containsan Order No attribute. So, each tuple in the Invoice relvar will have oneOrder No, which implies that there is precisely one Order for each Invoice.But in reality an invoice can be created against many orders, or indeed forno particular order. Additionally the Order relvar contains an Invoice Noattribute, implying that each Order has a corresponding Invoice. But againthis is not always true in the real world. An order is sometimes paidthrough several invoices, and sometimes paid without an invoice. In otherwords there can be many Invoices per Order and many Orders per Invoice.This is a many-to-many relationship between Order and Invoice (also calleda non-specific relationship). To represent this relationship in the databasea new relvar should be introduced whose role is to specify thecorrespondence between Orders and Invoices:

OrderInvoice(Order No,Invoice No)

Now, the Order relvar has a one-to-many relationship to the OrderInvoicetable, as does the Invoice relvar. If we want to retrieve every Invoicefor a particular Order, we can query for all orders where Order No in theOrder relation equals the Order No in OrderInvoice, and where Invoice Noin OrderInvoice equals the Invoice No in Invoice.

5.12   (DBMS) DataBase Management System

A database management system (DBMS) is computer software that manages databases. DBMSes may use any of a variety of database models, such as the network model or relational model. In large systems, a DBMS allows users and other software to store and retrieve data in a structured way.

5.12.1   Overview

A DBMS is a set of software programs that controls the organization, storage, management, and retrieval of data in a database. DBMS are categorized according to their data structures or types. It is a set of prewritten programs that are used to store, update and retrieve a Database. The DBMS accepts requests for data from the application program and instructs the operating system to transfer the appropriate data. When a DBMS is used, information systems can be changed much more easily as the organization's information requirements change. New categories of data can be added to the database without disruption to the existing system.

Organizations may use one kind of DBMS for daily transaction processing and then move the detail onto another computer that uses another DBMS better suited for random inquiries and analysis. Overall systems design decisions are performed by data administrators and systems analysts. Detailed database design is performed by database administrators.

Database servers are specially designed computers that hold the actual databases and run only the DBMS and related software. Database servers are usually multiprocessor computers, with RAID disk arrays used for stable storage. Connected to one or more servers via a high-speed channel, hardware database accelerators are also used in large volume transaction processing environments. DBMSs are found at the heart of most database applications. Sometimes DBMSs are built around a private multitasking kernel with built-in networking support although nowadays these functions are left to the operating system.

5.12.2   History

Databases have been in use since the earliest days of electronic computing. Unlike modern systems which can be applied to widely different databases and needs, the vast majority of older systems were tightly linked to the custom databases in order to gain speed at the expense of flexibility. Originally DBMSs were found only in large organizations with the computer hardware needed to support large data sets.

1960s Navigational DBMS

As computers grew in capability, this trade-off became increasingly unnecessary and a number of general-purpose database systems emerged; by the mid-1960s there were a number of such systems in commercial use. Interest in a standard began to grow, and Charles Bachman, author of one such product, Integrated Data Store (IDS), founded the "Database Task Group" within CODASYL, the group responsible for the creation and standardization of COBOL. In 1971 they delivered their standard, which generally became known as the "Codasyl approach", and soon there were a number of commercial products based on it available.

The Codasyl approach was based on the "manual" navigation of a linked data set which was formed into a large network. When the database was first opened, the program was handed back a link to the first record in the database, which also contained pointers to other pieces of data. To find any particular record the programmer had to step through these pointers one at a time until the required record was returned. Simple queries like "find all the people in Sweden" required the program to walk the entire data set and collect the matching results. There was, essentially, no concept of "find" or "search". This might sound like a serious limitation today, but in an era when the data was most often stored on magnetic tape such operations were too expensive to contemplate anyway.

IBM also had their own DBMS system in 1968, known as IMS. IMS was a development of software written for the Apollo program on the System/360. IMS was generally similar in concept to Codasyl, but used a strict hierarchy for its model of data navigation instead of Codasyl's network model. Both concepts later became known as navigational databases due to the way data was accessed, and Bachman's 1973 Turing Award award presentation was The Programmer as Navigator. IMS is classified as a hierarchical database. IDS and IDMS, both CODASYL databases, as well as CINCOMs TOTAL database are classified as network databases.

1970s Relational DBMS

Edgar Codd worked at IBM in San Jose, California, in one of their offshoot offices that was primarily involved in the development of hard disk systems. He was unhappy with the navigational model of the Codasyl approach, notably the lack of a "search" facility which was becoming increasingly useful. In 1970, he wrote a number of papers that outlined a new approach to database construction that eventually culminated in the groundbreaking A Relational Model of Data for Large Shared Data Banks.[1]

In this paper, he described a new system for storing and working with large databases. Instead of records being stored in some sort of linked list of free-form records as in Codasyl, Codd's idea was to use a "table" of fixed-length records. A linked-list system would be very inefficient when storing "sparse" databases where some of the data for any one record could be left empty. The relational model solved this by splitting the data into a series of normalized tables, with optional elements being moved out of the main table to where they would take up room only if needed.

For instance, a common use of a database system is to track information about users, their name, login information, various addresses and phone numbers. In the navigational approach all of these data would be placed in a single record, and unused items would simply not be placed in the database. In the relational approach, the data would be normalized into a user table, an address table and a phone number table (for instance). Records would be created in these optional tables only if the address or phone numbers were actually provided.

Linking the information back together is the key to this system. In the relational model, some bit of information was used as a "key", uniquely defining a particular record. When information was being collected about a user, information stored in the optional (or related) tables would be found by searching for this key. For instance, if the login name of a user is unique, addresses and phone numbers for that user would be recorded with the login name as its key. This "re-linking" of related data back into a single collection is something that traditional computer languages are not designed for.

Just as the navigational approach would require programs to loop in order to collect records, the relational approach would require loops to collect information about any one record. Codd's solution to the necessary looping was a set-oriented language, a suggestion that would later spawn the ubiquitous SQL. Using a branch of mathematics known as tuple calculus, he demonstrated that such a system could support all the operations of normal databases (inserting, updating etc.) as well as providing a simple system for finding and returning sets of data in a single operation.

Codd's paper was picked up by two people at Berkeley, Eugene Wong and Michael Stonebraker. They started a project known as INGRES using funding that had already been allocated for a geographical database project, using student programmers to produce code. Beginning in 1973, INGRES delivered its first test products which were generally ready for widespread use in 1979. During this time, a number of people had moved "through" the group — perhaps as many as 30 people worked on the project, about five at a time. INGRES was similar to System R in a number of ways, including the use of a "language" for data access, known as QUEL — QUEL was in fact relational, having been based on Codd's own Alpha language, but has since been corrupted to follow SQL, thus violating much the same concepts of the relational model as SQL itself.

IBM itself did only one test implementation of the relational model, PRTV, and a production one, Business System 12, both now discontinued. Honeywell did MRDS for Multics, and now there are two new implementations: Alphora Dataphor and Rel. All other DBMS implementations usually called relational are actually SQL DBMSs. In 1968, the University of Michigan began development of the Micro DBMS relational database management system. It was used to manage very large data sets by the US Department of Labor, the Environmental Protection Agency and researchers from University of Alberta, the University of Michigan and Wayne State University. It ran on mainframe computers using Michigan Terminal System. The system remained in production until 1996.

End 1970s SQL DBMS

IBM started working on a prototype system loosely based on Codd's concepts as System R in the early 1970s — unfortunately, System R was conceived as a way of proving Codd's ideas unimplementable[citation needed], and thus the project was delivered to a group of programmers who were not under Codd's supervision, never understood his ideas fully and ended up violating several fundamentals of the relational model. The first "quickie" version was ready in 1974/5, and work then started on multi-table systems in which the data could be broken down so that all of the data for a record (much of which is often optional) did not have to be stored in a single large "chunk". Subsequent multi-user versions were tested by customers in 1978 and 1979, by which time a standardized query language, SQL, had been added. Codd's ideas were establishing themselves as both workable and superior to Codasyl, pushing IBM to develop a true production version of System R, known as SQL/DS, and, later, Database 2 (DB2).

Many of the people involved with INGRES became convinced of the future commercial success of such systems, and formed their own companies to commercialize the work but with an SQL interface. Sybase, Informix, NonStop SQL and eventually Ingres itself were all being sold as offshoots to the original INGRES product in the 1980s. Even Microsoft SQL Server is actually a re-built version of Sybase, and thus, INGRES. Only Larry Ellison's Oracle started from a different chain, based on IBM's papers on System R, and beat IBM to market when the first version was released in 1978.

Stonebraker went on to apply the lessons from INGRES to develop a new database, Postgres, which is now known as PostgreSQL. PostgreSQL is primarily used for global mission critical applications (the .org and .info domain name registries use it as their primary data store, as do many large companies and financial institutions).

In Sweden, Codd's paper was also read and Mimer SQL was developed from the mid-70s at Uppsala University. In 1984, this project was consolidated into an independent enterprise. In the early 1980s, Mimer introduced transaction handling for high robustness in applications, an idea that was subsequently implemented on most other DBMS.

5.12.3   DBMS building blocks

A DBMS includes of four main parts: Modeling language, data structure, database query language, and transaction mechanism:

Modeling language

A data modeling language to define the schema of each database hosted in the DBMS, according to the DBMS database model. The four most common types of organizations are the:

  • hierarchical model,
  • network model,
  • relational model, and
  • object model.

Inverted lists and other methods are also used. A given database management system may provide one or more of the four models. The optimal structure depends on the natural organization of the application's data, and on the application's requirements (which include transaction rate (speed), reliability, maintainability, scalability, and cost).

The dominant model in use today is the ad hoc one embedded in SQL, despite the objections of purists who believe this model is a corruption of the relational model, since it violates several of its fundamental principles for the sake of practicality and performance. Many DBMSs also support the Open Database Connectivity API that supports a standard way for programmers to access the DBMS.

Data structure

Data structures (fields, records, files and objects) optimized to deal with very large amounts of data stored on a permanent data storage device (which implies relatively slow access compared to volatile main memory).

Database query language

A database query language and report writer to allow users to interactively interrogate the database, analyze its data and update it according to the users privileges on data. It also controls the security of the database. Data security prevents unauthorized users from viewing or updating the database. Using passwords, users are allowed access to the entire database or subsets of it called subschemas. For example, an employee database can contain all the data about an individual employee, but one group of users may be authorized to view only payroll data, while others are allowed access to only work history and medical data.

If the DBMS provides a way to interactively enter and update the database, as well as interrogate it, this capability allows for managing personal databases. However, it may not leave an audit trail of actions or provide the kinds of controls necessary in a multi-user organization. These controls are only available when a set of application programs are customized for each data entry and updating function.

Transaction mechanism

A database transaction mechanism, that ideally would guarantee the ACID properties, in order to ensure data integrity, despite concurrent user accesses (concurrency control), and faults (fault tolerance). It also maintains the integrity of the data in the database. The DBMS can maintain the integrity of the database by not allowing more than one user to update the same record at the same time. The DBMS can help prevent duplicate records via unique index constraints; for example, no two customers with the same customer numbers (key fields) can be entered into the database. See ACID properties for more information (Redundancy avoidance).

5.12.4   DBMS Topics

Logical and physical view

A database management system provides the ability for many different users to share data and process resources. But as there can be many different users, there are many different database needs. The question now is: How can a single, unified database meet the differing requirement of so many users?

A DBMS minimizes these problems by providing two views of the database data: a logical (external) view and physical (internal) view. The logical view/user’s view, of a database program represents data in a format that is meaningful to a user and to the software programs that process those data. That is, the logical view tells the user, in user terms, what is in the database. The physical view deals with the actual, physical arrangement and location of data in the direct access storage devices(DASDs). Database specialists use the physical view to make efficient use of storage and processing resources. With the logical view users can see data differently from how they are stored, and they do not want to know all the technical details of physical storage. After all, a business user is primarily interested in using the information, not in how it is stored.

One strength of a DBMS is that while there is only one physical view of the data, there can be an endless number of different logical views. This feature allows users to see database information in a more business-related way rather than from a technical, processing viewpoint. Thus the logical view refers to the way user views data, and the physical view to the way the data are physically stored and processed...

DBMS Features and capabilities

Alternatively, and especially in connection with the relational model of database management, the relation between attributes drawn from a specified set of domains can be seen as being primary. For instance, the database might indicate that a car that was originally "red" might fade to "pink" in time, provided it was of some particular "make" with an inferior paint job. Such higher arity relationships provide information on all of the underlying domains at the same time, with none of them being privileged above the others.

Throughout recent history specialized databases have existed for scientific, geospatial, imaging, document storage and like uses. Functionality drawn from such applications has lately begun appearing in mainstream DBMSs as well. However, the main focus there, at least when aimed at the commercial data processing market, is still on descriptive attributes on repetitive record structures.

Thus, the DBMSs of today roll together frequently-needed services or features of attribute management. By externalizing such functionality to the DBMS, applications effectively share code with each other and are relieved of much internal complexity. Features commonly offered by database management systems include:

Query ability
Querying is the process of requesting attribute information from various perspectives and combinations of factors. Example: "How many 2-door cars in Texas are green?" A database query language and report writer allow users to interactively interrogate the database, analyze its data and update it according to the users privileges on data.
Backup and replication
Copies of attributes need to be made regularly in case primary disks or other equipment fails. A periodic copy of attributes may also be created for a distant organization that cannot readily access the original. DBMS usually provide utilities to facilitate the process of extracting and disseminating attribute sets. When data is replicated between database servers, so that the information remains consistent throughout the database system and users cannot tell or even know which server in the DBMS they are using, the system is said to exhibit replication transparency.
Rule enforcement
Often one wants to apply rules to attributes so that the attributes are clean and reliable. For example, we may have a rule that says each car can have only one engine associated with it (identified by Engine Number). If somebody tries to associate a second engine with a given car, we want the DBMS to deny such a request and display an error message. However, with changes in the model specification such as, in this example, hybrid gas-electric cars, rules may need to change. Ideally such rules should be able to be added and removed as needed without significant data layout redesign.
Security
Often it is desirable to limit who can see or change which attributes or groups of attributes. This may be managed directly by individual, or by the assignment of individuals and privileges to groups, or (in the most elaborate models) through the assignment of individuals and groups to roles which are then granted entitlements.
Computation
There are common computations requested on attributes such as counting, summing, averaging, sorting, grouping, cross-referencing, etc. Rather than have each computer application implement these from scratch, they can rely on the DBMS to supply such calculations.
Change and access logging
Often one wants to know who accessed what attributes, what was changed, and when it was changed. Logging services allow this by keeping a record of access occurrences and changes.
Automated optimization
If there are frequently occurring usage patterns or requests, some DBMS can adjust themselves to improve the speed of those interactions. In some cases the DBMS will merely provide tools to monitor performance, allowing a human expert to make the necessary adjustments after reviewing the statistics collected.

Meta-data repository

Metadata is data describing data. For example, a listing that describes what attributes are allowed to be in data sets is called "meta-information". The meta-data is also known as data about data.

5.13   More on Normalization

In the field of relational database design, normalization is a systematic way of ensuring that a database structure is suitable for general-purpose querying and free of certain undesirable characteristics—insertion, update, and deletion anomalies—that could lead to a loss of data integrity.[1] E.F. Codd, the inventor of the relational model, introduced the concept of normalization and what we now know as the first normal form in 1970.[2] Codd went on to define the second and third normal forms in 1971[3]; and Codd and Raymond F. Boyce defined the Boyce-Codd normal form in 1974 .[4] Higher normal forms were defined by other theorists in subsequent years, the most recent being the sixth normal form introduced by Chris Date, Hugh Darwen, and Nikos Lorentzos in 2002.[5]

Informally, a relational database table (the computerized representation of a relation) is often described as "normalized" if it is in the third normal form (3NF).[6] Most 3NF tables are free of insertion, update, and deletion anomalies, i.e. in most cases 3NF tables adhere to BCNF, 4NF, and 5NF (but typically not 6NF).

A standard piece of database design guidance is that the designer should begin by fully normalizing the design, and selectively denormalize only in places where doing so is absolutely necessary to address performance issues.[7] However, some modeling disciplines, such as the dimensional modeling approach to data warehouse design, explicitly recommend non-normalized designs, i.e. designs that in large part do not adhere to 3NF.

5.13.1   Objectives of Normalization

A basic objective of the first normal form defined by Codd in 1970 was to permit data to be queried and manipulated using a "universal data sub-language" grounded in first-order logic.[9] (SQL is an example of such a data sub-language, albeit one that Codd regarded as seriously flawed.)[10] Querying and manipulating the data within an unnormalized data structure, such as the following non-1NF representation of customers' credit card transactions, involves more complexity than is really necessary:

CustomerTransactions
IDDateAmount
Jones1289014-Oct-2003-87
1290415-Oct-2003-50
Wilkins1289814-Oct-2003-21
Stevents1290715-Oct-2003-18
1492020-Nov-2003-70
1500327-Nov-2003-60

To each customer there corresponds a repeating group of transactions. The automated evaluation of any query relating to customers' transactions therefore would broadly involve two stages:

  1. Unpacking one or more customers' groups of transactions so that the individual transactions in a group can be examined, and
  2. Deriving a query result based upon the results of the first stage

For example, if we wanted to find out the monetary sum of all transactions that occurred in October 2003 for all customers, the system would have to know that it must first unpack the Transactions group of each customer, then sum the Amounts of all transactions thus obtained where the Date of the transaction falls in October 2003.

One of Codd's important insights was that this structural complexity could always be removed completely, leading to much greater power and flexibility in the way queries could be formulated (by users and applications) and evaluated (by the DBMS). The normalized equivalent of the structure above would look like this:

CustomerTr. IDDateAmount
Jones1289014-Oct-2003-87
Jones1290415-Oct-2003-50
Wilkins1289814-Oct-2003-21
Stevens1290715-Oct-2003-18
Stevens1492020-Nov-2003-70
Stevens1500327-Nov-2003-60

Now each row represents an individual credit card transaction, and the DBMS can obtain the answer we are interested in simply by finding all rows with a Date falling in October, and summing their Amounts. All of the values in the data structure are on an equal footing: they are all exposed to the DBMS directly, and can directly participate in queries, whereas in the previous situation some values were embedded in lower-level structures that had to be handled specially. Accordingly, the normalized design lends itself to general-purpose query processing, whereas the unnormalized design does not.

The objectives of normalization beyond 1NF were stated as follows by Codd:

  1. To free the collection of relations from undesirable insertion, update and deletion dependencies;
  2. To reduce the need for restructuring the collection of relations as new types of data are introduced, and thus increase the life span of application programs;
  3. To make the relational model more informative to users;
  4. To make the collection of relations neutral to the query statistics, where these statistics are liable to change as time goes by.

—E.F. Codd, "Further Normalization of the Data Base Relational Model"

The sections below give details of each of these objectives.

5.13.2   Free the database of modification anomalies

When an attempt is made to modify (update, insert into, or delete from) a table, undesired side-effects may follow. Not all tables can suffer from these side-effects; rather, the side-effects can only arise in tables that have not been sufficiently normalized. An insufficiently normalized table might have one or more of the following characteristics:

  • The same information can be expressed on multiple rows; therefore updates to the table may result in logical inconsistencies. For example, each record in an "Employees' Skills" table might contain an Employee ID, Employee Address, and Skill; thus a change of address for a particular employee will potentially need to be applied to multiple records (one for each of his skills). If the update is not carried through successfully—if, that is, the employee's address is updated on some records but not others—then the table is left in an inconsistent state. Specifically, the table provides conflicting answers to the question of what this particular employee's address is. This phenomenon is known as an update anomaly.
  • There are circumstances in which certain facts cannot be recorded at all. For example, each record in a "Faculty and Their Courses" table might contain a Faculty ID, Faculty Name, Faculty Hire Date, and Course Code—thus we can record the details of any faculty member who teaches at least one course, but we cannot record the details of a newly-hired faculty member who has not yet been assigned to teach any courses. This phenomenon is known as an insertion anomaly.
  • There are circumstances in which the deletion of data representing certain facts necessitates the deletion of data representing completely different facts. The "Faculty and Their Courses" table described in the previous example suffers from this type of anomaly, for if a faculty member temporarily ceases to be assigned to any courses, we must delete the last of the records on which that faculty member appears. This phenomenon is known as a deletion anomaly.
An update anomaly. Employee 519 is shown ashaving different addresses on different records.
Update_anomaly
An insertion anomaly. Until the new facultymember, Dr. Newsome, is assigned to teach atleast one course, his details cannot be recorded.
Insertion_anomaly
A deletion anomaly. All information about Dr.Giddens is lost when he temporarily ceases tobe assigned to any courses.
Deletion_anomaly
5.13.3   Minimize redesign when extending the database structure

When extending the structure of a database that is fully normalized, i.e. giving it the ability to accommodate new types of data, this can typically be done with minimal change to the database structure that is already there. As a result, applications interacting with the database are themselves minimally affected.

5.13.4   Make the data model more informative to users

Normalized tables, and the relationship between one normalized table and another, mirror real-world concepts and their interrelationships.

5.13.5   Avoid bias towards any particular pattern of querying

Normalized tables are suitable for general-purpose querying. This means any queries against these tables, including future queries whose details cannot be anticipated, are supported. In contrast, tables that are not normalized lend themselves to some types of queries, but not others.

5.13.6   Background to normalization: definitions
  • Functional dependency: Attribute B has a functional dependency on attribute A (i.e., A → B if, for each value of attribute A, there is exactly one value of attribute B). If value of A is repeating in tuples then value of B will also repeat. In our example, Employee Address has a functional dependency on Employee ID, because a particular Employee ID value corresponds to one and only one Employee Address value. (Note that the reverse need not be true: several employees could live at the same address and therefore one Employee Address value could correspond to more than one Employee ID. Employee ID is therefore not functionally dependent on Employee Address.) An attribute may be functionally dependent either on a single attribute or on a combination of attributes. It is not possible to determine the extent to which a design is normalized without understanding what functional dependencies apply to the attributes within its tables; understanding this, in turn, requires knowledge of the problem domain. For example, an Employer may require certain employees to split their time between two locations, such as New York City and London, and therefore want to allow Employees to have more than one Employee Address. In this case, Employee Address would no longer be functionally dependent on Employee ID.

Another way to look at the above is by reviewing basic mathematical functions:

Let F(x) be a mathematical function of one independent variable. The independent variable is analogous to the attribute A. The dependent variable (or the dependent attribute using the lingo above), and hence the term functional dependency, is the value of F(A); A is an independent attribute. As we know, mathematical functions can have only one output. Notationally speaking, it is common to express this relationship in mathematics as F(A) = B; or, B → F(A).

There are also functions of more than one independent variable--commonly, this is referred to as multivariable functions. This idea represents an attribute being functionally dependent on a combination of attributes. Hence, F(x,y,z) contains three independent variables, or independent attributes, and one dependent attribute, namely, F(x,y,z). In multivariable functions, there can only be one output, or one dependent variable, or attribute.

Trivial functional dependency
A trivial functional dependency is a functional dependency of an attribute on a superset of itself. {Employee ID, Employee Address} → {Employee Address} is trivial, as is {Employee Address} → {Employee Address}.
Full functional dependency

An attribute is fully functionally dependent on a set of attributes X if it is

  • functionally dependent on X, and
  • not functionally dependent on any proper subset of X. {Employee Address} has a functional dependency on {Employee ID, Skill}, but not a full functional dependency, because it is also dependent on {Employee ID}.
Transitive dependency
A transitive dependency is an indirect functional dependency, one in which X→Z only by virtue of X→Y and Y→Z.
Multivalued dependency
A multivalued dependency is a constraint according to which the presence of certain rows in a table implies the presence of certain other rows.
Join dependency
A table T is subject to a join dependency if T can always be recreated by joining multiple tables each having a subset of the attributes of T.
Superkey
A superkey is an attribute or set of attributes that uniquely identifies rows within a table; in other words, two distinct rows are always guaranteed to have distinct superkeys. {Employee ID, Employee Address, Skill} would be a superkey for the "Employees' Skills" table; {Employee ID, Skill} would also be a superkey.
Candidate key
A candidate key is a minimal superkey, that is, a superkey for which we can say that no proper subset of it is also a superkey. {Employee Id, Skill} would be a candidate key for the "Employees' Skills" table.
Non-prime attribute
A non-prime attribute is an attribute that does not occur in any candidate key. Employee Address would be a non-prime attribute in the "Employees' Skills" table.
Primary key
Most DBMSs require a table to be defined as having a single unique key, rather than a number of possible unique keys. A primary key is a key which the database designer has designated for this purpose.
5.13.7   Normal forms

The normal forms (abbrev. NF) of relational database theory provide criteria for determining a table's degree of vulnerability to logical inconsistencies and anomalies. The higher the normal form applicable to a table, the less vulnerable it is to inconsistencies and anomalies. Each table has a "highest normal form" (HNF): by definition, a table always meets the requirements of its HNF and of all normal forms lower than its HNF; also by definition, a table fails to meet the requirements of any normal form higher than its HNF.

The normal forms are applicable to individual tables; to say that an entire database is in normal form n is to say that all of its tables are in normal form n.

Newcomers to database design sometimes suppose that normalization proceeds in an iterative fashion, i.e. a 1NF design is first normalized to 2NF, then to 3NF, and so on. This is not an accurate description of how normalization typically works. A sensibly designed table is likely to be in 3NF on the first attempt; furthermore, if it is 3NF, it is overwhelmingly likely to have an HNF of 5NF. Achieving the "higher" normal forms (above 3NF) does not usually require an extra expenditure of effort on the part of the designer, because 3NF tables usually need no modification to meet the requirements of these higher normal forms.

The main normal forms are summarized below.

Normal FormBrief Definition
First Normal (1NF)Table represents a relation, no repeating groups.
Second Normal(2NF)No non-prime attribute in the tabile isfunctionally dependant on a part (proper subset)of a candidate key.
Third Normal (3NF)Every non-prime attribute is non-transitivelydependant on every key of teh table.
Fourth and BeyondWhat are you smokin pot?
5.13.8   Denormalization

Databases intended for Online Transaction Processing (OLTP) are typically more normalized than databases intended for Online Analytical Processing (OLAP). OLTP Applications are characterized by a high volume of small transactions such as updating a sales record at a super market checkout counter. The expectation is that each transaction will leave the database in a consistent state. By contrast, databases intended for OLAP operations are primarily "read mostly" databases. OLAP applications tend to extract historical data that has accumulated over a long period of time. For such databases, redundant or "denormalized" data may facilitate business intelligence applications. Specifically, dimensional tables in a star schema often contain denormalized data. The denormalized or redundant data must be carefully controlled during ETL processing, and users should not be permitted to see the data until it is in a consistent state. The normalized alternative to the star schema is the snowflake schema. It has never been proven that this denormalization itself provides any increase in performance, or if the concurrent removal of data constraints is what increases the performance. In many cases, the need for denormalization has waned as computers and RDBMS software have become more powerful, but since data volumes have generally increased along with hardware and software performance, OLAP databases often still use denormalized schemas.

Denormalization is also used to improve performance on smaller computers as in computerized cash-registers and mobile devices, since these may use the data for look-up only (e.g. price lookups). Denormalization may also be used when no RDBMS exists for a platform (such as Palm), or no changes are to be made to the data and a swift response is crucial.

5.13.9   Non-first normal form (NF² or N1NF)

In recognition that denormalization can be deliberate and useful, the non-first normal form is a definition of database designs which do not conform to the first normal form, by allowing "sets and sets of sets to be attribute domains" (Schek 1982). This extension is a (non-optimal) way of implementing hierarchies in relations. Some academics have dubbed this practitioner developed method, "First Ab-normal Form"; Codd defined a relational database as using relations, so any table not in 1NF could not be considered to be relational.

Consider the following table:Non-First Normal Form

PersonFavorite Colors
Bobblue, red
Janegreen, yellow, red

Assume a person has several favorite colors. Obviously, favorite colors consist of a set of colors modeled by the given table.

To transform this NF² table into a 1NF an "unnest" operator is required which extends the relational algebra of the higher normal forms. The reverse operator is called "nest" which is not always the mathematical inverse of "unnest", although "unnest" is the mathematical inverse to "nest". Another constraint required is for the operators to be bijective, which is covered by the Partitioned Normal Form (PNF).

6   Security

Security should be a top priority with all programmers. As withMDBMPS, many security issues will be solved or minimized. Here area few examples of security issues that are very prevalent.

6.1   Data security

Data security is the means of ensuring that data is kept safe fromcorruption and that access to it is suitably controlled. Thus data securityhelps to ensure privacy. It also helps in protecting personal data.

6.1.1   Data Security Technologies
6.1.1.1   Disk Encryption

Disk encryption refers to encryption technology that encrypts data on a harddisk drive. Disk encryption typically takes form in either software. Diskencryption is often referred to as on-the-fly encryption ("OTFE") ortransparent encryption

6.1.1.2   Backups

Backups are used to ensure data which is lost can be recovered.

6.1.2   International Laws and Standards
6.1.2.1   International Laws

In the UK, the Data Protection Act is used to ensure that personal data isaccessible to those whom it concerns, and provides redress to individuals ifthere are inaccuracies. This is particularly important to ensure individualsare treated fairly, for example for credit checking purposes. The DataProtection Act states that only individuals and companies with legitimateand lawful reasons can process personal information and cannot be shared.

6.1.2.2   International Standards

The International Standard ISO/IEC 17799 covers data security under thetopic of information security, and one of its cardinal principles is thatall stored information, i.e. data, should be owned so that it is clear whoseresponsibility it is to protect and control access to that data.

The Trusted Computing Group is an organization that helps standardizecomputing security technologies.

6.2   Data Validation

In computer science, data validation is the process of ensuring that aprogram operates on clean, correct and useful data. It uses routines, oftencalled "validation rules" or "check routines", that check for correctness,meaningfulness, and security of data that are input to the system. The rulesmay be implemented through the automated facilities of a data dictionary, orby the inclusion of explicit application program validation logic.

The simplest data validation verifies that the characters provided come froma valid set. For example, telephone numbers should include the digits andpossibly the characters +, -, (, and ) (plus, minus, and parentheses). Amore sophisticated data validation routine would check to see the user hadentered a valid country code, i.e., that the number of digits enteredmatched the convention for the country or area specified.

Incorrect data validation can lead to data corruption or a securityvulnerability. Data validation checks that data are valid, sensible,reasonable, and secure before they are processed.

6.2.1   Validation methods
6.2.1.1   Format or picture check

Checks that the data is in a specified format (template), e.g., dates haveto be in the format YYYY/MM/DD.

6.2.1.2   Data type checks

Checks the data type of the input and give an error message if the inputdata does not match with the chosen data type, e.g., In an input boxaccepting numeric data, if the letter 'O' was typed instead of the numberzero, an error message would appear.

6.2.1.3   Range check

Checks that the data lie within a specified range of values, e.g., the monthof a person's date of birth should lie between 1 and 12.

6.2.1.4   Limit check

Unlike range checks, data is checked for one limit only, upper OR lower,e.g., data should not be greater than 2 (>2).

6.2.1.5   Presence check

Checks that important data are actually present and have not been missedout, e.g., customers may be required to have their telephone numbers listed.

6.2.1.6   Check digits

Used for numerical data. An extra digit is added to a number which iscalculated from the digits. The computer checks this calculation when dataare entered, e.g., The ISBN for a book. The last digit is a check digitcalculated using a modulus 11 method.

6.2.1.7   Batch totals

Checks for missing records. Numerical fields may be added together for allrecords in a batch. The batch total is entered and the computer checks thatthe total is correct, e.g., add the 'Total Cost' field of a number oftransactions together.

6.2.1.8   Hash totals

This is just a batch total done on one or more numeric fields which appearsin every record, e.g., add the Telephone Numbers together for a number ofCustomers.

6.2.1.9   Spelling check

Looks for spelling and grammar errors.

6.2.1.10   Consistency Checks

Checks fields to ensure data in these fields corresponds, e.g., IfTitle = "Mr.", then Gender = "M".

6.2.1.11   Cross-system Consistency Checks

Compares data in different systems to ensure it is consistent, e.g., Theaddress for the customer with the same id is the same in both systems. Thedata may be represented differently in different systems and may need to betransformed to a common format to be compared, e.g., one system may storecustomer name in a single Name field as 'Doe, John Q', while another inthree different fields: First_Name (John), Last_Name (Doe) and Middle_Name(Quality); to compare the two, the validation engine would have to transformdata from the second system to match the data from the first, for example,using SQL: Last_Name || ', ' || First_Name || substr(Middle_Name, 1, 1)would convert the data from the second system to look like the data from thefirst 'Doe, John Q'.

6.3   Buffer Overflow

In computer security and programming, a buffer overflow, or buffer overrun,is an anomalous condition where a process attempts to store data beyond theboundaries of a fixed-length buffer. The result is that the extra dataoverwrites adjacent memory locations. The overwritten data may include otherbuffers, variables and program flow data, and may result in erratic programbehavior, a memory access exception, program termination (a crash),incorrect results or - especially if deliberately caused by a malicious user- a possible breach of system security.

Buffer overflows can be triggered by inputs specifically designed to executemalicious code or to make the program operate in an unintended way. As such,buffer overflows cause many software vulnerabilities and form the basis ofmany exploits. Sufficient bounds checking by either the programmer, thecompiler or the runtime can prevent buffer overflows.

The programming languages most commonly associated with buffer overflows areC and C++, because they provide no built-in protection against accessing oroverwriting data in any part of memory and do not check that data written toan array (the built-in buffer type) is within the boundaries of that array.

6.3.1   Technical description

A buffer overflow occurs when data written to a buffer, due to insufficientbounds checking, corrupts data values in memory addresses adjacent to theallocated buffer. Most commonly this occurs when copying strings ofcharacters from one buffer to another.

6.3.1.1   Basic example

In the following example, a program has defined two data items which areadjacent in memory: an 8-byte-long string buffer, A, and a two-byte integer,B. Initially, A contains nothing but zero bytes, and B contains the number3. Characters are one byte wide.

AB
0000000003

Now, the program attempts to store the character string "excessive" in the Abuffer, followed by a zero byte to mark the end of the string. By notchecking the length of the string, it overwrites the value of B:

AB
'e''x''c''e''s''s''i''v''e'0

Although the programmer did not intend to change B at all, B's value has nowbeen replaced by a number formed from part of the character string. In thisexample, on a big-endian system that uses ASCII, "e" followed by a zero bytewould become the number 25856. If B was the only other variable data itemdefined by the program, writing an even longer string that went past the endof B could cause an error such as a segmentation fault, terminating theprocess.

For more details on stack-based overflows, see Stack buffer overflow.
6.3.2   Exploitation

The techniques to exploit a buffer overflow vulnerability vary perarchitecture, operating system and memory region. For example, exploitationon the heap (used for dynamically allocated memory) is very different fromon the call stack.

6.3.2.1   Stack-based exploitation

A technically inclined and malicious user may exploit stack-based bufferoverflows to manipulate the program in one of several ways:

  • By overwriting a local variable that is near the buffer in memory on the stack to change the behaviour of the program which may benefit the attacker.
  • By overwriting the return address in a stack frame. Once the function returns, execution will resume at the return address as specified by the attacker, usually a user input filled buffer.
  • By overwriting a function pointer, or exception handler, which is subsequently executed.

With a method called "Trampolining", if the address of the user-supplieddata is unknown, but the location is stored in a register, then the returnaddress can be overwritten with the address of an opcode which will causeexecution to jump to the user supplied data. If the location is stored in aregister R, then a jump to the location containing the opcode for a jump R,call R or similar instruction, will cause execution of user supplied data.The locations of suitable opcodes, or bytes in memory, can be found in DLLsor the executable itself. However the address of the opcode typically cannotcontain any null characters and the locations of these opcodes can varybetween applications and versions of the operating system. The MetasploitProject is one such database of suitable opcodes, though only those found inthe Windows operating system are listed.

Stack-based buffer overflows are not to be confused with stack overflows.

6.3.2.2   Heap-based exploitation

A buffer overflow occurring in the heap data area is referred to as a heapoverflow and is exploitable in a different manner to that of stack-basedoverflows. Memory on the heap is dynamically allocated by the applicationat run-time and typically contains program data. Exploitation is performedby corrupting this data in specific ways to cause the application tooverwrite internal structures such as linked list pointers. The canonicalheap overflow technique overwrites dynamic memory allocation linkage (suchas malloc meta data) and uses the resulting pointer exchange to overwrite aprogram function pointer.

The Microsoft JPEG GDI+ vulnerability is an example of the danger a heapoverflow can represent to a computer user.

6.3.2.3   Barriers to exploitation

Manipulation of the buffer which occurs before it is read or executed maylead to the failure of an exploitation attempt. These manipulations canmitigate the threat of exploitation, but may not make it impossible.Manipulations could include conversion to upper or lower case, removal ofmetacharacters and filtering out of non-alphanumeric strings. However,techniques exist to bypass these filters and manipulations; alphanumericcode, polymorphic code, Self-modifying code and return to lib-C attacks. Thesame methods can be used to avoid detection by Intrusion detection systems.In some cases, including where code is converted into unicode, the threatof the vulnerability have been misrepresented by the disclosers as onlyDenial of Service when in fact the remote execution of arbitrary code ispossible.

6.3.2.4   Practicalities of exploitation

In real-world exploits there are a variety of issues which need to beovercome for exploits to operate reliably. Null bytes in addresses,variability in the location of shellcode, differences between differentenvironments and various counter-measures in operation.

6.3.2.5   NOP sled technique

Illustration of a NOP-sled payload on the stack.

A NOP-sled is the oldest and most widely known technique for successfullyexploiting a stack buffer overflow. It solves the problem of finding theexact address to the buffer by effectively increasing the size of the targetarea. To do this much larger sections of the stack are corrupted with theno-op machine instruction. At the end of the attacker supplied data, afterthe no-op instructions, is placed an instruction to perform a relative jumpto the top of the buffer where the shellcode is located. This collection ofno-ops is referred to as the "NOP-sled" because if the return address isoverwritten with any address within the no-op region of the buffer it will"slide" down the no-ops until it is redirected to the actual malicious codeby the jump at the end. This technique requires the attacker to guess whereon the stack the NOP-sled is instead of the comparatively small shellcode.

Because of the popularity of this technique many vendors of Intrusionprevention systems will search for this pattern of no-op machineinstructions in an attempt to detect shellcode in use. It is important tonote that a NOP-sled does not necessarily contain only traditional no-opmachine instructions; any instruction that does not corrupt the machinestate to a point where the shellcode will not run can be used in place ofthe hardware assisted no-op. As a result it has become common practice forexploit writers to compose the no-op sled with randomly chosen instructionswhich will have no real effect on the shellcode execution.

While this method greatly improves the chances that an attack will besuccessful, it is not without problems. Exploits using this techniquestill must rely on some amount of luck that they will guess offsets on thestack that are within the NOP-sled region. An incorrect guess willusually result in the target program crashing and could alert the systemadministrator to the attacker's activities. Another problem is that theNOP-sled requires a much larger amount of memory in which to hold a NOP-sledlarge enough to be of any use. This can be a problem when the allocatedsize of the affected buffer is too small and the current depth of the stackis shallow (i.e. there is not much space from the end of the current stackframe to the start of the stack). Despite its problems, the NOP-sled isoften the only method that will work for a given platform, environment, orsituation; as such it is still an important technique.

Illustration of a NOP-sled payload on the stack.
NopSled
6.3.2.6   The jump to register technique

The "jump to register" technique allows for reliable exploitation of stackbuffer overflows without the need for extra room for a NOP-sled and withouthaving to guess stack offsets. The strategy is to overwrite the returnpointer with something that will cause the program to jump to a knownpointer stored within a register which points to the controlled buffer andthus the shellcode. For example if register A contains a pointer to thestart of a buffer then any jump or call taking that register as an operandcan be used to gain control of the flow of execution.An instruction from ntdll.dll to call the DbgPrint() routine contains thei386 machine opcode for jmp esp.

In practice a program may not intentionally contain instructions to jump toa particular register. The traditional solution is to find an unintentionalinstance of a suitable opcode at a fixed location somewhere within theprogram memory. In figure E on the left you can see an example of such anunintentional instance of the i386 jmp esp instruction. The opcode for thisinstruction is FF E4. This two byte sequence can be found at a one byteoffset from the start of the instruction call DbgPrint at address0x7C941EED. If an attacker overwrites the program return address with thisaddress the program will first jump to 0x7C941EED, interpret the opcode FFE4 as the jmp esp instruction, and will then jump to the top of the stackand execute the attacker's code.

When this technique is possible the severity of the vulnerability increasesconsiderably. This is because exploitation will work reliably enough toautomate an attack with a virtual guarantee of success when it is run. Forthis reason, this is the technique most commonly used in Internet worms thatexploit stack buffer overflow vulnerabilities.

This method also allows shellcode to be placed after the overwritten returnaddress on the Windows platform. Since executables are based at address0x00400000 and x86 is a Little Endian architecture, the last byte of thereturn address must be a null, which terminates the buffer copy and nothingis written beyond that. This limits the size of the shellcode to the size ofthe buffer, which may be overly restrictive. DLLs are located in high memory(above 0x01000000 and so have addresses containing no null bytes, so thismethod can remove null bytes (or other disallowed characters) from theoverwritten return address. Used in this way, the method is often referredto as "DLL Trampolining".

An instruction from ntdll.dll to call theDbgPrint() routine contains the i386 machineopcode for jmp esp.
JumpToEsp
6.3.3   Protective countermeasures

Various techniques have been used to detect or prevent buffer overflows,with various tradeoffs. The most reliable way to avoid or prevent bufferoverflows is to use automatic protection at the language level. This sortof protection, however, cannot be applied to legacy code, and oftentechnical, business, or cultural constraints call for a vulnerable language.The following sections describe the choices and implementations available.

6.3.3.1   Choice of programming language

The choice of programming language can have a profound effect on theoccurrence of buffer overflows. As of 2008, among the most popularlanguages are C and its derivative, C++, with an enormous body of softwarehaving been written in these languages. C and C++ provide no built-inprotection against accessing or overwriting data in any part of memory; morespecifically, they do not check that data written to an array (theimplementation of a buffer) is within the boundaries of that array. However,the standard C++ libraries provide many ways of safely buffering data, andtechnology to avoid buffer overflows also exists for C.

Many other programming languages provide runtime checking and in some caseseven compile-time checking which might send a warning or raise an exceptionwhen C or C++ would overwrite data and continue to execute furtherinstructions until erroneous results are obtained which might or might notcause the program to crash. Examples of such languages include Ada, Lisp,Modula-2, Smalltalk, OCaml and such C-derivatives as Cyclone and D. The Javaand .NET bytecode environments also require bounds checking on all arrays.Nearly every interpreted language will protect against buffer overflows,signalling a well-defined error condition. Often where a language providesenough type information to do bounds checking an option is provided toenable or disable it. Static code analysis can remove many dynamic bound andtype checks, but poor implementations and awkward cases can significantlydecrease performance. Software engineers must carefully consider thetradeoffs of safety versus performance costs when deciding which languageand compiler setting to use.

6.3.3.2   Use of safe libraries

The problem of buffer overflows is common in the C and C++ languages becausethey expose low level representational details of buffers as containers fordata types. Buffer overflows must thus be avoided by maintaining a highdegree of correctness in code which performs buffer management. It has alsolong been recommended to avoid standard library functions which are notbounds checked, such as gets, scanf and strcpy. The Morris worm exploited agets call in fingerd.

Well-written and tested abstract data type libraries which centralize andautomatically perform buffer management, including bounds checking, canreduce the occurrence and impact of buffer overflows. The two mainbuilding-block data types in these languages in which buffer overflowscommonly occur are strings and arrays; thus, libraries preventing bufferoverflows in these data types can provide the vast majority of the necessarycoverage. Still, failure to use these safe libraries correctly can resultin buffer overflows and other vulnerabilities; and naturally, any bug in thelibrary itself is a potential vulnerability. "Safe" library implementationsinclude "The Better String Library", Vstr and Erwin. The OpenBSD operatingsystem's C library provides the strlcpy and strlcat functions, but these aremore limited than full safe library implementations.

In September 2006, Technical Report 24731, prepared by the C standardscommittee, was published; it specifies a set of functions which are based onthe standard C library's string and I/O functions, with additionalbuffer-size parameters. However, the efficacy of these functions for thepurpose of reducing buffer overflows is disputable; it requires programmerintervention on a per function call basis that is equivalent to interventionthat could make the analogous older standard library functions bufferoverflow safe.

6.3.3.3   Stack-smashing protection

Stack-smashing protection is used to detect the most common buffer overflowsby checking that the stack has not been altered when a function returns. Ifit has been altered, the program exits with a segmentation fault. Three suchsystems are Libsafe, and the StackGuard and ProPolice gccpatches.

Microsoft's Data Execution Prevention mode explicitly protects the pointerto the SEH Exception Handler from being overwritten.

Stronger stack protection is possible by splitting the stack in two: one fordata and one for function returns. This split is present in the Forthprogramming language, though it was not a security-based design decision.Regardless, this is not a complete solution to buffer overflows, assensitive data other than the return address may still be overwritten.

6.3.3.4   Pointer protection

Buffer overflows work by manipulating pointers (including stored addresses).PointGuard was proposed as a compiler-extension to prevent attackers frombeing able to reliably manipulate pointers and addresses. The approachworks having the compiler add code to automatically XOR-encode pointersbefore and after they are used. Because the attacker (theoretically) doesnot know what value will be used to encode/decode the pointer, he cannotpredict what it will point to if he overwrites it with a new value.PointGuard was never released, but Microsoft implemented a similar approachbeginning in Windows XP SP2 and Windows Server 2003 SP1. Rather thanimplement pointer protection as an automatic feature, Microsoft added an APIroutine that can be called at the discretion of the programmer. This allowsfor better performance (because it is not used all of the time), but placesthe burden on the programmer to know when it is necessary.

Because XOR is linear, an attacker may be able to manipulate an encodedpointer by overwriting only the lower bytes of an address. This can allow anattack to succeed if the attacker is able to attempt the exploit multipletimes and/or is able to complete an attack by causing a pointer to point toone of several locations (such as any location within a NOP sled).Microsoft added a random rotation to their encoding scheme to address thisweakness to partial overwrites.

6.3.3.5   Executable space protection

Executable space protection is an approach to buffer overflow protectionwhich prevents execution of code on the stack or the heap. An attacker mayuse buffer overflows to insert arbitrary code into the memory of a program,but with executable space protection, any attempt to execute that code willcause an exception.

Some CPUs support a feature called NX ("No eXecute") or XD ("eXecuteDisabled") bit, which in conjunction with software, can be used to markpages of data (such as those containing the stack and the heap) as readableand writeable but not executable.

Some Unix operating systems (e.g. OpenBSD, Mac OS X) ship with executablespace protection (e.g. W^X). Some optional packages include:

  • PaX
  • Exec Shield
  • Openwall

Newer variants of Microsoft Windows also support executable spaceprotection, called Data Execution Prevention. Proprietary add-ons include:

  • BufferShield
  • StackDefender

Executable space protection does not generally protect againstreturn-to-libc attacks, or any other attack which does not rely on theexecution of the attackers code. However, on 64-bit systems using ASLR, asdescribed below, executable space protection makes it far more difficult toexecute such attacks.

6.3.3.6   Address space layout randomization

Address space layout randomization (ASLR) is a computer security featurewhich involves arranging the positions of key data areas, usually includingthe base of the executable and position of libraries, heap, and stack,randomly in a process' address space.

Randomization of the virtual memory addresses at which functions andvariables can be found can make exploitation of a buffer overflow moredifficult, but not impossible. It also forces the attacker to tailor theexploitation attempt to the individual system, which foils the attempts ofinternet worms. A similar but less effective method is to rebase processesand libraries in the virtual address space.

6.3.3.7   Deep packet inspection

The use of deep packet inspection (DPI) can detect, at the networkperimeter, very basic remote attempts to exploit buffer overflows by use ofattack signatures and heuristics. These are able to block packets which havethe signature of a known attack, or if a long series of No-Operationinstructions (known as a nop-sled) is detected, these were once used whenthe location of the exploit's payload is slightly variable.

Packet scanning is not an effective method since it can only prevent knownattacks and there are many ways that a 'nop-sled' can be encoded. Attackershave begun to use alphanumeric, metamorphic, and self-modifying shellcodesto evade detection by heuristic packet scanners and Intrusion detectionsystems.

6.3.3.8   History of exploitation

Buffer overflows were understood as early as 1972, when the ComputerSecurity Technology Planning Study laid out the technique: "The codeperforming this function does not check the source and destination addressesproperly, permitting portions of the monitor to be overlaid by the user.This can be used to inject code into the monitor that will permit the userto seize control of the machine." (Page 61) Today, the monitor would bereferred to as the kernel.

The spread of personal computers in the 1980s increased the number of peoplewho were aware of the technique. On the Commodore PET for instance it was acommon practice to employ a rarely-used second tape buffer to store assemblylanguage routines. Some programmers, to save a few bytes of space on amachine with a maximum of 32K RAM, avoided use of the tedious BASIC "POKE"statement by changing the print buffer start to the tape buffer to print the6502 assembly language code (as strange looking characters) directly to thedesired location. Since the actual print buffer was longer than the tapebuffer, the BASIC string could easily overrun byte 1024 and interfere withthe Microsoft BASIC interpreter on the PET. The bare-bones boot imageloaders of the early personal computers, including the early Mac, Commodore,Atari and all Microsoft operating systems up to Windows 95 and 98, hadinadequate buffer protections and so many programmers became aware of bufferoverflows.

The earliest documented hostile exploitation of a buffer overflow was in1988. It was one of several exploits used by the Morris worm to propagateitself over the Internet. The program exploited was a Unix service calledfinger. Later, in 1995, Thomas Lopatic independently rediscovered the bufferoverflow and published his findings on the Bugtraq security mailing list.A year later, in 1996, Elias Levy (aka Aleph One) published in Phrackmagazine the paper "Smashing the Stack for Fun and Profit", a step-by-stepintroduction to exploiting stack-based buffer overflow vulnerabilities.

Since then, at least two major internet worms have exploited bufferoverflows to compromise a large number of systems. In 2001, the Code Redworm exploited a buffer overflow in Microsoft's Internet InformationServices (IIS) 5.0 and in 2003 the SQL Slammer worm compromised machinesrunning Microsoft SQL Server 2000.

In 2003, buffer overflows present in licensed Xbox games have been exploitedto allow unlicensed software, including homebrew games, to run on theconsole without the need for hardware modifications, known as modchips. ThePS2 Independence Exploit also used a buffer overflow to achieve the same forthe PlayStation 2. The Twilight Hack accomplished the same with the Wii,using a buffer overflow in The Legend of Zelda: Twilight Princess.

In 2009 the PSP 3000 was hacked with "The Sparta!!!" exploit."The Sparta!!!"exploit was allegedly discovered after overwriting the player's name in aGripShift (game) save file with "this is spartaaaaa..." -- with 57 a'stacked onto the end

6.4   Access control

In computer security, access control includes authentication, authorizationand audit. It also includes measures such as physical devices, includingbiometric scans and metal locks, hidden paths, digital signatures,encryption, social barriers, and monitoring by humans and automated systems.

In any access control model, the entities that can perform actions in thesystem are called subjects, and the entities representing resources to whichaccess may need to be controlled are called objects (see also Access ControlMatrix). Subjects and objects should both be considered as softwareentities, rather than as human users: any human user can only have an effecton the system via the software entities that they control. Although somesystems equate subjects with user IDs, so that all processes started by auser by default have the same authority, this level of control is notfine-grained enough to satisfy the Principle of least privilege, andarguably is responsible for the prevalence of malware in such systems.

In some models, for example the object-capability model, any software entitycan potentially act as both a subject and object.

Access control models used by current systems tend to fall into one of twoclasses: those based on capabilities and those based on access controllists (ACLs). In a capability-based model, holding an unforgeable referenceor capability to an object provides access to the object (roughly analogousto how possession of your house key grants you access to your house); accessis conveyed to another party by transmitting such a capability over a securechannel. In an ACL-based model, a subject's access to an object depends onwhether its identity is on a list associated with the object (roughlyanalogous to how a bouncer at a private party would check your ID to see ifyour name is on the guest list); access is conveyed by editing the list.(Different ACL systems have a variety of different conventions regarding whoor what is responsible for editing the list and how it is edited.)

Both capability-based and ACL-based models have mechanisms to allow accessrights to be granted to all members of a group of subjects (often the groupis itself modeled as a subject).

Access control systems provide the essential services of identification andauthentication (I&A), authorization, and accountability where:

  • identification and authentication determine who can log on to a system, and the association of users with the software subjects that they are able to control as a result of logging in;
  • authorization determines what a subject can do;
  • accountability identifies what a subject (or all subjects associated with a user) did.
6.4.1   Identification and authentication (I&A)

Identification and authentication (I&A) is the process of verifying that anidentity is bound to the entity that asserts it. The I&A process assumesthat there was an initial vetting of the identity, during which anauthenticator was established. Subsequently, the entity asserts an identitytogether with an authenticator as a means for validation. The onlyrequirements for the identifier is that it must be unique within itssecurity domain.

Authenticators are commonly based on at least one of these four factors:

  • Something you know, such as a password or a personal identification number (PIN). This assumes that only the owner of the account knows the password or PIN needed to access the account.
  • Something you have, such as a smart card or security token. This assumes that only the owner of the account has the necessary smart card or token needed to unlock the account.
  • Something you are, such as fingerprint, voice, retina, or iris characteristics.
  • Where you are, for example inside or outside a company firewall, or proximity of login location to a personal GPS device.
6.4.2   Authorization

Authorization applies to subjects rather than to users (the associationbetween a user and the subjects initially controlled by that user havingbeen determined by I&A). Authorization determines what a subject can do onthe system.

Most modern operating systems define sets of permissions that are variationsor extensions of three basic types of access:

  • Read (R): The subject can
    • Read file contents
    • List directory contents
  • Write (W): The subject can change the contents of a file or directory with the following tasks:
    • Add
    • Create
    • Delete
    • Rename

  • Execute (X): If the file is a program, the subject can cause the program to be run. (In Unix systems, the 'execute' permission doubles as a 'traverse directory' permission when granted for a directory.)

These rights and permissions are implemented differently in systems based ondiscretionary access control (DAC) and mandatory access control (MAC).

6.4.3   Accountability

Accountability uses such system components as audit trails (records) andlogs to associate a subject with its actions. The information recordedshould be sufficient to map the subject to a controlling user. Audit trailsand logs are important for

  • Detecting security violations
  • Re-creating security incidents

If no one is regularly reviewing your logs and they are not maintained in asecure and consistent manner, they may not be admissible as evidence.

Many systems can generate automated reports based on certain predefinedcriteria or thresholds, known as clipping levels. For example, a clippinglevel may be set to generate a report for the following:

  • More than three failed logon attempts in a given period
  • Any attempt to use a disabled user account

These reports help a system administrator or security administrator to moreeasily identify possible break-in attempts.

6.4.4   Access control techniques

Access control techniques are sometimes categorized as either discretionaryor non-discretionary. The three most widely recognized models areDiscretionary Access Control (DAC), Mandatory Access Control (MAC), and RoleBased Access Control (RBAC). MAC and RBAC are both non-discretionary.

6.4.5   Discretionary access control

Discretionary access control (DAC) is an access policy determined by theowner of an object. The owner decides who is allowed to access the objectand what privileges they have.

Two important concepts in DAC are

  • File and data ownership: Every object in the system has an owner. In most DAC systems, each object's initial owner is the subject that caused it to be created. The access policy for an object is determined by its owner.
  • Access rights and permissions: These are the controls that an owner can assign to other subjects for specific resources.

Access controls may be discretionary in ACL-based or capability-based accesscontrol systems. (In capability-based systems, there is usually no explicitconcept of 'owner', but the creator of an object has a similar degree ofcontrol over its access policy.)

6.4.6   Mandatory access control

Mandatory access control (MAC) is an access policy determined by the system,not the owner. MAC is used in multilevel systems that process highlysensitive data, such as classified government and military information. Amultilevel system is a single computer system that handles multipleclassification levels between subjects and objects.

  • Sensitivity labels: In a MAC-based system, all subjects and objects must have labels assigned to them. A subject's sensitivity label specifies its level of trust. An object's sensitivity label specifies the level of trust required for access. In order to access a given object, the subject must have a sensitivity level equal to or higher than the requested object.
  • Data import and export: Controlling the import of information from other systems and export to other systems (including printers) is a critical function of MAC-based systems, which must ensure that sensitivity labels are properly maintained and implemented so that sensitive information is appropriately protected at all times.

Two methods are commonly used for applying mandatory access control:

  • Rule-based access controls: This type of control further defines specific conditions for access to a requested object. All MAC-based systems implement a simple form of rule-based access control to determine whether access should be granted or denied by matching:
    • An object's sensitivity label
    • A subject's sensitivity label

  • Lattice-based access controls: These can be used for complex access control decisions involving multiple objects and/or subjects. A lattice model is a mathematical structure that defines greatest lower-bound and least upper-bound values for a pair of elements, such as a subject and an object.

Few systems implement MAC. XTS-400 is an example of one that does.

6.4.7   Role-based access control

Role-based access control (RBAC) is an access policy determined by thesystem, not the owner. RBAC is used in commercial applications and also inmilitary systems, where multi-level security requirements may also exist.RBAC differs from DAC in that DAC allows users to control access to theirresources, while in RBAC, access is controlled at the system level, outsideof the user's control. Although RBAC is non-discretionary, it can bedistinguished from MAC primarily in the way permissions are handled. MACcontrols read and write permissions based on a user's clearance level andadditional labels. RBAC controls collections of permissions that may includecomplex operations such as an e-commerce transaction, or may be as simple asread or write. A role in RBAC can be viewed as a set of permissions.

Three primary rules are defined for RBAC:

  1. Role assignment: A subject can execute a transaction only if the subject has selected or been assigned a role.
  2. Role authorization: A subject's active role must be authorized for the subject. With rule 1 above, this rule ensures that users can take on only roles for which they are authorized.
  3. Transaction authorization: A subject can execute a transaction only if the transaction is authorized for the subject's active role. With rules 1 and 2, this rule ensures that users can execute only transactions for which they are authorized.

Additional constraints may be applied as well, and roles can be combined ina hierarchy where higher-level roles subsume permissions owned by sub-roles.

Most IT vendors offer RBAC in one or more products.

6.4.8   Telecommunication

In telecommunication, the term access control is defined in U.S. FederalStandard 1037C with the following meanings:

  1. A service feature or technique used to permit or deny use of the components of a communication system.
  2. A technique used to define or restrict the rights of individuals or application programs to obtain data from, or place data onto, a storage device.
  3. The definition or restriction of the rights of individuals or application programs to obtain data from, or place data into, a storage device.
  4. The process of limiting access to the resources of an AIS to authorized users, programs, processes, or other systems.
  5. That function performed by the resource controller that allocates system resources to satisfy user requests.

Notice that this definition depends on several other technical terms fromFederal Standard 1037C.

6.4.9   Public policy

In public policy, access control to restrict access to systems("authorization") or to track or monitor behavior within systems("accountability") is an implementation feature of using trusted systems forsecurity or social control.

6.5   Security by design

The technologies of computer security are based on logic. There is nouniversal standard notion of what secure behavior is. "Security" is aconcept that is unique to each situation. Security is extraneous to thefunction of a computer application, rather than ancillary to it, thussecurity necessarily imposes restrictions on the application's behavior.

There are several approaches to security in computing, sometimes acombination of approaches is valid:

  1. Trust all the software to abide by a security policy but the software is not trustworthy (this is computer insecurity).
  2. Trust all the software to abide by a security policy and the software is validated as trustworthy (by tedious branch and path analysis for example).
  3. Trust no software but enforce a security policy with mechanisms that are not trustworthy (again this is computer insecurity).
  4. Trust no software but enforce a security policy with trustworthy mechanisms.

Many systems have unintentionally resulted in the first possibility. Sinceapproach two is expensive and non-deterministic, its use is very limited.Approaches one and three lead to failure. Because approach number four isoften based on hardware mechanisms and avoids abstractions and amultiplicity of degrees of freedom, it is more practical. Combinations ofapproaches two and four are often used in a layered architecture with thinlayers of two and thick layers of four.

There are myriad strategies and techniques used to design security systems.There are few, if any, effective strategies to enhance security afterdesign.

One technique enforces the principle of least privilege to great extent,where an entity has only the privileges that are needed for its function.That way even if an attacker gains access to one part of the system,fine-grained security ensures that it is just as difficult for them toaccess the rest

Furthermore, by breaking the system up into smaller components, thecomplexity of individual components is reduced, opening up the possibilityof using techniques such as automated theorem proving to prove thecorrectness of crucial software subsystems. This enables a closed formsolution to security that works well when only a single well-characterizedproperty can be isolated as critical, and that property is also assessableto math. Not surprisingly, it is impractical for generalized correctness,which probably cannot even be defined, much less proven. Where formalcorrectness proofs are not possible, rigorous use of code review and unittesting represent a best-effort approach to make modules secure.

The design should use "defense in depth", where more than one subsystemneeds to be violated to compromise the integrity of the system and theinformation it holds. Defense in depth works when the breaching of onesecurity measure does not provide a platform to facilitate subvertinganother. Also, the cascading principle acknowledges that several low hurdlesdoes not make a high hurdle. So cascading several weak mechanisms does notprovide the safety of a single stronger mechanism.

Subsystems should default to secure settings, and wherever possible shouldbe designed to "fail secure" rather than "fail insecure" (see fail safe forthe equivalent in safety engineering). Ideally, a secure system shouldrequire a deliberate, conscious, knowledgeable and free decision on the partof legitimate authorities in order to make it insecure.

In addition, security should not be an all or nothing issue. The designersand operators of systems should assume that security breaches areinevitable. Full audit trails should be kept of system activity, so thatwhen a security breach occurs, the mechanism and extent of the breach can bedetermined. Storing audit trails remotely, where they can only be appendedto, can keep intruders from covering their tracks. Finally, full disclosurehelps to ensure that when bugs are found the "window of vulnerability" iskept as short as possible.

6.5.1   Server/client architectures

In server/client architectures, the program at the other side may not be anauthorised client and the client's server may not be an authorised server.Even when they are, a man-in-the-middle attack could compromisecommunications.

Often the easiest way to break the security of a client/server system is notto go head on to the security mechanisms but instead to go around them. Aman in the middle attack is a simple example of this, because you can use itto collect details to impersonate a user. Which is why it is important toconsider encryption, hashing, and other security mechanisms in your designto ensure that information collected from a potential attacker wont allowaccess.

Another key feature to client server security in design is not only theobvious. But just general good coding practices. For example following aknown software design structure such as client and broker can help indesigning a well built structure with a solid foundation. Further more thatif the software is modified in the future it is even more important that itfollows a logical foundation of separation between the client and server.This is because if a programmer comes in and can not clearly understand thedynamics of the program they may end up adding or changing something thatcan add a security flaw. Even with the best design this is always apossibility, but the better we standardize the design the least chance wehave of this occurring.

It is also important to ensure that the data passed between the client andserver is necessary. A lazy programmer may use some simple functions from aserver program just so the implementation does not have to be reproduced onthe client. Though there are situations where this may be needed, excessdata flow like this can have many disadvantages. One of which is making yourconnection more obvious, showing more details about the structure ofcommunication, and possibly details that could expose a vulnerability.

7   Usability

Usability is a term used to denote the ease with which people can employ aparticular tool or other human-made object in order to achieve a particulargoal. Usability can also refer to the methods of measuring usability and thestudy of the principles behind an object's perceived efficiency or elegance.

In human-computer interaction and computer science, usability usually refersto the elegance and clarity with which the interaction with a computerprogram or a web site is designed. The term is also used often in thecontext of products like consumer electronics, or in the areas ofcommunication, and knowledge transfer objects (such as a cookbook, adocument or online help). It can also refer to the efficient design ofmechanical objects such as a door handle or a hammer.

The primary notion of usability is that an object designed with the users'psychology and physiology in mind is, for example:

  • More efficient to use—it takes less time to accomplish a particular task
  • Easier to learn—operation can be learned by observing the object
  • More satisfying to use

Complex computer systems are finding their way into everyday life, and atthe same time the market is becoming saturated with competing brands. Thishas led to usability becoming more popular and widely recognized in recentyears as companies see the benefits of researching and developing theirproducts with user-oriented instead of technology-oriented methods. Byunderstanding and researching the interaction between product and user, theusability expert can also provide insight that is unattainable bytraditional company-oriented market research. For example, after observingand interviewing users, the usability expert may identify neededfunctionality or design flaws that were not anticipated. Method called"contextual inquiry" does this in the naturally occurring context of theusers own environment.

In the user-centered design paradigm, the product is designed with itsintended users in mind at all times. In the user-driven or participatorydesign paradigm, some of the users become actual or de facto members of thedesign team.

The term user friendly is often used as a synonym for usable, though it mayalso refer to accessibility.

There is no consensus about the relation of the terms ergonomics (or humanfactors) and usability. Some think of usability as the softwarespecialization of the larger topic of ergonomics. Others view these topicsas tangential, with ergonomics focusing on physiological matters (e.g.,turning a door handle) and usability focusing on psychological matters(e.g., recognizing that a door can be opened by turning its handle).

Usability is also very important in website development. According toJakob Nielsen, "Studies of user behavior on the Web find a low tolerancefor difficult designs or slow sites. People don't want to wait. And theydon't want to learn how to use a home page. There's no such thing as atraining class or a manual for a Web site. People have to be able to graspthe functioning of the site immediately after scanning the home page—for afew seconds at most."

7.1   Definition

Usability is a qualitative attribute that assesses how easy user interfacesare to use. The word "usability" also refers to methods for improvingease-of-use during the design process. Usability consultant Jakob Nielsenand computer science professor Ben Shneiderman have written (separately)about a framework of system acceptability, where usability is a part of"usefulness" and is composed of:

  • Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?
  • Efficiency: Once users have learned the design, how quickly can they perform tasks?
  • Memorability: When users return to the design after a period of not using it, how easily can they re establish proficiency?
  • Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?
  • Satisfaction: How pleasant is it to use the design?

Usability is often associated with the functionalities of the product, inaddition to being solely a characteristic of the user interface (cf.framework of system acceptability, also below, which separates usefulnessinto utility and usability). For example, in the context of mainstreamconsumer products, an automobile lacking a reverse gear could be consideredunusable according to the former view, and lacking in utility according tothe latter view.

When evaluating user interfaces for usability, the definition can be assimple as "the perception of a target user of the effectiveness (fit forpurpose) and efficiency (work or time required to use) of the Interface".Each component may be measured subjectively against criteria e.g. Principlesof User Interface Design, to provide a metric, often expressed as apercentage.

It is important to distinguish between usability testing and usabilityengineering. Usability testing is the measurement of ease of use of aproduct or piece of software. In contrast, usability engineering (UE) is theresearch and design process that ensures a product with good usability.

Usability is an example of a non-functional requirement. As with othernon-functional requirements, usability cannot be directly measured but mustbe quantified by means of indirect measures or attributes such as, forexample, the number of reported problems with ease-of-use of a system.

7.2   Investigation

The key principle for maximizing usability is to employ iterative design,which progressively refines the design through evaluation from the earlystages of design. The evaluation steps enable the designers and developersto incorporate user and client feedback until the system reaches anacceptable level of usability.

The preferred method for ensuring usability is to test actual users on aworking system. Although, there are many methods for studying usability, themost basic and useful is user testing, which has three components:

  • Get some representative users.
  • Ask the users to perform representative tasks with the design.
  • Observe what the users do, where they succeed, and where they have difficulties with the user interface.

It's important to test users individually and let them solve any problems ontheir own. If you help them or direct their attention to any particular partof the screen, you will bias the test. Rather than running a big, expensivestudy, it's better to run many small tests and revise the design betweeneach one so you can fix the usability flaws as you identify them. Iterativedesign is the best way to increase the quality of user experience. The moreversions and interface ideas you test with users, the better.

Usability plays a role in each stage of the design process. The resultingneed for multiple studies is one reason to make individual studies fast andcheap, and to perform usability testing early in the design process. Hereare the main steps:

  • Before starting the new design, test the old design to identify the good parts that you should keep or emphasize, and the bad parts that give users trouble.
  • Test competitors' designs to get data on a range of alternative designs.
  • Conduct a field study to see how users behave in their natural habitat.
  • Make paper prototypes of one or more new design ideas and test them. The less time you invest in these design ideas the better, because you'll need to change them all based on the test results.
  • Refine the design ideas that test best through multiple iterations, gradually moving from low-fidelity prototyping to high-fidelity representations that run on the computer. Test each iteration.
  • Inspect the design relative to established usability guidelines, whether from your own earlier studies or published research.
  • Once you decide on and implement the final design, test it again. Subtle usability problems always creep in during implementation.

Don't defer user testing until you have a fully implemented design. If youdo, it will be impossible to fix the vast majority of the critical usabilityproblems that the test uncovers. Many of these problems are likely to bestructural, and fixing them would require major rearchitecting. The only wayto a high-quality user experience is to start user testing early in thedesign process and to keep testing every step of the way.

7.3   ISO standards

7.3.1   ISO/TR 16982:2002

ISO/TR 16982:2002 "Ergonomics of human-system interaction -- Usabilitymethods supporting human-centered design". This standard providesinformation on human-centred usability methods which can be used for designand evaluation. It details the advantages, disadvantages and other factorsrelevant to using each usability method.

It explains the implications of the stage of the life cycle and theindividual project characteristics for the selection of usability methodsand provides examples of usability methods in context.

The main users of ISO/TR 16982:2002 will be project managers. It thereforeaddresses technical human factors and ergonomics issues only to the extentnecessary to allow managers to understand their relevance and importance inthe design process as a whole.

The guidance in ISO/TR 16982:2002 can be tailored for specific designsituations by using the lists of issues characterizing the context of use ofthe product to be delivered. Selection of appropriate usability methodsshould also take account of the relevant life-cycle process.

ISO/TR 16982:2002 is restricted to methods that are widely used by usabilityspecialists and project managers.

ISO/TR 16982:2002 does not specify the details of how to implement or carryout the usability methods described.

7.3.2   ISO 9241

ISO 9241 is a multi-part standard covering a number of aspects for peopleworking with computers. Although originally titled Ergonomic requirementsfor office work with visual display terminals (VDTs) it is being retitled tothe more generic Ergonomics of Human System Interaction by ISO. As part ofthis change, ISO is renumbering the standard so that it can include manymore topics. The first part to be renumbered was part 10 (now renumbered topart 110).

Part 1 is a general introduction to the rest of the standard. Part 2addresses task design for working with computer systems. Parts 3–9 deal withphysical characteristics of computer equipment. Parts 110 and parts 11–19deal with usability aspects of software, including Part 110 (a general setof usability heuristics for the design of different types of dialogue) andPart 11 (general guidance on the specification and measurement ofusability).

7.4   Usability considerations

Usability includes considerations such as:

  • Who are the users, what do they know, and what can they learn?
  • What do users want or need to do?
  • What is the general background of the users?
  • What is the context in which the user is working?
  • What has to be left to the machine?

Answers to these can be obtained by conducting user and task analysis at thestart of the project.

7.4.1   Other considerations
  • Can users easily accomplish their intended tasks? For example, can users accomplish intended tasks at their intended speed?
  • How much training do users need?
  • What documentation or other supporting materials are available to help the user? Can users find the solutions they seek in these materials?
  • What and how many errors do users make when interacting with the product?
  • Can the user recover from errors? What do users have to do to recover from errors? Does the product help users recover from errors? For example, does software present comprehensible, informative, non-threatening error messages?
  • Are there provisions for meeting the special needs of users with disabilities? (accessibility)

Examples of ways to find answers to these and other questions are:user-focused requirements analysis, building user profiles, and usabilitytesting.

7.5   Evaluation methods

There are a variety of methods currently used to evaluate usability. Certainmethods make use of data gathered from users, while others rely on usabilityexperts. There are usability evaluation methods that apply to all stages ofdesign and development, from product definition to final designmodifications. When choosing a method you must consider the cost, timeconstraints, and appropriateness of the method. For a brief overview ofmethods, see Comparison of usability evaluation methods or continue readingbelow. Usability methods can be further classified into the followingsubcategories:

7.5.1   Cognitive modeling methods

Cognitive modeling involves creating a computational model to estimate howlong it takes people to perform a given task. Models are based onpsychological principles and experimental studies to determine times forcognitive processing and motor movements. Cognitive models can be used toimprove user interfaces or predict problem errors and pitfalls during thedesign process. A few examples of cognitive models include:

7.5.2   Parallel Design

With parallel design, several people create an initial design from the sameset of requirements. Each person works independently, and when finished,shares his/her concepts with the group. The design team considers eachsolution, and each designer uses the best ideas to further improve their ownsolution. This process helps to generate many different, diverse ideas andensures that the best ideas from each design are integrated into the finalconcept. This process can be repeated several times until the team issatisfied with the final concept.

7.5.3   GOMS

GOMS is an acronym that stands for Goals, Operator, Methods, and SelectionRules. It is a family of techniques that analyzes the user complexity ofinteractive systems. Goals are what the user has to accomplish. An operatoris an action performed in service of a goal. A method is a sequence ofoperators that accomplish a goal. Selection rules specify which methodshould be used to satisfy a given goal, based on the context.

7.5.4   Human Processor Model

Sometimes it is useful to break a task down and analyze each individualaspect separately. This allows the tester to locate specific areas forimprovement. To do this, it is necessary to understand how the human brainprocesses information. A model of the human processor is shown below.

HumanProcessorModel

Many studies have been done to estimate the cycle times, decay times, andcapacities of each of these processors. Variables that affect these caninclude subject age, ability, and the surrounding environment. For a youngeradult, reasonable estimates are:

ParameterMeanRange
Eye movement time230 ms70-700 ms
Decay half-life of visual image storage200 ms90-1000 ms
Perceptual processor cycle time100 ms50-200 ms
Cognitive processor cycle time70 ms25-170 ms
Motor processor cycle time70 ms30-100 ms
Effective working memory capacity7 items5-9 items

Long-term memory is believed to have an infinite capacity and decay time.

7.5.5   Keystroke level modeling

Keystroke level modeling is essentially a less comprehensive version of GOMSthat makes simplifying assumptions in order to reduce calculation time andcomplexity. See Keystroke level model for more information.

7.5.6   Inspection methods

These usability evaluation methods involve observation of users by anexperimenter, or the testing and evaluation of a program by an expertreviewer. They provide more quantitative data as tasks can be timed andrecorded.

7.5.6.1   Card Sorting

Card sorting is a way to involve users in grouping information for awebsite's usability review. Participants in a card sorting session areasked to organize the content from a Web site in a way that makes sense tothem. Participants review items from a Web site and then group these itemsinto categories. Card sorting helps to learn how users think about thecontent and how they would organize the information on the Web site. Cardsorting helps to build the structure for a Web site, decide what to put onthe home page, and label the home page categories. It also helps to ensurethat information is organized on the site in a way that is logical to users.

7.5.6.2   Ethnography

Ethnographic analysis is derived from anthropology. Field observations aretaken at a site of a possible user, which track the artifacts of work suchas Post-It notes, items on desktop, shortcuts, and items in trash bins.These observations also gather the sequence of work and interruptions thatdetermine the user’s typical day.

7.5.6.3   Heuristic Evaluation

Heuristic evaluation is a usability engineering method for finding andassesing usability problems in a user interface design as part of aniterative design process. It involves having a small set of evaluatorsexamining the interface and using recognized usability principles (the"heuristics"). It is the most popular of the usability inspection methods,as it is quick, cheap, and easy.

Heuristic evaluation was developed to aid in the design of computeruser-interface design. It relies on expert reviewers to discover usabilityproblems and then categorize and rate them by a set of principles(heuristics.) It is widely used based on its speed and cost-effectiveness.Jakob Nielsen's list of ten heuristics is the most commonly used inindustry. These are ten general principles for user interface design. Theyare called "heuristics" because they are more in the nature of rules ofthumb than specific usability guidelines.

  • Visibility of system status: The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
  • Match between system and the real world: The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.
  • User control and freedom: Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.
  • Consistency and standards: Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
  • Error prevention: Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.
  • Recognition rather than recall: Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
  • Flexibility and efficiency of use: Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
  • Aesthetic and minimalist design: Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
  • Help users recognize, diagnose, and recover from errors: Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
  • Help and documentation: Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.

Thus, by determining which guidelines are violated, the usability of adevice can be determined.

7.5.6.4   Usability Inspection

Usability inspection is a review of a system based on a set of guidelines.The review is conducted by a group of experts who are deeply familiar withthe concepts of usability in design. The experts focus on a list of areas indesign that have been shown to be troublesome for users.

7.5.6.5   Pluralistic Inspection

Pluralistic Inspections are meetings where users, developers, and humanfactors people meet together to discuss and evaluate step by step of a taskscenario. As more people inspect the scenario for problems, the higher theprobability to find problems. In addition, the more interaction in the team,the faster the usability issues are resolved.

7.5.6.6   Consistency Inspection

In consistency inspection, expert designers review products or projects toensure consistency across multiple products to look if it does things in thesame way as their own designs.

7.5.6.7   Activity Analysis

Activity analysis is a usability method used in preliminary stages ofdevelopment to get a sense of situation. It involves an investigatorobserving users as they work in the field. Also referred to as userobservation, it is useful for specifying user requirements and studyingcurrently used tasks and subtasks. The data collected is qualitative anduseful for defining the problem. It should be used when you wish to framewhat is needed, or “What do we want to know?”

7.5.7   Inquiry methods

The following usability evaluation methods involve collecting qualitativedata from users. Although the data collected is subjective, it providesvaluable information on what the user wants.

7.5.7.1   Task Analysis

Task analysis means learning about users' goals and users' ways of working.Task analysis can also mean figuring out what more specific tasks users mustdo to meet those goals and what steps they must take to accomplish thosetasks. Along with user and task analysis, we often do a third analysis:understanding users' environments (physical, social, cultural, andtechnological environments).

7.5.7.2   Focus Groups

A focus group is a focused discussion where a moderator leads a group ofparticipants through a set of questions on a particular topic. Althoughtypically used as a marketing tool, Focus Groups are sometimes used toevaluate usability. Used in the product definition stage, a group of 6 to 10users are gathered to discuss what they desire in a product. An experiencedfocus group facilitator is hired to guide the discussion to areas ofinterest for the developers. Focus groups are typically videotaped to helpget verbatim quotes, and clips are often used to summarize opinions. Thedata gathered not usually quantitative, but can help get an idea of a targetgroup's opinion.

7.5.7.3   Questionaires/Surveys

Surveys have the advantages of being inexpensive, require no testingequipment, and results reflect the users’ opinions. When written carefullyand given to actual users who have experience with the product and knowledgeof design, surveys provide useful feedback on the strong and weak areas ofthe usability of a design. This is a very common method and often does notappear to be a survey, but just a warranty card.

7.5.8   Prototyping methods
7.5.8.1   Rapid Prototyping

Rapid prototyping is a method used in early stages of development tovalidate and refine the usability of a system. It can be used to quickly andcheaply evaluate user-interface designs without the need for an expensiveworking model. This can help remove hesitation to change the design, sinceit is implemented before any real programming begins. One such method ofrapid prototyping is paper prototyping.

7.5.9   Testing methods

These usability evaluation methods involve testing of subjects for the mostquantitative data. Usually recorded on video, they provide task completiontime and allow for observation of attitude.

7.5.9.1   Remote usability testing

Remote usability testing is a technique that exploits users’ environment(e.g. home or office), transforming it into a usability laboratory whereuser observation can be done with screen sharing applications.

7.5.9.2   Thinking Aloud

The Think aloud protocol is a method of gathering data that is used in bothusability and psychology studies. It involves getting a user to verbalizetheir thought processes as they perform a task or set of tasks. Often aninstructor is present to prompt the user into being more vocal as they work.Similar to the Subjects-in-Tandem method, it is useful in pinpointingproblems and is relatively simple to set up. Additionally, it can provideinsight into the user's attitude, which can not usually be discerned from asurvey or questionnaire.

7.5.9.3   Subjects-in-Tandem

Subjects-in-tandem is pairing of subjects in a usability test to gatherimportant information on the ease of use of a product. Subjects tend tothink out loud and through their verbalized thoughts designers learn wherethe problem areas of a design are. Subjects very often provide solutions tothe problem areas to make the product easier to use.

7.5.10   Other methods
7.5.10.1   Cognitive walkthrough

Cognitive walkthrough is a method of evaluating the user interaction of aworking prototype or final product. It is used to evaluate the system’s easeof learning. Cognitive walkthrough is useful to understand the user’sthought processes and decision making when interacting with a system,specially for first-time or infrequent users.

7.5.10.2   Benchmarking

Benchmarking creates standardized test materials for a specific type ofdesign. Four key characteristics are considered when establishing abenchmark: time to do the core task, time to fix errors, time to learnapplications, and the functionality of the system. Once there is abenchmark, other designs can be compared to it to determine the usabilityof the system.

7.5.10.3   Meta-Analysis

Meta-Analysis is a statistical procedure to combine results across studiesto integrate the findings. This phrase was coined in 1976 as a quantitativeliterature review. This type of evaluation is very powerful for determiningthe usability of a device because it combines multiple studies to providevery accurate quantitative support.

7.5.10.4   Persona

Personas are fictitious characters that are created to represent thedifferent user types within a targeted demographic that might use a site orproduct. Alan Cooper introduced the concept of using personas as a part ofinteractive design in 1998 in his book The Inmates Are Running the Asylum,but had used this concept since as early as 1975. Personas are a usabilityevaluation method that can be used at various design stages. The mosttypical time to create personas is at the beginning of designing so thatdesigners have a tangible idea of who the users of their product will be.Personas are the archetypes that represent actual groups of users and theirneeds, which can be a general description of person, context, or usagescenario. This technique turns marketing data on target user population intoa few physical concepts of users to create empathy among the design team.

7.6   Evaluating with tests and metrics

Regardless to how carefully a system is designed, all theories must betested using usability tests. Usability tests involve typical users usingthe system (or product) in a realistic environment. Observation of theuser’s behavior, emotions, and difficulties while performing differenttasks, often identify areas of improvement for the system.

7.7   Prototypes

It is often very difficult for designers to conduct usability tests with theexact system being designed. Cost constraints, size, and design constraintsusually lead the designer to creating a prototype of the system. Instead ofcreating the complete final system, the designer may test different sectionsof the system, thus making several small models of each component of thesystem. The types of usability prototypes may vary from using paper models,index cards, hand drawn models, or storyboards.

Prototypes are able to be modified quickly, often are faster and easier tocreate with less time invested by designers and are more apt to changedesign; although sometimes are not an adequate representation of the wholesystem, are often not durable and testing results may not be parallel tothose of the actual system.

7.7.1   Metrics

While conducting usability tests, designers must use usability metrics toidentify what it is they are going to measure, or the usability metrics.These metrics are often variable, and change in conjunction with the scopeand goals of the project. The number of subjects being tested can alsoaffect usability metrics, as it is often easier to focus on specificdemographics. Qualitative design phases, such as general usability (can thetask be accomplished?), and user satisfaction are also typically done withsmaller groups of subjects. Using inexpensive prototype on small usergroups, provide more detailed information, because of the more interactiveatmosphere, and the designers ability to focus more on the individual user.

As the designs become more complex, the testing must become more formalized.Testing equipment will become more sophisticated and testing metrics becomemore quantitative. With a more refined prototype, designers often testeffectiveness, efficiency, and subjective satisfaction, by asking the userto complete various tasks. These categories are measured by the percent thatcomplete the task, how long it takes to complete the tasks, ratios ofsuccess to failure to complete the task, time spent on errors, the number oferrors, rating scale of satisfactions, number of times user seemsfrustrated, etc. Additional observations of the users give designers insighton navigation difficulties, controls, conceptual models, etc. The ultimategoal of analyzing these metrics is to find/create a prototype design thatusers like and use to successfully perform given tasks.

After conducting usability tests, it is important for a designer to recordwhat was observed, in addition to why such behavior occurred and modify themodel according to the results. Often it is quite difficult to distinguishthe source of the design errors, and what the user did wrong. However,effective usability tests will not generate a solution to the problems, butprovide modified design guidelines for continued testing.

7.8   Benefits of usability

The key benefits of usability are:

  • Higher revenues through increased sales
  • Increased user efficiency
  • Reduced development costs
  • Reduced support costs
7.8.1   Corporate integration

An increase in usability generally positively affects several facets of acompany’s output quality. In particular, the benefits fall into severalcommon areas:

  • Increased productivity
  • Decreased training and support costs
  • Increased sales and revenues
  • Reduced development time and costs
  • Reduced maintenance costs
  • Increased customer satisfaction

Increased usability in the workplace fosters several responses fromemployees. Along with any positive feedback, “workers who enjoy their workdo it better, stay longer in the face of temptation, and contribute ideasand enthusiasm to the evolution of enhanced productivity." In order tocreate standards, companies often implement experimental design techniquesthat create baseline levels. Areas of concern in an office environmentinclude (though are not necessarily limited to):

  • Working Posture
  • Design of Workstation Furniture
  • Screen Displays
  • Input Devices
  • Organizational Issues
  • Office Environment
  • Software Interface

By working to improve said factors, corporations can achieve their goals ofincreased output at lower costs, while potentially creating optimal levelsof customer satisfaction. There are numerous reasons why each of thesefactors correlates to overall improvement. For example, making a piece ofsoftware’s user interface easier to understand would reduce the need forextensive training. The improved interface would also tend to lower the timeneeded to perform necessary tasks, and so would both raise the productivitylevels for employees and reduce development time (and thus costs). It isimportant to note that each of the aforementioned factors are not mutuallyexclusive, rather should be understood to work in conjunction to form theoverall workplace environment.

8   Accessibility

Accessibility is a general term used to describe the degree to which aproduct (e.g., device, service, environment) is accessible by as many peopleas possible. Accessibility can be viewed as the "ability to access" thefunctionality, and possible benefit, of some system or entity. Accessibilityis often used to focus on people with disabilities and their right of accessto entities, often through use of assistive technology. Several definitionsof accessibility refer directly to access-based individual rights laws andregulations. Products or services designed to meet these regulations areoften termed Easy Access or Accessible.

Accessibility is not to be confused with usability which is used to describethe extent to which a product (e.g., device, service, environment) can beused by specified users to achieve specified goals with effectiveness,efficiency and satisfaction in a specified context of use.

Accessibility is strongly related to universal design when the approachinvolves "direct access." This is about making things accessible to allpeople (whether they have a disability or not). However, products marketedas having benefited from a Universal Design process are often actually thesame devices customized specifically for use by people with disabilities. Analternative is to provide "indirect access" by having the entity support theuse of a person's assistive technology to achieve access (e.g., screenreader).

8.1   Specific aims

8.1.1   Cognitive impairments and illiteracy

Caused by:

  • Developmental disabilities
    • Mental retardation
    • Cerebral palsy
    • Autism spectrum disorder
    • Various genetic and chromosomal disorders such as Down's syndrome and Fragile X syndrome
    • Fetal alcohol syndrome
  • Learning disabilities
    • Dyslexia
    • Dyscalculia
  • Cognitive Disabilities
    • Memory
    • Attention
    • Developmental Maturity
    • Problem solving
    • Logic skills

The biggest challenge in computer accessibility is to make resourcesaccessible to people with cognitive disabilities - particularly those withpoor communication skills - and those without reading skills.

Their further development relies on public domain icons being available.Many people with a learning disability learn and rely on proprietary symbols.They thus become tied to particular products. The copyright owners aregenerally unwilling to release them on the web.

Other examples include Web accessibility a set of guidelines and twoaccessible web portals designed for people developing reading skills arepeepo.com - try typing a letter with your keyboard for more - andpeepo.co.uk with enhanced graphics, unique style controls and improvedinteractivity (requires an SVG supported browser).

8.1.2   Visual impairment

Caused by:

  • Blindness
  • Low vision
  • Poor eyesight
  • Color blindness

Another significant challenge in computer accessibility is to make softwareusable by people with visual impairment, since computer interfaces oftensolicit input visually and provide visual feedback in response. Forindividuals with mild to medium vision impairment, it is helpful to uselarge fonts, high DPI displays, high-contrast themes and icons supplementedwith auditory feedback and screen magnifying software.

In the case of severe vision impairment such as blindness, screen readersoftware that provides feedback via text to speech or a refreshable brailledisplay is a necessary accommodation for interaction with a computer.

About 8% of people, mostly males, suffer from some form of color-blindness.In a well-designed user interface, color should not be the only way ofdistinguishing between different pieces of information. However, the onlycolor combinations that matter are those that people with a deficiencymight confuse, which generally means red and green and blue and green.

8.1.3   Motor and dexterity impairments

Caused by:

  • Difficulty/inability to use hands
  • Tremors
  • Muscle Slowness
  • Loss of fine muscle control
  • Parkinson's Disease
  • Muscular Dystrophy
  • Cerebral palsy
  • Stroke

Some people may not be able to use a conventional input device, such as themouse or the keyboard. Therefore it is important for software functions tobe accessible using both devices; ideally, software uses a generic input APIthat permits the use even of highly specialized devices unheard of at thetime of software development. Keyboard shortcuts and mouse gestures are waysto achieve this. More specialized solutions like on-screen softwarekeyboards and alternate input devices like switches, joysticks andtrackballs are also available. Speech recognition technology is also acompelling and suitable alternative to conventional keyboard and mouse inputas it simply requires a commonly available audio headset.

The astrophysicist Stephen Hawking is a famous example of a person sufferingfrom motor disability. He uses a switch, combined with special software,that allows him to control his wheelchair-mounted computer using hisremaining small movement ability. This performs as a normal computer,allowing him to research and produce his written work, and as anAugmentative and Alternative Communication and environmental control device.

8.1.4   Hearing impairment

Caused by:

  • Deafness
  • Hearing Impairments

While sound user interfaces have a secondary role in common desktopcomputing, usually limited to system sounds as feedback, software producerstake into account people who can't hear, either for personal disability,noisy environments, silence requirements or lack of sound hardware. Suchsystem sounds like beeps can be substituted or supplemented with visualnotifications and captioned text (akin to closed captions).

8.1.5   Photoepileptic seizures

Caused by:

  • Visual strobe
  • Flashing effects

Even though photoepileptic seizures are not caused (directly) by mostmodern computer monitors, the content displayed by monitors can causephotoepileptic seizures. Most users with photoepileptic seizures arespecifically sensitive to range of 16hz to 25hz. As long as the monitor'srefresh rate is not set to greater than 70hz and the monitor is notdefective, users should be without problems.

Rapidly changing images or highly regular patterns such as flashing contentor irregular fonts can trigger seizures in people with photosensitiveepilepsy. Avoid causing the user's screen to flicker with a frequencygreater than 2 Hz and less than 55 Hz.

8.2   Computer Software Restrictive Elements

These restrictive elements make utilization of a computer difficult foraccessibility users:

8.2.1   Collapsing Menus

Software that incorporates Collapsing Menus has a facility that makesfundamental menu options not appear when the package menus are activated.

Instead a downward facing arrow appears at the bottom of the menu.

In order to access the missing menu options, it is necessary to activate thedownward facing arrow. The menu will now change to include the missingoptions, and it is necessary to navigate the menu again to obtain themissing option. The problem reoccurs each time the menu is activated.

Problems

Problems with software incorporating this feature are:

The system requires more work for people who have difficulty in using themouse or keyboard.

The downward facing arrow is difficult to activate with the mouse or thetouchscreen.

The problem is recursive.

8.2.2   Right Click

A mouse with two buttons is required. Depression of the second mouse buttonprovides an alternative function to a normal mouse click.

Problems

Problems with software incorporating this feature are:

The interface is not compatible with touchscreen based systems becausetouchscreen devices generally do not generally provide alternative fingerrecognition or rightclick facilities.

Inclusion of a second mouse button causes confusion for left andright-handed users on shared computers.

Users unable to use a mouse cannot access the rightclick functionality.

8.2.3   Double Click

Double Click is a facility that requires depression of the mouse button inrapid succession to obtain an alternative function to a normal mouse click.

Problems

Problems with software incorporating this feature are:

The double click is difficult to achieve for some mouse and touchscreenusers.

8.2.4   Middle Click

Middle Click is a facility that requires a three button mouse or emulationthat requires pressing both mouse buttons simultaneously.

Problems

Problems with software incorporating this feature are:

The interface is not compatible with touchscreen based systems becausetouchscreens do not generally provide alternative finger recognition ormiddleclick facilities.

Users unable to use a mouse cannot access the middleclick functionality.

8.2.5   Drag

A drag event can be achieved by:

  • Moving the mouse or pointing device with a button pressed
  • A finger or pointer being moved whilst in continuous contact with a touchscreen or tablet device.
  • A joystick being moved whilst the button is continually depressed
  • The aiming device being moved whilst the trigger is continually depressed
Problems
One of the problems with software incorporating this feature is that someusers find interfaces requiring drag movements are difficult to use,particularly for accessibility users, or users with limited dexterity.
8.2.6   Neighbouring Widgets

Neighbouring Widgets is system design flaw built into some graphical userinterfaces caused by placement of widgets in close proximity to each other.

Problems
This causes a problem for users that lack precision targeting skills withthe mouse pointing device, or users that produce mouse jolt when theyattempt to click a mouse button.
8.2.7   No Keyboard Equivalents

This is a system design flaw, whereby some functionality provided by mousenavigation is not replicated by a keyboard equivalent.

Problems

The problem with software containing this flaw is that users with limiteddexterity will not be able to make use of the required functionality.

Examples of this include the lack of a facilities in Microsoft Word toSwitch Between Headers and Footers, or to resize boxes and tables using onlythe keyboard.

8.2.8   No Joystick Equivalents

Some windowing systems do not allow the joystick to be used to control thepointer or cursor. This flaw makes the system difficult to operate by userswith limited dexterity, because they cannot utilize the joystick for basicsystem operation.

8.2.9   Focus stealing

Focus stealing is a facility built into some graphical user interfaces andwindowing management systems that allows an application that is not in focusto suddenly gain focus and steal user input intended for the previouslyfocused application.

9   User Interface (UI)

The user interface (also known as Human Computer Interface or Man-Machine Interface (MMI)) is the aggregate of means by which people—the users—interact with the system—a particular machine, device, computer program or other complex tool. The user interface provides means of:

  • Input, allowing the users to manipulate a system
  • Output, allowing the system to indicate the effects of the users' manipulation.

To work with a system, users have to be able to control the system and assess the state of the system. For example, when driving an automobile, the driver uses the steering wheel to control the direction of the vehicle, and the accelerator pedal, brake pedal and gearstick to control the speed of the vehicle. The driver perceives the position of the vehicle by looking through the windscreen and exact speed of the vehicle by reading the speedometer. The user interface of the automobile is on the whole composed of the instruments the driver can use to accomplish the tasks of driving and maintaining the automobile.

The system may expose several user interfaces to serve different kinds of users. For example, a computerized library database might provide two user interfaces, one for library patrons (limited set of functions, optimized for ease of use) and the other for library personnel (wide set of functions, optimized for efficiency).

In some circumstance computers might observe the user, and react according to their actions without specific commands. A means of tracking parts of the body is required, and sensors noting the position of the head, direction of gaze and so on have been used experimentally. This is particularly relevant to immersive interfaces.

9.1   Usability

The design of a user interface affects the amount of effort the user must expend to provide input for the system and to interpret the output of the system, and how much effort it takes to learn how to do this. Usability is the degree to which the design of a particular user interface takes into account the human psychology and physiology of the users, and makes the process of using the system effective, efficient and satisfying.

Usability is mainly a characteristic of the user interface, but is also associated with the functionalities of the product and the process to design it. It describes how well a product can be used for its intended purpose by its target users with efficiency, effectiveness, and satisfaction, also taking into account the requirements from its context of use.

9.2   User interfaces in computing

In computer science and human-computer interaction, the user interface (of a computer program) refers to the graphical, textual and auditory information the program presents to the user, and the control sequences (such as keystrokes with the computer keyboard, movements of the computer mouse, and selections with the touchscreen) the user employs to control the program.

9.2.1   Types

Currently (as of 2009) the following types of user interface are the most common:

  • Graphical user interfaces (GUI) accept input via devices such as computer keyboard and mouse and provide articulated graphical output on the computer monitor. There are at least two different principles widely used in GUI design: Object-oriented user interfaces (OOUIs) and application oriented interfaces[verification needed].
  • Web-based user interfaces or web user interfaces (WUI) accept input and provide output by generating web pages which are transmitted via the Internet and viewed by the user using a web browser program. Newer implementations utilize Java, AJAX, Adobe Flex, Microsoft .NET, or similar technologies to provide realtime control in a separate program, eliminating the need to refresh a traditional HTML based web browser. Administrative web interfaces for web-servers, servers and networked computers are called Control panel.

User interfaces that are common in various fields outside desktop computing:

  • Command line interfaces, where the user provides the input by typing a command string with the computer keyboard and the system provides output by printing text on the computer monitor. Used for system administration tasks etc.
  • Tactile interfaces supplement or replace other forms of output with haptic feedback methods. Used in computerized simulators etc.
  • Touch interfaces are graphical user interfaces using a touchscreen display as a combined input and output device. Used in many types of point of sale, industrial processes and machines, self-service machines etc.

Other types of user interfaces:

  • Attentive user interfaces manage the user attention deciding when to interrupt the user, the kind of warnings, and the level of detail of the messages presented to the user.
  • Batch interfaces are non-interactive user interfaces, where the user specifies all the details of the batch job in advance to batch processing, and receives the output when all the processing is done. The computer does not prompt for further input after the processing has started.
  • Conversational Interface Agents attempt to personify the computer interface in the form of an animated person, robot, or other character (such as Microsoft's Clippy the paperclip), and present interactions in a conversational form.
  • Crossing-based interfaces are graphical user interfaces in which the primary task consists in crossing boundaries instead of pointing.
  • Gesture interfaces are graphical user interfaces which accept input in a form of hand gestures, or mouse gestures sketched with a computer mouse or a stylus.
  • Intelligent user interfaces are human-machine interfaces that aim to improve the efficiency, effectiveness, and naturalness of human-machine interaction by representing, reasoning, and acting on models of the user, domain, task, discourse, and media (e.g., graphics, natural language, gesture).
  • Multi-screen interfaces, employ multiple displays to provide a more flexible interaction. This is often employed in computer game interaction in both the commercial arcades and more recently the handheld markets.
  • Noncommand user interfaces, which observe the user to infer his / her needs and intentions, without requiring that he / she formulate explicit commands.
  • Object-oriented User Interface (OOUI)
  • Reflexive user interfaces where the users control and redefine the entire system via the user interface alone, for instance to change its command verbs. Typically this is only possible with very rich graphic user interfaces.
  • Tangible user interfaces, which place a greater emphasis on touch and physical environment or its element.
  • Text user interfaces are user interfaces which output text, but accept other form of input in addition to or in place of typed command strings.
  • Voice user interfaces, which accept input and provide output by generating voice prompts which are transmitted via a telephone network and heard by the user using a telephone. The user input is made by pressing telephone keys.
  • Natural-Language interfaces - Used for search engines and on webpages. User types in a question and waits for a response.
  • Zero-Input interfaces get inputs from a set of sensors instead of querying the user with input dialogs.
  • Zooming user interfaces are graphical user interfaces in which information objects are represented at different levels of scale and detail, and where the user can change the scale of the viewed area in order to show more detail.

9.3   Modalities and modes

A modality is a path of communication employed by the user interface to carry input and output. Examples of modalities:

  • Input — computer keyboard allows the user to enter typed text, digitizing tablet allows the user to create free-form drawing
  • Output — computer monitor allows the system to display text and graphics (vision modality), loudspeaker allows the system to produce sound (auditory modality)

The user interface may employ several redundant input modalities and output modalities, allowing the user to choose which ones to use for interaction.

A mode is a distinct method of operation within a computer program, in which the same input can produce different perceived results depending of the state of the computer program. Heavy use of modes often reduces the usability of a user interface, as the user must expend effort to remember current mode states, and switch between mode states as necessary.

11   Networking

A computer network is a group of interconnected computers. Networks may be classified according to a wide variety of characteristics. This article provides a general overview of some types and categories and also presents the basic components of a network.

A network is a collection of computers connected to each other. The network allows computers to communicate with each other and share resources and information. The Advance Research Projects Agency (ARPA) designed "Advanced Research Projects Agency Network" (ARPANET) for the United States Department of Defense. It was the first computer network in the world in late 1960's and early 1970's

11.1   Network classification

The following list presents categories used for classifying networks.

11.1.1   Scale

Based on their scale, networks can be classified as Local Area Network (LAN), Wide Area Network (WAN), Metropolitan Area Network (MAN), Personal Area Network (PAN), Virtual Private Network (VPN), Campus Area Network (CAN), Storage Area Network (SAN), etc.

11.1.2   Connection method

Computer networks can also be classified according to the hardware and software technology that is used to interconnect the individual devices in the network, such as Optical fiber, Ethernet, Wireless LAN, HomePNA, or Power line communication.

Ethernet uses physical wiring to connect devices. Frequently deployed devices include hubs, switches, bridges and/or routers.

Wireless LAN technology is designed to connect devices without wiring. These devices use radio waves or infrared signals as a transmission medium.

Functional relationship (network architectures)

Computer networks may be classified according to the functional relationships which exist among the elements of the network, e.g., Active Networking, Client-server and Peer-to-peer (workgroup) architecture.

11.2   Network topology

Computer networks may be classified according to the network topology upon which the network is based, such as bus network, star network, ring network, mesh network, star-bus network, tree or hierarchical topology network. Network topology signifies the way in which devices in the network see their logical relations to one another. The use of the term "logical" here is significant. That is, network topology is independent of the "physical" layout of the network. Even if networked computers are physically placed in a linear arrangement, if they are connected via a hub, the network has a Star topology, rather than a bus topology. In this regard the visual and operational characteristics of a network are distinct; the logical network topology is not necessarily the same as the physical layout. Networks may be classified based on the method of data used to convey the data, these include digital and analog networks.

11.3   Types of networks

Below is a list of the most common types of computer networks in order of scale.

11.3.1   Personal area network

A personal area network (PAN) is a computer network used for communication among computer devices close to one person. Some examples of devices that are used in a PAN are printers, fax machines, telephones, PDAs and scanners. The reach of a PAN is typically about 20-30 feet (approximately 6-9 meters), but this is expected to increase with technology improvements.

11.3.2   Local area network

A local area network (LAN) is a computer network covering a small physical area, like a home, office, or small group of buildings, such as a school, or an airport. Current LANs are most likely to be based on Ethernet technology. For example, a library may have a wired or wireless LAN for users to interconnect local devices (e.g., printers and servers) and to connect to the internet. On a wired LAN, PCs in the library are typically connected by category 5 (Cat5) cable, running the IEEE 802.3 protocol through a system of interconnected devices and eventually connect to the Internet. The cables to the servers are typically on Cat 5e enhanced cable, which will support IEEE 802.3 at 1 Gbit/s. A wireless LAN may exist using a different IEEE protocol, 802.11b, 802.11g or possibly 802.11n. The staff computers (bright green in the figure) can get to the color printer, checkout records, and the academic network and the Internet. All user computers can get to the Internet and the card catalog. Each workgroup can get to its local printer. Note that the printers are not accessible from outside their workgroup.

All interconnected devices must understand the network layer (layer 3), because they are handling multiple subnets (the different colors). Those inside the library, which have only 10/100 Mbit/s Ethernet connections to the user device and a Gigabit Ethernet connection to the central router, could be called "layer 3 switches" because they only have Ethernet interfaces and must understand IP. It would be more correct to call them access routers, where the router at the top is a distribution router that connects to the Internet and academic networks' customer access routers.

The defining characteristics of LANs, in contrast to WANs (wide area networks), include their higher data transfer rates, smaller geographic range, and lack of a need for leased telecommunication lines. Current Ethernet or other IEEE 802.3 LAN technologies operate at speeds up to 10 Gbit/s. This is the data transfer rate. IEEE has projects investigating the standardization of 100 Gbit/s, and possibly 40 Gbit/s.

Typical library network,in a branching tree topologyand controlled access to resources
NETWORK_Library_LAN
11.3.3   Campus area network

A campus area network (CAN) is a computer network made up of an interconnection of local area networks (LANs) within a limited geographical area. It can be considered one form of a metropolitan area network, specific to an academic setting.

In the case of a university campus-based campus area network, the network is likely to link a variety of campus buildings including; academic departments, the university library and student residence halls. A campus area network is larger than a local area network but smaller than a wide area network (WAN) (in some cases).

The main aim of a campus area network is to facilitate students accessing internet and university resources. This is a network that connects two or more LANs but that is limited to a specific and contiguous geographical area such as a college campus, industrial complex, office building, or a military base. A CAN may be considered a type of MAN (metropolitan area network), but is generally limited to a smaller area than a typical MAN. This term is most often used to discuss the implementation of networks for a contiguous area. This should not be confused with a Controller Area Network. A LAN connects network devices over a relatively short distance. A networked office building, school, or home usually contains a single LAN, though sometimes one building will contain a few small LANs (perhaps one per room), and occasionally a LAN will span a group of nearby buildings. In TCP/IP networking, a LAN is often but not always implemented as a single IP subnet.

11.3.4   Metropolitan area network

A metropolitan area network (MAN) is a network that connects two or more local area networks or campus area networks together but does not extend beyond the boundaries of the immediate town/city. Routers, switches and hubs are connected to create a metropolitan area network.

11.3.5   Wide area network

A wide area network (WAN) is a computer network that covers a broad area (i.e., any network whose communications links cross metropolitan, regional, or national boundaries [1]). Less formally, a WAN is a network that uses routers and public communications links [1]. Contrast with personal area networks (PANs), local area networks (LANs), campus area networks (CANs), or metropolitan area networks (MANs) are usually limited to a room, building, campus or specific metropolitan area (e.g., a city) respectively. The largest and most well-known example of a WAN is the Internet. A WAN is a data communications network that covers a relatively broad geographic area (i.e. one city to another and one country to another country) and that often uses transmission facilities provided by common carriers, such as telephone companies. WAN technologies generally function at the lower three layers of the OSI reference model: the physical layer, the data link layer, and the network layer.

11.3.6   Global area network

A global area networks (GAN) specification is in development by several groups, and there is no common definition. In general, however, a GAN is a model for supporting mobile communications across an arbitrary number of wireless LANs, satellite coverage areas, etc. The key challenge in mobile communications is "handing off" the user communications from one local coverage area to the next. In IEEE Project 802, this involves a succession of terrestrial WIRELESS local area networks (WLAN).[1]

11.3.7   Virtual private network

A virtual private network (VPN) is a computer network in which some of the links between nodes are carried by open connections or virtual circuits in some larger network (e.g., the Internet) instead of by physical wires. The link-layer protocols of the virtual network are said to be tunneled through the larger network when this is the case. One common application is secure communications through the public Internet, but a VPN need not have explicit security features, such as authentication or content encryption. VPNs, for example, can be used to separate the traffic of different user communities over an underlying network with strong security features.

A VPN may have best-effort performance, or may have a defined service level agreement (SLA) between the VPN customer and the VPN service provider. Generally, a VPN has a topology more complex than point-to-point.

A VPN allows computer users to appear to be editing from an IP address location other than the one which connects the actual computer to the Internet.

11.3.8   Internetwork

A Internetworking involves connecting two or more distinct computer networks or network segments via a common routing technology. The result is called an internetwork (often shortened to internet). Two or more networks or network segments connected using devices that operate at layer 3 (the 'network' layer) of the OSI Basic Reference Model, such as a router. Any interconnection among or between public, private, commercial, industrial, or governmental networks may also be defined as an internetwork.

In modern practice, the interconnected networks use the Internet Protocol. There are at least three variants of internetwork, depending on who administers and who participates in them:

  • Intranet
  • Extranet
  • Internet

Intranets and extranets may or may not have connections to the Internet. If connected to the Internet, the intranet or extranet is normally protected from being accessed from the Internet without proper authorization. The Internet is not considered to be a part of the intranet or extranet, although it may serve as a portal for access to portions of an extranet.

11.3.9   Intranet

An intranet is a set of networks, using the Internet Protocol and IP-based tools such as web browsers and file transfer applications, that is under the control of a single administrative entity. That administrative entity closes the intranet to all but specific, authorized users. Most commonly, an intranet is the internal network of an organization. A large intranet will typically have at least one web server to provide users with organizational information.

11.3.10   Extranet

An extranet is a network or internetwork that is limited in scope to a single organization or entity but which also has limited connections to the networks of one or more other usually, but not necessarily, trusted organizations or entities (e.g. a company's customers may be given access to some part of its intranet creating in this way an extranet, while at the same time the customers may not be considered 'trusted' from a security standpoint). Technically, an extranet may also be categorized as a CAN, MAN, WAN, or other type of network, although, by definition, an extranet cannot consist of a single LAN; it must have at least one connection with an external network.

11.3.11   Internet

The Internet is a specific internetwork. It consists of a worldwide interconnection of governmental, academic, public, and private networks based upon the networking technologies of the Internet Protocol Suite. It is the successor of the Advanced Research Projects Agency Network (ARPANET) developed by DARPA of the U.S. Department of Defense. The Internet is also the communications backbone underlying the World Wide Web (WWW). The 'Internet' is most commonly spelled with a capital 'I' as a proper noun, for historical reasons and to distinguish it from other generic internetworks.

Participants in the Internet use a diverse array of methods of several hundred documented, and often standardized, protocols compatible with the Internet Protocol Suite and an addressing system (IP Addresses) administered by the Internet Assigned Numbers Authority and address registries. Service providers and large enterprises exchange information about the reachability of their address spaces through the Border Gateway Protocol (BGP), forming a redundant worldwide mesh of transmission paths.

11.4   Basic hardware components

All networks are made up of basic hardware building blocks to interconnect network nodes, such as Network Interface Cards (NICs), Bridges, Hubs, Switches, and Routers. In addition, some method of connecting these building blocks is required, usually in the form of galvanic cable (most commonly Category 5 cable). Less common are microwave links (as in IEEE 802.12) or optical cable ("optical fiber"). An ethernet card may also be required.

11.4.1   Network interface cards

A network card, network adapter or NIC (network interface card) is a piece of computer hardware designed to allow computers to communicate over a computer network. It provides physical access to a networking medium and often provides a low-level addressing system through the use of MAC addresses. It allows users to connect to each other either by using cables or wirelessly.The NIC provides the transfer of data in megabytes.

11.4.2   Repeaters

A repeater is an electronic device that receives a signal and retransmits it at a higher power level, or to the other side of an obstruction, so that the signal can cover longer distances without degradation. In most twisted pair ethernet configurations, repeaters are required for cable runs longer than 100 meters away from the computer.

11.4.3   Hubs

A hub contains multiple ports. When a packet arrives at one port, it is copied to all the ports of the hub for transmission. When the packets are copied, the destination address in the frame does not change to a broadcast address. It does this in a rudimentary way: It simply copies the data to all of the Nodes connected to the hub.[2]

11.4.4   Bridges

A network bridge connects multiple network segments at the data link layer (layer 2) of the OSI model. Bridges do not promiscuously copy traffic to all ports, as hubs do, but learn which MAC addresses are reachable through specific ports. Once the bridge associates a port and an address, it will send traffic for that address only to that port. Bridges do send broadcasts to all ports except the one on which the broadcast was received.

Bridges learn the association of ports and addresses by examining the source address of frames that it sees on various ports. Once a frame arrives through a port, its source address is stored and the bridge assumes that MAC address is associated with that port. The first time that a previously unknown destination address is seen, the bridge will forward the frame to all ports other than the one on which the frame arrived.

Bridges come in three basic types:

  1. Local bridges: Directly connect local area networks (LANs)
  2. Remote bridges: Can be used to create a wide area network (WAN) link between LANs. Remote bridges, where the connecting link is slower than the end networks, largely have been replaced by routers.
  3. Wireless bridges: Can be used to join LANs or connect remote stations to LANs.
11.4.5   Switches

A switch is a device that forwards and filters OSI layer 2 datagrams (chunk of data communication) between ports (connected cables) based on the MAC addresses in the packets.[3] This is distinct from a hub in that it only forwards the datagrams to the ports involved in the communications rather than all ports connected. Strictly speaking, a switch is not capable of routing traffic based on IP address (layer 3) which is necessary for communicating between network segments or within a large or complex LAN. Some switches are capable of routing based on IP addresses but are still called switches as a marketing term. A switch normally has numerous ports, with the intention being that most or all of the network is connected directly to the switch, or another switch that is in turn connected to a switch.[4]

Switch is a marketing term that encompasses routers and bridges, as well as devices that may distribute traffic on load or by application content (e.g., a Web URL identifier). Switches may operate at one or more OSI model layers, including physical, data link, network, or transport (i.e., end-to-end). A device that operates simultaneously at more than one of these layers is called a multilayer switch.

Overemphasizing the ill-defined term "switch" often leads to confusion when first trying to understand networking. Many experienced network designers and operators recommend starting with the logic of devices dealing with only one protocol level, not all of which are covered by OSI. Multilayer device selection is an advanced topic that may lead to selecting particular implementations, but multilayer switching is simply not a real-world design concept.

11.4.6   Routers

Routers are networking devices that forward data packets between networks using headers and forwarding tables to determine the best path to forward the packets. Routers work at the network layer of the TCP/IP model or layer 3 of the OSI model. Routers also provide interconnectivity between like and unlike media (RFC 1812). This is accomplished by examining the Header of a data packet, and making a decision on the next hop to which it should be sent (RFC 1812) They use preconfigured static routes, status of their hardware interfaces, and routing protocols to select the best route between any two subnets. A router is connected to at least two networks, commonly two LANs or WANs or a LAN and its ISP's network. Some DSL and cable modems, for home (and even office) use, have been integrated with routers to allow multiple home/office computers to access the Internet through the same connection. Many of these new devices also consist of wireless access points (waps) or wireless routers to allow for IEEE 802.11g/b/n wireless enabled devices to connect to the network without the need for cabled connections.

14   Debugging

Debugging is a methodical process of finding and reducing the number of bugs, or defects, in a computer program or a piece of electronic hardware thus making it behave as expected. This can be done by fixing or bypassing found errors. Debugging tends to be harder when various subsystems are tightly coupled, as changes in one may cause bugs to emerge in another.

There is some controversy over the origin of the term "debugging."

The terms "bug" and "debugging" are both popularly attributed to Admiral Grace Hopper in the 1940s[1]. While she was working on a Mark II Computer at Harvard University, her associates discovered a moth stuck in a relay and thereby impeding operation, whereupon she remarked that they were "debugging" the system. However the term "bug" in the meaning of technical error dates back at least to 1878 and Thomas Edison (see the Software bug article for a full discussion), and "debugging" seems to have been used as a term in aeronautics before entering the world of computers. Indeed, in an interview Grace Hopper remarked that she was not coining the term. The moth fit the already existing terminology, so she saved it.

14.1   Tools

Debugging is, in general, a lengthy and tiresome task. The debugging skill of the programmer is probably the biggest factor in the ability to debug a problem, but the difficulty of software debugging varies greatly with the programming language used and the available tools, such as debuggers. Debuggers are software tools which enable the programmer to monitor the execution of a program, stop it, re-start it, set breakpoints, change values in memory and even, in some cases, go back in time. The term debugger can also refer to the person who is doing the debugging.

Generally, high-level programming languages, such as Java, make debugging easier, because they have features such as exception handling that make real sources of erratic behaviour easier to spot. In lower-level programming languages such as C or assembly, bugs may cause silent problems such as memory corruption, and it is often difficult to see where the initial problem happened. In those cases, memory debugger tools may be needed.

In certain situations, general purpose software tools that are language specific in nature can be very useful. These take the form of static code analysis tools. These tools look for a very specific set of known problems, some common and some rare, within the source code. All such issues detected by these tools would rarely be picked up by a compiler or interpreter, thus they are not syntax checkers, but more semantic checkers. Some tools claim to be able to detect 300+ unique problems. Both commercial and free tools exist in various languages. These tools can be extremely useful when checking very large source trees, where it is impractical to do code walkthroughs. A typical example of a problem detected would be a variable dereference that occurs before the variable is assigned a value. Another example would be to perform strong type checking when the language does not require such. Thus, they are better at locating likely errors, versus actual errors. As a result, these tools have a reputation of false positives. The old Unix lint program is an early example.

For debugging electronic hardware (e.g., computer hardware) as well as low-level software (e.g., BIOSes, device drivers) and firmware, instruments such as oscilloscopes, logic analyzers or in-circuit emulators (ICEs) are often used, alone or in combination. An ICE may perform many of the typical software debugger's tasks on low-level software and firmware.

14.2   Debugging process

Print debugging is the act of watching (live or recorded) trace statements, or print statements, that indicate the flow of execution of a process.

Often the first step in debugging is to attempt reproduce the problem. This can be a non-trivial task, for example in case of parallel processes or some unusual software bugs. Also specific user environment and usage history can make it difficult to reproduce the problem.

After the bug is reproduced, the input of the program needs to be simplified to make it easier to debug. For example, a bug in a compiler can make it crash when parsing some large source file. However, after simplification of the test case, only few lines from the original source file can be sufficient to reproduce the same crash. Such simplification can be made manually, using divide-and-conquer approach. The programmer will try to remove some parts of original test case and check if the problem still exists. When debugging the problem in GUI, the programmer will try to skip some user interaction from the original problem description and check if remaining actions are sufficient for bug to appear. To automate test case simplification, delta debugging methods can be used. Delta Debugging automates the scientific method of debugging. The algorithm isolates failure causes automatically by systematically narrowing down failure, inducing circumstances until a minimal set remains. Delta Debugging has been applied to isolate failure-inducing program input (e.g. an HTML page that makes a Web browser fail), failure-inducing user interaction (e.g. the keystrokes that make a program crash), or failure-inducing changes to the program code (e.g. after a failing regression test).

After the test case is sufficiently simplified, a programmer can use a debugger to examine program states (values of variables, call stack) and track down the origin of the problem. Alternatively a tracing can be used. In simple case the tracing is just a few print statements, which print out the values of variables in certain points of program execution.

Remote Debugging is the process of debugging a program running on a system different than the debugger. To start remote debugging, debugger connects to a remote system over a network. Once connected, debugger can control the execution of the program on the remote system and retrieve information about its state .

Post-Mortem Debugging is the act of debugging the (core) dump of process. The dump of the process space may be obtained automatically by the system, or manually by the interactive user. crash dumps (core dumps) are often generated after a process has terminated due to an unhandled exception .

14.3   Anti-debugging

Anti-debugging is "the implementation of one or more techniques within computer code that hinders attempts at reverse engineering or debugging a target process". The types of technique are:

  • API-based: check for the existence of a debugger using system information
  • Exception-based: check to see if exceptions are interfered with
  • Process and thread blocks: check whether process and thread blocks have been manipulated
  • Modified code: check for code modifications made by a debugger handling software breakpoints
  • Hardware- and register-based: check for hardware breakpoints and CPU registers
  • Timing and latency: check the time taken for the execution of instructions

Debugging can be inhibited by using one or more of the above techniques. There are enough anti-debugging techniques available to sufficiently protect software against most threats.

16   Project Planning

Project planning is part of project management, which relates to the use of schedules such as Gantt charts to plan and subsequently report progress within the project environment.

Initially, the project scope is defined and the appropriate methods for completing the project are determined. Following this step, the durations for the various tasks necessary to complete the work are listed and grouped into a work breakdown structure. The logical dependencies between tasks are defined using an activity network diagram that enables identification of the critical path. Float or slack time in the schedule can be calculated using project management software. Then the necessary resources can be estimated and costs for each activity can be allocated to each resource, giving the total project cost. At this stage, the project plan may be optimized to achieve the appropriate balance between resource usage and project duration to comply with the project objectives. Once established and agreed, the plan becomes what is known as the baseline. Progress will be measured against the baseline throughout the life of the project. Analyzing progress compared to the baseline is known as earned value management.

16.1   Mind Mapping

A mind map is a diagram used to represent words, ideas, tasks, or other items linked to and arranged radially around a central key word or idea. Mind maps are used to generate, visualize, structure, and classify ideas, and as an aid in study, organization, problem solving, decision making, and writing.

The elements of a given mind map are arranged intuitively according to the importance of the concepts, and are classified into groupings, branches, or areas, with the goal of representing semantic or other connections between portions of information. Mind maps may also aid recall of existing memories.

By presenting ideas in a radial, graphical, non-linear manner, mind maps encourage a brainstorming approach to planning and organizational tasks. Though the branches of a mindmap represent hierarchical tree structures, their radial arrangement disrupts the prioritizing of concepts typically associated with hierarchies presented with more linear visual cues. This orientation towards brainstorming encourages users to enumerate and connect concepts without a tendency to begin within a particular conceptual framework.

The mind map can be contrasted with the similar idea of concept mapping. The former is based on radial hierarchies and tree structures denoting relationships with a central governing concept, whereas concept maps are based on connections between concepts in more diverse patterns.

Guru_Mindmap

Mindmap

16.1.1   History

Mind maps (or similar concepts) have been used for centuries in learning, brainstorming, memory, visual thinking, and problem solving by educators, engineers, psychologists, and others. Some of the earliest examples of mind maps were developed by Porphyry of Tyros, a noted thinker of the 3rd century, as he graphically visualized the concept categories of Aristotle. Philosopher Ramon Llull (1235 - 1315) also used mind maps.

The semantic network was developed in the late 1950s as a theory to understand human learning and developed into mind maps by Allan Collins and M. Ross Quillian during the early 1960s. Due to his commitment and published research, and his work with learning, creativity, and graphical thinking, Collins can be considered the father of the modern mind map.

British popular psychology author Tony Buzan claims to have invented modern mind mapping. He claimed the idea was inspired by Alfred Korzybski's general semantics as popularized in science fiction novels, such as those of Robert A. Heinlein and A. E. van Vogt. Buzan argues that while 'traditional' outlines force readers to scan left to right and top to bottom, readers actually tend to scan the entire page in a non-linear fashion. Buzan also uses popular assumptions about the cerebral hemispheres in order to promote the exclusive use of mind mapping over other forms of note making.

The mind map continues to be used in various forms, and for various applications including learning and education (where it is often taught as 'Webs', 'Mind webs', or 'Webbing'), planning, and in engineering diagramming.

When compared with the concept map (which was developed by learning experts in the 1970s) the structure of a mind map is a similar radial, but is simplified by having one central key word.

16.1.2   Uses

A mind map is often created around a single word or text, placed in the center, to which associated ideas, words and concepts are added.

Mind maps have many applications in personal, family, educational, and business situations, including notetaking, brainstorming (wherein ideas are inserted into the map radially around the center node, without the implicit prioritization that comes from hierarchy or sequential arrangements, and wherein grouping and organizing is reserved for later stages), summarizing, revising, and general clarifying of thoughts. One could listen to a lecture, for example, and take down notes using mind maps for the most important points or keywords. One can also use mind maps as a mnemonic technique or to sort out a complicated idea. Mind maps are also promoted as a way to collaborate in color pen creativity sessions.

Mindmaps can be drawn by hand, either as 'rough notes' during a lecture or meeting, for example, or can be more sophisticated in quality. Examples of both are illustrated. There are also a number of software packages available for producing mind maps (see below).

16.1.3   Effectiveness in learning

Buzan[1] claims that the mind map is a vastly superior note taking method because it does not lead to a "semi-hypnotic trance" state induced by other note forms. Buzan also argues that the mind map utilizes the full range of left and right human cortical skills, balances the brain, taps into the alleged 99% of your unused mental potential, as well as intuition (which he calls "superlogic"). However, scholarly research suggests that such claims may actually be marketing hype based on misconceptions about the brain and the cerebral hemispheres. Critics argue that hemispheric specialization theory has been identified as pseudoscientific when applied to mind mapping.[2]

Scholarly research by Farrand, Hussain, and Hennessy (2002) found that the mind map technique had a limited but significant impact on memory recall in undergraduate students (a 10% increase over baseline for a 600-word text only) as compared to preferred study methods (a −6% increase over baseline). This improvement was only robust after a week for those in the mind map group, and there was a significant decrease in motivation compared to the subjects' preferred methods of note taking. Farrand et al. suggested that learners preferred to use other methods because using a mind map was an unfamiliar technique, and its status as a "memory enhancing" technique engendered reluctance to apply it.[3] Pressley, VanEtten, Yokoi, Freebern, and VanMeter (1998) found that learners tended to learn far better by focusing on the content of learning material rather than worrying over any one particular form of note taking.[4]

16.1.4   Tools

Mind mapping software can be used effectively to organize large amounts of information, combining spatial organization, dynamic hierarchical structuring and node folding. Software packages can extend the concept of mind mapping by allowing individuals to map more than thoughts & ideas with information on their computers and the internet, like spreadsheets, documents, internet sites and images.

16.1.5   Trademarks

The use of the term "Mind Maps" is trademarked by The Buzan Organisation, Ltd. in the UK[5] and the USA.[6] The trade-mark does not appear in the records of the Canadian Intellectual Property Office.[7] In the U.S. "Mind Maps" is trademarked as a "service mark" expressly for "EDUCATIONAL SERVICES, NAMELY, CONDUCTING COURSES IN SELF-IMPROVEMENT" – other products and services are not covered by the trademark.

16.2   Flow Charting

A flowchart is common type of chart, that represents an algorithm or process, showing the steps as boxes of various kinds, and their order by connecting these with arrows. Flowcharts are used in analyzing, designing, documenting or managing a process or program in various fields.

LampFlowchart

FlowchartExample

16.2.1   Overview

Flowcharts are used in designing and documenting complex processes. Like other types of diagram, they help visualize what is going on and thereby help the viewer to understand a process, and perhaps also find flaws, bottlenecks, and other less-obvious features within it. There are many different types of flowcharts, and each type has its own repertoire of boxes and notational conventions. The two most common types of boxes in a flowchart are:

  • a processing step, usually called activity, and denoted as a rectangular box
  • a decision, usually denoted as a diamond.

A flowchart is described as "cross-functional" when the page is divided into different swimlanes describing the control of different organizational units. A symbol appearing in a particular "lane" is within the control of that organizational unit. This technique allows the author to locate the responsibility for performing an action or making a decision correctly, showing the responsibility of each organizational unit for different parts of a single process.

Flowcharts depict certain aspects of processes and they are usually complemented by other types of diagram. For instance, Kaoru Ishikawa defined the flowchart as one of the seven basic tools of quality control, next to the histogram, Pareto chart, check sheet, control chart, cause-and-effect diagram, and the scatter diagram[2]. Similarly, in UML, a standard concept-modeling notation used in software development, the activity diagram, which is a type of flowchart, is just one of many different diagram types.

Nassi-Shneiderman diagrams are an alternative notation for process flow.

16.2.2   History

The first structured method for documenting process flow, the "flow process chart", was introduced by Frank Gilbreth to members of ASME in 1921 as the presentation “Process Charts—First Steps in Finding the One Best Way”. Gilbreth's tools quickly found their way into industrial engineering curricula. In the early 1930s, an industrial engineer, Allan H. Mogensen began training business people in the use of some of the tools of industrial engineering at his Work Simplification Conferences in Lake Placid, New York.

A 1944 graduate of Mogensen's class, Art Spinanger, took the tools back to Procter and Gamble where he developed their Deliberate Methods Change Program. Another 1944 graduate, Ben S. Graham, Director of Formcraft Engineering at Standard Register Corporation, adapted the flow process chart to information processing with his development of the multi-flow process chart to displays multiple documents and their relationships. In 1947, ASME adopted a symbol set derived from Gilbreth's original work as the ASME Standard for Process Charts.

According to Herman Goldstine, he developed flowcharts with John von Neumann at Princeton University in late 1946 and early 1947.[3]

Flowcharts used to be a popular means for describing computer algorithms. They are still used for this purpose; modern techniques such as UML activity diagrams can be considered to be extensions of the flowchart. However, their popularity decreased when, in the 1970s, interactive computer terminals and third-generation programming languages became the common tools of the trade, since algorithms can be expressed much more concisely and readably as source code in such a language. Often, pseudo-code is used, which uses the common idioms of such languages without strictly adhering to the details of a particular one.

16.2.3   Flow chart building blocks

Symbols

A typical flowchart from older Computer Science textbooks may have the following kinds of symbols:

Start and end symbols
Represented as lozenges, ovals or rounded rectangles, usually containing the word "Start" or "End", or another phrase signaling the start or end of a process, such as "submit enquiry" or "receive product".
Arrows
Showing what's called "flow of control" in computer science. An arrow coming from one symbol and ending at another symbol represents that control passes to the symbol the arrow points to.
Processing steps
Represented as rectangles. Examples: "Add 1 to X"; "replace identified part"; "save changes" or similar.
Input/Output
Represented as a parallelogram. Examples: Get X from the user; display X.
Conditional or decision
Represented as a diamond (rhombus). These typically contain a Yes/No question or True/False test. This symbol is unique in that it has two arrows coming out of it, usually from the bottom point and right point, one corresponding to Yes or True, and one corresponding to No or False. The arrows should always be labeled. More than two arrows can be used, but this is normally a clear indicator that a complex decision is being taken, in which case it may need to be broken-down further, or replaced with the "pre-defined process" symbol.

A number of other symbols that have less universal currency, such as:

  • A Document represented as a rectangle with a wavy base;
  • A Manual input represented by parallelogram, with the top irregularly sloping up from left to right. An example would be to signify data-entry from a form;
  • A Manual operation represented by a trapezoid with the longest parallel side at the top, to represent an operation or adjustment to process that can only be made manually.
  • A Data File represented by a cylinder

Flowcharts may contain other symbols, such as connectors, usually represented as circles, to represent converging paths in the flow chart. Circles will have more than one arrow coming into them but only one going out. Some flow charts may just have an arrow point to another arrow instead. These are useful to represent an iterative process (what in Computer Science is called a loop). A loop may, for example, consist of a connector where control first enters, processing steps, a conditional with one arrow exiting the loop, and one going back to the connector. Off-page connectors are often used to signify a connection to a (part of another) process held on another sheet or screen. It is important to remember to keep these connections logical in order. All processes should flow from top to bottom and left to right.

16.2.4   Examples

A flowchart for computing factorial N (N!) Where N! = 1 * 2 * 3 ... N. This flowchart represents a "loop and a half" — a situation discussed in introductory programming textbooks that requires either a duplication of a component (to be both inside and outside the loop) or the component to be put inside a branch in the loop.

16.2.5   Types of flow charts

There are many different types of flowcharts. On the one hand there are different types for different users, such as analysts, designers, engineers, managers, or programmers.[4] On the other hand those flowcharts can represent different types of objects. Sterneckert (2003) divides four more general types of flowcharts:[4]

  • Document flowcharts, showing a document flow through system
  • Data flowcharts, showing data flows in a system
  • System flowcharts showing controls at a physical or resource level
  • Program flowchart, showing the controls in a program within a system

However there are several of these classifications. For example Andrew Veronis (1978) named three basic types of flowcharts: the system flowchart, the general flowchart, and the detailed flowchart.[5] That same year Marilyn Bohl (1978) stated "in practice, two kinds of flowcharts are used in solution planning: system flowcharts and program flowcharts...".[6] More recently Mark A. Fryman (2001) stated that there are more differences. Decision flowcharts, logic flowcharts, systems flowcharts, product flowcharts, and process flowcharts are "just a few of the different types of flowcharts that are used in business and government.[7]

16.2.6   Software

Manual

Any vector-based drawing program can be used to create flowchart diagrams, but these will have no underlying data model to share data with databases or other programs such as project management systems or spreadsheets. Some tools offer special support for flowchart drawing, e.g., ConceptDraw, SmartDraw, EDraw Flowchart, Visio, and OmniGraffle.

Automatic

Many software packages exist that can create flowcharts automatically, either directly from source code, or from a flowchart description language. For example, Graph::Easy, a Perl package, takes a textual description of the graph, and uses the description to generate various output formats including HTML, ASCII or SVG.

17   Program Maintenance

Program maintenance in software engineering is the modification of software after delivery to correct faults, to improve performance or other attributes, or to adapt the product to a modified environment.”

This international standard describes the 6 software maintenance processes as:

  1. The implementation processes contains software preparation and transition activities, such as the conception and creation of the maintenance plan, the preparation for handling problems identified during development, and the follow-up on product configuration management.
  2. The problem and modification analysis process, which is executed once the application has become the responsibility of the maintenance group. The maintenance programmer must analyze each request, confirm it (by reproducing the situation) and check its validity, investigate it and propose a solution, document the request and the solution proposal, and, finally, obtain all the required authorizations to apply the modifications.
  3. The process considering the implementation of the modification itself.
  4. The process acceptance of the modification, by checking it with the individual who submitted the request in order to make sure the modification provided a solution.
  5. The migration process (platform migration, for example) is exceptional, and is not part of daily maintenance tasks. If the software must be ported to another platform without any change in functionality, this process will be used and a maintenance project team is likely to be assigned to this task.
  6. Finally, the last maintenance process, also an event which does not occur on a daily basis, is the retirement of a piece of software.

There are a number of processes, activities and practices that are unique to maintainers, for example:

  • Transition: a controlled and coordinated sequence of activities during which a system is transferred progressively from the developer to the maintainer;
  • Service Level Agreements (SLAs) and specialized (domain-specific) maintenance contracts negotiated by maintainers;
  • Modification Request and Problem Report Help Desk: a problem-handling process used by maintainers to prioritize, documents and route the requests they receive;
  • Modification Request acceptance/rejection: modification request work over a certain size/effort/complexity may be rejected by maintainers and rerouted to a developer.

Software maintenance and evolution of systems was first addressed by Meir M. Lehman in 1969. Over a period of twenty years, his research led to the formulation of eight Laws of Evolution. Key findings of his research include that maintenance is really evolutionary developments and that maintenance decisions are aided by understanding what happens to systems (and software) over time. Lehman demonstrated that systems continue to evolve over time. As they evolve, they grow more complex unless some action is taken to reduce the complexity.

The key software maintenance issues are both managerial and technical. Key management issues are: alignment with customer priorities, staffing, which organization does maintenance, estimating costs. Key technical issues are: limited understanding, impact analysis, testing, maintainability measurement.

17.1   Categories of maintenance

E.B. Swanson initially identified three categories of maintenance: corrective, adaptive, and perfective. These have since been updated and ISO/IEC 14764 presents:

  • Corrective maintenance: Reactive modification of a software product performed after delivery to correct discovered problems.
  • Adaptive maintenance: Modification of a software product performed after delivery to keep a software product usable in a changed or changing environment.
  • Perfective maintenance: Modification of a software product after delivery to improve performance or maintainability.
  • Preventive maintenance: Modification of a software product after delivery to detect and correct latent faults in the software product before they become effective faults.

All of the preceding take place when there is a known requirement for change.

18   MDBMPS Programming

Depending on the nature of the work involved, MDBMPS will make useof one or more of these previously mentioned methods. A basic overview ofsome of the patterns to be employed will be presented in this section. Wewill follow up with the specification sections Metadata Specifications,Code Generator Specifications, and Delivery Specifications.

Finally we arrive at the question (and answer), "What is the MetaData Based MetaProgramming System (MDBMPS)?"It is a system of code templates, code snippets, embedded code, and dataspecifications which, when combined, generate code for application programsusing the style, rules, preferences and code of the programmer using thesystem. We're now going to break each one of these down.

18.1   MetaData-Based

Data-based and data-driven applications are the factors that define thecontrol flow of the program. One of the key factors of manydata-based/data-driven applications is writing as little fixed code aspossible. One of the most important methods of implementingdata-based/data-driven applications or metaprogramming is to always pushproblems upstream. Finding a way to enhance the application instead ofhacking the application/metaprogram will save time and effort fromhand-patching code.

The MDBMPS follows this pattern. As much as possible, everythingabout an application, will be described in cascading levels of data. Thisprovides the further advantage that the application becomes languageagnostic. So that there are various generators for various languages. Infact one language might have different generators.

For example, rather than storing a bit of code to generate a screen, theinformation about the screen is saved. Such things as Screen size,background and foreground color, font and so on. Then these can be accessedand used or discarded by the language of any particular generator. Languageswhich allow for a different font can make use of this data, whereas thosethat do not, can simply discard it.

The data cascades both up and down. So that each part of the application,interacts with or influences the other parts. Screen colors can effect fieldcolors, but these can have their own values. As well a field when it appearsmight change the color of the entire window or screen.

Anything in an app would be part of this Metadata. Further details will beprovided in the repositories section. Some examples might include (butcertainly not be limited to) data about:

  • Fields
  • Tables
  • Databases
  • Screens
  • Popups
  • Buttons
  • Lists
  • Timed Events
  • On Save/Update/Delete events
  • Reports
  • Templates
  • Snippets
  • Embed Points
  • Security
  • Labels
  • Localization

As stated, this is only a small list. It will be added to as MDBMPSbecomes more clearly defined and the needs of the programmers comes closerinto focus.

18.2   Data-based vs Metaprogramming

Many developers have faced this dilemma and chosen a side. The concept ofcombining both methods will allow the benefits of both methods to be usedand increase productivity. It will though avoid the pitfalls of choosingone or the other method.

For instance as mentioned in the section on case tools, we'll find that manycode generation environments will save the programmers input as data,combine this with hard coded skeletons (templates) and then output codewhich the programmer has little control over. So in using a Data Basedmethod, the programmer has given up significant control.

If on the other hand a strictly metaprogramming method were to be employed,the programmer would find that many processes would have to be repeated fromone application to another. In the least, copying and pasting would stillhave to be one of his main methods of re-use.

As much as the programmer is able, he would wish to avoid any pitfalls inhis coding techniques.

18.3   Combining methods

The combined methodology is that important information about the applicationthat would normally be written in code is instead stored as data. Themetaprogram reads the stored data and combines the stored data withtemplates, snippets and other code to produce the required final output,which is a program in itself.

So this provides the programmer with a robust, fully functional system, acomplete cohesive unit. He then gains the benefits of both methods andindeed many other methods as well. To wit, important attributes about aprogram are stored merely as data. Templates allow different patterns ofmetaprogramming to be employed. These patterns could include differentcoding styles, improvement by a programmer of previous programming methods,or even generating the very same application in a completely differentprogramming language. Snippets find there use as well, so that not even thetiny bits that a programmer has created need to be discarded. Nor though, dothey need to be copied and pasted, as the system handles this on its own.

18.4   Templates

One might think of a template system as a stencil for code. Templating isused in many facets of MIS/IT industries. For instance HTML templates inlanguages such as PHP allow for websites to be designed with the same basiclayout, while presenting different information to the web surfer. Templatingis useful though, in nearly all programming languages. Consider the earlierCOBOL code fragments with some deletions and additions:

Pseudocode

Use file called customer.dat and refer to it as custom-filewith an index and create record key called customer-id

Allow file to set variable customer-record

Set variables for customer-record:

  1. customer-id
  2. customer-firstname
  3. customer-lastname

Open file custom-file

Set extra spaces to the variable customer-record to fill it completely

Set extra zeros to customer-id to fill it completely

Display the text "Enter ID:" and let user input data into customer-id

Display the text "Enter first name:" and let user inputdata into customer-firstname

Display the text "Enter last name:" and let the user inputdata into customer-lastname

Write all user inputted data into variable customer-record

Close file customer-file

Real World (COBOL)
ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.
FILE-CONTROL.

    SELECT CUSTOMER-FILE
        ASSIGN TO "CUSTOMER.DAT"
        ORGANIZATION IS INDEXED
        RECORD KEY IS CUSTOMER-ID
        ACCESS MODE IS DYNAMIC.

DATA DIVISION.
FILE SECTION.

FD  CUSTOMER-FILE
    LABEL RECORDS ARE STANDARD.
01  CUSTOMER-RECORD.
    05  CUSTOMER-ID                        PIC 9(10).
    05  CUSTOMER-FIRSTNAME                 PIC X(50).
    05  CUSTOMER-LASTNAME                  PIC X(50).

PROCEDURE DIVISION.

  OPEN I-O CUSTOMER-FILE
  MOVE SPACES TO CUSTOMER-RECORD
  MOVE ZEROES TO CUSTOMER-ID

  DISPLAY "ENTER ID:"
  ACCEPT CUSTOMER-ID

  DISPLAY "ENTER FIRST NAME:
  ACCEPT CUSTOMER-FIRSTNAME

  DISPLAY "ENTER LAST NAME:"
  ACCEPT CUSTOMER-LASTNAME

  WRITE CUSTOMER-RECORD
  CLOSE CUSTOMER FILE
  GOBACK.

Naturally a typical program would likely be much larger than this. Weremoved a number of fields to simplify the example. The program, opens afile for input and output (so that it can save data to the pre-existingfile), displays a request for each field, accepts user input for each field,saves the results and then closes the file and exist the program.

When we think about this simple program, we might imagine the probabilitythat nearly every program, or at least module, that would write one recordto a file from user input, would have this very same layout. There would bedifferences of course, such as the file name, field layouts and so on. Inthis example let's imagine that we have a similar layout, not just forCustomers, but for Friends, Patients, Neighbors, Club Members and so on. Allof these have only an ID, FIRSTNAME and LASTNAME. So let's turn this into apsuedo template to demonstrate how we might handle this:

Pseudocode

Use file called customer.dat and refer to it as {tablename}-filewith an index and create record key called {tablename}-id

Allow file to set variable {tablename}-record

Set variables for {tablename}-record:

  1. {tablename}-id
  2. {tablename}-firstname
  3. {tablename}-lastname

Open file {tablename}-file

Set extra spaces to the variable {tablename}-record to fill it completely

Set extra zeros to {tablename}-id to fill it completely

Display the text "Enter ID:" and let user input data into {tablename}-id

Display the text "Enter first name:" and let user inputdata into {tablename}-firstname

Display the text "Enter last name:" and let the user inputdata into {tablename}-lastname

Write all user inputted data into variable {tablename}-record

Close file {tablename}-file

Real World (COBOL)
ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.
FILE-CONTROL.

    SELECT {TABLENAME}-FILE
        ASSIGN TO "{TABLENAME}.dat"
        ORGANIZATION IS INDEXED
        RECORD KEY IS {TABLENAME}-ID
        ACCESS MODE IS DYNAMIC.

DATA DIVISION.
FILE SECTION.

FD  {TABLENAME}-FILE
    LABEL RECORDS ARE STANDARD.
01  {TABLENAME}-RECORD.
    05  {TABLENAME}-ID                        PIC 9(10).
    05  {TABLENAME}-FIRSTNAME                 PIC X(50).
    05  {TABLENAME}-LASTNAME                  PIC X(50).

PROCEDURE DIVISION.

  OPEN I-O {TABLENAME}-FILE
  MOVE SPACES TO {TABLENAME}-RECORD
  MOVE ZEROES TO {TABLENAME}-ID

  DISPLAY "ENTER ID:"
  ACCEPT {TABLENAME}-ID

  DISPLAY "ENTER FIRST NAME:
  ACCEPT {TABLENAME}-FIRSTNAME

  DISPLAY "ENTER LAST NAME:"
  ACCEPT {TABLENAME}-LASTNAME

  WRITE {TABLENAME}-RECORD
  CLOSE {TABLENAME} FILE
  GOBACK.

We would save this into an ordinary text file. Then using really just aboutany language under the sun, we could read in this text file, use thelangauge's facilities to replace {TABLENAME} with FRIENDS, NEIGHBORS,PATIENTS, CLUBMEMBERS and so on. To demonstrate how simple this is, I'll useboth COBOL and MySQL as examples. First MYSQL. We'll assume that the valuesabove are in a table called `templates` and the column which contains theactual code is `code` and lastly that we know the exact `id` of thetemplate we want to use is 23. So here is the example:

Pseudocode
Replace all occurrences of "*{TEMPLATE}*"
in variable called *code* with "*friends*" and
save into file called "*friends*"
Real World (SQL)
SET @group_concat_max = 99999999999;

SELECT REPLACE(`code`,'{TEMPLATE}','FRIENDS')
  INTO DUMPFILE '/tmp/friends.cob'
  FROM `templates`
  WHERE `id` = 23;

So then one could simply copy the file /tmp/friends.cob and be set to go(assuming the above was completely proper COBOL, it is missing some keyparts thoug.) Of course such code might be part of a larger applicationwritten in another language, such as COBOL or PHP which would prepare allthe needed values and then send that statement to the MySQL processor, thengrab the output itself (with or without the INTO DUMPFILE statement).

Now let's look at the example in cobol, making some of the differentassumptions. Our template is in a text file ./templates/saveperson.tpl. Wewant the output to go into ./output/friends.cob. It should be noted thatthe SUBSTITUTE function is unique to Open Cobol.Now let's see our example:

Pseudocode

Open file templates/saveperson.tpl and call it template-file and access byeach line and use as template-record

Open file output/friends.cob and call it output-file and accessby each line and use as output-record

For each template line, replace '{TABLE}' with 'friends' and store intooutput-record

close template-fileclose output-file

Real World (COBOL)
 ENVIRONMENT DIVISION.
 INPUT-OUTPUT SECTION.
 FILE-CONTROL.

  SELECT Template-File ASSIGN TO 'templates/saveperson.tpl'
  ORGANIZATION IS LINE SEQUENTIAL.

  SELECT Output-File ASSIGN TO 'output/friends.cob'
  ORGANIZATION IS LINE SEQUENTIAL.

 DATA DIVISION.
 FILE SECTION.

 FD Template-File.
 01 Template-Record.
    88 Template-File-EOF VALUE HIGH-VALUES.
    10 Template-Line     PIC X(500).

 FD Output-File.
 01 Output-Record.
    10 Output-Line     PIC X(500).

 WORKING-STORAGE SECTION.
 01 NEWLINE           CONSTANT x"0a".

 PROCEDURE DIVISION.

   OPEN INPUT Template-File
   OPEN OUTPUT Output-File

   READ Template-File
    AT END SET Template-File-EOF TO TRUE
   END-READ

   PERFORM UNTIL Template-File-EOF
     STRING
       FUNCTION TRIM(
         FUNCTION SUBSTITUTE(
           Template-Line,
           '{TABLE}',
           'FRIENDS'
       ) )
      DELIMITED BY SIZE NEWLINE
      INTO Output-Line

      WRITE Output-Record

     READ Template-File
       AT END SET Template-File-EOF TO TRUE
     END-READ
   END-PERFORM

 CLOSE Template-File
 CLOSE Output-File

GOBACK.

Of course for this application to be fully usable in generating code, aswith the MySQL example, one would want user interaction. This way, aprogrammer can choose from a list what templates he wants. Further changescould be made to the template so that the code would be generated properly,regardless of the number of fields. But it is our hope that this exampledemonstrates just how easy it is to set up a template based system.

The MDBMPS templating system will likely store the templatesdirectly in a DBMS. This will allow for searching and combining templatesin ways the creators will not have envisioned.

18.5   Snippets

Snippets are similar to templates. They differ in a few ways though. Whereasa template is created and meant to include other code, a snippet is meantto be included. Snippets might typically be smaller than a template aswell. Though it should be noted that they do not have to be small by anymeans. However, if you use too many large snippets, you will likely findthat you are not using the MDBMPS to its fullest capacity. Snippetsalso have the ability include other snippets.

Alright, now let's provide a couple of examples. Suppose that you wish tohave a copyright statement and an author mentioned in every application youwrite. In COBOL it might look something like this:

Pseudocode
:Author: John Hancock
:Copyright: 2009 All rights reserved
Real World (COBOL)
*>Author: John Hancock
*>Copyright: 2009 All rights reserved

Obviously this is a very small code fragment. Nevertheless, retyping, orcopying and pasting in every application you write can be tedious. Alsothere is the potential for an error, perhaps typing "All rights reserved" bymistake. By storing this as a snippet, you free yourself from thesepossibilities. So let's take our previous template and modify it to usethis snippet:

Pseudocode

{@snippet=2373}

Open file named {tablename}.dat using the variable {tablename}-fileand organize the index using variable {tablename}-id

for variable {tablename}-record, set the following variables:

  1. {tablename}-id
  2. {tablename}-firstname
  3. {tablename}-lastname

Use {template}-file as input and output file

Move spaces to variable {tablename}-record to fill it completely

Move zeroes to variable {tablename}-id to fill it completely

Show to user the text "Enter ID:"

Allow user to input text into variable {tablename}-id

Show to user the text "Enter First Name:"

Allow user to input text into variable {tablename}-firstname

Show to user the text "Enter Last Name:"

Allow user to input text into variable {tablename}-lastname

Store user input to variable {tablename}-record

Close file {tablename}-file

Real World (COBOL)
{@snippet-2373}
ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.

FILE-CONTROL.

    SELECT {TABLENAME}-FILE
        ASSIGN TO "{TABLENAME}.dat"
        ORGANIZATION IS INDEXED
        RECORD KEY IS {TABLENAME}-ID
        ACCESS MODE IS DYNAMIC.

DATA DIVISION.
FILE SECTION.

FD  {TABLENAME}-FILE
    LABEL RECORDS ARE STANDARD.
01  {TABLENAME}-RECORD.
    05  {TABLENAME}-ID                        PIC 9(10).
    05  {TABLENAME}-FIRSTNAME                 PIC X(50).
    05  {TABLENAME}-LASTNAME                  PIC X(50).

PROCEDURE DIVISION.
  OPEN I-O {TABLENAME}-FILE

  MOVE SPACES TO {TABLENAME}-RECORD
  MOVE ZEROES TO {TABLENAME}-ID

  DISPLAY "ENTER ID:"
  ACCEPT {TABLENAME}-ID

  DISPLAY "ENTER FIRST NAME:
  ACCEPT {TABLENAME}-FIRSTNAME

  DISPLAY "ENTER LAST NAME:"
  ACCEPT {TABLENAME}-LASTNAME

  WRITE {TABLENAME}-RECORD
  CLOSE {TABLENAME} FILE
  GOBACK.

Our code generator then would be designed to look additionally for anythinglabeled {@snippet- and then to use the number after the hymen as the IDlookup in the snippet file (table), then to replace the the {@snippet-###}with the contents from that record. As you can see, this allows common codeto be used across many templates.

Snippets can be used for a variety of functions. Setting up, cleaning up,functions, common calculations, are just a few of the areas that snippetswill find their use in the system.

18.6   Generated Code

Now that we have the templates and snippets where do we go? Well, togenerated code. Specifically, generated code is what will most often'fill in the blanks' of the templates (and even a snippet or two!). In ourexamples on templates we saw that filling in the blanks being done via hardcode, or in the case of MySQL, via user input. Indeed much of the generatedcode will in fact be created by user input. Table names, field names, fieldsizes, types and so on, will all be input into a database by the programmer.Depending on the language, these values will be wrapped as necessary, thensent to a template. Consider the example of FirstName. Perhaps we have theinformation about FirstName's requirements stored as simply as:

Pseudocode
Field name uses "FirstName",
Form label uses "First name",
Table label uses "Fname",
Field type is a "string",
Field size is "50"
Real World (COBOL)
Field Name : FirstName
Form Label : First Name
Table Label: Fname
Field Type : String
Field Size : 50

Different generators could read these values, interprete them as needed. Sofor the layout in the file (or table) the generator would output eitherCOBOL or MySQL as follows:

Pseudocode
Variable firstname is 50 characters
Real World (COBOL)
10 FirstName   PIC X(50).
Pseudocode
Variable firstname is 50 characters
Real World (SQL)
`FirstName` varchar(50) NOT NULL default '',

Of course we have also specified the label the field is to have when it ison a form or in a table, like a spreadsheet. Lets see the example COBOL codethat would be generated for a form and the MySQL code that would be generatedfor a table:

Pseudocode
Show to user the text "First Name:" and let user input data into firstname
Real World (COBOL)
DISPLAY "First Name:"
ACCEPT FirstName
Pseudocode
Grab table column firstname and use it as variable fname
Real World (SQL)
SELECT `FirstName` AS Fname
  FROM `FRIENDS`;

We hope that it is obvious that this facility could make the originaltemplate we created much more simple, robust and flexible. No longer wouldwe be limited to writing programs that just accessed one file, that had anID, FIRSTNAME and LASTNAME. Nor would we have to write a separate templatefor each file structure we wanted. Rather, using the facilities of the CodeGenerator, and a combination of logical data stored in tables, we would beable to generate rules based code into our templates. Of course thisprovides us with the further ability to combine templates in amazing ways.

It has to be clearly understood that, while we are using Fields in ourexamples, this concept of generated code, and settings stored in a table,apply across the MDBMPS. So there will be options for screens,reports, lookups, searching, deleting, and every other option that a modernapplication program is expected to have.

18.7   Embedded Code

What happens though, when you are using the code generator and and there isno option for a field to blank the screen before entering? Or turn thescreen blue, or fill the field with stars or any other option that ismissing? Or really any other option that you wish to perform?* Well, thisis where we turn to Embedded Code. Each section of MDBMPS willhave points for embedding code. Code can be embedded before a screen opens,after it does, before entering a field, after entering a field, while typingon a field, when opening or saving a file, when closing out of the program.Just about any place there is a Generated Code, there should be an optionto Embed code. So that makes it, hopefully, obvious that in your templatesany {OPTION} has at least a BEFORE_{OPTION} and an AFTER_{OPTION}.

* Of course these same questions apply to screen options or reportoptions and so on as mentioned in the section on Generated Code.

These embed points can be saved with the individual generic definitions offiles, fields, screens, programs, and so on. However, they can also bestored with a specific application's definition and use of these very samethings.

Let's consider an example. Suppose, that you have a password field.Typically you want *'s to appear as the user types. So, this has been addedas an option in the Field Definition specs. And for any password field, youturn it on. However, now you are writting an app designed to STOREpasswords. So you want the user to realize, when they type in their passwordhere, this is not a typical password field, after all it could displaycritial information on lots of their other accounts. So you decide toreplace * with % and you decide to make this password field use adifferent algorthym so it is even harder to crack.

In both cases the programmer adds an EMBED POINT. First an "AFTER KEYSTROKE"which replaces * with % for this particular password field. Then an "AFTERLEAVING FIELD" to change the algorthym. Realizing the utility of theseoptions, you naturally store the new password field in the FieldRepository as "SpecialPassword". From now on, all you have to do is'drop' the field "SpecialPassword" into a file, or on a screen, and all ofits attributes, including the new embed points will work.

Now, comes along another application, which needs the changed algorythm, butwish to have the password use & instead of %. Should you create a new field?Well you could of course. Certainly if this is a feature you are going towant in a lot of applications (say three or more) you certain shouldcreate a new field in the Field Repository. However, if this is a oneoff, you might be best served to choose the field from the Repository,and drop it into your app. Then, in your app, modify the Embed Point "AFTERKEYSTROKE" to suit your new needs. So the field retains it's originalvalues, but has your new wishes, but only for this single, one-offapplication.

It should be noted that embed points can cascade. So sometimes you mightcompletely override an embed point. Other times, you might add your changes,to the already existing ones. For instance keeping the % as the character tobe typed, but changing the color to Purple.

Embed points add a great deal of power to the MDBMPS. Especiallysince they can be saved on every part of the system.

18.8   Repositories

How then is all of this to be accomplished? Well the key is in thecentralization of all data and the reduction of all information about anapplication, and the system, into data. So that system will maintain anumber of databases in several formats. Each of these databases can bethought of as a respository, a storage warehouse, for the application andit's specifics. So each repository has like information stored.

For example there is the Code Repostitory. This will house any actual sourcecode that would be produced. So of course it would entail some of thefollowing:

  • Templates
  • Snippets
  • Embedded Code
  • Generated Code
  • Functions
  • Procedures

If the content is to be source code itself, programming statements and thelike, then one will find it in the Code Repository.

Then there will be the Attribute Repository. Attributes are key/value pairsthat will be read by the code generators to determine which templates andsnippets should be used when generating source code. Some examples for theAttribute Repository:

  • Field Attributes
  • Table Attributes
  • Screen Attributes
  • Label Attributes
  • User Attributes
  • List Attributes

So each of these describes its entity in full.

18.9   Decision Table to COBOL Translator

Revised: 06 January 2009 by V. B. Coen

18.9.1   Decision Tree Introduction

In the development of data processing procedures the most important and mostdifficult tasks are that of logical accumulation of the alternativespresented, the desired results, and the procedures necessary to attain thoseresults. The problem is made greater through lack of adequate communicationmethods between the analyst and the programmer who will adapt the problem toa computer's capabilities.

Computer languages are of little use to the analyst as they require theproblems to be defined in too much detail. Flow charts are a significantstep forward but the complexity of the problem tends to increase theconfusion factor in a flowchart.

Decision tables offer a simple, graphic means of defining all variables inquestion, the desired results, and the pertinent combinations of variablesthat may be presented to the problem. Decision tables provide a standardmeans of communication between the systems analyst and the programmers, andalso provides an excellent documentation device. And these are often usedwith and in Design Use Cases. The benefits of decision tables are:

  1. Programming is much faster.
  2. Debug time is reduced.
  3. Program maintenance is simpler.
  4. Use of tables leads to greater accuracy and completeness in problem formulation.
  5. Decision tables are a useful communication and documentation device.
18.9.2   Structure of a Decision Table

A decision table can be logically divided into four sections (see Figure 1below). The upper two sections (Condition Stub and Condition Entry) describethe set or string of conditions that is to be tested. The lower two sections(Action Stub and Action Entry) describe the set or string of actions that isto be taken upon the satisfaction of the above set of conditions. A ruleconsists of a set of conditions plus the actions to be performed uponsatisfaction of those conditions, and a decision table typically consists ofseveral rules.

Figure 1
Condition Stub      |         Condition Entry
____________________|________________________
                    |
Action Stub         |            Action Entry
Figure 2
     R1  R2                        R1  R2                  R1  R2
 C1  10  20                   C1  -50  XX             C1   Y   Y
 C2  AA  BB                   C2   Y    N             C2   Y   N
 A1   X                       A1   X                  A1   X
 A2   X   X                   A2        X

LIMITED-ENTRY               EXTENDED-ENTRY          MIXED-ENTRY
The three types of decision table in current useare the limited-, extended-, and mixed-entry
Figure 3. A limited entry table
      CREDIT-APPROVAL                 R1   R2   R3  ELS
SPECIAL-CLEARANCE = 1                            Y
SALARY = ACCEPTABLE                         Y    N
CREDIT-LIMIT > COST                    Y    N    N
PERFORM APPROVAL                       X    X    X
PERFORM ORDER-REJECT                                 X
A more detailed example of limited and extended entry tables
Figure 4. An example of an extended table
INSURANCE-APPROVAL          RULE 1       RULE 2       RULE 3    ELS
AGE >                       25           35           65
AGE IS LESS THAN            35           65
PHYSICAL CONDITION =        EXCELLENT    AVERAGE      AVERAGE
COMPUTE RATE-PER-1000 =     1.57         1.72         5.92
COMPUTE POLICY-LIMIT =      200000       100000       50000
PERFORM APPLICATION-REJECT                                       X

Note that the form of the individual conditions and actions is somewhatdifferent between Figure 3 & 4. In a limited entry table, the entirecondition or action is written in the stub and the condition entry islimited to "Y", "N" or blank, that is, asserting the presence of a truecondition, the presence of a false condition, or the ignoring of a specifiedcondition, respectively. An action entry is limited to "X" or blank. Thatis, executing (X) or skipping (blank) an action. In contrast, an extendedentry form has part of the condition or action extended directly into thecondition or action entry area. In mixed entry tables, both forms may beused within the same table, but any one horizontal row (condition) oraction) must be entirely limited or entirely extended.

Figure 4 is an example of a typical decision table. The use of such a tableis illustrated in the following statements about Rule 1:

Rule 1 says: If age is greater than 25, and age is less than 35, and healthis excellent, then rate per thousand is 1.57, and policy limit is 200,000.The underlined words are implied by the table layout. Each rule of adecision table is an alternative to each other rule. Therefore, logically itdoes not matter which rule is examined first; at most, one rule can besatisfied by a single set of conditions.

The program must still be told what to do if all rules have been consideredand the set of conditions that exist do not satisfy any of them. Therefore,the last rule in every decision table is the ELS-rule in which we tell theprogram what "else" to do if no rule is satisfied. An ELS is written in therule header entry as the last rule in the table.

The Decision Table Translator represents the next step in development oftable use by providing quick and accurate conversion of the graphic solution(Decision Table) to a COBOL source program.

18.9.3   Description of DECTRANS

Dectrans is written in Cobol and generates the same. Dectrans utilisesthree files. The input file is the user's program containing the decisiontables. It is immaterial to the program whether the input is a whole Cobolprogram or a group of disassociated tables. Any number of tables may beprocessed in one run. The first of two output files will be a filecontaining all of the source statements that were on the input file withthe tables being replaced by Cobol statements. The second output file is alisting showing all decision tables read, the source code produced fromeach table, and any warning or error messages. (See sample listing)

18.9.4   Program Format

The format for a Cobol program containing decision tables must conform tothe requirements for any Cobol program, except that each decision table isinserted in the PROCEDURE DIVISION at the point where it is to bereferenced.

The processor inserts the appropriate coding at the point where the table isencountered. The coding generated is in the form of a SECTION containing 4or more paragraphs, therefore, the entire table can be executed by a PERFORMstatement, a GO statement or an implied "fall through" to the firstinstruction in the SECTION.

If the output from Dectrans is to be in a compilable form, all data names orprocedure names referenced in a decision table must be defined in theappropriate Division in the program.

If a decision table is followed by user coding, a new SECTION header shouldfollow the decision table. (EODT entry)

The decision tables are placed in the PROCEDURE DIVISION at the point wherethe coding is to be executed. This is the only difference between a Cobolprogram with decision tables and the usual Cobol program.

Pseudocode
TO BE DONE
Real World (COBOL)
  IDENTIFICATION DIVISION.
  DATA DIVISION.
  FD  . . .


  PROCEDURE DIVISION.
  SECTION-NAME-L SECTION.
  ABC.
       ADD
       MOVE

THDR TABLE-2                         002001004
RHDR               001 002 003 ELS.
     A = B          Y   N   Y
     C = D          N   N   Y
     PERFORM XYZ    X   X   X
     PERFORM ERROR              X
EODT
  NEXT-SECTION SECTION.
  PAR-1.
       ADD
       MOVE
       IF
THDR
            etc.
18.9.5   Table Linkage

The coding generated by Dectrans will be in the form of a unique SECTION foreach table translated. Each routine will be complete with a section-nameconsisting of the table name with the word "SECTION" added. Each routinewill be terminated by an "EXIT" paragraph called "END000n" where n is 1 forthe first table, 2 for the second table, etc. Each string of actions willbe terminated by a "GO TO END000n" providing normal exit from the routine.It is permissible for the user to provide his own "GO TO ..." as an actionentry if he so desires. In this case a "G" must be in column 52 of the THDRentry. A "PERFORM ..." may be included in action entries referencing anothertable or a user provided routine.

The generated routine may be entered by use of (1) a "PERFORM" statement,(2) a "GO TO" statement, or (3) a fall through to the first instruction ofthe Section. If the user provides his own "GO TO .." in an action entry ofa table, the table should NOT be PERFORMED.

Pseudocode
TO BE DONE
Real World (COBOL)
     GO TO TABLE-1       (2)

          .

          .

     PERFORM TABLE-1.    (1)

     MOVE A TO B.

     ADD C TO B.         (3)

TABLE-1 SECTION.

PAR0001.

     IF____________
18.9.6   Table Header Entry THDR

The Header entry provides Dectrans with a unique name which will be used inthe SECTION header. The number of condition-stubs, action-stubs, and rulesare indicated as follows:

Entry Format
ColumnsContents
1 - 3Blank
4 - 7THDR
9 - 38Table name (30 chars)
41 - 43Number of conditions (3 digits)
44 - 46Number of actions (3 digits)
47 - 49Number of rules (3 digits) This must include the ELS Rule
50 - 51Blank
52GO TO Indicator. "G" will cause no GO TO END000n statement to begenerated and the user must provide the GO TO statement to leavethe table.
Entry Format Example
                                     4          5
4    9                               1          2

THDR TABLE-1                         007005004  G
18.9.7   Initialisation INIT (optional)

Actions that are always to be performed prior to the execution of the tablecan be specified by using the INIT entry followed by the statements to beexecuted. There is no limitations as to the coding that may appear in theINIT section except that another decision table may not appear.

The INIT entry follows the THDR entry and is terminated by the ENIT entry.The RHDR entry then follows the ENIT entry.

INIT Format Example
                                     4
4    9                               1

THDR XYZ                             001001001
INIT
        MOVE     SPACES TO DATA-NAME.
        ADD      1 TO COUNTER-A.
        SET      A UP BY 1.
        PERFORM  A THRU B.
ENIT

RHDR                    001  002  003  ELS.
18.9.8   Rule Header RHDR

The rule header entry specifies the format of the body of the table. Thedata area will begin in column 9. Rule numbers will be entered to definethe stub and entry fields. The stub area extends from column 9 to the firstcharacter of the first rule number. The stub area must not be more than 72characters or less than 10 characters.

The rule entry areas start at the first character of their respectivenumbers and extend to the first character of the next number. The rules mustnot be less than 3 nor more than 18 characters in length.

The last or ELSE rule is designated by an ELS rather than a rule number. Aperiod (full stop) terminates the ELS rule. If ELS is not present, the lastrule number is treated as the ELS rule. Even though the table may becomplete, the ELS rule is expected by the processor.

If more than one entry is needed to specify the table size, the informationcontinues in column 9 of subsequent entries.

It is not necessary for all rules to contain the same number of characters,but all entries within a rule must be the same length.

The Rule Header entry will be identified by a RHDR in columns 4 - 7.

Rule Header Example
4    9

RHDR                         001                002
     stub                    at least 3 and
     up to 72 characters     nor more than 18

001, 002, 003, etc, may be replaced by other characters if the user keeps inmind the not less than 3 and not greater than 18 rule. Example:

Example
001002003...009010011
1  2  3     9  10 11

001       002       003       ELS.

1         2         3
A         B         C
01        02        03
18.9.9   Body entries

The body entries will contain the body of the table and will correspond tothe format specified on the Rule Header. The placement of entries within aspecified field is free form. If more than one entry is needed for eachCondition or Action, sequence numbers should be used in columns 1 to 4.Columns 1 - 2 indicate the Condition/Action number and Columns 3 - 4indicate the entry number within the Condition/Action.

Example
  RHDR TABLE-A

                      001        002        003        004        005
       006        007        ELS

0101    A = B           Y          N          Y          Y          N
0102     N          Y
0201    FF > AX         N          Y          N          Y          N
0202     Y          N
18.9.10   Termination of the Decision Table EODT

The EODT entry signifies the end of the decision table and is formatted asfollows:

EODT Format
ColumnsContents
1 - 3Blank
4 - 7EODT
8Blank
9 - 80Comments
41 - 43Number of conditions (3 digits)
44 - 46Number of actions (3 digits)
47 - 49Number of rules (3 digits) This must include the ELS Rule
50 - 51Blank
52GO TO Indicator. "G" will cause no GO TO END000n statement to begenerated and the user must provide the GO TO statement to leavethe table.
Example
Col
  4

  EODT     end of validation check three
18.9.11   Discussion on Blank Entries

It has been stated earlier, that a blank entry in the condition entryportion of the table expresses a desire to ignore the indicated conditionfor the rule in question. In the case of an extended or mixed entry table,this is true. However, this cannot be taken so literally with limited entrytables as some user experiences have demonstrated.

In limited table processing, a blank entry causes the translator to assumethat the rule may be satisfied regardless of whether the condition is trueor false. This does not mean that the condition will be ignored. In somecases, it is possible that a blank entry will cause more coding to begenerated than the normal rule. This philosophy burdens the user with thetask of ensuring that a table contains logical and exclusive rules.

In the following examples, some common errors are demonstrated.

Example 1
_____________________________________________
_____________R1_______R2_______R3_______ELS__
___C1________Y________Y________Y_____________
___C2________Y________Y______________________
___C3_________________Y________N_____________

In example 1, the user's intention is fairly clear but the rules are notlogically unique. Rules 1 and 2 will be interpreted as being identicalbecause the blank entry may be true or false, and the true state would notbe unique. Rule 2 and 3 are unique, but 1 and 3 are not.

Example 2
_______________________________________
_____________R1_____R2_____R3_____ELS__
__X=1________Y_________________________
__X=2_______________Y__________________
__X=3______________________Y___________

Again, in example 2, it is clear what the user intends, since the conditionsare mutually exclusive. The translator cannot interpret the condition andtherefore can never assume that the rules are unique. This table can becorrected by inserting "N"s into the blank entries.

The problem can be avoided by mentally inserting Y/N into all blank entriesand then comparing all cases of all rules for being unique (example 3).

Example 3
__________________________________________
____________R1______R2______R3______ELS___
__X=1_______Y_______Y/N_____Y/N___________
__X=2_______Y/N_____Y_______Y/N___________
__X=3_______Y/N_____Y/N_____Y_____________
18.9.12   Conventions and Restrictions

  1. When using DECTRANS the following should be considered as reserved words:

    1. PARn where n is any 4 digit number
    2. RULn where n is any 4 digit number
    3. ACTn where n is any 4 digit number
    4. ELSn where n is any 4 digit number
    5. ENDn where n is any 4 digit number

  2. Maximum of 25 entries in Condition-Entry.

  3. Maximum of 25 entries in Action-Entry.

  4. Maximum of 25 rules including ELSE rule.

  5. Maximum of 18 and minimum of 3 columns in a rule.

  6. Maximum of 72 and minimum of 10 columns in condition and action stubs.

  7. ELS rule must be present or last rule will be treated as ELS rule.(A period must follow the last rule or ELS rule on the RHDR entry inorder to define the length of the field)

  8. Do not use periods to terminate table names, condition or action stubs.

18.9.13   Implementation Considerations

DECTRANS utilises three files

  1. Source Input file
  2. Output listing
  3. Source Output file
  1. The input file contains the tables to be translated to Cobol. The program accepts non associated tables or complete Cobol source programs with embedded tables. The processing of a table begins with a THDR entry and stops upon recognising a EODT entry. Tables must be in fixed format and the program outputs Cobol code in fixed format only.
  2. The output listing contains all table statements read from the input file, all of the statements generated from the tables and any diagnostic messages concerning the tables processed.
  3. The output file contains all of the COBOL source statements read from the input file and the COBOL statements generated from the tables. This output will be input to the COBOL compiler.
  4. Run Dectrans with the following parameters:
1: Input FileName (Mandatory), eg dec-run1.cbl2: Output FileName (Mandatory if not IFN.cpy) [IFN = Input FileName]3: Report FileName (Mandatory if not IFN.lst)

Minimum required params 1. The other two if omitted, will take their namefrom param 1 with (.cpy) and (.lst) as file extentions added to the name(without extention) used in parameter 1.

At completion, program will display a caution if any warning or errormessages have occurred.

18.9.14   Dectrans Processor Diagnostics
The following error messages are produced by Dectrans:
18.9.14.1   WARNING messages (translation will continue)
W01 SEQUENCE ERROR
The sequence numbers of the decision table are not in ascending order.Although this will not affectthe condition entries, the action entries may not be executed in the proper sequence.ACTION TAKEN: None, processing continues.
W02 LAST RULE WAS NOT ELSE RULE, TREATED AS SUCH
The last rule which must be "ELS" was not found. The last rule found will be considered the ELSE rule and any entries will be ignored.ACTION TAKEN: The entries in the last rule will not be processed. Source code will be generated.
W03 TABLE IS LIMITED BUT LOGICALLY INCOMPLETE
A limited table is being processed and one or more combinations of the conditions will cause the ELSE path to be taken. All of the possible rules are not present.ACTION TAKEN: None. This is only to indicate to the user that he has not covered all possibilities. The table is legal, he should be aware of the ELSE action that will result when one of the rules is not satisfied.
W04 ENIT ENTRY NOT FOUND, RHDR ENTRY TERMINATES INIT SECTION.
While processing the INIT section, a RHDR entry was encountered. Although the ENIT entry is missing, the processor can recover and finish processing the table.ACTION TAKEN: Table will be processed correctly.
W05 AUTO GOTO SUPPRESSED, GOTO MUST BE PROVIDED TO LEAVE EACH SET OF ACTIONS
The "GO TO ENDnnn" will not be generated. The user must specify a "GO TO" in the actions to leave the table. This table should not be PERFORMED since return to the exit may not be likely.ACTION TAKEN: None.
18.9.14.2   FATAL messages. (Processing continues with the next table)
E01 TABLE DELETED, ENTRY SIZE GREATER THAN RULE-ENTRY-LIMIT FOR RULE NUMBER n
As a rule header was being processed, an entry greater than 18 characters for an entry was found.ACTION TAKEN: Processing is discontinued for this table - no source code will be generated. The internal table limitations prevent the holding of all the information in the rule numbered n, therefore processing would be meaningless.
E02 STUB FIELD > 72 CHARACTERS
The first rule was not encountered within 72 positions from column 9. The maximum allowable length for the stub is 72 characters.ACTION TAKEN: Processing deleted - Correct table and retry.
E03 STUB FIELD < 10 CHARACTERS, TABLE DELETED
The first rule was encountered less than 10 positions from column 9. The minimum allowable length for the stub entry is 10.ACTION TAKEN: Source code will be generated but is probably in error. Correct and retry for correct results.
E04 NUMBER OF RULES ON RHDR EXCEEDS INDICATOR ON THDR

The number of rules specified on the THDR entry is less than the number of rules found on the RHDR entry.ACTION TAKEN: Processing is deleted until the next table is encountered.POSSIBLE CAUSES:

  1. Period missing following ELS rule or last rule specified.
  2. The field in the THDR entry indicating the number of rules does not begin in column 47.
E05 NUMBER OF ENTRIES < OR > THAN THDR INDICATORS, OR MISSING EODT

After processing the number of condition and action entries indicated on the THDR, the next record read was not an EODT.ACTION TAKEN: Table will not be processed.POSSIBLE CAUSES:

  1. EODT is missing.
  2. The fields in the THDR indicating the number of conditions andactions do not start in columns 41 and 44 respectively.
  3. The numbers of conditions and/or actions do not correspond to the THDR indicators.
E06 REDUNDANT OR CONTRADICTORY RULES HAVE BEEN FOUND, TABLE DELETED

Two or more rules are implicitly or implicitly equivalent and therefore the table must be incorrect. If the intention was for the two rules to be equal then the desired action should all be indicated under one of the rules and the others deleted.ACTION TAKEN: Table deleted. Correct and retry.POSSIBLE CAUSES:

  1. Two or more rules are identical.
  2. Due to the "ignoring" option, two or more rules could be interpreted as equal. (Check blank conditions.)
E07 SOURCE CODE GENERATION BYPASSED, CORRECT AND RETRY.
As stated, processing stops until next table. Also see any other message.
E08 MAXIMUM NUMBER OF CONDITIONS EXCEEDED

The number of conditions on the THDR is > than the limit allowed (25).ACTION TAKEN: Table deleted. Correct and retry.POSSIBLE CAUSE:

  1. If the number does not exceed the limit, check the starting column (41).
E09 MAXIMUM NUMBER OF ACTIONS EXCEEDED

The number of actions on the THDR is > than the limit allowed (25).ACTION TAKEN: Table deleted. Correct and retry.POSSIBLE CAUSE:

  1. If the number does not exceed the limit, check the starting column (44).
E10 MAXIMUM NUMBER OF RULES EXCEEDED

The number of rules on the THDR is > than the limit allowed (25).ACTION TAKEN: Table deleted. Correct and retry.POSSIBLE CAUSE:

  1. If the number does not exceed the limit, check the starting column (47).
E11 EODT ENCOUNTERED PRIOR TO END OF TABLE LENGTH, ENTRY MISSING

The total of the number of conditions and actions does not agree with the number of conditions and actions found.ACTION TAKEN: Table is not processed. Incorrect information. would cause a meaningless code generation.POSSIBLE CAUSES:

  1. The number of conditions and/or actions is not specified correctly.
  2. Each condition and action does not consist of the same number ofentries, i.e. If three entries are used to define the RHDR then eachcondition and action must be made up of three entries even though oneor two of the three might contain no information.
E12 TABLE DELETED, ENTRY SIZE < 3 CHARACTERS
See (E01)
E13 RULES nn AND nn ARE EQUAL, CHECK FOR REDUNDANT AND BLANK ENTRIES
See (E07)
E14 NO RULE HEADER (RHDR) FOUND, TABLE DELETED
The RHDR was not found immediately following the THDR or ENIT entries. The format of the table is unknown and cannot be processed.ACTION TAKEN: Table deleted. Correct and retry.
E15 CONTROLS NOT IN SEQUENCE, PROCESSING DISCONTINUED UNTIL NEXT THDR
A processor table control entry (THDR, INIT, EODT) was encountered while processing the INIT section and is logically incorrect. If both the ENIT and RHDR were missing and the EODT was detected, garbage will have been written to the source output file. If a THDR or INIT were read before the ENIT or RHDR, the results are unpredictable.ACTION TAKEN: Processing will be deleted until the next THDR is found. Correct and retry.
Table being processed is - TEST-003
   THDR TEST-003                        003001009

000010 TEST-003 SECTION.
000020 PAR0001.
   RHDR               001002003004005006007008ELS.
             C-1       Y  Y  Y  N  Y  N  N  N
             C-2       Y  Y  N  Y  N  N  Y  N
             C-3       Y  N  Y  Y  N  Y  N  N
         ACTION-1      X  X  X  X  X  X  X  X  X
   EODT
000030     IF C-1 GO TO RUL0001.
000040     IF C-2 GO TO RUL0002.
000050     IF C-3 GO TO ACT0006 ELSE GO TO ACT0008.
000060 RUL0002.
000070     IF C-3 GO TO ACT0004 ELSE GO TO ACT0007.
000080 RUL0001.
000090     IF C-2 GO TO RUL0003.
000100     IF C-3 GO TO ACT0003 ELSE GO TO ACT0005.
000110 RUL0003.
000120     IF C-3 GO TO ACT0001 ELSE GO TO ACT0002.
000130 ELS0001.
000140      ACTION-1.
000150      GO TO END0001.
000160 ACT0001.
000170      ACTION-1.
000180      GO TO END0001.
000190 ACT0002.
000200      ACTION-1.
000210      GO TO END0001.
000220 ACT0003.
000230      ACTION-1.
000240      GO TO END0001.
000250 ACT0004.
000260      ACTION-1.
000270      GO TO END0001.
000280 ACT0005.
000290      ACTION-1.
000300      GO TO END0001.
000310 ACT0006.
000320      ACTION-1.
000330      GO TO END0001.
000340 ACT0007.
000350      ACTION-1.
000360      GO TO END0001.
000370 ACT0008.
000380      ACTION-1.
000390      GO TO END0001.
000400 END0001.
000410     EXIT.
Table being processed is - TEST-C
   THDR TEST-C                          003004006  G

* W05 AUTO GOTO SUPPRESSED, GOTO MUST BE PROVIDED TO LEAVE EACH SET OF ACTIONS
000420 TEST-C SECTION.
000430 PAR0002.
   RHDR                       001 002 003 004 005 006 .
* W02 LAST RULE WAS NOT ELSE RULE, TREATED AS SUCH
         CONDITION-1           Y   Y   Y   N   N
         CONDITION-2           Y   Y   N   Y   N
         CONDITION-3           Y   N
         ACTION-1              X   X   X
         ACTION-2                          X   X
         NEVER-HAPPEN                              X
         GO TO PROCEDURE-NAME  X   X   X   X   X   X
   EODT
000440     IF CONDITION-1 GO TO RUL0005.
000450     IF CONDITION-2 GO TO ACT0013 ELSE GO TO ACT0014.
000460 RUL0005.
000470     IF CONDITION-2 GO TO RUL0007 ELSE GO TO ACT0012.
000480 RUL0007.
000490     IF CONDITION-3 GO TO ACT0010 ELSE GO TO ACT0011.
000500 ELS0002.
000510      NEVER-HAPPEN.
000520      GO TO PROCEDURE-NAME.
000530 ACT0010.
000540      ACTION-1.
000550      GO TO PROCEDURE-NAME.
000560 ACT0011.
000570      ACTION-1.
000580      GO TO PROCEDURE-NAME.
000590 ACT0012.
000600      ACTION-1.
000610      GO TO PROCEDURE-NAME.
000620 ACT0013.
000630      ACTION-2.
000640      GO TO PROCEDURE-NAME.
000650 ACT0014.
000660      ACTION-2.
000670      GO TO PROCEDURE-NAME.
000680 END0002.
000690     EXIT.
Table being processed is - CREDIT-APPROVAL
   THDR CREDIT-APPROVAL                 003002004

000700 CREDIT-APPROVAL SECTION.
000710 PAR0003.
   RHDR                                     001002003ELS.
         CREDIT-LIMIT IS GREATER THAN COST   Y  N  N
         SALARY ID EQUAL TO ACCEPTABLE          Y  N
         SPECIAL-CLEARANCE = 1                     Y
         PERFORM APPROVAL                    X  X  X
         PERFORM ORDER-REJECT                         X
   EODT  LIMITED ENTRY TEST 1
000720     IF CREDIT-LIMIT IS GREATER THAN COST GO TO ACT0016.
000730     IF SALARY ID EQUAL TO ACCEPTABLE GO TO ACT0017.
000740     IF SPECIAL-CLEARANCE = 1 GO TO ACT0018 ELSE GO TO
000750         ELS0003.
000760 ELS0003.
000770      PERFORM ORDER-REJECT.
000780      GO TO END0003.
000790 ACT0016.
000800      PERFORM APPROVAL.
000810      GO TO END0003.
000820 ACT0017.
000830      PERFORM APPROVAL.
000840      GO TO END0003.
000850 ACT0018.
000860      PERFORM APPROVAL.
000870      GO TO END0003.
000880 END0003.
000890     EXIT.
Table being processed is - TEST-DECTRANS-EXTENDED
   THDR TEST-DECTRANS-EXTENDED          008004004

000900 TEST-DECTRANS-EXTENDED SECTION.
000910 PAR0004.
   RHDR                         001       002       003       ELS   .
        FEMALE                    Y         Y         Y
        AGE GREATER THAN         21        25        18
        AGE <                    35        38        23
        HAIR IS EQUAL TO        BLONDE    RED       BLACK
        EYES =                  BLUE      RED       BLACK
        WEIGHT >                100       100       100
        WEIGHT <                140       140       130
        I/Q =                   110       115       100
        GO TO WINNER             X
        GO TO SO-SO                                  X
        GO TO POSSIBILITY                  X
        GO TO DOG                                             X
   EODT TEST FOR NO GO TO IN THDR
000920     IF FEMALE NEXT SENTENCE ELSE GO TO ELS0004.
000930     IF WEIGHT > 100 NEXT SENTENCE ELSE GO TO ELS0004.
000940     IF AGE GREATER THAN 21 NEXT SENTENCE ELSE GO TO RUL0012.
000950     IF AGE < 35 NEXT SENTENCE ELSE GO TO RUL0012.
000960     IF HAIR IS EQUAL TO BLONDE NEXT SENTENCE ELSE GO
000970         TO RUL0012.
000980     IF EYES = BLUE NEXT SENTENCE ELSE GO TO RUL0012.
000990     IF WEIGHT < 140 NEXT SENTENCE ELSE GO TO RUL0012.
001000     IF I/Q = 110 GO TO ACT0020.
001010 RUL0012.
001020     IF AGE GREATER THAN 25 NEXT SENTENCE ELSE GO TO RUL0013.
001030     IF AGE < 38 NEXT SENTENCE ELSE GO TO RUL0013.
001040     IF HAIR IS EQUAL TO RED NEXT SENTENCE ELSE GO TO
001050         RUL0013.
001060     IF EYES = RED NEXT SENTENCE ELSE GO TO RUL0013.
001070     IF WEIGHT < 140 NEXT SENTENCE ELSE GO TO RUL0013.
001080     IF I/Q = 115 GO TO ACT0021.
001090 RUL0013.
001100     IF AGE GREATER THAN 18 NEXT SENTENCE ELSE GO TO ELS0004.
001110     IF AGE < 23 NEXT SENTENCE ELSE GO TO ELS0004.
001120     IF HAIR IS EQUAL TO BLACK NEXT SENTENCE ELSE GO TO
001130         ELS0004.
001140     IF EYES = BLACK NEXT SENTENCE ELSE GO TO ELS0004.
001150     IF WEIGHT < 130 NEXT SENTENCE ELSE GO TO ELS0004.
001160     IF I/Q = 100 GO TO ACT0022.
001170 ELS0004.
001180      GO TO DOG.
001190      GO TO END0004.
001200 ACT0020.
001210      GO TO WINNER.
001220      GO TO END0004.
001230 ACT0021.
001240      GO TO POSSIBILITY.
001250      GO TO END0004.
001260 ACT0022.
001270      GO TO SO-SO.
001280      GO TO END0004.
001290 END0004.
001300     EXIT.
Table being processed is - TEST-B
   THDR TEST-B                          003003008

001310 TEST-B SECTION.
001320 PAR0005.
   RHDR                     1   2   3   4   5   6   7   E   .
        CONDITION-1         Y   Y   Y   N   N   N   N
        CONDITION-2         Y   N   N   N   N   Y   Y
        CONDITION-3             Y   N   N   Y   N   Y
        ACTION-1            X   X   X
        ACTION-2                        X   X   X   X
        NEVER-HAPPEN                                    X
   EODT
001330     IF CONDITION-1 GO TO RUL0015.
001340     IF CONDITION-2 GO TO RUL0016.
001350     IF CONDITION-3 GO TO ACT0028 ELSE GO TO ACT0027.
001360 RUL0016.
001370     IF CONDITION-3 GO TO ACT0030 ELSE GO TO ACT0029.
001380 RUL0015.
001390     IF CONDITION-2 GO TO ACT0024.
001400     IF CONDITION-3 GO TO ACT0025 ELSE GO TO ACT0026.
001410 ELS0005.
001420      NEVER-HAPPEN.
001430      GO TO END0005.
001440 ACT0024.
001450      ACTION-1.
001460      GO TO END0005.
001470 ACT0025.
001480      ACTION-1.
001490      GO TO END0005.
001500 ACT0026.
001510      ACTION-1.
001520      GO TO END0005.
001530 ACT0027.
001540      ACTION-2.
001550      GO TO END0005.
001560 ACT0028.
001570      ACTION-2.
001580      GO TO END0005.
001590 ACT0029.
001600      ACTION-2.
001610      GO TO END0005.
001620 ACT0030.
001630      ACTION-2.
001640      GO TO END0005.
001650 END0005.
001660     EXIT.
Table being processed is - TEST-012
   THDR TEST-012                        003001009

001670 TEST-012 SECTION.
001680 PAR0006.
   RHDR               001002003004005006007008ELS.
           C-1         N  N  N  N  Y  Y  Y  Y
           C-2         N  N  Y  N  Y  N  N  Y
           C-3         N  Y  N  Y  N  Y  N  Y
        ACTION-1       X  X  X  X  X  X  X  X  X
   EODT  E06 E07 TEST FAIL
**E06 REDUNDANT OR CONTRADICTORY RULES HAVE BEEN FOUND, TABLE DELETED
**E07 SOURCE CODE GENERATION BYPASSED, CORRECT AND RETRY
Table being processed is - TEST-D
   THDR TEST-D                          003001009

001690 TEST-D SECTION.
001700 PAR0007.
   INIT
001710     MOVE     SPACES TO BETA-NAMES.
001720     PERFORM  AA000-RESTART.
001730     SET      MASTER-CNT UP BY 1.
001740     INITIALIZE TOTALISERS.
001750*
001760*  ABOVE WILL BE FIRST SET OF CODE
001770*
   ENIT
   RHDR               001002003004005006007008ELS.
           C-1         N  N  N  N  Y  Y  Y  Y
           C-2         N  N  Y  Y  N  N  Y  Y
           C-3         N  Y  N  Y  N  Y  N  Y
        ACTION-1       X  X  X  X  X  X  X  X  X
   EODT  TEST FOR INIT
001780     IF C-1 GO TO RUL0023.
001790     IF C-2 GO TO RUL0022.
001800     IF C-3 GO TO ACT0034 ELSE GO TO ACT0033.
001810 RUL0022.
001820     IF C-3 GO TO ACT0036 ELSE GO TO ACT0035.
001830 RUL0023.
001840     IF C-2 GO TO RUL0024.
001850     IF C-3 GO TO ACT0038 ELSE GO TO ACT0037.
001860 RUL0024.
001870     IF C-3 GO TO ACT0040 ELSE GO TO ACT0039.
001880 ELS0006.
001890      ACTION-1.
001900      GO TO END0006.
001910 ACT0033.
001920      ACTION-1.
001930      GO TO END0006.
001940 ACT0034.
001950      ACTION-1.
001960      GO TO END0006.
001970 ACT0035.
001980      ACTION-1.
001990      GO TO END0006.
002000 ACT0036.
002010      ACTION-1.
002020      GO TO END0006.
002030 ACT0037.
002040      ACTION-1.
002050      GO TO END0006.
002060 ACT0038.
002070      ACTION-1.
002080      GO TO END0006.
002090 ACT0039.
002100      ACTION-1.
002110      GO TO END0006.
002120 ACT0040.
002130      ACTION-1.
002140      GO TO END0006.
002150 END0006.
002160     EXIT.

18.10   Code Generator Specifications

As with a metaprogramming code generator, much of the generated childrenprograms will contain parts of the parent generator. The main focus of themetaprogram will be the parent level and the spawned child levels. Theparent and spawned child levels can be the same programming language ortwo separate programming languages.

The spawned children levels are are independent applications from theparent level (metaprogram) after being generated fromdata-based/data-driven stored information. Keeping the separation of thedata and conventions from the actual generated programs are the main focusof data-based metaprogramming systems.

The generated/spawned children levels should be treated as code independentand as such with no reference, beyond the metadata, to what kind of code thechildren are.

19   Metadata Specifications

With Metadata, the system can be self-referential to itself and rely on itsown stored data to provide either the model and view of model-view-control(MVC) concept or the actual credentials of a metaprogram. One example wouldbe user preferences. With this example, the user desires to have datesdisplayed as month-date-year instead of the default year-month-date. Thiswould be stored into a database to be reused again by that individual user.Another example would involve user privileges set by administration ormanagement. This would be "metadata" by definition

19.1   Conceptual storage

Meta data is stored within a database that can be used throughout thesystem setup to allow the user to have a more productive and efficientexperience. The following is a list of possible storage capabilities:

  • Schema Example 1

    • content

    • DDR ( data driven registry)
      • content structures
      • structure configurations
      • default user permissions
      • default user preferences
  • Schema Example 2
    • content
    • content structures
    • structure configurations
    • default user permissions
    • default user preferences

19.2   Data driven registry

A data-driven registry would store within separated categores themeta data, such as the actual content structures, structureconfigurations, default user permissions, default user preferences, andmuch more. This would allow a centralized place for administrators to accessand control the generated application.

20   Delivery Specifications

There are multiple methods one could use for a delivery system. A fewmethods that are well known are listed in this specification for reference.Each are keeping to a simplistic separation of information and logic. Thisshould be referred to, here on, as the Presentation layer, the BusinessLayer, and the Data-based layer. Or collectively, as PBD.

20.1   Web Application Specification

Keeping with PBD, the MDBMPS aims to keep html as thePresentation layer, and the Business layer be the any programming languagedesired. The Data-Based layer would be in a desired database along with thecontent and metadata. This allows the MDBMPS to fullcapabilities to use metadata to generate the desired application.

20.1.1   Web Example A
  • Server-side

    • Data-Based: MySQL Database (with metadata and content stored)

    • Presentation: HTML 4.01

    • Business:
      • Top Layer: PHP
      • Bottom Layer: SQL Stored Procedures and SQL Functions
  • Client-side
    • Structure: HTML 4.01
    • Display: CSS
    • Behavior: Unobtrusive JavaScript
20.1.2   Web Example B
  • Server-side

    • Data-Based: PostgreSQL Database (with metadata and content stored)

    • Presentation: XHTML 1.1

    • Business:
      • Top Layer: Python
      • Bottom Layer: SQL Stored Procedures and SQL Functions
  • Client-side
    • Structure: XHTML 1.1
    • Display: CSS
    • Behavior: Unobtrusive JavaScript

20.2   Console Application Specification

The console (command-line interface [CLI]) application is a lightweightalternative to GUI and web and is operatable on a user's own terminalsession.

20.2.1   CLI Example A

  • Data-Based: MySQL Database (with metadata and content stored)

  • Presentation: Terminfo/Termcap

  • Business:
    • Top Layer: Python
    • Bottom Layer: SQL Stored Procedures and SQL Functions
20.2.2   CLI Example B

  • Data-Based: PostgreSQL Database (with metadata and content stored)

  • Presentation: ncurses

  • Business:
    • Top Layer: C/C++
    • Bottom Layer: SQL Stored Procedures and SQL Functions

20.3   GUI Application Specification

The Graphical User Interface (GUI) application would allow an applicationto operate in a user's own graphical session.

20.3.1   GUI Example A

  • Data-Based: MySQL Database (with metadata and content stored)

  • Presentation: GTK++

  • Business:
    • Top Layer: Python
    • Bottom Layer: SQL Stored Procedures and SQL Functions
20.3.2   GUI Example B

  • Data-Based: XML (with metadata and content stored)

  • Presentation: QT

  • Business:
    • Top Layer: XPCOM
    • Bottom Layer: C/C++

21   Notes

Items that need addressed:

21.1   Notes of Seth Keiper

  • Completed: Security
    • Completed: Cleaning I/O
      • Completed: User input
      • Completed: Output

    • Completed: Restrict filesystem access

  • Practice of genericism
    • works with any data
    • works with any user input
  • Accessibility in all forms
    • All input conforms to accessibility standards
    • All output conforms to accessibility standards
  • Completed: All code needs changed to pseudocode
    • Completed: is easier to work with
    • Completed: readable by those without programming experience
Pseudocode
  1. Read from file called TemplateFile
  2. At the end of file set the variable TemplateFileEOF to TRUE
  3. Stop reading file
Real World (COBOL)
READ Template-File
  AT END SET Template-File-EOF TO TRUE
END-READ

21.2   Notes of Joseph Frantz

2009-01-25

What needs done:

  • Better Introduction (more complete)
  • Chapter 10 Reporting
  • Chapter 12 Interfacing
  • Chapter 13 Communication
  • Chapter 15 Problem Solving

These are for the pre-spec. Obviously the planning still needs to be done.

22   Changes

  • In Progress

23   Summary

This is a conclusion of the MDBMPS. Combinding metadata withmetaprogramming can produce highly efficient code and allow a veryproductive environment.


石头
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
metadata-extractor依赖引入
01-17
metadata-extractor依赖引入
Hive学习笔记——metadata
weixin_30672295的博客
09-03 251
可以在hive的jdbc接口中使用getMetaData方法来获取hive表的相关元信息 statement = connection.createStatement(); DatabaseMetaData meta = connection.getMetaData(); 参考 https://blog.csdn.net/u010368839/article/de...
多数据源配置时出现的一些问题
YHJ
06-13 4256
自己写了一个SpringBoot结构的小项目，练习一下多数据源的配置。因为是从之前的项目中拷贝的代码，不清楚其实现原理，因此出现了一些错误。 一、Premature end of file. 控制台输出如下： Caused by: org.springframework.core.NestedIOException: Failed to parse mapping resource: 'file [E:\githubproject\rong_system\target\classes\mapping\
eclispe spring 配置报错：Error occured processing XML 'org.springframework.beans.MutablePropertyValues.ge
zx1323的博客
07-22 5631
Error occured processing XML 'org.springframework.beans.MutablePropertyValues.get(Ljava/lang/String;)Ljava/lang/Object;'. See Error Log for more details 转自：http://blog.csdn.net/niu_hao/article/
【Error: No available modular metadata for modular package】
m0_60983050的博客
07-13 3193
Error: No available modular metadata for modular package
Error: No available modular metadata for modular package
DeathlessDogface的博客
10-26 3238
centos 8 dnf 本地源处理module repo问题下载mariadb rpm包安装modular metadata生成工具创建repodata 和 modular metadata配置本地yum源安装mariadb类似问题 下载mariadb rpm包 配置有网络源的主机 dnf install mariadb-server --downloadonly --downloaddir=/root/pkgs/mariadb 安装modular metadata生成工具 dnf install
metadata.db
07-29
metadata.db 电子书库资源文件
metadata-extractor-2.16.0.jar
06-19
图片文件的文件元数据读取工具
深入理解 TypeScript Reflect Metadata
10-15
主要介绍了深入理解 TypeScript Reflect Metadata，文中通过示例代码介绍的非常详细，对大家的学习或者工作具有一定的参考学习价值，需要的朋友们下面随着小编来一起学习学习吧
metadata公共元数据项目源码
04-20
metadata公共元数据项目源码 https://github.com/songxingkeji/metadata
“Property or method “***“ is not defined on the instance but referenced during render.”报错的原因及解决方案
酷爱前端的小陈博客
04-29 18万+
报错问题： 在使用vue-cli运行项目的过程中，在VScode中不报错，但在浏览器调试工具中发出 [Vue warn]: Property or method "pic" is not defined on the instance but referenced during render. Make sure that this property is reactive, either in the data option, or for class-based components, by in.
如何解决vue项目报错Property or method “xxx“ is not defined on the instance but referenced during render
sunshine唯一的博客
07-25 1万+
如何解决vue项目报错Property or method "xxx" is not defined on the instance but referenced during render.
Springboot系列：@SpringBootApplication注解
youzizhu1126的博客
10-17 2068
在使用 Springboot 框架进行开发的时候，通常我们会在 main 函数上添加 @SpringBootApplication 注解，今天为大家解析一下 @SpringBootApplication，如有不正之处，欢迎批评指正。 @SpringBootApplication @SpringBootApplication源码如下： @Target({ElementType.TYPE
4: 模块化应用程序开发 Modular Application Development Using Prism Library 5.0 for WPF （英汉对照版）...
diegaifei1724的博客
04-30 368
A modular application is an application that is divided into a set of loosely coupled functional units (named modules) that can be integrated into a larger application. A client module encaps...
MySQL innodb parameters
yumushui的专栏
07-14 947
本文转载自：http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html
Multiple annotations found at this line: 解决办法
热门推荐
叶落无痕，枫过有情……
10-11 23万+
Multiple annotations found at this line: - schema_reference.4: Failed to read schema document 'http://www.springframework.org/schema/beans/ spring-beans-3.1.xsd', because 1) could not find the docum
Property or method xxxx is not defined on the instance but referenced during render
han8507900的博客
06-11 7073
做vue项目已经两个月了。从0开始学，现在想把之前遇到的问题一点点写下来以便他人和自己再次遇到。 第一个问题就是这个了： Property or method “xxxx” is not defined on the instance but referenced during render. 我遇到的问题是在form表单里定义了一个v-bind。 例如 :rules = “rule” 双引号里面应该要在methods写一个方法(函数)。我没写。就报这个错误了。 解决方法要么加个方法，要么把这个v
"Automation 服务器不能创建对象"　的解决方案
竹君子之家
11-11 2089
访问某些网站的时候，如果出现"Automation 服务器不能创建对象"的错误提示框，表明网页内的对象需要scrrun.dll的支持，可以尝试下面的方法：开始--运行--regsvr32 scrrun.dll不用重新安装操作系统和vs.net 有些脚本需要微软的 MSXML 控件才能进入。当使用 IE 5 以上版本的缺省安全模式时，会提示是否接受 MSXML 控件，如果接受，MSXML 将
mect: multi-metadata embedding based cross-transformer for chinese named entity recognition
最新发布
04-12
Q：「b'mect: multi-metadata embedding based cross-transformer for chinese named entity recognition'」是什么？ A：「b'mect: multi-metadata embedding based cross-transformer for chinese named entity recognition'」是一篇关于中文命名实体识别的论文，提出了一种新的算法来提高命名实体识别的准确率和效率。论文中提出的算法名为「BMECT」，利用多个元数据的嵌入和交叉变换的方式来提高模型的性能。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值