| content
| - %VOSWARNING%
%META:TOPICINFO{author="KingsleyIdehen" date="1144104148" format="1.1" version="1.1"}%
%VOSNAV%
---+ History of Virtuoso
Virtuoso is the offspring of a genealogy tree comprising Universal Data Access Middleware from <nop>OpenLink
Software and Kubl RDBMS (Relational Database Management System) Engine by Orri Erling.
%TOC%
---++Universal Data Access Middleware
In 1992 Kingsley Idehen departed Unisys (in the United Kingdom) to set up a consulting firm called
PAL Consulting Ltd. (which stood for Products, Applications, and Languages consulting). His initial
aim was to set up a consulting firm that specialized in data access middleware and associated integration
services; a natural progression since he possessed significant knowledge of all the major RDBMS engines
(at the time), and every first generation data access middleware product. Unfortunately, the competitive
realities of partnering with some early middleware vendors forced him change course and defend his vision
and passion for data access middleware. The net effect of this strategic change in direction was the
creation of <nop>OpenLink Software (by renaming PAL Consulting after its emerging collection of ODBC Drivers).
In the early 90's, <nop>OpenLink Software earned a stellar reputation in the data access middleware
market by delivering secure and high-performance ODBC Drivers for a range of industry leading databases.
These drivers were popularly known as the "<nop>OpenLink High-Performance" ODBC Drivers, and were the
only drivers at the time capable of dispelling the "ODBC means Poor Performance" myth. The company is
also responsible for the very first port of ODBC on Linux, and other UNIX variants (called: UDBC). It
also developed the very first ODBC Driver for Postgres, and has pursued the purity of platform independent
data access from its very inception. It combined its initial UDBC effort with a development effort with
Ke Jin to create what is known today as [[http://iodbc.org/][iODBC]], the earliest known Open Source project
associated with DBMS vendor independent data access.
By 1997 the company's initial portfolio of platform independent ODBC specific data access middleware
had grown to include drivers for JDBC, and OLE-DB; all available in a variety of architectural formats
(Single-Tier and Multi-Tier).
---++Kubl
In early 1994, Orri Erling was tasked with overseeing the porting of an existing DBMS known as VIA/DRE to
ten different Unix platforms and all versions of Windows then-extant. The portation effort was successful,
but the performance of the ports turned out to be inadequate for the network management application for
which they had been commissioned. While managing this effort, Orri was also developing a portable
distributed computing platform for C/C++ that provided [[AsyncRPC][asynchronous RPC]], [[Streaming][streaming]],
[[Serialization][serialization of C and C++ data structures]], [[PersistentObjects][persistent]] and
[[DistributedObjects][distributed objects]], etc.
The net effect of the VIA/DRE experience, combined with the completion of the Distributed Computing Platform,
spurred Orri toward a new goal. He embarked on a scientific pursuit, pushing the boundaries of SQL, aiming
to achieve the lofty goal of the fastest [[http://dbpedia.org/resource/Online_transaction_processing][OLTP]]-oriented
DBMS engine that could possibly be written. This effort bore fruit, in Kubl.
While working on Kubl, Orri was also engaged with Infosto Group, Finnish publisher of <i>Keltainen Porssi,</i>
a "Consumer to Consumer" ("[[http://dbpedia.org/resource/Consumer-to-consumer][C2C]]") national marketplace journal.
He was tasked with technical oversight of Infosto's participation in the [[http://dbpedia.org/resource/United_Nations_Conference_on_Trade_and_Development][United Nations]] [[http://www.tradepoint.org/][Trade Point]] initiative.
Trade Point's aim was to help network small businesses worldwide via an online marketplace on the then-nascent Web.
This effort became the first full-blown application of the Kubl DBMS Engine, which by then was a client-server RDBMS
core with basic SQL, transactions, and a stored procedure language. This project was soon followed by a web version of
<i>Keltainen Porssi,</i> which quickly gained 500.000 registered users out of a demographic of only 5 million. As
Finland of the mid 90s still had relatively little household Internet use, the market penetration of this effort was remarkable.
---++Virtuoso Project Commencement
In early 1998 Kingsley Idehen, Founder & CEO of <nop>OpenLink Software, felt the time was right to implement
the next stage of <nop>OpenLink's long-standing vision to provide complete separation of Application Logic,
Data Access, and Data Storage via a "Virtual Database Engine". The company had already successfully developed
and deployed high-performance data access drivers for ODBC, OLEDB, and JDBC that served traditional "Client-Server"
applications, but it now had its sights on addressing the same problems in the new architectural context presented
by the burgeoning World Wide Web. The main issue it had to address was the traditional "Build vs. Buy" approach
to building the Virtual Database Engine that it sought. In attempting to resolve the aforementioned dilemma
Kingsley decided to scour the internet for the existence of an ODBC CLI compliant Linux based SQL RDBMS Engine.
An engine that could be potentially morphed into a combined Middleware and DBMS product.
The extensive search and evaluation process lead Kingsley to Kubl, and ultimately Orri. Kingsley pitched the
<nop>OpenLink Virtual Database vision to Orri and within 4 weeks of their meeting <nop>OpenLink Software
acquired Kubl and the Virtuoso project was born.
---++Virtual Database Engine Development
As a vendor of high-performance ODBC Drivers covering all major backend DBMS engines, <nop>OpenLink Software was
uniquely positioned to build a Virtual DBMS Engine that truly abstracted data access across heterogeneous databases
without the expected challenges of obtaining functional drivers, or the political hang-ups of a traditional DBMS
vendor. In addition, the compatibility of technical prowess, vision and passion made the assimilation of Virtuoso
much easier than would be expected in conventional company and personnel mergers. Thus, in the fall of 1998 the
first release of the Virtuoso VDBMS Engine was unveiled. You now had a single high-performance ODBC, JDBC, or
OLE DB based access point for transparently connecting compliant Desktop Productivity Tools and Application
Development Environments (Traditional and Next Generation Web Oriented) to any ODBC- or JDBC-accessible backend
database.
IObox (one of the earliest hosted internet services) was one of the web applications/services built using Virtuoso.
It served up to half a million registered users from a single Sun SPARC 300MHz server with 32G of disk and catered
for about 3000 concurrent sessions. With the addition of more application logic in PHP, the data center became
larger and the end user count reached 1 Million. Telefonica eventually acquired IObox for about $300M in 2001.
<nop>OpenLink was a contracted developer and software vendor for the IObox project.
---++Transition to Virtual DBMS for SQL and XML
In early 1999, XML started to emerge as the preferred candidate for standardizing data representation for data
access, protocol definition, and data modeling. <nop>OpenLink responded to the impending XML market inflection
by adding XML support to Virtuoso. This included the following database engine hosted capabilities; Validating
XML Parser (supporting XML Schema), XSLT processor, SQL-XML transformation, XQuery, XPath, and a native XML
datatype (long before this functionality became roadmap and implementation items across Oracle, Microsoft SQL
Server, DB2, or any other major DBMS engine). Initial SQL-XML transformation functionality took the form of
a [[VirtXmlSchemaMapping][XML to SQL Schema Mapping]] system called [[VirtXmlViews][XML Views]] that also
preceded Microsoft's schema mapping system. It turned relations into XML and XPATH into SQL, allowing querying
of relational model data ([[RelationalTuple][tuples]]) using hierarchical data model navigation.
Virtuoso also supported shredded storage of XML for a short while but this was later dropped in favor of
document-style XML storage and [[VirtXmlTextIndex][XML specific Free Text indexing]]. <nop>OpenLink never fully
subscribed to the idea of XML being a storage format upon which transactional applications would be built. It
believed, as time has demonstrated, that XML would be the format of choice for data exposure and exchange. Thus,
its XML technology exploits remain centered on the position that XML (via "Universal Views") is the preferred
solution for unshackling "Logical Data Representation" from the confines of database model and engine specificity.
By exposing native and third party SQL Data as XML based views hosted by the Virtuoso Database engine, it was
clear to <nop>OpenLink that it had to surmount the metaphor level impedance challenge associated with exposing
DBMS engine resident XML views to XML developers and users that sought to interact with physical documents that
were URL accessible. In response to this reality, Virtuoso's SQL-XML functionality was extended to include
generation of HTTP accessible XML documents "on the fly". The XML documents (Valid or Well-Formed) that are
conditionally sensitive to changes in source databases. Naturally, this feature raised some security concerns
that ultimately led to the addition of WebDAV and Virtuoso specific ACL security to Virtuoso's in-built Web Server.
In addition to its sophisticated XML functionality, Virtuoso also possesses its own dynamic web page generation
language called VSP (Virtuoso Server Pages). The language enables the development of dynamic web pages using
Virtuoso's SQL procedure language interspersed in HTML (as is the case with PHP, ASP, JSP, and others). It also
includes a declarative XML based dynamic web page generation language called VSPX, which is conceptually similar
to ASP.NET. A key advantage that VSP/VSPX pages bring to bear is the fact that SQL data access occurs in-process,
as long as the tables being accessed are stored in Virtuoso.
---++Virtuoso the Universal Server
By 2000-2001, [[SOAP]] (Simple Object Access Protocol) started to gain prominence as the foundation protocol of
what is known today as the WS-* protocol stack. This protocol stack catalyzed the nascent Web Services paradigm
(including Service Oriented Architecture and ESB derivative paradigms). As Virtuoso already possessed DBMS hosted
(in-built) HTTP/WebDAV support (its web server functionality), and a very powerful SQL Stored Procedures language,
it was a very natural progression for the product to expose functionality that would simplify comprehension and
exploitation of the burgeoning Service Oriented Architecture (SOA) paradigm. The following features resulted:
* Execution of native or third party SQL Stored Procedures over HTTP
* Execution of native or third party SQL Stored Procedures via SOAP (by publishing SQL Stored Procedures as
WSDL and SOAP compliant Web Services)
* Proxy Generation for third party Web Services
* Exposure of saved SQL, SQL-XML, XQuery, and XPath queries as Web Services
Orri's Lisp and AI background provided Virtuoso with a rich programming language endowed with runtime data-typing
and self-hosted compilation making the incorporation of SOAP within an already powerful technology cocktail a
triviality. In addition, the Database / Data-Access Middleware essence of Virtuoso synergistically extended the
SOAP functionality to include Stored Procedures hosted in any third party ODBC accessible DBMS engine. Over time
Virtuoso has continued to implement relevant portions of the WS-* protocol stack as and when they are published.
It had one of the very first implementations of WS-Security and till this very day the only DBMS hosted
implementation of the Business Process Execution Language for Web Services (BPEL nee. BPEL4WS).
By the summer of 2001, Virtuoso underwent a major engine-rewrite that re-galvanized an erstwhile dormant aspect
of Virtuoso, namely its Object-Relational DBMS functionality. Outputs from this major development effort include:
* DBMS hosting of the Java, Microsoft .Net CLR, and Mono runtime environments
* User Defined Type (UDT) support with implementation in SQL, Java or .net
* Use of UDTs for abstracting web services, generating a UDT from a WSDL.
* SQL Stored Procedures extensibility via code associated with hosted runtimes
* Dynamic Language & Web page hosting for PHP, ASP.NET, JSP, Python, Perl, and Ruby (a recent addition)
* Procedure Views (Table Valued Functions in SQL Server and Table Functions in Oracle)
* Improved Cost based Distributed Query Optimizer for handling heterogeneous SQL joins
* Bi-Directional Transaction Replication
* XA-based 2-Phase Commit for Distributed Transactions
The combination of SQL-ORDBMS, HTTP/WebDAV, Web Services Platform for SOA, and other functionality realms covered
by Virtuoso, led to fundamental incompatibility between the product moniker "Virtual Database" and the actual
product feature set. Re-branding Virtuoso as a ?Universal Server? ultimately alleviated the aforementioned
product-branding challenge; a single server product offering that implements a plethora of industry-standard protocols.
From a traditional marketing and positioning point of view Virtuoso in its Universal Server is a complete renegade
by virtue of the fact that it straddles a number of functionality realms concurrently. At the same time, a closer
look at the product architecture and its evolution , unveils deep understanding and anticipation of the platform
requirements that will ultimately define next generation Intranet/Internet/Extranet (SOA, Web 2.0, Semantic Web/Data
Web, and beyond) solutions. This is all the more so as these solutions will demand loose coupling of application
logic, separation of data access & data storage, with an inherent need for collaboration driven integration that
comes naturally to Virtuoso.
---++Virtuoso and the Data Space Frontier
It is no secret that Virtuoso is a product way ahead of its time. <nop>OpenLink, as a result of finding the newest
member of its product portfolio in the enigmatic status of "Best Kept Secret", decided to address the emerging
product comprehension and uptake lull to further test and enhance Virtuoso by embarking upon a number of internal
application development projects. The goal of this effort was dual pronged: build next generation applications that
showcased the inherent prowess of Virtuoso, and "dogfood" the resulting applications (which would benefit the company
anyhow) prior to release. The net effect of this decision was a gradual change in engineering focus. No more pursuit
of high-end database functionality aimed solely at winning TPC benchmarks (such as transparent shared nothing
clustering). Instead, the focus turned to building a suite of collaboration oriented solutions aimed at showcasing
not only the prowess of Virtuoso, but the potential of emerging paradigms such as Web 2.0 and the "Semantic/Data Web".
The applications that emerged from this application development effort include:
* Web Application Framework
* Weblog Publishing Platform
* RSS/Atom/RDF Feed Aggregator
* Photo Sharing system
* Discussion Server
* Wiki Engine
* BPEL Process Manager (application layer above the in-built BPEL core)
* Unified Storage (that includes automatic metadata extraction and resource classification using RDFS and OWL)
* Social Networking Framework
As was the case when Virtuoso transitioned from a Virtual Database engine to a Universal Server that included
Virtual Database functionality, the suite of Virtuoso applications posed a number of marketing related product
branding challenges. Ultimately it was decided that the Universal Server would be packaged as a platform solution
(workbench of sorts) while the applications would be packaged as discrete parts of a collective DataSpace (since
all of the data and application logic that expressed the application behaviors resided in a Virtuoso Database).
---++Virtuoso Today
Having completed a slew of unreleased Virtuoso applications (as enumerated above), it was then decided in late 2005
that the time was right for finalizing engineering work that would unveil and revisit a deferred development effort
aimed at the "Semantic Web". There emergence of a definitive query language for RDF Data Management systems called
SPARQL, accompanied by a SOAP based protocol also called SPARQL, provided <nop>OpenLink with justification for
resuming its long stalled efforts on the RDF data storage front. Thus, from late 2005 to early 2006, RDF has been
a major development focus of the core Virtuoso engineering team. The end result of this effort is a full-blown
SPARQL to SQL translator (or rewriter) that successfully completes a majority (90 percentile) of tests in the DAWG
(RDF Data Access Workgroup) SPARQL test suite. The challenges posed by the Semantic Web vision compliment the
essence of Virtuoso's prowess and <nop>OpenLink's engineering skill set. After all, an RDF Triple store is
fundamentally a specialized database challenge that draws on deep database engineering and query optimization know-how.
A large part of Virtuoso was released in open-source form (dual-license mode similar to [[http://dbpedia.org/resource/MySQL][MySQL]]) in April 2006 with
the following strategic goals in mind:
* Developing and Fostering community around Virtuoso
* Making a significant contribution that potentially accelerates comprehension and exploitation of the "Semantic
Web"/"Data Web" vision
* Providing technology in a form palatable to early adopters and the extensive RDF research community
* Quicker product release cycles
* Bury the "Best-Kept Secret" and generally enigmatic status of Virtuoso
%VOSCOPY%
CategoryMarketing CategoryCorporateComms CategoryWebSite CategoryVirtuoso CategoryOpenSource
|
![[RDF Data]](/fct/images/sw-rdf-blue.png)
![[cxml]](/fct/images/cxml_doc.png)
![[csv]](/fct/images/csv_doc.png)
![[text]](/fct/images/ntriples_doc.png)
![[turtle]](/fct/images/n3turtle_doc.png)
![[ld+json]](/fct/images/jsonld_doc.png)
![[rdf+json]](/fct/images/json_doc.png)
![[rdf+xml]](/fct/images/xml_doc.png)
![[atom+xml]](/fct/images/atom_doc.png)
![[html]](/fct/images/html_doc.png)