. "2017-06-13T05:42:22.527845"^^ . . . "2017-06-13T05:42:22.527845"^^ . . . . . "---+Configuring Virtuoso for Scale\n\n%TOC%\n\n---++ Background\n\nOpenLink Virtuoso is an engine of many features. It incorporates a database engine, Web server, RDF Quad Store, \nSPARQL processor, and the OpenLink Data Spaces (ODS) suite of applications for bookmarks, briefcase, wiki, \nwebmail, etc. \n\nNaturally, each of these features has its dependencies and consequences for resource usage, be that in-memory \nor on disk.\n\nHere we present an overview of 3 possible configurations for Virtuoso.\n\nFor comparison, we use Virtuoso Open-Source Edition (VOS) compiled on\nDebian GNU/Linux (\"Testing\" distribution).\n\n\n---+++ Structure of a Virtuoso Installation\n\nA typical Virtuoso installation comprises the following files/areas:\n\n| *Directory* | *Description* | *Files* | *Size* |\n| /usr/bin/ | Vital operating binaries for server and command-line client | virtuoso-t %BR% isql-v %BR% isql-vw | 9.2 MB |\n| /usr/share/virtuoso/vad/ | ODS and other packages (cartridges/sponger, tutorial, demo, isparql, etc.) | cartridges_dav.vad %BR% doc_dav.vad | 131 MB |\n| /usr/lib/ | client drivers for ODBC & JDBC | jdbc-2.0/virtjdbc*.jar %BR% virtodbc*.so | 12 MB |\n| /usr/lib/virtuoso/hosting/ | Runtime loadable plugins for hosting and wiki markup | wikiv.so %BR% mediawiki.so %BR% creolewiki.so | 3.6 MB |\n| /var/lib/virtuoso/vsp/ | Splash-page for top-level web interface | vsp/ %BR% vsmx/ %BR% images/ %BR% frames/ %BR% css/ %BR% index.html %BR% etc. | 1.5 MB |\n| /usr/share/doc/virtuoso-opensource/ | Misc README data for OS platform | README.Debian %BR% README.CVS.gz %BR% etc. | 128 KB | \n| /usr/lib/jena/ | Java class modules for using Virtuoso from Jena | virt_jena.jar | 56 KB |\n| /usr/lib/sesame/ | Java class modules for using Virtuoso from Sesame | virt_sesame2.jar | 56 KB |\n| /var/lib/virtuoso/db | Default database directory | virtuoso.ini %BR% virtuoso.db | 12 MB |\n\n---+++ Operation\n\nWhen the server is run against a given .ini file, it looks for a database at\nthe name and location specified in that file. If none is found, the server\nwill create an empty database with minimal system schema.\n\nIf it finds an appropriate directory containing *.vad files (set in the\nvirtuoso.ini file during 'make install'), Virtuoso will install the\nConductor package by default.\n\nFrom there on, the Virtuoso administrator is expected to use the Conductor,\n<http://localhost:8890/conductor/>, to install further packages such\nas the ODS application suite, etc.\n\nAfter a checkpoint, a virtuoso.db file forms a portable unit encapsulating\nthe entirety of a database instance, and can be moved between servers, \nso you can implement custom applications and schemas using the client\ninterfaces (SQL/ODBC/JDBC) and avoid installing any of the packages, even\nthe Conductor, should you so desire.\n\n---+++ Configuration Options\n\nThe following parameters in a virtuoso.ini file defining a Virtuoso\ninstance control the resource consumption and performance:\n\n| [Database] | \n| DatabaseFile | virtuoso.db | Filename of database file | [[http://docs.openlinksw.com:80/virtuoso/dbadm.html#ini_Database][docs]] |\n| FileExtend | 200 | The amount of 8K-sized pages by which the database file automatically grows when the current file is not large enough. | [[http://docs.openlinksw.com:80/virtuoso/dbadm.html#ini_Database][docs]] |\n| MaxCheckpointRemap | 2000 | Controls how many pages may be stored other than their logical page during checkpoints | [[http://docs.openlinksw.com:80/virtuoso/CHECKPOINT.html#checkpointparams][docs]] |\n| Striping | 0 | Enables the database file-striping mechanism | [[http://docs.openlinksw.com:80/virtuoso/dbadm.html#ini_Striping][docs]] |\n| [Parameters] |\n| ServerThreads %BR% a/k/a %BR% MaxClientConnections | 20 | maximum number of threads (SQL+HTTP) used in the server - should typically be close to but not over the licensed number of concurrent connections | [[http://docs.openlinksw.com:80/virtuoso/dbadm.html#ini_Parameters][docs]] |\n| CheckpointInterval | 60 | interval (minutes) at which Virtuoso will automatically make a database checkpoint | [[http://docs.openlinksw.com:80/virtuoso/dbadm.html#ini_Parameters][docs]] |\n| NumberOfBuffers | 2000 | controls the amount of RAM (8K pages) used by Virtuoso to cache database files | [[http://docs.openlinksw.com:80/virtuoso/dbadm.html#ini_Parameters][docs]] |\n| MaxDirtyBuffers | 1200 | The maximum number of modified buffers to store before writing | |\n| MaxStaticCursorRows | 5000 | the maximum number of rows returned by a static cursor | [[http://docs.openlinksw.com:80/virtuoso/dbadm.html#fp_checkpointAuditTrail][docs]] |\n| FreeTextBatchSize | 100000 | the amount of text data processed in one batch of the free-text index during batch reindexing | [[http://docs.openlinksw.com:80/virtuoso/dbadm.html#ini_Parameters][docs]] |\n| [HTTPServer] |\n|ServerThreads %BR% a/k/a %BR% MaxClientConnections | 10 | Maximum concurrent HTTP sessions; must be less than the overall database ServerThreads | [[http://docs.openlinksw.com:80/virtuoso/webserver.html#vspconf][docs]] |\n| MaxKeepAlives | 10 | A maximum number of HTTP sockets with KeepAlive connections | [[http://docs.openlinksw.com:80/virtuoso/webserver.html#vspconf][docs]] |\n| KeepAliveTimeout | 10 | Timeout (in seconds) before an idle HTTP connection is closed | [[http://docs.openlinksw.com:80/virtuoso/webserver.html#vspconf][docs]] |\n| HTTPThreadSize | 280000 | Stack-size of an HTTP thread for handling connection and processing | [[http://docs.openlinksw.com:80/virtuoso/dbadm.html#ini_HTTPServer][docs]] |\n| [Striping] |\n| Segment1 | 100M, db-seg1-1.db, db-seg1-2.db | Segment-specification for disk-striping | [[http://docs.openlinksw.com:80/virtuoso/dbadm.html#ini_Striping][docs]] |\n| Segment2 | 100M, db-seg2-1.db | | |\n| [SPARQL] |\n| ResultSetMaxRows | 100000 | Maximum number of rows in a SPARQL resultset | [[http://docs.openlinksw.com:80/virtuoso/dbadm.html#ini_SPARQL][docs]] |\n| [Plugins] |\n| LoadPath | /usr/lib/virtuoso/hosting | Directory in which to search for plugins | [[http://docs.openlinksw.com:80/virtuoso/dbadm.html#ini_Plugins][docs]] |\n| Load1 | plain, wikiv | Main ODS-Wiki markup parser | |\n| Load2 | plain, mediawiki | Auxiliary MediaWiki markup parser module for ODS-Wiki | |\n| Load3 | plain, creolewiki | Auxiliary Creole wiki markup parser module for ODS-Wiki | |\n| Load4 | plain, im | ImageMagick plugin used by ODS-Gallery | |\n| Load5 | plain, wbxml2 | WbXML plugin used by the SyncML package | |\n| Load6 | plain, hslookup | Required for some Sponger Cartridge operations | |\n| Load7 | Hosting, hosting_php.so | Module for hosting PHP scripts within Virtuoso | |\n| ;Load8 | Hosting,hosting_perl.so | Module for hosting Perl scripts within Virtuoso (unsupported at present) | |\n| ;Load9 | Hosting,hosting_python.so | Module for hosting Python scripts within Virtuoso (unsupported at present) | |\n| ;Load10 | Hosting,hosting_ruby.so | Module for hosting Ruby scripts within Virtuoso (unsupported at present) | |\n| ;Load11 | msdtc,msdtc_sample | For Microsoft XA transaction support | |\n\n\n*Notes*: \n\n * Striping is an obvious way to control the amount of disk-space used; by default, \nstriping is off (0), so the [Striping] section does not come into play.\n * The checkpoint-interval setting is simply the amount of time for which\na temporary database will grow before being checkpointed into the main\nvirtuoso.db file, so the choice is scenario-dependent. For a given\nincoming transaction rate, a short interval will give frequent smaller\ncheckpoints, while specifying a longer interval will make fewer, slower,\ncheckpoints.\n * The numbers of threads in the database engine as a whole and\nspecifically allocated to the HTTP server will control performance,\nand each thread will cost a given amount of memory.\n * Naturally you can disable any or all unused plugins to further reduce\nthe memory footprint; for example, if not running ODS-Wiki, you can\nremove the wikiv, mediawiki, and creolewiki plugins.\n\nWe have a documentation page on\n[[http://docs.openlinksw.com/virtuoso/rdfperformancetuning.html#rdfperfgeneral][tuning Virtuoso for RDF usage]].\n\n---++ Analysis\n\nVirtuoso provides the status() command, which may be executed through the\nSQL interface (e.g., isql-v(1)).\n\nThe resultset from this command is \n[[http://docs.openlinksw.com:80/virtuoso/dbadm.html#DBSTAT][documented here]], but we\nspecifically highlight consideration of the NumberOfBuffers parameter;\nstatus() output shows how many buffers the server is actually\nusing, so you can tailor the allocated number accordingly.\n\n---++ Default\n\nBy default, the Debian virtuoso-opensource package enables all possible\nhosting options except Mono.\n\n| package .deb | 58 MB |\n| server binary, /usr/bin/virtuoso-t | 8.3 MB |\n| default database with Conductor installed | 37 MB |\n| Virtual memory allocation | 354 MB |\n| Resident memory used | 125 MB |\n\n---++ Embedded / Minimal\n\nThe minimum that is required to run Virtuoso is the server executable\n(virtuoso-t, or in Commercial Edition, virtuoso-iodbc-t) compiled with as\nfew options as possible, and the virtuoso.ini file. From there, the first\ntime the server is run against the virtuoso.ini, it will create the empty\ndatabase (virtuoso.db) with minimal schema.\n\nThe most important parameter to consider when optimizing for size is\nNumberOfBuffers. \n\nBy applying a few changes to virtuoso.ini, one can quite dramatically\nreduce the memory footprint:\n\n\n[Database]\nFileExtend = 100 ; down from 200\nMaxCheckpointRemap = 1000 ; down from 2000\n\n[TempDatabase]\nMaxCheckpointRemap = 1000 ; down from 2000\n \n[Parameters]\nServerThreads = 5 ; down from 10\n# MaxClientConnections = 5 ; alias for ServerThreads\nCheckpointInterval = 10 ; down from 60\nNumberOfBuffers = 100 ; down from 2000\nMaxDirtyBuffers = 50 ; down from 1200\nSchedulerInterval = 5 ; down from 10\nFreeTextBatchSize = 1000 ; down from 100000\n\n[HTTPServer]\nServerThreads = 2 ; down from 5\n# MaxClientConnections = 2 ; alias for ServerThreads\nKeepAliveTimeout = 5 ; down from 10\nHTTPThreadSize = 10000 ; down from 280000\n \n[Client]\nSQL_PREFETCH_ROWS = 10 ; down from 100\nSQL_PREFETCH_BYTES = 4096 ; down from 16000\n \n[Replication]\nServerEnable = 0 ; changed from 1\n\n[Zero Config]\n;ServerName = virtuoso (SAUCE) ; commented-out\n\n[SPARQL]\nResultSetMaxRows = 10000 ; down from 100000\n \n[Plugins]\n; ... ; comment-out all plugins\n\n\nResult:\n| default empty database with no Conductor installed | 12 MB |\n| Virtual memory allocation | 150 MB |\n| Resident memory used | 70 MB |\n\nExecuting status() shows that 1000 buffers are allocated and only 270 are in\nuse. \n\n---+++Comparison with LAMP\n\nOn the same machine, we installed Apache 2.x and MySQL 5.0 server using\nstandard Debian GNU/Linux packages:\n\n\nbash$ sudo apt-get install mysql-server apache2\n\n\nThis is not quite comparing like with like; Virtuoso includes not only \nHTTP and SQLinterfaces, but also a complete RDF Quad Store, SPARQL \nprocessor, free-text indexer, etc. However, out-of-the-box MySQL and \nApache consumptions compare to the above-tuned Virtuoso as follows:\n\n| | *Apache* | *MySQL* | *Virtuoso* |\n| package *.deb | 45.6 MB | - | 58 MB |\n| server binary | 0.3 MB | 7 MB | 8.3 MB | \n| default empty database | - | 21 MB | 12 MB |\n| virtual memory allocation | 229 MB | 123 MB | 350 MB |\n| resident memory usage | 2.8 MB | 15 MB | 70 MB |\n\n---++Enterprise-wide\n\nIn larger installations, the NumberOfBuffers parameter should be increased,\nbut there is no point in making the process so large it has to swap.\nTherefore we recommend about 60% memory should be allocated to buffers.\n\nAs examples, we consider two instances: our own web-server,\n[[http://www.openlinksw.com/][www.openlinksw.com]] (running on Debian\nGNU/Linux), and [[http://dbpedia.org/][DBpedia.org]] (running on Sun Solaris).\n\n| | *www* | *DBpedia* |\n|Virtual memory allocated | 2734 MB | 6257 MB |\n|Resident memory | 662 MB (65%) | 6216 MB (38%) |\n|Striping | 0 | 0 | \n|CheckpointInterval | 120 | 60 |\n|NumberOfBuffers | 20000, 18511 used | 100000, 99964 used |\n|MaxDirtyBuffers | 8000 | 40000 |\n|MaxCheckpointRemap | 16000 | 80000 |\n|FreeTextBatchSize | 100000 | 100000 | \n|ServerThreads (a/k/a MaxClientConnections) | 10 | 1000 |\n|HTTP ServerThreads (a/k/a MaxClientConnections) | 250 | 100 | \n|HTTPThreadSize | 280000 | 10000000 |\n|Plugins | only wikiv | wikiv, imagemagick, wbxml2 |\n|SPARQL ResultSetMaxRows | 1000 | 1000 | \n\n---++References\n\n * Virtuoso [[http://docs.openlinksw.com/virtuoso/perfdiag.html][performance diagnostics]]\n * Virtuoso [[http://docs.openlinksw.com/virtuoso/ptune.html][performance tuning]]\n" . "05e3a42e0551f57f5b43dd74c8bf1ef1" . . . "VirtConfigScale" . "2017-06-13T05:42:22Z" . "VirtConfigScale" . . . . "2017-06-13T05:42:22Z" . . . . . . . . . . . . . . . . . . . . . "VirtConfigScale" .