Talk:DB layer 2.0

Various Notes (before 20080508)

Agenda

From previous meeting + docs comments:
- multiple connections supported
- unit testing intro
- exercise and lams (we won't update these, but others, journal, workshop yes)
- $mdb or $db, global or static. (see "default $DB object" below)
- empty array returned instead of false
- $CFG->dbfamily, 100% out, possible? $CFG->dbtype out?
- performance profiling (MUST)
- logging (MUST)
- error/exception handling (Penny suggested at least looking at mahara's error handling for ideas) (MUST)
- $CFG->prefix vs {prefix} vs {tablename} - a la drupal or mahara.

To explain/review/discuss/decide:
- placeholders research and decision (:named vs ? vs $1) (both, not mixed)
- OOP architecture (see "Interface proposal" below)
- mdllib 2.0 own testing (functional real testing)
- first timeframe estimation

Other:
- discuss some new php inline docs policy
- review problems (see "Problems" below)
- ideas and comments
- next meeting (OOP architecture finished, mysql and postgresql working, XX tests against them (self testing), start collecting changes to search and destroy... with script, implementation details agreement)

Interface proposal

Sample implementation with basic API definition is in MDL-14679

All classes are stored in lib/dml/ directory. The naming convention is dbtype_dblibrary_moodle_database for database classes. For example oci8po_adodb_moodle_database and oci_pdo_moodle_database. There is a new configuration option $CFG->dblibrary which can be adodb, pdo or anything else in case of 3rd party modifications. The $CFG->dbtype is expected to contain internal driver name.

Each database class must fully abstract all operations with database including API for XMLDB editor.

To minimise conversion costs all function names are kept, though the prepared statements need a bit different method parameters:

$sql parameters need $params array with values, $sql query must not contain any user submitted data - instead use ? or :name parameters, both types are supported, but not both in the same query
$select parameters must be accompanied by $params too
$field1, $value1 are replaced by $conditions array - you can have more than one condition in get_records() and more than three in get_record() now

The use of recordsets was changed substantially - they can be used in foreach($rs as $record) directly. Recordset closing should be mandatory now, this might help with performance and memory consumption later. Each library abstraction or database class must define own recorset class with moodle_recordset interface.

While I like the foreach iterators here... this makes me think if that won't cause people to get confused between get_record() and get_recordset() functions. First ones doesn't need closing but second ones yes. Also, while thinking on this... do you think we could profile how many open recordsets are left open per request, debugging error if necessary? That will help developers, for sure. Eloy Lafuente (stronk7) 19:27, 6 May 2008 (CDT)

get_records() and similar functions that return arrays now return empty array if nothing found and false only if error occured, this should not cause any major regressions, all code should be verified during the conversion (proposed by Donal).

Database classes should not depend on $CFG settings - instead supplly them in constructors or use set_property() methods, this will allow us to use the same library for enrolment and auth plugins.

default $DB object

Martin proposed to use global $DB instead of function returning $mdb instance or factory methods. Nicolas confirmed that it should be suitable for unit testing purposes.

sample code:

function xyz($userid, $courseid) {
    global $DB, $CFG;
    if ($records = $DB->get_records_sql('SELECT * FROM {$CFG->prefix}abc WHERE userid=? AND courseid=?', array($userid, $courseid))) {
        //...
    }
}

Things like $db->debug = true; are changed to

$DB->set_debug(true);  // sets the database debug var

Implementation steps proposal

done:

import latest adodb for PHP5

in patch MDL-14679:

implement basic API as abstract moodle_database class and moodle_recordset interface
tweak lib/setup.php and related files - place all $DB setup code into one function setup_DB()
keep old global $db until everything is converted to new global $DB
all dmllib functions that do not accept $sql or $select can be rewritten to use new $DB - move those classes to lib/dmllib_deprecated.php for now - this allows us to run moodle in hybrid mode for testing purposes
move all unused dmllib functions to lib/dmllib_removed.php and remove body - this could be useful during the migration phase of contrib (make them to act as wrappers over new dmllib and emit debugging messages!)
implement basic mysql and postgresql classes

todo:

test, refactor, rewrite, improve, etc. the new dmllib classes
implement classes for all supported backends - oracle, mssql
majority of code can be converted to new classes while keeping magic quotes on (we are dealing with numbers mostly), the formslib can already return unslashed data and we can add new parameter to data_submitted() too
kill the magic quotes and convert the rest

We could also prepare an experimental mysql pdo driver in parallel - this could help to uncover potential problems in API design or implementation.

What parts of adodb do we use/need

connect to database, configure and set encoding
execute sql without result with bound parameters
get results as recorset with bound parameters
get results as array with bound parameters (can be emulated)
transactions - begin, end, rollback
get table columns information - needed for insert and update data validation (can be partially emulated, in fact this could be nice perf boost to use the data we already have in install.xml's)
get list of tables
debugging, logging , version info, etc.

PDO and native drivers in PHP5 offer the same level of functionality (except the legacy mysql driver).

Type hinting in sql parameters

All db drivers support some sort of type hinting when binding sql parameters. There is also option to autodetect the type from native PHP data type, but unfortunately this does not work well in our codebase (Example is error when passing empty string instead of 0 in pg). The API would be relatively simple when binding parameters one by one, but it is at present unsolved when defining parameters with array.

In case of insert and update we can get the types from our table definitions or by looking directly into database, but in case of general query parameters it is often hard to guess (especially in case of empty string ).

Solution could be to use special hints which would be part of parameter name, the last letter (if capital) would describe the type of parameter :paramN for integer number (seems better than I), :paramF for floating point numbers, paramT for text, paramC for char.

We are extremely lucky we do not use dates, blobs and other complex data types that differ in each database engine. In any case we should always try to submit valid parameter data, supply data types and report potential problems to developers and not rely on various conversions especially in mysql.

Documentation