#+TITLE: Postmodern
#+OPTIONS: num:nil
#+HTML_HEAD: <link rel="stylesheet" type="text/css" href="style.css" />
#+HTML_HEAD: <style>pre.src{background:#343131;color:white;} </style>
#+OPTIONS: ^:nil

Version 1.33.3

Postmodern is a Common Lisp library for interacting with [[https://postgresql.org][PostgreSQL databases]].
Features are:

- Efficient communication with the database server without need for foreign libraries.
- Support for UTF-8 on Unicode-aware Lisp implementations
- A syntax for mixing SQL and Lisp code
- Convenient support for prepared statements and stored procedures
- A metaclass for simple database-access objects

The biggest differences between this library and [[http://quickdocs.org/clsql/][clsql]] or [[https://github.com/fukamachi/cl-dbi][cl-dbi]]
are that Postmodern has no intention of being portable across different SQL
implementations (it embraces non-standard PostgreSQL features), and approaches
extensions like lispy SQL and database access objects in a quite different way.
This library was written because the CLSQL approach did not really work for
me. Your mileage may vary.

* Dependencies
  :PROPERTIES:
  :ID:       216c43d0-57ff-4ae3-a302-6d04a3d79665
  :CUSTOM_ID: 6887e7c3-c818-469a-b5f1-10a4b578b90b
  :END:
The library depends on [[http://quickdocs.org/usocket/][usocket]] (except on [[http://sbcl.org/][SBCL]] and [[https://franz.com/products/allegrocl/][ACL]], where the built-in socket library is used), [[https://github.com/pmai/md5.git][md5]], [[https://github.com/pcostanza/closer-mop.git][closer-mop]], [[https://github.com/sionescu/bordeaux-threads.git][bordeaux-threads]] if you want
thread-safe connection pools, and [[https://github.com/cl-plus-ssl/cl-plus-ssl.git][CL+SSL]] when SSL connections are needed.

As of version 1.3 it also depends on [[https://github.com/sharplispers/ironclad][ironclad]], [[https://github.com/massung/base64][base64]] and [[https://github.com/sabracrolleton/uax-15][uax-15]] because of the need to support scram-sha-256 authentication.

Postmodern itself is split into four different packages, some of which can be used independently:

- [[file:simple-date.html][Simple-date]] is a very basic implementation of date and time objects, used to support storing and retrieving time-related
SQL types. It is not loaded by default and you can use [[https://github.com/dlowe-net/local-time][local-time]] instead.

- [[file:cl-postgres.html][CL-postgres]] is the low-level library used for interfacing with a PostgreSQL server over a socket.

- [[file:s-sql.html][S-SQL]] is used to compile s-expressions to strings of SQL code, escaping any Lisp values inside, and doing as much as
possible of the work at compile time. Finally,

- [[file:postmodern.html][Postmodern]] itself is a wrapper around these packages and provides higher level functions, a very simple data access object that can be mapped directly to database tables and some convient utilities. It then tries to put all these things together into a convenient programming interface.itself is the library that tries to put all these things together into a convenient programming interface.

* License
  :PROPERTIES:
  :ID:       8ba6488f-4b3c-47f7-8a50-844363c5f484
  :CUSTOM_ID: 554e0dee-93b3-47b1-b808-3bd6c366b784
  :END:
Postmodern is released under a zlib-style license. Which approximately
means you can use the code in whatever way you like, except for passing
it off as your own or releasing a modified version without indication
that it is not the original.

The functions execute-file.lisp were ported from [[https://github.com/dimitri/pgloader][pgloader]] with grateful thanks to
Dimitri Fontaine and are released under a BSD-3 license.

* Download and installation
  :PROPERTIES:
  :ID:       d4cca0ee-ff7f-4530-9be7-e9b3de62bdb4
  :CUSTOM_ID: 6f05b344-12c2-42b0-b231-3aaced30afb8
  :END:
We suggest using [[https://quicklisp.org][quicklisp.org]] for installation.

A git repository with the most recent changes can be viewed or checked out at:

> git clone [[https://github.com/marijnh/Postmodern]]

* Quickstart
  :PROPERTIES:
  :ID:       f55510fb-3715-4cdd-9e37-4ab0e968e72d
  :CUSTOM_ID: b5bb7222-8134-4dcb-83b7-f764b7f2bb33
  :END:
This quickstart is intended to give you a feel of the way coding with
Postmodern works. Further details about the workings of the library
can be found in the reference manuals linked below.

Assuming you have already installed quicklisp, load postmodern.
#+BEGIN_SRC lisp
(ql:quickload :postmodern)
(use-package :postmodern)
#+END_SRC

If you have a PostgreSQL server running on localhost, with a database called 'testdb'
on it, which is accessible for user 'foucault' with password 'surveiller', there are
two basic ways to connect to a database. If your role/application/database(s) looks
like a 1:1 relationship and you are not using threads, you can connect like this:
#+BEGIN_SRC lisp
(connect-toplevel "testdb" "foucault" "surveiller" "localhost")
#+END_SRC

This will establish a connection to be used by all code, except for that wrapped
in a with-connection form, which takes the same arguments but only establishes
the connection within that lexical scope.

Connect-toplevel will maintain a single connection for the life of the session.

If the server is not on localhost, replace that string with a string containing the ip address of the server:
#+BEGIN_SRC lisp
(connect-toplevel "testdb" "foucault" "surveiller" "216.27.61.135")
#+END_SRC

If the Postgresql server is running on a port other than 5432,
you would also pass the appropriate keyword port parameter. E.g.:

#+BEGIN_SRC lisp
(connect-toplevel "testdb" "foucault" "surveiller" "localhost" :port 5434)
#+END_SRC

Ssl connections would similarly use the keyword parameter :use-ssl and
pass :no, :try, :require, :yes, or :full

- :try means if the server supports it
- :require means use provided ssl certificate with no verification
- :yes means verify that the server cert is issued by a trusted CA, but does not verify the server hostname
- :full means expect a CA-signed cert for the supplied hostname and verify the server hostname

If you have multiple roles connecting to one or more databases, i.e. 1:many or
many:1, (in other words, changing connections) or you are using threads (each thread must have its own connection) then with Postmodern you can use the =with-connection= form which establishes a connection with a lexical scope is more appropriate.
#+BEGIN_SRC lisp
(with-connection '("testdb" "foucault" "surveiller" "localhost")
  ...)
#+END_SRC
For other connection options, please see [[https://marijnhaverbeke.nl/postmodern.html#function-connection][https://marijnhaverbeke.nl/postmodern.html#function-connection]]

If you are creating a database, you need to have established a connection
to a currently existing database (typically "postgres"). Assuming the foucault role
is a superuser and you want to stay in a development connection with your new database
afterwards, you would first use with-connection to connect to postgres, create the
database and then switch to connect-toplevel for development ease.
#+BEGIN_SRC lisp
(with-connection '("postgres" "foucault" "surveiller" "localhost")
  (create-database 'testdb :limit-public-access t
                     :comment "This database is for testing silly theories"))

(connect-toplevel "testdb" "foucault" "surveiller" "localhost")
#+END_SRC
Note: (create-database) functionality is new to postmodern v. 1.32. Setting the
:limit-public-access parameter to t will block connections to that database from
anyone who you have not explicitly given permission (except other superusers).

A word about Postgresql connections. Postgresql connections are not lightweight
threads. They actually consume about 10 MB of memory per connection. In addition, any connections
which require security (ssl or scram authentication) will take additiona time and create
more overhead.

Postgresql can be tuned to limit the number of connections allowed at any one time. It defaults to 100. The parameter is set in the postgresql.conf file. Depending on the size of your server and what you are doing, the sweet spot generally seems to be between 200-400 connections before you need to bring in connection pooling.

If your application is threaded, each thread should use its own connection. Connections are stateful and attempts to use the same connection for multiple threads will

If you have an application like a web app which will make many connections, you also
generally do not want to create and drop connections for every query. The usual solution
is to use connection pools so that the application is grabbing an already existing connection
and returning it to the pool when finished, saving connection time and memory.

To use postmodern's simple connection pooler, the =with-connection= call would look like:
#+BEGIN_SRC lisp
(with-connection '("testdb" "foucault" "surveiller" "localhost" :pooled-p t)
      ...)
#+END_SRC

The maximum number of connections in the pool is set in the special variable
*max-pool-size*, which defaults to nil (no maximum).

Now for a basic sanity test which does not need a database connection at all:

#+BEGIN_SRC lisp
(query "select 22, 'Folie et déraison', 4.5")
;; => ((22 "Folie et déraison" 9/2))
#+END_SRC
That should work. query is the basic way to send queries to the database.
The same query can be expressed like this:
#+BEGIN_SRC lisp
(query (:select 22 "Folie et déraison" 4.5))
;; => ((22 "Folie et déraison" 9/2))
#+END_SRC

In many contexts, query strings and lists starting with keywords can be used
interchangeably. The lists will be compiled to SQL. The S-SQL manual describes
the syntax used by these expressions. Lisp values occurring in them are
automatically escaped. In the above query, only constant values are used, but
it is possible to transparently use run-time values as well:
#+BEGIN_SRC lisp
(defun database-powered-addition (a b)
  (query (:select (:+ a b)) :single))

(database-powered-addition 1030 204)
;; => 1234
#+END_SRC

That last argument, :single, indicates that we want the result not as a list
of lists (for the result rows), but as a single value, since we know that we
are only selecting one value. Some other options are :rows, :row, :column, :alists,
:plists, :array-hash, :json-strs, :json-str, :json-array-str, :dao and :none.
Their precise effect is documented in the [[file:postmodern.html][Postmodern reference manual]] under [[file:postmodern.html#querying][Queries]]

You do not have to pull in the whole result of a query at once, you can
also iterate over it with the doquery macro:
#+BEGIN_SRC lisp
(doquery (:select 'x 'y :from 'some-imaginary-table) (x y)
  (format t "On this row, x = ~A and y = ~A.~%" x y))
#+END_SRC

You can work directly with the database or you can use a simple
database-access-class (aka dao) which would cover all the fields in a row.
This is what a database-access class looks like:
#+BEGIN_SRC lisp
(defclass country ()
  ((name :col-type string :initarg :name
         :reader country-name)
   (inhabitants :col-type integer :initarg :inhabitants
                :accessor country-inhabitants)
   (sovereign :col-type (or db-null string) :initarg :sovereign
              :accessor country-sovereign))
  (:metaclass dao-class)
  (:keys name))
#+END_SRC
The above defines a class that can be used to handle records in a table named
'country' with three columns: name, inhabitants, and sovereign. The :keys
parameter specifies which column(s) are used for the primary key. Once you have
created the class, you can return an instance of the country class by calling

#+BEGIN_SRC lisp
(get-dao 'country "Croatia")
#+END_SRC

You can also define classes that use multiple columns in the primary key:

#+BEGIN_SRC lisp
(defclass points ()
  ((x :col-type integer :initarg :x
      :reader point-x)
   (y :col-type integer :initarg :y
      :reader point-y)
   (value :col-type integer :initarg :value
          :accessor value))
  (:metaclass dao-class)
  (:keys x y))
#+END_SRC

In this case, retrieving a points record would look like the following where
12 and 34 would be the values you are looking to find in the x column and y
column respectively.:

#+BEGIN_SRC lisp
(get-dao 'points 12 34)
#+END_SRC

Consider a slightly more complicated version of country:
#+BEGIN_SRC lisp
(defclass country-c ()
  ((id :col-type integer :col-identity t :accessor id)
   (name :col-type string :col-unique t :check (:<> 'name "")
         :initarg :name :reader country-name)
   (inhabitants :col-type integer :initarg :inhabitants
                :accessor country-inhabitants)
   (sovereign :col-type (or db-null string) :initarg :sovereign
              :accessor country-sovereign)
   (region-id :col-type integer :col-references ((regions id))
              :initarg :region-id :accessor region-id))
  (:metaclass dao-class)
  (:table-name countries))
#+END_SRC

In this example we have an id column which is specified to be an identity column.
Postgresql will automatically generate a sequence of of integers and this will
be the primary key.

We have a name column which is specified as unique and is not null and the
check will ensure that the database refuses to accept an empty string as the name.

We have a region-id column which references the id column in the regions table.
This is a foreign key constraint and Postgresql will not accept inserting a country
into the database unless there is an existing region with an id that matches this
number. Postgresql will also not allow deleting a region if there are countries
that reference that region's id. If we wanted Postgresql to delete countries when
regions are deleted, that column would be specified as:
#+BEGIN_SRC lisp
(region-id :col-type integer :col-references ((regions id) :cascade)
  :initarg :region-id :accessor region-id)
#+END_SRC

Now you can see why the double parens.

We also specify that the table name is not "country" but "countries". (Some style guides
recommend that table names be plural and references to rows be singular.)

** Table Creation
   :PROPERTIES:
   :ID:       6ac2dcab-bd3b-48de-9ea0-fec010d76879
   :CUSTOM_ID: eccad49e-8df8-4451-89ff-4987b103c9dd
   :END:
*** With SQL or S-SQL
    :PROPERTIES:
    :CUSTOM_ID: 7629810d-8ccb-4236-b540-6aff596a042f
    :END:
You can create tables directly without the need to define a class, and in more
complicated cases, you may need to use the s-sql :create-table operator or plain
vanilla sql. Staying with examples that will match our slightly more complicated
dao-class above (but ignoring the fact that the references parameter would
actually require us to create the regions table first) and using s-sql rather
than plain vanilla sql would be the following:
#+BEGIN_SRC lisp
(query (:create-table 'countries
      ((id :type integer  :primary-key t :identity-always t)
       (name :type string :unique t :check (:<> 'name ""))
       (inhabitants :type integer)
       (sovereign :type (or db-null string))
       (region-id :type integer :references ((regions id))))))
#+END_SRC

Restated using vanilla sql:
#+BEGIN_SRC sql
(query "CREATE TABLE countries (
           id INTEGER NOT NULL PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
           name TEXT NOT NULL UNIQUE CHECK (NAME <> E''),
           inhabitants INTEGER NOT NULL,
           sovereign TEXT,
           region_id INTEGER NOT NULL REFERENCES regions(id)
             MATCH SIMPLE ON DELETE RESTRICT ON UPDATE RESTRICT)")
#+END_SRC
Let's look at a slightly different example:
#+BEGIN_SRC lisp
(query (:create-table so-items
         ((item-id :type integer)
          (so-id :type (or integer db-null) :references ((so-headers id)))
          (product-id :type (or integer db-null))
          (qty :type (or integer db-null))
          (net-price :type (or numeric db-null)))
         (:primary-key item-id so-id)))
#+END_SRC

Restated using plain sql:
#+BEGIN_SRC sql
(query "CREATE TABLE so_items (
           item_id INTEGER NOT NULL,
           so_id INTEGER REFERENCES so_headers(id)
                 MATCH SIMPLE ON DELETE RESTRICT ON UPDATE RESTRICT,
           product_id INTEGER,
           qty INTEGER,
           net_price NUMERIC,
           PRIMARY KEY (item_id, so_id)
     );")
#+END_SRC
In the above case, the new table's name will be so_items because sql does not
allow hyphens and plain vanilla sql will require that. Postmodern will
generally allow you to use the quoted symbol 'so-items. This is also true for
all the column names. The column item-id is an integer and cannot be null. The
column so-id is also an integer, but is allowed to be null and is a foreign key
to the id field in the so-headers table so-headers. The primary key is actually
a composite of item-id and so-id. (If we wanted the primary key to be just
item-id, we could have specified that in the form defining item-id.)

For more detail and examples on building tables using the s-sql approach,
see [[file:create-tables.html][create-tables.html]]

*** With a Previously Created DAO
    :PROPERTIES:
    :CUSTOM_ID: 5129ed89-2ff3-45b4-ae70-41c2c286eacc
    :END:
You can also use a previously defined dao to create a table as well. The
dao-table-definition function examines a dao class and generates the plain vanilla
sql for creating a table. That can be passed on to the execute function to create
a table. For example the simple country dao would generate:
#+BEGIN_SRC lisp
(dao-table-definition 'country)

"CREATE TABLE country
  (name TEXT NOT NULL,
   inhabitants INTEGER NOT NULL,
   sovereign TEXT DEFAULT NULL,
  PRIMARY KEY (name))"

(execute (dao-table-definition 'country))
#+END_SRC
(Execute works like query, but does not expect any results back.)

The slightly more complicated country-c version would generate:
#+BEGIN_SRC lisp
(dao-table-definition 'country-c)

;; => "CREATE TABLE countries (
;;       id INTEGER NOT NULL PRIMARY KEY generated always as identity,
;;       name TEXT NOT NULL UNIQUE,
;;       inhabitants INTEGER NOT NULL,
;;       sovereign TEXT DEFAULT NULL,
;;       region_id INTEGER NOT NULL REFERENCES regions(id)
;;         MATCH SIMPLE ON DELETE RESTRICT ON UPDATE RESTRICT)

(execute (dao-table-definition 'country-c))
#+END_SRC
For the rest of the discussion on this page, we will use the simpler version
and save the more complicated version for the [[file:postmodern.html][postmodern]] manuals.

** Inserting Data
   :PROPERTIES:
   :ID:       980103dd-9593-4047-9954-92e80f3785a9
   :CUSTOM_ID: 0b22f6d2-20e4-49cd-a311-083aade58cbf
   :END:
Similarly to table creation, you can insert data using the s-sql wrapper, plain
vanilla sql or daos.

The s-sql approach would be:

#+BEGIN_SRC lisp
(query (:insert-into 'country :set 'name "The Netherlands"
                                   'inhabitants 16800000
                                   'sovereign "Willem-Alexander"))

(query (:insert-into 'country :set 'name "Croatia"
                                   'inhabitants 4400000))
#+END_SRC

You could also insert multiple rows at a time but that requires the same columns for each row:

#+BEGIN_SRC lisp
(query (:insert-rows-into 'country :columns 'name 'inhabitants 'sovereign
                                   :values '(("The Netherlands" 16800000 "Willem-Alexander")
                                             ("Croatia" 4400000 :null))))
#+END_SRC

The sql approach would be:
#+BEGIN_SRC lisp
(query "insert into country (name, inhabitants, sovereign)
                            values ('The Netherlands', 16800000, 'Willem-Alexander')")

(query "insert into country (name, inhabitants)
                            values ('Croatia', 4400000)")
#+END_SRC

The multiple row sql approach would be:
#+BEGIN_SRC lisp
(query "insert into country (name, inhabitants, sovereign)
                            values
                              ('The Netherlands', 16800000, 'Willem-Alexander'),
                              ('Croatia', 4400000, NULL)")
#+END_SRC

Using dao classes would look like:
Let us go back to our approach using a dao class and add a few countries:
#+BEGIN_SRC lisp
(insert-dao (make-instance 'country :name "The Netherlands"
                                    :inhabitants 16800000
                                    :sovereign "Willem-Alexander"))
(insert-dao (make-instance 'country :name "Croatia"
                                    :inhabitants 4400000))
#+END_SRC
Postmodern does not yet have an insert-daos (plural) function.

Staying with the dao class approach, to update Croatia's population, we could do this:
#+BEGIN_SRC lisp
(let ((croatia (get-dao 'country "Croatia")))
  (setf (country-inhabitants croatia) 4500000)
  (update-dao croatia))
(query (:select '* :from 'country))

;; => (("The Netherlands" 16800000 "Willem-Alexander")
;;     ("Croatia" 4500000 :NULL))
#+END_SRC
Please see the documentation for s-sql for more examples of using s-sql rather than daos.

Next, to demonstrate a bit more of the S-SQL syntax, here is the query the
utility function list-tables uses to get a list of the tables in a database:
#+BEGIN_SRC lisp
(sql (:select 'relname :from 'pg-catalog.pg-class
      :inner-join 'pg-catalog.pg-namespace :on (:= 'relnamespace
                                                   'pg-namespace.oid)
      :where (:and (:= 'relkind "r")
                   (:not-in 'nspname (:set "pg_catalog" "pg_toast"))
                   (:pg-catalog.pg-table-is-visible 'pg-class.oid))))

;; => "(SELECT relname FROM pg_catalog.pg_class
;;      INNER JOIN pg_catalog.pg_namespace ON (relnamespace = pg_namespace.oid)
;;      WHERE ((relkind = 'r') and (nspname NOT IN ('pg_catalog', 'pg_toast'))
;;             and pg_catalog.pg_table_is_visible(pg_class.oid)))"
#+END_SRC

sql is a macro that will simply compile a query, it can be useful for seeing
how your queries are expanded or if you want to do something unexpected with
them.

As you can see, lists starting with keywords are used to express SQL commands
and operators (lists starting with something else will be evaluated and then
inserted into the query). Quoted symbols name columns or tables (keywords can
also be used but might introduce ambiguities). The syntax supports subqueries,
multiple joins, stored procedures, etc. See the [[file:s-sql.html][S-SQL reference manual]] for a
complete treatment.

Finally, here is an example of the use of prepared statements:
#+BEGIN_SRC lisp
(defprepared sovereign-of
  (:select 'sovereign :from 'country :where (:= 'name '$1))
  :single!)
(sovereign-of "The Netherlands")
;; => "Willem-Alexander"
#+END_SRC

The defprepared macro creates a function that takes the same amount of
arguments as there are $X placeholders in the given query. The query will
only be parsed and planned once (per database connection), which can be
faster, especially for complex queries.
#+BEGIN_SRC lisp
(disconnect-toplevel)
#+END_SRC

* Authentication
  :PROPERTIES:
  :ID:       5ad46584-6887-4866-9c40-633768c0d39a
  :CUSTOM_ID: b15533d8-efa3-43a9-b632-a3256cea261f
  :END:
Postmodern can use either md5 or scram-sha-256 authentication. Scram-sha-256
authentication is obviously more secure, but slower than md5, so take that into
account if you are planning on opening and closing many connections without
using a connection pooling setup..

Other authentication methods have not been tested. Please let us know if there
is a authentication method that you believe should be considered.

* Reference
  :PROPERTIES:
  :ID:       27b39236-15ee-42c3-958a-3c9c903c4567
  :CUSTOM_ID: 8993c7bd-4ba2-4080-8c5a-ff90de45eca5
  :END:
The reference manuals for the different components of Postmodern are kept
in separate files. For using the library in the most straightforward way,
you only really need to read the Postmodern reference and glance over the
S-SQL reference. The simple-date reference explains the time-related data
types included in Postmodern, and the CL-postgres reference might be useful
if you just want a low-level library for talking to a PostgreSQL server.

- [[file:postmodern.html][Postmodern]]
- [[file:s-sql.html][S-SQL]]
- [[file:simple-date.html][Simple-date]]
- [[file:cl-postgres.html][CL-postgres]]

Some specific topics in more detail

- [[file:array-notes.html][Array-Notes]]
- [[file:create-tables.html][Creating Tables]]
- [[file:dao-classes.html][Database Access Object (Dao) Classes]]
- [[file:dynamic-queries.html][Dynamic Queries]]
- [[file:interval-notes.html][Interval Notes]]
- [[file:isolation-notes.html][Isolation Notes]]

* Data Types
  :PROPERTIES:
  :ID:       d089d05b-4485-4fb5-9097-5a66492bc470
  :CUSTOM_ID: e2475974-6131-40ef-9ca3-54bf111a5dd0
  :END:
** Data Types
   :PROPERTIES:
   :ID:       deae4656-1b87-4518-9718-3b3e8c35f6c5
   :CUSTOM_ID: b850ea6c-b61e-4601-8423-65a8a626cb58
   :END:
For a short comparison of lisp and Postgresql data types (date and time datatypes are described in the next section)

| SQL type         | Description                                                       |
|------------------+-------------------------------------------------------------------|
| smallint         | -32,768 to +32,768 2-byte storage                                 |
| integer          | -2147483648 to +2147483647 integer, 4-byte storage                |
| bigint           | -9223372036854775808 to 9223372036854775807 8-byte storage        |
| numeric(X, Y)    | User specified, see notes below                                   |
| real             | float, 6 decimal digit precision 4-byte storage                   |
| double-precision | floating, 15 decimal digit precision 8-byte storage               |
| text             | variable length string, no limit specified                        |
| char(X)          | char(length), blank-padded string, fixed storage length           |
| varchar(X)       | varchar(length), non-blank-padded string, variable storage length |
| boolean          | boolean, 'true'/'false', 1 byte                                   |
| bytea            | binary string allowing non-printable octets                       |
| date             | date range: 4713 BC to 5874897 AD                                 |
| interval         | See [[file:interval-notes.html][interval]]                                                      |
| array            | See discussion at [[file:array-notes.html][array-notes.html]]                                |

Numeric and decimal are variable storage size numbers with user specified precision.
Up to 131072 digits before the decimal point; up to 16383 digits after the decimal point.
The syntax is numeric(precision, scale). Numeric columns with a specified scale will coerce input
values to that scale. For more detail, see https://www.postgresql.org/docs/current/datatype-numeric.html

| PG Type          | Sample Postmodern Return Value                                              | Lisp Type (per sbcl)                 |
|------------------+-----------------------------------------------------------------------------+--------------------------------------|
| boolean          | T                                                                           | BOOLEAN                              |
| boolean          | NIL  (Note: within Postgresql this will show 'f')                           | BOOLEAN                              |
| int2             | 273                                                                         | (INTEGER 0 4611686018427387903)      |
| int4             | 2                                                                           | (INTEGER 0 4611686018427387903)      |
| char             | A                                                                           | (VECTOR CHARACTER 64)                |
| varchar          | id&wl;19                                                                    | (VECTOR CHARACTER 64)                |
| numeric          | 78239/100                                                                   | RATIO                                |
| json             | { "customer": "John Doe", "items": {"product": "Beer","qty": 6}}            | (VECTOR CHARACTER 64)                |
| jsonb            | {"title": "Sleeping Beauties", "genres": ["Fiction", "Thriller", "Horror"]} | (VECTOR CHARACTER 128)               |
| float            | 782.31                                                                      | SINGLE-FLOAT                         |
| point            | (0.0d0 0.0d0)                                                               | CONS                                 |
| lseg             | ((-1.0d0 0.0d0) (2.0d0 4.0d0))                                              | CONS                                 |
| path             | ((1,0),(2,4))                                                               | (VECTOR CHARACTER 64)                |
| box              | ((1.0d0 1.0d0) (0.0d0 0.0d0))                                               | CONS                                 |
| polygon          | ((21,0),(2,4))                                                              | (VECTOR CHARACTER 64)                |
| line             | {2,-1,0}                                                                    | (VECTOR CHARACTER 64)                |
| double_precision | 2.38921379231d8                                                             | DOUBLE-FLOAT                         |
| double_float     | 2.3892137923231d8                                                           | DOUBLE-FLOAT                         |
| circle           | <(0,0),2>                                                                   | (VECTOR CHARACTER 64)                |
| cidr             | 100.24.10.0/24                                                              | (VECTOR CHARACTER 64)                |
| inet             | 100.24.10.0/24                                                              | (VECTOR CHARACTER 64)                |
| interval         | #<INTERVAL P1Y3H20m>                                                        | INTERVAL                             |
| bit              | #*1                                                                         | (SIMPLE-BIT-VECTOR 1)                |
| int4range        | [11,24)                                                                     | (VECTOR CHARACTER 64)                |
| uuid             | 40e6215d-b5c6-4896-987c-f30f3678f608                                        | (VECTOR CHARACTER 64)                |
| text_array       | #(text one text two text three)                                             | (SIMPLE-VECTOR 3)                    |
| integer_array    | #(3 5 7 8)                                                                  | (SIMPLE-VECTOR 4)                    |
| bytea            | #(222 173 190 239)                                                          | (SIMPLE-ARRAY (UNSIGNED-BYTE 8) (4)) |
| text             | Lorem ipsum dolor sit amet, consectetur adipiscing elit                     | (VECTOR CHARACTER 64)                |
| enum_mood        | happy *Note: enum_mood was defined as 'sad','ok' or 'happy'                 | (VECTOR CHARACTER 64)                |

* Passing Parameters as Text or Binary
:PROPERTIES:
:CUSTOM_ID: passing-binary-parameters
:END:
To avoid sql injection attacks you should be using parameterized queries. They also help avoid the hassle of worrying about double quotes and single quotes. They are also a required part of prepared queries. The following shows an example of parameterized query in regular sql and s-sql.
#+begin_src lisp
(query "select name, address from customers where id = $1" 237)

(query (:select 'name 'address :from 'customers :where (:= 'id '$1)) 237)
#+end_src
Postmodern defaults to passing query parameters as text. In an example like the above, Postgresql would look at the customer table, see that "id" must be an integer or big integer and proceed accordingly. If Postgresql does not have a table column which would allow it to determine the appropriate data type and you do not specify the date type in the query, Postgresql treats the parameter as text. The following example in plain sql demonstrates:
#+BEGIN_SRC lisp
(query "select $1" 1 :single)
"1"
(query "select $1" 1.5 :single)
"1.5"
(query "select $1" T :single)
"true"
(query "select $1" nil :single)
"false"
(query "select $1" :NULL :single)
:NULL
#+END_SRC

You can specify the type and Postgresql will attempt to follow that:
#+begin_src lisp
(query "select $1::integer" 1 :single)
1
#+end_src
As of version 1.33, Postmodern provides an optional setting in the database connection object which will cause Postmodern to pass parameters to Postgresql in binary format if that format is available for that datatype. Currently this means int2, int4, int8, float, double-float (except clisp) and boolean. Rational numbers continue to be passed as text.

The default for cl-postgres/Postmodern is to continue to pass parameters to Postgresql as text (not in binary format) in order to avoid breaking existing user code. If you want to pass parameters to Postgresql in binary format, you can either provide the keyword parameter :use-binary in creating a connection like so:
#+begin_src lisp
  (connect "test-db" "test-user" "test-password" "192.168.5.223"
           :port 5434 :pooled-p t :use-ssl try :application-name "test-app" :use-binary t)
#+end_src
or use the postmodern =with-connection= macro:
#+begin_src lisp
  (with-connection '("database-name" "user-name" "user-password" "localhost or IP address"
                     :use-binary t [any other keyword parameters you want to apply])
    ...)
#+end_src
You can also change the flag after the connection is established with the =use-binary-parameters= function, passing T to use binary parameters or nil to use text parameters:
#+begin_src lisp
  (use-binary-parameters *database* t)

  (use-binary-parameters some-database-connection t)
#+end_src
Using binary parameters the results has the following results:
#+BEGIN_SRC lisp
(query "select $1" 1 :single)
1
(query "select $1" 1.5 :single)
1.5
(query "select $1" T :single)
T
(query "select $1" nil :single)
NIL
(query "select $1" :NULL :single)
:NULL
#+END_SRC
Using binary parameters does tighten type checking when using prepared queries. You will not be able to use prepared queries with varying formats. In other words, if you have a prepared query that you pass an integer as the first parameter and a string as the second parameter the first time it is used, any subsequent uses of that prepared query during that session will also have to pass an integer as the first parameter and a string as the second parameter.

Benchmarking does indicate a slight speed and consing benefit to passing parameters as binary, but your mileage will vary depending on your use case.

* Caveats and to-dos
  :PROPERTIES:
  :ID:       157ea05f-4c49-4e49-8cf9-a3df4bf16b09
  :CUSTOM_ID: fc095960-ba9d-4a98-ac89-0db7a56777f1
  :END:
** Timezones and Simple-Date and Local-Time
   :PROPERTIES:
   :ID:       8ff631c8-994f-4658-bbc4-779afc80bdf2
   :CUSTOM_ID: e5c68251-0ca6-4f96-8d36-175cec626eeb
   :END:
It is important to understand how postgresql (not postmodern) handles
timestamps and timestamps with time zones. Postgresql keeps everything
in UTC, it does not store a timezone even in a timezone aware column.
If you use a timestamp with timezone column, postgresql will calculate
the UTC time and will normalize the timestamp data to UTC. When you
later select the record, postgresql will look at the timezone for the
postgresql session, retrieve the data and then provide the data
recalculated from UTC to the timezone for that postgresql session.
There is a good writeup of timezones at
[[http://blog.untrod.com/2016/08/actually-understanding-timezones-in-postgresql.html]]
and [[http://phili.pe/posts/timestamps-and-time-zones-in-postgresql/][http://phili.pe/posts/timestamps-and-time-zones-in-postgresql/]].

Without simple-date or local-time properly loaded and without use of the
Postgresql to_char function, sample date and time data from postgresql will
look like:

| PG Type         |                                        Sample Return Value | Lisp Type                       |
|-----------------+------------------------------------------------------------+---------------------------------|
| date            |                                                 3798576000 | (INTEGER 0 4611686018427387903) |
| time_wo_tz      | ((HOURS 9) (MINUTES 47) (SECONDS 9) (MICROSECONDS 926531)) | CONS                            |
| time_w_tz       |                                         09:47:16.510459-04 | (VECTOR CHARACTER 64)           |
| timestamp_wo_tz |                                                 3798611253 | (INTEGER 0 4611686018427387903) |
| timestamp_w_tz  |                                                 3798625647 | (INTEGER 0 4611686018427387903) |

YOU DO NOT NEED TO ADD ANY OTHER LIBRARIES IF ALL YOU WANT TO DO IS GET ISO 8601 or
[[https://tools.ietf.org/html/rfc3339][RFC 3339]] PROPERLY FORMATTED TIME AND DATE STRINGS WITH THE [[https://www.postgresql.org/docs/current/functions-formatting.html][TO_CHAR]] POSTGRESQL
FUNCTION.

Assume a data table with columns  "col_timestamp_without_time_zone", "col_timestamp_with_time_zone", "col_timestamptz", "col_timestamp", "col_time"
and "col_date".

First, just basic sql. In this example we ask for the timestamp_with_time_zone field
three times to show differences in the result dealing with time zones. First we do not
add a timezone parameter to the pattern, the second time  we ask for the time zone
using TZ, the third time we ask for the offset from UTC and get back -04. We would get
the same result using those additional parameters with the col_timestamptz field.

#+BEGIN_SRC sql
(query "(SELECT to_char(col_timestamp_with_time_zone, E'YYYY-MM-DD HH24:MI:SS'),
                to_char(col_timestamp_with_time_zone, E'YYYY-MM-DD HH24:MI:SS TZ'),
                to_char(col_timestamp_with_time_zone, E'YYYY-MM-DD HH24:MI:SS OF'),
                to_char(col_timestamptz, E'YYYY-MM-DD HH24:MI:SS'),
                to_char(col_timestamp, E'YYYY-MM-DD HH24:MI:SS'),
                to_char(col_time, E'HH24:MI:SS'),
                to_char(col_date, E'YYYY-MM-DD')
         FROM short_data_type_tests
         WHERE (id = 1))")
(("2020-10-30 19:30:54" "2020-10-30 19:30:54 EDT" "2020-10-30 19:30:54 -04"
"2020-10-30 19:30:54" "2020-10-30 19:30:54" "19:30:54" "2020-10-30"))
#+END_SRC

Now the s-sql version:
#+BEGIN_SRC lisp
(query (:select  (:to-char 'col_timestamp_with_time_zone "YYYY-MM-DD HH24:MI:SS TZ")
                 (:to-char 'col_timestamp_with_time_zone "YYYY-MM-DD HH24:MI:SS OF")
                 (:to-char 'col_timestamptz "YYYY-MM-DD HH24:MI:SS")
                 (:to-char 'col_timestamp "YYYY-MM-DD HH24:MI:SS")
                 (:to-char 'col_time "HH24:MI:SS")
                 (:to-char 'col_date "YYYY-MM-DD")
        :from 'short-data-type-tests
        :where (:= 'id 1)))

(("2020-10-30 19:30:54 EDT" "2020-10-30 19:30:54 -04" "2020-10-30 19:30:54"
  "2020-10-30 19:30:54" "19:30:54" "2020-10-30"))
#+END_SRC
*** Simple-Date Library Use
The Simple-date add-on library (not enabled by default)
provides types (CLOS classes) for dates, timestamps, and intervals
similar to the ones SQL databases use, in order to be able to store and read
these to and from a database in a straighforward way. A few obvious operations
are defined on these types.

To use simple-date with cl-postgres or postmodern,
load simple-date-cl-postgres-glue and register suitable SQL
readers and writers for the associated database types.

#+BEGIN_SRC lisp
(ql:quickload :simple-date/postgres-glue)

(setf cl-postgres:*sql-readtable*
        (cl-postgres:copy-sql-readtable
          simple-date-cl-postgres-glue:*simple-date-sql-readtable*))
#+END_SRC

With simple date loaded, the same data will look like this:

| PG Type                    | Sample Return Value                  | Lisp Type             |
|----------------------------+--------------------------------------+-----------------------|
| date                       | #<DATE 16-05-2020>                   | DATE                  |
| time_without_timezone      | #<TIME-OF-DAY 09:47:09.926531>       | TIME-OF-DAY           |
| time_with_timezone         | 09:47:16.510459-04                   | (VECTOR CHARACTER 64) |
| timestamp_without_timezone | #<TIMESTAMP 16-05-2020T09:47:33,315> | TIMESTAMP             |
| timestamp_with_timezone    | #<TIMESTAMP 16-05-2020T13:47:27,855> | TIMESTAMP             |

You can export these as json-strings with the encode-json-to-string function. E.g.
#+BEGIN_SRC lisp
(encode-json-to-string (query (:select 'timestamp-without-time-zone
                               :from 'test-data
                               :where (:= 'id 1))
                               :single))
"\"2020-12-30 13:30:54:0\""
#+END_SRC
Or more simply with passing one of the json result type parameters to the query
function. E.g.
#+BEGIN_SRC lisp
(query (:select 'timestamp-with-time-zone
        :from 'test-data
        :where (:< 'id 3))
  :json-strs)

'("{\"timestampWithTimeZone\":\"2019-12-30 18:30:54:0\"}"
  "{\"timestampWithTimeZone\":\"1919-12-30 18:30:54:0\"}")
#+END_SRC
To get back to the default cl-postgres reader:
#+BEGIN_SRC lisp
(setf cl-postgres:*sql-readtable*
        (cl-postgres:copy-sql-readtable
          cl-postgres::*default-sql-readtable*))
#+END_SRC
However [[http://marijnhaverbeke.nl/postmodern/simple-date.html][Simple-date]] has no concept of time zones. Many users use another library,
[[https://github.com/dlowe-net/local-time][local-time]], which solves the same problem as simple-date, but does understand
time zones. One thing to remember is that PostgreSQL doesn't store timezone
information when using `timestamp with time zone`. Time zone information only
used to convert it to proper UTC timestamp.

*** Local-Time Library Use
For those who want to use local-time, to enable the local-time reader:
#+BEGIN_SRC lisp
(ql:quickload :cl-postgres+local-time)
(local-time:set-local-time-cl-postgres-readers)
#+END_SRC

With that set postgresql time datatype returns look like:
With local-time loaded and local-time:set-local-time-cl-postgres-readers run,
the same sample data looks like:

| PG Type                    | Sample Return Value              | Lisp Type             |
|----------------------------+----------------------------------+-----------------------|
| date                       | 2020-05-15T20:00:00.000000-04:00 | TIMESTAMP             |
| time_without_timezone      | 2000-03-01T04:47:09.926531-05:00 | TIMESTAMP             |
| time_with_timezone         | 09:47:16.510459-04               | (VECTOR CHARACTER 64) |
| timestamp_without_timezone | 2020-05-16T05:47:33.315622-04:00 | TIMESTAMP             |
| timestamp_with_timezone    | 2020-05-16T09:47:27.855146-04:00 | TIMESTAMP             |

Similarly to simple-date timestamps, these can be exported as json-strings with the encode-json-to-string function. E.g.
#+BEGIN_SRC lisp
(encode-json-to-string (query (:select 'timestamp-with-time-zone
                               :from 'test-data
                               :where (:= 'id 1))
                               :single))
"\"{2020-12-30T08:30:54.000000-05:00}\""
#+END_SRC
Or more simply with passing one of the json result type parameters to the query
function. E.g.
#+BEGIN_SRC lisp
(query (:select 'timestamp-with-time-zone
        :from 'test-data
        :where (:< 'id 3))
  :json-strs)

'("{\"timestampWithTimeZone\":\"{2019-12-30T13:30:54.000000-05:00}\"}"
  "{\"timestampWithTimeZone\":\"{1919-12-30T13:30:54.000000-05:00}\"}")
#+END_SRC
** Portability
   :PROPERTIES:
   :ID:       769a0e88-de54-4356-a474-72708accbafb
   :CUSTOM_ID: bdf9ddb0-5f95-4807-8862-8970b35bd142
   :END:
The Lisp code in Postmodern is theoretically portable across implementations,
and seems to work on all major ones as well as some minor ones such as Genera.
It is regularly tested on ccl, sbcl, ecl, abcl and cmucl.

ABCL version 1.8.0 broke the dao class inheritance. See [[https://abcl.org/trac/ticket/479]].
Everything other than dao-classes works.

Clisp currently has issues with executing a file of sql statements (Postmodern's execute-file function).

Implementations that do not have meta-object protocol support will not have
DAOs, but all other parts of the library should work (all widely used
implementations do support this).

The library is not likely to work for PostgreSQL versions older than 8.4.
Other features only work in newer Postgresql versions as the features
were only introduced in those newer versions.

** Reserved Words
   :PROPERTIES:
   :ID:       671c5e6a-f548-4791-86a5-575b3fcc0aa5
   :CUSTOM_ID: 565fad0b-aef4-497c-92d8-096a0a78c804
   :END:
It is highly suggested that you do not use words that are reserved by Postgresql
and the sql standard as identifiers (e.g. table names, columns). The reserved
words are:

"all" "analyse" "analyze" "and" "any" "array" "as" "asc" "asymmetric"
"authorization" "between" "binary" "both" "case" "cast" "check" "collate"
"column" "concurrently" "constraint" "create" "cross" "current-catalog"
"current-date" "current-role" "current-schema" "current-time"
"current-timestamp" "current-user" "default" "deferrable" "desc" "distinct" "do"
"else" "end" "except" "false" "fetch" "filter" "for" "foreign" "freeze" "from"
"full" "grant" "group" "having" "ilike" "in" "initially" "inner" "intersect"
"into" "is" "isnull" "join" "lateral" "leading" "left" "like" "limit"
"localtime" "localtimestamp" "natural" "new" "not" "notnull"  "nowait" "null"
"off" "offset" "old" "on" "only" "or" "order" "outer" "overlaps" "placing"
"primary" "references" "returning" "right" "select" "session-user" "share"
"similar" "some" "symmetric" "table" "then" "to" "trailing" "true" "union"
"unique" "user" "using" "variadic" "verbose" "when" "where" "window" "with"

** Things that should be implemented
   :PROPERTIES:
   :ID:       50d91126-93f1-4f50-96ad-bd63a7278866
   :CUSTOM_ID: 4ac8f5f4-d3b0-41c5-a222-fe3086049279
   :END:
Postmodern is under active development so Issues and feature requests should
be flagged on [[https://github.com/marijnh/Postmodern][Postmodern's site on github]].

Some areas that are currently under consideration can be found in the ROADMAP.md
file.

* Resources
  :PROPERTIES:
  :ID:       eb969965-5221-48f8-bb79-5a93fe451454
  :CUSTOM_ID: 773ee4ef-a685-484e-bc6a-6daa849a7d04
  :END:
- [[https://mailman.common-lisp.net/listinfo/postmodern-devel][Mailing List]]
- [[https://sites.google.com/site/sabraonthehill/postmodern-examples][A collection of Postmodern examples]]
- [[http://www.postgresql.org/docs/current/static/index.html][The PostgreSQL manuals]]
- [[http://www.postgresql.org/docs/current/static/protocol.html][The wire protocol Postmodern uses]]
- [[http://clsql.b9.com/][CLSQL]]
- [[https://github.com/filonenko-mikhail/cl-ewkb][Common Lisp Postgis library]]
- [[http://common-lisp.net/project/local-time/][Local-time]]

* Running tests
  :PROPERTIES:
  :ID:       bd354217-9828-444b-afbf-41e9f0d522ca
  :CUSTOM_ID: 844add79-070b-4e89-8797-3bc21ea3173b
  :END:

Postmodern uses [[https://github.com/sionescu/fiveam][FiveAM]] for
testing.  The different component systems of Postmodern have tests
defined in corresponding test systems, each defining a test suite.
The test systems and corresponding top-level test suites are:

- `:postmodern` in `postmodern/tests`,
- `:cl-postgres` in `cl-postgres/tests`,
- `:s-sql` in `s-sql/tests`, and
- `:simple-date` in `simple-date/tests`.

Before running the tests make sure PostgreSQL is running and a test
database is created.  By default tests use the following connection
parameters to run the tests:

- Database name: test
- User: test
- Password: <empty>
- Hostname: localhost
- Port: 5432
- Use-SSL :NO

If connection with these parameters fails then you will be asked to
provide the connection parameters interactively.  The parameters will
be stored in `cl-postgres-tests:*test-connection*` variable and
automatically used on successive test runs.  This variable can also be
set manually before running the tests.

To test a particular component one would first load the corresponding
test system, and then run the test suite.  For example, to test the
`postmodern` system in the REPL one would do the following:
#+BEGIN_SRC lisp
(ql:quickload "postmodern/tests")
(5am:run! :postmodern)

    ;; ... test output ...
#+END_SRC


It is also possible to test multiple components at once by first
loading test systems and then running all tests:
#+BEGIN_SRC lisp
(ql:quickload '("cl-postgres/tests" "s-sql/tests"))
(5am:run-all-tests)

    ;; ... test output ...
#+END_SRC

To run the tests from command-line specify the same forms using your
implementation's command-line syntax.  For instance, to test all
Postmodern components on SBCL, use the following command:

    env DB_USER=$USER sbcl --noinform \
        --eval '(ql:quickload "postmodern/tests")' \
        --eval '(ql:quickload "cl-postgres/tests")' \
        --eval '(ql:quickload "s-sql/tests")' \
        --eval '(ql:quickload "simple-date/tests")' \
        --eval '(progn (setq 5am:*print-names* nil) (5am:run-all-tests))' \
        --eval '(sb-ext:exit)'

As you can see from above, database connection parameters can be
provided using environment variables:

- `DB_NAME`: database name,
- `DB_USER`: user,
- `DB_PASS`: password,
- `DB_HOST`: hostname.
