pgloader - Online in the Cloud

PROGRAM:

NAME

pgloader - PostgreSQL data loader

SYNOPSIS

pgloader [<options>] [<command-file>]...
pgloader [<options>] SOURCE TARGET

DESCRIPTION

pgloader loads data from various sources into PostgreSQL. It can transform the data it
reads on the fly and submit raw SQL before and after the loading. It uses the COPY
PostgreSQL protocol to stream the data into the server, and manages errors by filling a
pair of reject.dat and reject.log files.

pgloader operates either using commands which are read from files:

pgloader commands.load

or by using arguments and options all provided on the command line:

pgloader SOURCE TARGET

ARGUMENTS

The pgloader arguments can be as many load files as needed, or a couple of connection
strings to a specific input file.

SOURCE CONNECTION STRING
The source connection string format is as follows:

format:///absolute/path/to/file.ext
format://./relative/path/to/file.ext

Where format might be one of csv, fixed, copy, dbf, db3 or ixf.

db://user:pass@host:port/dbname

Where db might be of sqlite, mysql or mssql.

When using a file based source format, pgloader also support natively fetching the file
from an http location and decompressing an archive if needed. In that case it´s necessary
to use the --type option to specify the expected format of the file. See the examples
below.

Also note that some file formats require describing some implementation details such as
columns to be read and delimiters and quoting when loading from csv.

For more complex loading scenarios, you will need to write a full fledge load command in
the syntax described later in this document.

TARGET CONNECTION STRING
The target connection string format is described in details later in this document, see
Section Connection String.

OPTIONS

INQUIRY OPTIONS
Use these options when you want to know more about how to use pgloader, as those options
will cause pgloader not to load any data.

-h, --help
Show command usage summary and exit.

-V, --version
Show pgloader version string and exit.

-E, --list-encodings
List known encodings in this version of pgloader.

-U, --upgrade-config
Parse given files in the command line as pgloader.conf files with the INI syntax
that was in use in pgloader versions 2.x, and output the new command syntax for
pgloader on standard output.

GENERAL OPTIONS
Those options are meant to tweak pgloader behavior when loading data.

-v, --verbose
Be verbose.

-q, --quiet
Be quiet.

-d, --debug
Show debug level information messages.

-D, --root-dir
Set the root working directory (default to "/tmp/pgloader").

-L, --logfile
Set the pgloader log file (default to "/tmp/pgloader.log").

--log-min-messages
Minimum level of verbosity needed for log message to make it to the logfile. One of
critical, log, error, warning, notice, info or debug.

--client-min-messages
Minimum level of verbosity needed for log message to make it to the console. One of
critical, log, error, warning, notice, info or debug.

-S, --summary
A filename where to copy the summary output. When relative, the filename is
expanded into *root-dir*.

The format of the filename defaults to being human readable. It is possible to have
the output in machine friendly formats such as CSV, COPY (PostgreSQL´s own COPY
format) or JSON by specifying a filename with the extension resp. .csv, .copy or
.json.

-l <file>, --load-lisp-file <file>
Specify a lisp file to compile and load into the pgloader image before reading the
commands, allowing to define extra transformation function. Those functions should
be defined in the pgloader.transforms package. This option can appear more than
once in the command line.

--self-upgrade <directory>:

Specify a directory where to find pgloader sources so that one of the very first
things it does is dynamically loading-in (and compiling to machine code) another
version of itself, usually a newer one like a very recent git checkout.

COMMAND LINE ONLY OPERATIONS
Those options are meant to be used when using pgloader from the command line only, rather
than using a command file and the rich command clauses and parser. In simple cases, it can
be much easier to use the SOURCE and TARGET directly on the command line, then tweak the
loading with those options:

· --with "option":

Allows setting options from the command line. You can use that option as many times as
you want. The option arguments must follow the WITH clause for the source type of the
SOURCE specification, as described later in this document.

· --set "guc_name=´value´"

Allows setting PostgreSQL configuration from the command line. Note that the option
parsing is the same as when used from the SET command clause, in particular you must
enclose the guc value with single-quotes.

· --field "..."

Allows setting a source field definition. Fields are accumulated in the order given on
the command line. It´s possible to either use a --field option per field in the source
file, or to separate field definitions by a comma, as you would do in the HAVING
FIELDS clause.

· --cast "..."

Allows setting a specific casting rule for loading the data.

· --type csv|fixed|db3|ixf|sqlite|mysql|mssql

Allows forcing the source type, in case when the SOURCE parsing isn´t satisfying.

· --encoding <encoding>

Set the encoding of the source file to load data from.

· --before <filename>

Parse given filename for SQL queries and run them against the target database before
loading the data from the source. The queries are parsed by pgloader itself: they need
to be terminated by a semi-colon (;) and the file may include \i or \ir commands to
include another file.

· --after <filename>

Parse given filename for SQL queries and run them against the target database after
having loaded the data from the source. The queries are parsed in the same way as with
the --before option, see above.

MORE DEBUG INFORMATION
To get the maximum amount of debug information, you can use both the --verbose and the
--debug switches at the same time, which is equivalent to saying --client-min-messages
data. Then the log messages will show the data being processed, in the cases where the
code has explicit support for it.

USAGE EXAMPLES

Review the command line options and pgloader´s version:

pgloader --help
pgloader --version

Loading from a complex command
Use the command file as the pgloader command argument, pgloader will parse that file and
execute the commands found in it:

pgloader --verbose ./test/csv-districts.load

CSV
Load data from a CSV file into a pre-existing table in your database:

pgloader --type csv \
--field id --field field \
--with truncate \
--with "fields terminated by ´,´" \
./test/data/matching-1.csv \
postgres:///pgloader?tablename=matching

In that example the whole loading is driven from the command line, bypassing the need for
writing a command in the pgloader command syntax entirely. As there´s no command though,
the extra inforamtion needed must be provided on the command line using the --type and
--field and --with switches.

For documentation about the available syntaxes for the --field and --with switches, please
refer to the CSV section later in the man page.

Note also that the PostgreSQL URI includes the target tablename.

Reading from STDIN
File based pgloader sources can be loaded from the standard input, as in the following
example:

pgloader --type csv \
--field "usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong" \
--with "skip header = 1" \
--with "fields terminated by ´\t´" \
- \
postgresql:///pgloader?districts_longlat \
< test/data/2013_Gaz_113CDs_national.txt

The dash (-) character as a source is used to mean standard input, as usual in Unix
command lines. It´s possible to stream compressed content to pgloader with this technique,
using the Unix pipe:

gunzip -c source.gz | pgloader --type csv ... - pgsql:///target?foo

Loading from CSV available through HTTP
The same command as just above can also be run if the CSV file happens to be found on a
remote HTTP location:

pgloader --type csv \
--field "usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong" \
--with "skip header = 1" \
--with "fields terminated by ´\t´" \
http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt \
postgresql:///pgloader?districts_longlat

Some more options have to be used in that case, as the file contains a one-line header
(most commonly that´s column names, could be a copyright notice). Also, in that case, we
specify all the fields right into a single --field option argument.

Again, the PostgreSQL target connection string must contain the tablename option and you
have to ensure that the target table exists and may fit the data. Here´s the SQL command
used in that example in case you want to try it yourself:

create table districts_longlat
(
usps text,
geoid text,
aland bigint,
awater bigint,
aland_sqmi double precision,
awater_sqmi double precision,
intptlat double precision,
intptlong double precision
);

Also notice that the same command will work against an archived version of the same data,
e.g. http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt.gz.

Finally, it´s important to note that pgloader first fetches the content from the HTTP URL
it to a local file, then expand the archive when it´s recognized to be one, and only then
processes the locally expanded file.

In some cases, either because pgloader has no direct support for your archive format or
maybe because expanding the archive is not feasible in your environment, you might want to
stream the content straight from its remote location into PostgreSQL. Here´s how to do
that, using the old battle tested Unix Pipes trick:

curl http://pgsql.tapoueh.org/temp/2013_Gaz_113CDs_national.txt.gz \
| gunzip -c \
| pgloader --type csv \
--field "usps,geoid,aland,awater,aland_sqmi,awater_sqmi,intptlat,intptlong"
--with "skip header = 1" \
--with "fields terminated by ´\t´" \
- \
postgresql:///pgloader?districts_longlat

Now the OS will take care of the streaming and buffering between the network and the
commands and pgloader will take care of streaming the data down to PostgreSQL.

Migrating from SQLite
The following command will open the SQLite database, discover its tables definitions
including indexes and foreign keys, migrate those definitions while casting the data type
specifications to their PostgreSQL equivalent and then migrate the data over:

createdb newdb
pgloader ./test/sqlite/sqlite.db postgresql:///newdb

Migrating from MySQL
Just create a database where to host the MySQL data and definitions and have pgloader do
the migration for you in a single command line:

createdb pagila
pgloader mysql://user@localhost/sakila postgresql:///pagila

Fetching an archived DBF file from a HTTP remote location
It´s possible for pgloader to download a file from HTTP, unarchive it, and only then open
it to discover the schema then load the data:

createdb foo
pgloader --type dbf http://www.insee.fr/fr/methodes/nomenclatures/cog/telechargement/2013/dbf/historiq2013.zip postgresql:///foo

Here it´s not possible for pgloader to guess the kind of data source it´s being given, so
it´s necessary to use the --type command line switch.

BATCHES AND RETRY BEHAVIOUR

To load data to PostgreSQL, pgloader uses the COPY streaming protocol. While this is the
faster way to load data, COPY has an important drawback: as soon as PostgreSQL emits an
error with any bit of data sent to it, whatever the problem is, the whole data set is
rejected by PostgreSQL.

To work around that, pgloader cuts the data into batches of 25000 rows each, so that when
a problem occurs it´s only impacting that many rows of data. Each batch is kept in memory
while the COPY streaming happens, in order to be able to handle errors should some happen.

When PostgreSQL rejects the whole batch, pgloader logs the error message then isolates the
bad row(s) from the accepted ones by retrying the batched rows in smaller batches. To do
that, pgloader parses the CONTEXT error message from the failed COPY, as the message
contains the line number where the error was found in the batch, as in the following
example:

CONTEXT: COPY errors, line 3, column b: "2006-13-11"

Using that information, pgloader will reload all rows in the batch before the erroneous
one, log the erroneous one as rejected, then try loading the remaining of the batch in a
single attempt, which may or may not contain other erroneous data.

At the end of a load containing rejected rows, you will find two files in the root-dir
location, under a directory named the same as the target database of your setup. The
filenames are the target table, and their extensions are .dat for the rejected data and
.log for the file containing the full PostgreSQL client side logs about the rejected data.

The .dat file is formatted in PostgreSQL the text COPY format as documented in
http://www.postgresql.org/docs/9.2/static/sql-copy.html#AEN66609 .

A NOTE ABOUT PERFORMANCES

pgloader has been developed with performances in mind, to be able to cope with ever
growing needs in loading large amounts of data into PostgreSQL.

The basic architecture it uses is the old Unix pipe model, where a thread is responsible
for loading the data (reading a CSV file, querying MySQL, etc) and fills pre-processed
data into a queue. Another threads feeds from the queue, apply some more transformations
to the input data and stream the end result to PostgreSQL using the COPY protocol.

When given a file that the PostgreSQL COPY command knows how to parse, and if the file
contains no erroneous data, then pgloader will never be as fast as just using the
PostgreSQL COPY command.

Note that while the COPY command is restricted to read either from its standard input or
from a local file on the server´s file system, the command line tool psql implements a
\copy command that knows how to stream a file local to the client over the network and
into the PostgreSQL server, using the same protocol as pgloader uses.

SOURCE FORMATS

pgloader supports the following input formats:

· csv, which includes also tsv and other common variants where you can change the
separator and the quoting rules and how to escape the quotes themselves;

· fixed columns file, where pgloader is flexible enough to accomodate with source files
missing columns (ragged fixed length column files do exist);

· PostgreSLQ COPY formatted files, following the COPY TEXT documentation of PostgreSQL,
such as the reject files prepared by pgloader;

· dbase files known as db3 or dbf file;

· ixf formated files, ixf being a binary storage format from IBM;

· sqlite databases with fully automated discovery of the schema and advanced cast rules;

· mysql databases with fully automated discovery of the schema and advanced cast rules;

· MS SQL databases with fully automated discovery of the schema and advanced cast rules.

PGLOADER COMMANDS SYNTAX

pgloader implements a Domain Specific Language allowing to setup complex data loading
scripts handling computed columns and on-the-fly sanitization of the input data. For more
complex data loading scenarios, you will be required to learn that DSL´s syntax. It´s
meant to look familiar to DBA by being inspired by SQL where it makes sense, which is not
that much after all.

The pgloader commands follow the same global grammar rules. Each of them might support
only a subset of the general options and provide specific options.

LOAD <source-type>
FROM <source-url> [ HAVING FIELDS <source-level-options> ]
INTO <postgresql-url> [ TARGET COLUMNS <columns-and-options> ]

[ WITH <load-options> ]

[ SET <postgresql-settings> ]

[ BEFORE LOAD [ DO <sql statements> | EXECUTE <sql file> ] ... ]
[ AFTER LOAD [ DO <sql statements> | EXECUTE <sql file> ] ... ]
;

The main clauses are the LOAD, FROM, INTO and WITH clauses that each command implements.
Some command then implement the SET command, or some specific clauses such as the CAST
clause.

COMMON CLAUSES

Some clauses are common to all commands:

· FROM

The FROM clause specifies where to read the data from, and each command introduces its
own variant of sources. For instance, the CSV source supports inline, stdin, a
filename, a quoted filename, and a FILENAME MATCHING clause (see above); whereas the
MySQL source only supports a MySQL database URI specification.

In all cases, the FROM clause is able to read its value from an environment variable
when using the form GETENV ´varname´.

· INTO

The PostgreSQL connection URI must contains the name of the target table where to load
the data into. That table must have already been created in PostgreSQL, and the name
might be schema qualified.

The INTO target database connection URI can be parsed from the value of an environment
variable when using the form GETENV ´varname´.

Then INTO option also supports an optional comma separated list of target columns,
which are either the name of an input field or the white space separated list of the
target column name, its PostgreSQL data type and a USING expression.

The USING expression can be any valid Common Lisp form and will be read with the
current package set to pgloader.transforms, so that you can use functions defined in
that package, such as functions loaded dynamically with the --load command line
parameter.

Each USING expression is compiled at runtime to native code.

This feature allows pgloader to load any number of fields in a CSV file into a
possibly different number of columns in the database, using custom code for that
projection.

· WITH

Set of options to apply to the command, using a global syntax of either:

· key = value

· use option

· do not use option

See each specific command for details.

· SET

This clause allows to specify session parameters to be set for all the sessions opened
by pgloader. It expects a list of parameter name, the equal sign, then the
single-quoted value as a comma separated list.

The names and values of the parameters are not validated by pgloader, they are given
as-is to PostgreSQL.

· BEFORE LOAD DO

You can run SQL queries against the database before loading the data from the CSV
file. Most common SQL queries are CREATE TABLE IF NOT EXISTS so that the data can be
loaded.

Each command must be dollar-quoted: it must begin and end with a double dollar sign,
$$. Dollar-quoted queries are then comma separated. No extra punctuation is expected
after the last SQL query.

· BEFORE LOAD EXECUTE

Same behaviour as in the BEFORE LOAD DO clause. Allows you to read the SQL queries
from a SQL file. Implements support for PostgreSQL dollar-quoting and the \i and \ir
include facilities as in psql batch mode (where they are the same thing).

· AFTER LOAD DO

Same format as BEFORE LOAD DO, the dollar-quoted queries found in that section are
executed once the load is done. That´s the right time to create indexes and
constraints, or re-enable triggers.

· AFTER LOAD EXECUTE

Same behaviour as in the AFTER LOAD DO clause. Allows you to read the SQL queries from
a SQL file. Implements support for PostgreSQL dollar-quoting and the \i and \ir
include facilities as in psql batch mode (where they are the same thing).

Connection String
The <postgresql-url> parameter is expected to be given as a Connection URI as documented
in the PostgreSQL documentation at
http://www.postgresql.org/docs/9.3/static/libpq-connect.html#LIBPQ-CONNSTRING.

postgresql://[user[:password]@][netloc][:port][/dbname][?option=value&...]

Where:

· user

Can contain any character, including colon (:) which must then be doubled (::) and
at-sign (@) which must then be doubled (@@).

When omitted, the user name defaults to the value of the PGUSER environment variable,
and if it is unset, the value of the USER environment variable.

· password

Can contain any character, including the at sign (@) which must then be doubled (@@).
To leave the password empty, when the user name ends with at at sign, you then have to
use the syntax user:@.

When omitted, the password defaults to the value of the PGPASSWORD environment
variable if it is set, otherwise the password is left unset.

· netloc

Can be either a hostname in dotted notation, or an ipv4, or an Unix domain socket
path. Empty is the default network location, under a system providing unix domain
socket that method is preferred, otherwise the netloc default to localhost.

It´s possible to force the unix domain socket path by using the syntax
unix:/path/to/where/the/socket/file/is, so to force a non default socket path and a
non default port, you would have:

postgresql://unix:/tmp:54321/dbname

The netloc defaults to the value of the PGHOST environment variable, and if it is unset,
to either the default unix socket path when running on a Unix system, and localhost
otherwise.

· dbname

Should be a proper identifier (letter followed by a mix of letters, digits and the
punctuation signs comma (,), dash (-) and underscore (_).

When omitted, the dbname defaults to the value of the environment variable PGDATABASE,
and if that is unset, to the user value as determined above.

· options

The optional parameters must be supplied with the form name=value, and you may use
several parameters by separating them away using an ampersand (&) character.

Only some options are supported here, tablename (which might be qualified with a
schema name) sslmode, host, port, dbname, user and password.

The sslmode parameter values can be one of disable, allow, prefer or require.

For backward compatibility reasons, it´s possible to specify the tablename option
directly, without spelling out the tablename= parts.

The options override the main URI components when both are given, and using the
percent-encoded option parameters allow using passwords starting with a colon and
bypassing other URI components parsing limitations.

Regular Expressions
Several clauses listed in the following accept regular expressions with the following
input rules:

· A regular expression begins with a tilde sign (~),

· is then followed with an opening sign,

· then any character is allowed and considered part of the regular expression, except
for the closing sign,

· then a closing sign is expected.

The opening and closing sign are allowed by pair, here´s the complete list of allowed
delimiters:

~//
~[]
~{}
~()
~<>
~""
~´´
~||
~##

Pick the set of delimiters that don´t collide with the regular expression you´re trying to
input. If your expression is such that none of the solutions allow you to enter it, the
places where such expressions are allowed should allow for a list of expressions.

Comments
Any command may contain comments, following those input rules:

· the -- delimiter begins a comment that ends with the end of the current line,

· the delimiters /* and */ respectively start and end a comment, which can be found in
the middle of a command or span several lines.

Any place where you could enter a whitespace will accept a comment too.

Batch behaviour options
All pgloader commands have support for a WITH clause that allows for specifying options.
Some options are generic and accepted by all commands, such as the batch behaviour
options, and some options are specific to a data source kind, such as the CSV skip header
option.

The global batch behaviour options are:

· batch rows

Takes a numeric value as argument, used as the maximum number of rows allowed in a
batch. The default is 25 000 and can be changed to try having better performances
characteristics or to control pgloader memory usage;

· batch size

Takes a memory unit as argument, such as 20 MB, its default value. Accepted
multipliers are kB, MB, GB, TB and PB. The case is important so as not to be confused
about bits versus bytes, we´re only talking bytes here.

· batch concurrency

Takes a numeric value as argument, defaults to 10. That´s the number of batches that
pgloader is allows to build in memory, even when only a single batch at a time might
be sent to PostgreSQL.

Supporting more than a single batch being sent at a time is on the TODO list of
pgloader, but is not implemented yet. This option is about controlling the memory
needs of pgloader as a trade-off to the performances characteristics, and not about
parallel activity of pgloader.

Other options are specific to each input source, please refer to specific parts of the
documentation for their listing and covering.

A batch is then closed as soon as either the batch rows or the batch size threshold is
crossed, whichever comes first. In cases when a batch has to be closed because of the
batch size setting, a debug level log message is printed with how many rows did fit in the
oversized batch.

LOAD CSV

This command instructs pgloader to load data from a CSV file. Here´s an example:

LOAD CSV
FROM ´GeoLiteCity-Blocks.csv´ WITH ENCODING iso-646-us
HAVING FIELDS
(
startIpNum, endIpNum, locId
)
INTO postgresql://user@localhost:54393/dbname?geolite.blocks
TARGET COLUMNS
(
iprange ip4r using (ip-range startIpNum endIpNum),
locId
)
WITH truncate,
skip header = 2,
fields optionally enclosed by ´"´,
fields escaped by backslash-quote,
fields terminated by ´\t´

SET work_mem to ´32 MB´, maintenance_work_mem to ´64 MB´;

The csv format command accepts the following clauses and options:

· FROM

Filename where to load the data from. Accepts an ENCODING option. Use the
--list-encodings option to know which encoding names are supported.

The filename may be enclosed by single quotes, and could be one of the following
special values:

· inline

The data is found after the end of the parsed commands. Any number of empty lines
between the end of the commands and the beginning of the data is accepted.

· stdin

Reads the data from the standard input stream.

· FILENAMES MATCHING

The whole matching clause must follow the following rule:

[ ALL FILENAMES | [ FIRST ] FILENAME ]
MATCHING regexp
[ IN DIRECTORY ´...´ ]

The matching clause applies given regular expression (see above for exact syntax, several
options can be used here) to filenames. It´s then possible to load data from only the
first match of all of them.

The optional IN DIRECTORY clause allows specifying which directory to walk for finding the
data files, and can be either relative to where the command file is read from, or
absolute. The given directory must exists.

The FROM option also supports an optional comma separated list of field names describing
what is expected in the CSV data file, optionally introduced by the clause HAVING FIELDS.

Each field name can be either only one name or a name following with specific reader
options for that field, enclosed in square brackets and comma-separated. Supported
per-field reader options are:

· terminated by

See the description of field terminated by below.

The processing of this option is not currently implemented.

· date format

When the field is expected of the date type, then this option allows to specify the
date format used in the file.

Date format string are template strings modeled against the PostgreSQL to_char
template strings support, limited to the following patterns:

· YYYY, YYY, YY for the year part

· MM for the numeric month part

· DD for the numeric day part

· HH, HH12, HH24 for the hour part

· am, AM, a.m., A.M.

· pm, PM, p.m., P.M.

· MI for the minutes part

· SS for the seconds part

· MS for the milliseconds part (4 digits)

· US for the microseconds part (6 digits)

· unparsed punctuation signs: - . * # @ T / \ and space

Here´s an example of a date format specification:

column-name [date format ´YYYY-MM-DD HH24-MI-SS.US´]

· null if

This option takes an argument which is either the keyword blanks or a double-quoted
string.

When blanks is used and the field value that is read contains only space characters,
then it´s automatically converted to an SQL NULL value.

When a double-quoted string is used and that string is read as the field value, then
the field value is automatically converted to an SQL NULL value.

· trim both whitespace, trim left whitespace, trim right whitespace

This option allows to trim whitespaces in the read data, either from both sides of the
data, or only the whitespace characters found on the left of the streaing, or only
those on the right of the string.

· WITH

When loading from a CSV file, the following options are supported:

· truncate

When this option is listed, pgloader issues a TRUNCATE command against the PostgreSQL
target table before reading the data file.

· drop indexes

When this option is listed, pgloader issues DROP INDEX commands against all the
indexes defined on the target table before copying the data, then CREATE INDEX
commands once the COPY is done.

In order to get the best performances possible, all the indexes are created in
parallel and when done the primary keys are built again from the unique indexes just
created. This two step process allows creating the primary key index in parallel with
the other indexes, as only the ALTER TABLE command needs an access exclusive lock on
the target table.

· disable triggers

When this option is listed, pgloader issues an ALTER TABLE ... DISABLE TRIGGER ALL
command against the PostgreSQL target table before copying the data, then the command
ALTER TABLE ... ENABLE TRIGGER ALL once the COPY is done.

This option allows loading data into a pre-existing table ignoring the foreign key
constraints and user defined triggers and may result in invalid foreign key
constraints once the data is loaded. Use with care.

· skip header

Takes a numeric value as argument. Instruct pgloader to skip that many lines at the
beginning of the input file.

· csv header

Use the first line read after skip header as the list of csv field names to be found
in the CSV file, using the same CSV parameters as for the CSV data.

· trim unquoted blanks

When reading unquoted values in the CSV file, remove the blanks found in between the
separator and the value. That behaviour is the default.

· keep unquoted blanks

When reading unquoted values in the CSV file, keep blanks found in between the
separator and the value.

· fields optionally enclosed by

Takes a single character as argument, which must be found inside single quotes, and
might be given as the printable character itself, the special value \t to denote a
tabulation character, or 0x then an hexadecimal value read as the ASCII code for the
character.

This character is used as the quoting character in the CSV file, and defaults to
double-quote.

· fields not enclosed

By default, pgloader will use the double-quote character as the enclosing character.
If you have a CSV file where fields are not enclosed and are using double-quote as an
expected ordinary character, then use the option fields not enclosed for the CSV
parser to accept those values.

· fields escaped by

Takes either the special value backslash-quote or double-quote, or any value supported
by the fields terminated by option (see below). This value is used to recognize
escaped field separators when they are to be found within the data fields themselves.
Defaults to double-quote.

· csv escape mode

Takes either the special value quote (the default) or following and allows the CSV
parser to parse either only escaped field separator or any character (including CSV
data) when using the following value.

· fields terminated by

Takes a single character as argument, which must be found inside single quotes, and
might be given as the printable character itself, the special value \t to denote a
tabulation character, or 0x then an hexadecimal value read as the ASCII code for the
character.

This character is used as the field separator when reading the CSV data.

· lines terminated by

Takes a single character as argument, which must be found inside single quotes, and
might be given as the printable character itself, the special value \t to denote a
tabulation character, or 0x then an hexadecimal value read as the ASCII code for the
character.

This character is used to recognize end-of-line condition when reading the CSV data.

LOAD FIXED COLS

This command instructs pgloader to load data from a text file containing columns arranged
in a fixed size manner. Here´s an example:

LOAD FIXED
FROM inline
(
a from 0 for 10,
b from 10 for 8,
c from 18 for 8,
d from 26 for 17 [null if blanks, trim right whitespace]
)
INTO postgresql:///pgloader?fixed
(
a, b,
c time using (time-with-no-separator c),
d
)

WITH truncate

SET client_encoding to ´latin1´,
work_mem to ´14MB´,
standard_conforming_strings to ´on´

BEFORE LOAD DO
$$ drop table if exists fixed; $$,
$$ create table fixed (
a integer,
b date,
c time,
d text
);
$$;

01234567892008052011431250firstline
01234562008052115182300left blank-padded
12345678902008052208231560another line
2345609872014092914371500
2345678902014092914371520

The fixed format command accepts the following clauses and options:

· FROM

Filename where to load the data from. Accepts an ENCODING option. Use the
--list-encodings option to know which encoding names are supported.

The filename may be enclosed by single quotes, and could be one of the following
special values:

· inline

The data is found after the end of the parsed commands. Any number of empty lines
between the end of the commands and the beginning of the data is accepted.

· stdin

Reads the data from the standard input stream.

The FROM option also supports an optional comma separated list of field names describing
what is expected in the FIXED data file.

Each field name is composed of the field name followed with specific reader options for
that field. Supported per-field reader options are the following, where only start and
length are required.

· start

Position in the line where to start reading that field´s value. Can be entered with
decimal digits or 0x then hexadecimal digits.

· length

How many bytes to read from the start position to read that field´s value. Same format
as start.

Those optional parameters must be enclosed in square brackets and comma-separated:

· terminated by

See the description of field terminated by below.

The processing of this option is not currently implemented.

· date format

When the field is expected of the date type, then this option allows to specify the
date format used in the file.

Date format string are template strings modeled against the PostgreSQL to_char
template strings support, limited to the following patterns:

· YYYY, YYY, YY for the year part

· MM for the numeric month part

· DD for the numeric day part

· HH, HH12, HH24 for the hour part

· am, AM, a.m., A.M.

· pm, PM, p.m., P.M.

· MI for the minutes part

· SS for the seconds part

· MS for the milliseconds part (4 digits)

· US for the microseconds part (6 digits)

· unparsed punctuation signs: - . * # @ T / \ and space

Here´s an example of a date format specification:

column-name [date format ´YYYY-MM-DD HH24-MI-SS.US´]

· null if

This option takes an argument which is either the keyword blanks or a double-quoted
string.

When blanks is used and the field value that is read contains only space characters,
then it´s automatically converted to an SQL NULL value.

When a double-quoted string is used and that string is read as the field value, then
the field value is automatically converted to an SQL NULL value.

· trim both whitespace, trim left whitespace, trim right whitespace

This option allows to trim whitespaces in the read data, either from both sides of the
data, or only the whitespace characters found on the left of the streaing, or only
those on the right of the string.

· WITH

When loading from a FIXED file, the following options are supported:

· truncate

When this option is listed, pgloader issues a TRUNCATE command against the PostgreSQL
target table before reading the data file.

· disable triggers

When this option is listed, pgloader issues an ALTER TABLE ... DISABLE TRIGGER ALL
command against the PostgreSQL target table before copying the data, then the command
ALTER TABLE ... ENABLE TRIGGER ALL once the COPY is done.

This option allows loading data into a pre-existing table ignoring the foreign key
constraints and user defined triggers and may result in invalid foreign key
constraints once the data is loaded. Use with care.

· skip header

Takes a numeric value as argument. Instruct pgloader to skip that many lines at the
beginning of the input file.

LOAD COPY FORMATTED FILES

This commands instructs pgloader to load from a file containing COPY TEXT data as
described in the PostgreSQL documentation. Here´s an example:

LOAD COPY
FROM copy://./data/track.copy
(
trackid, track, album, media, genre, composer,
milliseconds, bytes, unitprice
)
INTO postgresql:///pgloader?track_full

WITH truncate

SET client_encoding to ´latin1´,
work_mem to ´14MB´,
standard_conforming_strings to ´on´

BEFORE LOAD DO
$$ drop table if exists track_full; $$,
$$ create table track_full (
trackid bigserial,
track text,
album text,
media text,
genre text,
composer text,
milliseconds bigint,
bytes bigint,
unitprice numeric
);
$$;

The COPY format command accepts the following clauses and options:

· FROM

Filename where to load the data from. This support local files, HTTP URLs and zip
files containing a single dbf file of the same name. Fetch such a zip file from an
HTTP address is of course supported.

· WITH

When loading from a COPY file, the following options are supported:

· delimiter

Takes a single character as argument, which must be found inside single quotes, and
might be given as the printable character itself, the special value \t to denote a
tabulation character, or 0x then an hexadecimal value read as the ASCII code for the
character.

This character is used as the delimiter when reading the data, in a similar way to the
PostgreSQL COPY option.

· null

Takes a quoted string as an argument (quotes can be either double quotes or single
quotes) and uses that string as the NULL representation in the data.

This is similar to the null COPY option in PostgreSQL.

· truncate

When this option is listed, pgloader issues a TRUNCATE command against the PostgreSQL
target table before reading the data file.

· disable triggers

When this option is listed, pgloader issues an ALTER TABLE ... DISABLE TRIGGER ALL
command against the PostgreSQL target table before copying the data, then the command
ALTER TABLE ... ENABLE TRIGGER ALL once the COPY is done.

This option allows loading data into a pre-existing table ignoring the foreign key
constraints and user defined triggers and may result in invalid foreign key
constraints once the data is loaded. Use with care.

· skip header

Takes a numeric value as argument. Instruct pgloader to skip that many lines at the
beginning of the input file.

LOAD DBF

This command instructs pgloader to load data from a DBF file. Here´s an example:

LOAD DBF
FROM http://www.insee.fr/fr/methodes/nomenclatures/cog/telechargement/2013/dbf/reg2013.dbf
INTO postgresql://user@localhost/dbname
WITH truncate, create table;

The dbf format command accepts the following clauses and options:

· FROM

Filename where to load the data from. This support local files, HTTP URLs and zip
files containing a single dbf file of the same name. Fetch such a zip file from an
HTTP address is of course supported.

· WITH

When loading from a DBF file, the following options are supported:

· truncate

When this option is listed, pgloader issues a TRUNCATE command against the PostgreSQL
target table before reading the data file.

· disable triggers

When this option is listed, pgloader issues an ALTER TABLE ... DISABLE TRIGGER ALL
command against the PostgreSQL target table before copying the data, then the command
ALTER TABLE ... ENABLE TRIGGER ALL once the COPY is done.

This option allows loading data into a pre-existing table ignoring the foreign key
constraints and user defined triggers and may result in invalid foreign key
constraints once the data is loaded. Use with care.

· create table

When this option is listed, pgloader creates the table using the meta data found in
the DBF file, which must contain a list of fields with their data type. A standard
data type conversion from DBF to PostgreSQL is done.

· table name

This options expects as its value the possibly qualified name of the table to create.

LOAD IXF

This command instructs pgloader to load data from an IBM IXF file. Here´s an example:

LOAD IXF
FROM data/nsitra.test1.ixf
INTO postgresql:///pgloader?nsitra.test1
WITH truncate, create table

BEFORE LOAD DO
$$ create schema if not exists nsitra; $$,
$$ drop table if exists nsitra.test1; $$;

The ixf format command accepts the following clauses and options:

· FROM

Filename where to load the data from. This support local files, HTTP URLs and zip
files containing a single ixf file of the same name. Fetch such a zip file from an
HTTP address is of course supported.

· WITH

When loading from a IXF file, the following options are supported:

· truncate

When this option is listed, pgloader issues a TRUNCATE command against the PostgreSQL
target table before reading the data file.

· disable triggers

When this option is listed, pgloader issues an ALTER TABLE ... DISABLE TRIGGER ALL
command against the PostgreSQL target table before copying the data, then the command
ALTER TABLE ... ENABLE TRIGGER ALL once the COPY is done.

This option allows loading data into a pre-existing table ignoring the foreign key
constraints and user defined triggers and may result in invalid foreign key
constraints once the data is loaded. Use with care.

· create table

When this option is listed, pgloader creates the table using the meta data found in
the DBF file, which must contain a list of fields with their data type. A standard
data type conversion from DBF to PostgreSQL is done.

· table name

This options expects as its value the possibly qualified name of the table to create.

LOAD ARCHIVE

This command instructs pgloader to load data from one or more files contained in an
archive. Currently the only supported archive format is ZIP, and the archive might be
downloaded from an HTTP URL.

Here´s an example:

LOAD ARCHIVE
FROM /Users/dim/Downloads/GeoLiteCity-latest.zip
INTO postgresql:///ip4r

BEFORE LOAD
DO $$ create extension if not exists ip4r; $$,
$$ create schema if not exists geolite; $$,

EXECUTE ´geolite.sql´

LOAD CSV
FROM FILENAME MATCHING ~/GeoLiteCity-Location.csv/
WITH ENCODING iso-8859-1
(
locId,
country,
region null if blanks,
city null if blanks,
postalCode null if blanks,
latitude,
longitude,
metroCode null if blanks,
areaCode null if blanks
)
INTO postgresql:///ip4r?geolite.location
(
locid,country,region,city,postalCode,
location point using (format nil "(~a,~a)" longitude latitude),
metroCode,areaCode
)
WITH skip header = 2,
fields optionally enclosed by ´"´,
fields escaped by double-quote,
fields terminated by ´,´

AND LOAD CSV
FROM FILENAME MATCHING ~/GeoLiteCity-Blocks.csv/
WITH ENCODING iso-8859-1
(
startIpNum, endIpNum, locId
)
INTO postgresql:///ip4r?geolite.blocks
(
iprange ip4r using (ip-range startIpNum endIpNum),
locId
)
WITH skip header = 2,
fields optionally enclosed by ´"´,
fields escaped by double-quote,
fields terminated by ´,´

FINALLY DO
$$ create index blocks_ip4r_idx on geolite.blocks using gist(iprange); $$;

The archive command accepts the following clauses and options:

· FROM

Filename or HTTP URI where to load the data from. When given an HTTP URL the linked
file will get downloaded locally before processing.

If the file is a zip file, the command line utility unzip is used to expand the
archive into files in $TMPDIR, or /tmp if $TMPDIR is unset or set to a non-existing
directory.

Then the following commands are used from the top level directory where the archive
has been expanded.

· command [ AND command ... ]

A series of commands against the contents of the archive, at the moment only
CSV,´FIXED and DBF commands are supported.

Note that commands are supporting the clause FROM FILENAME MATCHING which allows the
pgloader command not to depend on the exact names of the archive directories.

The same clause can also be applied to several files with using the spelling FROM ALL
FILENAMES MATCHING and a regular expression.

The whole matching clause must follow the following rule:

FROM [ ALL FILENAMES | [ FIRST ] FILENAME ] MATCHING

· FINALLY DO

SQL Queries to run once the data is loaded, such as CREATE INDEX.

LOAD MYSQL DATABASE

This command instructs pgloader to load data from a database connection. The only
supported database source is currently MySQL, and pgloader supports dynamically converting
the schema of the source database and the indexes building.

A default set of casting rules are provided and might be overloaded and appended to by the
command.

Here´s an example:

LOAD DATABASE
FROM mysql://root@localhost/sakila
INTO postgresql://localhost:54393/sakila

WITH include drop, create tables, create indexes, reset sequences

SET maintenance_work_mem to ´128MB´,
work_mem to ´12MB´,
search_path to ´sakila´

CAST type datetime to timestamptz drop default drop not null using zero-dates-to-null,
type date drop not null drop default using zero-dates-to-null,
-- type tinyint to boolean using tinyint-to-boolean,
type year to integer

MATERIALIZE VIEWS film_list, staff_list

-- INCLUDING ONLY TABLE NAMES MATCHING ~/film/, ´actor´
-- EXCLUDING TABLE NAMES MATCHING ~<ory>
-- DECODING TABLE NAMES MATCHING ~/messed/, ~/encoding/ AS utf8

BEFORE LOAD DO
$$ create schema if not exists sakila; $$;

The database command accepts the following clauses and options:

· FROM

Must be a connection URL pointing to a MySQL database. At the moment only MySQL is
supported as a pgloader source.

If the connection URI contains a table name, then only this table is migrated from
MySQL to PostgreSQL.

· WITH

When loading from a MySQL database, the following options are supported, and the
efault WITH clause is: no truncate, create tables, include drop, create indexes, reset
sequences, foreign keys, downcase identifiers.

WITH options:

· include drop

When this option is listed, pgloader drops all the tables in the target PostgreSQL
database whose names appear in the SQLite database. This option allows for using the
same command several times in a row until you figure out all the options, starting
automatically from a clean environment. Please note that CASCADE is used to ensure
that tables are dropped even if there are foreign keys pointing to them. This is
precisely what include drop is intended to do: drop all target tables and recreate
them.

Great care needs to be taken when using include drop, as it will cascade to all
objects referencing the target tables, possibly including other tables that are not
being loaded from the source DB.

· include no drop

When this option is listed, pgloader will not include any DROP statement when loading
the data.

· truncate

When this option is listed, pgloader issue the TRUNCATE command against each
PostgreSQL table just before loading data into it.

· no truncate

When this option is listed, pgloader issues no TRUNCATE command.

· disable triggers

When this option is listed, pgloader issues an ALTER TABLE ... DISABLE TRIGGER ALL
command against the PostgreSQL target table before copying the data, then the command
ALTER TABLE ... ENABLE TRIGGER ALL once the COPY is done.

This option allows loading data into a pre-existing table ignoring the foreign key
constraints and user defined triggers and may result in invalid foreign key
constraints once the data is loaded. Use with care.

· create tables

When this option is listed, pgloader creates the table using the meta data found in
the MySQL file, which must contain a list of fields with their data type. A standard
data type conversion from DBF to PostgreSQL is done.

· create no tables

When this option is listed, pgloader skips the creation of table before lading data,
target tables must then already exist.

· create indexes

When this option is listed, pgloader gets the definitions of all the indexes found in
the MySQL database and create the same set of index definitions against the PostgreSQL
database.

· create no indexes

When this option is listed, pgloader skips the creating indexes.

· uniquify index names, preserve index names

MySQL index names are unique per-table whereas in PostgreSQL index names have to be
unique per-schema. The default for pgloader is to change the index name by prefixing
it with idx_OID where OID is the internal numeric identifier of the table the index is
built against.

In somes cases like when the DDL are entirely left to a framework it might be sensible
for pgloader to refrain from handling index unique names, that is achieved by using
the preserve index names option.

The default is to uniquify index names.

Even when using the option preserve index names, MySQL primary key indexes named
"PRIMARY" will get their names uniquified. Failing to do so would prevent the primary
keys to be created again in PostgreSQL where the index names must be unique per
schema.

· foreign keys

When this option is listed, pgloader gets the definitions of all the foreign keys
found in the MySQL database and create the same set of foreign key definitions against
the PostgreSQL database.

· no foreign keys

When this option is listed, pgloader skips creating foreign keys.

· reset sequences

When this option is listed, at the end of the data loading and after the indexes have
all been created, pgloader resets all the PostgreSQL sequences created to the current
maximum value of the column they are attached to.

The options schema only and data only have no effects on this option.

· reset no sequences

When this option is listed, pgloader skips resetting sequences after the load.

The options schema only and data only have no effects on this option.

· downcase identifiers

When this option is listed, pgloader converts all MySQL identifiers (table names,
index names, column names) to downcase, except for PostgreSQL reserved keywords.

The PostgreSQL reserved keywords are determined dynamically by using the system
function pg_get_keywords().

· quote identifiers

When this option is listed, pgloader quotes all MySQL identifiers so that their case
is respected. Note that you will then have to do the same thing in your application
code queries.

· schema only

When this option is listed pgloader refrains from migrating the data over. Note that
the schema in this context includes the indexes when the option create indexes has
been listed.

· data only

When this option is listed pgloader only issues the COPY statements, without doing any
other processing.

· CAST

The cast clause allows to specify custom casting rules, either to overload the default
casting rules or to amend them with special cases.

A casting rule is expected to follow one of the forms:

type <mysql-type-name> [ <guard> ... ] to <pgsql-type-name> [ <option> ... ]
column <table-name>.<column-name> [ <guards> ] to ...

It´s possible for a casting rule to either match against a MySQL data type or against a
given column name in a given table name. That flexibility allows to cope with cases where
the type tinyint might have been used as a boolean in some cases but as a smallint in
others.

The casting rules are applied in order, the first match prevents following rules to be
applied, and user defined rules are evaluated first.

The supported guards are:

· when default ´value´

The casting rule is only applied against MySQL columns of the source type that have
given value, which must be a single-quoted or a double-quoted string.

· when typemod expression

The casting rule is only applied against MySQL columns of the source type that have a
typemod value matching the given typemod expression. The typemod is separated into its
precision and scale components.

Example of a cast rule using a typemod guard:

type char when (= precision 1) to char keep typemod

This expression casts MySQL char(1) column to a PostgreSQL column of type char(1) while
allowing for the general case char(N) will be converted by the default cast rule into a
PostgreSQL type varchar(N).

· with extra auto_increment

The casting rule is only applied against MySQL columns having the extra column
auto_increment option set, so that it´s possible to target e.g. serial rather than
integer.

The default matching behavior, when this option isn´t set, is to match both columns
with the extra definition and without.

This means that if you want to implement a casting rule that target either serial or
integer from a smallint definition depending on the auto_increment extra bit of
information from MySQL, then you need to spell out two casting rules as following:

type smallint with extra auto_increment
to serial drop typemod keep default keep not null,
type smallint
to integer drop typemod keep default keep not null

The supported casting options are:

· drop default, keep default

When the option drop default is listed, pgloader drops any existing default expression
in the MySQL database for columns of the source type from the CREATE TABLE statement
it generates.

The spelling keep default explicitly prevents that behaviour and can be used to
overload the default casting rules.

· drop not null, keep not null

When the option drop not null is listed, pgloader drops any existing NOT NULL
constraint associated with the given source MySQL datatype when it creates the tables
in the PostgreSQL database.

The spelling keep not null explicitly prevents that behaviour and can be used to
overload the default casting rules.

· drop typemod, keep typemod

When the option drop typemod is listed, pgloader drops any existing typemod definition
(e.g. precision and scale) from the datatype definition found in the MySQL columns of
the source type when it created the tables in the PostgreSQL database.

The spelling keep typemod explicitly prevents that behaviour and can be used to
overload the default casting rules.

· using

This option takes as its single argument the name of a function to be found in the
pgloader.transforms Common Lisp package. See above for details.

It´s possible to augment a default cast rule (such as one that applies against ENUM
data type for example) with a transformation function by omitting entirely the type
parts of the casting rule, as in the following example:

column enumerate.foo using empty-string-to-null

· MATERIALIZE VIEWS

This clause allows you to implement custom data processing at the data source by
providing a view definition against which pgloader will query the data. It´s not
possible to just allow for plain SQL because we want to know a lot about the exact
data types of each column involved in the query output.

This clause expect a comma separated list of view definitions, each one being either
the name of an existing view in your database or the following expression:

name AS $$ sql query $$

The name and the sql query will be used in a CREATE VIEW statement at the beginning of
the data loading, and the resulting view will then be dropped at the end of the data
loading.

· MATERIALIZE ALL VIEWS

Same behaviour as MATERIALIZE VIEWS using the dynamic list of views as returned by
MySQL rather than asking the user to specify the list.

· INCLUDING ONLY TABLE NAMES MATCHING

Introduce a comma separated list of table names or regular expression used to limit
the tables to migrate to a sublist.

Example:

INCLUDING ONLY TABLE NAMES MATCHING ~/film/, ´actor´

· EXCLUDING TABLE NAMES MATCHING

Introduce a comma separated list of table names or regular expression used to exclude
table names from the migration. This filter only applies to the result of the
INCLUDING filter.

EXCLUDING TABLE NAMES MATCHING ~<ory>

· DECODING TABLE NAMES MATCHING

Introduce a comma separated list of table names or regular expressions used to force
the encoding to use when processing data from MySQL. If the data encoding known to you
is different from MySQL´s idea about it, this is the option to use.

DECODING TABLE NAMES MATCHING ~/messed/, ~/encoding/ AS utf8

You can use as many such rules as you need, all with possibly different encodings.

LIMITATIONS
The database command currently only supports MySQL source database and has the following
limitations:

· Views are not migrated,

Supporting views might require implementing a full SQL parser for the MySQL dialect
with a porting engine to rewrite the SQL against PostgreSQL, including renaming
functions and changing some constructs.

While it´s not theoretically impossible, don´t hold your breath.

· Triggers are not migrated

The difficulty of doing so is not yet assessed.

· ON UPDATE CURRENT_TIMESTAMP is currently not migrated

It´s simple enough to implement, just not on the priority list yet.

· Of the geometric datatypes, only the POINT database has been covered. The other ones
should be easy enough to implement now, it´s just not done yet.

DEFAULT MySQL CASTING RULES
When migrating from MySQL the following Casting Rules are provided:

Numbers:

· type int with extra auto_increment to serial when (< precision 10)

· type int with extra auto_increment to bigserial when (<= 10 precision)

· type int to int when (< precision 10)

· type int to bigint when (<= 10 precision)

· type tinyint with extra auto_increment to serial

· type smallint with extra auto_increment to serial

· type mediumint with extra auto_increment to serial

· type bigint with extra auto_increment to bigserial

· type tinyint to boolean when (= 1 precision) using tinyint-to-boolean

· type tinyint to smallint drop typemod

· type smallint to smallint drop typemod

· type mediumint to integer drop typemod

· type integer to integer drop typemod

· type float to float drop typemod

· type bigint to bigint drop typemod

· type double to double precision drop typemod

· type numeric to numeric keep typemod

· type decimal to decimal keep typemod

Texts:

· type char to varchar keep typemod

· type varchar to text

· type tinytext to text

· type text to text

· type mediumtext to text

· type longtext to text

Binary:

· type binary to bytea

· type varbinary to bytea

· type tinyblob to bytea

· type blob to bytea

· type mediumblob to bytea

· type longblob to bytea

Date:

· type datetime when default "0000-00-00 00:00:00" and not null to timestamptz drop not
null drop default using zero-dates-to-null

· type datetime when default "0000-00-00 00:00:00" to timestamptz drop default using
zero-dates-to-null

· type timestamp when default "0000-00-00 00:00:00" and not null to timestamptz drop not
null drop default using zero-dates-to-null

· type timestamp when default "0000-00-00 00:00:00" to timestamptz drop default using
zero-dates-to-null

· type date when default "0000-00-00" to date drop default using zero-dates-to-null

· type date to date

· type datetime to timestamptz

· type timestamp to timestamptz

· type year to integer drop typemod

Geometric:

· type point to point using pgloader.transforms::convert-mysql-point

Enum types are declared inline in MySQL and separately with a CREATE TYPE command in
PostgreSQL, so each column of Enum Type is converted to a type named after the table and
column names defined with the same labels in the same order.

When the source type definition is not matched in the default casting rules nor in the
casting rules provided in the command, then the type name with the typemod is used.

LOAD SQLite DATABASE

This command instructs pgloader to load data from a SQLite file. Automatic discovery of
the schema is supported, including build of the indexes.

Here´s an example:

load database
from sqlite:///Users/dim/Downloads/lastfm_tags.db
into postgresql:///tags

with include drop, create tables, create indexes, reset sequences

set work_mem to ´16MB´, maintenance_work_mem to ´512 MB´;

The sqlite command accepts the following clauses and options:

· FROM

Path or HTTP URL to a SQLite file, might be a .zip file.

· WITH

When loading from a SQLite database, the following options are supported:

When loading from a SQLite database, the following options are supported, and the
default WITH clause is: no truncate, create tables, include drop, create indexes,
reset sequences, downcase identifiers, encoding ´utf-8´.

· include drop

When this option is listed, pgloader drops all the tables in the target PostgreSQL
database whose names appear in the SQLite database. This option allows for using the
same command several times in a row until you figure out all the options, starting
automatically from a clean environment. Please note that CASCADE is used to ensure
that tables are dropped even if there are foreign keys pointing to them. This is
precisely what include drop is intended to do: drop all target tables and recreate
them.

Great care needs to be taken when using include drop, as it will cascade to all
objects referencing the target tables, possibly including other tables that are not
being loaded from the source DB.

· include no drop

When this option is listed, pgloader will not include any DROP statement when loading
the data.

· truncate

When this option is listed, pgloader issue the TRUNCATE command against each
PostgreSQL table just before loading data into it.

· no truncate

When this option is listed, pgloader issues no TRUNCATE command.

· disable triggers

When this option is listed, pgloader issues an ALTER TABLE ... DISABLE TRIGGER ALL
command against the PostgreSQL target table before copying the data, then the command
ALTER TABLE ... ENABLE TRIGGER ALL once the COPY is done.

This option allows loading data into a pre-existing table ignoring the foreign key
constraints and user defined triggers and may result in invalid foreign key
constraints once the data is loaded. Use with care.

· create tables

When this option is listed, pgloader creates the table using the meta data found in
the SQLite file, which must contain a list of fields with their data type. A standard
data type conversion from DBF to PostgreSQL is done.

· create no tables

When this option is listed, pgloader skips the creation of table before lading data,
target tables must then already exist.

· create indexes

When this option is listed, pgloader gets the definitions of all the indexes found in
the SQLite database and create the same set of index definitions against the
PostgreSQL database.

· create no indexes

When this option is listed, pgloader skips the creating indexes.

· reset sequences

When this option is listed, at the end of the data loading and after the indexes have
all been created, pgloader resets all the PostgreSQL sequences created to the current
maximum value of the column they are attached to.

· reset no sequences

When this option is listed, pgloader skips resetting sequences after the load.

The options schema only and data only have no effects on this option.

· schema only

When this option is listed pgloader will refrain from migrating the data over. Note
that the schema in this context includes the indexes when the option create indexes
has been listed.

· data only

When this option is listed pgloader only issues the COPY statements, without doing any
other processing.

· encoding

This option allows to control which encoding to parse the SQLite text data with.
Defaults to UTF-8.

· CAST

The cast clause allows to specify custom casting rules, either to overload the default
casting rules or to amend them with special cases.

Please refer to the MySQL CAST clause for details.

· INCLUDING ONLY TABLE NAMES MATCHING

Introduce a comma separated list of table names or regular expression used to limit
the tables to migrate to a sublist.

Example:

INCLUDING ONLY TABLE NAMES MATCHING ~/film/, ´actor´

· EXCLUDING TABLE NAMES MATCHING

Introduce a comma separated list of table names or regular expression used to exclude
table names from the migration. This filter only applies to the result of the
INCLUDING filter.

EXCLUDING TABLE NAMES MATCHING ~<ory>

DEFAULT SQLite CASTING RULES
When migrating from SQLite the following Casting Rules are provided:

Numbers:

· type tinyint to smallint

· type integer to bigint

· type float to float using float-to-string

· type real to real using float-to-string

· type double to double precision using float-to-string

· type numeric to numeric using float-to-string

Texts:

· type character to text drop typemod

· type varchar to text drop typemod

· type nvarchar to text drop typemod

· type char to text drop typemod

· type nchar to text drop typemod

· type nvarchar to text drop typemod

· type clob to text drop typemod

Binary:

· type blob to bytea

Date:

· type datetime to timestamptz using sqlite-timestamp-to-timestamp

· type timestamp to timestamptz using sqlite-timestamp-to-timestamp

· type timestamptz to timestamptz using sqlite-timestamp-to-timestamp

LOAD MS SQL DATABASE

This command instructs pgloader to load data from a MS SQL database. Automatic discovery
of the schema is supported, including build of the indexes, primary and foreign keys
constraints.

Here´s an example:

load database
from mssql://user@host/dbname
into postgresql:///dbname

including only table names like ´GlobalAccount´ in schema ´dbo´

set work_mem to ´16MB´, maintenance_work_mem to ´512 MB´

before load do $$ drop schema if exists dbo cascade; $$;

The mssql command accepts the following clauses and options:

· FROM

Connection string to an existing MS SQL database server that listens and welcome
external TCP/IP connection. As pgloader currently piggybacks on the FreeTDS driver, to
change the port of the server please export the TDSPORT environment variable.

· WITH

When loading from a MS SQL database, the same options as when loading a MySQL database
are supported. Please refer to the MySQL section. The following options are added:

· create schemas

When this option is listed, pgloader creates the same schemas as found on the MS SQL
instance. This is the default.

· create no schemas

When this option is listed, pgloader refrains from creating any schemas at all, you
must then ensure that the target schema do exist.

· CAST

The cast clause allows to specify custom casting rules, either to overload the default
casting rules or to amend them with special cases.

Please refer to the MySQL CAST clause for details.

· INCLUDING ONLY TABLE NAMES LIKE ´...´ [, ´...´] IN SCHEMA ´...´

Introduce a comma separated list of table name patterns used to limit the tables to
migrate to a sublist. More than one such clause may be used, they will be accumulated
together.

Example:

including only table names lile ´GlobalAccount´ in schema ´dbo´

· EXCLUDING TABLE NAMES LIKE ´...´ [, ´...´] IN SCHEMA ´...´

Introduce a comma separated list of table name patterns used to exclude table names
from the migration. This filter only applies to the result of the INCLUDING filter.

EXCLUDING TABLE NAMES MATCHING ´LocalAccount´ in schema ´dbo´

DEFAULT MS SQL CASTING RULES
When migrating from MS SQL the following Casting Rules are provided:

Numbers:

· type tinyint to smallint

· type float to float using float-to-string

· type real to real using float-to-string

· type double to double precision using float-to-string

· type numeric to numeric using float-to-string

· type decimal to numeric using float-to-string

· type money to numeric using float-to-string

· type smallmoney to numeric using float-to-string

Texts:

· type char to text drop typemod

· type nchat to text drop typemod

· type varchar to text drop typemod

· type nvarchar to text drop typemod

· type xml to text drop typemod

Binary:

· type binary to bytea using byte-vector-to-bytea

· type varbinary to bytea using byte-vector-to-bytea

Date:

· type datetime to timestamptz

· type datetime2 to timestamptz

Others:

· type bit to boolean

· type hierarchyid to bytea

· type geography to bytea

· type uniqueidentifier to uuid using sql-server-uniqueidentifier-to-uuid

TRANSFORMATION FUNCTIONS

Some data types are implemented in a different enough way that a transformation function
is necessary. This function must be written in Common lisp and is searched in the
pgloader.transforms package.

Some default transformation function are provided with pgloader, and you can use the
--load command line option to load and compile your own lisp file into pgloader at
runtime. For your functions to be found, remember to begin your lisp file with the
following form:

(in-package #:pgloader.transforms)

The provided transformation functions are:

· zero-dates-to-null

When the input date is all zeroes, return nil, which gets loaded as a PostgreSQL NULL
value.

· date-with-no-separator

Applies zero-dates-to-null then transform the given date into a format that PostgreSQL
will actually process:

In: "20041002152952"
Out: "2004-10-02 15:29:52"

· time-with-no-separator

Transform the given time into a format that PostgreSQL will actually process:

In: "08231560"
Out: "08:23:15.60"

· tinyint-to-boolean

As MySQL lacks a proper boolean type, tinyint is often used to implement that. This
function transforms 0 to ´false´ and anything else to ´true´.

· bits-to-boolean

As MySQL lacks a proper boolean type, BIT is often used to implement that. This
function transforms 1-bit bit vectors from 0 to f and any other value to t..

· int-to-ip

Convert an integer into a dotted representation of an ip4.

In: 18435761
Out: "1.25.78.177"

· ip-range

Converts a couple of integers given as strings into a range of ip4.

In: "16825344" "16825599"
Out: "1.0.188.0-1.0.188.255"

· convert-mysql-point

Converts from the astext representation of points in MySQL to the PostgreSQL
representation.

In: "POINT(48.5513589 7.6926827)"
Out: "(48.5513589,7.6926827)"

· float-to-string

Converts a Common Lisp float into a string suitable for a PostgreSQL float:

In: 100.0d0
Out: "100.0"

· set-to-enum-array

Converts a string representing a MySQL SET into a PostgreSQL Array of Enum values from
the set.

In: "foo,bar"
Out: "{foo,bar}"

· empty-string-to-null

Convert an empty string to a null.

· right-trimg

Remove whitespace at end of string.

· byte-vector-to-bytea

Transform a simple array of unsigned bytes to the PostgreSQL bytea Hex Format
representation as documented at
http://www.postgresql.org/docs/9.3/interactive/datatype-binary.html

· sqlite-timestamp-to-timestamp

SQLite type system is quite interesting, so cope with it here to produce timestamp
literals as expected by PostgreSQL. That covers year only on 4 digits, 0 dates to
null, and proper date strings.

· sql-server-uniqueidentifier-to-uuid

The SQL Server driver receives data fo type uniqueidentifier as byte vector that we
then need to convert to an UUID string for PostgreSQL COPY input format to process.

· unix-timestamp-to-timestamptz

Converts a unix timestamp (number of seconds elapsed since beginning of 1970) into a
proper PostgreSQL timestamp format.

LOAD MESSAGES

This command is still experimental and allows receiving messages via UDP using a syslog
like format, and, depending on rule matching, loads named portions of the data stream into
a destination table.

LOAD MESSAGES
FROM syslog://localhost:10514/

WHEN MATCHES rsyslog-msg IN apache
REGISTERING timestamp, ip, rest
INTO postgresql://localhost/db?logs.apache
SET guc_1 = ´value´, guc_2 = ´other value´

WHEN MATCHES rsyslog-msg IN others
REGISTERING timestamp, app-name, data
INTO postgresql://localhost/db?logs.others
SET guc_1 = ´value´, guc_2 = ´other value´

WITH apache = rsyslog
DATA = IP REST
IP = 1*3DIGIT "." 1*3DIGIT "."1*3DIGIT "."1*3DIGIT
REST = ~/.*/

WITH others = rsyslog;

As the command is still experimental the options might be changed in the future and the
details are not documented.

Use pgloader online using onworks.net services