資料大挪移之ora2pg使用指南
阿新 • • 發佈：2018-12-06
ora2pg是一款免費的小工具，用於將Oracle資料庫遷移到PostgreSQL。ora2pg通過自動掃描oracle資料庫獲取表結構以及資料，然後生成PostgreSQL資料庫的SQL指令碼或者直接在PostgreSQL上建立表並插入資料。而且它還支援檢視、序列以及分割槽表，其功能非常強大。這裡有一個120G左右的oracle 11g測試庫，因業務需求將其遷移至PostgreSQL 10.6。
一、安裝ora2pg
1.1 安裝OS依賴軟體包
[[email protected] ~]# yum install perl-DBD-Pg perl perl-devel perl-DBI perl-CPAN bzip2 perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker
1.2 安裝Oracle瘦客戶端
從Oracle官方網站下載oracle 12c的瘦客戶端，然後進行安裝並配置。
[[email protected] u02]# la /u02/software/oracle/
total 55M
drwxr-xr-x 2 root root 4.0K Dec 3 15:04 .
drwxr-xr-x 7 root root 76 Dec 3 15:04 ..
-rw-r--r-- 1 root root 51M Nov 28 14:34 oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 593K Nov 28 14:32 oracle-instantclient12.2-devel-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 1.5M Nov 28 14:32 oracle-instantclient12.2-jdbc-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 692K Nov 28 14:32 oracle-instantclient12.2-sqlplus-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 922K Nov 28 14:35 oracle-instantclient12.2-tools-12.2.0.1.0-1.x86_64.rpm
[ 編輯root環境變數，加入以下內容：
[[email protected] ~]# vi .bash_profile
export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib
export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK
export TNS_ADMIN=/usr/lib/oracle/12.2/client64/network/admin
export ORACLE_HOME=/usr/lib/oracle/12.2/client64/
export PATH=$PATH:$ORACLE_HOME/bin:$HOME/bin
[ 1.3 安裝ora2pg
[[email protected] ~]# wget https://sourceforge.net/projects/ora2pg/files/19.1/ora2pg-19.1.tar.bz2
[[email protected] ~]# tar -xjf ora2pg-19.1.tar.bz2
[[email protected] ~]# cd ora2pg-19.1
[[email protected] ora2pg-19.1]# perl Makefile.PL
Checking if your kit is complete...
Looks good
Writing Makefile for Ora2Pg
Done...
------------------------------------------------------------------------------
Please read documentation at http://ora2pg.darold.net/ before asking for help
------------------------------------------------------------------------------
Now type: make && make install
[[email protected] ora2pg-19.1]# make && make install
cp lib/Ora2Pg.pm blib/lib/Ora2Pg.pm
cp lib/Ora2Pg/GEOM.pm blib/lib/Ora2Pg/GEOM.pm
cp lib/Ora2Pg/PLSQL.pm blib/lib/Ora2Pg/PLSQL.pm
cp lib/Ora2Pg/MySQL.pm blib/lib/Ora2Pg/MySQL.pm
cp scripts/ora2pg blib/script/ora2pg
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/ora2pg
cp scripts/ora2pg_scanner blib/script/ora2pg_scanner
/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/ora2pg_scanner
Manifying blib/man3/ora2pg.3
Installing /usr/local/share/perl5/Ora2Pg.pm
Installing /usr/local/share/perl5/Ora2Pg/GEOM.pm
Installing /usr/local/share/perl5/Ora2Pg/PLSQL.pm
Installing /usr/local/share/perl5/Ora2Pg/MySQL.pm
Installing /usr/local/share/man/man3/ora2pg.3
Installing /usr/local/bin/ora2pg
Installing /usr/local/bin/ora2pg_scanner
Installing default configuration file (ora2pg.conf.dist) to /etc/ora2pg
Appending installation info to /usr/lib64/perl5/perllocal.pod1.4 安裝DBD::Oracle
[[email protected] ~]# perl -MCPAN -e shell
Terminal does not support AddHistory.
cpan shell -- CPAN exploration and modules installation (v1.9800)
Enter 'h' for help.
cpan[1]> get DBD::Oracle
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/authors/01mailrc.txt.gz
Reading '/root/.cpan/sources/authors/01mailrc.txt.gz'
............................................................................DONE
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/modules/02packages.details.txt.gz
Reading '/root/.cpan/sources/modules/02packages.details.txt.gz'
Database was generated on Wed, 28 Nov 2018 23:54:25 GMT
HTTP::Date not available
.............
New CPAN.pm version (v2.16) available.
[Currently running version is v1.9800]
You might want to try
install CPAN
reload cpan
to both upgrade CPAN.pm and run the new version without leaving
the current session.
...............................................................DONE
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/modules/03modlist.data.gz
Reading '/root/.cpan/sources/modules/03modlist.data.gz'
DONE
Writing /root/.cpan/Metadata
Running get for module 'DBD::Oracle'
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/authors/id/Z/ZA/ZARQUON/DBD-Oracle-1.76.tar.gz
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/authors/id/Z/ZA/ZARQUON/CHECKSUMS
Checksum for /root/.cpan/sources/authors/id/Z/ZA/ZARQUON/DBD-Oracle-1.76.tar.gz ok
Scanning cache /root/.cpan/build for sizes
DONE
cpan[2]> quit
Terminal does not support GetHistory.
Lockfile removed.
[[email protected] ~]# cd .cpan/build/DBD-Oracle-1.76-sU5yu5/
[[email protected] DBD-Oracle-1.76-sU5yu5]# perl Makefile.PL
[[email protected] DBD-Oracle-1.76-sU5yu5]# make && make install1.5 建立ora2pg配置檔案
首先從示例檔案複製一份，然後根據實際情況更改配置檔案相關引數：
[[email protected] ~]# cp /etc/ora2pg/ora2pg.conf.dist /etc/ora2pg/ora2pg.conf
[[email protected] ~]# vi /etc/ora2pg/ora2pg.conf
--Oracle連線引數
ORACLE_HOME /usr/lib/oracle/12.2/client64
ORACLE_DSN dbi:Oracle:host=172.16.100.19;sid=kkdb;port=1521
ORACLE_USER system --如果這裡的使用者是sys、system，則user_grants選項必須設定為1
ORACLE_PWD abcABC12
USER_GRANTS 1
ORA_INITIAL_COMMAND commit
--schema引數
EXPORT_SCHEMA 1
SCHEMA XXKIN
CREATE_SCHEMA 0
COMPILE_SCHEMA 0
PG_SCHEMA public
--匯出物件(根據實際情況選擇)
TYPE TABLE COPY VIEW TYPE
DISABLE_COMMENT 0
EXTERNAL_TO_FDW 1
TRUNCATE_TABLE 0
USE_TABLESPACE 0
REORDERING_COLUMNS 0
--約束引數
KEEP_PKEY_NAMES 0
PKEY_IN_CREATE 0
FKEY_ADD_UPDATE never
FKEY_DEFERRABLE 0
DEFER_FKEY 0
DROP_FKEY 1
--觸發器引數
DISABLE_SEQUENCE 1
DISABLE_TRIGGERS 1
--物件修改引數
PRESERVE_CASE 0
INDEXES_RENAMING 0
USE_INDEX_OPCLASS 0
PREFIX_PARTITION 0
DISABLE_PARTITION 1
WITH_OID 0
ORA_RESERVED_WORDS audit,comment
USE_RESERVED_WORDS 1
DISABLE_UNLOGGED 0
--PostgreSQL引數
PG_DSN dbi:Pg:dbname=kkdb;host=192.168.120.149;port=5432
PG_USER xxkin
PG_PWD redhat
OUTPUT xxkin.sql
OUTPUT_DIR /tmp
BZIP2
FILE_PER_CONSTRAINT 0
FILE_PER_INDEX 0
FILE_PER_FKEYS 0
FILE_PER_TABLE 0
FILE_PER_FUNCTION 0
STOP_ON_ERROR 1
COPY_FREEZE 0
CREATE_OR_REPLACE 1
--TYPE引數
PG_NUMERIC_TYPE 1
PG_INTEGER_TYPE 1
DEFAULT_NUMERIC bigint
ENABLE_MICROSECOND 1
TO_NUMBER_CONVERSION numeric
GEN_USER_PWD 0
FORCE_OWNER 0
FORCE_SECURITY_INVOKER 0
--資料引數
DATA_LIMIT 10000
NOESCAPE 0
TRANSACTION serializable
STANDARD_CONFORMING_STRINGS 1
NO_LOB_LOCATOR 1
XML_PRETTY 0
LOG_ON_ERROR 0
TRIM_TYPE BOTH
INTERNAL_DATE_MAX 49
FUNCTION_CHECK 1
--Export/Import效能引數
JOBS 1
ORACLE_COPIES 1
PARALLEL_TABLES 1
DEFAULT_PARALLELISM_DEGREE 0
PARALLEL_MIN_ROWS 100000
DROP_INDEXES 0
SYNCHRONOUS_COMMIT 0
--PLSQL引數
EXPORT_INVALID 0
PLSQL_PGSQL 1
NULL_EQUAL_EMPTY 0
EMPTY_LOB_NULL 0
PACKAGE_AS_SCHEMA 1
REWRITE_OUTER_JOIN 1
FUNCTION_STABLE 1
COMMENT_COMMIT_ROLLBACK 0
COMMENT_SAVEPOINT 0
USE_ORAFCE 0
AUTONOMOUS_TRANSACTION 1
--ASSESSMENT引數
ESTIMATE_COST 0
COST_UNIT_VALUE 5
DUMP_AS_HTML 0
TOP_MAX 10
HUMAN_DAYS_LIMIT 5
--PostgreSQL特性引數
PG_SUPPORTS_WHEN 1
PG_SUPPORTS_INSTEADOF 1
PG_SUPPORTS_MVIEW 1
PG_SUPPORTS_CHECKOPTION 1
PG_SUPPORTS_IFEXISTS 1
PG_SUPPORTS_LATERAL 1
PG_SUPPORTS_PARTITION 1
BITMAP_AS_GIN 1
PG_BACKGROUND 0
PG_SUPPORTS_SUBSTR 1
PG_SUPPORTS_NAMED_OPERATOR 1
PG_SUPPORTS_IDENTITY 1
PG_SUPPORTS_PROCEDURE 0
--SPATIAL引數
AUTODETECT_SPATIAL_TYPE 1
CONVERT_SRID 1
DEFAULT_SRID 4326
GEOMETRY_EXTRACT_TYPE INTERNAL
--FDW引數
FDW_SERVER orcl
--MySQL引數
MYSQL_PIPES_AS_CONCAT 0
MYSQL_INTERNAL_EXTRACT_FORMAT 0到此，整個ora2pg的安裝配置已完成。
二、使用ora2pg
2.1 匯出全部表結構
如果在配置檔案裡指定的type包含了COPY，導完表結構後，表的資料也會隨之匯出，加-t引數即可只匯出表結構不匯出資料。
[[email protected] ~]# ora2pg -t table view
[========================>] 777/777 tables (100.0%) end of scanning.
[========================>] 777/777 tables (100.0%) end of table export.
[========================>] 0/0 views (100.0%) end of output. 匯出完成後，在PostgreSQL中建立資料庫以及對應的表。
-bash-4.2$ psql
psql (10.6)
Type "help" for help.
postgres=# CREATE USER xxkin WITH password 'redhat' CREATEDB SUPERUSER replication bypa***ls createrole login;
postgres=# create database kkdb;
postgres=# alter database kkdb owner to xxkin;
postgres=# \c kkdb
You are now connected to database "kkdb" as user "postgres".
kkdb=# \i /tmp/output.sqlora2pg有很多引數可供使用，具體請參考其幫助檔案。
2.2 通過ora2pg遷移資料
之前已建立好資料庫以及相關表結構，現在只需通過ora2pg將資料同步到PostgreSQL資料庫上即可。
[[email protected] ~]# ora2pg -d -t copy
[2018-11-30 11:38:51] Ora2Pg version: 19.1
[2018-11-30 11:38:51] Trying to connect to database: dbi:Oracle:host=172.16.100.19;sid=kkdb;port=1521
[2018-11-30 11:38:51] Isolation level: SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
[2018-11-30 11:38:51] DEBUG: executing initial command to Oracle: commit
[2018-11-30 11:38:51] Retrieving table information...
[2018-11-30 11:40:55] [1] Scanning table ABC_DEL_CLASS (1 rows)...
[2018-11-30 11:40:55] [2] Scanning table ACTIVITY_DATA (41 rows)...
[2018-11-30 11:40:55] [3] Scanning table ACTIVITY_GBEAN_STAT (243897 rows)...
......如果在遷移過程中，需要排除某些表，加-e引數，如下:
[[email protected] ~]# ora2pg -d -t copy -e usr_dx_tbs,hr_info_pg,op_log總結：
以下是在使用過程遇到的問題，提出的建議：
1、先匯出表結構，然後在同步資料；
2、在匯出表結構的時候禁用分割槽表；
3、注意PostgreSQL的保留的關鍵字是否為oracle表字段名；
4、刪除所有外來鍵約束。
5、直接通過ora2pg遷移資料，不要將oracle表資料匯出為csv格式，然後再匯入PostgreSQL資料庫。
參考文獻：ora2pg DOCUMENTATION