一.官網說明

A datawarehousing configuration is a many-to-one configuration. Multiple sourcedatabases send data to one target warehouse database. Oracle GoldenGatesupports like-to-like or heterogeneous transfer of data, with capabilities forfiltering and conversion on any system in the configuration (support varies bydatabase platform).

--多個Source DB傳送到一個Target warehouse DB。

1.1 Considerations for a data warehousing configuration

1.1.1 Isolation of data records

Thisconfiguration assumes that each source database contributes different recordsto the target system. If the same record exists in the same table on two ormore source systems and can be changed on any of those systems, conflictresolution routines are needed to resolve conflicts when changes to that recordare made on both sources at the same time and replicated to the target table.

這裡假設每個source DB 傳送給TargetDB的都是不同的record，如果是相同的record，可能就會在TargetDB 上產生衝突。

1.1.2 Data storage

You can dividethe data storage between the source systems and the target system to reduce theneed for massive amounts of disk space on the target system. This is accomplishedby using a data pump on each source, rather than sending data directly from eachExtract across the network to the target.

(1)A primary Extract writes to alocal trail on each source.

(2)A data-pump Extract on eachsource reads the local trail and sends it across TCP/IP to a dedicated Replicatgroup.

1.1.3 Filtering and conversion

If not all ofthe data from a source system will be sent to the data warehouse, you can use thedata pump to perform the filtering. This removes that processing overhead fromthe primary Extract group, and it reduces the amount of data that is sentacross the network.

1.1.3.1 To filter data,you can use:

(1)A FILTER or WHERE clause in a TABLEstatement (Extract) or in a MAP statement (Replicat).

(2)A SQL query or procedure

(3)User exits

1.1.3.2 To transformdata, you can use:

(1)Native Oracle GoldenGateconversion functions

(2)A user exit from the Extract orReplicat process that applies rules from an external

(3)transformation solution, thenreturns the manipulated data to Oracle GoldenGate.

(4)Replicat to deliver datadirectly to an ETL solution or other transformation engine.

1.1.4 Data volume

The standard configuration is sufficientif:

(1)The transaction load isconsistent and of moderate volume that is spread out more or less evenly amongall of the objects to be replicated.

(2)There are none of thefollowing: tables that are subject to long-running transactions, tables thathave a very large number of columns that change, or tables that contain columnsfor which Oracle GoldenGate must fetch from the database (generally columnswith LOBs, columns that are affected by SQL procedures executed by Oracle GoldenGate,and columns that are not logged to the transaction log).

If yourenvironment does not satisfy those conditions, consider adding one or more setsof parallel processes.

1.2 Creating a data warehousing configuration

Figure 13 Configuration for datawarehousing

1.2.1 Source systems

--To configure theManager process

1. On each source, configure the Managerprocess.

2. In each Manager parameter file, use the PURGEOLDEXTRACTSparameter to control the purging of files from the trail on the local system.

--To configure theprimary Extract groups

3. On each source, use the ADD EXTRACT commandto create a primary Extract group. For documentation purposes, these groups arecalled ext_1 and ext_2.

Extract_1

ADD EXTRACT<ext_1>, TRANLOG, BEGIN <time> [, THREADS <n>]

Extract_2

ADD EXTRACT<ext_2>, TRANLOG, BEGIN <time> [, THREADS <n>]

注意： Use TRANLOG as the data source option. For DB2 on Z/OS, specify thebootstrap data set (BSDS) name following TRANLOG.

4. On each source, use the ADD EXTTRAIL commandto create a local trail.

Extract_1

ADD EXTTRAIL<local_trail_1>, EXTRACT <ext_1>

Extract_2

ADD EXTTRAIL<local_trail_2>, EXTRACT <ext_2>

注意： Use the EXTRACT argument to link each Extract group to the localtrail on the same system. The primary Extract writes to this trail, and thedata-pump reads it.

5. On each source, use the EDIT PARAMS commandto create a parameter file for the primary Extract. Include the followingparameters plus any others that apply to your database environment.

Extract_1

-- Identify the Extract group:

EXTRACT <ext_1>

-- Specify database login information asneeded for the database:

[SOURCEDB <dsn_1>,][USERID<user>[, PASSWORD <pw>]]

-- Specify the local trail that thisExtract writes to:

EXTTRAIL <local_trail_1>

-- Specify tables to be captured:

TABLE <owner>.<table>;

Extract_2

-- Identify the Extract group:

EXTRACT <ext_2>

-- Specify database login information asneeded for the database:

[SOURCEDB <dsn_2>,][USERID<user>[, PASSWORD <pw>]]

-- Specify the local trail that thisExtract writes to:

EXTTRAIL <local_trail_2>

-- Specify tables to be captured:

TABLE <owner>.<table>;

--To configure the datapumps

6. On each source, use the ADD EXTRACT commandto create a data pump Extract group. For documentation purposes, these pumpsare called pump_1 and pump_2.

Data pump_1

ADD EXTRACT <pump_1>, EXTTRAILSOURCE<local_trail_1>, BEGIN <time>

Data pump_2

ADD EXTRACT <pump_2>, EXTTRAILSOURCE<local_trail_2>, BEGIN <time>

注意：Use EXTTRAILSOURCE as the data source option, and specify the nameof the trail on the local system.

7. On each source, use the ADD RMTTRAIL commandto create a remote trail on the target.

Source_1

ADD RMTTRAIL<remote_trail_1>, EXTRACT <pump_1>

Source_2

ADD RMTTRAIL<remote_trail_2>, EXTRACT <pump_2>

注意：Use the EXTRACT argumentto link each remote trail to a different data pump. The data pump writes tothis trail over TCP/IP, and a Replicat reads from it.

8. On each source, use the EDIT PARAMS commandto create a parameter file for the data pump group. Include the followingparameters plus any others that apply to your

database environment.

Data pump_1

-- Identify the data pump group:

EXTRACT <pump_1>

-- Specify database login information asneeded for the database:

[SOURCEDB <dsn_1>,][USERID<user>[, PASSWORD <pw>]]

-- Specify the name or IP address of thetarget system:

RMTHOST <target>, MGRPORT<portnumber>

-- Specify the remote trail on the target system:

RMTTRAIL <remote_trail_1>

-- Allow mapping, filtering, conversion orpass data through as-is:

[PASSTHRU | NOPASSTHRU]

-- Specify tables to be captured:

TABLE <owner>.<table>;

Data pump_2

-- Identify the data pump group:

EXTRACT <pump_2>

-- Specify database login information asneeded for the database:

[SOURCEDB <dsn_2>,][USERID<user>[, PASSWORD <pw>]]

-- Specify the name or IP address of thetarget system:

RMTHOST <target>, MGRPORT<portnumber>

-- Specify the remote trail on the targetsystem:

RMTTRAIL <remote_trail_2>

-- Allow mapping, filtering, conversion orpass data through as-is:

[PASSTHRU | NOPASSTHRU]

-- Specify tables to be captured:

TABLE <owner>.<table>;

注意：Use NOPASSTHRU if the data pump will be filtering or convertingdata, and also use the SOURCEDB and USERID parameters as appropriate for thedatabase, to enable definitions lookups. If the data pump will not be filteringor converting data, use PASSTHRU to bypass the lookups.

NOTE：

To use PASSTHRU mode,the names of the source and target objects must beidentical. No column mapping, filtering, SQLEXECfunctions, transformation, or other functions that require data manipulationcan be specified in the parameter file. You can combine normal processing withpass-through processing by pairing PASSTHRU and NOPASSTHRU with different TABLEstatements.

1.2.2 Target system

--To configure theManager process

9. Configure the Manager process.

10. In the Manager parameter file, use the PURGEOLDEXTRACTSparameter to control the purging of files from the trail.

--To configure the Replicat groups

11. On the target, use the ADD REPLICAT commandto create a Replicat group for each remote trail that you created. Fordocumentation purposes, these groups are called rep_1 and rep_2.

Replicat_1

ADD REPLICAT <rep_1>, EXTTRAIL<remote_trail_1>, BEGIN <time>

Replicat_2

ADD REPLICAT <rep_2>, EXTTRAIL<remote_trail_2>, BEGIN <time>

注意： Use the EXTTRAIL argumentto link the Replicat group to the trail.

每個Data Pump 對應一個Replicat程序。

12. On the target, use the EDIT PARAMS commandto create a parameter file for each Replicat group. Include the followingparameters plus any others that apply to your database environment.

Replicat_1

-- Identify the Replicat group:

REPLICAT <rep_1>

-- State whether or not source and targetdefinitions are identical:

SOURCEDEFS <full_pathname> |ASSUMETARGETDEFS

-- Specify database login information asneeded for the database:

[TARGETDB <dsn_3>,] [USERID <userid>[, PASSWORD <pw>]]

-- Specify error handling rules:

REPERROR (<error>, <response>)

-- Specify tables for delivery:

MAP <owner>.<table>, TARGET<owner>.<table>[, DEF <template name>];

Replicat_2

-- Identify the Replicat group:

REPLICAT <rep_2>

-- State whether or not source and targetdefinitions are identical:

SOURCEDEFS <full_pathname> |ASSUMETARGETDEFS

-- Specify database login information asneeded for the database:

[TARGETDB <dsn_3>,] [USERID <userid>[, PASSWORD <pw>]]

-- Specify error handling rules:

REPERROR (<error>, <response>)

-- Specify tables for delivery:

MAP <owner>.<table>, TARGET<owner>.<table>[, DEF <template name>];

注意：You can use anynumber of MAP statements for any given Replicat group. All MAP statements for agiven Replicat group must specify the same objects that are contained in thetrail that is linked to the group.

二.示例

這裡還是3個節點的GG，我們這裡從節點2和節點3傳送到節點1.

DB： oracle 11.2.0.3

OS:redhat 5.4

2.1 在所有節點上設定Manager

GGSCI (gg1) 2> edit params mgr

PORT 7809

2.2 分別在GG2和GG3上新增Extract程序

--GG2

GGSCI (gg2) 4> add extract ext1,tranlog,begin now

GGSCI (gg2) 5> add exttrail/u01/ggate/dirdat/lt, extract ext1

GGSCI (gg2) 7> view params ext1

extract ext1

userid [email protected],password ggate

exttrail /u01/ggate/dirdat/lt

ddl include all objname dave.pdba;

table dave.pdba;

--GG3

GGSCI (gg3) 2> add extract ext1,tranlog,begin now

GGSCI (gg3) 4> add exttrail/u01/ggate/dirdat/lt, extract ext1

GGSCI (gg3) 6> view params ext1

extract ext1

userid [email protected],password ggate

exttrail /u01/ggate/dirdat/lt

ddl include all objname dave.pdba;

table dave.pdba;

2.3 分別在GG2和GG3上新增DataPump 程序

--GG2

GGSCI (gg2) 8> add extractdpump,exttrailsource /u01/ggate/dirdat/lt

GGSCI (gg2) 9> add rmttrail/u01/ggate/dirdat/d1, extract dpump

--注意這裡指定的Target端的位置，我們以d1 開頭。

GGSCI (gg2) 11> view params dpump

extract dpump

userid [email protected], password ggate

rmthost gg1, mgrport 7809,compress,compressthreshold0

rmttrail /u01/ggate/dirdat/d1

passthru

table dave.pdba;

--GG3

GGSCI (gg3) 7> add extractdpump,exttrailsource /u01/ggate/dirdat/lt

GGSCI (gg3) 8> add rmttrail/u01/ggate/dirdat/d2, extract dpump

GGSCI (gg3) 10> view params dpump

extract dpump

userid [email protected], password ggate

rmthost gg1, mgrport 7809,compress,compressthreshold0

rmttrail /u01/ggate/dirdat/d2

passthru

table dave.pdba;

2.4 在GG1上配置2個Replicat程序，分別對應Data Pump程序

--配置checkpoint

GGSCI (gg1) 3> view params ./GLOBALS

GGSCHEMA ggate

CHECKPOINTTABLE ggate.checkpoint

GGSCI (gg1) 5> dblogin [email protected],password ggate

Successfully logged into database.

GGSCI (gg1) 7> add checkpointtableggate.checkpoint

Successfully created checkpoint tableGGATE.CHECKPOINT.

這部分，參考：

--配置2個Replicat：

--rep1:

GGSCI (gg1) 8> add replicatrep1,exttrail /u01/ggate/dirdat/d1, checkpointtable ggate.checkpoint

REPLICAT added.

GGSCI (gg1) 11> view params rep1

replicat rep1

ASSUMETARGETDEFS

userid [email protected],password ggate

discardfile/u01/ggate/dirdat/rep1_discard.txt, append, megabytes 10

--HANDLECOLLISIONS

ddl include all

ddlerror default ignore retryop

map dave.pdba, target dave.pdba;

--rep2:

GGSCI (gg1) 9> add replicatrep2,exttrail /u01/ggate/dirdat/d2, checkpointtable ggate.checkpoint

REPLICAT added.

GGSCI (gg1) 13> view params rep2

replicat rep2

ASSUMETARGETDEFS

userid [email protected],password ggate

discardfile/u01/ggate/dirdat/rep2_discard.txt, append, megabytes 10

--HANDLECOLLISIONS

ddl include all

ddlerror default ignore retryop

map dave.pdba, target dave.pdba;

資料初始化的問題，具體參考：

2.5 啟動相關程序進行測試

--GG2：

GGSCI (gg2) 20> start ext1

確定ext1的trails 檔案：

GGSCI (gg2) 25> view report ext1

…

2011-11-20 11:31:43 INFOOGG-01056 Recovery initializationcompleted for target file /u01/ggate

/dirdat/lt000018, at RBA 132350, CSN1525929.

2011-11-20 11:31:43 INFOOGG-01478 Output file/u01/ggate/dirdat/lt is using format RELEASE 10.

4/11.1.

2011-11-20 11:31:43 WARNING OGG-01438 Checkpoint marked as from graceful shutdown,but records fou

nd after checkpoint in trail/u01/ggate/dirdat/lt. Expected EOF Seqno0, RBA 0. Found Seqno 18, RB

A 132350.

2011-11-20 11:31:43 INFOOGG-01026 Rolling over remotefile /u01/ggate/dirdat/lt000018.

…

GGSCI (gg2) 24> info dpump

EXTRACTDPUMP Initialized 2011-11-20 11:11 Status STOPPED

Checkpoint Lag 00:00:00 (updated 00:20:16 ago)

Log Read Checkpoint File /u01/ggate/dirdat/lt000000

First Record RBA 0

GGSCI (gg2) 26> alter extractdpump,extseqno 18,extrba 0

EXTRACT altered.

GGSCI (gg2) 27> start dpump

Sending START request to MANAGER ...

EXTRACT DPUMP starting

GGSCI (gg2) 40> info all

ProgramStatus Group Lag Time Since Chkpt

MANAGERRUNNING

EXTRACTRUNNING DPUMP 00:00:00 00:02:05

EXTRACTRUNNING EXT1 00:00:00 00:00:02

REPLICATABENDED REP1 00:00:00 12:24:42

GGSCI (gg2) 41> info dpump

EXTRACTDPUMP Last Started 2011-11-2011:37 Status RUNNING

Checkpoint Lag 00:00:00 (updated 00:02:24 ago)

Log Read Checkpoint File /u01/ggate/dirdat/lt000018

First Record RBA 0

這裡注意Data Pump 預設是從lt000000 開始讀取trail 檔案，如果是新搭建的GG 同步，那麼都是從lt000000開始處理，所以沒有問題，如果是後來修改稱data pump，就需要根據extract 程序的錯誤提示資訊，使用如下命令：

GGSCI(gg1) 82> alter extract dpump,extseqno 18,extrba 0

具體參考：

--GG3:

GGSCI (gg3) 14> start ext1

Sending START request to MANAGER ...

EXTRACT EXT1 starting

GGSCI (gg3) 18> view report ext1

…

2011-11-20 11:39:50 INFOOGG-01056 Recovery initializationcompleted for target file /u01/ggate

/dirdat/lt000016, at RBA 132672, CSN1525929.

2011-11-20 11:39:50 INFOOGG-01478 Output file/u01/ggate/dirdat/lt is using format RELEASE 10.

4/11.1.

2011-11-20 11:39:50 WARNING OGG-01438 Checkpoint marked as from graceful shutdown,but records fou

nd after checkpoint in trail/u01/ggate/dirdat/lt. Expected EOF Seqno0, RBA 0. Found Seqno 16, RB

A 132672.

2011-11-20 11:39:50 INFOOGG-01026 Rolling over remotefile /u01/ggate/dirdat/lt000016.

…

GGSCI (gg3) 19> alter extractdpump,extseqno 16,extrba 0

EXTRACT altered.

GGSCI (gg3) 20> info dpump

EXTRACTDPUMP Initialized 2011-11-20 11:41 Status STOPPED

Checkpoint Lag 00:00:00 (updated 00:00:06 ago)

Log Read Checkpoint File /u01/ggate/dirdat/lt000016

First Record RBA 0

GGSCI (gg3) 21> start dpump

Sending START request to MANAGER ...

EXTRACT DPUMP starting

GGSCI (gg3) 22> info all

ProgramStatus Group Lag Time Since Chkpt

MANAGERRUNNING

EXTRACTRUNNING DPUMP 00:00:00 00:00:13

EXTRACTRUNNING EXT1 00:00:00 00:00:05

REPLICATABENDED REP1 00:00:00 12:28:42

--GG1:

GGSCI (gg1) 18> start rep1

Sending START request to MANAGER ...

REPLICAT REP1 starting

GGSCI (gg1) 21> start rep2

Sending START request to MANAGER ...

REPLICAT REP2 starting

GGSCI (gg1) 22> info all

ProgramStatus Group Lag Time Since Chkpt

MANAGERRUNNING

EXTRACTABENDED DPUMP 00:00:00 12:34:55

EXTRACTABENDED DPUMP2 00:00:00 12:34:49

EXTRACTABENDED EXT1 00:00:00 12:34:51

REPLICATRUNNING REP1 00:00:00 00:17:37

REPLICATRUNNING REP2 00:00:00 00:17:15

2.6 驗證同步

--GG2的pdba 表insert 一條記錄：

SQL> conn dave/dave;

Connected.

SQL> insert into pdba values(2222,sysdate);

1 row created.

SQL> commit;

Commit complete.

--GG3的pdba 表insert 一條記錄：

SQL> conn dave/dave;

Connected.

SQL> insert into pdba values(3333,sysdate);

1 row created.

SQL> commit;

Commit complete.

--在GG1 的pdba 表查詢這條記錄：

SQL> select * from pdba where idin(2222,3333);

no rows selected

沒有同步過來。

GGSCI (gg1) 86> info rep1

REPLICATREP1 Last Started 2011-11-2012:22 Status RUNNING

Checkpoint Lag 00:00:00 (updated 00:00:01 ago)

Log Read Checkpoint File /u01/ggate/dirdat/d1000000

First Record RBA 0

GGSCI (gg1) 87> stop rep1

Sending STOP request to REPLICAT REP1 ...

Request processed.

GGSCI (gg1) 88> stop rep2

Sending STOP request to REPLICAT REP2 ...

Request processed.

GGSCI (gg1) 89> alter replicat rep1extseqno 1 extrba 0

REPLICAT altered.

GGSCI (gg1) 91> alter replicat rep2 extseqno1 extrba 0

REPLICAT altered.

GGSCI (gg1) 93> start rep1

Sending START request to MANAGER ...

REPLICAT REP1 starting

GGSCI (gg1) 94> start rep2

Sending START request to MANAGER ...

REPLICAT REP2 starting

--最後確認一下：

SQL>select * from pdba where id<4444;

ID TIME

---------- ------------

2222 20-NOV-11

3333 20-NOV-11

多對一的測試同步成功。

注意這裡有2個小技巧：

1.如果同步有問題，但程序report又沒有提供更多資訊，可以使用Logdump工具檢視trails裡的內容，從而確認是哪一步出現了問題。關於logdump ，參考：

2.如果tail 搞不定，一個簡單的解決方法：

把所有程序delete 掉，所有的tails 全部delete 掉，重新來過，這種方法比檢視trail 要節省很多時間。

-------------------------------------------------------------------------------------------------------

版權所有，文章允許轉載，但必須以連結方式註明源地址，否則追究法律責任!

Blog: http://blog.csdn.net/tianlesoftware

Weibo: http://weibo.com/tianlesoftware

Skype: tianlesoftware

-------加群需要在備註說明Oracle表空間和資料檔案的關係，否則拒絕申請----

DBA1 群：62697716(滿); DBA2 群：62697977(滿)DBA3 群：62697850(滿)

DBA 超級群：63306533(滿); DBA4 群：83829929(滿) DBA5群： 142216823(滿)

DBA6 群：158654907(滿) DBA7 群：69087192(滿)DBA8 群：172855474

DBA 超級群2：151508914 DBA9群：102954821 聊天 群：40132017(滿)