Use data define file and colmap to map different column

阿新 • • 發佈：2019-01-21

通常我們使用OGG同步一個表的時候，遇見兩個表的列不一樣，這時候我們就需要使用資料定義檔案和colmap將行進行對映

下面我們來看看源表的結構：

SQL> desc test3;
Name                                                  Null?    Type
----------------------------------------------------- -------- ------------------------------------
OWNER                                                 NOT NULL VARCHAR2(30)
TABLE_NAME                                            NOT NULL VARCHAR2(30)
TABLESPACE_NAME                                                VARCHAR2(30)
CLUSTER_NAME                                                   VARCHAR2(30)
IOT_NAME                                                       VARCHAR2(30)
STATUS                                                         VARCHAR2(8)
PCT_FREE                                                       NUMBER
PCT_USED                                                       NUMBER
INI_TRANS                                                      NUMBER
MAX_TRANS                                                      NUMBER
INITIAL_EXTENT                                                 NUMBER
NEXT_EXTENT                                                    NUMBER
MIN_EXTENTS                                                    NUMBER
MAX_EXTENTS                                                    NUMBER
PCT_INCREASE                                                   NUMBER
FREELISTS                                                      NUMBER
FREELIST_GROUPS                                                NUMBER
LOGGING                                                        VARCHAR2(3)
BACKED_UP                                                      VARCHAR2(1)
NUM_ROWS                                                       NUMBER
BLOCKS                                                         NUMBER
EMPTY_BLOCKS                                                   NUMBER
AVG_SPACE                                                      NUMBER
CHAIN_CNT                                                      NUMBER
AVG_ROW_LEN                                                    NUMBER
AVG_SPACE_FREELIST_BLOCKS                                      NUMBER
NUM_FREELIST_BLOCKS                                            NUMBER
DEGREE                                                         VARCHAR2(20)
INSTANCES                                                      VARCHAR2(20)
CACHE                                                          VARCHAR2(10)
TABLE_LOCK                                                     VARCHAR2(8)
SAMPLE_SIZE                                                    NUMBER
LAST_ANALYZED                                                  DATE
PARTITIONED                                                    VARCHAR2(3)
IOT_TYPE                                                       VARCHAR2(12)
TEMPORARY                                                      VARCHAR2(1)
SECONDARY                                                      VARCHAR2(1)
NESTED                                                         VARCHAR2(3)
BUFFER_POOL                                                    VARCHAR2(7)
FLASH_CACHE                                                    VARCHAR2(7)
CELL_FLASH_CACHE                                               VARCHAR2(7)
ROW_MOVEMENT                                                   VARCHAR2(8)
GLOBAL_STATS                                                   VARCHAR2(3)
USER_STATS                                                     VARCHAR2(3)
DURATION                                                       VARCHAR2(15)
SKIP_CORRUPT                                                   VARCHAR2(8)
MONITORING                                                     VARCHAR2(3)
CLUSTER_OWNER                                                  VARCHAR2(30)
DEPENDENCIES                                                   VARCHAR2(8)
COMPRESSION                                                    VARCHAR2(8)
COMPRESS_FOR                                                   VARCHAR2(12)
DROPPED                                                        VARCHAR2(3)
READ_ONLY                                                      VARCHAR2(3)
SEGMENT_CREATED                                                VARCHAR2(3)
RESULT_CACHE                                                   VARCHAR2(7)


SQL> select count(1) from test3;

  COUNT(1)
----------
      2822

目標表的結構：

SQL> desc test3;
Name                                      Null?    Type
----------------------------------------- -------- ----------------------------
OWNER                                     NOT NULL VARCHAR2(30)
TABLE_NAME                                NOT NULL VARCHAR2(30)
TABLESPACE_NAME                                    VARCHAR2(30)

SQL>
SQL>
SQL>
SQL> select count(1) from test3;

  COUNT(1)
----------
         0

SQL>

具體操作如下：
前期安裝配置就不詳細介紹了，請參考前面的文章。
這裡先使用ogg的initload 方法同步源表已有的資料：
Source：

add extract initload,sourceistable;

edit param initload

extract initload
userid ogg, password ogg
rmthost 192.168.56.61,mgrport 7839
rmttask replicat, group repload
table zbdba.test3;

建立資料定義檔案：

GGSCI (zbdba1) 7> edit param defgen

defsfile ./dirsql/convert.sql
userid ogg,password ogg
table zbdba.test3;

[ 
[email protected] ogg]$ ./defgen paramfile /ogg/dirprm/defgen.prm

***********************************************************************
        Oracle GoldenGate Table Definition Generator for Oracle
      Version 11.2.1.0.1 OGGCORE_11.2.1.0.1_PLATFORMS_120423.0230
   Linux, x64, 64bit (optimized), Oracle 11g on Apr 23 2012 05:08:19

Copyright (C) 1995, 2012, Oracle and/or its affiliates. All rights reserved.


                    Starting at 2015-03-02 15:59:20

***********************************************************************

Operating System Version:
Linux
Version #1 SMP Wed Jun 13 18:24:36 EDT 2012, Release 2.6.32-279.el6.x86_64
Node: zbdba1
Machine: x86_64
                         soft limit   hard limit
Address Space Size   :    unlimited    unlimited
Heap Size            :    unlimited    unlimited
File Size            :    unlimited    unlimited
CPU Time             :    unlimited    unlimited

Process id: 7481

***********************************************************************
**            Running with the following parameters                  **
***********************************************************************
defsfile ./dirsql/convert.sql
userid ogg,password ***
table zbdba.test3;
Retrieving definition for ZBDBA.TEST3

2015-03-02 15:59:24  WARNING OGG-00869  No unique key is defined for table 'TEST3'. All viable columns will be used to represent the key, but may not guarantee uniqueness.  KEYCOLS may be used to define the key.



Definitions generated for 1 table in ./dirsql/convert.sql


[[email protected] dirsql]$ cat convert.sql
*+- Defgen version 2.0, Encoding UTF-8
*
* Definitions created/modified  2015-03-02 15:59
*
*  Field descriptions for each column entry:
*
*     1    Name
*     2    Data Type
*     3    External Length
*     4    Fetch Offset
*     5    Scale
*     6    Level
*     7    Null
*     8    Bump if Odd
*     9    Internal Length
*    10    Binary Length
*    11    Table Length
*    12    Most Significant DT
*    13    Least Significant DT
*    14    High Precision
*    15    Low Precision
*    16    Elementary Item
*    17    Occurs
*    18    Key Column
*    19    Sub Data Type
*
Database type: ORACLE
Character set ID: windows-936
National character set ID: UTF-16
Locale: neutral
Case sensitivity: 14 14 14 14 14 14 14 14 14 14 14 14 11 14 14 14
*
Definition for table ZBDBA.TEST3
Record length: 1704
Syskey: 0
Columns: 55
OWNER                       64     30        0  0  0 1 0     30     30      0 0 0 0 0 1    0 1 0
TABLE_NAME                  64     30       36  0  0 1 0     30     30      0 0 0 0 0 1    0 1 0
TABLESPACE_NAME             64     30       72  0  0 1 0     30     30      0 0 0 0 0 1    0 1 0
CLUSTER_NAME                64     30      108  0  0 1 0     30     30      0 0 0 0 0 1    0 1 0
IOT_NAME                    64     30      144  0  0 1 0     30     30      0 0 0 0 0 1    0 1 0
STATUS                      64      8      180  0  0 1 0      8      8      0 0 0 0 0 1    0 1 0
PCT_FREE                    64     50      194  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
PCT_USED                    64     50      250  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
INI_TRANS                   64     50      306  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
MAX_TRANS                   64     50      362  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
INITIAL_EXTENT              64     50      418  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
NEXT_EXTENT                 64     50      474  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
MIN_EXTENTS                 64     50      530  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
MAX_EXTENTS                 64     50      586  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
PCT_INCREASE                64     50      642  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
FREELISTS                   64     50      698  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
FREELIST_GROUPS             64     50      754  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
LOGGING                     64      3      810  0  0 1 0      3      3      0 0 0 0 0 1    0 1 0
BACKED_UP                   64      1      818  0  0 1 0      1      1      0 0 0 0 0 1    0 1 0
NUM_ROWS                    64     50      824  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
BLOCKS                      64     50      880  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
EMPTY_BLOCKS                64     50      936  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
AVG_SPACE                   64     50      992  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
CHAIN_CNT                   64     50     1048  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
AVG_ROW_LEN                 64     50     1104  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
AVG_SPACE_FREELIST_BLOCKS   64     50     1160  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
NUM_FREELIST_BLOCKS         64     50     1216  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
DEGREE                      64     20     1272  0  0 1 0     20     20      0 0 0 0 0 1    0 1 0
INSTANCES                   64     20     1298  0  0 1 0     20     20      0 0 0 0 0 1    0 1 0
CACHE                       64     10     1324  0  0 1 0     10     10      0 0 0 0 0 1    0 1 0
TABLE_LOCK                  64      8     1340  0  0 1 0      8      8      0 0 0 0 0 1    0 1 0
SAMPLE_SIZE                 64     50     1354  0  0 1 0     50     50     50 0 0 0 0 1    0 1 2
LAST_ANALYZED              192     19     1410  0  0 1 0     19     19     19 0 5 0 0 1    0 1 0
PARTITIONED                 64      3     1432  0  0 1 0      3      3      0 0 0 0 0 1    0 1 0
IOT_TYPE                    64     12     1440  0  0 1 0     12     12      0 0 0 0 0 1    0 1 0
TEMPORARY                   64      1     1458  0  0 1 0      1      1      0 0 0 0 0 1    0 1 0
SECONDARY                   64      1     1464  0  0 1 0      1      1      0 0 0 0 0 1    0 1 0
NESTED                      64      3     1470  0  0 1 0      3      3      0 0 0 0 0 1    0 1 0
BUFFER_POOL                 64      7     1478  0  0 1 0      7      7      0 0 0 0 0 1    0 1 0
FLASH_CACHE                 64      7     1490  0  0 1 0      7      7      0 0 0 0 0 1    0 1 0
CELL_FLASH_CACHE            64      7     1502  0  0 1 0      7      7      0 0 0 0 0 1    0 1 0
ROW_MOVEMENT                64      8     1514  0  0 1 0      8      8      0 0 0 0 0 1    0 1 0
GLOBAL_STATS                64      3     1528  0  0 1 0      3      3      0 0 0 0 0 1    0 1 0
USER_STATS                  64      3     1536  0  0 1 0      3      3      0 0 0 0 0 1    0 1 0
DURATION                    64     15     1544  0  0 1 0     15     15      0 0 0 0 0 1    0 1 0
SKIP_CORRUPT                64      8     1564  0  0 1 0      8      8      0 0 0 0 0 1    0 1 0
MONITORING                  64      3     1578  0  0 1 0      3      3      0 0 0 0 0 1    0 1 0
CLUSTER_OWNER               64     30     1586  0  0 1 0     30     30      0 0 0 0 0 1    0 1 0
DEPENDENCIES                64      8     1622  0  0 1 0      8      8      0 0 0 0 0 1    0 1 0
COMPRESSION                 64      8     1636  0  0 1 0      8      8      0 0 0 0 0 1    0 1 0
COMPRESS_FOR                64     12     1650  0  0 1 0     12     12      0 0 0 0 0 1    0 1 0
DROPPED                     64      3     1668  0  0 1 0      3      3      0 0 0 0 0 1    0 1 0
READ_ONLY                   64      3     1676  0  0 1 0      3      3      0 0 0 0 0 1    0 1 0
SEGMENT_CREATED             64      3     1684  0  0 1 0      3      3      0 0 0 0 0 1    0 1 0
RESULT_CACHE                64      7     1692  0  0 1 0      7      7      0 0 0 0 0 1    0 1 0
End of definition

Target：

GGSCI (zbdba2) 6> add replicat repload,specialrun;
REPLICAT added.


GGSCI (zbdba2) 7>

GGSCI (zbdba2) 7>

GGSCI (zbdba2) 7> edit param repload

REPLICAT repload
USERID ogg, PASSWORD ogg
SOURCEDEFS ./dirsql/convert.sql
MAP zbdba.test3, TARGET zbdba.test3,
colmap (
owner=owner, table_name=table_name,tablespace_name=tablespace_name)
;

開啟抽取程序：

GGSCI (zbdba1) 6> start initload

Sending START request to MANAGER ...
EXTRACT INITLOAD starting

檢視日誌：

2015-03-02 16:11:12  INFO    OGG-00993  Oracle GoldenGate Capture for Oracle, initload.prm:  EXTRACT INITLOAD started.
2015-03-02 16:11:42  INFO    OGG-00991  Oracle GoldenGate Capture for Oracle, initload.prm:  EXTRACT INITLOAD stopped normally.

Target：
2015-03-03 02:44:33  INFO    OGG-00973  Oracle GoldenGate Manager for Oracle, mgr.prm:  Manager started replicat task process (Port 7840).
2015-03-03 02:44:48  INFO    OGG-01229  Oracle GoldenGate Delivery for Oracle:  Connected to :37943.
2015-03-03 02:44:48  INFO    OGG-00995  Oracle GoldenGate Delivery for Oracle, repload.prm:  REPLICAT REPLOAD starting.
2015-03-03 02:44:48  INFO    OGG-03035  Oracle GoldenGate Delivery for Oracle, repload.prm:  Operating system character set identified as UTF-8. Locale: en_US, LC_ALL:.
2015-03-03 02:44:48  INFO    OGG-01815  Oracle GoldenGate Delivery for Oracle, repload.prm:  Virtual Memory Facilities for: COM
    anon alloc: mmap(MAP_ANON)  anon free: munmap
    file alloc: mmap(MAP_SHARED)  file free: munmap
    target directories:
    /ogg/dirtmp.
2015-03-03 02:44:48  INFO    OGG-00996  Oracle GoldenGate Delivery for Oracle, repload.prm:  REPLICAT REPLOAD started.
2015-03-03 02:44:52  WARNING OGG-00869  Oracle GoldenGate Delivery for Oracle, repload.prm:  No unique key is defined for table 'TEST3'. All viable columns will be used to represent the key, but may not guarantee uniqueness.  KEYCOLS may be used to define the key.
2015-03-03 02:44:57  INFO    OGG-00994  Oracle GoldenGate Delivery for Oracle, repload.prm:  REPLICAT REPLOAD stopped normally.

已經同步完成：
檢視目標庫資料：

SQL> select count(1) from test3;

  COUNT(1)
----------
      2822

SQL>

自此已完成初始化

Use data define file and colmap to map different column

通常我們使用OGG同步一個表的時候，遇見兩個表的列不一樣，這時候我們就需要使用資料定義檔案和colmap將行進行對映下面我們來看看源表的結構： SQL> desc test3; Name

Use Snapshots and Clones to Recover Storage Gateway Volume Data

Amazon Web Services is Hiring. Amazon Web Services (AWS) is a dynamic, growing business unit within Amazon.com. We are currently hiring So

Use DropzondJS and Spring to upload and download file

Easy File Upload Using DropzoneJS and Spring is an open source library that provides drag and drop file uploads with imag

SpringBoot中測試Elasticsearch，在測試ElasticsearchRepository文件操作中的查詢時報錯org.springframework.data.elasticsearch.ElasticsearchException: failed to map source **

問題描述：在SpringBoot中測試Elasticsearch的時候，當測試ElasticsearchRepository的文件操作的時候，在呼叫find()查詢方法時報錯，錯誤如下： com.fasterxml.jackson.databind.exc.InvalidDefinitionExc

Why (and how) to use eslint in your project

Why (and how) to use eslint in your projectThis story was written by Sam Roberts, a Senior Software Engineer at IBM Canada. It was first published in IBM d

From Cats to Categories: Processing Geospatial Data with Machine and Deep Learning

With the exponential growth of the number of images (and radar data, and point clouds, and…) that are being collected, we must answer this question: how ar

How to Use Netflix’s Eureka and Spring Cloud for Service Registry

How to Use Netflix’s Eureka and Spring Cloud for Service RegistryOne of the main tenets of the microservice architecture pattern is that a set of loosely-c

How to use APIs with Pandas and store the results in Redshift

How to use APIs with Pandas and store the results in RedshiftHere is an easy tutorial to help understand how you can use Pandas to get data from a RESTFUL

Use WebP images along with other fallback sources to and placeholder to get max image optimization…

Use WebP images along with other fallback sources and placeholder to get max image optimization on the websiteI was figuring out the best way to AUTO optim

Swiss Artificial Intelligence Startup Map and Link to Fintech Q3/ 2018 Fintech Schweiz Digital Finance News

Swisscom published the 3rd Swiss Artificial Intelligence Startup Map. Out of this list, we have listed again below for your convenience all the Startups ou

A quick intro to Dependency Injection: what it is, and when to use it

Why should I use dependency injection?Let’s say we have a car class which contains various objects such as wheels, engine, etc.Without dependency injection

Three Popular Clustering Methods and When to Use Each

Three Popular Clustering Methods and When to Use EachIn the mad rush to find new ways of teasing apart labeled data, we often forget about everything we ca

Why we use and contribute to open source software

Why we use and contribute to open source softwareThis is Kevin McEntee, VP of Systems & ECommerce Engineering here at Netflix. Netflix is a technology

How We Use Data to Suggest Tags for Your Story

How We Use Data to Suggest Tags for Your StoryHere on Medium, we envision tags to be central in organizing and connecting ideas. Follow the tags you’re int

Use Data Pipeline to Copy Tables to Another Database

Download and use these scripts to copy a table from one database to another using Data Pipeline. Before you begin, modify the sample definition

Use data science to up your game performance

Spencer Krum shows you how he uses Jupyter Notebooks and IBM Data Science Experience (DSX) to analyze professional StarCraft II matches (which represent t

How to read *.data in Matlab and Python

See P2 of "HW2_545_2018_"In Matlab:z = dlmread('spambase.data',',');In Python:import numpy as npz = np.genfromtxt('spambase.data', dtype=float, delimi

python trojan development 2nd —— use python to send mail and listen to the key board then combine them

conf ssa rst 創建 fin com mat os.path onf 今晚先上代碼，明天再說怎麽寫的，請勿用於非法用途！！！！！ 1 from pynput.keyboard import Key,Listener 2 import logging

[React] Configure a React & Redux Application For Production Deployment and Deploy to Now

hang scrip app class pack pub relative pts als In this lesson, we’ll make a few small changes to our scripts and add some environme

tar(child):bzip2:Cannot exec: No such file and directory

bzip2 下午從阿裏雲拷貝一個比較大的壓縮包到另外一臺阿裏雲主機，使用scp發現拷貝速度有點慢，想起以前用過pv+lz+tar的方法傳輸Linux下的大數據，二話不說，開始幹。因為是翻墻下載的包，一開始以為是自己下載的安裝包有問題，但仔細看了下提示“bzip2:Cannot exec...”，想了下，

Use data define file and colmap to map different column

相關推薦