1. 程式人生 > >Ambari資料庫分析-配置相關

Ambari資料庫分析-配置相關

重點介紹配置相關儲存。
主要是服務對應的配置問題;主機對應的配置問題;使用者修改的配置問題.

image

clusterconfig

叢集配置表,包含各種服務元件的相關配置

41  version1    1   core-site   2   2   {"fs.defaultFS":"hdfs://hadoop181:8020","ipc.server.tcpnodelay":"true","mapreduce.jobtracker.webinterface.trusted":"false","hadoop.security.auth_to_local":"DEFAULT","ipc.client.idlethreshold"
:"8000","hadoop.proxyuser.hdfs.groups":"*","fs.trash.interval":"360","hadoop.security.authorization":"false","hadoop.http.authentication.simple.anonymous.allowed":"true","ipc.client.connection.maxidletime":"30000","hadoop.proxyuser.hcat.groups":"*","hadoop.proxyuser.hive.hosts":"hadoop182","ha.failover-controller.active-standby-elector.zk.op.retries"
:"120","hadoop.proxyuser.ambari.hosts":"hadoop181","hadoop.security.authentication":"simple","hadoop.proxyuser.hdfs.hosts":"*","hadoop.proxyuser.ambari.groups":"*","ipc.client.connect.max.retries":"50","io.file.buffer.size":"131072","hadoop.proxyuser.hcat.hosts":"hadoop182","net.topology.script.file.name"
:"/etc/hadoop/conf/topology_script.py","io.compression.codecs":"org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec","hadoop.proxyuser.hive.groups":"*","io.serializations":"org.apache.hadoop.io.serializer.WritableSerialization"} {"final":{"fs.defaultFS":"true"}} 1489658974532 65 version1 1 yarn-site 2 2 {"yarn.resourcemanager.zk-address":"hadoop183:2181,hadoop182:2181,hadoop181:2181","yarn.timeline-service.entity-group-fs-store.active-dir":"/ats/active/","yarn.nodemanager.container-monitor.interval-ms":"3000","yarn.nodemanager.linux-container-executor.group":"hadoop","yarn.application.classpath":"$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*","yarn.admin.acl":"activity_analyzer,yarn","yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds":"3600","yarn.nodemanager.remote-app-log-dir-suffix":"logs","yarn.nodemanager.address":"0.0.0.0:45454","yarn.resourcemanager.webapp.address":"hadoop182:8088","yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms":"300000","yarn.scheduler.maximum-allocation-vcores":"3","yarn.timeline-service.address":"hadoop182:10200","yarn.timeline-service.enabled":"true","yarn.nodemanager.aux-services":"mapreduce_shuffle,spark_shuffle,spark2_shuffle","yarn.nodemanager.resource.cpu-vcores":"3","yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled":"false","yarn.nodemanager.aux-services.spark_shuffle.class":"org.apache.spark.network.yarn.YarnShuffleService","yarn.nodemanager.log-aggregation.debug-enabled":"false","yarn.resourcemanager.zk-acl":"world:anyone:rwcda","yarn.http.policy":"HTTP_ONLY","yarn.client.nodemanager-connect.max-wait-ms":"60000","yarn.timeline-service.http-authentication.simple.anonymous.allowed":"true","yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size":"10000","yarn.resourcemanager.zk-state-store.parent-path":"/rmstore","yarn.resourcemanager.admin.address":"hadoop182:8141","yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage":"false","yarn.resourcemanager.resource-tracker.address":"hadoop182:8025","yarn.resourcemanager.fs.state-store.retry-policy-spec":"2000, 500","yarn.nodemanager.delete.debug-delay-sec":"0","yarn.timeline-service.http-authentication.type":"simple","yarn.timeline-service.ttl-enable":"true","yarn.timeline-service.entity-group-fs-store.retain-seconds":"604800","yarn.log-aggregation.retain-seconds":"2592000","yarn.nodemanager.resource.memory-mb":"4096","yarn.timeline-service.http-authentication.proxyuser.ambari.groups":"*","yarn.nodemanager.disk-health-checker.min-healthy-disks":"0.25","yarn.node-labels.enabled":"false","yarn.nodemanager.log.retain-second":"604800","yarn.resourcemanager.connect.max-wait.ms":"900000","yarn.scheduler.minimum-allocation-vcores":"1","yarn.resourcemanager.zk-num-retries":"1000","yarn.nodemanager.aux-services.spark2_shuffle.classpath":"{{stack_root}}/${hdp.version}/spark2/aux/*","yarn.resourcemanager.scheduler.class":"org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler","yarn.resourcemanager.system-metrics-publisher.enabled":"true","yarn.timeline-service.client.max-retries":"30","yarn.timeline-service.client.retry-interval-ms":"1000","yarn.timeline-service.store-class":"org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore","yarn.timeline-service.bind-host":"0.0.0.0","yarn.nodemanager.container-metrics.unregister-delay-ms":"60000","yarn.timeline-service.generic-application-history.store-class":"org.apache.hadoop.yarn.server.applicationhistoryservice.NullApplicationHistoryStore","yarn.resourcemanager.fs.state-store.uri":" ","yarn.nodemanager.linux-container-executor.cgroups.hierarchy":"hadoop-yarn","yarn.resourcemanager.recovery.enabled":"true","yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath":"","yarn.resourcemanager.scheduler.address":"hadoop182:8030","yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size":"10000","yarn.timeline-service.ttl-ms":"2678400000","yarn.nodemanager.health-checker.script.timeout-ms":"60000","yarn.nodemanager.aux-services.spark2_shuffle.class":"org.apache.spark.network.yarn.YarnShuffleService","yarn.log.server.url":"http://hadoop182:19888/jobhistory/logs","yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes":"org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl","yarn.resourcemanager.address":"hadoop182:8050","yarn.timeline-service.recovery.enabled":"true","yarn.resourcemanager.scheduler.monitor.enable":"false","yarn.nodemanager.recovery.dir":"{{yarn_log_dir_prefix}}/nodemanager/recovery-state","hadoop.registry.rm.enabled":"true","yarn.resourcemanager.bind-host":"0.0.0.0","yarn.nodemanager.log-aggregation.num-log-files-per-app":"30","yarn.nodemanager.admin-env":"MALLOC_ARENA_MAX\u003d$MALLOC_ARENA_MAX","yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size":"10","yarn.acl.enable":"false","yarn.resourcemanager.ha.enabled":"false","yarn.client.nodemanager-connect.retry-interval-ms":"10000","yarn.timeline-service.version":"1.5","yarn.nodemanager.linux-container-executor.resources-handler.class":"org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler","yarn.timeline-service.leveldb-timeline-store.read-cache-size":"104857600","yarn.nodemanager.local-dirs":"/hadoop/yarn/local","yarn.timeline-service.leveldb-timeline-store.path":"/hadoop/yarn/timeline","yarn.nodemanager.recovery.enabled":"true","yarn.resourcemanager.zk-timeout-ms":"10000","yarn.resourcemanager.am.max-attempts":"2","yarn.resourcemanager.state-store.max-completed-applications":"${yarn.resourcemanager.max-completed-applications}","yarn.nodemanager.linux-container-executor.cgroups.mount":"false","yarn.node-labels.fs-store.root-dir":"/system/yarn/node-labels","yarn.node-labels.fs-store.retry-policy-spec":"2000, 500","yarn.nodemanager.aux-services.mapreduce_shuffle.class":"org.apache.hadoop.mapred.ShuffleHandler","yarn.resourcemanager.work-preserving-recovery.enabled":"true","yarn.log-aggregation-enable":"true","yarn.nodemanager.aux-services.spark_shuffle.classpath":"{{stack_root}}/${hdp.version}/spark/aux/*","yarn.resourcemanager.store.class":"org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore","yarn.timeline-service.leveldb-state-store.path":"/hadoop/yarn/timeline","yarn.timeline-service.entity-group-fs-store.done-dir":"/ats/done/","yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage":"90","hadoop.registry.zk.quorum":"hadoop183:2181,hadoop182:2181,hadoop181:2181","yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds":"3600","yarn.nodemanager.container-executor.class":"org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor","yarn.resourcemanager.nodes.exclude-path":"/etc/hadoop/conf/yarn.exclude","yarn.timeline-service.state-store-class":"org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore","yarn.timeline-service.http-authentication.proxyuser.ambari.hosts":"hadoop181","yarn.nodemanager.bind-host":"0.0.0.0","yarn.resourcemanager.hostname":"hadoop182","yarn.resourcemanager.connect.retry-interval.ms":"30000","yarn.timeline-service.webapp.address":"hadoop182:8188","yarn.scheduler.minimum-allocation-mb":"512","yarn.timeline-service.entity-group-fs-store.summary-store":"org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore","yarn.nodemanager.health-checker.interval-ms":"135000","yarn.resourcemanager.zk-retry-interval-ms":"1000","yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb":"1000","yarn.nodemanager.remote-app-log-dir":"/app-logs","yarn.scheduler.maximum-allocation-mb":"4096","yarn.nodemanager.vmem-check-enabled":"false","yarn.timeline-service.entity-group-fs-store.scan-interval-seconds":"60","yarn.resourcemanager.webapp.https.address":"hadoop182:8090","yarn.timeline-service.webapp.https.address":"hadoop182:8190","yarn.nodemanager.resource.percentage-physical-cpu-limit":"80","yarn.nodemanager.vmem-pmem-ratio":"2.1","yarn.nodemanager.log-aggregation.compression-type":"gz","yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms":"10000","yarn.nodemanager.log-dirs":"/hadoop/yarn/log"} {} 1489658977060

script

CREATE TABLE clusterconfig (
  config_id BIGINT NOT NULL,
  version_tag VARCHAR(100) NOT NULL,
  version BIGINT NOT NULL,
  type_name VARCHAR(100) NOT NULL,
  cluster_id BIGINT NOT NULL,
  stack_id BIGINT NOT NULL,
  selected SMALLINT NOT NULL DEFAULT 0,
  config_data LONGTEXT NOT NULL,
  config_attributes LONGTEXT,
  create_timestamp BIGINT NOT NULL,
  selected_timestamp BIGINT NOT NULL DEFAULT 0,
  CONSTRAINT PK_clusterconfig PRIMARY KEY (config_id),
  CONSTRAINT FK_clusterconfig_cluster_id FOREIGN KEY (cluster_id) REFERENCES clusters (cluster_id),
  CONSTRAINT FK_clusterconfig_stack_id FOREIGN KEY (stack_id) REFERENCES stack(stack_id),
  CONSTRAINT UQ_config_type_tag UNIQUE (cluster_id, type_name, version_tag),
  CONSTRAINT UQ_config_type_version UNIQUE (cluster_id, type_name, version));

重點說明

  • /**
    • The most recent time that this configuration was marked as
    • {@link #selected}. This is useful when configruations are being reverted
    • since a reversion does not create a new instance. Another configuration may
    • technically be newer via its creation date ({@link #timestamp}), however
    • that does not indicate it was the most recently enabled configuration.
      */

serviceconfig

服務配置,通過serviceconfigmapping關聯表與clusterconfig相關聯。

CREATE TABLE serviceconfig (
  service_config_id BIGINT NOT NULL,
  cluster_id BIGINT NOT NULL,
  service_name VARCHAR(255) NOT NULL,
  version BIGINT NOT NULL,
  create_timestamp BIGINT NOT NULL,
  stack_id BIGINT NOT NULL,
  user_name VARCHAR(255) NOT NULL DEFAULT '_db',
  group_id BIGINT,
  note LONGTEXT,
  CONSTRAINT PK_serviceconfig PRIMARY KEY (service_config_id),
  CONSTRAINT FK_serviceconfig_stack_id FOREIGN KEY (stack_id) REFERENCES stack(stack_id),
  CONSTRAINT UQ_scv_service_version UNIQUE (cluster_id, service_name, version));

 serviceconfigmapping

關聯service配置表與cluster配置表

serviceconfighosts

關聯servic配置表與主機表

configgroup

配置組,用於儲存使用者自定義配置.

2   2   tianbaochao YARN    New configuration group created on Fri Mar 17 2017  1489716220882   YARN
CREATE TABLE configgroup (
  group_id BIGINT,
  cluster_id BIGINT NOT NULL,
  group_name VARCHAR(255) NOT NULL,
  tag VARCHAR(1024) NOT NULL,
  description VARCHAR(1024),
  create_timestamp BIGINT NOT NULL,
  service_name VARCHAR(255),
  CONSTRAINT PK_configgroup PRIMARY KEY (group_id),
  CONSTRAINT FK_configgroup_cluster_id FOREIGN KEY (cluster_id) REFERENCES clusters (cluster_id));

configgrouphostmapping

配置組和主機對映關係

CREATE TABLE configgrouphostmapping (
  config_group_id BIGINT NOT NULL,
  host_id BIGINT NOT NULL,
  CONSTRAINT PK_configgrouphostmapping PRIMARY KEY (config_group_id, host_id),
  CONSTRAINT FK_cghm_cgid FOREIGN KEY (config_group_id) REFERENCES configgroup (group_id),
  CONSTRAINT FK_cghm_host_id FOREIGN KEY (host_id) REFERENCES hosts (host_id));

confgroupclusterconfigmapping

配置組與叢集配置對映關係

資料

2   2   yarn-site   version1493012679312    _db 1493012680084

script

CREATE TABLE confgroupclusterconfigmapping (
  config_group_id BIGINT NOT NULL,
  cluster_id BIGINT NOT NULL,
  config_type VARCHAR(100) NOT NULL,
  version_tag VARCHAR(100) NOT NULL,
  user_name VARCHAR(100) DEFAULT '_db',
  create_timestamp BIGINT NOT NULL,
  CONSTRAINT PK_confgroupclustercfgmapping PRIMARY KEY (config_group_id, cluster_id, config_type),
  CONSTRAINT FK_cgccm_gid FOREIGN KEY (config_group_id) REFERENCES configgroup (group_id),
  CONSTRAINT FK_confg FOREIGN KEY (cluster_id, config_type, version_tag) REFERENCES clusterconfig (cluster_id, type_name, version_tag));

hostconfigmapping

/**
* Entity that represents a host config mapping and override.
*/
主機

CREATE TABLE hostconfigmapping (
  create_timestamp BIGINT NOT NULL,
  host_id BIGINT NOT NULL,
  cluster_id BIGINT NOT NULL,
  type_name VARCHAR(255) NOT NULL,
  selected INTEGER NOT NULL DEFAULT 0,
  service_name VARCHAR(255),
  version_tag VARCHAR(255) NOT NULL,
  user_name VARCHAR(255) NOT NULL DEFAULT '_db',
  CONSTRAINT PK_hostconfigmapping PRIMARY KEY (create_timestamp, host_id, cluster_id, type_name),
  CONSTRAINT FK_hostconfmapping_cluster_id FOREIGN KEY (cluster_id) REFERENCES clusters (cluster_id),
  CONSTRAINT FK_hostconfmapping_host_id FOREIGN KEY (host_id) REFERENCES hosts (host_id));

相關推薦

Ambari資料庫分析-配置相關

重點介紹配置相關儲存。 主要是服務對應的配置問題;主機對應的配置問題;使用者修改的配置問題. clusterconfig 叢集配置表,包含各種服務元件的相關配置 41 version1 1 core-site 2 2

資料庫配置相關檔案

applicationConfig.properties 資料庫配置 #application all config #jdbc c3p0 jdbc.driver =com.microsoft.sqlserver.jdbc.SQLServerDriver jdbc.url = jdbc:sq

使用maven profile實現多環境配置相關打包

本地 ces 軟件測試 測試 project oca rect fault uil 項目開發需要有多個環境,一般為開發,測試,預發,正式4個環境,通過maven可以實現按不同環境進行打包部署,命令為: mvn package -P dev 在eclipse中可以右擊選項r

背景圖background的例子分析相關屬性

完全 attach 輸入 定位 以及 p s ... microsoft center 今天需要做一個占滿設備寬度的輪播圖,這裏作為demo僅展示一張圖,下面分別是要操作的圖片(這裏做了縮放處理,實際的圖比較大),以及要實現的效果圖,很明顯兩者是不成比例的:   (

數據源連接數據庫配置相關xml文件

servle b- 內容 筆記 icu web-inf tom ica 得出 學完數據源連接數據後,做個筆記,當我們的程序對數據庫訪問頻繁時,為了提高程序運行效率,我們可以通過 數據源連接數據庫,從數據庫連接池中直接取得出於空閑狀態的數據庫連接對象,以下是相關xml文件的配

linux網絡配置相關命令、虛擬網絡接口eth0:0

命令 ftw onf network gre .org -h ocs ngs 網絡接口(interface)是網絡硬件設備在操作系統中的表示方法,比如網卡在Linux操作系統中用ethX,是由0開始的正整數,比如eth0、eth1...... ethX。而普通貓和ADSL的

linux 網絡設備,網卡配置 ,相關

ifconfig value 地址 nat ace spa 排查 glibc 計算機名稱 網絡設備,網卡配置: Eth0是物理網卡:唯一mac地址,Bcast:廣播地址,MAsk:子網掩碼, Lo:系統自帶的回環的ip地址,可以做一些基本的測試應用,比如沒有網卡就用127

bash配置相關

相關 src bubuko 方式 OS tro info gpo body 登錄方式 登錄方式分為兩種方式: bash配置相關

centos網絡配置相關

ifconfig 網絡配置 相關 ip配置文件目錄:/etc/sysconfig/network-scripts/ ifcfg-IFNAME配置文件的格式: DEVICE=IFNAME: 此配置文件所關聯到的設備,設備名稱要與本文件名ifcfg-後面保持一致;

Android Studio Could not GET gradle-3.2.0.pom (安卓包配置相關問題)

could 9.png too spa not pom .com sharp 安卓 安卓出現的錯誤如下 Could not GET ‘https://dl.google.com/dl/android/maven2/com/android/tools/build/gradl

區塊鏈教程Fabric1.0源代碼分析配置交易-生成通道配置

tran 路徑 實現 消息 org 代碼筆記 ssi bat 初始 Fabric 1.0源代碼筆記 之 configtx(配置交易) #configtxgen(生成通道配置) 1、configtxgen概述 configtxgen,用於生成通道配置,具體有如下三種用法:

區塊鏈教程Fabric1.0源代碼分析配置交易-生成通道配置

class def main函數 amp solo inspect like load dset 區塊鏈教程Fabric1.0源代碼分析配置交易-生成通道配置二。Generator接口實現,即bootstrapper。 type bootstrapper struct {

區塊鏈教程Fabric1.0源代碼分析配置交易體系介

sco ring mon group 區塊 comm templates copy marshal 區塊鏈教程之Fabric1.0源代碼分析配置交易體系介紹,2018年下半年,區塊鏈行業正逐漸褪去發展之初的浮躁、回歸理性,表面上看相關人才需求與身價似乎正在回落。但事實上,正

Kettle學習筆記003之資料庫連線配置

1.  Kettle的資料庫連線方式常用的是jdbc連線方式 2. 資料庫連線配置:      在任務設計器Spoon的主物件樹的DB連線選項進行配置 3. 共享資料庫連線      a. 新建的資料庫連線只能在當

關於MySQL資料庫編碼修復相關問題

  本篇主要是本人在實際開發過程中遇到的MySQL字元編碼等bug修復相關問題。   在使用下列語句在執行資料庫表通過flask-sqlacodegen 進行ORM對映成模型類的時候發生的bug: flask-sqlacodegen mysql+pymysql://root:[email 

Android資料庫GreenDao配置

1.新增依賴 在moudle的build.gradle中新增依賴 implementation 'org.greenrobot:greendao:3.2.2' implementation 'org.greenrobot:greendao-generator:3.2.2'

centos 7.4安裝jdk配置相關

下載jdk http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html 進入伺服器/usr/local目錄 cd /usr/local/ 建立jav

H3 GPIO 暫存器配置相關函式(ioremap iounmap)

對暫存器操作遵循一個原則:  讀------改 ------寫 1. 配置暫存器操作 (ioremap   //  iounmap)         #define BASE_

從SpringBoot原始碼分析 配置檔案的載入原理和優先順序

從SpringBoot原始碼分析 配置檔案的載入原理和優先順序 本文從SpringBoot原始碼分析 配置檔案的載入原理和配置檔案的優先順序     跟入原始碼之前,先提一個問題:   SpringBoot 既可以載入指定目錄下的配置檔案獲取配置項,也可

Django框架(二)—— 基本配置:app註冊、模板配置、靜態檔案配置資料庫連線配置post和get

app註冊、模板配置、靜態檔案配置、資料庫連線配置post和get 一、app 在Django中,APP可以用以下類比 大學 --------------------專案 計算機學院------------app01 土木學院 ------------ app02 1、app建立 方