TEZUI搭建

官方 TEZUI 需要TEZ6+hadoop2.6.0

成都创新互联专注于商都网站建设服务及定制，我们拥有丰富的企业做网站经验。 热诚为您提供商都营销型网站建设，商都网站制作、商都网页设计、商都网站官网定制、小程序设计服务，打造商都网络公司原创品牌,更为您提供商都网站排名全网营销落地服务。

其实 TEZUI  TEZ0.53+hadoop2.4.0+也可以玩 只要hadoop有timelineserver

但是hadoop2.4.0 hadoop2.5.0 timelineserver不支持跨域请求。。所以用ambari2.2中的tez view来搭建可以实现，并且方便快捷。

tez-site.xml

      tez.lib.uris

      hdfs:///apps/tez-0.5.3/tez-0.5.3.tar.gz

      tez.task.generate.counters.per.io

      true

  Log history using the Timeline Server

  tez.history.logging.service.class

  org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService

  Publish configuration information to Timeline server.

  tez.runtime.convert.user-payload.to.history-text

  true

yarn-site.xml

加上

  Indicate to clients whether Timeline service is enabled or not.

  If enabled, the TimelineClient library used by end-users will post entities

  and events to the Timeline server.

  yarn.timeline-service.enabled

  true

  The hostname of the Timeline service web application.

  yarn.timeline-service.hostname

  192.168.117.117

  yarn.resourcemanager.system-metrics-publisher.enabled

  true

  Enables cross-origin support (CORS) for web services where

  cross-origin web response headers are needed. For example, javascript making

  a web services request to the timeline server.

  yarn.timeline-service.http-cross-origin.enabled

  true 详细信息见

http://search-hadoop.com/m/tQlTsMD%26subj=Tez+nbsp+taskcount+log+visualization 

  Address for the Timeline server to start the RPC server.

  yarn.timeline-service.address

  ${yarn.timeline-service.hostname}:10201

  The http address of the Timeline service web application.

  yarn.timeline-service.webapp.address

  ${yarn.timeline-service.hostname}:8188

  The https address of the Timeline service web application.

  yarn.timeline-service.webapp.https.address

  ${yarn.timeline-service.hostname}:2191

yarn timelineserver start  可以启动timelineserver

Tez config 

https://issues.apache.org/jira/browse/TEZ-2294

Tez 优化

Listing some details at very high level,

- Set "tez.task.generate.counters.per.io=true" to get more details on the task counters. Basically this starts printinng the counters per edge, which can be a lot more useful for debugging.

- In case you want to avoid container launches etc when you analyze for first time, try hive.prewarm.enabled=true & hive.prewarm.numcontainers=

- Container reuse is enabled by default in tez. (tez.am.container.idle.release-timeout-min.millis, tez.am.container.idle.release-timeout-max.millis controls the amount of time a container is held by AM before releasing it)

- Set tez.runtime.io.sort.mb appropriately to avoid spills (you can check task counters in the logs to find out the spills and adjust it accordingly)

- Set tez.runtime.sort.threads=2 to enable PipelinedSorter which is a lot performant than DefaultSorter (this is the default in master branch. But if you are using earlier releases, you can turn it on by setting tez.runtime.sort.threads=2).

- Set tez.runtime.compress=true and set tez.runtime.compress.codec (SnappyCodec is preferred, but it is upto you to choose)

- Set tez.runtime.shuffle.keep-alive.enabled=true in case you have shuffle heavy workload. This reduces number of connections in shuffle.

- Adjust memory allocated to different inputs/outputs based on tez.task.scale.memory.ratios (but this is more of expert level setting which you might want to touch after nailing down any memory pressure)

- Adjusting shuffle buffers are also possible, but would advise only when you nail down an issue related to shuffle/merge codepath.

- Set "tez.runtime.optimize.local.fetch=true" to bypass http fetches (when data is locally present)

Feel free to refer to https://github.com/t3rmin4t0r/tez-autobuild/blob/master/tez-site.xml for any commonly used settings for benchmarks.

Rajesh,

   What are the problems with having tez.runtime.shuffle.keep-alive.enabled and tez.runtime.optimize.local.fetch set to true always by default?

@r7raul1984, would you mind filing a documentation jira for your question. The list that Rajesh provided might be good to formalize into a doc and/or wiki. 

Also, please take a look at https://issues.apache.org/jira/browse/TEZ-2294 to see all the list of parameters. If you see something off or not clear enough, please add your comments to the jira.

@Rohini,

We recently changed tez.runtime.optimize.local.fetch to true as the default value in master. The feature was introduced and probably kept as false initially as it had not been fully battle tested.

The latter I am assuming depends on how many open connections a cluster’s setup can sustain and needs to be tuned in combination with “tez.runtime.shuffle.keep-alive.max.connections”.  Good point on whether we should make this true by default. Will wait for @Rajesh/@Gopal/@Sid to chime in and they can open a new jira if this is generally beneficial in most setups.

>   What are the problems with having

>tez.runtime.shuffle.keep-alive.enabled and

>tez.runtime.optimize.local.fetch set to true always by default?

>   What are the problems with having

>tez.runtime.shuffle.keep-alive.enabled and

>tez.runtime.optimize.local.fetch set to true always by default?

Nothing has failed due to these so far - we