系統技術非業餘研究 » 為什麼Erlang是軟實時的
之前在微博上@老師木 同學發起了個關於Erlang搶佔式排程的討論,相對於其他語言Erlang是真正的搶佔式排程服務,這也是erlang能夠號稱軟實時的很重要的原因之一,另外二個原因分別是erlang的GC是針對每個程序的,每次GC通常只需要收集幾K或者幾十個物件;BIF是用trap機制來保證公平的。
今天在微博上看到erlang排程器的工作細節的文章來源見這裡。這篇文章很準確的描述了Erlang排程器是如何工作的,很好的澄清了大家對各種語言排程器的誤解。 本來我前段時間想自己寫,看來沒必要了,就順手貼出來給大家。
How Erlang does scheduling
In this, I describe why Erlang is different from most other language runtimes. I also describe why it often forgoes throughput for lower latency.
TL;DR – Erlang is different from most other language runtimes in that it targets different values. This describes why it often seem to perform worse if you have few processes, but well if you have many.
From time to time the question of Erlang scheduling gets asked by different people. While this is an abridged version of the real thing, it can act as a way to describe how Erlang operates its processes. Do note that I am taking Erlang R15 as the base point here. If you are a reader from the future, things might have changed quite a lot—though it is usually fair to assume things only got better, in Erlang and other systems.
Toward the operating system, Erlang usually has a thread per core you have in the machine. Each of these threads runs what is known as a scheduler. This is to make sure all cores of the machine can potentially do work for the Erlang system. The cores may be bound to schedulers, through the +sbt flag, which means the schedulers will not “jump around” between cores. It only works on modern operating systems, so OSX can’t do it, naturally. It means that the Erlang system knows about processor layout and associated affinities which is important due to caches, migration times and so on. Often the +sbt flag can speed up your system. And at times by quite a lot.
The +A flag defines a number of async threads for the async thread pool. This pool can be used by drivers to block an operation, such that the schedulers can still do useful work while one of the pool-threads are blocked. Most notably the thread pool is used by the file driver to speed up file I/O – but not network I/O.
While the above describes a rough layout towards the OS kernel, we still need to address the concept of an Erlang (userland) process. When you call spawn(fun worker/0) a new process is constructed, by allocating its process control block in userland. This usually amounts to some 600+ bytes and it varies from 32 to 64 bit architectures. Runnable processes are placed in the run-queue of a scheduler and will thus be run later when they get a time-slice.
Before diving into a single scheduler, I want to describe a little bit about how migration works. Every once in a while, processes are migrated between schedulers according to a quite intricate process. The aim of the heuristic is to balance load over multiple schedulers so all cores get utilized fully. But the algorithm also considers if there is enough work to warrant starting up new schedulers. If not, it is better to keep the scheduler turned off as this means the thread has nothing to do. And in turn this means the core can enter power save mode and get turned off. Yes, Erlang conserves power if possible. Schedulers can also work-steal if they are out of work. For the details of this, see [1].
IMPORTANT: In R15, schedulers are started and stopped in a “lagged” fashion. What this means is that Erlang/OTP recognizes that starting a scheduler or stopping one is rather expensive so it only does this if really needed. Suppose there is no work for a scheduler. Rather than immediately taking it to sleep, it will spin for a little while in the hope that work arrives soon. If work arrives, it can be handled immediately with low latency. On the other hand, this means you cannot use tools like top(1) or the OS kernel to measure how efficient your system is executing. You must use the internal calls in the Erlang system. Many people were incorrectly assuming that R15 was worse than R14 for exactly this reason.
Each scheduler runs two types of jobs: process jobs and port jobs. These are run with priorities like in an operating system kernel and is subject to the same worries and heuristics. You can flag processes to be high-priority, low-priority and so on. A process job executes a process for a little while. A port job considers ports. To the uninformed, a “port” in Erlang is a mechanism for communicating with the outside world. Files, network sockets, pipes to other programs are all ports. Programmers can add “port drivers” to the Erlang system in order to support new types of ports, but that does require writing C code. One scheduler will also run polling on network sockets to read in new data from those.
Both processes and ports have a “reduction budget” of 2000 reductions. Any operation in the system costs reductions. This includes function calls in loops, calling built-in-functions (BIFs), garbage collecting heaps of that process[n1], storing/reading from ETS, sending messages (The size of the recipients mailbox counts, large mailboxes are more expensive to send to). This is quite pervasive, by the way. The Erlang regular expression library has been modified and instrumented even if it is written in C code. So when you have a long-running regular expression, you will be counted against it and preempted several times while it runs. Ports as well! Doing I/O on a port costs reductions, sending distributed messages has a cost, and so on. Much time has been spent to ensure that any kind of progress in the system has a reduction cost[n2].
In effect, this is what makes me say that Erlang is one of a few languages that actually does preemptive multitasking and gets soft-realtime right. Also it values low latency over raw throughput, which is not common in programming language runtimes.
To be precise, preemption[2] means that the scheduler can force a task off execution. Everything based on cooperation cannot do this: Python twisted, Node.js, LWT (Ocaml) and so on. But more interestingly, neither Go (golang.org) nor Haskell (GHC) is fully preemptive. Go only switches context on communication, so a tight loop can hog a core. GHC switches upon memory allocation (which admittedly is a very common occurrence in Haskell programs). The problem in these systems are that hogging a core for a while—one might imagine doing an array-operation in both languages—will affect the latency of the system.
This leads to soft-realtime[3] which means that the system will degrade if we fail to meet a timing deadline. Say we have 100 processes on our run-queue. The first one is doing an array-operation which takes 50ms. Now, in Go or Haskell/GHC[n3] this means that tasks 2-100 will take at least 50ms. In Erlang, on the other hand, task 1 would get 2000 reductions, which is sub 1ms. Then it would be put in the back of the queue and tasks 2-100 would be allowed to run. Naturally this means that all tasks are given a fair share.
Erlang is meticously built around ensuring low-latency soft-realtime properties. The reduction count of 2000 is quite low and forces many small context switches. It is quite expensive to break up long-running BIFs so they can be preempted mid-computation. But this also ensures an Erlang system tend to degrade in a graceful manner when loaded with more work. It also means that for a company like Ericsson, where low latency matters, there is no other alternative out there. You can’t magically take another throughput-oriented language and obtain low latency. You will have to work for it. And if low latency matters to you, then frankly not picking Erlang is in many cases an odd choice.
[1] “Characterizing the Scalability of Erlang VM on Many-core Processors” http://kth.diva-portal.org/smash/record.jsf?searchId=2&pid=diva2:392243
[2] http://en.wikipedia.org/wiki/Preemption_(computing)
[3] http://en.wikipedia.org/wiki/Real-time_computing
[n1] Process heaps are per-process so one process can’t affect the GC time of other processes too much.
[n2] This section is also why one must beware of long-running NIFs. They do not per default preempt, nor do they bump the reduction counter. So they can introduce latency in your system.
[n3] Imagine a single core here, multicore sort of “absorbs” this problem up to core-count, but the problem still persists.
(Smaller edits made to the document at Mon 14th Jan 2013)
祝玩得開心!
Post Footer automatically generated by wp-posturl plugin for wordpress.
相關推薦
系統技術非業餘研究 » Erlang 17.5引入+hpds命令列控制程序預設字典大小
Erlang 17.5釋出引入控制程序預設字典大小的命令列引數: Erlang/OTP 17.5 has been released Written by Henrik, 01 Apr 2015 Some highlights of the release are: ERTS: Added co
系統技術非業餘研究 » Erlang R16B03釋出,R17已發力
Erlang R16B03釋出了,通常03版本是bug fix版本,進入生產版本,官方的說明如下: OTP R16B03 is a service release with mostly a number of small corrections and user contributions. B
系統技術非業餘研究 » Erlang R13B04 Installation
R13B04後erlang的原始碼編譯為了考慮移植性,就改變了編譯方式,以下是官方wiki上的安裝文件: 1. Cloning Here are the basic steps to build Erlang/OTP in the Git repository. Start by cloning:
系統技術非業餘研究 » Erlang R15的記憶體delayed dealloc特性對訊息密集型程式的影響
在新的NUMA體系結構下,每個CPU都有自己的本地記憶體,如果要訪問其他CPU的記憶體,那算remote了,要走CPU之間的QPI通道,通常這樣速度會有40%的下降。 那麼對於多執行緒的程式來講,這個硬體的變化對軟體也有很大的影響。在多執行緒程式裡面,通常一個執行緒會為一個物件分配記憶體,然後把這
系統技術非業餘研究 » Erlang R17新特性淺評
Erlang R17RC2 原始碼已經就緒, 參見 這裡 後續版本的釋出時間,官方的時間安排參見 這裡,摘抄如下: Preliminary dates for the upcoming release: Release: erts, emu,comp |Code stop
系統技術非業餘研究 » Erlang R16支援帶顏色的控制檯
Erlang通過fix tty驅動的過濾,在R16版本支援帶顏色的控制檯,這個特性在我們做各種監控工具高亮非常有幫助,參見R16的Readme: Support ANSI in console Unix platforms will no longer filter control sequenc
系統技術非業餘研究 » erlang coredump問題
早上成立濤同學問道: : :)我們最近發生了幾次宕機。。節點無緣無故就沒有了。也沒有crash dump,也不知道任何線索。 我們知道erlang的VM在正常運作的時候,如果發現erlang程式的異常或者虛擬機器資源不夠如記憶體不夠的時候,會產生erl_crash.dump檔案,裡面把crash的
系統技術非業餘研究 » Erlang open_port極度影響效能的因素
Erlang的port相當於系統的IO,打開了Erlang世界通往外界的通道,可以很方便的執行外部程式。 但是open_port的效能對整個系統來講非常的重要,我就帶領大家看看open_port影響效能的因素。 首先看下open_port的文件: {spawn, Command} Star
系統技術非業餘研究 » Erlang節點重啟導致的incarnation問題
今天晚上mingchaoyan同學在線上問以下這個問題: 152489 =ERROR REPORT==== 2013-06-28 19:57:53 === 152490 Discarding message {send,<<19 bytes>>} from <0.8
系統技術非業餘研究 » Erlang port巧用環境變數
Erlang與外面世界的互動主要通過port來進行的,特別是和外部程式的協作,通常是通過管道進行的。 基本上有2種方法可以呼叫外部程式: 1. os:cmd 2. erlang:open_port, 這二種方式各有利弊,先看文件: os:cmd的文件參見這裡 cmd(Command) ->
系統技術非業餘研究 » Erlang新新增選項 +zerts_de_busy_limit 控制節點間通訊的資料量
erlang節點間通訊預設是通過tcp通道進行的, 而且每對節點間只有一個tcp連結,所有的rpc和內建的類似monitor這樣的訊息也都是通過這個通道進行的. 當資料量過大的時候, 系統就會發出 busy distribution port警告, 同時限制資料的吞吐. 這個值預設是128k. 現
系統技術非業餘研究 » Erlang程式碼反編譯以及檢視彙編碼
Erlang的程式碼是先翻譯成abstract_code,再到目的碼的,如果有符號資訊很容易恢復原始碼,通常我們部署系統的時候需要把符號資訊去掉,reltool就可以幹這個事情! 我們演示下: $ cat server.erl -module(server). -compile(export
系統技術非業餘研究 » Erlang Shell實用小技巧
Erlang Shell下有很多內建的命令,在平時互動的時候很好用,文件裡面都是一行帶過,大家可能沒什麼感覺。 我來重點講解和演示下: $ erl Erlang R14B04 (erts-5.8.5) [/source] [smp:2:2] [rq:2] [async-threads:0] [h
系統技術非業餘研究 » ”Erlang supervisor 極其白痴的 Bug“的澄清
2008-05-26的時候, 著名的Trustno1發表了這篇文章 http://www.iteye.com/topic/197097 抱怨Erlang supervisor 極其白痴的一個bug. 今天 @淘李福 同學重新提起這個事情: 翻到一個老帖子: http://www.iteye.com
系統技術非業餘研究 » Erlang叢集RPC通道擁塞問題及解決方案
Erlang的叢集預設情況下是全聯通的,也就是當一個節點加入叢集的時候,介紹人會推薦叢集裡面所有的節點主動來和新加入的節點建立聯絡, 效果如下圖: 我們這次不講如何避免全聯通而是來講這個節點間通道的問題。 我們知道erlang的訊息傳送是透明的,只要呼叫Pid!Msg, 虛擬機器和叢集的基礎設
系統技術非業餘研究 » Erlang match_spec引擎介紹和應用
match_spec是什麼呢? A “match specification” (match_spec) is an Erlang term describing a small “program” that will try to match something (either the para
系統技術非業餘研究 » Erlang虛擬機器基礎設施dtrace探測點介紹和使用
最新的Erlang虛擬機器(R15B01)很大的一個改進就是加入了對dtrace探測點的支援了, 具體參見這裡, 主要目標是方便在生產實踐中定位複雜的效能問題。 目前Erlang的虛擬機器的探測點支援Linux的systemtap和freebsd的dtrace,我們剛好能夠享受的到。 作者Scot
系統技術非業餘研究 » erlang的abstract code
erlang的abstract code是編譯的中間程式碼,很多工具如 erl_pp lint什麼的都是根據這個做調整的。還有進一步的parse_transform也是基於它的。 所以,瞭解它非常重要。 erts user guide裡面詳細了描述了它的定義。我這裡展示的是如何獲取到某個模組的ab
系統技術非業餘研究 » Erlang原始碼彙編格式
我們在編碼的時候, 通常會好奇, 這時候需要觀察erl原始碼生成的VM opcode. Erlang的VM是register based的VM, 生產的opcode很容易理解. 生成彙編格式有2種方式: 1. 從原始碼生成抽象碼. erlc +”‘S'” mod.erl, 生成mod.S 2.
系統技術非業餘研究 » Erlang 網路密集型伺服器的瓶頸和解決思路
最近我們的Erlang IO密集型的伺服器程式要做細緻的效能提升,從每秒40萬包處理提升到60萬目標,需要對程序和IO排程器的原理很熟悉,並且對行為進行微調,花了不少時間參閱了相關的文件和程式碼。 其中最有價值的二篇文章是: 1. Characterizing the Scalability of