This is one of my end applications of the Overo.  I want to mount this device onto a rover/uav and get real-time streaming, good quality video.  I also didn't want the operation to dominate the CPU.  Ideally my other applications will be able to run along side the video stream.  Therefore, I had to use the DSP.  So once I got it running it was time to stream.

這是Overo的其中一個終端應用程式。我想要這個裝置連線在rover/uav上面，並且獲取到實時的質量好的視訊流媒體。我不想這個功能佔用CPU。這樣我的其他應用程式可以理想地與視訊流媒體一起執行。

Scott's discussion and example pipelines were great but I had previously tested some gstreamer code on Linux machines that I wanted to try.  I got the code fromhere, there's a whole bunch of sample files.  What I initially did was change the server

scott討論和管道使用的例子非常好：如下：

GStreamer Pipelines

On the Ubuntu client with an of IP 192.168.10.4, run a GStreamer command like the following

$ gst-launch-0.10 -v udpsrc port=4000 \
    caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' \
    ! rtph264depay ! ffdec_h264 ! xvimagesink sync=false

Then on the gumstix side run this, substituting your workstation's IP address

[email protected]:~# gst-launch -v videotestsrc ! video/x-raw-yuv,width=640,height=480 \
    ! TIVidenc1 codecName=h264enc engineName=codecServer ! rtph264pay pt=96 \
    ! udpsink host=192.168.10.4 port=4000

在imx6端，streaming to PC：

[email protected] ~$ gst-launch -v videotestsrc ! video/x-raw-yuv,width=640,height=
480 ! vpuenc codec=avc ! rtph264pay pt=96 ! udpsink host=172.21.78.36 port=4000

在ubuntu PC端  接收：

[email protected]:Desktop$ gst-launch -vvv udpsrc port=4000 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264" ! rtph264depay ! ffdec_h264 ! xvimagesink

這裡理解一下：（所有pipeline屬性都可以通過gst-inspect來檢視）

在imx6端streaming一個基於udp&rtp的視訊測試源videotestsrc,並且設定這個測試源的屬性或者叫caps。

格式為video/x-raw-yuv,width=640,height=480,這些屬性是屬於videotestsrc的pipeline。

raw的視訊源通過imx6的硬體編碼器vpuenc（ gst-fsl-plugins裡面的），這個元件的sink&sout是有規定範圍的:

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw-yuv
                 format: TNVP
      video/x-raw-yuv
                 format: NV12
      video/x-raw-yuv
                 format: I420

  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/mpeg
            mpegversion: 4
      video/x-h263
      video/x-h264
      image/jpeg
顯然video/x-raw-yuv符合規定的sink範圍。SRC可以定義為上面幾種編碼方式。這裡定義為codec=avc（即video/x-h264）

OK，編碼為H264之後，再通過rtph264pay元件打包為rtp包發出去。其中定義The payload type of the packets 96.同樣這個

rtph264pay也是有規定的，他的sink必須要為video/x-h264。傳輸出去的caps就是rtp視訊流媒體，9W的clock，H264的編碼。如下：

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      application/x-rtp
                  media: video
                payload: [ 96, 127 ]
             clock-rate: 90000
          encoding-name: H264

  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h264

最後src out出來的給udpsink元件，即封裝好的rtp包通過udp協議來傳輸。並且指定這個元件的相關屬性如host port。

OK===>以上的測試例子：要先在接收端執行命令，然後再imx6進行傳送。 如何解決：sprop-parameter-sets

上面的命令要先client端接收，然後再server傳送。“不能先發送command，再接收command”。如果不想要這樣的限制。需在接收端的caps裡面新增sprop-parameter-sets引數。這個引數是從傳送command中得到的。
[email protected]:Desktop$ gst-launch -v udpsrc port=4000 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264,sprop-parameter-sets=(string)\"Z0JAHqaAoD2QAA\\=\\=\\,aM4wpIAA\" " ! rtph264depay ! ffdec_h264 ! xvimagesink

With that change you can start and stop the client or server in any order.

接收端的解析：

[email protected]:Desktop$ gst-launch -vvv udpsrc port=4000 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264" ! rtph264depay ! ffdec_h264 ! xvimagesink sync=false

上面等價於
[email protected]:Desktop$ gst-launch udpsrc uri=udp://172.21.78.36:4000 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264" ! rtph264depay ! ffdec_h264 ! xvimagesink sync=false

接收端等待接收udpsrc，這個源out出來的是資料是通過udp傳輸的，裡面的資料是rtp&H264封裝的。所以指定了其caps。

接著用rtph264depay將rtp解包，剩下H264的資料，再通過ffdec_h264進行解碼，src out為raw rgb或yuv

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw-rgb
      video/x-raw-yuv

  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-h264
                  width: [ 16, 4096 ]
                 height: [ 16, 4096 ]
              framerate: [ 0/1, 2147483647/1 ]

OK,我們再來看一個例子：將ubuntu的camera streaming to imx6 displaying。延遲非常少，但ubuntu很佔CPU。

OK===>
ubuntu:
[email protected]:Desktop$ gst-launch v4l2src device=/dev/video0 ! 'video/x-raw-yuv,width=640,height=480' !  x264enc pass=qual quantizer=20 tune=zerolatency ! rtph264pay ! udpsink host=172.21.78.63 port=4000 -v

可以將device=/dev/video0省略，因為camera預設對應video0

  device              : Device location
                        flags: readable, writable
                        String. Default: "/dev/video0"

imx6:
gst-launch -v udpsrc port=4000 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000,encoding-name=(string)H264" ! rtph264depay  ! vpudec ! mfw_v4lsink  sync=false

可以簡化為如下：

imx6:
gst-launch -v udpsrc port=4000 caps="application/x-rtp" ! rtph264depay  ! vpudec ! mfw_v4lsink  sync=false

並且VPUDEC可以自適應的解析H264，不想要指定具體的codec。

注意：與第一個例子，imx6 streaming to PC。相比。

第二個PC camera streaming to imx6 displaying。不用指定sprop-parameter-sets，都可以無限制的，client server誰先執行都可以。不知道為什麼？

Without that change you canalso start and stop the client or server in any order.

========================================

我們再來看看原文作者的方法：

這裡的方法借鑑了gst-plugins-good/tests/examples/rtp裡面的例子，相應HW codec進行更正。

gst-launch -v gstrtpbin name=rtpbin v4l2src ! queue ! videorate ! ffmpegcolorspace ! video/x-raw-yuv,width=640,height=480,framerate=15/1 ! TIVidenc1 codecName=h264enc engineName=codecServer ! rtph264pay ! rtpbin.send_rtp_sink_0 rtpbin.send_rtp_src_0 ! udpsink port=5000 host=$DEST ts-offset=0 name=vrtpsink rtpbin.send_rtcp_src_0 ! udpsink port=5001 host=$DEST sync=false async=false name=vrtcpsink udpsrc port=5005 name=vrtpsrc ! rtpbin.recv_rtcp_sink_0

檢視一下gstrtpbin的用法：

Pad Templates:                                                                                                                                                  
  SINK template: 'recv_rtp_sink_%d'                                                                                                                             
    Availability: On request                                                                                                                                    
      Has request_new_pad() function: gst_rtp_bin_request_new_pad                                                                                               
    Capabilities:                                                                                                                                               
      application/x-rtp                                                                                                                                         
                                                                                                                                                                
  SINK template: 'recv_rtcp_sink_%d'                                                                                                                            
    Availability: On request                                                                                                                                    
      Has request_new_pad() function: gst_rtp_bin_request_new_pad                                                                                               
    Capabilities:                                                                                                                                               
      application/x-rtcp                                                                                                                                        
                                                                                                                                                                
  SINK template: 'send_rtp_sink_%d'                                                                                                                             
    Availability: On request                                                                                                                                    
      Has request_new_pad() function: gst_rtp_bin_request_new_pad                                                                                               
    Capabilities:                                                                                                                                               
      application/x-rtp                                                                                                                                         
                                                                                                                                                                
  SRC template: 'recv_rtp_src_%d_%d_%d'                                                                                                                         
    Availability: Sometimes                                                                                                                                     
    Capabilities:                                                                                                                                               
      application/x-rtp                                                                                                                                         
                                                                                                                                                                
  SRC template: 'send_rtcp_src_%d'                                                                                                                              
    Availability: On request                                                                                                                                    
      Has request_new_pad() function: gst_rtp_bin_request_new_pad                                                                                               
    Capabilities:                                                                                                                                               
      application/x-rtcp                                                                                                                                        
                                                                                                                                                                
  SRC template: 'send_rtp_src_%d'                                                                                                                               
    Availability: Sometimes                                                                                                                                     
    Capabilities:                                                                                                                                               
      application/x-rtp

然後我們根據gst-plugins-good裡面的例子，逐一分析：

先看這個指令碼：server-alsasrc-PCMA.sh

#!/bin/sh
#
# A simple RTP server
#  sends the output of autoaudiosrc as alaw encoded RTP on port 5002, RTCP is sent on
#  port 5003. The destination is 127.0.0.1.
#  the receiver RTCP reports are received on port 5007
#
# .--------.    .-------.    .-------.      .----------.     .-------.
# |audiosrc|    |alawenc|    |pcmapay|      | rtpbin   |     |udpsink|  RTP
# |       src->sink    src->sink    src->send_rtp send_rtp->sink     | port=5002
# '--------'    '-------'    '-------'      |          |     '-------'
#                                           |          |
#                                           |          |     .-------.
#                                           |          |     |udpsink|  RTCP
#                                           |    send_rtcp->sink     | port=5003
#                            .-------.      |          |     '-------' sync=false
#                 RTCP       |udpsrc |      |          |               async=false
#               port=5007    |     src->recv_rtcp      |
#                            '-------'      '----------'

# change this to send the RTP data and RTCP to another host
DEST=127.0.0.1

#AELEM=autoaudiosrc
AELEM=audiotestsrc

# PCMA encode from an the source
ASOURCE="$AELEM ! audioconvert"
AENC="alawenc ! rtppcmapay"

gst-launch -v gstrtpbin name=rtpbin \
     $ASOURCE ! $AENC ! rtpbin.send_rtp_sink_0  \
            rtpbin.send_rtp_src_0 ! udpsink port=5002 host=$DEST                      \
            rtpbin.send_rtcp_src_0 ! udpsink port=5003 host=$DEST sync=false async=false \
         udpsrc port=5007 ! rtpbin.recv_rtcp_sink_0

這裡使用到了gstrtpbin這個元件。這個元件有3個sink，3個sout。

1、編碼後的資料給這個元件的send_rtp_sink，再通過send_rtp_src給udpsink傳送rtp資料出去

2、同時通過send_rtcp_src傳送rtcp資料給udpsink出去。

3、用recv_rtcp_sink接收udpsrc的資料。

這個server，用到4個元素。其中兩個用來發送send(sink->sout)rtp資料，一個用來send（sout）rtcp資料。一個用來recv（sink）rtcp資料。

剩下兩個用來接收rtp資料的元素用於client端。recv（sink->sout）。======>一共六個！剩下兩個的用法是在client裡面講述。

另外：udpsink的埠，rtp和rtcp獨立。udpsrc的埠，用來接收rtcp也是獨立開的。

帶續未完～～～～～～～～～～

You'll notice I removed all the audio chunks.  At the moment I don't have a mic so what's the point.  I did however leave in all the control flow (RTCP).  This is nice to sync up the streams.  I don't fully understand what's going on but I have played with the timing and seen the video change using the RTCP settings, basically performance seemed to improve.  In the above code, I setDEST to the ip address or DNS name of my client.nbsp;

The first client I tested was a windows maching with VLC and this client fileclient-PCMA.sdp.  Things worked ok (as long as you open up the client first and before the connection times out you fire up the server).  However, there was a pretty good lag (roughly 1 sec) in the video.  So I tried out the linux sh fileclient-H264.sh and things looked much better.  I also don't know if it had anything to do with it but I setup the ntpdate package on the Overo so that it would correctly set the date (I think the RTCP uses time stamping to improve performance but I could be wrong).  I figured having the correct date/time would be good.  To get it to display the correct time zone I used  export TZ=PST8PDT.  However I can't get it to be persistent.  I tried putting it in a file at /etc/TZ but that didn't work, I'll play around with it later.

Now I wanted to stream stuff to a windows machine.  I did a quick search and foundthis website.  The OSSBuild comes with some precompiled binaries, one of which is gst-launch.  With that binary I used the following command

gst-launch -v gstrtpbin name=rtpbin latency=50 udpsrc caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, sprop-parameter-sets=(string)\"Z0KAHukBQHpCAAAH0AAB1MAIAA\\=\\=\\,aM48gAA\\=\"" port=5000 ! rtpbin.recv_rtp_sink_0 rtpbin. ! rtph264depay ! ffdec_h264 ! ffmpegcolorspace ! autovideosink udpsrc port=5001 ! rtpbin.recv_rtcp_sink_0 rtpbin.send_rtcp_src_0 ! udpsink port=5005 host=$OVERO sync=false async=false

I set OVERO to be the ip/dns name of the Overo wifi adapter.  You'll notice I added thesprop-parameter-sets as per Scott'sinstructions so that client or server can be started first.  I also messed around with thelatency value.  It's definitely dependent on the speed of the client.  Some I could set low, others needed to be higher.  I believe it adds a bit of lag to the video.  Anyway, what I got was really good video with very little lag.  Also, one of the reasons I was doing this test was to see the CPU load.  So ... with 640x480 at 15 frames per second over wifi I got about 20% cpu load.

top - 20:17:26 up 37 min,  2 users,  load average: 0.08, 0.06, 0.01
Tasks:  67 total,   1 running,  66 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.0%us,  0.8%sy,  0.0%ni, 95.9%id,  0.8%wa,  0.6%hi,  0.0%si,  0.0%st
Mem:    469008k total,    71380k used,   397628k free,     3200k buffers
Swap:        0k total,        0k used,        0k free,    46712k cached  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1013 root      20   0 87720 7660 5132 S 19.2  1.6   0:34.72 gst-launch-0.10

This is pretty sweet, ready to test other stuff while streaming video. I still need to play with the latency. It's not very large but definitely noticable. I haven't determined if it's the client or the server. Now that I have an LCD screen (thanks Don) I'm going to try looping back the video and see if the latency is apparent, stay tuned.

Last modified Sun, 4 Sep, 2011 at 16:41