從網路rtp封包中分離hevc/h265
上篇博文介紹了修復ffmpeg分離PS流的BUG,有同學關心定位BUG時抓網路RTP包分離HEVC碼流的問題,本次重開一博文介紹此問題,並在結尾附上原始碼。
一、rtpdump檔案解析
使用tcpdump或wireshark抓取rtp網路包以後存為pcap檔案,然後用wireshark匯出位更簡潔的rtpdump檔案。方法如下:1)在wireshark中,滑鼠右擊其中一個網路包,在彈出選單中選擇“解碼為”,將該網路包的UDP埠設定為RTP協議。2)依次選擇選單欄“電話”\“RTP”\“RTP流”,根據需要選定一個RTP流,然後點選下方的Export匯出為rtpdump格式的檔案,我們下面的分析都基於此檔案。
rtpdump檔案以"#!rtpplay1.0
addrss/port\n"為檔案頭起始,其後跟一個結構頭:
typedef struct { struct timeval32 { uint32_t tv_sec; /* start of recording (GMT) (seconds) */ uint32_t tv_usec; /* start of recording (GMT) (microseconds)*/ } start; uint32_t source; /* network source (multicast address) */ uint16_t port; /* UDP port */ uint16_t padding; /* padding */ } RD_hdr_t;
在上述檔案頭之後,就是一個個的rtp buffer,每個rtp buffer都有一個buffer頭,buffer結構如下:
typedef struct { uint16_t length; /* length of packet, including this header (may be smaller than plen if not whole packet recorded) */ uint16_t plen; /* actual header+payload length for RTP, 0 for RTCP */ uint32_t offset; /* milliseconds since the start of recording */ } RD_packet_t; typedef union { struct { RD_packet_t hdr; char data[8000]; } p; char byte[8192]; } RD_buffer_t;
RD_buffer_t中的data即為rtp資料。有了上述資料結構就可以寫出一個解析rtpdump檔案頭的函式:
/*
* Read header. Return -1 if not valid, 0 if ok.
*/
int RD_header(FILE *in, struct sockaddr_in *sin, int verbose)
{
RD_hdr_t hdr;
time_t tt;
char line[80], magic[80];
if (fgets(line, sizeof(line), in) == NULL) return -1;
sprintf(magic, "#!rtpplay%s ", RTPFILE_VERSION);
if (strncmp(line, magic, strlen(magic)) != 0) return -1;
if (fread((char *)&hdr, sizeof(hdr), 1, in) == 0) return -1;
hdr.start.tv_sec = ntohl(hdr.start.tv_sec);
hdr.port = ntohs(hdr.port);
if (verbose) {
struct tm *tm;
struct in_addr in;
in.s_addr = hdr.source;
tt = (time_t)(hdr.start.tv_sec);
tm = localtime(&tt);
strftime(line, sizeof(line), "%c", tm);
printf("Start: %s\n", line);
printf("Source: %s (%d)\n", inet_ntoa(in), hdr.port);
}
if (sin && sin->sin_addr.s_addr == 0) {
sin->sin_addr.s_addr = hdr.source;
sin->sin_port = htons(hdr.port);
}
return 0;
} /* RD_header */
當然也會寫出一個解析rtp buffer的函式:
int RD_read(FILE *in, RD_buffer_t *b)
{
/* read packet header from file */
if (fread((char *)b->byte, sizeof(b->p.hdr), 1, in) == 0) {
/* we are done */
return 0;
}
/* convert to host byte order */
b->p.hdr.length = ntohs(b->p.hdr.length) - sizeof(b->p.hdr);
b->p.hdr.offset = ntohl(b->p.hdr.offset);
b->p.hdr.plen = ntohs(b->p.hdr.plen);
/* read actual packet */
if (fread(b->p.data, b->p.hdr.length, 1, in) == 0) {
perror("fread body");
return 0;
}
return b->p.hdr.length;
} /* RD_read */
上述程式碼改自自rtptools。
二、從RTP中取出HEVC碼流
HEVC的NAL跟H264相比最大的區別是,HEVC NAL頭有2個位元組。由於HEVC PACI packets通常較少使用,所以HEVC的RTP解析一般要解決2個主要的問題:1)一個NALU由多個RTP包組成;2)一個RTP包含有多個NALU。除了這2個問題,其他的RTP包基本上在其RTP負載的前面加上00 00 00 01頭即可。
1)一個NALU由多個RTP組成,即fragmentation unit,簡稱FU,它有一個位元組的FU header。
/*
* decode the HEVC payload header according to section 4 of draft version 6:
*
* 0 1
* 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* |F| Type | LayerId | TID |
* +-------------+-----------------+
*
* Forbidden zero (F): 1 bit
* NAL unit type (Type): 6 bits
* NUH layer ID (LayerId): 6 bits
* NUH temporal ID plus 1 (TID): 3 bits
* decode the FU header
*
* 0 1 2 3 4 5 6 7
* +-+-+-+-+-+-+-+-+
* |S|E| FuType |
* +---------------+
*
* Start fragment (S): 1 bit
* End fragment (E): 1 bit
* FuType: 6 bits
*/
FU的NAL型別為49,一個HEVC幀如果長度超過最大RTP打包長度,則使用FU打包,FU頭中有起始標記,S為開始標記,E為結束標記,SE都沒有標記,則為中間標記,我們可以稱為M FU。即一組FU包由1個S FU,N(>=0)個M FU,以及一個E FU組成,所以要根據FU的NAL頭和FU頭將一個序列的FU HEVC包,還原為一個真正的HEVC NALU。
/* fragmentation unit (FU) */
case 49:
/* pass the HEVC payload header (two bytes) */
buf += 2;
len -= 2;
/*
* decode the FU header
*
* 0 1 2 3 4 5 6 7
* +-+-+-+-+-+-+-+-+
* |S|E| FuType |
* +---------------+
*
* Start fragment (S): 1 bit
* End fragment (E): 1 bit
* FuType: 6 bits
*/
first_fragment = buf[0] & 0x80;
last_fragment = buf[0] & 0x40;
fu_type = buf[0] & 0x3f;
m_firstfrag = first_fragment;
m_lastfrag = last_fragment;
/* pass the HEVC FU header (one byte) */
buf += 1;
len -= 1;
//cout << "nal_type:" << nal_type << " FU type:" << fu_type << " first_frag:" << first_fragment << " last_frag:" << last_fragment << " with " << len <<" bytes\n";
/* sanity check for size of input packet: 1 byte payload at least */
if (len <= 0) {
return -1;
}
if (first_fragment && last_fragment) {
return -1;
}
ret = nal_type;
/*modify nal header*/
new_nal_header[0] = (rtp_pl[0] & 0x81) | (fu_type << 1);
new_nal_header[1] = rtp_pl[1];
handleFragPackage(buf, len, first_fragment,new_nal_header, sizeof(new_nal_header), outbuf,outlen);
break;
/*
* process one hevc frag package,add the startcode and nal header only when the package set start_bit
*/
void H265Frame::handleFragPackage(const uint8_t *buf, int len,int start_bit,const uint8_t *nal_header,int nal_header_len,uint8_t *outbuf,int *outlen){
int tot_len = len;
int pos = 0;
if (start_bit)
tot_len += sizeof(start_sequence) + nal_header_len;
if (start_bit) {
memcpy(outbuf + pos, start_sequence, sizeof(start_sequence));
pos += sizeof(start_sequence);
memcpy(outbuf + pos, nal_header, nal_header_len);
pos += nal_header_len;
}
memcpy(outbuf + pos, buf, len);
*outlen = tot_len;
}
2)一個RTP包含有N(N>=2)個NALU,即aggregated packet,簡稱AP,其NAL值為48。在HEVC的RTP打包中,一般將VPS、SPS、PPS以及SEI等NALU打入一個RTP包中。AP包的解析相對比較簡單,將各部分拆開並分別增加00 00 00 01頭即可。在HEVC的RTP解包中,AP因為包含了如此重要的引數,所以比較重要,AP包解錯,HEVC解碼將無法進行。
/* aggregated packet (AP) - with two or more NAL units */
case 48:
//cout << "nal_type:" << nal_type << "\n";
buf += 2;
len -= 2;
//handl aggregated p
if(handleAggregatedPacket(buf, len,outbuf,outlen) == 0){
ret = nal_type;
}
break;
/* aggregated packet (AP) - with two or more NAL units
* we add start_sequence_code_header for every NAL unit
*/
int H265Frame::handleAggregatedPacket(const uint8_t *buf, int len,uint8_t *outbuf, int *outlen)
{
int pass = 0;
int total_length = 0;
uint8_t *dst = NULL;
// first we are going to figure out the total size
for (pass = 0; pass < 2; pass++) {
const uint8_t *src = buf;
int src_len = len;
while (src_len > 2) {
uint16_t nal_size = AV_RB16(src);
// consume the length of the aggregate
src += 2;
src_len -= 2;
if (nal_size <= src_len) {
if (pass == 0) {
// counting
total_length += sizeof(start_sequence) + nal_size;
} else {
// copying
memcpy(dst, start_sequence, sizeof(start_sequence));
dst += sizeof(start_sequence);
memcpy(dst, src, nal_size);
dst += nal_size;
}
} else {
cout << "[HEVC] Aggregated error,nal size exceeds length: " << nal_size << " " << src_len << "\n";
return -1;
}
// eat what we handled
src += nal_size;
src_len -= nal_size;
}
if (pass == 0) {
dst = outbuf;
*outlen = total_length;
}
}
return 0;
}
上述RTP封包的HEVC解析程式碼改自ffmpeg。