Tagged: gnutv

An Interim Report on the Project

It has been two months since the last blog entry and a lot of stuff has happened. We began to rewrite the audio module to get rid of all external libraries. The idea is that the OpenMAX interface (of the Raspberry) has everything we need to decode the audio directly. However, there are still minor problems with the code which prevent the release of a new prototype.

This was the first issue. The second issue is that we are very busy with the module for the personalized recording. Our current approach, the use of a artificial neural network, is very promising but due to the lack of proper meta data and the fact that the existing data is very sparse, it is challenging to create good predictions. Since the recommender part is the heart of our system, we are focusing on this issue right now.

What about DVB-T?

Short after we uploaded the first prototype in the subversion repository of our new project, we had a chance to test it with a different type of DVB device; in this case, it was DVB-T. All we needed then was a channels.conf file. With the scan utility from the dvb-apps package, it was straightforward to create it from scratch.

The test environment was a Raspberry Pi with MPEG2 enabled. After the firmware of the DVB-T stick was placed in /lib/firmware folder, the stick was recognized without any problems. A quick test with raberrytv $CHANNEL confirmed that the device was working and that it received a valid signal. The last necessary step was to start the prototype in the ‘omx’ mode raberrytv -out omx $CHANNEL which also worked without any problems.

However, there are some conclusions we have drawn from this lesson. First, the lip-sync code is not yet perfect and needs more work. Second, the nature of DVB-T makes the signal less reliable, at least with a small antenna, than DVB-S and thus failures are more likely. Nevertheless, the test was a great success and confirmed that the prototype can handle any DVB-* device in general without any modifications of the source. Of course it is very likely that fine tuning is required here and there to improve the quality and the user experiences, but we think that this is doable with only minor effort.

Finally, we would like to thank all people who contributed to this test. This includes especially the sponsoring of the hardware. The next step is to use the gained experiences to finish the work on a first prototype that is ready for end-users.

Some Problems with DVB-S2

We recently finished to clean-up the code for our prototype. The main goal was to assure to be lip-sync for the A/V data and to get rid of lengthy code parts that were derived from very generic and somewhat older MPEG code. The additional tests we made revealed some minor problems, but all of them could be corrected very easily. That were the good news.

After we finished our DVB-S tests, we recorded some DVB-S2 streams with the device that we intended to use for further tests. We did this by using gnutv. However, the recorded streams could not be displayed on any platform. Since the DVB-S2 device is working perfectly on other platforms, we assume that this is an issue with the available kernel on the Rasbperry. For that reason, we will delay further HD tests until we upgraded the device to the latest image and kernel version.

In the meantime, we continue our work on the first public release of Raberry TV.

Eureka

Today we performed a first full smoke test on the prototype. The easy part was to playback recorded MPEG media files in TS or PS format. Easy, because there is usually not much jitter when you read data from a fixed drive (hard disk, flash, …). And as expected, the prototype performed very well on this test. We did not encounter any problems, neither video nor audio issues.

The more challenging part was to playback MPEG data which arrives at the DVB device and is then handed over to our (user space) process. We had to tweak the size of the cache and the block size to get rid of underflow problems that occurred occasionally. With the new settings, and some minor changes to the multi-threading code, we were able to watch TV without any audio static or picture artifacts. However, with the simple approach we had chosen, the audio and video drifted out of sync after a short while, but this can be easily fixed by adjusting the existing approach and the help of the timestamps in the MPEG data itself.

To sum up the situation, except for the ‘lip sync’ issue the prototype only needs some polishing, minor cleanups and more in-depth testing. As soon as the last issue is fixed, we plan to make the source code public to get some feedback from others. We are especially interested in feedback from users with different devices and configurations, for instance DVB-T or DVB-C.

An Incomplete GnuTV Integration

Introduction

After we previously implemented a sketch of each single software component that we need for our TV prototype on the Raspberry, the next step was to create an actual proof-of-concept prototype. As noted before, we use the gnutv application from the Debian package dvb-apps because it already provides generic DVB functions which means that we can focus on the data decoding with OpenMAX. The idea is to provide a hardware specific output mode which allows users to watch TV like that:
gnutv -buffer 1048576 -out omx $CHANNEL

Implementation Details

Because the MPEG-TS streams may contain several video streams as well as audio streams, we first need to select the PIDs that should be used for playback. More precisely, one PID for audio and one for video. To keep the prototype as simple as possible, we automatically select the first video stream and the first audio stream. These parameters are required to decode the MPEG-TS data into MPEG-ES.
Then we define a callback function for each type of data which passes the elementary data to the corresponding module (omx_video, omx_audio). At last we need to adjust the gnutv code to provide the new omx output mode, but that is straightforward and not an OpenMax issue.

Results

Again we were surprised that even such a simple prototype performed so well. In short, the video decoding worked flawlessly and except for some noise in the decoded audio data, we could already watch TV. However, there are some open issues. It is probably a good idea to provide a dedicated cache for the MPEG-TS data to cope with jitter and similar problems. Furthermore, the audio module also needs some cleanup to get rid of the noise. But what is most important is to synchronize the audio and video data. During our tests, the video stream was always ahead of the audio stream. A solution for the problem should not be too hard since the MPEG data provides meta data and it is straightforward to enhance the prototype with threads and events for data synchronization.

A Simple DVB Player

After we gained some experiences with the OpenMAX interface API we began to work on the audio part of the MPEG-TS stream. Basically it is the same procedure that we used for the video data. First we convert the TS data into PS and then the PS data into ES which can be used by the OpenMAX routines. Since the our mpeg utility code from dvbstream can be used for both types of data (audio and video) we just need to provide additional callbacks for the audio data.

Again, we used the hello_pi code examples, more precisely the file audio.c, for our proof-of-concept prototype. The aim was to demultiplex a complete MPEG-TS stream into audio and video, the video data is ignored for the moment, to decode the data on the fly and to pass it to the audio device via the OpenMAX interface. That worked surprisingly well, at least for our very simple environment that only used the audio jack in 2 channel stereo mode.

So, let us check our inventory what we got:

  • A module to decode MPEG-TS streams into elementary video and audio data
  • A module to utilize the MPEG decoder of the GPU to handle the video data
  • A module to decode the audio data
  • A module to process IR events issued by an IR receiver

Now, we just need to glue all these components together by defining some simple interfaces and to add some missing features. For instance, to synchronize audio and video streams.

OpenMAX Experiments

When we stumbled about the hello_video code example that is delivered with the Raspberry image we decided to use the code as a testbed for a prototype to do media decoding. The code uses the OpenMAX, Open Media Acceleration, interface to provide the actual video routines for the decoding. Without an additional license for MPEG-2, only H.264 decoding is possible.
Since most of the DVB data is still in MPEG-2, we focus on the decoding of this type of data. However, since OpenMAX provides an abstract interface, the adjustments for H.264 decoding are straightforward and can be easily done.

Because the sample video file that is provided in the source example is H.264, the first step is to create some media data in MPEG-2 format. That can easily be done with gnutv, for instance, gnutv -timeout 10 -out file test.mpeg $CHANNEL. The raw output of DVB is MPEG-TS, a container format of MPEG that is usually used in broadcasting systems. The data container contains at least one audio and one video stream and because the OpenMAX interface can only work with raw data of a dedicated type, the next step is to demultiplex the data into single streams. This step involves actually some decoding to convert the MPEG-TS data into MPEG-ES which then can be used directly by the OpenMAX video decoding interface.

Sample code for the described procedure is available in some libraries which are part of most Linux distributions. But because we wanted to minimize dependencies and provide a stand-alone prototype, we decided to use GPLed code from the package dvbstream.
For those, who just want to test it without typing much code, the output data can be also created with 'mplayer -dumpvideo -dumpfile test.mpeg test.ts'. The file test.mpeg is in MPEG-ES format which can be directly passed to the hello_video.bin program. However, a small modification to the video.c is required to change the decoder from H.264 to MPEG-2:

video.c:125 -- format.eCompressionFormat = OMX_VIDEO_CodingAVC;
video.c:125 ++ format.eCompressionFormat = OMX_VIDEO_CodingMPEG2;

If everything worked, the screen should be filled with some moving pictures.

After we gathered all the information and code we needed, we decided to implement a proof-of-concept prototype that adds omx as an output to gnutv. In short, instead of sending the output to a file or socket, the media output will be directly send to the video decoder which means the data is directly shown on the screen. As noted before, we first need to decode the MPEG-TS data into MPEG-ES and then we can use the OpenMAX routines.

After we finished to cleanup the code and performed more tests, we need to work on the audio integration. However, with the insights we have gained, a small, lightweight TV solution for the Raspberry is one step closer.

Recipe to Watch TV

What follows is a step by step list what to do to watch TV on your Raspberry. First the checklist:

  • Raspberry Pi device
  • Optional: a license for decoding MPEG 2 data
  • Supported DVB-S, DVB-S2 USB device

Kernel Update

Because the Kernel that is provided with the Raspberry image does not contain the required modules for DVB USB devices, the Kernel must be upgraded. The easiest way to do this is to use an automatic script, for instance rpi-update. For safety reasons, the old image from the card should be backuped. After the process is done, the device can be booted.

Setting up the DVB environment

In case the DVB devices needs a special firmware, it has to be downloaded first and placed in /lib/firmware. To make sure that all required Kernel modules are available on startup, they should be placed in the /etc/modules file. We will demonstrate the steps for the Technotrend S-2400. The name of the required firmware is dvb-usb-tt-s2400-01.fw and the required modules are dvb-usb-ttusb2, dvb-usb. To test the process the command modprobe dvb-usb-ttusb2 can be used (as root).

Testing the DVB environment

As a first step, the package dvb-apps has to be installed. The packet contains the application gnutv that is used to receive the DVB data. The omxplayer should be already installed. However, it might be a good idea to update it to make sure you always have the newest version. We assume that a channels.conf has already been created or is available and is stored in /etc.
To test the actual reception, the command gnutv CHANNEL can be started, where CHANNEL is the name of the channel to tune-in. In case of any problems, a short error description is shown and the process will exit. Otherwise, if everything works, the channel will be locked in and the program goes into an infinite loop. The process can be now aborted, because everything works fine.

Watching TV

To make sure that the sound card is working and omxplayer is able to playback MPEG data, omxplayer should be manually started once with an MPEG file. 
Since the DVB streams are not available as files, we will create a FIFO to pass the data from gnutv directly to omxplayer. This can be easily done with mkfifo /tmp/mystdin. The next step is to feed the FIFO with data from gnutv and let omxplayer read from the fifo.


gnutv -buffer 1048576 -out file /tmp/mystdin "COMEDY CHANNEL" &
sleep 1
omxplayer /tmp/mystdin

After some seconds the MPEG stream should be directly displayed in front of the console text. To quit the player, ‘q’ should be pressed.

We are working on a python framework to automate these steps and to provide support for IR receivers that are usually bundled with DVB-S USB devices. The first test was very promising but more tests need to be done and the code needs cleanups before we will release it.

Introduction

In contrast to a lot of other users we were never interested to turn the mini computer (Raspberry Pi) into a full-blown media center. What we had in mind was to create a simple but nevertheless elegant TV player capable of processing both MPEG2-based and H.264-based media. Therefore, we first had to activate the MPEG2 hardware decoding in the store. After this step was done, all hardware requirements were met.

For the initial prototype the Technisat SkyStar USB device is used. We have chosen the device because of the HD-capability and the fact of the seamless Linux integration. The card has been successfully tested on the mini computer and except for some artificats in the H.264 playback we did not encounter any problems. Other supported cards should also work but we do not have any user feedback yet.

The next step was an inventory control. Since only the omxplayer is capable of GPU decoding media data, we had to find a way to pipe the raw MPEG stream from the DVB device directly into the program. To simplify data handling, and to avoid stdin/stdout issues, we created a FIFO with ‘mkfifo’. The next step was to actually receive the DVB-S data. We used the gnutv application from the dvb-apps package because it is simple and it supports DVB-S2 out of the box.

This procedure may seem unnecessarily complex but because traditional media applications with DVB support cannot be used because of the missing GPU acceleration, it is absolutely necessary to use omxplayer for now. And because the player does not support to receive media from DVB directly, a second program is required to receive the actual data and to pass it to the player.

To sum it up, we do something like 'gnutv -out file /tmp/my-dvbs-fifo CHANNEL'; omxplayer /tmp/my-dvbs-fifo to display the media. On a side note we would like to add that no X is required for this approach. The omxplayer works fine on the console and the same applies for the sound device. Thus, we can use the very limited memory for caching and other tasks.

The drawback of the console-only environment is that there is none or a very limited mouse support. But we do not believe this is a real problem, because the DVB device comes with an IR receiver that is supported by the Kernel and this allows us to control the prototype directly, for instance to switch channels or to adjust the sound volume.

As a final remark, the prototype is not ready yet. We are not even sure if the elicitation of requirements is done. In any case there still are a lot open challenges we have to work on, a lot of code to write and a lot of coffee to drink. We are currently working on a first release but it will definitely take some time until it is ready for a public beta test.

The intention of this blog is to report about the progress we made, insights we gained and to discuss (new) ideas with other people.