View source for How to configure SPI (Service and Programme Information) for ODR-DabMux

Howto by Ulrik Brinck on odr-radioepg-bridge installation on Debian Jessie. Nick Piggott has contributed with some additions (and edited to show installation for Python 3)

This is now updated to run on Python 3. Instuctions on how to make it (mostly) run on Python 2 are below, but this is now deprecated.

'''This guide assumes your default python envrionment is Python 3.'''

===Introduction===

This tool (or rather series of tools) will pick up your SPI (Service and Programme Information) from the internet via RadioDNS and generate a DAB SPI from it. It was written by Ben Poor and Nick Piggott. In this HowTo, I will describe how it can be installed alongside Python 3.

* Confused about EPG and SPI? In this context, EPG and SPI are basically the same thing. While EPG (Electronic Programme Guide) is an old, but commonly well-known term, SPI (Service and Programme Information) is now the official name of this feature.
* Some Service Providers only allow access to the SI (& PI) files if you get a Client ID (like an API key) from them first. This ClientID is associated with their FQDN (fully qualified domain name). You can create a CSV format file which holds the fqdns and associated ClientIDs, which allows the generator to provide the ClientID when it's asking for the SI & PI files.

===Notes===

'''This tool uses RadioDNS to locate and acquire Service Information (SI.xml), Programme Information (PI.xml) and Logos.'''
Each station you want to add to the SPI must be registered in RadioDNS using the correct information (Ensemble ID, Service ID, Service Component ID), and have correctly formatted SI, PI and logo files in PNG format. Find out more at https://radiodns.org/technical/documentation/howto/

I have assumed that you are logged in to your Linux system as a standard user and have sudo installed on your system. If sudo is not installed, you can see here, how to install it on Debian Jessie:
https://www.geek17.com/en/content/debian-8-jessie-install-and-configure-sudo-25

===Installation===

====Step 0: Update your system====
Begin with

 sudo apt-get update

to make sure that you can install all that you need.

====Step 1: Miscellaneous dependencies====
First there are some dependencies to install with apt-get:
 
 sudo apt-get install python3-setuptools python3-pip python3-crcmod python3-dateutil python3-isodate python3-pil git

Next there are some dependencies and subcomponents which must be downloaded from github and installed. Establish yourself in a suitable working directory (e.g. ~/src)

====Step 2: python-dabmot====


 git clone https://github.com/Opendigitalradio/python-dabmot
 cd python-dabmot
 sudo pip3 install .
 cd ..

====Step 3: python-mot-epg====

 git clone https://github.com/Opendigitalradio/python-mot-epg
 cd python-mot-epg
 sudo pip3 install .
 cd ..

====Step 4: python-msc====

 git clone https://github.com/Opendigitalradio/python-dabmsc
 cd python-dabmsc
 sudo pip3 install .
 cd ..

====Step 5: pyradiodns====

 git clone https://github.com/radiodns/pyradiodns
 cd pyradiodns
 sudo python3 setup.py install
 cd ..

'''NOTE'''
The installer specifies that dnspython must be at or greater than version 2.0 when used with python3. Most distributions ship with this version.
If the installer can't find dnspython>=2.0, you need to download, unpack, and manually install the package from https://pypi.org/simple/dnspython/

====Step 6: odr-radiodns-bridge====

 git clone https://github.com/Opendigitalradio/odr-radiodns-bridge
 cd odr-radiodns-bridge
 sudo pip3 install .
 cd ..

====Step 7: python-hybridspi====

 git clone https://github.com/Opendigitalradio/python-hybridspi
 cd python-hybridspi
 sudo pip3 install .
 cd ..

====Step 8: odr-radioepg-bridge====

 git clone https://github.com/Opendigitalradio/odr-radioepg-bridge
 sudo ln -f odr-radioepg-bridge/generate-epg /usr/local/bin/generate-epg



That's it. You should now be able to run this command and see a help text:
 generate-epg -h

===Generating and broadcasting your SPI (EPG)===

====Generating your SPI====

Odr-radioepg-bridge is a command-line tool, which will:
* First read your mux.conf (from your ODR-DabMux configuration) to get information about your services.
* Then try to fetch SPI data for your services from the internet using RadioDNS.
* And lastly generate an output file containing a complete SPI.

If you type "generate-epg -h" you will get a list of available command-line arguments:

 usage: generate-epg [-h] [-o OUTPUT] [-X] [-d DAYS] [-p PACKET_SIZE]
                     [-a PACKET_ADDRESS] [-D]
                     f
 
 Encodes an SPI bitstream for services in a multiplex configuration file
 
 positional arguments:
   f                  multiplex configuration file
 
 optional arguments:
   -h, --help         show this help message and exit
   -o OUTPUT          output bitstream file
   -X                 turn debug on
   -d DAYS            number of days ahead to encode schedule files
   -p PACKET_SIZE     Packet size in bytes
   -a PACKET_ADDRESS  Packet address
   -D                 output Datagroups instead of Packets
   -c CLIENTIDS       filename of CSV file of fqdn,ClientID key values


The positional argument tells the tool where to find your mux.conf.
The other arguments are optional, and some default values will be used if you don't set them:

* -o can be used to set filename and path to your output file (your SPI). Default is output.dat, placed in the folder from where you run the tool.
* -d is the number of days of programme schedule you want in you SPI. From 1 til 4 days, default is 2.
* -p is packet size. Available values are 24, 48, 72 and 96. Default is 96. Which value you should use, depends on the bitrate in which you are broadcasting your SPI (see below).
* -a is the packet address for your SPI. Default is 1, which will normally be fine. Important is, that you must set the same address in your mux.conf (see below).
* -D will tell the script to output Datagroups instead of Packets. This is useful for some commercial multiplexers or in case you want to broadcast your SPI as X-PAD instead of packet data. Note: You do not need this if you plan to broadcast your SPI the way it's described on this wiki page.
* -c gives the filename of a CSV formatted file which contains Fully Qualified Domain Names (fqdns) and Client IDs. Some service providers will only allow access to their SI file with a Client ID. Contact them directly for a Client ID.

   example.org,wkljew324cx


Example:
If your mux.conf is placed in /home/user/odr/config/mux.conf, and you want 4 days in your SPI (today and the next 3 days) and you want to broadcast it in 16 kbit/s, your command line could look like this:

 generate-epg -d 4 -p 48 /home/user/odr/config/mux.conf

This will generate a file named output.dat containing your SPI. In this example, I have set the packet size to 48, which is suitable for 16 kbit/s. I have not set the packet address, which means that the default value of 1 will be used.

====Packet sizes and bitrates====

The packet size, which you set when you generate your SPI, should match the bitrate in which you intend to broadcast it. If the packet size is too large, your SPI will not be transmitted. If the packet size is too small, your SPI will (probably) be transmitted, but it will be slower than necessary, because more space will be used for packet headers and maybe partly empty data packets. I will recommend these values:

{| class="wikitable"
! Bitrate
! Packet size
|-
| 8
| 24
|-
| 16
| 48
|-
| 24
| 72
|-
| 32
| 96
|-
| 48
| 72
|-
| 64
| 96
|}

====Broadcasting your SPI====

To broadcast your SPI using ODR-DabMux, create a packet data service in your mux.conf like in this example:

In the services section:

 srv-spi {
 label "EPG & Logos"
 shortlabel "EPGLogos"
 ; the service-ID consists of 8 hex digits for packet data services. Usually the first two digits will
 ; be the same as your ECC, and the third digit will be the same as the first digit in your Ensemble Id.
 id 0xe1912345
 }

In the subchannels section:

 sub-spi {
 ; type packet or enhancedpacket can be used for SPI.
 type enhancedpacket
 ; if you change the bitrate, remember to also change the packet size accordingly (see above) and generate your SPI again.
 bitrate 16
 protection-profile EEP_A
 protection 3
 ; inputuri is the path and filename for your output file from generate-epg
 inputuri "/home/user/odr-radioepg-bridge-master/output.dat"
 inputproto file
 } 

And in the components section (if you are using version 3.1.1 or earlier of odr-dabmux)

 comp-spi {
 type 60
 service srv-spi
 subchannel sub-spi
 figtype 0x7
 ; The address must be the same as your packet address set when you generate the SPI (default value is 1).
 address 0x1
 datagroup true
 }

If you are using version 3.1.2 or later of odr-dabmux, the construction of the component section is slightly different

 comp-spi {
 type 60
 service srv-spi
 subchannel sub-spi
 user-applications {
   userapp "spi"
   }
 address 0x1
 datagroup true
 }

That's it. Now (re)start your ODR-DabMux, and your SPI should be transmitted.

===Updating your SPI automatically every day===

====Feeding the SPI from a fifo will ensure that you can update it without restarting the MUX====

While the setup shown above will allow you to broadcast your SPI, it ''might'' not allow you to update your SPI without having to restart your MUX.
* If the SPI output file is ''rewritten'', the new content will be picked up by ODR-DabMux automatically, and your SPI will immediately be updated. This will be the case if you use generate-epg to generate a new file directly upon the the old one.
* However, if the SPI output file is ''replaced'', the new file will '''not''' be picked up by ODR-DabMux, until you restart it. This will be the case, if you generate a new file elsewhere (e.g. in a different folder or on another machine) and then ''copy'' it into where the old one is.

To get around this, and be sure that you can always update your SPI, any way you want, you can feed your SPI to ODR-DabMux from a named fifo instead of a file.

Navigate to the folder where you have your SPI and create a fifo like this:

 mkfifo spififo

In the subchannel section of your mux.conf, you should replace your SPI file (output.dat) with your fifo, and also add a new line, "nonblock true", so that your SPI subchannel looks like this:

 sub-spi {
 ; type packet or enhancedpacket can be used for SPI.
 type enhancedpacket
 ; if you change the bitrate, remember to also change the packet size accordingly (see above) and generate your SPI again.
 bitrate 16
 protection-profile EEP_A
 protection 3
 ; inputuri is the path and filename for your output from generate-epg
 inputuri "/home/user/odr-radioepg-bridge-master/spififo"
 inputproto file
 nonblock true
 } 

Your SPI must be continously sent to the fifo. You can make a shell script for that. Create a file, sendspi.sh, with this content:

 #!/bin/bash
  
 while :
 do
  
 cat /home/user/odr-radioepg-bridge-master/output.dat >> /home/user/odr-radioepg-bridge-master/spififo
  
 done

And make it executable (chmod +x sendspi.sh).

This script must be running all the time. If you're already using Supervisor to keep your MUX running (as described here: https://github.com/Opendigitalradio/dab-scripts), it's easy to also use supervisor for this script. Create a new file, sendspi.conf, in your /home/user/odr/config/supervisor folder with this content:

 [program:sendspi]
 command=bash -c "/home/user/odr-radioepg-bridge-master/sendspi.sh"
 autostart=true
 autorestart=true
 startretries=99999
 priority=10
 stderr_logfile=/dev/stdout
 stdout_logfile=/dev/stdout

Then run these commands:

 sudo ln -s /home/user/odr/config/supervisor/sendspi.conf /etc/supervisor/conf.d/sendspi.conf
 sudo supervisorctl reread
 sudo supervisorctl update

Now your SPI is sent to ODR-DabMux from the fifo, and you will now be able to update your SPI, any way you want, without having to restart ODR-DabMux.

====Updating your SPI automatically every day====

You can update your SPI every 24 hours by adding a new daily task to cron.

Create a new file

 /etc/cron.daily/spi

with the single line (with the same parameters as when you generate your SPI yourself)

 /usr/local/bin/generate-epg -d 4 -p 48 /home/user/odr/config/mux.conf -o /home/user/odr-radioepg-bridge-master/output.dat

By default, the cron job will run every day at around 06:00. If you want your SPI to update more frequently (every 4 hours), you could add it to the user cron job with

 crontab -e

and add the line

 0 */4 * * * /usr/local/bin/generate-epg -d 4 -p 48 /home/user/odr/config/mux.conf -o /home/user/odr-radioepg-bridge-master/output.dat

'''NOTE:'''
The SPI creates new MOT TransportIDs for every object each time it is run. A receiver will not recognise that objects have been previously received and cached. If there is an interruption in reception, the SPI will be incomplete. The more frequently you update the SPI, more likely it is that a receiver will not be able to utilise the cached objects.

===Some observations===

* According to specs, logo slides in DAB EPG must be PNG. Although JPG is allowed in hybrid radio SPI, JPG logos will not be added to your DAB EPG. Be sure that your logos (at least those up to 320x240) are PNG.
* Resolved on 12APR21: (Programme descriptions (shortDescription) seem to be processed but not shown. Perhaps we will have to look a little further into this.)
* Updated on 15MAR22: Instructions are now for Python 3