Serial RapidIO
From LinuxC6xWiki
Contents |
Serial RapidIO package content
The Serial RapidIO (SRIO) is supported in linux-c6x-0.7.1 and later releases.
The Serial RapidIO package under the directory ~/my-linux-c6x/projects/rio-utils includes the following file and directories:
Makefile -> top heading Makefile
Misc -> examples and prebuilt BIOS images directory
utils -> user-land utility directory
Building the Serial RapidIO package sources
The SRIO utilities are built from the top level Makefile when selecting the full-root and performing the ‘make product’ or if calling the one-rio rule (‘make one-rio’).
Configuring the kernel RapidIO driver
The TCI648x/647x SRIO hardware configuration is defined in the following file:
linux-c6x/arch/c6x/platforms/board-<board>.c
You can override this configuration with Linux kernel command line parameters as in Table 1.
Table 1. Linux kernel command line Serial RapidIO parameters
Example:
riohdid=0 rioports=1 rioinit=enum riomode=0 rioedma-threshold=32768 (for TCI6474 device #1) riohdid=-1 rioports=1 rioinit=discov riomode=0 rioedma-threshold=32768 (for TCI6474 device #2)
This will enumerate the SRIO bus on port 1 with the SRIO serdes configuration of the SRIO bootmode 0. The EDMA transfer is used for transfer greater than 32KB.
Using the kernel RapidIO driver
Boot the board with the standard Linux kernel then log on the target.
During the kernel boot, you must get the following traces:
RIO: register sRIO controller for hostid 0
RIO: enumerate master port 0, RIO0 mport
TCI648x RapidIO driver v2.1
RIO: setting EDMA threshold to 0x8000
To check that the RIO driver is running:
# ls -la /dev/ | grep rio
crw-rw---- 1 root root 154, 0 Nov 30 00:00 /dev/rio1.1
This example indicates that a remote SRIO device is found with device id 1 on the port 1.
The syntax is /dev/rio<port>.<device_id>
This file is a character special device. It allows to perform Direct I/O and doorbell operations to a remote SRIO site.
You can read and write to remote memory address space by reading/writing in this file. The offset 0 of this file is mapped to the remote Direct I/O base (by default at the DDR start, i.e. 0x80000000 on C6474). You can change this base address by performing ioctl() commands.
For other operations, the following IOCTL (in include/linux/rio.h) are available:
ioctl(fd, RIO_DIO_BASE_SET, &base); /* Set the base offset for a site */
ioctl(fd, RIO_DIO_BASE_GET, &base); /* Get the base offset of a site */
ioctl(fd, RIO_DIO_MODE_SET, &mode); /* Set the Direct I/O mode for a site */
ioctl(fd, RIO_DIO_MODE_GET, &mode); /* Get the Direct I/O mode of a site */
ioctl(fd, RIO_DBELL_TX, &info); /* Sent a doorbell to a site */
ioctl(fd, RIO_DBELL_RX, &info); /* Wait for a doorbell from a site */
'base' is an address value, 'mode' is the Direct I/O mode (RIO_DIO_MODE_WRITER, RIO_DIO_MODE_WRITE, RIO_DIO_MODE_SWRITE), 'info' is the doorbell info data.
Using the RIONET driver
The kernel Serial RapidIO driver also offers the capabilities to use Ethernet over the Serial RapidIO message passing interface.
The following SRIO tests require two TMS320C647x devices connected through serial RapidIO. The TMS320C6474 Evaluation Module from Spectrum Digital, Inc. http://www.spectrumdigital.com has two TMS320C6474 devices and thus can be used for this purpose.
Set the boot switch of the EVM to sRIO boot config n (n must be with the same value as riomode=n in the kernel command line), for example, riomode=0 (BOOTM0 on, BOOTM1 on, BOOTM2 on and BOOTM3 off).
During the kernel boot, you must get the following traces:
eth1: rionet Ethernet over RapidIO Version 0.2, MAC 00:01:00:01:00:00
Using 00:e:0001 (vid 0030 did 0092)
...
Log on the target and configure the eth1 interface:
# ifconfig eth1 192.168.0.1
Boot another Linux kernel on another C647x device connected by SRIO, log on the second target and configure the eth1 interface too. Be sure that the second kernel command line contains riohdid=-1 and rioinit=discov.
# ifconfig eth1 192.168.0.2
You can now communicate between the two C647x devices with TCP/IP like with a standard Ethernet device, e.g by using telnet as below.
The rioconfig utility
To administrate and test the Serial RapidIO driver, a 'rioconfig' utility is provided in the package:
# rioconfig
Usage:
rioconfig {-b ADDR|--base=ADDR} DEV
Changing DirectIO base offset to ADDR
rioconfig {-m MODE|--mode=MODE} DEV
Changing DirectIO mode to MODE
riocongig {-d DBELL|--dbell=DBELL} DEV
Sending doorbell number DBELL
rioconfig {-w DBELL|--dbell-wait=DBELL} DEV
Waiting doorbell number DBELL
rioconfig {-t DIO_MODE |--bench=DIO_MODE} [-l LOOP|--
loop=LOOP] [-B BUF_SIZE|--buf-size=BUF_SIZE] [-b BASE|--base=BASE] [--no-check]
Launch DirectIO performance benchmark
- rioconfig -b <addr> /dev/rio<port>.<num> will change the Direct I/O base address of site <num> to <addr>.
- rioconfig -m <mode> /dev/rio<port>.<num> will change the Direct I/O default mode of site <num> to <mode>. (<mode> can be 0 for NWRITE_R, 1 for NWRITE and 2 for SWRITE).
- rioconfig -d <dbell> /dev/rio<port>.<num> will post a doorbell with <dbell> info to a site <num>.
- rioconfig -w <dbell> /dev/rio<port>.<num> will wait a doorbell with <dbell> info from a site <num>.
- rioconfig -t <mode> /dev/rio<port>/.num> will bench Direct I/O read and write operations with <mode> mode on site <num>. <mode> can by 'n' for NWRITE_R, 'a' for NWRITE and 's' for STREAM. The --no-check flag will do the test without performing data verification. Base offset of the test can be changed with the -b flag, default buffer size can be changed with the -B flag and the test can be performed in loop using the -l flag.
Examples
1) Read/Write with Direct I/O
- Boot Linux on TCI6474 device #1
- Connect with CCS the TCI6474 device #2 in order to set up the DDR timing.
- On the target, do the following commands:
# echo 'test' > /dev/rio1.1
# od -a /dev/rio1.1 | more
You must get the 'test' string.
- On the TCI6474 device #2 with CCS, view memory at address 0x80000000, you must observe the 'test' string too in CCS.
2) Boot of remote TCI6474 and doorbell Tx/Rx
- Set the Jumper of the board to BOOTMODE_0 (BOOTM0 on, BOOTM1 on, BOOTM2 on and BOOTM3 off).
- Boot Linux on the TCI6474 device #1
- Log on the target
- Issue the following commands:
# loadremotecore /dev/rio1.1 0 /opt/images/SRIO_boot.out
It will load the SRIO_boot.out COFF image to the core 0 of the second TCI6474.
# rioconfig -w 1 /dev/rio1.1
To wait a doorbell with info=1 from the second TCI6474.
- Open a second terminal or telnet session and issue the following commands:
# rioconfig -d 0 /dev/rio1.1
It will send a doorbell to the second TCI6474, which will boot with the SRIO_boot.out DSP/BIOS application that will send a doorbell 1 to the first TCI6474. The 'rioconfig -w 1 /dev/rio1.1' command must exit without error as below.
3) Direct I/O performance benchmark
- Boot Linux on TCI6474 device #1
- Connect with CCS the TCI6474 device #2 in order to set up the DDR timing.
- On the target, issue the following command:
# rioconfig -t s --buf-size=8192 --no-check --loop=10 /dev/rio1.1
Below is the example log:
