events
news
The Linux Foundation
 
 
Platform Tests

From The Linux Foundation

Contents

Installing the LSB 3.1 Platform Tests

Installing from the Test Bundle

The entire set of platform tests is available in an installable bundle. The easiest way to install all of the tests is to download the tarball from the Bundled packages section of the download page. Here is an example session: 

# wget -c http://ftp.freestandards.org/pub/lsb/bundles/released-3.1.0/\
testkit/lsb-runtime-tests-3.1.0-3.x86_64.tar.gz     (line broken for display purposes)
...
# tar -xzf lsb-runtime-tests-3.1.0-3.x86_64.tar.gz
# cd lsb-rt
# ./install.sh

This system appears to be a variant of Debian GNU/Linux, such as
Debian itself, Ubuntu, MEPIS, Xandros, Linspire, etc.

Is this correct? y

Installing packages...

#

Installing Tests Individually

If the testkit bundle is not used, test suites may be downloaded and installed individually. For more details, see Platform_Tests_Individual.

Running the LSB 3.1 Platform Tests

Executing the lsb-libchk Test Suite

  1. Start lsblibchk. These tests do not need root access to run. 
    $ /opt/lsb/bin/lsblibchk

The test will leave a result file named journal.libchk in the current directory.

Executing the lsb-cmdchk Test Suite

  1. Start lsbcmdchk. These tests do not need root access to run. 
    $ /opt/lsb/bin/lsbcmdchk

The test will leave a result file named journal.lsbcmdchk in the current directory.

Executing the Runtime Test Suite

  1. Check that loopback mounting support is enabled: 
    # grep loop /proc/devices
  7 loop

    If you don't see the "7 loop" line above, load the loop module: 

    # modprobe loop
  2. Change the password for vsx0, vsx1, and vsx2 to "vsx,1234" 
    # echo "vsx0:vsx,1234" | /usr/sbin/chpasswd
# echo "vsx1:vsx,1234" | /usr/sbin/chpasswd
# echo "vsx2:vsx,1234" | /usr/sbin/chpasswd
  3. Select a virtual terminal and log in as the vsx0 user. It is not recommended to use su or sudo to become the vsx0 user, as on some configurations the login-related tests will not find a utmp entry and will have problems.
  4. Start the runtime tests. Use the answers given below as a template. 
    sh-3.00$ ./run_tests
Name of person running tests (Automated)? Fred Tester
Organisation (NONE)? LSB
Test System (UNKNOWN)? BlackBox 2.0 for IA32
Block special filename (/home/tet/test_sets/nonexistb)? <RETURN>
Does the implementation provide bash as /bin/sh..? [y] <RETURN>
Does the implementation provide a C shell..? [y] n
Enter the name of the Kernel (typically vmlinuz)..? [vmlinuz] <RETURN>
Does the implementation allow users to create devices using MAKEDEV..? [n] <RETURN>
Does the implementation support the file command? [y] <RETURN>
Does the implementation support process accounting..? [n] <RETURN>
Does the implementation support NIS..? [n] <RETURN>
 Enter name of the user for PAM tests [vsx0] : <RETURN>
 Enter password of the user for PAM tests : vsx,1234
 You will next be prompted for two passwords for the pam test user
 accounts. Please choose a robust password for your platform.
 For example a combination of digits and upper and lower case letters.
 Enter password of the test user vsx1 : vsx,1234
 Enter password of the test user vsx2 : vsx,1234
Installing locales for the i18n test suite ... enter the root password:
Password: 
Create locales... LTP_1.utf8 LTP_IL1.UTF-8 LTP_IL2.UTF-8 Done.
Install /etc/pam.d/lsbpam_conf ..? [y] <RETURN>
Enter root password: 
Password: 

Install nonexistent devices ..? [y] <RETURN>
Enter the root password:
Password: 
Updating the account properties of users vsx1 and vsx2
  sets the vsx1 account to have a password that needs to be updated
  sets the vsx2 account to be expired
Enter root password: 
Password: 
2000+0 records in
2000+0 records out
mke2fs 1.38 (30-Jun-2005)
Filesystem label=
OS type: Linux
...
This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
tune2fs 1.38 (30-Jun-2005)
Setting maximal mount count to -1
Setting interval between checks to 0 seconds
/dev/loop0
Setting up loopback device
Enter the root password:
Password: 
Mount loopback device
Password: 
Partially filling disk
Password: 
Filling file system mounted on /mnt
If the file system is large, this will take a long time ...
1+0 records in
1+0 records out

File system should now be not quite full:

Unmount loopback device
Password: 
----------------------------------------------------------------------
Executing the test suites
It is not unusual for these test suites to take several hours to run.
...
  5. Enter your root passwd when prompted (there will be a about eight separate prompts for this).

The runtime tests will take several hours to complete (approx 4 to 8 hours). If more than 8 hours have elapsed and the tests are still not complete it indicates the currently running test has hung for some reason. In this situation enter control-C in the window where the tests are running to terminate the hung test and the runtime tests should then continue.

The test will leave a result file named journal in a numbered subdirectory of the results directory. The number code increments each time the test is started so that older result files are preserved. The first time the test is run the file will thus be /home/tet/test_sets/results/0001e/journal.


Executing the lsb-c++ Test Suite

  1. Make sure the following locales are available on your system: it_IT en_US deDE en_HK de_DE@euro es_MX fr_FR en_PH fr_FR@euro.
  2. Add /usr/opt/lsb/appbat/bin to your PATH. The following command should do this: 
    # export PATH=$PATH:/usr/opt/lsb/appbat/bin
  3. Start the c++ test suite. These tests should not need root access to run, however due to a bug the run_tests script itself requires it for the 3.4.3-5 version. Use the answers given below as a template. 
    # /opt/lsb/test/share/qmtest_libstdcpp_3.4.3/run_tests
Where do you want to create the test database (/home/fred/qmtest_libstdcpp_3.4.3)? <ENTER>
What is the GNU triplet for your operating system (i486-unknown-linux-gnu)? <ENTER>
Name of person running tests (Automated)? Fred Tester
Organisation (NONE)? LSB
Test System (Automated)? BlackBox 2.0 for IA32
...

The C++ test suite will take approximately twenty to thirty minutes to complete. The test will leave a result file in the v3db subdirectory of the location selected to create the test database. The file is date stamped, so an example name would be journal.200607100947.

Executing the lsb-vsw4 Test Suite

  1. Make sure a copy of Xvfb is in the search path: 
    # which Xvfb

    If not found, place a copy of Xfvb in a directory in the search path or add the correct directory to $PATH if it is already installed but not in the path.

  2. Start the vsw4 tests (become root before doing this). Use the answers given below as a template. 
    # cd /opt/lsb/test/vsw4
# ./run_vsw4.sh
Name of person running tests (Not_defined)? Fred Tester
Organisation (Not_defined)? LSB
Test System (Not_defined)? BlackBox 2.0 for IA32
Hostname of client for X tests (localhost)? <ENTER>
DISPLAY of the Xvfb (ie., :1) (:1.0)? :5.0
XT_FONTPATH (/opt/lsb/test/vsw4/xtest/fonts/,/usr/X11R6/lib/X11/fonts/misc/)? <ENTER>   
XT_FONTPATH_GOOD (/usr/X11R6/lib/X11/fonts/100dpi/,/usr/X11R6/lib/X11/fonts/75dpi/)? <ENTER>
Starting X Virtual Frame Buffer
Testing the test fonts are installed .... 
...

    For newer X.org-based releases (7.0 and later at least), the fonts will not be in /usr/X11R6. In this case, answer the font questions differently: 

    XT_FONTPATH ( ... )? /opt/lsb/test/vsw4/xtest/fonts/,/usr/share/X11/fonts/misc/
XT_FONTPATH_GOOD ( ... )? /usr/share/X11/fonts/100dpi/,/usr/share/X11/fonts/75dpi/

The vsw4 tests will take approximately thirty minutes to complete.

The test will leave a result file named journal in a numbered subdirectory of the results directory which in turn is found in the xtest directory. The number code increments each time the test is started so that older result files are preserved. The first time the test is run the file will thus be /opt/lsb/test/vsw4/xtest/results/0001e/journal.

Executing the lsb-test-desktop Test Suite

  1. Make sure a copy of Xvfb is in the search path: 
    # which Xvfb

    If not found, place a copy of Xfvb in a directory in the search path or add the correct directory to $PATH if it is already installed but not in the path.

  2. Start the desktop tests (become root before doing this). Use the answers given below as a template. 
    # cd /opt/lsb/test/desktop
# ./run_tests
Name of person running tests (Not_defined)? Fred Tester
Organisation (Not_defined)? LSB
Test System (Not_defined)? BlackBox 2.0 for IA32
Hostname of client for X tests (localhost)? <ENTER>
DISPLAY of the Xvfb (ie., :1) (:1.0)? :5.0
X11 font path (/usr/X11R6/lib/X11/fonts/misc/)? <ENTER>
Starting X Virtual Frame Buffer
...

    For newer X.org-based releases (7.0 and later at least), the fonts will not be in /usr/X11R6. In this case, answer the font question differently: 

    X11 font path (/usr/X11R6/lib/X11/fonts/misc/)? /usr/share/X11/fonts/misc/

When the test is complete, it will report the locations of the test journal files. There will be five result files produced by this test suite. The message will be something like this: 

LSB desktop test is done now. The test journals can be found at
    /opt/lsb/test/desktop/gtkvts/results/{max-num}e/journal
    /opt/lsb/test/desktop/qt3/results/{max-num}e/journal
    /opt/lsb/test/desktop/fontconfig/results/{max-num}e/journal
    /opt/lsb/test/desktop/xml/journal.libxml2
    /opt/lsb/test/desktop/libpng/journal.pngtest

Analyzing Test Results

Each of the test suites above generates one or more results files known as journals. The first step in analysis involves running the summary tool, tjreport. This tool is a Perl script included in the lsb-tet3-lite package, and will be found at the path /opt/lsb-tet3-lite/bin/tjreport. An output of this tool might be: 

$ tjreport journal.lsbcmdchk
Loading networked waivers information 
...
Warning, wget was unable to locate network waivers to retrieve..
lp 1 FAIL

Test was run: 20060629 07:27:22
Test Suite Name: lsbcmdchk
Test Suite Version: 3.1.0.20060514-1
Test Suite Architecture: IA32
Total Tests Passed: 135
Total Tests Failed (including waived): 1
Total Tests Failed (excluding waived): 1

Test Result Breakdown:
PASS: 135
WARNING: 0
NOTIMP: 0
UNAPPROVE: 0
UNSUPPORTED: 0
TEST_ERROR: 0
FIP: 0
NOTINUSE: 0
UNRESOLVED: 0
UNINITIATED: 0
UNTESTED: 0
FAIL: 1
UNKNOWN: 0
UNREPORTED: 0

For conformance, we're seeking a result of "Total Tests Failed (excluding waived): 0" for each test suite. When there are registered waivers for a particular version of a particular test suite, the reporting tool will successfully fetch a waiver file and "check off" those tests which are waived in producing this summary.

In the above test run, there was a single failure, a test designated as "lp 1". If you search for the "lp" test in the journal file, you will see the entire test sequence: 

10|67 lp 07:27:22|TC Start
15|67 tetj-1.0 1|TCM Start
400|67 1 1 07:27:22|IC Start
200|67 1 07:27:22|Looking for command lp
220|67 1 1 07:27:22|FAIL
410|67 1 1 07:27:22|IC End
80|67 0 07:27:22|TC End

Which indicates quite simply that the required command lp is missing from this system; this problem must be corrected before the system can pass this test suite.

For more information on test suite analysis, see the Failure Analysis page. (Note: this page is slightly outdated, will be updated soon)

Additional ways to analyze test journals may be found on the TetWorks page. The linked page has a way to upload a journal file and see several different forms of report, the grw tool is suggested as it will generate a colorized html report that is easy to read.

Retrieved from "http://www.linuxstandards.org/en/Platform_Tests"

[Article] [Discussion] [View source] [History]

Copyright © 2008 Linux Foundation. All rights reserved.
LSB is a trademark of the Linux Foundation. Linux is a registered trademark of Linus Torvalds.

Please see our privacy policy.