Bug reporting

To begin to deal the problem, it is necessary to provide the trace file, performed with level 6 (except in cases when the problem can be easily reproduced by the described action). In addition to trace, if possible, to the error message, you can also attach Backtrace.

Trace file

General info

Trace - a permanent record to the file information about what is happening «inside» OpenMCU-ru during the conference. Trace will almost certainly need to try to understand why you have something was wrong with the sound or video, why OpenMCU-ru suddenly freezes, spontaneously stopped work or will something else than have to deal with.

Trace is performed with the specified trace level: from 0 (turned off) to 6 (the most detailed). Trace level can be set in the web interface on the settings on page.

The trace file can be very large if OpenMCU-ru worked for a long time. In order to obtain optimal image resolution, preferably:

  • restart service OpenMCU-ru;
  • as soon as possible provoke the problem;
  • stop service;
  • copy the file (on next start the trace file will be overwritten).

Before posting trace file to forum, it is desirable to archive it.

Trace file level 6, as a rule, contains information about the type of equipment, user names and IP-addresses used during video conferencing session. By sending someone a file with trace or publishing it elsewhere (eg by uploading to the forum), you consent to the processing of this information. As a rule, it is not very big secret, however, need you to understand that.

Location of the trace file depends on the operating system:

  • on Linux this folder is /var/log/openmcu-ru/
  • on Windows this folder is C:\Program Files\OpenMCU-ru\log
  • on FreeBSD this folder is /usr/local/share/openmcu-ru/log/

If you use a non-standard build - trace can theoretically be located elsewhere or be disabled entirely.

If the trace rotation is enabled in the settings, then the previous trace can be downloaded via the web interface:

http://openmcu:1420/trace0.txt

Trace example

Trace only at first glance seems complicated. You can start to learn it yourself, and learn many new and exciting things about H.323 and SIP.

In the first column is written time since the start of OpenMCU-ru in the format (HOURS:)MINUTES:SECONDS.milliseconds. In the second column (and sometimes third) - source of the event. Next - the event itself. Form of the logging is not very strict, especially at level 6, where many different dumps are written - in this case, the timestamp and the source of the event will be only in the first line of the dump.

At the beginning of the trace (when starting) you will see a lot of these lines:

  0:00.042	  OpenMCU.ru	H323	Added capability: SILK_B40{sw} <1>

This information about the codecs that can be used by OpenMCU-ru. If there are no codecs in ​​the list - maybe you forgot to plugin it.

The second column may contain (after the colon) identifier of the computing thread, for example:

H225 Answe...er:8120d40

This simplifies debugging, as most likely, the problem occurs only in one thread while the rest work as intended.

A few examples of what should be in «good» trace (if OpenMCU-ru is installed and working correctly):

Opened TCP port 1720 for incoming calls:

  0:00.041        OpenMCU.ru    PWLib   Created thread 0x8120328 H323 Listener:%0x
  0:00.041        OpenMCU.ru    PWLib   File handle high water mark set: 17 PTCPSocket
  0:00.041        OpenMCU.ru    H323    Started listener Listener[ip$*:1720]
  0:00.042        OpenMCU.ru    PWLib   Thread high water mark set: 6
  0:00.042      H323 Liste...er:8120328 PWLib   Started thread 0x8120328 H323 Listener:8120328
  0:00.042      H323 Liste...er:8120328 H323    Awaiting TCP connections on port 1720
  0:00.042      H323 Liste...er:8120328 TCP     Waiting on socket accept on ip$*:1720

Your time and thread IDs will be different, but everything else should be approximately similar.

Started the web server (with which you manage the conference) based on the ptlib library:

  0:00.054	  OpenMCU.ru	PWLib	Created thread 0x8121300 HTTP Service:%x
  0:00.054	  OpenMCU.ru	PWLib	Thread high water mark set: 7
  0:00.054	HTTP Servi...ce:8121300	PWLib	Started thread 0x8121300 HTTP Service:8121300

Created videoconference room101:

  0:00.055	  OpenMCU.ru	PColCnv	PColourConverter constructed: YUV420P 1408x1152 -> YUV420P 352x288
  0:00.055	  OpenMCU.ru	Conference	New conference started: ID=ec8e232a-68eb-e111-9471-001a64cafed8, number = room101
  0:00.055	  OpenMCU.ru	PColCnv	PColourConverter constructed: YUV420P 704x576 -> YUV420P 352x288
  0:00.056	  OpenMCU.ru	Conference	About to add member  to conference ec8e232a-68eb-e111-9471-001a64cafed8
  0:00.056	  OpenMCU.ru	Conference	Adding member 135404784 to conference ec8e232a-68eb-e111-9471-001a64cafed8
  0:00.056	  OpenMCU.ru	PWLib	File handle high water mark set: 22 Thread unblock pipe
  0:00.056	  OpenMCU.ru	PWLib	Created thread 0x8122710 
  0:00.056	  OpenMCU.ru	PWLib	Thread high water mark set: 8
  0:00.056	  OpenMCU.ru	PWLib	File handle high water mark set: 24 Thread unblock pipe
  0:00.056	  OpenMCU.ru	PWLib	Created thread 0x8122ad8 
  0:00.056	PSimpleThr...d:08122710	PWLib	Started thread 0x8122710 PSimpleThread:08122710
  0:00.056	  OpenMCU.ru	PWLib	Thread high water mark set: 9
  0:00.056	PSimpleThr...d:08122ad8	PWLib	Started thread 0x8122ad8 PSimpleThread:08122ad8

Backtrace

General info

Backtrace (also called a «stack trace») - is a list of functions called by the application during the application crash. This list can be useful when troubleshooting. As it allows developers to know which function failed.

First of all must be installed GNU Debugger. In Debian/Ubuntu this can be done by command:

apt-get install gdb

in CentOS/Fedora:

yum install gdb

To make a backtrace there are two ways:

Method 1 - «manually».

  • Run the application in the debugger (gdb).
  • Wait for the crash of the application (or generate it, if you know the sequence of actions leading to the crash).
  • Execute the necessary commands in the debugger to display backtrace.

This method has its advantages, but there are a few drawbacks. In particular, if the application crashs rarely and for unknown reasons, then wait for its crash in debugger - not the most convenient way.

Method 2 - «save coredump».

  • Enable saving memory dumps for application.
  • Run application and work with it as usual.
  • In the case of a application crash - coredump be will automatically created.
  • Once the application has crashed, it can be (automatically) restarted and continue to work - created coredump will not be harmed.
  • Obtained memory dumps can also be analyzed with gdb and get the backtrace.

The second method look at details.

Enable saving coredump

1. First, you need to specify for operating system where to save the coredump:

mkdir /var/cores
chmod 777 /var/cores
echo "kernel.core_pattern=/var/cores/core.%e.%t.%p.%s" >> /etc/sysctl.conf
sysctl kernel.core_pattern=/var/cores/core.%e.%t.%p.%s

In this example dumps will be saved to /var/cores in the format:

core.$EXECUTABLE_FILENAME.$UNIX_TIME.$PID.$SIGNAL_NUMBER

For example:

/var/cores/core.openmcu-ru.1381844450.32503.3

2. Need to allow the creation of dump directly for OpenMCU-ru service. To do this, uncomment line in the file /etc/default/openmcu-ru:

DAEMON_COREDUMP="unlimited"

And restart the service.

3. If necessary install debug symbols for OpenMCU-ru. In some distributions debug symbols located in separate packages. For example, for Ubuntu must be installed:

apt-get install openmcu-ru-dbg

At this preliminary stage is completed. Now we have to wait a crash of OpenMCU-ru. To verify that everything is done correctly, you can manually crash the application (it is better not to do this during a videoconference with director):

killall -SIGSEGV openmcu-ru

As a result of this command, the service must be crashed, and in the folder /var/cores/ debug dump should appear.

4. To obtain backtrace by coredump you can use this command:

gdb -batch \
  -ex 'set pagination 0' \
  -ex 'echo \n' \
  -ex 'backtrace' \
  -ex 'echo \n' \
  -ex 'info registers' \
  -ex 'echo \n' \
  -ex 'x/16i $pc' \
  -ex 'echo \n' \
  -ex 'thread apply all backtrace' \
  -ex 'quit' \
  /usr/bin/openmcu-ru \
  /var/cores/core.openmcu-ru.1381844450.32503.3 \
  > openmcu-ru_backtrace.log

Instead of the coredump in example you must substitute to yours. If the executable file openmcu-ru located on different path (eg /opt/openmcu-ru/bin/openmcu-ru) - you must also correct to yours.

As a result of this command will be created a file openmcu-ru_backtrace.log (in the current directory) which you must archive and attach to topic on the forum together with a description of the error.

See also