VIDIOC_DQBUF hangs

[ghost]

User: Bad news! It stayed up from 9/2/2014 through 9/11/2014. All analog cameras has stopped working; however, the 2 IP cameras are still up. As we can see, it has been 5 days since it stopped working.

lspci: 04:05.0 Multimedia video controller: Bluecherry BC-H16480A 16 port H.264 video and audio encoder / decoder
solo6010-dkms ver 2.4.10
bluecherry ver 2.2.4

GDB traces show that ioctl(VIDIOC_DQBUF) is hanging on all card's video ports.

[ghost]

GDB info dumps are here: http://lizard.bluecherry.net/~autkin/solo_issue_74/

[ghost]

# uname -a
Linux fwcams 3.13.0-34-generic #60-Ubuntu SMP Wed Aug 13 15:45:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[ghost]

CPU: 4-cores Intel(R) Core(TM) i5 CPU 650 @ 3.20GHz

[ghost]

Probably the same happens in other case. There's installation with 3 solo cards. Streams from solo channels freeze after approx an hour of running, either with driver included in recent kernel or with solo6x10-edge from our repo.
With 3.17.0 kernel and upstream driver, if we restart bluecherry service after the freeze, such warning is given in dmesg:

[ 3046.156811] ------------[ cut here ]------------
[ 3046.156818] WARNING: CPU: 1 PID: 2469 at drivers/media/v4l2-core/videobuf2-core.c:2144 __vb2_queue_cancel+0x11e/0x180 [videobuf2_core]()
[ 3046.156819] Modules linked in: solo6x10 videobuf2_dma_contig videobuf2_dma_sg videobuf2_memops videobuf2_core v4l2_common videodev snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi rfcomm bnep bluetooth intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper lpc_ich shpchp winbond_cir cryptd rc_core ipmi_si video mac_hid parport_pc ppdev dm_mirror dm_region_hash dm_log dm_snapshot dm_bufio lp parport hid_generic ahci e1000e libahci usbhid hid ptp megaraid_sas pps_core
[ 3046.156848] CPU: 1 PID: 2469 Comm: bc-server Not tainted 3.17.0 #1
[ 3046.156849] Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS S1200BT.86B.02.00.0041.120520121743 12/05/2012
[ 3046.156850]  0000000000000009 ffff880226e9fb80 ffffffff817480e7 0000000000000000
[ 3046.156852]  ffff880226e9fbb8 ffffffff8106c97d 0000000000000000 ffff88021c1c0090
[ 3046.156853]  ffff88002dd80ad8 ffff88021c1e0550 ffff8800bddef740 ffff880226e9fbc8
[ 3046.156855] Call Trace:
[ 3046.156860]  [<ffffffff817480e7>] dump_stack+0x45/0x56
[ 3046.156863]  [<ffffffff8106c97d>] warn_slowpath_common+0x7d/0xa0
[ 3046.156872]  [<ffffffff8106ca5a>] warn_slowpath_null+0x1a/0x20
[ 3046.156874]  [<ffffffffa03f873e>] __vb2_queue_cancel+0x11e/0x180 [videobuf2_core]
[ 3046.156882]  [<ffffffffa03fbe5a>] vb2_queue_release+0x1a/0x30 [videobuf2_core]
[ 3046.156888]  [<ffffffffa03fbed1>] _vb2_fop_release+0x61/0xa0 [videobuf2_core]
[ 3046.156890]  [<ffffffffa03fbf3a>] vb2_fop_release+0x2a/0x50 [videobuf2_core]
[ 3046.156894]  [<ffffffffa03d0434>] v4l2_release+0x34/0x80 [videodev]
[ 3046.156898]  [<ffffffff811d5d74>] __fput+0xe4/0x220
[ 3046.156899]  [<ffffffff811d5efe>] ____fput+0xe/0x10
[ 3046.156903]  [<ffffffff81088b54>] task_work_run+0xc4/0xe0
[ 3046.156905]  [<ffffffff8106eec9>] do_exit+0x2d9/0xa80
[ 3046.156908]  [<ffffffff810a2fc2>] ? enqueue_task_fair+0x422/0x6d0
[ 3046.156910]  [<ffffffff8101c1e9>] ? sched_clock+0x9/0x10
[ 3046.156912]  [<ffffffff8106f6ef>] do_group_exit+0x3f/0xa0
[ 3046.156914]  [<ffffffff8107b17b>] get_signal+0x1db/0x720
[ 3046.156926]  [<ffffffff81012528>] do_signal+0x28/0x9c0
[ 3046.156929]  [<ffffffffa03d0614>] ? v4l2_ioctl+0x84/0x150 [videodev]
[ 3046.156932]  [<ffffffff811e70b0>] ? do_vfs_ioctl+0x2e0/0x4c0
[ 3046.156933]  [<ffffffff81012f29>] do_notify_resume+0x69/0xb0
[ 3046.156935]  [<ffffffff81750a2a>] int_signal+0x12/0x17
[ 3046.156937] ---[ end trace 71d6cd303051b8d7 ]---

BTW Upstream driver also produces a lot of artifacts - frames from other cameras get into video randomly.

[ghost]

Some context for last warning:

drivers/media/v4l2-core/videobuf2-core.c +2144

        /*
         * If you see this warning, then the driver isn't cleaning up properly
         * in stop_streaming(). See the stop_streaming() documentation in
         * videobuf2-core.h for more information how buffers should be returned
         * to vb2 in stop_streaming().
         */
        if (WARN_ON(atomic_read(&q->owned_by_drv_count))) {
                for (i = 0; i < q->num_buffers; ++i)
                        if (q->bufs[i]->state == VB2_BUF_STATE_ACTIVE)
                                vb2_buffer_done(q->bufs[i], VB2_BUF_STATE_ERROR);
                /* Must be zero now */
                WARN_ON(atomic_read(&q->owned_by_drv_count));
        }

include/media/videobuf2-core.h
 * @stop_streaming:     called when 'streaming' state must be disabled; driver
 *                      should stop any DMA transactions or wait until they
 *                      finish and give back all buffers it got from buf_queue()
 *                      callback by calling @vb2_buffer_done() with either
 *                      VB2_BUF_STATE_DONE or VB2_BUF_STATE_ERROR; may use
 *                      vb2_wait_for_all_buffers() function

[ghost]

I have noticed that solo cards don't stop all at the same time; I have met a condition/momentum with only single card's channels being frozen (but all of them).

[ghost]

Suspicious dmesg output from early boot time (before the loading of solo6x10 driver)

[   23.315042] ACPI Warning: SystemIO range 0x0000000000000428-0x000000000000042f conflicts with OpRegion 0x0000000000000400-0x000000000000047f (\PMIO) (20140724/utaddress-258)
[   23.315048] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[   23.315051] ACPI Warning: SystemIO range 0x0000000000000540-0x000000000000054f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140724/utaddress-258)
[   23.315053] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[   23.315054] ACPI Warning: SystemIO range 0x0000000000000530-0x000000000000053f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140724/utaddress-258)
[   23.315055] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[   23.315056] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000052f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140724/utaddress-258)
[   23.315058] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000052f conflicts with OpRegion 0x0000000000000500-0x000000000000052f (\_SI_.SIOR) (20140724/utaddress-258)
[   23.315060] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver

[ghost]

Driver module reloading doesn't help.

[curtishall]

Maybe ask someone on the v4l list?

On Mon, Oct 20, 2014 at 7:31 PM, Andrey Utkin notifications@github.com
wrote:

Driver module reloading doesn't help.

—
Reply to this email directly or view it on GitHub
#74 (comment)
.

Curtis Hall (curt@bluecherry.net curt@corp.bluecherry.net)
Bluecherry - www.bluecherrydvr.com / store.bluecherry.net
(877) 418-3391 x 283
(573) 642-6161 x 283