Delays between requests

[stiebrs]

Hi,
First of all - thanks for nice driver and support, love it :)

I'm trying to utilize FT232H in 8MBaud UART mode hooked to RS485, that seems to work all right, don't even get too many errors, apart from overflow :) What bugs me, is the delay between packets:

Capture is taken from RS485 bus. First peak (at marker 1) is request for data packet, then comes data packet itself (1024 bytes) and then next request is sent only 11-12 ms later. These delays seem to be consistent and independent of baudrate (same delays are present for 6MB and 3MB). Which limits me to 1000/13 = 76 kBps RX rate.

I understand the height of software stack (pyftdi, pyusb, libusb, kernel, etc), but 10ms is a long time in multi-gigahertz world. Is there any way to reduce these delays to some reasonable amount (tweaking USB settings/kernel/whatever?). Do you have any experience with their C library? Is it faster?

[eblot]

Hi,

Are these delays captured with read, write or both directions?

[stiebrs]

Wow, you're fast :)
Traces are half-duplex RS485. First is write request (at marker), then read (about 250us it takes for my device to fetch the data and dump it), then write request at second marker, followed by read.
I read exact number of bytes and timeout is set small enough:
self.ser.timeout = 0.000006
self.ser.write(request)
self.ser.read(1024)

[eblot]

Could you try to apply #107?

This is an issue I never took time to validate as I was more focused on I2C/SPI/GPIO support. I'll try to find time to reproduce this issue, but as it is a one-liner patch, it is worth trying it.

[eblot]

I wrote two tests:

• ftdi://:2232h/1 TXD -> ftdi://:2232h/2 RXD (one port write to the other port)
• ftdi://:232h/1 TXD -> ftdi://:232h/1 RXD (loopback)

In both cases, on a 1000000 continuous byte sequence, I do not observe stall delays greater than 1.2 ms (w/o applying #107).

Maybe I did not get how you actually sequence the calls to PyFtdi. Would you have a code snippet to share?

[stiebrs]

I'm trying to receive 1024 byte packets at a time in a loop (for my communications protocol, it contains headers, CRC and other crap, which I ignore for debugging of this):

		self.ser.timeout = 0.0006
		for p in packet_requests:
			self.ser.write(p)
			r = self.ser.read(1024)

"packet_requests" in my case is just an array of preformatted strings with valid commands for my peripheral device.

I run it with python unittest framework, maybe that adds some overhead. Also I have tons of USB devices attached, which might add extra delays. I thought I would try to minimize external influence on USB bus first, before reporting back (use external oscilloscope, remove all other unnecessary USB peripherals, etc). Applying a version of #107 (that commit seems a bit dated, reduced in serialext/protocol_ftdy.py line 83 (v0.50.2) instead) did not seem to change much. Which made me think about busy USB host.

[eblot]

You can also try to shorten your packets. 1024 is the exact FT232H FIFO size, and unfortunately there is no proper way to manage it finely. If you write packets, trying spitting them @ 512 bytes -max- (protocol overhead included. 512 is also the max USB packet size for FT232H btw. If you read packets, try also reading “shorter” packets to see if it helps with this latency.

[stiebrs]

From FT232H documentation v2.0 section 4.2 (page 19) I understood, that it has 2x1024 byte FIFOs:

Port FIFO TX Buffer (1Kbytes). Data from the Host PC is stored in these buffers to be used by the Multi-purpose UART/FIFO controllers. This is controlled by the USB Protocol Engine and FIFO control block.
Port FIFO RX Buffer (1Kbytes). Data from the Multi-purpose UART/FIFO controllers is stored in these blocks to be sent back to the Host PC when requested. This is controlled by the USB Protocol Engine and FIFO control block.

That's why I chose 1KB packet size. I'll try to reduce packet size as well.

[eblot]

Yes I do know. There is an PyFtdi API to retrieve the FIFO size btw :)

The problem is that there is no flag but “Tx FIFO empty” in FTDI HW, which means it is impossible to optimize the instant when to send a new packet. If you push 1024 byte buffer into a not empty 1024 byte FIFO, you force a stall, and new retries...

It is worth on RX size as if you poll too much the Host CPU load rises quickly, if you poll a bit too slow, you lose bytes. There is no USB interrupt in FTDI HW....

[stiebrs]

Maybe it works fine if you adjust your TX/RX sizes to fill the buffer exactly, but in real-life scenarios you'll have different lengths of data. Found out, that the issue is related to latency timer. I removed the 12ms limit in driver, empirically found that in my particular case 6-8ms is reasonably stable, pushing it any lower wreaks havoc in comms (python only).
There's a whole another can of worms with how the driver sets/resets latency (I have to set my target latency each time before TX/RX and once execution is done system-wide latency is set to 255), but for my intent and purpose (functional testing, not performance) this is working fine. Final interfacing driver will be written in C with that I can get 4-6MBaud reasonably reliably.

Thanks for the help!

[eblot]

I do not think it should be closed now. Need to give a try to a 1 ms latency.
See also #199.