Debian - open-iscsi error

[Schnippzle]

Next issue!

Code:

Oct 22 19:12:45 tinkerboard kernel: [92868.298146]  connection1:0: detected conn error (1020)
Oct 22 19:13:18 tinkerboard kernel: [92900.958163]  connection1:0: detected conn error (1020)
Oct 22 19:13:40 tinkerboard kernel: [92923.378172]  connection1:0: detected conn error (1020)

Seeing this error as soon as I try to write to the drive.  Reading is fine, no issues, but writing creates these errors in /var/log/messages and write speeds are sub 1MB/s.

Any further assistance would be greatly appreciated and thank you in advance!

[Craz_tyle]

I have no any idea about iscsi...
here is some from google's forum https://groups.google.com/forum/#!topic/...y8Y3UyCEFA
and some data shows, you can try to let server-side disable "IMMEDIATEDATA".

I think you can try to find more reference from Google, by using this kind keyword "open-iscsi detected conn error (1020)"

[Schnippzle]

Thanks for the quick reply.  Will have a look, thanks.  I did have a Google beforehand, but as this worked fine with Raspbian before I wasn't sure if it was something TinkerOS-specific.

Will investigate further and update.

[Schnippzle]

Update:  I set the following parameters in iscsi.config and rediscovered the node.

Code:

node.conn[0].timeo.noop_out_interval = 0

node.conn[0].timeo.noop_out_timeout = 0

I have now got a more consistent write speed, however it doesn't top 4mbit/s....which is a bit poor.  Continuing troubleshooting.

I have also had a look at this:

Code:

$ cat /sys/block/sdg/queue/max_hw_sectors_kb
32767
$ cat /sys/block/sdg/queue/max_sectors_kb
1280

And found that my max_sectors_kb was set to 1280 whilst my block size was actually 512.  Changed max_sectors_kb to 512 has done nothing though.

[Schnippzle]

It has been suggested that as a workaround I enable multi-pathing.  Thankfully I have two available nodes:

Code:

linaro@tinkerboard:~$ sudo iscsiadm --mode session
tcp: [6] 192.168.1.29:3260,0 iqn.2004-10.com.thecus:iseries-000a40e02:default-target (non-flash)
tcp: [7] 192.168.1.13:3260,0 iqn.2004-10.com.thecus:iseries-000a40e02:default-target (non-flash)

However as expected when I go to install the multi-path service I get this:

Code:

linaro@tinkerboard:/mnt/seedbox$ sudo systemctl status multipathd.service
● multipathd.service - Device-Mapper Multipath Device Controller
  Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
  Active: failed (Result: exit-code) since Fri 2018-10-26 12:29:02 UTC; 1min 15s ago
 Process: 5418 ExecStartPre=/sbin/modprobe -a scsi_dh_alua scsi_dh_emc scsi_dh_rdac dm-multipath (code=exited, status=1/

Oct 26 12:29:02 tinkerboard modprobe[5418]: modprobe: WARNING: Module scsi_dh_alua not found in directory /lib/modules/4.
Oct 26 12:29:02 tinkerboard modprobe[5418]: modprobe: WARNING: Module scsi_dh_emc not found in directory /lib/modules/4.4
Oct 26 12:29:02 tinkerboard modprobe[5418]: modprobe: WARNING: Module scsi_dh_rdac not found in directory /lib/modules/4.
Oct 26 12:29:02 tinkerboard modprobe[5418]: modprobe: WARNING: Module dm-multipath not found in directory /lib/modules/4.
Oct 26 12:29:02 tinkerboard systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Oct 26 12:29:02 tinkerboard systemd[1]: multipathd.service: Failed with result 'exit-code'.

Will this require an additional kernel rebuild?

[Schnippzle]

Been playing around with the kernel configs and added the following:

Code:

CONFIG_SCSI_DH_ALUA=y
CONFIG_SCSI_DH_EMC=y
CONFIG_SCSI_DH_RDAC=y

I have also previously tried with 

Code:

CONFIG_SCSI_DH_ALUA=m
CONFIG_SCSI_DH_EMC=m
CONFIG_SCSI_DH_RDAC=m

I have also amended "CONFIG_MD_MULTIPATH=" to both m and y with the same result.

The results when trying to install multipath-tools is the same.

Quite stuck here and feel like I'm bashing my head against a wall.  Any help or assistance would be greatly appreciated.

[Schnippzle]

Enabled more granular debugging on the open-iscsi service and found this:

Code:

Oct 29 20:12:07 tinkerboard kernel: [18697.821045]  connection1:0: detected conn error (1020)
Oct 29 20:12:08 tinkerboard iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Oct 29 20:12:10 tinkerboard iscsid: connection1:0 is operational after recovery (1 attempts)
Oct 29 20:12:33 tinkerboard kernel: [18723.082548]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 1841307, last ping 1841808, now 1842309
Oct 29 20:12:33 tinkerboard kernel: [18723.082608]  connection1:0: detected conn error (1022)
Oct 29 20:12:33 tinkerboard iscsid: Kernel reported iSCSI connection 1:0 error (1022 - Invalid or unknown error code) state (3)
Oct 29 20:12:35 tinkerboard iscsid: connection1:0 is operational after recovery (1 attempts)
Oct 29 20:12:57 tinkerboard kernel: [18747.051035]  connection1:0: detected conn error (1020)
Oct 29 20:12:57 tinkerboard iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Oct 29 20:12:59 tinkerboard iscsid: connection1:0 is operational after recovery (1 attempts)
Oct 29 20:13:19 tinkerboard kernel: [18769.110999]  connection1:0: detected conn error (1020)
Oct 29 20:13:19 tinkerboard iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Oct 29 20:13:21 tinkerboard iscsid: connection1:0 is operational after recovery (1 attempts)
Oct 29 20:13:44 tinkerboard kernel: [18794.581050]  connection1:0: detected conn error (1020)
Oct 29 20:13:45 tinkerboard iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Oct 29 20:13:47 tinkerboard iscsid: connection1:0 is operational after recovery (1 attempts)

It would appear that the TCP connection issue persists!

[Craz_tyle]

Been playing around with the kernel configs and added the following:

Code:

CONFIG_SCSI_DH_ALUA=y
CONFIG_SCSI_DH_EMC=y
CONFIG_SCSI_DH_RDAC=y

I have also previously tried with 

Code:

CONFIG_SCSI_DH_ALUA=m
CONFIG_SCSI_DH_EMC=m
CONFIG_SCSI_DH_RDAC=m

I have also amended "CONFIG_MD_MULTIPATH=" to both m and y with the same result.

The results when trying to install multipath-tools is the same.

Quite stuck here and feel like I'm bashing my head against a wall.  Any help or assistance would be greatly appreciated.

About...multipath-tools needs,

need modify below Kernel config:
Adding following lines in "/arch/arm/configs/miniarm-rk3288_defconfig"'s last line.

Code:

CONFIG_SCSI_DH=y
CONFIG_SCSI_DH_ALUA=m
CONFIG_SCSI_DH_EMC=m
CONFIG_SCSI_DH_RDAC=m
CONFIG_DM_MULTIPATH=m

Then build again.
The result should be as below:

Code:

$ sudo systemctl status multipathd.service
● multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-10-30 08:01:13 CST; 11h ago
 Main PID: 259 (multipathd)
   Status: "running"
   CGroup: /system.slice/multipathd.service
           └─259 /sbin/multipathd -d -s

Oct 30 08:01:14 tinkerboard multipathd[259]: 0x59a95dfd: failed in domap for addition of new path mmcblk1
Oct 30 08:01:14 tinkerboard multipathd[259]: uevent trigger error
Oct 30 08:01:14 tinkerboard multipathd[259]: zram0: add path (uevent)
Oct 30 08:01:14 tinkerboard multipathd[259]: zram0: HDIO_GETGEO failed with 25
Oct 30 08:01:14 tinkerboard multipathd[259]: zram0: failed to get udev uid: Invalid argument
Oct 30 08:01:14 tinkerboard multipathd[259]: zram0: failed to get path uid
Oct 30 08:01:14 tinkerboard multipathd[259]: uevent trigger error
Oct 30 08:01:14 tinkerboard multipathd[259]: dm-0: remove map (uevent)
Oct 30 19:32:03 tinkerboard multipathd[259]: zram0: triggering change event to reinitialize
Oct 30 19:32:03 tinkerboard multipathd[259]: zram0: add path (uevent)

[Schnippzle]

Thanks for the advice, appreciated. Will rebuild and try again!

[Schnippzle]

Multipath successfully installed, but issues persist. I have decided to brute-force this with an rsync workaround:

rsync -avP <source> <destination>

I'm still getting the TCP drop errors, but the rsync command will resume again. Unfortunate, but it's a temporary way forward. Big thanks to Craz for helping out.

Feel free to close this thread.