Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read until very slow comparing to Paramiko #425

Open
michalwitwicki opened this issue Feb 9, 2023 · 2 comments · May be fixed by #431
Open

Read until very slow comparing to Paramiko #425

michalwitwicki opened this issue Feb 9, 2023 · 2 comments · May be fixed by #431

Comments

@michalwitwicki
Copy link

Hi,
I have similar (or same) issue as discussed in following tickets: #338, #347, #151 and #203 - basically Read Until is slow.

I have created setup to compare Read Until directly to Paramiko, that comparison is not exactly 1-to-1 but I think it is enough to show the problem.

requirements.txt

bcrypt==4.0.1
cffi==1.15.1
cryptography==39.0.0
paramiko==2.12.0
pycparser==2.21
PyNaCl==1.5.0
robotframework==6.0.2
robotframework-sshlibrary==3.8.0
scp==0.14.4
six==1.16.0

long_output.sh

#!/bin/bash
for i in {1..10000}
do
   echo "$i. long output long output long output"
done
echo "DONE"

long_output.robot

*** Settings ***
Library                SSHLibrary
Library                utils.py

*** Variables ***
${HOST}                
${USERNAME}            
${PASSWORD}            

*** Test Cases ***
Send file
    Open Connection     ${HOST}            timeout=120
    Login               ${USERNAME}        ${PASSWORD}
    Put File            ./long_output.sh   /root/
    Close All Connections

Paramiko test
    paramiko_write      cd /root && time ./long_output.sh    ${HOST}    ${USERNAME}    ${PASSWORD}

SSHLibrary test
    Open Connection     ${HOST}            timeout=120
    Login               ${USERNAME}        ${PASSWORD}
    Write               cd /root && time ./long_output.sh
    Read Until          DONE     
    Close All Connections

utils.py

import paramiko
from robot.api.deco import keyword, library
import robot.api.logger as logger

@library(scope='GLOBAL')
class utils:
    @keyword
    def paramiko_write(self, cmd, host, username, password):
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        ssh.connect(host, 22, username, password, timeout=10)

        stdin, stdout, stderr = ssh.exec_command(cmd)

        output = []
        for line in stdout.readlines():
            logger.info(line)
            output.append(line)

        ssh.close()

        return "\n".join(output)
    

This is how I started Robot: robot -d ./results -v HOST:<ip> -v USERNAME:<user> -v PASSWORD:<pass> long_output.robot

And finally the result of running above:
Zrzut ekranu 2023-02-09 152241

I am running this from virtual machine with Fedora and machine I am connecting to has CentOS.
I am using Python3.

All files to download:
shhlib_slow_read_until.zip

Any idea where problem might be?

Thanks,
Michał Witwicki

@michalwitwicki michalwitwicki changed the title Read until very slow comparing to direct Paramiko Read until very slow comparing to Paramiko Feb 9, 2023
@raychenv
Copy link

raychenv commented Feb 10, 2023

Check commit 8e77942
The bottleneck may be here in _read_until.
self.read_char reads only a character each loop. And need match every output. It is really time consuming for long output.

Should use Execute Command keyword to get the output and check.

@michalwitwicki
Copy link
Author

@raychenv
Thanks for explanation.
Execute Command is indeed much faster but still not as fast as paramiko approach.
Thank you for suggestion anyway.

@uruun uruun linked a pull request Jun 30, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants