python – SSHException ('No existing session') when starting 500 multiprocessing processes creating ssh connections using paramiko

Question:

I am trying to create a script for configuring hosts over ssh via paramiko using the multiprocessing module. Since the nodes are about 500, processing them in turn takes a lot of time. There is the following code:

import time
import sys
import paramiko
import socket
import re
import multiprocessing
host = open('/home/user/hosts', 'r')
hosts = host.readlines()
host.close()
def connect_ssh(host, queue):
        try:
            remote_conn_pre = paramiko.SSHClient()

            remote_conn_pre.set_missing_host_key_policy(
            paramiko.AutoAddPolicy())

            remote_conn_pre.connect(host, username=username, password=password, look_for_keys=False, allow_agent=False, timeout=5)
            time.sleep(0.2)

            print("SSH connection established to %s" % host)
            remote_conn = remote_conn_pre.invoke_shell()
            print("Interactive SSH session established")

            ###здесь операции с хостами####

               queue.put({host: output})

        except (socket.gaierror,socket.timeout,TimeoutError):
                print("Could not connect to %s \n" % host)
                sys.exit(1) 


def send_commands(function):
    processes = []
    queue = multiprocessing.Queue()
    for l in range(len(hosts)):
        p = multiprocessing.Process(target = function, args = (hosts[l], queue))
        p.start()
        processes.append(p) 
    for p in processes:
        p.join()

    results = []
    for p in processes:
        results.append(queue.get())
    return results

send_commands(connect_ssh)

If you apply it to a small group of nodes (about 10 for example), then everything works correctly. But one has only to start execution on all 500 devices, then on half (and randomly) the following error falls out:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "scripts/qos_add_mp.py", line 24, in connect_ssh
    remote_conn_pre.connect(host, username=username, password=password, look_for_keys=False, allow_agent=False, timeout=5)
  File "/usr/local/lib/python3.5/dist-packages/paramiko/client.py", line 341, in connect
    server_key = t.get_remote_server_key()
  File "/usr/local/lib/python3.5/dist-packages/paramiko/transport.py", line 670, in get_remote_server_key
    raise SSHException('No existing session')

At the same time, as far as I understand, the error is typical for cases when the keys look_for_keys = False and allow_agent = False are not specified in connect paramiko. But I have them. What could be the catch?

Answer:

If you want to configure 500 hosts over ssh, check out a very cool tool – ansible

A lot of recipes have already been made for him, and in life the knowledge of such a tool is more useful than the invention of the bicycle.