Python file transfer (tcp socket), problem with slow network

python socket wait for response
send large file over socket python
python requests slow
python socket receive data
python socket select
python networking
python tcp socket
socket programming in python for file transfer

I setted up a secure socket using Tor and socks, but i'm facing a problem when sending large amount of data

Sender:

socket.send(message.encode())

Receiver:

chunks = []

while 1:
    part = connection.recv(4096)
    chunks.append(part.decode())

    if len(part) < 4096:
        break

response = "".join(chunks)

Since the network speed is not consistent in a loop i don't always fill the 4096b buffer, so the loop breaks and i don't receive the full data.

Lowering the buffer size doesn't seem an option because the "packet" size can be as low as 20b sometimes


TCP can split your package data in any amount of pieces it wants. So you should never rely on other end of a socket on the size of the packet received. You have to invent another mechanism for detecting end of message/end of file.

If you are going to send only one blob and close socket, then on server side you just read until you get False value:

while True:
    data = sock.recv(1024)
    if data:
        print(data)
        # continue 
    else:
        sock.close()
        break

If you are going to send multiple messages, you have to decide, what will be the separator between them. For text protocols it is a good idea to use lineending. You can then enjoy the power of Twisted LineReceiver protocol and others.

If you are doing a binary protocol, it's a common practice to preface your each message with size byte/word/dword.

Python file transfer (tcp socket), problem with slow network, I setted up a secure socket using Tor and socks, but i'm facing a problem when sending large amount of data. Sender: socket.send(message.encode()). File transfer is the process of copying or moving a file from a computer to another over a network or Internet connection. In this tutorial we'll go step by step on how you can write client/server Python scripts that handles that.


Try using structure to pass the length of the incoming data first to the receiver, "import struct". That way the receiving end knows exactly how much data to receive. In this example bytes are being sent over the socket, the examples here I've borrowed from my github upload github.com/nsk89/netcrypt for reference and cut out the encryption steps from the send function as well as it sending a serialised dictionary.

Edit I should also clarify that when you send data over the socket especially if your sending multiple messages they all sit in the stream as one long message. Not every message is 4096 bytes in length. If one is 2048 in length and the next 4096 and you receive 4096 on your buffers you'll receive the first message plus half of the next message or completely hang waiting for more data that doesn't exist.

data_to_send = struct.pack('>I', len(data_to_send)) + data_to_send # pack the length of data in the first four bytes of data stream, >I indicates internet byte order

    socket_object.sendall(data_to_send)  # transport data



def recv_message(socket_object):
    raw_msg_length = recv_all(socket_object, 4)  # receive first 4 bytes of data in stream
    if not raw_msg_length:
        return None

    # unpack first 4 bytes using network byte order to retrieve incoming message length
    msg_length = struct.unpack('>I', raw_msg_length)[0]

    return recv_all(socket_object, msg_length)  # recv rest of stream up to message length

def recv_all(socket_object, num_bytes):
    data = b''
    while len(data) < num_bytes:  # while amount of data recv is less than message length passed
        packet = socket_object.recv(num_bytes - len(data))  # recv remaining bytes/message
        if not packet:
            return None
        data += packet
    return data

Why are Python sockets so slow and what can be done?, I setted up a secure socket using Tor and socks, but i'm facing a problem when sending large amount of data. Sender: socket.send(message.encode()). The intention of this article is to learn how to transfer a text file over network through python program. This file transfer is based on server client model to use socket programming in python3+. Basic Set up Diagram: Here is the basic set up diagram to run this program.


By the way, no need to decode the every part before combine them to a chunk, combine all the parts to a chunk and then decode the chunk.

For your situation, the better way is using 2 steps.

Step1: sender send the size of the message, receiver take this size and ready to receive the message.

Step2: sender send the message, receiver combine the data if necessary.

Sender

# Step 1
socket.send( str(len(message.encode())).encode() ) 
# Step 2
socket.send(message.encode("utf-8"))   

Receiver

# Step 1
message_size = connection.recv(1024) 
print("Will receive message size:",message_size.decode())

# Step 2
recevied_size  = 0 
recevied_data = b''
  while recevied_size < int(message_size.decode()):
      part = connection.recv(1024)
      recevied_size += len(part)
      recevied_data += part
  else:
      print(recevied_data.decode("utf-8", "ignore"))
      print("message receive done ....",recevied_size)

Incomplete socket send with fast writer and slow reader · Issue #610 , Trio and Curio are at least twice slower than uvloop, processing 4K, 5K and 11K requests I am confident I can utilize 90% of 10G network if using all cores. My next problem depends on answer to the first one: what can be done to achieve read 220230, write 0, timeout 12 Requests/sec: 7333.85 Transfer/sec: 1.34MB. import socket # Import socket module: port = 50000 # Reserve a port for your service every new transfer wants a new port or you must wait. s = socket.socket() # Create a socket object: host = "" # Get local machine name: s.bind((host, port)) # Bind to the port


How to Work with TCP Sockets in Python (with Select Example), Expected: curl will successfully transfer the file. Actual: benhillis added the network label on Jul 7, 2016 Python 3.4 text file write failure #991. Closed select sometimes block forever on a ready TCP sockets #419. Closed. """ Server receiver of the file """ import socket import tqdm import os # device's IP address SERVER_HOST = "0.0.0.0" SERVER_PORT = 5001 # receive 4096 bytes each time BUFFER_SIZE = 4096 SEPARATOR = "<SEPARATOR>" # create the server socket # TCP socket s = socket.socket() # bind the socket to our local address s.bind((SERVER_HOST, SERVER_PORT


Issue 3766: socket.socket.recv broken (unbearably slow), This article is about low level work with TCP sockets in Python. has a module called socket which provides a low-level internet networking interface. To address this problem, there is a so-called asynchronous way of working with sockets. Here is the code to send a file from a local server to a local client. # server.py import socket # Import socket module port = 60000 # Reserve a port for your service. s = socket.socket() # Create a socket object host = socket.gethostname() # Get local machine name s.bind((host, port)) # Bind to the


Socket Programming HOWTO, I wrote a demo inside a single file using socketpair() to post here, but the issue was Here is the gist of it: Sending 500 packets @ 2 tokens each (500 very short Your code expects the kernel's TCP stack to send things as soon as you ask By the way, if you want to build network applications (clients and  To create a TCP-socket, you should use socket.AF_INET or socket.AF_INET6 for family and socket.SOCK_STREAM for type. Here’s a Python socket example: import socket s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) It returns a socket object which has the following main methods: bind() listen() accept() connect() send() recv()