Hot questions for Using ZeroMQ in django

Question:

I'm migrating a project to python on docker for a better "facility" for development. Ut's a project that runs with python 2.7, Django 1.6.8 and several RPC dependencies

I find myself with a crash of the application with this stacktrace that I can't find the missing module (No module named coros).

I'm thinking it should either be an addiction problem related to a version that is not good or an addiction installed on the default bone but not on the docker bone. Do you have any ideas or suggestions to correct this problem?

My Dockerfile

FROM python:2

WORKDIR /var/www
RUN echo "deb http://download.opensuse.org/repositories/network:/messaging:/zeromq:/release-stable/Debian_9.0/ ./" >> /etc/apt/sources.list
RUN wget https://download.opensuse.org/repositories/network:/messaging:/zeromq:/release-stable/Debian_9.0/Release.key -O- | apt-key add
RUN apt-get update 
RUN apt-get install libzmq3-dev  libev-dev -y 


RUN apt-get install -y  bash \
            git \
            default-libmysqlclient-dev \
            libldap2-dev \
            libsasl2-dev \
            libssl-dev \
            python-gevent \
            python-gevent-websocket \
            build-essential \
            python-dev \
            locales

RUN pip install django-debug-toolbar==1.2.1
RUN apt-get install libevent-dev -y 
RUN apt-get install python-all-dev -y 
RUN pip install gevent
RUN pip install greenlet
RUN pip install --upgrade pip
ADD requirements.txt ./
RUN pip install -r requirements.txt
RUN pip install pyzmq

My requirements file

crispy-forms-foundation==0.2.3.1
django-crispy-forms==1.3.2
Django==1.6.8
django-auth-ldap==1.1.4
django-pipeline==1.3.14
iso8601==0.1.8
MySQL-python==1.2.5
nameko==2.1.2
python-ldap==2.4.10
python-memcached==1.53
pytz==2013.8
django-widget-tweaks==1.3
Babel==1.3
django-braces==1.2.2
PyReact==0.2.0
raven==3.2.1
djangorestframework==3.1.3
django-statsd-mozilla==0.3.12
boto==2.8.0
libthumbor==1.0.1
Pillow==2.5.1
django-debug-toolbar==1.2.2
South==1.0.1
zerorpc==0.4.4
requests>=2.7.0,<3.0.0
python-updict==0.1.2
graphene==0.10.2
beautifulsoup4==4.5.1
ciso8601==1.0.2

Error traceback

app_1       | Traceback (most recent call last):
app_1       |   File "/usr/local/lib/python2.7/wsgiref/handlers.py", line 85, in run
app_1       |     self.result = application(self.environ, self.start_response)
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/contrib/staticfiles/handlers.py", line 67, in __call__
app_1       |     return self.application(environ, start_response)
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 187, in __call__
app_1       |     self.load_middleware()
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 45, in load_middleware
app_1       |     mw_class = import_by_path(middleware_path)
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/utils/module_loading.py", line 26, in import_by_path
app_1       |     sys.exc_info()[2])
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/utils/module_loading.py", line 21, in import_by_path
app_1       |     module = import_module(module_path)
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/utils/importlib.py", line 40, in import_module
app_1       |     __import__(name)
app_1       |   File "/var/www/ofsdashboards/common/middleware/login_required.py", line 7, in <module>
app_1       |     compile(reverse('ofsdashboards.account.views.login').lstrip('/')),
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 536, in reverse
app_1       |     return iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs))
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 403, in _reverse_with_prefix
app_1       |     self._populate()
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 267, in _populate
app_1       |     for pattern in reversed(self.url_patterns):
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 365, in url_patterns
app_1       |     patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 360, in urlconf_module
app_1       |     self._urlconf_module = import_module(self.urlconf_name)
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/utils/importlib.py", line 40, in import_module
app_1       |     __import__(name)
app_1       |   File "/var/www/ofsdashboards/urls.py", line 33, in <module>
app_1       |     url(r'^homes/', include('ofsdashboards.homes.urls')),
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/conf/urls/__init__.py", line 26, in include
app_1       |     urlconf_module = import_module(urlconf_module)
app_1       |   File "/usr/local/lib/python2.7/site-packages/django/utils/importlib.py", line 40, in import_module
app_1       |     __import__(name)
app_1       |   File "/var/www/ofsdashboards/homes/urls.py", line 5, in <module>
app_1       |     from .dashboard import views as dashboard_views
app_1       |   File "/var/www/ofsdashboards/homes/dashboard/views.py", line 17, in <module>
app_1       |     from ofsdashboards.common.react_renderer import render
app_1       |   File "/var/www/ofsdashboards/common/react_renderer.py", line 2, in <module>
app_1       |     import zerorpc
app_1       |   File "/usr/local/lib/python2.7/site-packages/zerorpc/__init__.py", line 28, in <module>
app_1       |     from .socket import *
app_1       |   File "/usr/local/lib/python2.7/site-packages/zerorpc/socket.py", line 27, in <module>
app_1       |     from .events import Events
app_1       |   File "/usr/local/lib/python2.7/site-packages/zerorpc/events.py", line 31, in <module>
app_1       |     import gevent.coros
app_1       | ImproperlyConfigured: Error importing module ofsdashboards.common.middleware.login_required: "No module named coros"

Answer:

The traceback shows that zerorpc 0.4.4 is trying to import gevent.coreos, which was removed in gevent 1.2.

Eventually, you want to upgrade zerorpc to a newer version that doesn't try to import gevent.coreos, but that may require further dependency/code changes.

A more minimal change would be to try installing gevent<1.2 instead.

Question:

I use python 3.6.6 anaconda 64 bit windows, zmq 4.2.5 I have very strange behavior: function can't return.

Server:

import zmq
import json

def main():
    context = zmq.Context()
    socket = context.socket(zmq.REP)
    #socket.setsockopt(zmq.RCVTIMEO, 1000)
    socket.setsockopt(zmq.SNDTIMEO, 1000)
    socket.bind(my_address)

    def send_response(**kwargs):
        try:
            #... Some kwargs standard preprocessing
            socket.send_string(json.dumps(kwargs))
        except zmq.ZMQBaseError:
            return False
        return True

    while True:
        try:
            msg = socket.recv().decode("utf-8")
        except zmq.ZMQBaseError:
            continue
        #... Processing
        if not send_response(error='Everything is wrong'):
            continue
        #... Processing

Client:

import zmq
import json

def do_request(**kwargs):
    context = zmq.Context()
    socket = context.socket(zmq.REQ)
    socket.setsockopt(zmq.RCVTIMEO, 5000)
    socket.setsockopt(zmq.SNDTIMEO, 1000)
    socket.connect(my_address)

    #... kwargs pre-rocessing
    try:
        socket.send_string(json.dumps(kwargs))
        response = json.loads(socket.recv().decode("utf-8"))
        #... Processing
        return response
    except zmq.ZMQBaseError as e:
        print(e)
        print("PING1!!!")
        return dict(error='Service temporarily unavailable', r_status=503)

def called_from_view():
    response = do_request(command='ping')
    print("PING2!!!")

It was working. I don't know what has changed. But now it is broken. When server is shut down, client's do_request doesn't return the Service temporarily unavailable. The response = json.loads(socket.recv().decode("utf-8")) throws an exception. In except section python prints the PING1, but it doesn't print PING2 in called_from_view after that. Even with the empty return. I can't figure out why.

I create new socket in do_request because the client's called_from_view is called from django's view. I had other problems before with the global socket.


Answer:

Add this option after creating your socket in the client (you can keep your other options too):

socket = context.socket(zmq.REQ)
socket.setsockopt(zmq.LINGER, True) # or False, my testing says it works both ways.

See this SO answer for reference. Basically, the garbage collector is trying to pick up the pieces when returning from do_request and the zmq socket is stopping it from finishing and it hangs indefinitely.

Question:

I am writing a Django web application where the user can do different operations from every view and when the user submit the form a JSON is pushed to ZeroMQ and waits for a response (REQ-REP).

My problem is that the operations are too slow. I don't know if the problems occur becouse I am using a low performance Debian virtualized in VirtualBox but I think the problem is that when I call the function (see below), I do a new connection every time and the connection take like 5 seconds.

def push(obj):
    try:
        context = zmq.Context()
        socket = context.socket(zmq.REQ)
        socket.setsockopt(zmq.LINGER, 0)
        socket.connect('tcp://127.0.0.1:8888')
        socket.send_json(obj)
        poller = zmq.Poller()
        poller.register(socket, zmq.POLLIN)
        if poller.poll(3*1000):
            socket.recv()
            return True
        else:
            return False
    except:
        return False
    finally:
        socket.close()
        context.term()

This is the other side code:

context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind('tcp://127.0.0.1:8888')

while True:
    obj = str(socket.recv_json())
    print obj
    socket.send('ACK')

What am I doing wrong? Maybe should I think I need to keep the connection opened for all the user session, but how? Any tips are appreciated.

Thank you!


Answer:

I have finelly solved the problem unistalling libzmq-dev installed from debian repository (I think I was using 2.2 but I am not sure) and installing PyZMQ and ZMQ using pip with this command:

pip install --no-use-wheel pyzmq

I am not sure which version is installed now or which version is the best for a stable production environment with python 2.7, but it works fast now!

Thanks for the help.