Mixing Python with Elixir II

Important: Read Part I of this post before continuing.

In Part I, we looked at how to call Python functions from Elixir using Erlport!

ErlPort is an Elixir library which makes it easier to connect Erlang to a number of other programming languages using the Erlang port protocol.

However, as you may have noticed, the calling process had to wait for Python to finish processing and return the results. This works for operations that are instant and doesn’t take much time. Otherwise, it’s not much fun when you are running an intensive Python function. Because your Elixir program is gonna hang waiting for the Python code to finish.

For operations that take a while to complete we need a way to call the Python function and get notified in our elixir code when the function is complete — similar to how we write elixir program with processes talking to each asynchronously. This is useful, especially in cases when you want an Elixir process to control/supervise several Python operations. We can run the Python functions concurrently which makes life a little better.

In this post, we’ll explore how to achieve this — Asynchronous communication between Python and Elixir!

We can reason about the challenge the same way we do in communicating between Elixir processes — by using cast — which allows us to communicate between processes asynchronously.

Previously, we used erlport’s :python.call/4 mostly when calling functions from Python. This is synchronous and the elixir program had to wait for the python function to finish and send response.

In this post, we will use :python.cast/2

Understanding :python.cast/2

:python.cast/2 works the same as GenServer.cast/2. You give it process id (or registered process name) and the message to send to that process. Unlike :python.call variant, :python.cast is used to send message instead of calling a Python function. Same way your Elixir processes are sent messages. Erlport provides a mechanism for us to receive messages sent to the python instance.

Erlport Python Module

Erlport offers a python library that helps with working with elixir data types and processes. You can install erlport using pip.

pip install erlport #don't run this yet!

However, the Erlport Elixir repo already comes with the Python library which are automatically loaded at runtime. So you don’t have to install the Erlport Python library if you are using Erlport through Elixir.

Below is how data is mapped between Erlang/Elixir and Python

Erlang/Elixir data types mapped to Python data types (http://erlport.org/docs/python.html)Python data types mapped to Erlang/Elixir data types (http://erlport.org/docs/python.html)

Handling Message Sent from Elixir to Python

For Elixir processes you have a receive block in which all messages are handled. Or the handle_call, handle_cast and handle_info functions if you are using GenServer. Erlport provides a similar mechanism for handling messages sent to the Python process. set_message_handler is the function. It’s in the erlport.erlang module. It takes a Python function with one argument-message i.e the incoming message. Erlport will then call the function passed to set_message_handler whenever a message is received. It’s like a callback function.

Sending Message from Python to Elixir

Since an Elixir process isn’t aware of who sent a cast message, we have to find a way to know which process to send the result to once the python function completes. One way is to register the process id, of the Elixir process that you want the result sent to.

The Erlport python module provides cast function for sending a message to Elixir process. With this we can send asynchronously message back to the elixir process when the function is done!

Putting it all together

Open Terminal and create a new Elixir project using mix

$ mix new elixir_python

Add dependency

Add erlport to your dependencies mix.exs

defp deps do    [     {:erlport, "~> 0.9"},    ]  end

Then install project dependencies.

$ cd elixir_python$ mix deps.get

Create priv/python directory where you’ll keep our python modules

$ mkdir -p priv/python

Create an elixir module wrapper for Erlport related functions.

#lib/python.exdefmodule ElixirPython.Python do

    @doc """      Start python instance with custom modules dir priv/python      """def start() do         path = [             :code.priv_dir(:elixir_python),              "python"        ]|> Path.join()        {:ok, pid} = :python.start([           {:python_path, to_charlist(path)}        ])        pid

    end

    def call(pid, m, f, a \\ []) do        :python.call(pid, m, f, a)enddef cast(pid, message) do        :python.cast(pid, message)enddef stop(pid) do        :python.stop(pid)end

end

In order for Elixir process to receive message asynchronously, we will create a simple GenServer module

#lib/python_server.exdefmodule ElixirPython.PythonServer do    use GenServer   alias ElixirPython.Python

   def start_link() do GenServer.start_link(__MODULE__, [])end

   def init(args) do#start the python session and keep pid in state      python_session = Python.start()      #register this process as the message handler      Python.call(python_session, :test, :register_handler, [self()])      {:ok, python_session}end

   def cast_count(count) do {:ok, pid} = start_link()      GenServer.cast(pid, {:count, count})end   def call_count(count) do{:ok, pid} = start_link()      # :infinity timeout only for demo purposes      GenServer.call(pid, {:count, count}, :infinity)end   def handle_call({:count, count}, from, session) do result = Python.call(session, :test, :long_counter, [count])      {:reply, result, session}end

   def handle_cast({:count, count}, session) do Python.cast(session, count)     {:noreply, session}end

   def handle_info({:python, message}, session) do         IO.puts("Received message from python: #{inspect message}")      #stop elixir process      {:stop, :normal,  session}end

   def terminate(_reason, session) do Python.stop(session)     :okend

end

Now lets create the python module priv/python/test.py

#priv/python/test.pyimport timeimport sys#import erlport modules and functionsfrom erlport.erlang import set_message_handler, castfrom erlport.erlterms import Atommessage_handler = None #reference to the elixir process to send result todef cast_message(pid, message):    cast(pid, message)def register_handler(pid):    #save message handler pidglobal message_handler    message_handler = piddef handle_message(count):try:        print "Received message from Elixir"        print count        result = long_counter(count)        if message_handler:           #build a tuple to atom {:python, result}           cast_message(message_handler, (Atom('python'), result))except Exception, e:      # you can send error to elixir process here too      # print epassdef long_counter(count=100):    #simluate a time consuming python function    i = 0    data = []while i < count:        time.sleep(1) #sleep for 1 sec        data.append(i+1)        i = i + 1return dataset_message_handler(handle_message) #set handle_message to receive all messages sent to this python instance

Compile!

$ mix compile

Run!

$ iex -S mixiex(1)> ElixirPython.PythonServer.call_count(4)[1, 2, 3, 4] #printed after waiting 4seciex(2)> ElixirPython.PythonServer.call_count(10)[1, 2, 3, 4, 5, 6, 7, 8, 9, 10] #printed after waiting 10sec!

Also notice that call_count causes iex to hang till we get the back result from python

Now let’s call same python function asynchronously!

iex(3)> ElixirPython.PythonServer.cast_count(4):okReceived message from Elixir4Received message from python: [1, 2, 3, 4] # printed after 4sec, no waitiex(4)> ElixirPython.PythonServer.cast_count(10):okReceived message from Elixir10iex(5)>nilReceived message from python: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] #printed after 10sec without blocking

Notice how we can continue to work in iex and get the result once our python function completes. Try callingcast_count with different numbers — large and small.

That is it! Now you can communicate between Elixir and Python asynchronously.

Happy coding!

Mixing Python with Elixir II was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.