Python bindings for Clojure
EPL-2.0 License
import libpython_clj2.java_api;
[cnuernber/libpython-clj "1.36"]
(note name change).require-python
calls. These namespace will notWe aim to integrate Python into Clojure at a deep level. This means that we want to be able to load/use python modules almost as if they were Clojure namespaces. We also want to be able to use Clojure to extend Python objects. I gave a talk at Clojure Conj 2019 that outlines more of what is going on.
This code is a concrete example that generates an embedding for faces:
(ns facial-rec.face-feature
(:require [libpython-clj2.require :refer [require-python]]
[libpython-clj2.python :refer [py. py.. py.-] :as py]
[tech.v3.datatype :as dtype]))
(require-python 'mxnet
'(mxnet ndarray module io model))
(require-python 'cv2)
(require-python '[numpy :as np])
(defn load-model
[& {:keys [model-path checkpoint]
:or {model-path "models/recognition/model"
checkpoint 0}}]
(let [[sym arg-params aux-params] (mxnet.model/load_checkpoint model-path checkpoint)
all-layers (py. sym get_internals)
target-layer (py/get-item all-layers "fc1_output")
model (mxnet.module/Module :symbol target-layer
:context (mxnet/cpu)
:label_names nil)]
(py. model bind :data_shapes [["data" [1 3 112 112]]])
(py. model set_params arg-params aux-params)
model))
(defonce model (load-model))
(defn face->feature
[img-path]
(py/with-gil-stack-rc-context
(if-let [new-img (cv2/imread img-path)]
(let [new-img (cv2/cvtColor new-img cv2/COLOR_BGR2RGB)
new-img (np/transpose new-img [2 0 1])
input-blob (np/expand_dims new-img :axis 0)
data (mxnet.ndarray/array input-blob)
batch (mxnet.io/DataBatch :data [data])]
(py. model forward batch :is_train false)
(-> (py. model get_outputs)
first
(py. asnumpy)
(#(dtype/make-container :java-array :float32 %))))
(throw (Exception. (format "Failed to load img: %s" img-path))))))
(ns my-py-clj.config
(:require [libpython-clj2.python :as py]))
;; When you use conda, it should look like this.
(py/initialize! :python-executable "/opt/anaconda3/envs/my_env/bin/python3.7"
:library-path "/opt/anaconda3/envs/my_env/lib/libpython3.7m.dylib")
{...
;; This namespace going to run when the REPL is up.
:repl-options {:init-ns my-py-clj.config}
...}
user> (require '[libpython-clj2.require :refer [require-python]])
...logging info....
nil
user> (require-python '[numpy :as np])
nil
user> (def test-ary (np/array [[1 2][3 4]]))
#'user/test-ary
user> test-ary
[[1 2]
[3 4]]
We have a document on all the features but beginning usage is pretty simple. Import your modules, use the things from Clojure. We have put effort into making sure things like sequences and ranges transfer between the two languages.
One very complimentary aspect of Python with respect to Clojure is its integration with cutting edge native libraries. Our support isn't perfect so some understanding of the mechanism is important to diagnose errors and issues.
Current, we launch the python3 executable and print out various different bits of
configuration as json. We parse the json and use the output to attempt to find
the libpython3.Xm.so
shared library so for example if we are loading python
3.6 we look for libpython3.6m.so
on Linux or libpython3.6m.dylib
on the Mac.
If we are unable to find a dynamic library such as libpythonx.y.so
or libpythonx.z.dylib
,
it may be because Python is statically linked and the library is not present at all.
This is dependent on the operating system and installation, and it is not always possible to detect it.
In this case, we will receive an error message saying "Failed to find a valid python library!".
To fix this, you may need to install additional OS packages or manually set the precise library location during py/initialize!
.
This pathway has allowed us support Conda albeit with some work. For examples using Conda, check out the facial rec repository a)bove or look into how we build our test docker containers.
The scicloj community is maintaining a devcontainer
template on which libpython-clj
is know to work
out of the box.
This can be used as a starting point for projects using libpython-clj
or as reference for debuging issues.
We like to talk about libpython-clj on Zulip as the conversations are persistent and searchable.
New to Clojure or the JVM? Try remixing the nextjournal entry and playing around there. For more resources on learning and getting more comfortable with Clojure, we have an introductory document.
To install jar to local .m2 :
$ clj -X:depstar
After building and process pom.xml
file you can run:
$ clj -X:install
$ clj -X:deploy
This command will sign jar before deploy, using your gpg key.
Copyright 2019 Chris Nuernberger
This program and the accompanying materials are made available under the terms of the Eclipse Public License 2.0 which is available at http://www.eclipse.org/legal/epl-2.0.