F# and Iron Python

Iron Python - .NET implementation of Python, tightly integrated with .NET framework, has a wide range of applications. It can be used as an embedded scripting language, as a full-fledged language for creating complex apps and as a bridge for reusing existing Python code in managed programs. The latter benefit is very important because Python offers huge amount of various libraries distributed as a source code and significant part of it can be used with Iron Python (maybe with few minor changes).

Today’s post will be devoted to various ways of integration between Iron Python and F#. I’ll try to skip the details of DLR configuration, because this is vast topic that worth separate post (maybe even a few posts). Instead I’ll focus on questions of integration. All samples were created and tested with VS 2010 and Iron Python 6.1 RC for .NET 4.0. First part for all the samples is the same: create new console F# project and add references to IronPython, Microsoft.Dynamic and Microsoft.Scripting.

Running standalone script. This task is trivial and needs no comments.

open IronPython.Hosting

let python = Python.CreateEngine() // create Python script engine with default settings
let script = @"
def fact(x) :
    return x > 1 and x * fact( x - 1) or 1
print fact(5)
"
python.Execute(script) |> ignore;

120

Brilliant, first success. However this script is mostly useless, because it cannot accept parameters and return values. Of course it is possible to use some external stuff like files for passing\returning values or insert parameters by declaring placeholders in script and then applying something like printf or string.Format but IMO both approaches are non-elegant,heavy-weighted and error-prone. We will make everything in a more accurate way.

Scopes and delegates. There is a term scope in Python, it defines visibility of name. Roughly speaking scope is a dictionary that maps name to instance. Scopes are organized in hierarchy and if request for name was not satisfied in child scope - the call will be delegated to parent. In previous sample we didn't create the scope, so engine makes it implicitly. Next sample will create scope explititly.

open System
open IronPython.Hosting

let python = Python.CreateEngine() // create Python script engine with default settings
let script = @"
class A(object) :
    def __init__(self, x, y) : 
        self.x = x
        self.y = y
    def write(self) :
        print ""x=%s, y=%s"" %(self.x, self.y)

def create(x, y) : 
    return A(x, y)

def write(s) : 
    s.write()
"
let scope = python.CreateScope()

python.Execute(script, scope) |> ignore; 
for name in scope.GetVariableNames() do
    printfn "%s" name
let create = scope.GetVariable<Func<_, _, obj>>("create")
let write = scope.GetVariable<Action<obj>>("write")

let o = create.Invoke(1, "!")
do write.Invoke(o)

__builtins__
__file__
__name__
__doc__
A
create
write
x=1, y=!

Script execution doesn't produce any result, instead it will add names A, create and write to scope. After execution we can extract corresponding values from scope, DLR will automatically convert them to required type. In out case we obtain two delegates, one acts as a factory, another invokes some hardcoded method on given instance.

Direct access. This approach is also not flawless, the necessity to create a function for every method in type is extermly annoying: if I already has instance returned by factory, why cannot I invoke method directly. And in fact I can, DLR allows doing it through using Operations set provided by concrete ScriptEngine. We can combine this fact with F# dynamic lookup operator and make something ve-e-ery interesting.

open System
open IronPython.Hosting
open Microsoft.FSharp.Reflection

let python = Python.CreateEngine() // create Python script engine with default settings
let (?) (o : obj) m : 'Result = 
    if FSharpType.IsFunction typeof<'Result>    
    then 
        // if it was function call then we need to take requested callable member from instance
        let func = python.Operations.GetMember(o, m)
        let domain, _ = FSharpType.GetFunctionElements(typeof<'Result>)
        let getArgs = 
            if domain = typeof<unit> then fun _ -> [||]
            elif FSharpType.IsTuple domain then fun a -> FSharpValue.GetTupleFields(a)
            else fun a -> [|a|]

        downcast FSharpValue.MakeFunction(typeof<'Result>, fun args -> 
            python.Operations.Invoke(func, getArgs(args))
            )
    else 
        downcast python.Operations.GetMember(o, m)

let (?<-) (o : obj) m v = 
    python.Operations.SetMember(o, m, v)

let script = @"
class A(object) :
    def __init__(self, x, y) : 
        self.x = x
        self.y = y
    def write(self, prefix) :
        print ""%s: x=%s, y=%s"" %(prefix, self.x, self.y)

A(100, 100)
"
let a = python.Execute(script);
a?y <- 500
a?write("Test") 
printfn "%d" a?x

Test: x=100, y=500
100

As you may notice we've made few changes in script to simplify it: all functions are removed and now it returns instance of A as a result. Implementation of (?<-) is pretty simple but (?) may need some comments:

• 'Result type defines what member should be invoked.
• Functions and properties are processed separately: properties are accessed via calling GetMember operation on given instance. Functions are processed in two steps: first - get callable member, second - perform call with it.
• Arguments processing forks in 3 cases. Unit arguments are represented as null, instead of null we need to pass empty array. Single argument follows as it is and tupled arguments should be unpacked from tuple and put into array.

Note: this version also needs improvements, now dynamic lookup operator is tightly bound to particular instance of ScriptEngine, and this is bad. This can be solved by using technique similar to C# 4.0 compiler: create DLR call sites, use binder etc. This fix shall be the subject for one of my next posts.

We can take values from scope right? Right! But who can forbid us to put some predefined items in scope and refer to them in script? Nobody! And this is perfectly valid approach to invoke existing managed code from Iron Python script. Its worth noting that Iron Python have access to all the might of BCL, all you need to do is import clr module.

open System
open IronPython.Hosting
open Microsoft.FSharp.Reflection

let python = Python.CreateEngine() // create Python script engine with default settings
let (?) (o : obj) m : 'Result = 
    if FSharpType.IsFunction typeof<'Result>    
    then 
        // if it was function call then we need to take requested callable member from instance
        let func = python.Operations.GetMember(o, m)
        let domain, _ = FSharpType.GetFunctionElements(typeof<'Result>)
        let getArgs = 
            if domain = typeof<unit> then fun _ -> [||]
            elif FSharpType.IsTuple domain then fun a -> FSharpValue.GetTupleFields(a)
            else fun a -> [|a|]

        downcast FSharpValue.MakeFunction(typeof<'Result>, fun args -> 
            python.Operations.Invoke(func, getArgs(args))
            )
    else 
        downcast python.Operations.GetMember(o, m)

let (?<-) (o : obj) m v = 
    python.Operations.SetMember(o, m, v)

type public SomeVeryUsefulType() = 
    member x.write(path, content) = System.IO.File.WriteAllText(path, content) 
type AlsoUsefulStaticType = 
    static member read(path) = System.IO.File.ReadAllText(path)

let script = @"
import clr

from System.IO import Path

class A(object) :
    def write(self) :
        tempPath = Path.GetTempFileName() // direct access to BCL
        writer.write(tempPath, ""Hi from Iron Python"")
        return ReaderType.read(tempPath)
A()
"
let scope = python.CreateScope()
scope.SetVariable("writer", SomeVeryUsefulType())
scope.SetVariable("ReaderType", IronPython.Runtime.Types.DynamicHelpers.GetPythonTypeFromType(typeof<AlsoUsefulStaticType>))
let a = python.Execute(script, scope);

printfn "%s" (a?write())

Hi from Iron Python

The sample shows that we can equally successfully give IronPython instance and types. The entire picture will be incomplete without mentioning possibility to create whole Python modules in managed code. It is irreplaceable feature because many native Python libraries is implemented in C and cannot be used in Iron Python directly.

open System
open IronPython.Hosting
open IronPython.Runtime

open System.Net

type public FSWeb = 
    static member download(ctx : CodeContext, uri : string) = 
        let req = HttpWebRequest.Create(uri)
        let resp = req.GetResponse()
        use stream = resp.GetResponseStream()
        use reader = new System.IO.StreamReader(stream)
        reader.ReadToEnd()

// annotation of managed Python module in current assembly
[<assembly : PythonModule("fsweb", typeof<FSWeb>)>]
do()

let script = @"
import fsweb
fsweb.download(""http://google.com"")
"
let python = Python.CreateEngine()

python.Runtime.LoadAssembly(typeof<FSWeb>.Assembly)
let content : string = downcast python.Execute(script)
printfn "size: %d" content.Length

size: 7110