Friday, October 2, 2015

Richards Benchmark


baseline

javascript backend: 0.012 seconds

with manual pre-jitting

javascript backend + JIT Natives: 0.0106 seconds

The Chrome V8 JIT is already optimized for the Richards benchmark, so its harder to make it run any faster. Using a V8 native JIT call to force a method to be optimized before the benchmark timer, it is possible to get a tiny 1/50th of a second more speed. It would be interesting if V8 exposed other native calls that could improve performance. This is enabled with the command line option `--v8-natives`, and the syntax `v8->(F(args))` where `F(args)` is a function or method called with dummy arguments that will warm up the function so V8 can infer its static types.

richards.py

rusthon_0.9.4_all.deb

Wednesday, September 23, 2015

Benchmarks updated


add.py

fannkuch.py

fannkuch old version vs fast setslice

old profile
  • __jsdict_pop replaced by direct call to __array_insert
  • __array_getslice replaced by __array_getslice_lowerstep: 200ms faster
  • __array_setslice 90ms faster

float.py

nbody.py

operator_overloading_functor.py

pystone.py

rusthon_0.9.3_all.deb

Monday, September 21, 2015

Operator Overloading Benchmark


JavaScript has no support for operator overloading, and this is generally a good thing; because in languages that do support it, programmers often abuse it and create code that is harder to read. The only real exception is with math libraries, where having operator overloading for vectors, matrices, and complex number classes can be very helpful, and make code more readable.

Emulation of operator overloading in JavaScript has a very high performance overhead, so much so that it is simply unusable if used traditionally with no extra syntax. Both Rusthon and RapydScript, after careful consideration, have adopted many of the same optimization patterns: and first is no direct support for operator overloading.

Brython, on the other hand, has attempted to fully support all of the dynamic features of Python, including operator overloading. As you will see in this benchmark, the price is very fucking high, Brython is 50 times slower than regular CPython3, and 295 times slower than Rusthon.

operator_overloading.py

The Brython FAQ states that Brython is "somewhere between 3 to 5 times slower", nice try guys, that should be updated to say something like: "Brython is a toy, and can not actually be used for anything".

with oo:

def benchmark(n):
 a = [ Vector(i*0.09,i*0.05, i*0.01) for i in range(n)]
 b = [ Vector(i*0.08,i*0.04, i*0.02) for i in range(n)]
 c = []
 d = []
 for j in range(n):
  with oo:
   u = a[j]
   v = b[j]
   c.append( u+v )
   d.append( u*v )
 return [c,d]

Above is a snippet of code from the benchmark, the special syntax with oo: enables operator overloading for just those statements under it in Rusthon. If you hate this syntax, you can also use with operator_overloading:, or fore go operator overloading and directly use the method names like this:

def benchmark(n):
 a = [ Vector(i*0.09,i*0.05, i*0.01) for i in range(n)]
 b = [ Vector(i*0.08,i*0.04, i*0.02) for i in range(n)]
 c = []
 d = []
 for j in range(n):
  u = a[j]
  v = b[j]
  c.append( u.__add__(v) )
  d.append( u.__mul__(v) )
 return [c,d]

Functor Benchmark


operator_overloading_functor.py

Above is the result of my first implementation of __call__ in the Rusthon JavaScript backend. Why is it so much slower than regular CPython? The Chrome V8 JIT is very sensitive to dynamic behavior, and it will not be able to optimize a function if it is too dynamic. What is happening in this implementation is the method __call__ becomes a nested function in the class constructor, which dynamically rebinds its calling context to this, copies all attributes and methods from this to __call__, and then returns __call__. See my commit here that enabled this syntax.

Above is the result of my second implementation. The difference is huge, what did i change? Instead of running this.__init__(args) in the constructor, the __init__ method (and all other methods) gets attached to the nested function __call__ and not rebound, then when calling __call__.__init__(args) the calling context of this is already __call__, and so we can skip coping values from this. This allows the V8 JIT to know that __call__ is not so dynamic, and to optimize it however it can. The only performance question now is, what is the overhead of returning a function object, rather than a normal object from the class constructor? Lets compare to another version of the benchmark where __call__ gets replaced by a normal method named call

operator_overloading_nonfunctor.py

Above is the result of the same benchmark but without using the special method __call__, here it is replaced by a normal method call. The class constructor returns a regular object, not a function-object like above. This is less dynamic, and the V8 JIT is able to optimize it even more. In conclusion, there is indeed a price to pay for having callable objects, the cost being about six times slower.

What about Brython?

As you have seen above, the implementation details of a transpiler can have a huge impact on performance, if not carefully designed, tuned and tested, performance can quickly become many times slower then CPython. Let's take a look at a project that made all the wrong decisions, Brython. By wrong decisions, i mean trying to emulate all the dynamic features of CPython in JavaScript.

Above are the absolutely terrible results of Brython running the same benchmark. In this test, Brython is 80 times slower than CPython, and 88 times slower than Rusthon.

Sunday, September 20, 2015

Benchmarks2


What is killing RapydScript in the Fannkuch benchmark? I am not sure, looking over the output from RapydScript, it looks almost the same as Rusthon, except for the way it performs array slicing, could that be the cause of the slow down? See the transpiled code here: RapydScript output and Rusthon output


fannkuch.py

add.py

float.py

What about Brython?

Brython is another python to javascript transpiler that claims to be fast, see their FAQ here, there they claim it is "somewhere between 3 to 5 times slower"... this is complete bullshit, nice try guys. I wanted to include Brython in these benchmarks, but Brython is so fucking slow, it is not even practical to run benchmarks on it. I ran the fannkuch benchmark using Brython, and waited, and waited, and after a really long time, it finally finished, with an average time of 170seconds. Recall that Python3 completes fannkuch in 0.3seconds. Brython is about 560 times slower than regular CPython3, and 850 times slower than Rusthon.

update: I ran the Fannkuch benchmark again in Brython, this time unchecking the "debug" box, but the results were not much better, Brython still takes 150seconds. That makes it 500 times slower than CPython3.

First Benchmarks



add.py

fannkuch.py

float.py

Tuesday, September 15, 2015

Rusthon 1.0 Release Comming Soon


After almost two years of development, Rusthon is nearing the stable 1.0 release. The project originally began with my fork of PythonScript by Amirouche, who had written a two-pass transpiler that was easy to understand and modify. PythonScript was then renamed to PythonJS, and optimized for speed. Early benchmarks tests showed that many dynamic pythonic features would have to be thrown away, because of the high performance overhead. The minimal python syntax that evolved over several months, ended up having many of the same behavior and rules that Alexander Tsepkov's RapydScript also had.

Rusthon is now fully regression tested, you can see the source and output of the tests that transpile and execute without errors below. The syntax has been almost fully unified for each of the backends, the syntax developed for the Go and C++ backends has made its way back into the JavaScript backend to provide the same kind of type safety and new features.

JavaScript Regression Test Results

Native Backend Regression Test Results

Sunday, September 6, 2015

js backend: typed maps


There was alot of hype on Dart not long ago, but already it has faded away into the trash bin of so many other failed languages, with over 33,000 commits and less than 400 github stars, nobody gives a fuck, nice try Google+.

The Dart developers made some really bad choices early on when implementing hash-maps, like keeping the order of items [fuckup #1], and not optimizing for translation to Javascript [fuckup #2]. And things get even worse, we can also give up on the Dart VM as part of Chrome in the future, see here.

A major drawback in JavaScript is the lack of typed hash maps, and object keys always being coerced into strings. Objects in JavaScript are unfortunately just associative arrays whose keys are always strings. This can lead to bugs that are hard to trace down. Dart tried to solve this problem with compile time type checking; but this fails in the real world where your code is interfacing with huge amounts of external JavaScript which can not be compile time checked, so you still have to deal with runtime errors.

Typed hashmaps in Rusthon are checked at runtime when you transpile your project without the --release command line option. This allows you to debug your code with runtime errors that make sense, and enforce static types even when working with external JavaScript libraries.

map[K]V{}

Above is the syntax for typed hashmaps in Rusthon, it is directly inspired by the hashmap syntax in Golang. Where K is the key type, and V is the value type. Note this the same syntax used for the Rust, C++ and Go backends. See this example: javascript_typed_dict.md

FullStack NoSQL on SQL


There are several full-stack frameworks for python like: Django, Web2Py, and CherryPy. This PDF from the Web2Py docs has a pretty good overview of these frameworks, see web2py_vs_others.pdf. What you will notice with these frameworks is they each have their own fucked up way of embedding code and logic into HTML templates. These huge frameworks try to hide the details of the object relational mapping and database, but in doing so force you to use their HTML template language. Andy Shora has some interesting insights on full-stack dev, check out http://andyshora.com/full-stack-developers.html

Writing a full-stack system in Rusthon is pretty simple, this demo is just 300 lines and includes: backend, database, and frontend. See source code here. The HTML is raw and can be easily integrated with modern solutions like Angular.js. By not having some huge framework in your way, you can directly code the server to send data in the way you want, with whatever ad-hoc protocol you choose. In this demo all data is moved around over a websocket, and both clients are kept in sync with the server and database in realtime.

The Python library Dataset is used to hide all the SQL, and provides a NoSQL like solution in combination with property getter/setters on the client side, you never have to worry about directly writing SQL code.

Monday, August 10, 2015

Elm Frontend


Today I implemented a front-end for Elm, a language that touts itself as "the best of functional programming in your browser". Integration is simple, you just include Elm code in its own fenced code block inside your markdown file, see this example hello_elm.md. This works the same way as the other JS transpiler frontends, Rusthon calls the other transpiler in a subprocess, and inserting its output into the final html. Checkout the other frontends in Rusthon for: CoffeeScript and Rapydscript.

Elm first impressions

My first try with Elm was a big disappointment. Things look bad starting with the install page, there it lists packages for "OSX", "Windows" and "Anywhere". First off I dont give a fuck if you have packages for OSX and Windows, support the NPM "Anywhere" package and make it work well. Next problem, the NPM package, at the bottom of the page it says, "Also, set the ELM_HOME environment variable to /usr/local/lib/node_modules/elm/share or the corresponding directory created on your machine." What the fuck is your problem Kevin MÃ¥rtensson? Do you really think i am going to play around with my environment variables just try out a transpiler.

Next problem, i run elm-make on my test and it asks me if i want to download some extra bullshit, it then drops this bullshit into the current directory i am running the command in! What the fuck, cache that in ~/.elm or something, do not just start junking up my file system.

first test code fails

import Text
myfunc a =
  Text.plainText a

I was pretty sure that was going to work, i took it pretty much right from chapter1 of the Elm guide here. Searching around i found that Text.plainText is deprecated, and this pisses me off. The problem with Elm and some other transpiled languages (like Dart) is they try to define a standard library of modules and functions, often this grows into some huge mess, which then gets refactored and parts deprecated. A language that transpiles to JavaScript does not need to reinvent all these things, there are plenty of JS libraries out there already. So why did Elm go down the wrong path? Probably because it has such shitty JS interop using ports, and so needs its own fat standard library because it is such a pain for the user to simply call a JavaScript function.

Wednesday, August 5, 2015

JavaScript backend: Debugger


This first reason why people give up when trying to learn JavaScript and HTML, they load up their test page in Google Chrome, and then see this fucking bullshit. It can be a blank white page, or your page half drawn with missing elements...

This second reason why people give up on JavaScript and HTML, they have to press Ctrl+Alt+J to open the god forsaken Google Chrome Developer Tools window. They see some error message in red and then have to click to the far right of it to jump to that point in the code.

Then the user sees the interface above and runs outside to puke. The line that caused the error is highlighted in yellow, for a split second, and then fades out so fast your eye can easily miss it. What the fuck?

Maybe then they give the FireFox debugger and try and see something even more hideous. Then they decide it is time to give up on JavaScript and HTML forever.


A Real JavaScript Debugger

This is the new debugger that is built into the JavaScript backend of Rusthon. When you transpile your project without using the --release command line option, it will insert runtime code to catch any uncaught errors, it parses the trace-back, and generates an HTML interface, with human readable error messages, and editable code that updates without recompile or reload.

note: if you do not include Ace.js in your HTML, then the debugger falls back to using a regular HTML TextArea, which looks like this.

Thursday, July 30, 2015

Javascript backend: Static Type Syntax


Every fucking time i see this error message in my chrome js console, Uncaught TypeError: Cannot read property `position` of undefined makes me want to walk into Google's head office and beat every mother fucker down. It is time for Lars Bak and Kasper Lund at Google to give up on Dart, and provide a real debugger for Chrome.

Programmers can get into heated debates about syntax and static typing. Sure, it is nice to have a compiler check the types of every variable at compile time, and then throw errors for any typing mistakes you have made, and this works well for native apps written in a statically typed language, but not in the web browser.

Dart and TypeScript compile to JavaScript, and try to offer the same safety that static types provide. However, this is not actually useful because JavaScript is itself not typed, and the type of a variable can change at anytime.

The reality is that in any real web application, you are going to be interfacing with many external JavaScript libraries, and most of your type errors are going to happen at runtime when using those libraries. So you really need a good runtime type checker that will provide useful errors so you can debug your code quickly. JavaScript without runtime type checking, is very hard to debug, because many types of errors are silent, and often this causes later code to break with confusing error messages about something being undefined, forcing you to manually trace the logic back to the original thing that was wrong and caused the whole chain of problems.

Runtime Type Checking Syntax

example source code

def setup_camera_untyped( c ):
 c.position.z = 60
 c.position.x = 5

The function above expects the argument c to be an instance of THREE.PerspectiveCamera. However, if you make a mistake and call the function and pass undefined or some other type, it will fail at runtime with this error message which is almost completely useless.

typed version

def setup_camera( c:THREE.PerspectiveCamera ):
 c.position.z = 60
 c.position.x = 5

Above is the same function with c typed as THREE.PerspectiveCamera, this instructs the transpiler to inject runtime type checking code, that when fails, throws and error message you can actually use to fix the problem, like what is the function and variable name.

For more info see the wiki page, here.

Monday, July 27, 2015

Javascript Backend: NodeJS Tornado Web


NodeJS allows you easily move code back and forth from the client to server, and use the same libraries client-side and server-side. Python3 offers nothing new: still GIL locked, no JIT, and no new syntax worth even mentioning. The only thing Python3 is good for is wasting CPU cycles, and increasing the rate of climate change.

Its nice to have the option to move your server code to NodeJS without having to manually rewrite it. Rusthon now includes a fake Tornado web module, that emulates the basic features of Tornado by wrapping the NodeJS modules: http, url, and ws. The wrapper source code is in nodejs_tornado.py. You load this with from nodejs.tornado import *.

example

example source code
#backend:javascript
from runtime import *
from nodejs import *
from nodejs.tornado import *

PORT = 8000

class MainHandler( tornado.web.RequestHandler ):

 def get(self, path=None):
  print('path', path)
  if path == 'favicon.ico' or path.endswith('.map'):
   self.write('')
  else:
   self.write( open('index.html').read() )


Handlers = [
 ('/', MainHandler),
]

def main():
 print('')
 app = new(tornado.web.Application( Handlers ))
 app.listen( PORT )
 tornado.ioloop.IOLoop.instance().start()

## start main ##
main()

Saturday, July 4, 2015

Javascript Backend: timeout syntax


Performance critical games and web applications often need to maintain consistent and high frame rates. In this situation it is better to drop some of the logic update rather than dropping render frames. This way the rendering frame rate can maintain some target like 60fps, while the logic updates at a variable rate.

Ian Webster has an article about a technique he calls Timed Array Processing, which is able to drop logic and maintain consistent a frame rate regardless of the number of objects in the scene. It works by taking the current time, and when looping over the scene objects, it only updates objects while under the time limit. This automatically spreads the logic computation load over multiple frames. This highly useful feature should actually be abstracted to the language syntax level, so it can be used in a clear and consistent way.

Rusthon has a smallish code base written in Python, maintained entirely by me, so it only takes me a few days to add a feature like Timed Processing. The new syntax to mark a block of code for timed processing is: with timeout(ms):, where ms is the time to finish in milliseconds. The Rusthon JavaScript transpiler will insert time checking code after each function-calling-expression and inside for/while-loops within that indented block. Runtime execution of the code block will then suspend if the time limit is reached. This allows the programmer to write scaleable code, and not scrafice readabity or maintainablity.

Rusthon Input

 q = []
 with timeout( 10 ):
  i = 0
  while True:
   q.append( CalcFib(i) )
   i += 1

JavaScript Output

 q = [];
 var __clk__ = (new Date()).getTime();
 while (true) {  /* timeout: 10 */
  i = 0;
  while (true)
  {
   if ( (new Date()).getTime() - __clk__ >= 10 )  { break; }
   q.append(CalcFib(i));
   i ++;
  }
  break; 
 }

The example above computes the Fibonaci series for up to 10 milliseconds, on my laptop it reaches the 28th place in that time, example source code. Timed processing is useful when used together with the sleep builtin and the select statement, see this example for more.

Sunday, June 21, 2015

JavaScript Backend: WebWorkers

Generation of async callbacks at transpilation time, from simple blocking logic, into callback-hell.

Async Transform

Using WebWorkers directly requires you define functions for postMessage and onmessage, split your JavaScript into two files, and load libraries in the worker with importScripts. Only low-level-chumps think dealing with these API's directly is a good idea. All these API details are easily abstracted away with syntax inspired from Golang. One of the best things about Golang is the message passing syntax: it makes code more readable, and it prevents a library-war by having this syntax builtin.

result1 = <- mychannel
print result1
result2 = <- mychannel
print result2

The code above gets the result of mychannel that computes something in a WebWorker. This requires waiting for the result, so if written directly by hand in JavaScript, you would need to write a mess of async callbacks. Rusthon automatically translates above into async code, and deals with routing messages to each channel.

__workerpool__.recv( mychannel, function (result1) {
 console.log(result1);
 __workerpool__.recv( mychannel, function (result2) {
  console.log(result2);
 });
});

source code

Channel Select Syntax

Select syntax inspired by Golang, works pretty much the same as Golangs select statement.


for i in range(n):
 print 'trying to select on workers'
 select:
  case res = <- worker1:
   show('case-worker1:' + res)
  case res = <- worker2:
   show('case-worker2:' + res)

example source code

Monday, June 15, 2015

Rusthon Overview


The Old Way

Above is the traditional way software gets developed. It is a slow and complex process, it creates extra work for the coder, because they must manage files, folders, and multiple build tools. This is a headache and it is not secure.

The New Way

Above the new way of software development using Rusthon. Its simple and more secure using only a single markdown file that combines all config files, and source code in multiple languages. Rusthon also includes a powerful Python to Javascript transpiler, so you can do almost everything from Python syntax if you want.

Sunday, May 10, 2015

WebGL CSS3D UI Prototype


I have updated this prototype to work with the latest THREE.js and only use one CSS3DRenderer layer, you can view the live demo on heroku rusthon.herokuapp.com. The shading over the CSS3D windows is done with the bloom fx in THREE.js, some of the stock fx are compatible with RGBA targets, and able to render overtop of the interactive HTML layer.

You can drag and drop Collada (dae) files from your desktop onto the view and they will be loaded and attached to the 3D window. See the source code, here. The prototype works in Google Chrome, Safari, new iphones and androids, but not very well in Firefox see this bug webgl alpha overlay css3d bug

Saturday, April 18, 2015

Unreal Engine4 - CPython Plugin


Rusthon has special syntax for directly using PyObject's from the CPython CAPI. The -> operator is used on PyObject's, and is transpiled into: PyObject_GetAttrString(ob,name), for more info see this wiki page. This allows you to directly manage calling CPython functions, cast and move data back and forth from C++ to the CPython interpreter. This makes using complex libraries simple and safer, you do not need to depend on complex wrapper generators, or depend on C++ reflection to dynamically bind CPython to C++.

Unreal has a C++ reflection system, and this can be used to generate bindings to a scripting language. Unreal users want scripting support, see this thread, and this one asking about CPython in Unreal.

Scripting is great when you have a command-prompt and can type code at runtime, or write quick scripts and simply run them. Scripting fails when you need to do something complex and use multiple threads, for this you need C++. Rusthon allows you to write in a Python-like language and create fast glue code that bridges CPython together with any other C++ library, no reflection or generated bindings required.

This example: unreal_cpython.md shows you how to connect CPython2.7 with Unreal. The example contains no handwritten C++, instead the code is transpiled to C++ at compile time. The embedded Python script is also inlined into the final binary.

Friday, April 17, 2015

C++ Backend Part3


Calling into C++ from another language normally requires an FFI and writing extra broiler plate code and having to worry about garbage collection and threading issues. Just take a look at how fucked up interfacing D is with C++. Don't miss the bottom of that doc on C++ Templates, from the doc: "D templates have little in common with C++ templates, and it is very unlikely that any sort of reasonable method could be found to express C++ templates in a link-compatible way with D. This means that the C++ STL, and C++ Boost, likely will never be accessible from D." There are many other system languages, like Rust, that want to kill off C++, but do not have a good way of interfacing with C++.

C++ is deeply entrenched, its not going anywhere. Rusthon transpiles to human readable C++, and you can configure the transpiler to use regular or smart pointers. Interfacing with an external C++ library is simple and requires no FFI, you can use C++ namespaces, template classes (via macros) and directly handle memory, passing the address of some variable or returning a pointer.

Caffe Deep Learning

Caffe is a large deep learning framework with many different types of neural networks. Getting it to compile and working on Fedora took me a couple of days, along the way I had to make lots of notes on the build and defines it required. The notes, build script, and code is all put into a single markdown file, see hello_caffe.md.

Testing Caffe with Rusthon has given me the chance to iron out the last details for working with external C++. Some functions in Caffe need to be passed the address of a variable, not the value or a pointer to it, the new builtin addr(X) takes the address of X and passes it to the function. If you need to initialize an array of template classes you can now use this trick.

with MyType as "some_template<float>":
    x = new( MyType() )
    y = []MyType()
    y.append( x )

Wednesday, April 8, 2015

C++ Backend Part2


New syntax and built-in functions have been added to Rusthon's C++ transpiler to make interfacing with external C++ libraries as simple as possible. Namespace syntax is supported using ::, just like in normal C++, more info here. Macro syntax is support with the macro function (to inline simple macros) and new with statement with T as "foo(%s)": that makes T a callable macro, this is useful for calling template functions from external libraries, more info see here.

Functions can also now use C++ type syntax to define the function return type. This allows you to override the default behavior of returning shared pointers (to objects, arrays, and maps).

  • def f()->T* returns a pointer to T
  • def f()->T& returns a reference to T

Calling methods on strings has now been fixed and always works, before there was the problem of pointers|references (which the transpiler had to keep track of). This has been fixed with a c++ template class function, the transpiler when it is not sure what the type of something is will use this template function to properly wrap the value. This also solves the general problem of working with objects from external c++ libs, where you are not sure if something is a pointer, or a pointer to something.

Tuesday, April 7, 2015

Unreal Engine4 Test Plugin


Above, screen shot of Unreal Editor being opened by the installer script in Fedora21, and starting the compile process. update: april 14: got the plugin to compile and load, fixed a bug in the Unreal source code (3dluvr's version), get my fork of Unreal4.7 here that is compatible with Rusthon. The command below will compile the plugin and launch Unreal Editor.

./rusthon.py ./examples/unreal_plugin.md --run=install-plugin.py --output-dir=~/Documents/Unreal\ Projects/MyProject/

source code

https://github.com/rusthon/Rusthon/blob/master/examples/unreal_plugin.md

Friday, April 3, 2015

Custom C++ Types


custom types wiki page

Which is the best smart pointer library for C++, that works everywhere and is fast? The standard smart pointers in C++11 can raise exceptions for some operations and this adds performance overhead. High performance libraries often use custom smart pointers because of this. Each custom smart pointer library has a similar API and behavior, and sometimes it is just the method names that differ. These implementation details get in the way of clear and readable code.

Rusthon solves this problem by allowing you to configure which smart pointers to use, for each block of code. This allows you to write code at a higher level, and control the low level details of translation using a single line with syntax('mytypes.json'). Code in that block will use the custom types and methods you define for: vectors, maps, shared and weak references. Outside of the block it defaults back to standard C++11 types.

Unreal Engine4

This example shows you how to configure the translator to use Unreal Engine smart pointers, string type, and template arrays. The translator configuration is at the top of the markdown, and redefines template and method names. Here is the output of main:

int main() {
 TArray<TSharedRef<Child>> children = {};
 auto p = TSharedRef<Parent>(
  (new Parent())->__init__(1000, children)
 );
 std::cout << TEXT("parent:");
 std::cout << p;std::cout << std::endl;
 auto c1 = make_child(p, 1);
 auto c2 = make_child(p, 20);
 auto c3 = make_child(p, 300);
 std::cout << TEXT("children:") << std::endl;
 std::cout << c1 << std::endl;
 std::cout << c2 << std::endl;
 std::cout << c3 << std::endl;
 std::cout << TEXT("calling foo") << std::endl;
 std::cout << c1->foo() << std::endl;
 std::cout << TEXT("calling bar") << std::endl;
 c1->bar();
 p.Reset();
 std::cout << c1->foo() << std::endl;
 c1->bar();
 return 0;
}

Saturday, March 14, 2015

CPython inside Rusthon


Normally when mixing C++ with CPython you compile your C++ code and load it as an extension module. Sometimes, but less often CPython is embedded inside a C++ program. To interact with the Python interpreter from C++ requires writing verbose C-API calls all over the place. Rusthon simplifies using the C-API with special syntax ->, dedicated to PyObjects, that will generate the required PyObject_XXX calls for you.

simple tutorial: cpython_embed.md

The simple tutorial above shows you how to call methods on PyObject's and convert the return value to a C++ native type. Below shows how to use Go style channels to send PyObject's around to different threads created with spawn (c++11 std::thread).

multi-threaded GIL example: cpython_multithreaded.md

with gil:
    value = pyob->pymethod( cppob.cppmethod() as pyint ) as int
above example shows different calling styles for the PyObject and C++ object. note: as pyint becomes PyInt_FromLong from the CPython C-API, this is used to convert the return value of `cppmethod` to a PyObject.

Sunday, March 1, 2015

Literate Programming


Literate Programming, invented by Donald Knuth, makes programs human readable, secure and of higher quality.

Rusthon's multi-language markdown allows you use literate programming for mixing: python, c++, c, java, nim, rust, go, javascript and html. The transpiler/markdown-parser also acts as a compiler frontend to: gcc, g++, java, rustc, and go. This allows you to create multiple compiled executeables from a single markdown.

The source code of Rusthon itself is now mostly written in markdown, the main modules are:

Thursday, February 26, 2015

Nim and Java and C++


This single markdown file, nim_java.md mixes: Nim, Java, and C++. The output is a single executeable that includes the Nim code translated to C, the Nim C runtime, Rusthon code translated to C++, and links to the JVM. This allows you to call Nim and Java functions from Rusthon, and manage passing data between them. For how to load a JAR library, see here.

#backend:c++

import nim
import jvm
jvm.namespace('mymod')

@jvm
class PointSubclass( Point ):
 def __init__(self, x:int, y:int):
  self.set(x,y)

 def show(self):
  print self.getx()
  print self.gety()


def main():
 nim.main()
 p1 = jvm( PointSubclass(1,2) )
 p2 = jvm( PointSubclass(10,20) )
 p1.show()
 p2.show()
 p2.scale( nim_sum(100, 1000) )
 p2.show()

Mixing Nim, Java and C++ in a single binary would normally require some complex build, and running a wrapper generator like Nimrod-Java. But Nimrod-Java has not been updated in a year, and calling JNI through Nim is going to complicate threading.

Nim and Rusthon


Nim is a new language with some very powerful features. Nim's weakness is interfacing with C++ and external threads, Hankiewicz has a good article on some of these issues here. Nim and Rusthon integration helps solve this problem, allowing you to call Nim functions from C++. This works by including your Nim program in the same markdown file along with a Rusthon program. The embedded Nim program is parsed and wrappers are generated for Nim functions that export to C. During the build Rusthon will use Nim to translate the program to C, and GCC is called to compile a staticlib without a main. The static C library is then linked to the final C++ executeable.


markdown source code

nim

proc my_nim_function(a:cint,b:cint,s:cstring): cint {.cdecl, exportc.} =
 echo("calling my_nim_function")
 echo(s)
 echo("a:", a)
 echo("b:", b)
 result = a+b

rusthon

You need to import nim and in your main function call nim.main() before calling any Nim functions.
#backend:c++
import nim

def main():
 nim.main()
 print 'calling nim function'
 s = 'mymessage to nim'
 msg = my_nim_function( 10, 20, cstr(s) )
 print msg
 print 'ok'

program output

running: /tmp/rusthon-c++-bin
calling nim function
calling my_nim_function
mymessage to nim
a:10
b:20
30
ok

Saturday, February 21, 2015

Subclass from Java


You can subclass from a Java class in Rusthon using the C++11 backend. JNI wrappers are generated using Giws and automatically compiled with the final exe by putting the Giws XML inside your markdown project.

Giws only supports regular and class methods, and basic types. Constructors and properties of Java classes are not exposed, and require extra wrapping by hand. Hopefully this will get fixed soon, github issues: constructors, properties

source code: https://github.com/rusthon/Rusthon/blob/master/examples/giws_subclass.md

java

package mymod;

public class A{
 public A(){}
 public void foo(){
  System.out.println("A.foo from java.");
 }
}

public class B extends A{
 public int y;
 public B(){ this.y=420; }
 public void bar(int i){
  System.out.println("B.bar from java.");
  this.y = i;
 }
}

rusthon

The example below shows how the property y (defined in the Java class) is exposed to Rusthon with inlined C++ that directly uses JNI calls. This is not actually the best way to do this, it would be better to write getter/setter functions in Java, and wrap those normally with Giws XML. This way of going direct with inlined JNI is only useful if you were loading a JAR and did not have access to the original Java source code to add the getter/setters.

import jvm
jvm.namespace('mymod')

@jvm
class C(B):
 def __init__(self, x:int):
  self.x = x

 def get_y(self) ->int:
  inline('''
  auto env = getCurrentEnv();
  auto fid = env->GetFieldID(this->instanceClass, "y", "I");
  return env->GetIntField(this->instance, fid);
  ''')

 def hey(self):
  print('hey from rusthon')

c++ output of C

using namespace mymod;
class C:  public B {
  public:
 std::string __class__;
 int  x;
 C* __init__(int x);
 int get_y();
 void hey();
 C(JavaVM* _jvm) : B(_jvm) {__class__ = std::string("C");}
 std::string getclassname() {return this->__class__;}
};
 int C::get_y() {
  auto env = getCurrentEnv();
  auto fid = env->GetFieldID(this->instanceClass, "y", "I");
  return env->GetIntField(this->instance, fid);
 }
 void C::hey() {
  std::cout << std::string("hey from rusthon") << std::endl;
 }

 C* C::__init__(int x) {
  this->x = x;
  return this;
 }

When creating an instance of C you still need to wrap it with the special jvm call so that the Java proxy object is also created.

def main():
 c = jvm( C(999) )
 print c.x  ## x can be used directly, prints 999
 print c.get_y()  ## prints 420
 c.foo()
 c.bar( 100 )     ## sets y to 100
 print c.get_y()  ## prints 100
 c.hey()

Thursday, February 19, 2015

Java Frontend with Giws


The Java Frontend has been updated to work with the Giws JNI wrapper generator, this lets you create Java objects and call their methods from C++, and also from Rusthon when using the C++ backend. Giws supports all of Java's primitive types like numbers, strings, and arrays; one problem is class constructors see this git hub issue.

The new syntax is documented in the wiki, here. Java source that you embed in the markdown is compiled with javac, xml source in the markdown that is marked with @giws will run giws and generate the c++ wrappers. In rusthon code using import jvm will link the final exe to libjvm.so and also create an instance of the JavaVM.

Jython Hello World

source code: giws_jython.md

See the source code in giws_jython.md to see the xml wrapper code the exposes Jython's PythonInterpreter, this simple example only wraps the exec method.

rusthon input

import jvm
jvm.load( 'jython.jar' )
jvm.namespace('org.python.util')

def main():
 interp = jvm( PythonInterpreter() )
 script = "print 'hello world'"
 interp.__exec__(cstr(script))

c++ output

using namespace org_python_util;
int main() {
 auto interp = std::make_shared(__javavm__);
 auto script = std::string("print 'hello world'");
 interp->exec(cstr(script));
 return 0;
}

Similar projects that allow Java to be used in CPython: Py4J, and Jpype.

Tuesday, February 17, 2015

C++11 Weak Pointers


C++11 introduces std::shared_ptr and std::weak_ptr to manage shared memory with reference counting and weak references. Weak references are used by children in a tree structure to reference their parent, and break the reference cycle. Methods of the child class need to take extra care with the parent pointer (the weak pointer), to promote it to a shared pointer, and only use it if the pointer is still alive.

Rusthon detects when you have a reference cycle between two classes and automatically uses weak references to break the cycle. The parent class must contain a list of child objects, then when the parent is set on the child it becomes a weak pointer. For more info, check out the wiki page.

rusthon input

The line below par = self.parent promotes the weak pointer to a reference counted shared pointer that could also be null if the parent has been deleted, so you need to check if it is None before it can be used. The last line c1.bar() will crash with a segmentation fault because it is used after the parent has been deleted, and the bar function is not checking if self.parent is not None.

class Parent:
 def __init__(self, children:[]Child ):
  self.children = children

class Child:
 def __init__(self, parent:Parent ):
  self.parent = parent

 def foo(self) ->int:
  par = self.parent
  if par is not None:
   return 1
  else:
   print('parent is gone..')

 def bar(self):
  print self.parent.children

def make_child(p:Parent) -> Child:
 c = Child(p)
 p.children.push_back(c)
 return c


def main():
 children = []Child()
 p = Parent( children )
 c1 = make_child(p)
 c2 = make_child(p)
 print c1.foo()
 c1.bar()
 del p
 print c1.foo()
 #uncomment to segfault##c1.bar()

c++11 output

class Parent {
  public:
 std::string __class__;
 std::shared_ptr<std::vector< std::shared_ptr<Child> >>  children;
 void __init__(std::shared_ptr<std::vector<std::shared_ptr<Child>>> children);
 Parent() {__class__ = std::string("Parent");}
 std::string getclassname() {return this->__class__;}
};
class Child {
  public:
 std::string __class__;
 std::weak_ptr<Parent>  parent;
 void __init__(std::shared_ptr<Parent> parent);
 int foo();
 void bar();
 Child() {__class__ = std::string("Child");}
 std::string getclassname() {return this->__class__;}
};

 void Parent::__init__(std::shared_ptr<std::vector<std::shared_ptr<Child>>> children) {
  this->children = children;
 }

 void Child::__init__(std::shared_ptr<Parent> parent) {
  this->parent = parent;
 }


 int Child::foo() {
  auto par = this->parent.lock();
  if (( par ) != nullptr) {
   return 1;
  } else {
   std::cout << std::string("parent is gone..") << std::endl;
  }
 }

 void Child::bar() {
  std::cout << this->parent.lock()->children << std::endl;
 }

std::shared_ptr<Child> make_child(std::shared_ptr<Parent> p) {
 Child  _ref_c = Child{};_ref_c.__init__(p);
 std::shared_ptr<Child> c = std::make_shared<Child>(_ref_c);
 p->children->push_back(c);
 return c;
}

int main() {

 std::vector<std::shared_ptr<Child>> _ref_children = {};
 std::shared_ptr<std::vector<std::shared_ptr<Child>>> children = std::make_shared<std::vector<std::shared_ptr<Child>>>(_ref_children);
 Parent  _ref_p = Parent{};_ref_p.__init__(children);
 std::shared_ptr<Parent> p = std::make_shared<Parent>(_ref_p);
 auto c1 = make_child(p);
 auto c2 = make_child(p);
 std::cout << c1->foo() << std::endl;
 c1->bar();
 p.reset();
 std::cout << c1->foo() << std::endl;
 return 0;
}

Thursday, February 12, 2015

Java Frontend


https://github.com/rusthon/java2python

Mixing languages, generated bindings, it makes development hard. Calling a Java function from C++ or Python through an FFI, converting types and objects, things quickly break apart. An alternative way to integrate with Java is with a frontend that converts it into another language, and completely bypass the JVM and Java compiler.

Java was supposed to run everywhere, but there is no official support for it on iOS, and you need to use a transpiler of some kind like Tom's j2objc to convert it to Objective-C. Rusthon's Java frontend can be used in a similar way, by converting Java into Rusthon syntax, and then translating that with the C++ backend, you have portable C++ that could be then compiled on iOS.

source code hello_java.md


Java Input

hand written java code example program with two class methods, foo takes a string.
public class HelloWorld {
    public static void test() {
        System.out.println("Hello, World test");
    }

    public static void foo(String msg) {
        System.out.println(msg);
    }

}

Rusthon Output

output from java2python in rusthon syntax.

class HelloWorld(object):
    @classmethod
    def test(cls:object):
        print("Hello, World test")

    @classmethod
    def foo(cls:object, msg:string):
        print(msg)

C++ Output

final translation to c++11
class HelloWorld {
  public:
 std::string __class__;
 static void test();
 static void foo(std::string msg);
 HelloWorld() {__class__ = std::string("HelloWorld");}
 virtual std::string getclassname() {return this->__class__;}
};
 void HelloWorld::test() {
  std::cout << std::string("Hello, World test") << std::endl;
 }

 void HelloWorld::foo(std::string msg) {
  std::cout << msg << std::endl;
 }

Wednesday, February 11, 2015

5X more Pystones than CPython


Two versions of Pystone are used, untyped pystone.py and typed pystone-typed.py. Taking a look at those files will show you most of what you need to know for porting code to Rusthon. This benchmark was done on 64bit linuxmint, 2Ghz. Command to run this bench:

cd Rusthon/regtests
./run.py ./bench/pystone.py
./run.py ./bench/pystone-typed.py


above pystones per-second.

Tuesday, February 10, 2015

Rusthon C++11 Backend


These are the first benchmarks testing the new C++ backend of Rusthon. As we would expect, its faster than CPython and the JavaScript backend. And, with one extra line of code, that enables raw pointers and noexcept, Rusthon can out perform PyPy.

This benchmark is short and straight to the point, less than 50 lines, it is testing object allocation/deallocation, calling methods, and adding integers. The outer loop iterates, 1000 times, and the inner loop iterates 100 times.

source code: add.py

The only surprising result from above is the big difference in performance between NodeJS and NodeWebkit. Both are using Rusthon's JavaScript backend with the same options, the JavaScript is the same. NodeJS is more than twice as slow as NodeWebkit, most likely because my version of NodeJS is older, and must not have been compiled with as many optimizations as NodeWebkit. This may show that while V8 is super fast, it's speed many vary alot depending on version and compilation options.

Optimized Version

Here is the same benchmark as above, but with one extra line at the top with (pointers, noexcept):, this makes rusthon use real pointers, and functions do not throw exceptions. Most of the performance gain comes from changing from reference counted std::shared_ptr to a real pointers. Using direct pointers requires more care in how your code is written, and how memory will be automatically and manually be free'ed, in the future we will provide an option for something faster than std::shared_ptr but safer than real pointers, using arena management or multi-core cleanup.

source code: add-opt.py

from time import clock

with (pointers, noexcept):
 class A:
  def __init__(self, x:int,y:int,z:int):
   self.x = x; self.y = y; self.z = z
  def add(self, other:A) ->A:
   a = A(self.x+other.x, self.y+other.y, self.z+other.z)
   return a
  def iadd(self, other:A):
   self.x += other.x
   self.y += other.y
   self.z += other.z

 class M:

  def f2(self, step:int, a:A, b:A, c:A, x:int,y:int,z:int) ->A:
   s = A(0,0,0)
   for j in range(step):
    u = A(x,y,z)
    w = a.add(u).add(b).add(c)
    s.iadd(w)
   return s

  def f1(self, x:int, y:int, a:A, b:A, c:A ) -> A:
   w = A(0,0,0)
   flip = False
   for i in range(x):
    if flip:
     flip = False
     w.iadd(self.f2(y, a,b,c, 1,2,3))
    else:
     flip = True
     w.iadd(self.f2(y, a,b,c, 4,5,6))
   return w

 def main():
  m = M()
  xsteps = 1000
  ysteps = 100
  start = clock()
  n = -1000000
  a = A(n,n+1,n)
  b = A(n,n+2,n)
  c = A(n,n+3,n)
  w = m.f1(xsteps, ysteps, a,b,c)
  print(clock()-start)

Sometimes people complain my benchmarks are unfair because Rusthon is static while Python and PyPy are dynamic. Or the fact that CPython has an entire ecosystem of 3rd party libraries. First of all, those 3rd party libraries are most often wrappers around a C or C++ library, and it is always a worry if they are being maintained. It is better to use C/C++ libraries directly, because Rusthon is translated to C++, there is no need for FFI, it can simply link-in and directly call into another C++ library. Rusthon is not dynamic, and static is a good thing. Do you really want to ship a Python interpreter with your application? The PyPy interpreter is over 100MB. Rusthon gives you a tiny statically linked exe, this benchmark exe is just 28KB. This lets you compile and run your application anywhere where C++ can be compiled, like mobile platforms where CPython and PyPy can not be used.

c++ output (pointers and noexcept)

class A {
  public:
 std::string __class__;
 int  y;
 int  x;
 int  z;
 A* __init__(int x, int y, int z) noexcept;
 A* add(A* other) noexcept;
 void iadd(A* other) noexcept;
 A() {__class__ = std::string("A");}
 virtual std::string getclassname() {return this->__class__;}
};
A* A::__init__(int x, int y, int z) noexcept {
 this->x = x;
 this->y = y;
 this->z = z;
 return this;
}
A* A::add(A* other) noexcept {
 A  _ref_a = A{};
 _ref_a.__init__((this->x + other->x), (this->y + other->y), (this->z + other->z));
 A* a = &_ref_a;
 return a;
}

void A::iadd(A* other) noexcept {
 this->x += other->x;
 this->y += other->y;
 this->z += other->z;
}

class M {
  public:
 std::string __class__;
 A* f2(int step, A* a, A* b, A* c, int x, int y, int z) noexcept;
 A* f1(int x, int y, A* a, A* b, A* c) noexcept;
 M() {__class__ = std::string("M");}
 virtual std::string getclassname() {return this->__class__;}
};
A* M::f2(int step, A* a, A* b, A* c, int x, int y, int z) noexcept {
  A  _ref_s = A{};_ref_s.__init__(0, 0, 0);
  A* s = &_ref_s;
  auto j = 0;
  auto j__end__ = step;
  while (( j ) < j__end__) {
   A  _ref_u = A{};
   _ref_u.__init__(x, y, z);
   A* u = &_ref_u;
   auto w = a->add(u)->add(b)->add(c);
   s->iadd(w);
   j ++;
  }
  return s;
}
A* M::f1(int x, int y, A* a, A* b, A* c) noexcept {
  A  _ref_w = A{};_ref_w.__init__(0, 0, 0);
  A* w = &_ref_w;
  auto flip = false;
  auto i = 0;
  auto i__end__ = x;
  while (( i ) < i__end__) {
   if (flip==true) {
    flip = false;
    w->iadd(this->f2(y, a, b, c, 1, 2, 3));
   } else {
    flip = true;
    w->iadd(this->f2(y, a, b, c, 4, 5, 6));
   }
   i ++;
  }
  return w;
}

int main() noexcept {

 M  _ref_m = M{};M* m = &_ref_m;
 auto xsteps = 1000;
 auto ysteps = 100;
 auto start = __clock__();
 auto n = -1000000;
 A  _ref_a = A{};_ref_a.__init__(n, (n + 1), n);
 A* a = &_ref_a;
 A  _ref_b = A{};_ref_b.__init__(n, (n + 2), n);
 A* b = &_ref_b;
 A  _ref_c = A{};_ref_c.__init__(n, (n + 3), n);
 A* c = &_ref_c;
 auto w = m->f1(xsteps, ysteps, a, b, c);
 std::cout << (__clock__() - start) << std::endl;
 return 0;
}

Sunday, February 1, 2015

Downtown Sacramento Meetup


I have started a new meetup, the "High Performance Computing Club" for hackers in the Sacramento area to meetup everyday at lunch time, and talk about writing fast programs that run on the CPU and specialized FPGA hardware. Rusthon has a new SystemVerilog backend that I will be demoing. The meetup is free, join here.

Sunday, January 25, 2015

The Rusthon Team


There has been a sudden jump up in activity in development of Rusthon, the C++ backend is starting to work well, and we have a new literate markdown format to assemble and compile multiple languages. Three new developers have joined the project on Github, and are pushing commits at a very fast pace, today we have already reached 19 commits. The new developers are: Kud1ing, Shooza, and James Rubino. The best commit of the day goes to Shooza, who fixed a bug that had me totally stumped, here is his fix for calling C functions from the C++ backend, commit.

Sunday, January 18, 2015

Python Burning

Want to use Python to make high performance mobile games? Forget about it, Python is not even officially supported on iOS. PyPy is not the solution either, its not working on iOS or Android. Python is embarrassingly slow and inefficient compared to compiled languages like C++. Power efficiency is a big concern on mobile, and it should be on the server side as well.

Python has become one of the most popular languages for server side programming. Running applications on perhaps hundreds of thousands of servers world wide. This is having a huge environmental impact, and nobody in the Python community is talking about it. The fact is that if those applications were rewritten in C++, Go or even JavaScript, it would require dramatically less power and less servers.

Save our planet by ditching the CPython interpreter now!

"Facebook, which is getting some negative PR on environmental issues, recently announced several green initiatives. Some of them are intuitive: getting employees to cut down on water usage, a shuttle bus, composting. Most intriguing, though, is one initiative to ensure that the very code written by Facebook's programmers is itself as green as possible, something achieved by converting Facebook's code from one language called PHP to another called C++." http://www.fastcompany.com/1702079/can-computer-code-be-greener-facebook-thinks-so