Ruminations

Code better not harder

Freestyle or Racing?

When I started the hobby, I am aware that there are two ways of flying FPV which is racing and freestyle. Drone racing has been going on for a while with professional competition all over the world such as DRL and MultiGP. But, freestyle doesn’t seem to have a specific venue, because as the name suggests, it’s free flying with style.

I would like to explore a bit on how does someone who started with FPV approach these two styles of flying. In racing, there are specific tracks, obstacles, goals, and time tracker. It’s structured, and the goal is obvious: to be the fastest to finish the track. But does that mean being the fastest is all it takes? Depends on what it means to be fast. I have been using the DRL (Drone Racing League) simulator, and have participated in the 2020 Tryout tracks, which is a course in Drone Park, and as someone who is just starting, it’s extremely hard to get to the top. Even on a simulator, not the real race.

How difficult is it to race at Drone Park?

As you can see from the video, the track is the most challenging I’ve ever seen compared to DRL tracks in previous years, as it involves multi-directional full-throttle maneuver, power-loops, corkscrew, and more. A slight interruption to flow will just ruin everything and it will definitely drop your lap time. I could finish the track but with about 2-3 of the time it takes compared to the #1 in the leaderboard. Also, it’s very difficult to do inverted power loop, while staying on track with full throttle. Alpha Pilot? Let’s just say robotics and AI will need to learn animal instinct before it can compete with the best FPV pilots.

In comparison, flying freestyle is about learning tricks and maneuver in various interesting location and spots. The goal is mostly about how each style transition smoothly to the next style. What is a good style is also subjective, but in general if you have a very good and precise control of how to fly a racing quad, you can transition to freestyle easily. Because the dynamics are similar, just the goals are different.

I found that practicing racing skills are hard but more motivating than a non-targeted practice of freestyle tricks. This is especially true in a simulator (DRL or VelociDrone), where you don’t really have real-world immersion. In the real world, practically it’s easier to fly freestyle in most cases. Thus, I tend to view freestyle as a relaxation or exploration flying that you couldn’t do with racing. It’s likely that the better someone is at racing, the better someone could be at freestyle.

FPV - How did I get started?

If you are a drone pilot or have been working with drones, quadcopter or fixed wing, I am sure you have heard of FPV, and this post is not targeted towards those who are already in the hobby. With a quick search one can find a lot of tutorials, articles, and even complete training on how to become an FPV pilot. I am writing mostly to give another perspective on how did someone get started on it. Although, I have to state beforehand that my experience is neither unique nor it’s a good way to start with FPV.

Before that I would like to provide a few information where you can seriously learn and understand the process of entering the hobby, building a racing / freestyle quad, and becoming an FPV pilot for racing, freestyle, or however you want to do it for. The sky and your hard-work is the only limit.

  1. OscarLiang: My favorite tutorial and reviews site on anything FPV-related. The writings are very comprehensive and easy to understand compared to a few others who can be too technical, incomplete, too verbose, boring, or lacking in details.
  2. Propwashed: Similar level of comprehensiveness to OscarLiang. I do read specific articles such as How to fly in Rate mode?, PID Tuning guide, LiPo battery guide, and all the technical materials needed to fly a racing quad. But, there can be a few posts that could only be found in OscarLiang. Such as the review or benchmarks of various FPV gears and electronic components.
  3. GetFPV: Also similar to the above two sites. They might also have specific articles that I could not find elsewhere. The difference is GetFPV is also a shop for FPV gears or products. Although, I haven’t really compare them side-by-side. I found that OscarLiang reviewed more products and have more variation on the topics being discussed.
  4. FPV Know-it-All: Created by none other than Joshua Bardwell himself. I do enjoy his explanation of various topics in a friendly manner and easy to understand for beginners. Everyone should watch his PID Tuning Masterclass. But, he seem to be only producing videos, not written articles.
  5. Ethix: Steele Davis a.k.a Mr. Steele is a part of them. His videos and piloting skills are what I would consider as crazy but in a positive way. Although there’s a lot to be learned from his content, he doesn’t seem to write articles or provide a comprehensive guide for beginners. There’s however a few guidelines on FPV.
  6. Johnny FPV: Johnny Schaer, he’s a racer and FPV pilot. I think he has one of the most spectacular videos ever made with FPV. He doesn’t have lots of tutorials or content for learning, but definitely you can learn his equipment setup, and even buy the same set of equipment. But, even though you do, it doesnt’t mean you can match his quality of work.
  7. Sozo: The YouTube channel has videos mostly for long range expedition. It also has tutorials for FPV and racing pilot, accompanied with LiftOff simulator.
  8. TheJumperWire: Not very specific to FPV, but more on electronics, quadcopter, and programming it. The author, Sean Kelly has been working with bitcraze on Crazyflie which is a famous programmable and open source quadcopter platform to do research on autonomous drone. I enjoy his series of articles on LiPo battery characteristics.
  9. Drone Racing League: A leading organization that promotes drone racing all over the world. They have racing competition. All you need to do is download the simulator, practice, and enter a race. If you just want to be a racer. This is where to start.
  10. There’s definitely more, but I can’t recall them.

Also, I have to mention Team BlackSheep and VelociDrone where one is an FPV shop that has been a pioneer in the hobby, and VelociDrone which has been one of the go-to simulators to practice racing, freestyle, or online race. There are others such as LiftOff or DRL (Drone Racing League) simulator, but I haven’t personally used them, because I haven’t got the chance or a high-end gaming computer to also try them out.

Next, how did I get started? As I have mentioned before, I was brought into the hobby after I was involved with a project on autonomous drone with DJI Matrice 100 using ROS. Afterwards, I discovered FPV while searching for parts and components to custom build a quadcopter. But because I do not have a personal space or workbench to work with electronics, I decided not to build one for now, and instead bought a BNF 3-inch quadcopter built by GEPRC from Banggood.

A 3-inch quadcopter is a good balance between being able to fly freestyle, racing and but not throwing yourself immediately to higher-powered 5-inch or 7-inch ones which will require larger capacity LiPo batteries with more weight, size, and more experience on how to handle its speed and range. The bigger quads (although they are still mini-quad) can give a false sense of inspiration, and then you started to take more risks without having a good sense of how it flies, how it’s being tuned, and how to manage all aspects of controlling it.

If I could restart I would start with micro quads that are often referred to as Whoops such as the Emax TinyHawk, Eachine Trashcan, Beta65 / 75 / 85 from BetaFPV. They are good for flying indoors, and getting a feel of flying FPV without flight assistance from sensors such as GPS or Optical Flow to hold its position for hovering. But, I don’t want to spend extra money unless if I actually have a chance to fly indoors — which is unlikely as indoor space tend to be occupied by people — unless if I join a race. Although, they are okay for outdoors too, I’d rather use a Toothpick, 3-inch or larger.

It’s recommended to practice with simulator before flying the real quad, but only after a few flights with my first 3-inch, then I started to really feel how real-life flying is compared to simulator. Aside from physics, real flying require more awareness of the environment, and the configuration of the hardware or flight controller (BetaFlight):

  • LiPo battery characteristics, balanced charging, discharged, what is storage voltage, etc. is very important
  • Electronics and how to recognize symptoms when things are broken (motors, capacitors blowed up, short circuit)
  • Basics of Radio Frequency Communication
  • What’s the difference between TBS transceiver vs FrSky
  • Why do some people need long range transceiver at 900 MHz
  • What kind of antenna to use (Pagoda, Leaf, RHCP or LHCP polarized, SMA or RP-SMA connector)
  • FPV frequency selection on video transmitter (VTX), which channel, band, and transmission power numbers and how it’s being regulated. In Singapore the limit is 100mW, otherwise you would need permission.
  • Without watching the recorded video of my real flight — VelociDrone doesn’t do recording automatically — I wouldn’t be aware of my weaknesses.
  • Throttle management is different on a 3-inch compared to 5-inch. With smaller propellers I have to make sure that throttle doesn’t drop for too long at low altitude, otherwise the quad can easily drop to the ground and flip.
  • VelociDrone’s smallest mini quad is a 4-inch. Although they have micro quads, it’s still not a 2-3 inch quad.
  • The list doesn’t stop here while on the simulator you can simply ignore regulation, hardware, electronics, and focus on flying.

I realized that having an understanding of the fundamentals of building a quadcopter does help with developing an interest on FPV. But, having a good understanding or even mastery of the technology doesn’t always translate to an interest in racing or freestyle flying. First and foremost FPV requires time, a certain personality trait and adventurous mindset. Because there is not much rationale behind practicing hard to learn the basics of high speed racing and acrobatic movements other than a sense of adventure and curiosity. Not to mention understanding the regulation of Unmanned Aerial Vehicle in countries you’re flying in.

Some people are either simply lazy (it does require hard work), or their lifestyle doesn’t lead them into that direction. With all the news of people landing drones at the airport and spying at people’s home, it discouraged people from understanding the various aspects of the hobby whether it’s quadcopter, FPV, or fixed wing. The result is the activity tend to be associated with pranks and immaturity, while in reality the hobby has a lot more variation in it other than people who bought drones for photography or videography.

2020: symmetrical year

It has been a long time since I wrote anything, and I have decided that I had to at least once in 2020. As it’s the only year in my lifetime that is symmetrical. Do numbers mean anything? Probably yes or no, either way I have recently worked on more robotics related software than a consumer app on iOS. I still wrote an iOS app to interface with a backend running on a headless computer such as Raspberry Pi, Intel NUC, NVIDIA Jetson or it can be any kind of computer. I have to admit there aren’t many nice solution out there. Because most robotics software came out from research lab, and it doesn’t often consider smartphone as their frontend. There isn’t a lot of software engineers working in the robotics industry apparently for a valid reason.

If we talk about robotics, probably an arm manipulator, roomba-like differential drive, humanoid, or a dog robot came into our imagination. But, those are not the only robot. A “drone” can be a robot. A car could also be a robot. The only element needed to drive this is sequence of commands to achieve semi or fully autonomous behaviour, without us controlling every aspect of their movement.

Now, I thought robotics is cool. Yes, it’s cool to have a computer to move on its own. But, when we delegate our responsibilities to autonomous robots, our job is done. We don’t consider them as exciting anymore. Who would be excited with a washing machine for an example? Unless a robot could interact with us like a human or animal does, it’s boring.

As I have worked with drones for a few months, another world has revealed itself to me. The first drone I have touched is a DJI Matrice 100. It’s an amazing piece of equipment, as I never worked with an aircraft before. However, after a while as any other piece of equipment, it’s not something that I get excited over.

I guess the main reason is the difficulty with which to use it on various activities. It’s too large, noisy, and although it’s programmable, how far can we get with it without years of work, to make it on par to the latest and significantly cheaper drone on the market?

Then, I have also tried the newer DJI Mavic 2 Pro. It’s much better, more stable, and easier to control. It does what it’s designed to do (flying, taking photos and videos), but it’s not an open platform where you can replace the hardware or software. But, just last year Skydio 2 was released. It has far better autonomous flying capabilities, and for some reason they’d rather state that it’s using NVIDIA Jetson TX2 as a computer. This is not the case with Mavic. Though DJI is using an Ambarella vision processor, they don’t advertise it anywhere in their marketing materials. Skydio is another drone targeted at Mavic’s customer base, and it’s also not a programmable drone, though it might be in the future. I assume any product equipped with NVIDIA Jetson series board need to provide some attribution. This is also the case with DJI Matrice 2xx series.

Now, if I need a fully autonomous drone or I prefer to call it quadcopter — as it’s a multirotor with 4 motors driving its propellers — the only way is by building it myself. But, while I have a thought of doing that, I found that quite a few of DIY builders are doing it for racing or freestyle flying with an FPV (First Person View) goggles. I found this activity to be very active and growing, and you can search on the internet to find various setup of racing / freestyle quad with various configuration. It’s also an art in flying in remote areas, nature, abandoned space, and video shooting at its best.

I could consider that a racing / freestyle quad and FPV goggles has pushed the miniaturization in electronics and innovation in radio transmission technology which an autonomous drone project has not achieved so far or they are moving at a slower rate. The differentiating factor is the kind of people or flyers that used them. An autonomous drone require heavy computation, and there aren’t a lot of hobbyist in programming and electronics, while an FPV hobby on the other hand is more appealing to anyone who is seeking an adrenaline rush. Anyone with little background in electronics and software can enter the hobby with persistent practice. There’s always those who seek thrill, thus there are always demands. It’s not very hard to manufacture electronics that could drive motors aggressively. But, it’s very hard to manufacture electronics that can simultaneously drive motors and do a lot of heavy matrix calculations in real time, in other words to put a GPU on a mini or micro quadcopter.

The other side of casual user is probably better served with a stable drone that flies by itself, which is so far only Skydio that could achieve this. Any Mavic series so far is relatively okay at obstacle avoidance but it’s very limited, you need to fly it at a certain height to track you. Also you can’t send Mavic for an autonomous mission without good GPS signal. Because in order to move itself without GPS it needs a point of reference. At the moment it’s just the person flying it, another person, or a car. While ideally it should build a map of its environment from its own motion and visual processing a.k.a SLAM, and that’s what Skydio is doing so far and why it has an amazing tracking and obstacle avoidance capability.

In the case of aggressive quadrotor movement such as drone racing, it might be possible in the future to have it in the form of pilot assistance system. But instead of helping to avoid obstacles — although that would be good too — imagine that a pilot could learn the optimized flight characteristics of a racing / freestyle quad as a feedback system to the manual flight in Acro or Rate mode via the remote control. With digital HD FPV system such as the latest one from DJI, this feedback information can be provided in real time via the goggles as well. Combined with a simulator system, professional pilot datasets, and flight logs, a pilot at any level is able to improve him / herself via this training-in-the-loop system.

Rewind view controller states (observables)

A pattern that was not immediately obvious when you began to use an Rx-based view controller setup is to undo/reset all of the observables that have been subscribed to after it re-appears.

This is a common behaviour in a navigation controller stack of view controllers. We might have various controls in a view controller, it could be a text field, text view, or anything that you might want to return to from the next controller in a navigation stack.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
import UIKit
import RxSwift
import RxCocoa

final class FormViewController: UIViewController {
var disposeBag: DisposeBag = DisposeBag()

var completed: PublishSubject<Void>()

lazy var textField: UITextField = {
let textField = UITextField()
textField.autocorrectionType = .No
textField.autocapitalizationType = .None
textField.returnKeyType = .Done
return textField
}()

func viewDidLoad() {
super.viewDidLoad()

// We indicate that a certain action should be completed,
// once the text field's text reaches a certain length
textField.rx_text
.asObservable()
.subscribeNext { text in
if text.length > 6 {
completed.onCompleted()
}
}
.addDisposableTo(disposeBag)
}

func viewWillAppear(animated: Bool) {
super.viewWillAppear(animated)

// Observe when returning
completed = PublishSubject<Void>()
completed
.subscribeCompleted {
// A subscribed action is completed
}
.addDisposableTo(disposeBag)
}

func viewWillDisappear(animated: Bool) {
super.viewWillDisappear(animated)

// Clean up all observables
disposeBag = DisposeBag()
}
}

Typically, we would want to do a certain action in a view controller, that could be observed, and it should be known when does it complete. One example is network request, but the above example is simplified to illustrate the idea of running an action that is triggered by an observed event (text field event). PublishSubject is a common example to represent an action subject that can be subscribed to and could subscribe to an observable.

In the snippet above, on view appearance, the action is being reset, and re-subscribed, while on view disappearance, the disposeBag is being re-created. Once it returns to this view controller from the navigation stack it will have the completed subject re-created and re-subscribed.

Without the above setup, a completed action remains completed and it will not emit anymore items that is being observed.

To think about it a little bit further, before Rx I often do not have an idea on how to “replay” a certain action within scenes of view controllers. Traditionally this can be done by re-creating whatever it’s an object responsible for such action (e.g. NSURLRequest).

It can always be done traditionally without Rx, but in that way there doesn’t seem to be a generalisation of idea in how to represent an action that can be anything.

Revisiting Size Classes

Articles on iOS size classes never went beyond the ideal that we have to think in terms of Any, Regular, Compact Horizontal or Vertical size classes for adaptive layout, instead of pixels and devices screen sizes.

Ever since size classes was introduced in iOS 8, I have always wondered, why do iPhone 4s, 5, and 6 are all under Compact Horizontal and Regular Vertical size classes?

While in reality they have different dimensions, and during design phase, we have to think about the dimensions of views and their subviews, together with the distances between them due to these varying screen sizes.

We could of course declare auto layout constraints programmatically with a multiplier based on screen width or height. But, from what I understood size classes are supposed to move us forward from this way of thinking towards designing layout specifically just for the abstractions of them. Until I encountered one article that describes my confusion on size classes:

Size Classes - Close, but Not Quite
By Kevin Packard @kevinpackard One Storyboard to Rule Them All At WWDC 14, Apple introduced Unified Storyboards and Size
blog.blackfrog.com

Note that, the above article was posted about 2 years ago.

Exactly, if a designer should consider how an iPhone 4s is way shorter than an iPhone 6, resulting in individual layout design for each iPhone 4s, 5, 6, 6 plus. Then, aren’t we still doing what we are doing before size classes were introduced?

Well, I hope this will change in WWDC 2016.

Moving On

After about 4.5 years of working primarily as an iOS Software Engineer at 2359 Media — including server side development of VoicePing with node.js that I have been doing for a year — I am finally ready to move on to my next challenge to work as a Lead iOS Developer at Touché Singapore.

Touché core product is a seamless payment system together with a device using biometric (fingerprint) as an identity system. It’s something that I have never explored before, and it’s only recently that fingerprint based identity started to appear on mainstream devices such as iPhone (Touch ID) and then Android. Thus, it’s intriguing that they’ve managed to produced a standalone device and to continue pushing in that direction.

Working in a payment industry is something that has been in the back of my mind for quite a while, though I never got a chance to really understand how all the system works.

In relation to that, I have been increasingly comfortable with cards, digital, or online payment, and I hope someday cash and coins are no longer necessary.

This reminded me of the time when I visited Sweden (Uppsala, and sometimes Stockholm) for 3 months around 6 years ago, and that was the first time that I had witnessed for myself the widespread usage of cards, as opposed to cash. As can be read from here as well, which ignited my brain cells, though these thoughts have pretty much stayed there and ignored for the past 4 years.

On the contrary, after reading the Preface of bitcoin book from Princeton course, I have realised that the major advantage of using cash is the anonymity of transactions, and it’s independent of third party verification such as payment processors (Visa, MasterCard, or Banks, etc.).

I agree that these are the properties that don’t exist in a cashless system. But, we have moved on from a barter system, to cash, credit cards, and NFCs on smartphones, with a bit of sacrifice on our privacy and security. These made transactions much easier, and now it’s easier than ever to transfer money and purchase goods across the world. So, in the end the progress is really worth something.

Back to the topic of my career move, I would mostly still be doing mobile development on the iOS side, which is still exciting given the recent movements by Apple (Swift, open source, etc.), but in a new industry, a new team, and a new focus.

Playing around with Data Science

After a few influences here and there from colleagues, and how I kept hearing this buzzword time and time again. I am finally curious to understand what is data science all about.

I started with an online course from Coursera in my spare time as an introduction, then I’m currently in the process of finishing the R programming course at the end of October.

Overall, as a long time programmer, I consider the level of difficulty of the R programming assignments as easy. You don’t need to spend days to solve one of them. Mostly, it can be done in an hour or more within a day. Moreover, the course team are very generous in providing examples, and all of the lecture materials are very organized.

Though, it’s likely to get harder if I proceed to the next course in the data science specialization track. Which, I probably will. But before that, I’ve accidentally discovered data science on the command line by Jeroen Janssens which introduced a set of toolbox that I perceived as a more complete tooling than just utilizing R.

I generally believe that programming toolsets should not be confined to one environment, one editor, with their own custom interfaces or protocols written by a vendor that is not extensible or interoperable with another toolset. Although this is not the case with R, but I have seen other programming tools — which I can’t recall right now — that are built this way.

What about Python for data science? Yes, Python is a general purpose programming language, but I assume that the data science community are more productive within the R programming environment. Proven by how Johns Hopkins University are using R to teach data science. It would take more learning and time for those with little programming experience to solve these problems using Python.

The data scientists have done a lot for the R development tools. I have searched and peeked a bit into a few of the stuff that they had built. Though, honestly I haven’t explored what kind of tools or libraries in Python that is made for data science.

So, I’m thinking along the lines of using what is already been done on R, and we should be able to interface with it from a different programming environment.

For example:
An interactive application (desktop, web, mobile, whatever) that continuosly records data from user input, or other sources which is then formatted into CSV or data frames which is the data type that R usually interacts with.
These data should be fed into R programs that are being specifically written to do a unit of calculation. The raw data of the result should be sent back to the application which are optimized for user interface and visualization purpose.

We need this interfacing because I’m not sure that R tools (even though it has Shiny and other plotting libraries) are complete or well developed enough to be able to do advanced UI or visualization, it’s not their focus.
Regardless, it’s still very early for me to tell whether it’s even better to remove R from this sort of system. Because it does seem like it’s just a tool for data scientist, it’s not something that you’ll use for an automated stand alone performance critical system that will process and compute huge amounts of data.

VoicePing

Lesson learned while building VoicePing.

It’s neither about pursuing the big ideas nor technical challenges associated to building a highly mutable state machine with multitude of side effects.

Not even about achieving the elegant architecture of cooperating microservices which would scale to billion clients.

Yes, at some point the above big goals will seem to be the overall purpose or the end result of a real-time system. But right now:

80% are about making sure things are working as expected at the right time for:

  1. Consumers of the services (mobile clients).
  2. End users of the mobile clients.

20% are about making sure every little part of the software should be written and configured in the most understandable way, with as little overhead as possible in terms of memory or processing resources.

More HTTP server benchmark

Following from my last post on benchmarking hello world http server on various platforms, I continued with increasing the number of requests and concurrency. In addition, I added scalatra, play, and greenlet tornado to the list.

What is greenlet? To make it short, it’s just like goroutine in golang. What’s goroutine? Read about it here.

This time, I excluded node.js, twisted, and eventmachine from the list, because apache bench could not finish the benchmark at 100,000 number of requests with 100 multiple requests at a time.

node.js and eventmachine could still perform well at 15000 requests with 100 multiple requests, and the results of both don’t have a significant difference compared to the last time I tested it. Beyond that, the test was aborted after 10 failures, with the same error:

apr_socket_connect(): Operation already in progress (37)

I have also made an upgrade on ab executable to Version 2.3 Revision 1430300. This could be built from the latest httpd source code at version 2.4.4.

That being said, I may try to find out what are the problems in my test setup — Mac OS X 10.8.4, Retina MacBook Pro, Mid 2012 with Quad Core i7 at 2.3 GHz — that fails apache bench to finish it for node.js, twisted, and eventmachine. I figured that to do a full-scale benchmark with apache bench, it needs to be run on a Linux server.

The apache bench command that I was using:

% ab -r -k -n 100000 -c 100 -e ~/Documents/each_platform_results.csv http://localhost:port/

The source code for greenlet with tornado:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import tornado.httpserver
import tornado.web
from greenlet_tornado import greenlet_asynchronous, greenlet_fetch

class MainHandler(tornado.web.RequestHandler):
@greenlet_asynchronous
def get(self):
self.write("Hello World!")

application = tornado.web.Application([
(r"/", MainHandler),
])

if __name__ == "__main__":
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(8001)
print("Tornado listening at 8001")
tornado.ioloop.IOLoop.instance().start()

As you can see, there’s not much difference except for the @greenlet_asynchronous decorator.

The result is not that different either:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Server Software:        TornadoServer/3.0.2
Server Hostname: 0.0.0.0
Server Port: 8001

Document Path: /
Document Length: 12 bytes

Concurrency Level: 100
Time taken for tests: 32.202 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 100000
Total transferred: 23100000 bytes
HTML transferred: 1200000 bytes
Requests per second: 3105.36 [#/sec] (mean)
Time per request: 32.202 [ms] (mean)
Time per request: 0.322 [ms] (mean, across all concurrent requests)
Transfer rate: 700.52 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 5
Processing: 6 32 10.7 29 100
Waiting: 6 32 10.7 29 100
Total: 10 32 10.7 29 100

Percentage of the requests served within a certain time (ms)
50% 29
66% 29
75% 30
80% 30
90% 51
95% 61
98% 63
99% 68
100% 100 (longest request)

Actually, tornado without greenlet is slightly faster by 0.01 ms.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Server Software:        TornadoServer/3.0.2
Server Hostname: 0.0.0.0
Server Port: 8001

Document Path: /
Document Length: 12 bytes

Concurrency Level: 100
Time taken for tests: 31.168 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 100000
Total transferred: 23100000 bytes
HTML transferred: 1200000 bytes
Requests per second: 3208.39 [#/sec] (mean)
Time per request: 31.168 [ms] (mean)
Time per request: 0.312 [ms] (mean, across all concurrent requests)
Transfer rate: 723.77 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 4
Processing: 7 31 11.0 27 108
Waiting: 7 31 11.0 27 108
Total: 8 31 11.0 27 108

Percentage of the requests served within a certain time (ms)
50% 27
66% 28
75% 28
80% 29
90% 50
95% 59
98% 63
99% 73
100% 108 (longest request)

Next up, Scalatra which is a sinatra inspired web framework written in scala, but it takes much longer to setup compared to its predecessor.

Here’s the code:

1
2
3
4
5
6
7
8
9
10
package com.jessearmand.helloscala

import org.scalatra._
import scalate.ScalateSupport

class MyScalatraServlet extends HelloScalaAppStack {
get("/") {
"Hello World."
}
}

Yes, that simple! Once you set it up with some effort, if you never done any Scala programming.

How does it perform? Roughly, it’s twice as fast as python’s tornado.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Server Software:        Jetty(8.1.8.v20121106)
Server Hostname: 0.0.0.0
Server Port: 8080

Document Path: /
Document Length: 12 bytes

Concurrency Level: 100
Time taken for tests: 16.332 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 100000
Total transferred: 14700000 bytes
HTML transferred: 1200000 bytes
Requests per second: 6122.82 [#/sec] (mean)
Time per request: 16.332 [ms] (mean)
Time per request: 0.163 [ms] (mean, across all concurrent requests)
Transfer rate: 878.96 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 4
Processing: 0 16 4.2 16 79
Waiting: 0 16 4.2 16 79
Total: 0 16 4.2 16 80

Percentage of the requests served within a certain time (ms)
50% 16
66% 17
75% 18
80% 19
90% 21
95% 23
98% 26
99% 29
100% 80 (longest request)

Now, let’s try Play framework which I guess is the best framework written in Scala to build a web application with non-blocking IO.

1
2
3
4
5
6
7
8
9
10
11
package controllers

import play.api._
import play.api.mvc._

object Application extends Controller {
def index = Action {
Ok("Hello World!")
// Ok(views.html.index("Your new application is ready."))
}
}

Note how I commented out the code responsible for rendering the default index.html view. This is only because I want a pure “Hello World” comparison. Although, I do understand that it’s not a comprehensive way to benchmark web apps for what it can do in various other cases. Read here for a good example of doing this.

Result on initial launch of the app:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
Server Software:
Server Hostname: 0.0.0.0
Server Port: 9000

Document Path: /
Document Length: 12 bytes

Concurrency Level: 100
Time taken for tests: 6.425 seconds
Complete requests: 100000
Failed requests: 10
(Connect: 5, Receive: 5, Length: 0, Exceptions: 0)
Write errors: 0
Keep-Alive requests: 99995
Total transferred: 11599420 bytes
HTML transferred: 1199940 bytes
Requests per second: 15563.80 [#/sec] (mean)
Time per request: 6.425 [ms] (mean)
Time per request: 0.064 [ms] (mean, across all concurrent requests)
Transfer rate: 1763.00 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.9 0 105
Processing: 2 6 4.9 5 96
Waiting: 0 6 4.9 5 96
Total: 2 6 5.1 5 157

Percentage of the requests served within a certain time (ms)
50% 5
66% 6
75% 7
80% 8
90% 11
95% 15
98% 22
99% 27
100% 157 (longest request)

Then, after the first benchmark:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Server Software:
Server Hostname: 0.0.0.0
Server Port: 9000

Document Path: /
Document Length: 12 bytes

Concurrency Level: 100
Time taken for tests: 3.933 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 100000
Total transferred: 11600000 bytes
HTML transferred: 1200000 bytes
Requests per second: 25422.96 [#/sec] (mean)
Time per request: 3.933 [ms] (mean)
Time per request: 0.039 [ms] (mean, across all concurrent requests)
Transfer rate: 2879.94 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 4
Processing: 2 4 1.2 3 13
Waiting: 2 4 1.2 3 13
Total: 2 4 1.3 3 16

Percentage of the requests served within a certain time (ms)
50% 3
66% 4
75% 4
80% 5
90% 6
95% 7
98% 7
99% 8
100% 16 (longest request)

It seems Play framework takes some time from the initial launch to achieve some kind of a “steady state”.

Last, let’s do some benchmark with go lang again. This time, I didn’t set GOMAXPROCS which sets the maximum number of CPUs that can be executing simultaneously. Thus, I let it use a default value that I don’t know of.

Result:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Server Software:
Server Hostname: 127.0.0.1
Server Port: 3000

Document Path: /
Document Length: 12 bytes

Concurrency Level: 100
Time taken for tests: 2.559 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 100000
Total transferred: 13800000 bytes
HTML transferred: 1200000 bytes
Requests per second: 39077.64 [#/sec] (mean)
Time per request: 2.559 [ms] (mean)
Time per request: 0.026 [ms] (mean, across all concurrent requests)
Transfer rate: 5266.32 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 4
Processing: 0 3 0.6 3 6
Waiting: 0 3 0.6 3 6
Total: 0 3 0.6 3 6

Percentage of the requests served within a certain time (ms)
50% 3
66% 3
75% 3
80% 3
90% 3
95% 4
98% 4
99% 5
100% 6 (longest request)

Go is definitely still a clear winner with half the speed compared to my last benchmark. Next to Go performance is Play framework.

On a related note, there’s an interesting article for Python developers if you’re thinking about migrating to Go.

I'm sold with golang

While I was waiting for WWDC, I did an apache bench for simple hello world server written in go, ruby’s eventmachine, python’s tornado and twisted, and the well-known node.js.

% ab -r -k -n 15000 -c 50 -e ~/Documents/each_platform_results.csv http://localhost:port/

15000 requests and 50 concurrent requests is the only setting that I could test on all platforms without failing tests with an error like this:

apr_socket_connect(): Operation already in progress (37)

How is the result?

OK, let’s start with tornado (I believe it’s used by facebook for realtime news feed?). Here’s the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import tornado.httpserver
import tornado.ioloop
import tornado.web

class MainHandler(tornado.web.RequestHandler):
@tornado.web.asynchronous
def get(self):
self.write("Hello World\n")
self.finish()

application = tornado.web.Application([
(r"/", MainHandler),
])

if __name__ == "__main__":
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(8001)
print("Tornado listening at 8001")
tornado.ioloop.IOLoop.instance().start()

Result:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Server Software:        TornadoServer/3.0.2
Server Hostname: 0.0.0.0
Server Port: 8001

Document Path: /
Document Length: 12 bytes

Concurrency Level: 50
Time taken for tests: 5.072 seconds
Complete requests: 15000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 15000
Total transferred: 3465000 bytes
HTML transferred: 180000 bytes
Requests per second: 2957.67 [#/sec] (mean)
Time per request: 16.905 [ms] (mean)
Time per request: 0.338 [ms] (mean, across all concurrent requests)
Transfer rate: 667.21 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 6 17 6.0 15 65
Waiting: 6 17 5.9 15 65
Total: 7 17 6.0 15 65

Percentage of the requests served within a certain time (ms)
50% 15
66% 16
75% 16
80% 17
90% 27
95% 30
98% 35
99% 38
100% 65 (longest request)

Then… the reputable Twisted:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
from twisted.web import server, resource
from twisted.internet import reactor

class HelloResource(resource.Resource):
isLeaf = True

def render_GET(self, request):
request.setHeader("Content-Type", "text/plain")
return "Hello World\n"

port = 8001
reactor.listenTCP(port, server.Site(HelloResource()))
print "Twisted running at %d" % port
reactor.run()

Result:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Server Software:        TwistedWeb/13.0.0
Server Hostname: 0.0.0.0
Server Port: 8001

Document Path: /
Document Length: 12 bytes

Concurrency Level: 50
Time taken for tests: 5.893 seconds
Complete requests: 15000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 2115000 bytes
HTML transferred: 180000 bytes
Requests per second: 2545.19 [#/sec] (mean)
Time per request: 19.645 [ms] (mean)
Time per request: 0.393 [ms] (mean, across all concurrent requests)
Transfer rate: 350.46 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 2
Processing: 11 20 2.2 19 34
Waiting: 11 20 2.2 19 34
Total: 11 20 2.2 19 34

Percentage of the requests served within a certain time (ms)
50% 19
66% 20
75% 20
80% 21
90% 22
95% 23
98% 25
99% 27
100% 34 (longest request)

What about ruby eventmachine?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
require 'eventmachine'
require 'evma_httpserver'

class Handler < EventMachine::Connection
include EventMachine::HttpServer

def process_http_request
resp = EventMachine::DelegatedHttpResponse.new( self )
resp.status = 200
resp.content = "Hello World!"
resp.send_response
end
end

EventMachine::run {
EventMachine::start_server("0.0.0.0", 8002, Handler)
puts "Listening... at 8002"
}

Result:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Server Software:
Server Hostname: 0.0.0.0
Server Port: 8002

Document Path: /
Document Length: 12 bytes

Concurrency Level: 50
Time taken for tests: 1.909 seconds
Complete requests: 15000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 780000 bytes
HTML transferred: 180000 bytes
Requests per second: 7858.99 [#/sec] (mean)
Time per request: 6.362 [ms] (mean)
Time per request: 0.127 [ms] (mean, across all concurrent requests)
Transfer rate: 399.09 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 4.4 0 245
Processing: 1 6 19.5 4 247
Waiting: 1 5 19.3 3 247
Total: 2 6 20.0 4 247

Percentage of the requests served within a certain time (ms)
50% 4
66% 4
75% 4
80% 4
90% 6
95% 8
98% 11
99% 80
100% 247 (longest request)

Wow, that’s faster than tornado or twisted in terms of requests per second, but it transferred less data for some reason. OK, we move on to node.js, I expect it should be faster.

1
2
3
4
5
6
7
8
9
var http = require('http');

http.createServer(function (request, response) {
response.writeHead(200, { "Content-Type": "text/plain" });
response.write("Hello World\n");
response.end();
}).listen(8000);

console.log('Listening on port 8000...');

Result:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Server Software:
Server Hostname: 0.0.0.0
Server Port: 8000

Document Path: /
Document Length: 12 bytes

Concurrency Level: 50
Time taken for tests: 2.181 seconds
Complete requests: 15000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 1695000 bytes
HTML transferred: 180000 bytes
Requests per second: 6877.80 [#/sec] (mean)
Time per request: 7.270 [ms] (mean)
Time per request: 0.145 [ms] (mean, across all concurrent requests)
Transfer rate: 758.98 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.1 0 137
Processing: 1 7 11.9 6 142
Waiting: 1 7 11.8 6 142
Total: 2 7 11.9 6 142

Percentage of the requests served within a certain time (ms)
50% 6
66% 6
75% 6
80% 6
90% 7
95% 9
98% 12
99% 101
100% 142 (longest request)

Yes, unsurprisingly it has the highest transfer rate, although it has less mean requests / sec compared to ruby event machine. Based on the percentage of the requests served, node.js is faster.

Are we done? No, the last but not least:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
package main

import (
"fmt"
"io"
"net/http"
"runtime"
)

func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/plain")
io.WriteString(w, "Hello World\n")
})

runtime.GOMAXPROCS(4)
fmt.Println("Server running at http://127.0.0.1:3000/ with GOMAXPROCS(4) on i7")
http.ListenAndServe("127.0.0.1:3000", nil)
}

Result:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Server Software:
Server Hostname: 0.0.0.0
Server Port: 3000

Document Path: /
Document Length: 12 bytes

Concurrency Level: 50
Time taken for tests: 0.228 seconds
Complete requests: 15000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 15000
Total transferred: 2070000 bytes
HTML transferred: 180000 bytes
Requests per second: 65771.30 [#/sec] (mean)
Time per request: 0.760 [ms] (mean)
Time per request: 0.015 [ms] (mean, across all concurrent requests)
Transfer rate: 8863.71 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 2
Processing: 0 1 0.1 1 2
Waiting: 0 1 0.1 1 2
Total: 0 1 0.2 1 3

Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 1
95% 1
98% 1
99% 1
100% 3 (longest request)

Transfer rate is 8863.71 Kbytes/sec with 65771.3 requests per second?

Go lang, I’m sold.