Profiling Ruby Code

This should be a small primer on how to profile Ruby/Rails code.

Tools of the trade

Thankfully, there are plenty of awesome gems to help us improve performance in Ruby apps. My particular favorites include:

  • rack-mini-profiler

    Shows a small speed counter, enabled for every HTML page in your app. Highly configurable and can even be used in production! Also has plugins to generate fancy flamegraphs, thanks to gems such as flamegraph and stackprof.

  • bullet

    Warns you about N + 1 queries, unused eager loading, and probable cache counter performance optimizations. Can be configured to send notifications to multiple channels (e.g.: Growl, XMPP, Honeybadger, Bugsnag, Airbrake, Rollbar, Slack), and also raise errors if necessary (particularly useful in specs).

  • ruby-prof (MRI-only)

    Code profiler that can output to many different formats, including the Valgrind calltree format (to be used by tools such as KCacheGrind) and graph profiles. It profiles multiple threads simultaneously.

    KCacheGrind output

    Ruby-prof call stack

  • Custom tracing code

    If you want to try a poor-man’s version of the ruby-prof call stack, feel free to use this implementation (taken from this one and a little improved upon):

def trace(filename = '/tmp/trace', event_types = [:call, :return], *matchers)
  points = []

  tracer = TracePoint.new(*event_types) do |trace_point|
    if matchers.all? { |match| trace_point.path.match(match) }
      points << { event: trace_point.event,
                  file: trace_point.path, line: trace_point.lineno,
                  class: trace_point.defined_class,
                  method: trace_point.method_id }.merge(
        trace_point.event == :return ? { return: trace_point.return_value } : {}
      )
    end
  end

  result = tracer.enable { yield }

  File.open("#{filename}.rb_trace", 'w') do |file|
    points.each do |point|
      event_prefix = point[:event] == :return ? 'return' : 'call'
      return_value = point[:return] ? "(#{point[:return]})" : ''
      file.puts "#{point[:file]}:#{point[:line]}:#{event_prefix} #{point[:class]}##{point[:method]} #{return_value}"
    end
  end

  result
end

This works wonders with Emacs’ grep-mode, it’s very IDE-like. Kinda like this:

Testing for performance

There’s good advice in the Rails performance guides.

Only thing I want to add is that the performance tests only make sense if they’re run consistently in similar machines (or, even better, always the same one), otherwise you run the risk of getting different results based on hardware, system load and other sources of noise. It pays to establish the most isolated environment possible here - refer to your CI documentation for that.

Fixing a bottleneck

My favorite method is: profile, analyze results, make the smallest changes that seem to fix the problem, rinse and repeat. Never assume you know what the bottleneck is - you’ll be surprised with real life.

That being said, here are some common tips to fix degraded performance in apps:

  • Choose good algorithms

    No way to squeeze performance out of a bad algorithm, even if you’re writing it in hand-optimized assembly. Be wary of O(n^2) algorithms for seemingly-simple things, iterating over map-like structures, and things like that.

    One caveat: sometimes an algorithm has good worst-case bounds, but its performance characteristics only show up for very large input values. In this case, it’s usually better to use a “worse” algorithm that fits your usual input.

  • Lazily evaluate what you need

    Got the result you want? Good, now break out of the loop you’re in. Seems simple, but sometimes it’s the cause of very degraded performance.

    Also, you can use the Enumerator class to stream potentially long operations. Or, if you’re lucky enough to be writing Haskell, the language does that for you!

    Main takeaway here is: do not run any code you don’t need to.

  • Cache expensive results

    If you have an object that looks like a pure function, you can usually cache its results, and save these cycles for other operations. It trades CPU for memory, but that is usually a valid tradeoff.

  • Push very expensive operations to background jobs

    If you’ve done all of the above and the app still does not perform as you would like to, try to push slow operations to background jobs. If you don’t have a hard requirement to recalculate this data on-the-fly (this is particularly true of complex metrics), this approach can work wonders.

Useful bibliography

Refer to these to get a better grasp of performance improvements for your Rails app:

Conclusion

Hopefully, you now know how and where to start optimizing after reading this article. If there’s interest, I can write more detailed articles explaining each of the layers of a performance optimization job - from profiling to tuning different aspects of the Rails app (SQL queries, ActiveRecord, view logic, assets delivery, front-end code, protocols).

Compiling Emacs 25 - X11 woes and toolkit issues

I was prompted by the requirements of this very important package to recompile my Emacs from source. I thought that’d be easy. Oh boy, was I in for a surprise.

First try

Everything went smoothly in the beginning. I would just run git fetch && git pull, ./autogen.sh, ./configure && make, cd src/ && ./emacs, and I would be looking at the Spacemacs window in no time. “This is so easy, I can’t believe it won’t fail”.

I was correct in my suspicion.

First, Emacs complained about (void-function define-inline). After reading INSTALL, I learned that I should use the make bootstrap command.

After that, it did compile, but with none of the libraries I intended to use. The graphics were weird, too - probably because I wasn’t using GTK, but the Lucid X toolkit.

“Just configure it with your desired toolkit!”

When I ran ./configure --with-x-toolkit=gtk3 --with-modules && make bootstrap, my problems just started:

configure: error: Package 'xproto', required by 'xau', not found

OK, where is ‘xproto’? After running locate xproto, I found out that there was a .pc file in a system directory. If you look at the documentation for pkg-config, you’ll see the following:

DESCRIPTION The pkg-config program is used to retrieve information about installed libraries in the system. It is typically used to compile and link against one or more libraries. Here is a typical usage scenario in a Makefile:

   program: program.c
        cc program.c $(pkg-config --cflags --libs gnomeui)

   pkg-config retrieves information about packages from special metadata files. These files  are  named  after
   the  package,  and  has  a  .pc  extension.   On  most  systems,  pkg-config  looks  in /usr/lib/pkgconfig,
   /usr/share/pkgconfig, /usr/local/lib/pkgconfig and /usr/local/share/pkgconfig for  these  files.   It  will
   additionally look in the colon-separated (on Windows, semicolon-separated) list of directories specified by
   the PKG_CONFIG_PATH environment variable.

So, basically this program ensures we’re compiling with all the right flags. I’m running a 64-bit Linux, and this means that libraries might not be in their standard places (that, and the fact that I also run a symlink-hodge-podge between 3 different partitions. More on that later, if I’m not too ashamed of talking about that in public).

List of commands for installation

X11_PATH=`dirname $(locate x11 | grep pkgconfig | head -1)`
XPROTO_PATH=`dirname $(locate xproto | grep pkgconfig | head -1)`
export PKG_CONFIG_PATH="$XPROTO_PATH:$X11_PATH:$PKG_CONFIG_PATH"
./configure --with-x-toolkit=gtk3 --with-modules
make bootstrap

Other issues

After that I faced this known bug on Spacemacs. Thankfully, the devs already found a workaround:

dotspacemacs-configuration-layers
'(
  (python :variables python-test-runner '(nose pytest))
  ;; other layers here
  )

After that, I had a problem with the background color - it always defaulted to white. Thanks to this SO answer, I traced the root of the problem to a “Customize” option in my .spacemacs. Deleting it solved the problem:

(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 ;; '(default
 ;; a bunch of color definitions here that weren't helpful
 ;; )
 )

XWidgets (optional)

After that, I figured “hey, why don’t I compile with xwidgets support? Looks cool, and it’ll help while I don’t get a second monitor”. For that, I needed to install this dependency:

sudo apt-get install libwebkitgtk-3.0-dev

And the command for compiling Emacs with this optional module became:

./configure --with-x-toolkit=gtk3 --with-modules --with-xwidgets

Takeaways

Now I can meme inside Spacemacs to my heart’s content. Hopefully, this blog post might help someone who’s going through the same problems as I did.

sass-lint, why don't you just work?

Tonight I came up with this, to see if I could have some saner error reporting (other than standard sass complaining about non-consequential stuff) in my stylesheets.

Results were a bit lackluster, as the following image shows (plain sass works fine, though):

Am I missing something obvious, or it’s just that sass-lint is works this way?

(BTW, this gist is not entirely correct, but that’s the best I have…for now. If time and patience allow, hopefully I’ll submit a patch to flycheck when I get this working)

Thoughts on Shen

What is Shen?

  • Practical version

Shen is an hosted Lisp that comes with a full-featured macro system, a Prolog, an optional type system more powerful than Haskell’s, and does it all in under 5000 lines of code. It has been ported to many platforms including Java, JavaScript and Ruby.

  • Emotional version

It is the magnum opus of a lone programmer who is well-versed in theory AND practice. Even if you don’t think the language might be worth more attention outside of this talk, it might give you a few cool insights.

The essay linked under `practice` is one I feel so strongly about that I must restrain myself not to tell it over and over to every new coworker. Guy tells a bit about my personal story - only I didn’t return to academia to see what I could gain by studying more.

Why Lisp?

Most people could rephrase it as ‘these parens hurt my eyes!’. Rest assured that they matter. We Lispers understand that they’re there for a very important reason, and that reason is:

Lisp is the programmable programming language.

Btw, there are tools to alleviate most problems with parens. There’s the awesome Paredit, and the subject of M-Expressions always comes up (we only need more developers to actively work on this!)

Problems in paradise?

As told in “The bipolar Lisp programmer”, Lisp loses culturally because it’s so easy to do complicated stuff in it that no one cares to document and package nicely for “lesser” mortals.

(Ironically enough, I’m writing this while in one of my own insomniac intellectual frenzy of productivity. Especially in my country - Brazil - I find a teacher who understands the finer psychological aspects of learning and productivity a very rare sight. Perhaps we can learn from other cultures how to do university better?)

There’s a lot of historical baggage and lingo that is very off-putting for new developers (“head is car and tail is cdr? What do these even mean?”)

The very nature of the language gives a lot of space for the programmer to make mistakes. It is powerful, but should be wielded wisely.

(As with most dynamic languages, in fact.)

One coworker wrote a very cool piece on some of the problems he found in the (Common) Lisp language. I agree with the sentiment (I have spent some time with Haskell because of this), but I feel that Shen is a step ahead, by joining the best of both worlds.

Got it, Lisp is cool but not perfect. How does Shen compare?

Shen has a Haskell-style type system that is entirely optional. This allows one to opt-in or out of this feature whenever needed - so you get the best of both worlds: the flexibility of dynamic languages and the feedback on compiling mistakes of “static” languages.

It respects and improves on the venerable Lisp tradition of easily creating DSLs by providing a built-in, baked-inside-the-language YACC. This, coupled with the fact that Shen is homoiconic, gives you a lot of power with very low operational overhead. It is used to implement the built-in Prolog in what I consider a very small LOC for the amount of functionality it provides.

(Yeah, there’s a defcc “primitive”. How cool is that? Perl 5 people should be envious.)

(To clarify: what is “operational overhead”? I want to say that it’s easier to do JetBrains-style stuff when the language allows you to. Better than having to muck around parsing code, and deal with the problems related to it. No syntax = one less problem to focus on.)

Strangely enough, it is a Lisp with the Unix-virusesque quality (as argued in the body of essays collectively named as Worse is Better). It compiles down to a mini-Lisp called KLambda that is very easy to port to different environments, and this has been done many, many times.

Shen sounds very cool. Now, I’m sure it’s not perfect. Can you tell me about its problems?

Sure! There are a few:

The community is definitely niche. At the time of this writing, the mailing list for the project is around 500 people - IMHO, it looks like it’s mostly academics who bend towards the practical and like to “get their hands dirty” on code. Almost the same demographic one would expect to find in Haskell mailing lists.

The language’s author is known for being a bit harsh towards open source. It difficults adoption (no self-respecting developer wants to be forced to use something [s]he can not read the source and tweak, if the need ever arises). The author argues eloquently about this to some extent here, but, as one of the so-called “Twitter generation” myself, I have to say that I only started taking Shen more seriously after their website was revamped.

Wait, what do you mean? There’s no open source?

Yes, there is.

You can understand the author’s stance on open source by looking at it through the point of view of the Lisp community. There is a recurring theme of pride in using what are called professional-level tools. The community understands that such a degree of power can only be used by people who committed themselves to understanding how to use it - and the level of dedication required for that usually means “for my entire lifetime”.

Given that the Lisp arts are alien to most programmers, the level of investment necessary to use them “in the real world” is much bigger than less “weird” counterparts. So, paradoxically, Lisp respects its users’ intellectual ability, but exactly because of that, it demands more of them, while other languages might give a “quick fix” by appealing to what is easily relatable (as a famous example, JavaScript having C-style syntax, opposed to Scheme syntax as Brendan Eich originally envisioned).

And so, it remains (and probably will remain) a niche language. It gives you very, very much, but it “only” requires that you dedicate yourself completely to it. Ever read the piece about Haskell being “that girl, the special one”? It’s kinda like that.

Obligatory xkcd on this matter. It tells more than I can put into words, really. Also, I need it to finish this.

Takeaways

Shen is a cool language in the spirit of Lisp that modernizes it and is backed by cutting-edge research on programming languages. It gives very thoughtful answers to problems we, the programmers’ global tribe, are collectively facing.

The community is small, but well-meaning and full of smart people. Exactly the sort of people you want covering your back when you’re facing a difficult bug.

We still need to solve the “Lisp paradox”, but we’ll get there, one step at a time.

Resist the seductive argument of using “real world” languages and choose one that has the best of both worlds. We are the rebels. Join us!

Linking to Github directly from Emacs

GitHub has a RESTful interface for showing files in different branches and commits. In my daily tasks, it’s common to refer to files in different branches and commits for code review, so I knew I had to make this more efficient. Thus the following code was born:

(defun github/copy-file-url (curbranch)
  (interactive (list (magit-read-branch "Branch: ")))
  (let* ((toplevel (replace-regexp-in-string "\/$" "" (magit-toplevel)))
         (curbranch (or curbranch (magit-get-current-branch)))
         (pathtofile (replace-regexp-in-string (regexp-quote toplevel) "" (buffer-file-name))))
    (message
     ;; format: $REMOTE-URL/blob/$BRANCH/$PATHTOFILE
     (kill-new (format "%s/blob/%s%s#%s"
                       (replace-regexp-in-string "\.git$" "" (magit-get "remote.origin.url"))
                       curbranch
                       pathtofile
                       (mapconcat (lambda (pos) (format "L%s" (line-number-at-pos pos)))
                                  (if (region-active-p)
                                      (list (region-beginning) (region-end))
                                    (list (point))) "-"))))))

In the top let* expression, we get the current branch and the path to the current file relative to the repository’s toplevel. I usually clone via HTTP, so remote.origin.url returns an almost correct URL (we need to strip it of the trailing ‘.git’ to work). Finally, we get the line number(s - plural if there’s an active region) from Emacs and build the Github URL.

I didn’t test it, but would be happy to discover that this function also works with git-timemachine.