Continuation support in ruby

I heard today that Ruby 2.0 is going to abandon support for continuation, at least initially until things have stabilised a bit. It is no longer a news, it is just that I have not been active in Ruby community for a while.

The Problem with Continuation Support in Ruby 1.x

My initial reaction upon hearing that was "What a pity". On second thought, I realised that dropping continuation initially makes sense. The support for continuation in Ruby 1.x is missing an ensure procedure that is continuation-friendly; meaning one that is called only once one a given path is no longer accessible.

$create_cont_k=nil

def do_something(k)
  puts "Do something called"
  $create_cont_k = k
end

def do_something_else
  return unless $create_cont_k
  puts "Do something else called"
  k = $create_cont_k
  $create_cont_k = nil
  k.call
end


def create_cont
  callcc{|create_cont_k| do_something(create_cont_k)}
ensure
  puts "Ensure called"
end

create_cont
do_something_else

# /tmp $ ruby1.8  /tmp/ensure-run-multiply.rb 
# Do something called
# Ensure called
# Do something else called
# Ensure called

A proper ensure mechanism that supports explicit continuation formation system, like Ruby 1.8, would have produced:

# /tmp $ ruby1.8  /tmp/ensure-run-multiply.rb 
# Do something called
# Do something else called
# Ensure called

Notice how "ensure" is supposed to be called only once. I have a sample implementation of such continuation-observant ensure mechanism in system-managed-unwind-protect-in-sisc for SISC, a scheme implementation.

"But, but, what if I really want something like ensure that is called each time a code section is called?", you asked. In other words, you want something like Ruby 1.8's multiple-shot ensure in this new world of single-shot ensure.

That is not a problem and don't think that you are being unreasonable for asking this feature. It is useful to be able to guarantee execution of some code upon exiting a dynamic environment.

A dynamic environment is the environment that your program can access at any given time. The environment can change with each instruction executed by the machine.

But such one-sided guarantee does not really do anything interesting. How about a guarantee that some code is also executed upon entering a dynamic environment, thus having a symmetric aspect.

Scheme has dynamic-wind that does the above:

(define *CREATE-CONT-K* #f)

(define (do-something k)
  (display "Do something called\n")
  (set! *CREATE-CONT-K* k))

(define (do-something-else)
  (when *CREATE-CONT-K*
    (display "Do something else called\n")
    (let ((k *CREATE-CONT-K*))
      (set! *CREATE-CONT-K* #f)
      (k))))

(define (create-cont)
  (dynamic-wind
      (lambda () (display "Entering...\n"))
      (lambda ()
        (call/cc
         (lambda (create-cont-k)
           (do-something create-cont-k))))
      (lambda () (display "Exiting...\n"))))

(create-cont)
(do-something-else)

;; /tmp $ sisc -x /tmp/dynwind.scm 
;; Entering...
;; Do something called
;; Exiting...
;; Do something else called
;; Entering...
;; Exiting...

The utility in having a in/excursion guard can't be understated. In SRFI-34, dynamic-wind is used to install a custom exception handler. Yes, a custom exception handler; this is probably a strange concept for some people. Why would one have different exception handlers on the same piece of code? The first reason is "why not?". A dynamic language restricted to having a static exception handler feels incomplete. The second reason is more practical. Since you can in/ex-curse from/to any context (via calling a provided continuation object), how an exception is handled in a given context is not necessarily the same as in other different contexts.

Thus, a complete example of how multiple-shot and single-shot ensure example co-existing (sorry, in scheme as there is no Ruby implementation with single-shot ensure yet):

(class-path-extension-append! (list "file:/home/ysantoso/share/project/scheme/unwind-protect/"))

(require-library 'unwind-protect)
(import unwind-protect)

(import s2j)
(define (do-gc) ((generic-java-method '|gc|) (java-null (java-class '|java.lang.System|))))

;; do-something and do-something-else functions are elided for brevity

(define (create-cont)
  (dynamic-wind
      (lambda () (display "Entering...\n"))
      (lambda ()
        (unwind-protect
          (call/cc
            (lambda (create-cont-k)
              (do-something create-cont-k)))
         (display "Ensure\n")))
      (lambda () (display "Exiting...\n"))))


(create-cont)
(do-gc)
(do-something-else)
(do-gc)

;; /tmp $ sisc -x /tmp/dynwind.scm 
;; Entering...
;; Do something called
;; Exiting...
;; Do something else called
;; Entering...
;; Exiting...
;; Ensure

Notice how "ensure" is called just once even though there are two incursion into the protected code block.

The last deficiency in Ruby 1.x is the restriction that a continuation has to be resumed from the same thread that reifies it.

def do_something_else
  return unless $create_cont_k
  puts "Do something else called"
  k = $create_cont_k
  $create_cont_k = nil
  t=Thread.new { k.call}
  t.join
end

# /tmp $ ruby1.8  ensure-run-multiply.rb 
# Do something called
# Ensure called
# Do something else called
# ensure-run-multiply.rb:13:in `call': continuation called across threads (RuntimeError)
#       from ensure-run-multiply.rb:14:in `join'
#       from ensure-run-multiply.rb:14:in `do_something_else'
#       from ensure-run-multiply.rb:25

(originally from http://microjet.ath.cx/WebWiki/2006.11.05_Continuation_Support_in_Ruby.html)

perl-like lock for ruby

Authentication

Login with:New JS-Kit AccountMy Google ProfileMy Twitter AccountMy FriendFeed AccountMy Yahoo! AccountMy Blogger AccountMy JS-Kit AccountMy Haloscan AccountMy OpenID

cdfh was asking in #ruby-lang about a locking library that allows one to do:

lock(an_obj) { ..... }

similar to the lock library in perl.

"That won't not that hard to build", I thought. I did the first version in about 5 minutes. Then thought that it would be nice to make it robust to object_id wrap-around and also to discard unnecessary records associate to an_obj when an_obj has been garbage collected.

require 'thread'
require 'weakref'


class Lock
  Record = Struct.new("Record", :weakref, :mutex)

  def initialize(auto_cleanup_per_record = 1000)
    @mutex = Mutex.new
    @obj_records = {}
    @create_count = 0
    @auto_cleanup_per_record = auto_cleanup_per_record
  end

  def lock(what, &block)
    record = acquire_obj_record(what)
    record.mutex.synchronize {
      if block_given?
        block.call
      end
    }
  end
  

  private

  def cleanup_proc
    lambda {|id|
      @mutex.synchronize {
        to_delete = []
        @obj_records.each_pair{|k,v|
          if not v.weakref.weakref_alive?
            to_delete << k
          end
        }
        to_delete.each{|id|
          @obj_records.delete(id)
        }
        #puts "Cleanup id: #{to_delete.join(",")}. Size: #{@obj_records.size}: #{@obj_records.keys.join(",")}"
      }
    }
  end

  def acquire_obj_record(what)
    @mutex.synchronize {
      if record = @obj_records[what.object_id] and record.weakref.weakref_alive? 
        record
      else
        ObjectSpace.define_finalizer(what, cleanup_proc)
        @obj_records[what.object_id] = Record.new(WeakRef.new(what), Mutex.new)
      end
    }
  end

  public
  
  @instance = Lock.new
  def self.lock(what, &block)
    @instance.lock(what, &block)
  end
  def self.cleanup(verbose=false)
    @instance.cleanup(verbose)
  end
end



if $0 == __FILE__
  module Test
    def self.test1(finalizer_block)
      puts "Test1"
      s="str"*1024*1024
      ObjectSpace.define_finalizer(s, finalizer_block)
      puts s.object_id
      
      t1 = Thread.new {
        Lock.lock(s) {
          4.times {
            puts "Thread1"
            sleep(0.25)
          }
        }
      }
      
      t2 = Thread.new {
        puts "Thread2 started"
        Lock.lock(s) {
          4.times { 

            puts "Thread2"
            sleep(0.25)
          }
        }
      }
      
      t1.join
      t2.join
    end

    def self.test2(finalizer_block)
      puts "Test2"
      s="noo"*1024*1024
      puts s.object_id
      ObjectSpace.define_finalizer(s, finalizer_block)
      Lock.lock(s)
    end

    def self.test3(finalizer_block)
      puts "Test3"
      10.times {
        s="noo"*1024*1024
        puts s.object_id
        ObjectSpace.define_finalizer(s, finalizer_block)
        Lock.lock(s)
      }
    end
  end



  Test.test1(lambda {|id| puts "Finalized: #{id}"})
  GC.start
  puts "GC invoked"

  Test.test2(lambda {|id| puts "Finalized: #{id}"})
  GC.start
  puts "GC invoked"

  Test.test3(lambda {|id| puts "Finalized: #{id}"})
  GC.start
  puts "GC invoked"
end


=begin
/mnt/vg0.home/ysantoso/tmp/ruby-postgres/tests $ ruby /tmp/lock.rb 
Test1
-604990708
Thread1
Thread2 started
Thread1
Thread1
Thread1
Thread2
Thread2
Thread2
Thread2
GC invoked
Test2
-610787788
GC invoked
Test3
-610788148
-610953348
-615730956
-615809476
-615887996
Finalized: -615730956
Finalized: -610953348
-615902566
-615981086
-615697446
Finalized: -615809476
Finalized: -615887996
Finalized: -615902566
-614121002
-614199522
Finalized: -614121002
Cleanup id: -615809476,-615730956,-610953348,-615902566,-615887996. Size: 7: -614199522,-615697446,-615981086,-610787788,-614121002,-610788148,-604990708
Finalized: -614199522
Cleanup id: -614121002. Size: 6: -614199522,-615697446,-615981086,-610787788,-610788148,-604990708
Finalized: -615697446
Cleanup id: -614199522. Size: 5: -615697446,-615981086,-610787788,-610788148,-604990708
Finalized: -615981086
Cleanup id: -615697446. Size: 4: -615981086,-610787788,-610788148,-604990708
Finalized: -610787788
Cleanup id: -615981086. Size: 3: -610787788,-610788148,-604990708
Finalized: -610788148
Cleanup id: -610787788. Size: 2: -610788148,-604990708
GC invoked
Finalized: -604990708

=end 

(originally from http://microjet.ath.cx/WebWiki/2006.08.07_LockLibraryForRuby.html)

Ruby's hash implementation

Authentication

Login with:New JS-Kit AccountMy Google ProfileMy Twitter AccountMy FriendFeed AccountMy Yahoo! AccountMy Blogger AccountMy JS-Kit AccountMy Haloscan AccountMy OpenID

Ruby's Hash class is implemented using the same engine that it uses for symbol table.

It is not meant for high-volume usage.

/tmp $ ruby testspeed.rb 
Rehearsal ----------------------------------------------------------------------------------------------
reading with while, name=1million                            1.240000   0.030000   1.270000 (  1.272282)
reading with readline, name=1million                         0.980000   0.210000   1.190000 (  1.215839)
reading with while and inserting into hash, name=1million    5.330000   0.200000   5.530000 (  5.715192)
reading with while and inserting into array, name=1million   5.640000   0.210000   5.850000 (  5.975710)
------------------------------------------------------------------------------------ total: 13.840000sec

                                                                 user     system      total        real
reading with while, name=1million                            1.750000   0.020000   1.770000 (  1.785138)
reading with readline, name=1million                         1.440000   0.010000   1.450000 (  1.454656)
reading with while and inserting into hash, name=1million    4.050000   0.020000   4.070000 (  4.102691)
reading with while and inserting into array, name=1million   2.290000   0.020000   2.310000 (  2.320377)

def create_file(name, size)
  File.open("/tmp/largefile_#{name}", "w") {|f| size.times {|i|f.puts "foo#{i}"; } }
end

# do these once
# create_file("1million", 1*1000*1000)
# create_file("5million", 5*1000*1000)

def read_with_while(name)
  File.open("/tmp/largefile_#{name}") {|fh|
    while line = fh.gets
      line.chomp!
    end
  }
end

def read_with_readlines(name)
  File.readlines("/tmp/largefile_#{name}")
end

def read_into_hash(name)
  hash={}; array=[]; File.open("/tmp/largefile_#{name}"){ |fh| while line = fh.gets; line.chomp!; 
                                                                 hash[line] = 1; 
                                                               end} 
end
def read_into_array(name)
  array=[]; File.open("/tmp/largefile_#{name}"){ |fh| while line = fh.gets; line.chomp!; 
                                                                 array << line
                                                               end} 
end

require 'benchmark'
Benchmark.bmbm {|r|
  ["1million"].each{|name|
    GC.start
    r.report("reading with while, name=#{name}") {read_with_while(name)}
    GC.start
    r.report("reading with readline, name=#{name}") {read_with_readlines(name)}
    GC.start
    r.report("reading with while and inserting into hash, name=#{name}") { read_into_hash(name)}
    GC.start
    r.report("reading with while and inserting into array, name=#{name}") { read_into_array(name)}
  }
} 

(originally from http://microjet.ath.cx/WebWiki/2006.04.15_Ruby%27sHash.html)

Using symbols for the wrong reason

Authentication

Login with:New JS-Kit AccountMy Google ProfileMy Twitter AccountMy FriendFeed AccountMy Yahoo! AccountMy Blogger AccountMy JS-Kit AccountMy Haloscan AccountMy OpenID

The concept of symbols have been popular among the lisp community, yet not many people know about it mainly due to the general ignorance.
Then ruby came and made symbols be a commodity programming construct. People who were not aware of symbols now are.

And they are asking about it numerous times. There is not a week in the ruby mailing list that there isn't a question about symbols: what are they and what are they good for?

Many people have tried to answer that, but the answer has been along the lines presented in these two articles: http://glu.ttono.us/articles/2005/08/19/understanding-ruby-symbols and http://zephyrfalcon.org/weblog2/arch_e10_00850.html#e857.

The answer has been putting undue emphasise on the way current ruby VM implements symbols. Ruby string is mutable, and it is not efficiently implemented in current ruby VM. So, use symbols for efficient, immutable, and string-like objects.

It is not wrong and it is correct for current ruby VM. However, I think, that is a misguided answer to the questions. The answer should, on the other hand, put an emphasise on the programmer's intention.
rubyists uses #each() method more frequently than a for loop because it clarifies their intention of iterating over some sequence even though they could have used the more efficient for loop or even the if and goto constructs.

One does not tell another to use if and goto over for loop for iterating a sequence simply because if and goto may be more efficient. No matter how inefficient a compiler/interpreter implements the for loop construct, the possibility of an efficient for loop implementation remains. In fact, by using the for construct, the compiler could have an easier time deducing your intent of looping, and if some conditions are met (e.g., closed looping of certain numbers of times), it could unroll your loop for a better performance if your systems allows it.

In short, any answer that depends on a particular implementation is doomed to be short-lived. What happens if the next ruby VM implements COW (copy-on-write) strings? A COW string would share initial instances. Only f there is a modification to the instance, then the initial instance is copied and the modification is performed on the copy. As long as one does not try to modify COW string instance, it can be as efficient as how the current VM implements symbols. IOW, any answer that rallies around so-called efficiency while abandoning intent would become obsolete and there is a new scramble to get at an updated answer.

Thus, I finally come to say that one should not use symbols just for efficiency gain. Symbols are not meant to be an immutable string-like object. It is really meant to be used to construct user-defined identifiers. The user in this case would be the programmers.

Consider:

foo1 = { 
   :host => 'localhost',
   :port => 80
}
foo2 = {
   'host' => 'localhost',
   'port' => 80

In foo1, symbols are being used to identify the following data. The string 'localhost' is identified as a host, and 80 is not just any number, but rather a port number.

In foo2, it is a bit unclear as to what purpose 'host' and 'port' serves. Is foo2 a macro replacement list? That is, if the program reads the string 'host', would it be replaced to 'localhost'? What is the purpose of 'host' there? Is it an identifier for the string 'localhost'?

The programming world should borrow the real estate's adage of "location, location, location". It should be translated to: "intention, intention, intention". It is the main reason why comments that clarifies the intention of the programmer are so valuable. It is the main reason why there are a variety language constructs. It should also be the main reason for you to decide whether or not to use symbols.

2005.12.28 update: I am joyful that not everyone resorted to dumbing down the concept of symbols. http://onestepback.org/index.cgi/Tech/Ruby/SymbolsAreNotImmutableStrings.red
2006.01.06 update: What an amazing interest on symbol! I don't think I've seen any one topic in ruby that has generated 142 posts in a single thread before this.
http://groups.google.com/group/comp.lang.ruby/browse_frm/thread/164ae5f5cbbac02e?q=differences+between+%3Afoo&hl=en&
2007.07.03 update: a related article: 2007.07.03_WhatAreSymbols

(originally from http://microjet.ath.cx/WebWiki/2005.12.27_UsingSymbolsForTheWrongReason.html)

Connection from ruby to MS SQL Server

Authentication

Login with:New JS-Kit AccountMy Google ProfileMy Twitter AccountMy FriendFeed AccountMy Yahoo! AccountMy Blogger AccountMy JS-Kit AccountMy Haloscan AccountMy OpenID

Quick setup guide for: debian, iodbc, ruby, mssql2k

apt-get install libdbi-ruby libdbd-odbc-ruby freetds-dev odbcinst1 iodbc

Installing freetds-dev and odbcinst1 should cause apt-get to offer you a choice of having freetds managed by odbcinst. Say yes. That will create a file /etc/odbcinst.ini

ysantoso@helen:~$ cat /etc/odbcinst.ini
[FreeTDS]
Description     = TDS driver (Sybase/MS SQL)
Driver          = /usr/lib/odbc/libtdsodbc.so
Setup           = /usr/lib/odbc/libtdsS.so
CPTimeout       =
CPReuse         =
FileUsage       = 1

Next thing to do is to setup ~/.freetds.conf. I started with the template at /etc/freetds/freetds.conf.

ysantoso@helen:~$ cat ~/.freetds.conf
[global]
        # Default TDS protocol version. 
        tds version = 4.2

        initial block size = 512

        swap broken dates = no

        swap broken money = no

        # Database server login method, if both server and domain
        # logins are enabled, domain login is tried first if a domain
        # is specified, and if that fails the server login will be
        # used.
        try server login = yes
        try domain login = no

        # Whether to write a TDSDUMP file for diagnostic purposes
        # (setting this to /tmp is insecure on a multi-user system)
;       dump file = /tmp/freetds.log
;       debug level = 10

        # If you get out of memory errors, it may mean that your
          client
        # is trying to allocate a huge buffer for a TEXT field.
        # (Microsoft servers sometimes pretend TEXT columns are
        # 4 GB wide!)   If you have this problem, try setting
        # 'text size' to a more reasonable limit
        text size = 64512


[FooServer]
host            =       fooserver.example.com
port            =       1433
tds version     =       7.0

Then we setup ~/.odbc.ini

ysantoso@helen:~$ cat ~/.odbc.ini
[ODBC Data Sources]
FooDSN = description about FooDSN

[FooDSN]
Driver          = /usr/lib/odbc/libtdsodbc.so
ServerName      = FooServer
Database        = Bar

Next, we test:

ysantoso@helen:~$ iodbctest
iODBC Demonstration program
This program shows an interactive SQL processor
Driver Manager: 03.52.0205.0204

Enter ODBC connect string (? shows list): ?

DSN                              | Driver
------------------------------------------------------------------------------
FooDSN                           | FooDSN

Enter ODBC connect string (? shows list):
DSN=FooDSN;UID=sa;PWD=lookmanopassword

SQL>select * from tablebaz;
.
.
.
.

Alright, ODBC is setup. Let's try connecting from Ruby. There are two ways: one is to use the ODBC driver directly or using DBI.

I am not familiar with the ODBC driver's API. Looks similar to DBI, but I'm sure it differs in places. So, I just did a quick trial at it just to ensure that the DSN is seen.

irb(main):002:0> require 'odbc'
true
irb(main):005:0> ODBC.datasources
[#]

irb(main):006:0> require 'dbi'
true
irb(main):008:0> dbh=DBI.connect('dbi:ODBC:FooDSN', 'sa', 'lookmanopassword')
#, @attr={}>, @trace_output=#, @trace_mode=\ 2>
irb(main):015:0> dbh.select_one('select count(*) from websession')
[2495950] 

(originally from http://microjet.ath.cx/WebWiki/2005.11.03_Connecting_From_Ruby_To_MSSQL.html)

Ruby on Debian

Authentication

Login with:New JS-Kit AccountMy Google ProfileMy Twitter AccountMy FriendFeed AccountMy Yahoo! AccountMy Blogger AccountMy JS-Kit AccountMy Haloscan AccountMy OpenID

In Debian Woody and Sarge (previous stable and current stable trees as of today), the ruby packages are split up into many packages. This has caused problems to Debian users not having enough dpkg-foo skill, which are many.

The situation is corrected in Debian unstable, but it would be a while before the correction trickles down to testing, and even longer to stable.

In the meantime, you can do this instead:

apt-get install grep-dctrl    # gives you grep-availablea
apt-get install `grep-available -ns Package -F Source -X ruby-defaults`
pt-get install `grep-available -ns Package -F Source -X ruby1.8`
apt-get install libopenssl-ruby

which would install all ruby packages produced from the official ruby1.8 source tarball.

~ $ grep-available -ns Package -F Source -X ruby-defaults
libgdbm-ruby
libruby
libtcltk-ruby
libiconv-ruby
rdoc
libcurses-ruby
libsyslog-ruby
libsdbm-ruby
libreadline-ruby
ri
libdbm-ruby
libxmlrpc-ruby
irb
ruby
libyaml-ruby
libpty-ruby
libtk-ruby
libtest-unit-ruby
libdl-ruby
ruby-elisp

~$ grep-available -ns Package -F Source -X ruby1.8
ruby1.8-elisp
libopenssl-ruby1.8
ri1.8
ruby1.8-examples
libdbm-ruby1.8
libreadline-ruby1.8
libruby1.8
libgdbm-ruby1.8
libruby1.8-dbg
irb1.8
libtcltk-ruby1.8
rdoc1.8
ruby1.8-dev

Total packages: 23 packages.

~ $ grep-available -ns Package -F Source -X ruby1.8|wc -l
13

~ $ grep-available -ns Package -F Source -X ruby-defaults|wc -l
20 

(originally from http://microjet.ath.cx/WebWiki/RubyOnDebian.html)