| How I built libyaml on Windows with MSVC++ |
[Mar. 17th, 2012|09:57 pm] |
So, my foray into building libyaml on Windows. I don't know if this is the right solution, but it worked for me. Anyway, back to the story.
The rubygems library, or at least the gem command line tool, requires a yaml parser to be installed in order to work properly. That means either Syck or Psych. In Ruby 1.9 it wants Pysch.
If you're using the Ruby Installer this is no big deal. It's already built for you and the gem command just works. If you want to build it using the Microsoft toolchain, things get a little trickier.
First, grab and unzip + untar libyaml. In Visual Studio, which I'm assuming you have installed, go to File -> Open -> Project/Solution and selection libyaml-x.y.z\win32\vs2008\libyaml.sln (or just libyaml if it doesn't show file extensions). From there you'll need to build the yamldll project. Ignore the rest, just right click on that one and select "build solution". If all goes well, you should have a .lib and .dll file under libyaml-x.y.z\win32\vs2008\Output\Debug\lib\DLL.
Then, copy the yaml.lib and yaml.dll files in your favorite location under the 'lib' directory. I put mine in c:\usr\yaml\lib. Then copy the yaml.h file (found under libyaml-x.y.z\include) in the same location, but in an 'include' directory. I put mine in c:\usr\yaml\include. I also had to copy the yaml.dll file to the same directory as the Ruby executable to get things to work, but I'm not sure why.
From there you should be able to build Psych. Go to wherever you unpacked the Ruby source, and cd to ext/psych. Then run "ruby extconf.rb --with-libyaml-dir=c:\usr\yaml; nmake; nmake install."
After I installed zlib (using pr-zlib instead), gem install worked just fine. :) |
|
|
| More FFI |
[Dec. 14th, 2011|10:30 am] |
Lately I've converted a few of my smaller Ruby libraries that were using C extensions to use the ffi gem. Specifically, sys-cpu, sys-uname and sys-uptime now all use FFI behind the scenes. They all now work with JRuby, too. Yay.
I'm still not enamored with FFI, though. It has added some features for automatic struct allocation which is good, but it still has most of the issues I complained about originally. In fact, another issue I didn't mention last time is that there's no way to automatically import simple macro definitions from header files. This again means manually inspecting header files if you want to use sensical constant names instead of raw numbers. To make matters worse, sometimes different platforms have the same macro with different values. To wit HW_MACHINE_ARCH is set to 11 on BSD, but 12 on other platforms. Fun.
And, converting those three gems to use FFI took much, much longer than it should have. It's still much easier for me to use C extensions. So, my more complex C extensions will not be converted any time soon, and probably only if I'm actually paid to do it. |
|
|
| Ruby, Solaris, Compiler Flags |
[Jul. 30th, 2011|12:30 am] |
After finally getting my Solaris 10 VM reinstalled, along with a copy of Sun Studio Compiler 12, I decided to do some benchmarking. These are the results of some Array benchmarks that are included in the "berger_spec" project, using Ruby 1.8.7-p352. The first set of results are with no compiler flags set. The second set are the results using "-g -fast -xipo -xrestrict -xpagesize=2M".
With no flags set:
Array[] 0.310000 0.000000 0.310000 ( 0.314656)
Array.new(int) 0.240000 0.000000 0.240000 ( 0.238028)
Array.new(int, obj) 0.300000 0.000000 0.300000 ( 0.300978)
Array.new(array) 0.370000 0.000000 0.370000 ( 0.370444)
Array.new(size){ block } 0.490000 0.000000 0.490000 ( 0.487876)
Array#& 0.460000 0.000000 0.460000 ( 0.467969)
Array#* (int) 0.310000 0.010000 0.320000 ( 0.305176)
Array#* (join) 0.970000 0.000000 0.970000 ( 0.972113)
Array#- 0.520000 0.000000 0.520000 ( 0.526612)
Array#<< 0.120000 0.010000 0.130000 ( 0.137587)
Array#<=> 0.280000 0.000000 0.280000 ( 0.282714)
Array#== 0.320000 0.000000 0.320000 ( 0.320758)
Array#[] 0.070000 0.000000 0.070000 ( 0.067216)
Array#[]= 0.080000 0.000000 0.080000 ( 0.075855)
Array#| 0.570000 0.010000 0.580000 ( 0.591650)
Array#assoc 0.130000 0.000000 0.130000 ( 0.130801)
Array#at 0.060000 0.000000 0.060000 ( 0.065698)
Array#clear 0.070000 0.000000 0.070000 ( 0.065757)
Array#collect 0.400000 0.000000 0.400000 ( 0.404131)
Array#collect! 0.330000 0.000000 0.330000 ( 0.331329)
Array#compact 0.150000 0.000000 0.150000 ( 0.152455)
Array#compact! 0.060000 0.000000 0.060000 ( 0.062184)
Array#concat 0.130000 0.010000 0.140000 ( 0.139257)
Array#delete(obj) 0.250000 0.000000 0.250000 ( 0.248629)
Array#delete(obj){ block } 0.310000 0.000000 0.310000 ( 0.308989)
Array#delete_at 0.070000 0.000000 0.070000 ( 0.067850)
Array#delete_if 0.360000 0.000000 0.360000 ( 0.361809)
Array#each 0.310000 0.000000 0.310000 ( 0.317559)
Array#each_index 0.390000 0.000000 0.390000 ( 0.384572)
Array#empty? 0.060000 0.000000 0.060000 ( 0.058954)
Array#eql? 0.260000 0.000000 0.260000 ( 0.262421)
Array#fetch(index) 0.070000 0.000000 0.070000 ( 0.073590)
Array#fetch(index, default) 0.110000 0.000000 0.110000 ( 0.109604)
Array#fetch(index){ block } 0.160000 0.000000 0.160000 ( 0.155640)
Array#fill(obj) 0.110000 0.000000 0.110000 ( 0.115837)
Array#fill(obj, start) 0.130000 0.000000 0.130000 ( 0.129365)
Array#fill(obj, start, length) 0.120000 0.000000 0.120000 ( 0.119590)
Array#fill(obj, range) 0.160000 0.000000 0.160000 ( 0.160754)
Array#fill{ block } 0.250000 0.000000 0.250000 ( 0.247193)
Array#fill(start){ block } 0.230000 0.000000 0.230000 ( 0.232438)
Array#fill(start, length){ block } 0.140000 0.000000 0.140000 ( 0.144361)
Array#fill(range){ block } 0.230000 0.000000 0.230000 ( 0.227871)
Array#first 0.060000 0.000000 0.060000 ( 0.056156)
Array#flatten 1.000000 0.050000 1.050000 ( 1.044090)
Array#flatten! 0.810000 0.000000 0.810000 ( 0.812008)
Array#include? 0.290000 0.000000 0.290000 ( 0.288412)
Array#index 0.480000 0.000000 0.480000 ( 0.485483)
Array#insert 34.160000 0.040000 34.200000 ( 34.253063)
Array#join 0.950000 0.060000 1.010000 ( 1.017708)
Array#last 0.070000 0.000000 0.070000 ( 0.060686)
Array#length 0.050000 0.000000 0.050000 ( 0.058534)
Array#nitems 0.060000 0.000000 0.060000 ( 0.060758)
Array#pack 0.280000 0.070000 0.350000 ( 0.346961)
Array#pop 0.060000 0.000000 0.060000 ( 0.061161)
Array#push 0.270000 0.010000 0.280000 ( 0.280934)
Array#rassoc 0.150000 0.000000 0.150000 ( 0.152078)
Array#reject 0.380000 0.000000 0.380000 ( 0.377071)
Array#reject! 0.310000 0.000000 0.310000 ( 0.315611)
Array#replace 0.220000 0.000000 0.220000 ( 0.212419)
Array#reverse 0.110000 0.000000 0.110000 ( 0.118272)
Array#reverse! 0.070000 0.000000 0.070000 ( 0.065353)
Array#reverse_each 0.310000 0.000000 0.310000 ( 0.307441)
Array#rindex 0.070000 0.000000 0.070000 ( 0.070481)
Array#shift 0.060000 0.000000 0.060000 ( 0.061049)
Array#slice(int) 0.060000 0.000000 0.060000 ( 0.064987)
Array#slice(start, length) 0.100000 0.000000 0.100000 ( 0.098006)
Array#slice(range) 0.190000 0.000000 0.190000 ( 0.184972)
Array#slice!(int)! 0.070000 0.000000 0.070000 ( 0.072685)
Array#slice!(start, length) 0.120000 0.000000 0.120000 ( 0.125541)
Array#slice!(range) 0.160000 0.000000 0.160000 ( 0.157826)
Array#sort 0.150000 0.000000 0.150000 ( 0.143664)
Array#sort{ block } 1.200000 0.000000 1.200000 ( 1.212228)
Array#sort! 0.090000 0.000000 0.090000 ( 0.083815)
Array#sort!{ block } 0.880000 0.000000 0.880000 ( 0.887263)
Array#to_a 0.070000 0.000000 0.070000 ( 0.063464)
Array#to_ary 0.050000 0.000000 0.050000 ( 0.057507)
Array#to_s 1.010000 0.000000 1.010000 ( 1.011248)
Array#transpose 0.320000 0.000000 0.320000 ( 0.316514)
Array#uniq 0.540000 0.000000 0.540000 ( 0.543465)
Array#uniq! 0.350000 0.000000 0.350000 ( 0.345773)
Array#unshift 0.050000 0.000000 0.050000 ( 0.057270)
Array#values_at(int) 0.200000 0.000000 0.200000 ( 0.192256)
Array#values_at(range) 0.250000 0.000000 0.250000 ( 0.253189)
With compiler flags -g -fast -xipo -xrestrict -xpagesize=2M:
Array[] 0.170000 0.000000 0.170000 ( 0.172121)
Array.new(int) 0.140000 0.000000 0.140000 ( 0.135445)
Array.new(int, obj) 0.170000 0.000000 0.170000 ( 0.166088)
Array.new(array) 0.190000 0.000000 0.190000 ( 0.197206)
Array.new(size){ block } 0.240000 0.000000 0.240000 ( 0.237472)
Array#& 0.310000 0.000000 0.310000 ( 0.310026)
Array#* (int) 0.210000 0.010000 0.220000 ( 0.217617)
Array#* (join) 0.520000 0.000000 0.520000 ( 0.525306)
Array#- 0.340000 0.000000 0.340000 ( 0.340506)
Array#<< 0.060000 0.050000 0.110000 ( 0.109793)
Array#<=> 0.170000 0.000000 0.170000 ( 0.162905)
Array#== 0.200000 0.050000 0.250000 ( 0.253786)
Array#[] 0.030000 0.000000 0.030000 ( 0.037419)
Array#[]= 0.050000 0.000000 0.050000 ( 0.042840)
Array#| 0.390000 0.050000 0.440000 ( 0.439489)
Array#assoc 0.070000 0.000000 0.070000 ( 0.064447)
Array#at 0.030000 0.000000 0.030000 ( 0.036351)
Array#clear 0.030000 0.000000 0.030000 ( 0.031723)
Array#collect 0.170000 0.000000 0.170000 ( 0.163739)
Array#collect! 0.130000 0.000000 0.130000 ( 0.131613)
Array#compact 0.080000 0.010000 0.090000 ( 0.085052)
Array#compact! 0.030000 0.000000 0.030000 ( 0.033368)
Array#concat 0.090000 0.020000 0.110000 ( 0.111146)
Array#delete(obj) 0.130000 0.000000 0.130000 ( 0.129340)
Array#delete(obj){ block } 0.170000 0.000000 0.170000 ( 0.170104)
Array#delete_at 0.040000 0.000000 0.040000 ( 0.036985)
Array#delete_if 0.180000 0.000000 0.180000 ( 0.184568)
Array#each 0.120000 0.000000 0.120000 ( 0.115847)
Array#each_index 0.130000 0.000000 0.130000 ( 0.133975)
Array#empty? 0.030000 0.000000 0.030000 ( 0.030337)
Array#eql? 0.160000 0.000000 0.160000 ( 0.156354)
Array#fetch(index) 0.040000 0.000000 0.040000 ( 0.040687)
Array#fetch(index, default) 0.050000 0.000000 0.050000 ( 0.054539)
Array#fetch(index){ block } 0.070000 0.000000 0.070000 ( 0.074566)
Array#fill(obj) 0.050000 0.000000 0.050000 ( 0.053080)
Array#fill(obj, start) 0.070000 0.000000 0.070000 ( 0.064187)
Array#fill(obj, start, length) 0.060000 0.000000 0.060000 ( 0.060635)
Array#fill(obj, range) 0.090000 0.000000 0.090000 ( 0.087738)
Array#fill{ block } 0.130000 0.000000 0.130000 ( 0.130971)
Array#fill(start){ block } 0.120000 0.000000 0.120000 ( 0.122547)
Array#fill(start, length){ block } 0.080000 0.000000 0.080000 ( 0.083082)
Array#fill(range){ block } 0.130000 0.000000 0.130000 ( 0.129588)
Array#first 0.030000 0.000000 0.030000 ( 0.031762)
Array#flatten 0.610000 0.060000 0.670000 ( 0.671874)
Array#flatten! 0.480000 0.000000 0.480000 ( 0.476308)
Array#include? 0.160000 0.000000 0.160000 ( 0.163326)
Array#index 0.260000 0.000000 0.260000 ( 0.264573)
Array#insert 30.930000 0.020000 30.950000 ( 30.960786)
Array#join 0.560000 0.260000 0.820000 ( 0.832717)
Array#last 0.030000 0.000000 0.030000 ( 0.030846)
Array#length 0.030000 0.000000 0.030000 ( 0.030950)
Array#nitems 0.030000 0.000000 0.030000 ( 0.032106)
Array#pack 0.250000 0.230000 0.480000 ( 0.475462)
Array#pop 0.030000 0.000000 0.030000 ( 0.031812)
Array#push 0.150000 0.160000 0.310000 ( 0.296990)
Array#rassoc 0.080000 0.000000 0.080000 ( 0.082723)
Array#reject 0.180000 0.000000 0.180000 ( 0.182116)
Array#reject! 0.120000 0.000000 0.120000 ( 0.116769)
Array#replace 0.090000 0.000000 0.090000 ( 0.092382)
Array#reverse 0.060000 0.000000 0.060000 ( 0.061858)
Array#reverse! 0.040000 0.000000 0.040000 ( 0.032183)
Array#reverse_each 0.110000 0.000000 0.110000 ( 0.113048)
Array#rindex 0.040000 0.000000 0.040000 ( 0.039673)
Array#shift 0.030000 0.000000 0.030000 ( 0.033855)
Array#slice(int) 0.040000 0.000000 0.040000 ( 0.036325)
Array#slice(start, length) 0.090000 0.000000 0.090000 ( 0.091806)
Array#slice(range) 0.070000 0.000000 0.070000 ( 0.072233)
Array#slice!(int)! 0.040000 0.000000 0.040000 ( 0.041699)
Array#slice!(start, length) 0.070000 0.000000 0.070000 ( 0.065852)
Array#slice!(range) 0.090000 0.000000 0.090000 ( 0.087642)
Array#sort 0.080000 0.000000 0.080000 ( 0.086419)
Array#sort{ block } 0.560000 0.000000 0.560000 ( 0.559803)
Array#sort! 0.050000 0.000000 0.050000 ( 0.048642)
Array#sort!{ block } 0.470000 0.000000 0.470000 ( 0.463398)
Array#to_a 0.030000 0.000000 0.030000 ( 0.033682)
Array#to_ary 0.030000 0.000000 0.030000 ( 0.032244)
Array#to_s 0.560000 0.350000 0.910000 ( 0.911277)
Array#transpose 0.260000 0.000000 0.260000 ( 0.258392)
Array#uniq 0.370000 0.000000 0.370000 ( 0.364935)
Array#uniq! 0.160000 0.000000 0.160000 ( 0.160942)
Array#unshift 0.030000 0.000000 0.030000 ( 0.030089)
Array#values_at(int) 0.060000 0.000000 0.060000 ( 0.066309)
Array#values_at(range) 0.300000 0.010000 0.310000 ( 0.310101)
So, generally speaking, some nice improvements. :) |
|
|
| ActiveRecord::Base.all (without conditions) Considered Harmful |
[Feb. 16th, 2011|02:35 pm] |
We've pretty much reached the conclusion that ActiveRecord::Base.all, or .find(:all) if you prefer, should never be used without conditionals. If you want to iterate over every record in the table, you should use my custom .foreach method.
Some recent site flakiness inspired me to do some comparisons of a .all vs a .foreach. The details are below. For the lazy, the short version is that a .all call will suck down rather large amounts of RAM and CPU, growing larger as the number of records grows. The .foreach call consumes about the same amount of CPU, but with low and steady RAM usage.
Table: Teacher
Overview:
61572 records.
28 columns, mostly varchar (character) fields of with
a size of about 25, with one field at 4000 characters, but which is usually empty.
Specs:
Ubuntu 10.04 (lucid)
Kernel Linux 2.6.32-28-generic
Memory: 1.9 GB
2 CPU, dual core 3GHz Intel
Ruby 1.8.7-p330
Rails 3.0.4
=> Teacher.all
Memory: 163mb
%62.5 CPU
%7.61 MEM.
Total time: About 30 seconds.
=> Teacher.foreach
Memory: 41mb
%68.34 CPU
%1.42 MEM
Total time: About 38 seconds.
The figures I'm showing are what the process finally capped out at. The .all call starts off at 41mb and climbs until it finishes the process, while the .foreach call sits steady at 41mb, though its CPU usage climbs about as much as the .all call.
This, of course, will only get worse on tables with larger datasets, and will absolutely hammer your server. Frankly, I think the find_each implementation ought to be the default implementation, though last I checked CPK broke it. |
|
|
| Specification files |
[Dec. 19th, 2010|08:26 am] |
Why do people overthink gem specification files? Here's an example:
s.files = Dir.glob(%w[{lib,test}/**/*.rb bin/* [A-Z]*.{txt,rdoc} ext/**/*.{rb,c} **/deps.rip]) + %w{Rakefile .gemspec}
s.files += Dir.glob('man/*')
Here's what I do for all of my gems:
Dir['**/*'].reject{ |f| f.include?('git') }
THE END. |
|
|
| My stock clean task |
[Dec. 4th, 2010|06:15 am] |
Anyone got a better one?
require 'rake/clean'
require 'rbconfig'
include Config
CLEAN.include(
'**/*.gem', # Gem files
'**/*.rbc', # Rubinius
'**/*.o', # C object file
'**/*.log', # Ruby extension build log
'**/Makefile', # C Makefile
'**/conftest.dSYM', # OS X build directory
"**/*.#{CONFIG['DLEXT']}" # C shared object
)
|
|
|
| Fast Forward |
[Nov. 21st, 2010|11:01 am] |
| [ | Tags | | | poetry | ] |
| [ | mood |
| | creative | ] |
Here's some sci-fi poetry that popped into my head this morning. Inspired by the movie Skyline.
Fast Forward.
Fast forward
A cure for cancer is found.
People live longer.
Humanity rejoices.
But we want more.
People can still die.
This is unacceptable.
Fast forward
The Hayflick limit is broken.
People live longer.
Humanity rejoices.
But we want more.
People can still die.
This is unacceptable.
Fast forward
DNA recombination eliminates aging entirely.
Humanity rejoices.
People are effectively immortal.
But we want more.
People are not invulnerable.
This is unnacceptable.
Fast forward
Advances in bioengineering allow people to shed their mortal bodies.
There is rejoicing.
Bodies are more durable.
But we want more.
Our bodies are not invulnerable.
This is unnacceptable.
Fast forward
Regenerative biotechnology virtually eliminates all vulnerabilities.
There is contentment.
Bodies are effectively invincible.
But there are too many of us now.
This is unnacceptable.
Fast forward
Exploration and colonization of space occurs.
There is satisfaction.
But there aren't enough of us any more.
We need fresh minds and resources to create and run new bodies.
This is unnacceptable.
Fast forward
Exploration of space reveals other worlds with intelligent species with mortal bodies.
This is unnacceptable.
We want more. |
|
|
| Git Project Hell |
[Nov. 2nd, 2010|01:41 pm] |
We're in a bad situation with the composite_primary_keys library. Just take a look at its github network.
For those of you not familiar with ActiveRecord it's the default ORM for Rails. Unfortunately, it doesn't support composite primary keys out of the box. That hoses Oracle users, who must use this library to make AR work properly with Oracle for any table with more than 1 column used for a primary key.
Back to the github project. In short, 46 forks, including forks of forks, and at least 4 independently published gems, all in various states. Few of the fork owners seem to communicating with each other.
So, what the frack are we supposed to do now? This is the sort of problem you don't see in the wild with centralized VCS' because either everyone is committing to a central project (instead of forking it). Or, in the extreme case, one guy will create a separate project (effectively forking it once), and programmers will then commit to that project going forward. Our choices are to either pick one fork and hope for the best, or fork the project ourselves, and devote resources to maintaining it.
TOTAL SUCK.
This is an awful situation to be in, especially from a business perspective. In short, we're seeing the failings of the whole "social programming" paradigm IMO. Sometimes you need a dictator to steer the project, and crush the opposition. Instead, we've got a bunch of senators all vying for power and there's no easy way to tell which one is the future Augustus. |
|
|
| Achieving C99 compliance via C++ |
[Oct. 2nd, 2010|10:59 am] |
I thought it was a well understood joke that achieving C99 compliance requires a C++ compiler. A recent entry on reddit (which I can't find right off hand) bemoaned the fact that with Visual C++ you can't compile this because of the inline type declaration:
// for.c
#include <stdio.h>
int main(){
for(int i = 0; i < 10; i++)
printf("I is: %i\n", i);
return 0;
}
With a standard C compiler (including gcc 3, by the way), this will fail to compile. So, how do we make it work? By passing the -Tp switch, which builds your program using C++ instead of C:
cl /TP for.c
Ta-da! |
|
|
| navigation |
| [ |
viewing |
| |
most recent entries |
] |
| [ |
go |
| |
earlier |
] |
| |
|
|
![[info]](http://l-stat.livejournal.com/img/userinfo.gif?v=92.1){kind=small}
djberg96's journal
calm
confused
blah