Dump csv reports using Ruby on Rails and Postgres
This method is the most efficient, super simple way to generate CSV files with Rails. It uses an active record connection and the Postgres COPY command writes each csv generated line from Postgres to a file.
File.open(file_path, 'w') do |f|
ActiveRecord::Base.connection.raw_connection.copy_data "COPY (SELECT * FROM users) TO STDOUT WITH (FORMAT CSV, HEADER TRUE, FORCE_QUOTE *);" do
while line = connection.raw_connection.get_copy_data do
f.write line.force_encoding('UTF-8')
end
end
end
Leveraging Postgres to do this, results in tens of millions of rows being able to be exported in under 5 mins.
Support SVG's and PNG/JPG in your Rails Dragonfly workflow
I was struggling to find a solution to easily provide thumb pngs, as well as display svgs inline with the img tag with the ruby gem Dragonfly.
For example something simple like this:
= image_tag user.image.thumb('100x100#').url
But that breaks when showing SVG files, as it looks like it strips out a bunch of important info from the file.
To set the dimensions of the svg correctly. You can add an additional gem, gem 'dragonfly_svg' and you'll also need to add that as a plugin to your dragonfly.rb file.
require 'dragonfly'
# Configure
Dragonfly.app.configure do
plugin :imagemagick
plugin :svg
...
To render the SVG correctly you'd write this:
= image_tag user.image.set_dimensions(100,100).url
So, in order to support both seamlessly. I built a custom Dragonfly processor.
Then you can setup a custom processor, that checks the mime type of the image, and processes it either way.
processor :custom_thumb do |content, *args|
if content.mime_type == 'image/svg+xml'
dimensions = args.first.split('x')
content.process!(:set_dimensions, dimensions[0].to_i, dimensions[1].to_i)
else
content.process!(:thumb, *args)
end
end
Then, throughout the code, you can easily refer to the custom_thumb processor:
= image_tag user.image.custom_thumb('100x100#').url
Removing sensitive data from GitHub wiki revision history
If you, like me, by mistake added a password to a Github Wiki, and would like to remove it from history. It's actually quite easy.
The whole wiki, is actually a git repo, which you can checkout locally like this:
git clone https://github.com/<user>/<repo>.wiki.git
You can then do a rebase, to amend the the wiki changes (commits). Remove the sensitive commits, and then do a force push.
git rebase --interactive HEAD~2
git push -f
Please note: that even though you won't be able to see it on the UI, the sensitive data will still exist in the git history, if you would like to be sure that nothing exists, you can destroy the whole wiki's history, by following this gist.
Playing with Googles Cloud Vision API
The age of intelligent api's is here.
If you want to get a taste of what Google's new "Cloud Vision API" can do. Sign up here https://cloud.google.com/vision/.
Once you're accepted, get an api key from the Google console
Then at your fingertips you have:
- Optical Character Recognition
- Face detection, and sentiment analysis i.e Happy, Sad
- Landmark detection
I wrote a tiny little ruby script if you want to give a whirl. This specifically uses the landmark detection api.
require 'base64'
require 'json'
require 'faraday'
filepath = '/path/to/my/image.jpg'
content = Base64.encode64(File.binread(filepath))
conn = Faraday.new(:url => 'https://vision.googleapis.com')
data = {requests: [{'image' => {'content' => content}, 'features' => [{'type' => 'LANDMARK_DETECTION', 'maxResults' => 10}]}]};
response = conn.post do |req|
req.url '/v1alpha1/images:annotate?key=<your api key>'
req.headers['Content-Type'] = 'application/json'
req.body = data.to_json
end
puts response
So here is a picture I took in 2006, without any geolocation info attached in exif data.
What does the api return?
It gives a confidence score, a description of the landmark and its rough latitude and longitude, and a bunch of other cool stuff I'm yet to understand.
[
[0] {
"landmarkAnnotations" => [
[0] {
"mid" => "/m/01k_5m",
"description" => "Lake Tahoe",
"score" => 0.46974313,
"boundingPoly" => {
"vertices" => [
[0] {
"x" => 1092,
"y" => 555
},
[1] {
"x" => 1273,
"y" => 555
},
[2] {
"x" => 1273,
"y" => 944
},
[3] {
"x" => 1092,
"y" => 944
}
]
},
"locations" => [
[0] {
"latLng" => {
"latitude" => 38.940395,
"longitude" => -119.91884
}
}
]
},
[1] {
"score" => 0.29812887,
"boundingPoly" => {
"vertices" => [
[0] {
"x" => 960,
"y" => 902
},
[1] {
"x" => 1558,
"y" => 902
},
[2] {
"x" => 1558,
"y" => 1073
},
[3] {
"x" => 960,
"y" => 1073
}
]
},
"locations" => [
[0] {
"latLng" => {
"latitude" => 38.943923,
"longitude" => -119.928989
}
}
]
}
]
}
]
My new Hobby
I used to think I had a few hobbies, mountain biking, rugby, vege gardening, coding, etc, etc, but then my wife and I bought a Queenslander workers cottage that needed a bit of attention.
I had zero interest in renovation. But this beast has taken over, it consumes every minute of my spare time, and has for the past 2 years.
We're nearing the end of the whole experience, and although feeling exhausted, we're really stoked that a lot of things have come together.
If anyone is interested here is our progress in pictures (apologies for the blogger blog :) Egbert Blog
Quickly setup a ruby on rails edge app
Setup your app directory
mkdir rails-edge-test-app
cd rails-ege-test-app
Specify your Ruby version
touch .ruby-version
echo 'ruby-2.3.1' > .ruby-version
Setup Gemfile with edge rails, rack and arel
touch Gemfile
printf "source 'https://rubygems.org'\nruby '2.3.1'\n\n\ngem 'rails', github: 'rails/rails'" >> Gemfile
bundle
Use rails generators to create new app, and run server
bundle exec rails new . --dev --force
bundle exec rails s
Machines are learning from us, and that's not always a good thing
I was recently playing with Google's new Photo application. An incredibly powerful little application, yet has an incredibly simple UI.
Google has gone through my entire library of 30k photos and classified them against their boat loads of data.
It's always nice to get a visual sense of what Machine Learning is doing, and after playing with the photo app it's quite clear how it classifies photos.
For example, if I search for 'festival', it finds photo's of all sorts of festivals, from Glastonbury:
To the Soccer World Cup:
To the Notting Hill Carnival:
To a few incorrect ones like these:
You can understand why machine learning might classify these, and it's not really a problem. When we have more data, those results will continue to get better, they will almost definitely result in more accuracy going into the future.
So, what's the problem?
The problem is here; do a search for something like 'desert', I get this result:
Do a search for 'dessert' and I get this result:
So, through our own stupidity, we have trained the machines to think 'dessert' is the same as 'desert'.
A trivial example, but in a time when we are ramping up our dependence on machine learning, we need to remember that the machine is not always correct and it is for the same reason as the crowd is not always correct.
Safety nets
I've had a few close calls when switching between the Dev/Staging/Prod environments. These will save you from any nasty suprises!
iTerm Colour
Change the colour of the terminal window when you ssh into your staging or production environments. I use a simple bash script that switches when I ssh into certain hosts.
Banner/Ribbon on Staging/Development
I just used a fork of the Github ribbon
Show App Name and Environment when in console
We used this handy gem Marco-Polo
That looks like this in the console:
~/Sites/myapp$ rails c
myapp(dev)>