onnxruntime.cr

ONNX Runtime bindings for Crystal

Installation

1. Install ONNX Runtime

   Download and install the ONNX Runtime from the official releases.

   Option A: System-wide installation (Recommended)

   For Linux:

   VERSION_TAG=$(cat ONNXRUNTIME_VERSION)
   VERSION=${VERSION_TAG#v}
   wget https://github.com/microsoft/onnxruntime/releases/download/$VERSION_TAG/onnxruntime-linux-x64-$VERSION.tgz
   tar -xzf onnxruntime-linux-x64-$VERSION.tgz
   
   # Install to system directories
   sudo cp onnxruntime-linux-x64-$VERSION/lib/* /usr/local/lib/
   sudo cp -r onnxruntime-linux-x64-$VERSION/include/* /usr/local/include/
   sudo ldconfig

   For macOS:

   VERSION_TAG=$(cat ONNXRUNTIME_VERSION)
   VERSION=${VERSION_TAG#v}
   curl -L https://github.com/microsoft/onnxruntime/releases/download/$VERSION_TAG/onnxruntime-osx-arm64-$VERSION.tgz -o onnxruntime-osx-arm64-$VERSION.tgz
   tar -xzf onnxruntime-osx-arm64-$VERSION.tgz
   
   # Install to system directories
   sudo cp onnxruntime-osx-arm64-$VERSION/lib/* /usr/local/lib/
   sudo cp -r onnxruntime-osx-arm64-$VERSION/include/* /usr/local/include/

   Option B: Using local installation

   If you prefer not to install system-wide, set library paths:

   # Download and extract as above, then:
   export LIBRARY_PATH=/path/to/onnxruntime-linux-x64-$VERSION/lib:$LIBRARY_PATH  # For build time
   export LD_LIBRARY_PATH=/path/to/onnxruntime-linux-x64-$VERSION/lib:$LD_LIBRARY_PATH  # For runtime
   
   # Build and run your Crystal program
   crystal build your_program.cr
   ./your_program

   Alternatively, use --link-flags:

   # Set path variable for convenience
   ORT_LIB=/path/to/onnxruntime-linux-x64-$VERSION/lib
   
   # Build with embedded rpath
   crystal build your_program.cr --link-flags="-L$ORT_LIB -Wl,-rpath,$ORT_LIB"
   
   # Run without LD_LIBRARY_PATH
   ./your_program

2. Add the dependency to your shard.yml:

   dependencies:
     onnxruntime:
       github: kojix2/onnxruntime.cr

3. Run shards install

Usage

require "onnxruntime"

# Recommended: RAII block style
OnnxRuntime::InferenceSession.open("path/to/model.onnx", release_env: true) do |session|
  # Print model inputs and outputs
  puts "Inputs:"
  session.inputs.each do |input|
    puts "  #{input.name}: #{input.type} #{input.shape}"
  end

  puts "Outputs:"
  session.outputs.each do |output|
    puts "  #{output.name}: #{output.type} #{output.shape}"
  end

  # Prepare input data
  input_data = {
    "input_name" => [1.0_f32, 2.0_f32, 3.0_f32]
  }

  # Run inference
  result = session.run(input_data)

  # Process results
  result.each do |name, data|
    puts "#{name}: #{data}"
  end
end

MNIST Example

Download the MNIST model: mnist-12.onnx (raw

require "onnxruntime"

# Load the MNIST model
session = OnnxRuntime::InferenceSession.new("mnist-12.onnx")

# Create a dummy input (28x28 image draw 1)
input_data = Array(Float32).new(28 * 28) { |i| (i % 14 == 0 ? 1.0 : 0.0).to_f32 }

# Run inference
result = session.run({"Input3" => input_data}, ["Plus214_Output_0"], shape: {"Input3" => [1_i64, 1_i64, 28_i64, 28_i64]})

# Get the output probabilities
probabilities = result["Plus214_Output_0"].as(Array(Float32))

# Find the digit with highest probability
predicted_digit = probabilities.index(probabilities.max)
puts "Predicted digit: #{predicted_digit}"

# Explicitly release resources
session.release
OnnxRuntime::InferenceSession.release_env

Memory Management

You can use either explicit release or block-based RAII.

Recommended for short scripts: block-based RAII

OnnxRuntime::InferenceSession.open("path/to/model.onnx", release_env: true) do |session|
  result = session.run(input_data)
  # session is always released, even if an error is raised
end

Explicit release (useful for long-running apps):

# Create and use session
session = OnnxRuntime::InferenceSession.new("path/to/model.onnx")
result = session.run(input_data)

# When finished, explicitly release resources
session.release
OnnxRuntime::InferenceSession.release_env

release_session remains available for backward compatibility.

For long-running applications like web servers, use explicit release with signal handlers:

Signal::INT.trap do
  puts "Shutting down..."
  session.release
  OnnxRuntime::InferenceSession.release_env
  exit
end

See the examples directory for more detailed implementations.

Why do you need to manually free memory?

Previously, the following error was displayed on macOS.

libc++abi: terminating due to uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
Program received and didn't handle signal ABRT (6)

This seems to be related to multithreading and mutex. According to the AI, this is difficult to solve with finalize, so we tried to solve it by creating a reference counter, but we were unable to solve it in the end. If you can solve this problem, please create a pull request!

Development

The code is generated by AI and may not be perfect. Please feel free to contribute and improve it.

Contributing

1. Fork it (https://github.com/kojix2/onnxruntime.cr/fork)
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request