Oil Shell — Using Oil to improve a Bash script

created: 29.09.2020

The Oil project aims to transform the Unix shell into a better programming language. Under its umbrella live two languages: OSH and the Oil language. OSH is a bash compatible shell, while the Oil language is an improved, safer, and nicer shell. I played with Oil for a while, and I talked to the creator of the Oil project, Andy Chu. I like Oil, and I think you might like it too.

Disclaimer: The Oil language is still in early development. It absolutely can be tried out already, but it’s not complete and you will run into some bugs and crashes. OSH is more stable. In the following text, I will refer to the Oil language as “Oil” out of laziness.

Let’s start with a simple problem and solve it with bash first, then with Oil. In node projects, there is typically a package.json file that includes dependencies and build scripts. The build scripts can be run with CLI tools like npm or yarn. My idea for a super simple script: Let’s execute the scripts from a package.json without using npm or yarn. (You shouldn’t do this in real life, because the npm dependencies would be missing.)

A simple package.json may look like this:

{
  "name": "example",
  "version": "1.0.0",
  "description": "This is just an example package.json",
  "main": "index.js",
  "scripts": {
    "test": "echo \"all good :)\"",
    "say hello": "echo \"Hello world!\""
  },
  "author": "Till",
  "license": "WTFPL"
}

With this package.json, running npm run test will print “all good :)”. And I want to be able to run something like my-script.sh test to do the same, but without calling npm.

First, I need to get the script out of the json file. For this, I can use jq, a nice tool for handling json in the command line. To get the test command, you can write jq -r .scripts.test package.json. In a shell script, the first argument of the script is called $1, so I’ll use jq -r .scripts.$1 package.json. Then I just need to run eval on that to execute the script:

#!/bin/bash
eval $(jq -r .scripts.$1 package.json)

Easy enough. Let’s make it slightly less simple by also printing the script before we run it.

 #!/bin/bash
script= $(jq -r .scripts.$1 package.json)
echo $script :
eval $script

Hmmmm, when I run this, the results look a bit odd:

$ ./bash-simple.sh test
"all good :)"
:

That’s not what I want. Even worse, ./bash-simple.sh "say hello" doesn’t work at all, complaining about hello not being a file. Can you spot all the mistakes I made?

Let me try explaining them:

1. Variable definitions in bash can’t have spaces. Bash, therefore, evaluates script= (with a space in the end) just like script="", and then executes the rest as a command. The $(...) causes the script to be executed right away.
2. The $1 is not quoted, therefore it’s split up into jq -r .scripts.say hello, which causes jq to think we want to read from a file called hello.
3. Fun fact: Should our script variable be “-n”, the echo command would only print :, without a new line at the end. That’s because “-n” is an option for echo.

It’s a very simple shell script, yet it would be understandable for a beginner to overlook all three of these mistakes. Especially quoting is often a problem. The issues can only be solved by making the script uglier.

script=$(jq -r .scripts."\"$1\"" package.json)
echo "$script :"
eval $script

But bash is already ugly enough, I don’t want to write stuff like this. Also, bash has a lot more gotchas, and it’s just annoying to fight with it all the time. I recommend reading the article in Greg’s wiki on bash pitfalls.

Oil as a better way

Oil has built-in json support. That means you can not only work on flat text, but you can use nested json like in every other language:

json read :pkg < ./package.json

Oil has a setvar keyword for defining and updating variables:

setvar scripts = pkg["scripts"]

Yes, you can put white space around the =! There is also the var keyword, used only for defining new variables. The set keyword updates them.

Oil also changes the defaults. Splitting requires explicitly writing @split(variable), while $variable will not be split. No double quoting is needed.

So the complete Oil script now looks like this:

#!/usr/local/bin/oil

json read :pkg < ./package.json
setvar script = pkg["scripts"][$1]
write -- $script:
eval $script

This looks much nicer in my opinion, and you’re way less likely to make mistakes. Using a dedicated keyword makes it possible to differentiate declaring and updating variables.

Some other nice features

Let me show you some other cool things in Oil.

Oil has better arrays:

oil$ setvar files = ["image with spaces.png", "notes.txt"]
oil$ rm @files

“Egg Expressions” are nicer Regular Expressions. You can assign them to variables and compose them easily. They “compile” to normal regular expressions, so you can use them with existing tools like grep.

# Define a subpattern that matches three digits
oil$ setvar D = / digit{1,3} /

# Use the subpattern to describe ip addresses
oil$ setvar ip_pat = / D '.' D '.' D '.' D /

# Use the pattern with grep
oil$ grep -E $ip_pat log.txt

Currently, you can fry your Egg Expressions only in Oil, but hopefully, other tools will adopt something like this as well.

Also “functions” (now reasonably renamed to “proc”) can declare their parameters and don’t have dynamic scoping.

Overall, I believe that lots of the changes Oil is making are an improvement over bash.

One planned Oil feature that I’m particularly looking forward too is blocks. Andy Chu plans to add Ruby-like blocks, that could allow simple meta programming in Oil. This relatively simple feature, if done well, could allow shell to grow into several areas that are already close to its usual uses: Shell as configuration language? Shell as a build system?

Adoption and compatibility

I’m convinced that Oil is already better than bash for most scripting tasks, and Oil is rapidly improving. But is better good enough? Bash is preinstalled on most unix systems, and there are large shell scripts that already exist that nobody wants to rewrite.

Oil Shell attempts to solve this with the backward compatible OSH. OSH runs bash scripts just fine, and you can still use some of the new features, such as the built-in json support.

OSH has fine-grained settings to make it stricter and enable more advanced features. Oil is just OSH with a bunch of these settings enabled. Step by step, you can turn the crude OSH into the refined Oil language. So you don’t have to adopt Oil all at once, you just enable the nicer features one at a time when you want them.

Can you hear the C?

Unix shell never was the cleanest or prettiest language. But at some point it was simple and when it grew to more platforms new implementations added new features. Now shell is a big family of complex languages. Shell is a language that grows in two ways: It grows with each new implementation, and it grows in features because everyone can easily make CLI programs. Almost every language can write CLI apps since everything is just plain text. Shell is the glue that holds these apps together.

In a way, Oil is the logical next step, yet another shell dialect that solves the issues that bash has accumulated over the decades. On the other hand, Oil adds new data structures that can’t be trivially expressed in plain text, so it’s harder to extend it naturally with CLI apps. This way, shell loses its connection with C.

To give an example: in shell code like if [ $variable = "something ]; then ... the [ is a command. That means the variable, =, the string on the right and even ] are passed to it as arguments. Practically, that’s super inconvenient since you have to have spaces between all operands. But conceptually it’s almost amazing, it means you can just write your own [, in whatever language you like! Do you want to be able to do math inside your if? Feel free to write that in Haskell or VisualBasic.Net or whatever.

As a counter-example, take the json built-in from Oil. json can assign a Dict to a variable. If you want to write your own json command or add an xml command, you can only write it in Oil.

This problem isn’t new, Bash already has associative arrays that aren’t just plaintext. This might be the necessary step shell programming has to take to improve. It might be worth pointing out that PowerShell is the extreme conclusion to this, in PowerShell everything is an object instead of a string. PowerShell is in this way very anti-unix. And Oil is therefore also not really following the “unix philosophy”. But I think it’s doing the right thing.

Oil has a sales problem

While I believe that Oil is an improvement over the status quo, its features aren’t “flashy”. Shell has two purposes: It’s a scripting language, and an interactive enviroment. Oil focuses on the scripting part. If you run it interactively, it’s not (yet) as convenient as the alternatives. Its tooling is not yet as good as bash’s (No shellcheck, no debuggers, no proper syntax highlighting, etc). As with any upcoming programming language, this will take time.

In contrast, the fish shell focuses on the interactive experience. If you get someone to install fish, they immediately see cool things that make their life easier right away. Pitching Oil needs more time. You have to explain how Oil allows better-structured code, what kind of bugs you can avoid in Oil, etc. I started using Oil for some small private scripts, and I’m happy with those. I will continue writing personal scripts in Oil. But I’ll go back to fish as my default interactive shell, probably until “fish oil” becomes a reality.

This kind of goes back to the old “worse is better” problem. I think Oil is better (than bash), but that doesn’t necessarily mean it will see adoption. I hope it will. I’d prefer a world where /bin/sh is typically a link to OSH or Oil.

If you agree with me and have some free time over, consider helping out with the development of Oil. There are lots of exciting challenges left!