piston/readme.md

398 lines
12 KiB
Markdown
Raw Normal View History

2018-09-21 20:45:09 +02:00
## Piston
Piston is the underlying engine for running untrusted and possibly malicious
code that originates from EMKC contests and challenges. It's also used in the
Engineer Man Discord server via [felix bot](https://github.com/engineer-man/felix).
2018-09-21 20:45:09 +02:00
2020-07-04 07:07:21 +02:00
#### Use Public API (new)
Requires no installation and you can use it immediately. Reference the API Usage section below to learn
about the request format but rather than using the local URLs, use the following URLs:
- `GET` `https://emkc.org/api/v1/piston/versions`
- `POST` `https://emkc.org/api/v1/piston/execute`
2020-07-04 07:08:17 +02:00
2020-07-04 07:07:21 +02:00
Important Note: The Piston API is rate limited to 1 request per second
2018-09-21 20:45:09 +02:00
#### Installation
2019-05-31 20:09:47 +02:00
```
# clone and enter repo
git clone https://github.com/engineer-man/piston
cd piston/lxc
# install dependencies
# centos:
2019-05-31 20:59:51 +02:00
yum install -y epel-release
yum install -y lxc lxc-templates debootstrap libvirt
2019-06-17 07:37:41 +02:00
systemctl start libvirtd
2019-05-31 20:09:47 +02:00
2019-06-17 07:37:41 +02:00
# ubuntu server 18.04:
apt install lxc lxc-templates debootstrap libvirt0
2019-05-31 20:09:47 +02:00
# arch:
sudo pacman -S lxc libvirt unzip
2019-06-17 07:37:41 +02:00
# everything else:
# not documented, please open pull requests with commands for debian/arch/macos
2019-05-31 20:09:47 +02:00
# create and start container
lxc-create -t download -n piston -- --dist ubuntu --release bionic --arch amd64
./start
2019-05-31 22:34:48 +02:00
# open a shell to the container
2019-05-31 20:09:47 +02:00
./shell
# install all necessary piston dependencies
echo 'source /opt/.profile' >> /opt/.bashrc
echo 'export HOME=/opt' >> /opt/.profile
echo 'export TERM=linux' >> /opt/.profile
2020-10-17 20:57:26 +02:00
echo 'export PATH=$PATH:/opt/.local/bin' >> /opt/.profile
2019-06-19 07:09:56 +02:00
export HOME=/opt
export TERM=linux
sed -i 's/\/root/\/opt/' /etc/passwd
2019-05-31 20:14:24 +02:00
sed -i \
's/http:\/\/archive.ubuntu.com\/ubuntu/http:\/\/mirror.math.princeton.edu\/pub\/ubuntu/' \
/etc/apt/sources.list
2019-05-31 20:09:47 +02:00
apt-get update
apt-get install -y \
nano wget build-essential pkg-config libxml2-dev \
libsqlite3-dev mono-complete curl cmake libpython2.7-dev \
ruby libtinfo-dev unzip
# install python2
2020-06-09 04:24:15 +02:00
# final binary: /opt/python2/Python-2.7.17/python
# get version: /opt/python2/Python-2.7.17/python -V
cd /opt && mkdir python2 && cd python2
wget https://www.python.org/ftp/python/2.7.17/Python-2.7.17.tar.xz
unxz Python-2.7.17.tar.xz
tar -xf Python-2.7.17.tar
cd Python-2.7.17
./configure
# open Modules/Setup and uncomment zlib line
make
echo 'export PATH=$PATH:/opt/python2/Python-2.7.17' >> /opt/.profile
source /opt/.profile
# install python3
2020-06-09 04:24:15 +02:00
# final binary: /opt/python3/Python-3.8.2/python
# get version: /opt/python3/Python-3.8.2/python -V
cd /opt && mkdir python3 && cd python3
wget https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tar.xz
unxz Python-3.8.2.tar.xz
tar -xf Python-3.8.2.tar
cd Python-3.8.2
./configure
make
ln -s python python3.8
echo 'export PATH=$PATH:/opt/python3/Python-3.8.2' >> /opt/.profile
source /opt/.profile
# install node.js
2020-06-09 04:24:15 +02:00
# final binary: /opt/nodejs/node-v12.16.1-linux-x64/bin/node
# get version: /opt/nodejs/node-v12.16.1-linux-x64/bin/node -v
cd /opt && mkdir nodejs && cd nodejs
wget https://nodejs.org/dist/v12.16.1/node-v12.16.1-linux-x64.tar.xz
unxz node-v12.16.1-linux-x64.tar.xz
tar -xf node-v12.16.1-linux-x64.tar
echo 'export PATH=$PATH:/opt/nodejs/node-v12.16.1-linux-x64/bin' >> /opt/.profile
source /opt/.profile
# install typescript
2020-06-09 04:24:15 +02:00
# final binary: /opt/nodejs/node-v12.16.1-linux-x64/bin/tsc
# get version: /opt/nodejs/node-v12.16.1-linux-x64/bin/tsc -v
/opt/nodejs/node-v12.16.1-linux-x64/bin/npm i -g typescript
# install golang
2020-06-09 04:24:15 +02:00
# final binary: /opt/go/go/bin/go
# get version: /opt/go/go/bin/go version
cd /opt && mkdir go && cd go
wget https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz
tar -xzf go1.14.1.linux-amd64.tar.gz
echo 'export PATH=$PATH:/opt/go/go/bin' >> /opt/.profile
echo 'export GOROOT=/opt/go/go' >> /opt/.profile
echo 'export GOCACHE=/tmp' >> /opt/.profile
source /opt/.profile
# install php
2020-06-09 04:24:15 +02:00
# final binary: /usr/local/bin/php
# get version: /usr/local/bin/php -v
cd /opt && mkdir php && cd php
wget https://www.php.net/distributions/php-7.4.4.tar.gz
tar -xzf php-7.4.4.tar.gz
cd php-7.4.4
./configure
make
make install
# install rust
2020-06-09 04:24:15 +02:00
# final binary: /opt/.cargo/bin/rustc
# get version: /opt/.cargo/bin/rustc --version
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
echo 'export PATH=$PATH:/opt/.cargo/bin' >> /opt/.profile
source /opt/.profile
# install swift
2020-06-09 04:24:15 +02:00
# final binary: /opt/swift/swift-5.1.5-RELEASE-ubuntu18.04/usr/bin/swift
# get version: /opt/swift/swift-5.1.5-RELEASE-ubuntu18.04/usr/bin/swift --version
cd /opt && mkdir swift && cd swift
wget https://swift.org/builds/swift-5.1.5-release/ubuntu1804/swift-5.1.5-RELEASE/swift-5.1.5-RELEASE-ubuntu18.04.tar.gz
tar -xzf swift-5.1.5-RELEASE-ubuntu18.04.tar.gz
echo 'export PATH=$PATH:/opt/swift/swift-5.1.5-RELEASE-ubuntu18.04/usr/bin' >> /opt/.profile
source /opt/.profile
# install nasm
2020-06-09 04:24:15 +02:00
# final binary: /opt/nasm/nasm-2.14.02/nasm
# get version: /opt/nasm/nasm-2.14.02/nasm -v
cd /opt && mkdir nasm && cd nasm
wget https://www.nasm.us/pub/nasm/releasebuilds/2.14.02/nasm-2.14.02.tar.gz
tar -xzf nasm-2.14.02.tar.gz
cd nasm-2.14.02
./configure
make
echo 'export PATH=$PATH:/opt/nasm/nasm-2.14.02' >> /opt/.profile
source /opt/.profile
# install java
2020-06-09 04:24:15 +02:00
# final binary: /opt/java/jdk-14/bin/java
# get version: /opt/java/jdk-14/bin/java -version
cd /opt && mkdir java && cd java
wget https://download.java.net/java/GA/jdk14/076bab302c7b4508975440c56f6cc26a/36/GPL/openjdk-14_linux-x64_bin.tar.gz
tar -xzf openjdk-14_linux-x64_bin.tar.gz
echo 'export PATH=$PATH:/opt/java/jdk-14/bin' >> /opt/.profile
source /opt/.profile
2019-05-31 20:09:47 +02:00
2020-10-17 19:56:02 +02:00
# install jelly
2020-10-17 20:57:26 +02:00
cd /opt && mkdir jelly && cd jelly
wget https://github.com/DennisMitchell/jellylanguage/archive/master.zip
unzip master.zip
cd jellylanguage-master
2020-10-17 21:14:19 +02:00
pip3.8 install .
2020-10-17 19:56:02 +02:00
2020-05-02 06:19:36 +02:00
# install julia
2020-08-09 01:30:08 +02:00
# final binary: /opt/julia/julia-1.5.0/bin/julia
# get version: /opt/julia/julia-1.5.0/bin/julia --version
2020-05-02 06:19:36 +02:00
cd /opt && mkdir julia && cd julia
2020-08-09 01:30:08 +02:00
wget https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.0-linux-x86_64.tar.gz
tar -xzf julia-1.5.0-linux-x86_64.tar.gz
echo 'export PATH=$PATH:/opt/julia/julia-1.5.0/bin' >> /opt/.profile
2020-05-02 06:19:36 +02:00
source /opt/.profile
2020-06-05 02:17:26 +02:00
# install kotlin
2020-06-09 04:24:15 +02:00
# final binary: /opt/kotlinc/bin/kotlinc
# get version: /opt/kotlinc/bin/kotlinc -version
2020-06-04 21:01:14 +02:00
cd /opt
2020-10-29 22:39:14 +01:00
wget https://github.com/JetBrains/kotlin/releases/download/v1.4.10/kotlin-compiler-1.4.10.zip
unzip kotlin-compiler-1.4.10.zip
rm kotlin-compiler-1.4.10.zip
2020-06-04 21:01:14 +02:00
echo 'export PATH=$PATH:/opt/kotlinc/bin' >> /opt/.profile
source /opt/.profile
2020-05-02 06:19:36 +02:00
# install elixir and erlang
2020-06-09 04:24:15 +02:00
# final binary: /opt/elixir/bin/elixir
# get version: /opt/elixir/bin/elixir --version
# erlang
cd /opt && mkdir erlang && cd erlang
wget http://erlang.org/download/otp_src_23.0.tar.gz
gunzip -c otp_src_23.0.tar.gz | tar xf -
cd otp_src_23.0 && ./configure
make
echo 'export PATH=$PATH:/opt/erlang/otp_src_23.0/bin' >> /opt/.profile
source /opt/.profile
2020-06-05 02:17:26 +02:00
# elixir
cd /opt && mkdir elixir && cd elixir
wget https://github.com/elixir-lang/elixir/releases/download/v1.10.3/Precompiled.zip
mkdir elixir-1.10.3 && unzip Precompiled.zip -d elixir-1.10.3/
echo 'export PATH=$PATH:/opt/elixir/elixir-1.10.3/bin' >> /opt/.profile
source /opt/.profile
2020-06-05 03:01:21 +02:00
# install emacs
2020-06-09 04:24:15 +02:00
# final binary: /opt/emacs/emacs-26.3/src/emacs
# get version: /opt/emacs/emacs-26.3/src/emacs --version
2020-06-05 03:01:21 +02:00
cd /opt && mkdir emacs && cd emacs
wget https://mirrors.ocf.berkeley.edu/gnu/emacs/emacs-26.3.tar.xz
tar -xf emacs-26.3.tar.xz
cd emacs-26.3
./configure --with-gnutls=no
2020-06-09 04:42:21 +02:00
make
2020-06-05 03:01:21 +02:00
echo 'export PATH=$PATH:/opt/emacs/emacs-26.3/src' >> /opt/.profile
source /opt/.profile
2020-10-07 06:40:14 +02:00
# install lua
# final binary: /opt/lua/lua54/src/lua
# get version: /opt/lua/lua54/src/lua -v
cd /opt && mkdir lua && cd lua
wget https://sourceforge.net/projects/luabinaries/files/5.4.0/Docs%20and%20Sources/lua-5.4.0_Sources.tar.gz/download
tar -xzf download
cd lua54
make
echo 'export PATH=$PATH:/opt/lua/lua54/src' >> /opt/.profile
source /opt/.profile
2020-10-17 05:57:45 +02:00
# install haskell
# final binary: /usr/bin/ghc
# get version: /usr/bin/ghc --version
apt install -y ghc
2020-10-17 05:57:45 +02:00
# install deno
# final binary: /opt/.deno/bin/deno
# get version: /opt/.deno/bin/deno --version
2020-10-17 20:57:26 +02:00
cd /opt && mkdir deno && cd deno
curl -fsSL https://deno.land/x/install/install.sh | sh
2020-10-17 08:20:35 +02:00
echo 'export DENO_INSTALL="/opt/.deno"' >> /opt/.profile
echo 'export PATH="$DENO_INSTALL/bin:$PATH"' >> /opt/.profile
source /opt/.profile
2020-10-31 20:10:27 +01:00
# install nim
# final binary: /opt/nim/bin/nim
# get version: /opt/nim/bin/nim -v
cd /opt && mkdir nim && cd nim
wget https://nim-lang.org/download/nim-1.4.0-linux_x64.tar.xz
unxz nim-1.4.0-linux_x64.tar.xz
tar -xf nim-1.4.0-linux_x64.tar
cd nim-1.4.0
./install.sh /opt
2020-10-31 20:36:04 +01:00
echo 'export PATH=$PATH:/opt/nim/bin' >> /opt/.profile
source /opt/.profile
2020-10-31 20:10:27 +01:00
2019-05-31 20:09:47 +02:00
# create runnable users and apply limits
for i in {1..150}; do
useradd -M runner$i
usermod -d /tmp runner$i
echo "runner$i soft nproc 64" >> /etc/security/limits.conf
echo "runner$i hard nproc 64" >> /etc/security/limits.conf
echo "runner$i soft nofile 2048" >> /etc/security/limits.conf
echo "runner$i hard nofile 2048" >> /etc/security/limits.conf
done
# cleanup
rm -rf /home/ubuntu
chmod 777 /tmp
2019-05-31 20:09:47 +02:00
# leave container
exit
# optionally run tests
cd ../tests
./test_all_lxc
```
2018-09-21 20:45:09 +02:00
2019-06-17 07:06:38 +02:00
#### CLI Usage
- `lxc/execute [language] [file path] [arg]...`
2018-09-21 20:45:09 +02:00
2019-06-17 07:06:38 +02:00
#### API Usage
2019-06-17 07:09:35 +02:00
To use the API, it must first be started. To start the API, run the following:
```
cd api
./start
```
2020-07-04 07:07:21 +02:00
#### Base URLs
For your own local installation, use:
```
http://127.0.0.1:2000
```
When using the public Piston API, use:
```
https://emkc.org/api/v1/piston
```
#### Versions Endpoint
`GET /versions`
This endpoint takes no input and returns a JSON array of the currently installed languages.
Truncated response sample:
```json
[
{
"name": "awk",
"version": "1.3.3"
},
{
"name": "bash",
"version": "4.4.20"
},
{
"name": "c",
"version": "7.5.0"
}
]
```
#### Execution Endpoint
`POST /execute`
2019-06-17 07:06:38 +02:00
This endpoint takes the following JSON payload and expects at least the language and source. If
source is not provided, a blank file is passed as the source.
```json
{
"language": "js",
"source": "console.log(process.argv)",
"args": [
"1",
"2",
"3"
]
}
```
A typical response when everything succeeds will be similar to the following:
```json
{
"ran": true,
"language": "js",
"version": "12.13.0",
2019-06-17 07:06:38 +02:00
"output": "[ '/usr/bin/node',\n '/tmp/code.code',\n '1',\n '2',\n '3' ]"
}
```
If an invalid language is supplied, a typical response will look like the following:
```json
{
"code": "unsupported_language",
"message": "whatever is not supported by Piston"
}
```
2018-09-21 20:45:09 +02:00
#### Supported Languages
2020-06-09 04:24:15 +02:00
- awk
- bash
- c
- cpp
- csharp
2020-10-17 19:06:23 +02:00
- deno
2020-06-09 04:24:15 +02:00
- elixir
- emacs
- go
2020-10-17 05:57:45 +02:00
- haskell
2020-06-09 04:24:15 +02:00
- java
2020-10-17 19:56:02 +02:00
- jelly
2020-06-09 04:24:15 +02:00
- julia
- kotlin
- nasm
- node
- perl
- php
- python2
- python3
- ruby
- rust
- swift
- typescript
2018-09-21 20:45:09 +02:00
2018-09-22 06:15:24 +02:00
#### Principle of Operation
2019-05-31 20:09:47 +02:00
Piston utilizes LXC as the primary mechanism for sandboxing. There is a small API written in Go which takes
in execution requests and executes them in the container. High level, the API writes
a temporary source and args file to `/tmp` and that gets mounted read-only along with the execution scripts into the container.
2018-09-22 18:52:19 +02:00
The source file is either ran or compiled and ran (in the case of languages like c, c++, c#, go, etc.).
2018-09-22 06:15:24 +02:00
#### Security
2019-05-31 20:09:47 +02:00
LXC provides a great deal of security out of the box in that it's separate from the system.
Piston takes additional steps to make it resistant to
2018-09-22 06:15:24 +02:00
various privilege escalation, denial-of-service, and resource saturation threats. These steps include:
- Disabling outgoing network interaction
- Capping max processes at 64 (resists `:(){ :|: &}:;`, `while True: os.fork()`, etc.)
- Capping max files at 2048 (resists various file based attacks)
2018-09-22 06:15:24 +02:00
- Mounting all resources read-only (resists `sudo rm -rf --no-preserve-root /`)
- Running as a variety of unprivileged users
2018-09-22 18:52:19 +02:00
- Capping runtime execution at 3 seconds
2018-09-22 06:15:24 +02:00
- Capping stdout to 65536 characters (resists yes/no bombs and runaway output)
- SIGKILLing misbehaving code
2018-09-21 20:45:09 +02:00
#### License
Piston is licensed under the MIT license.