Mascali/piston
1
0
Fork 0
mirror of https://github.com/engineer-man/piston.git synced 2025-04-21 04:26:28 +02:00
Code Issues Projects Releases Wiki Activity

Update for new API

Browse source
This commit is contained in:
Vrganj 2021-01-23 21:31:23 +01:00
commit c10a30b69a
78 changed files with 2063 additions and 868 deletions
Download patch file Download diff file Expand all files Collapse all files

547
readme.md
View file

@ -1,348 +1,220 @@
## Piston
Piston is the underlying engine for running untrusted and possibly malicious
code that originates from EMKC contests and challenges. It's also used in the
Engineer Man Discord server via [I Run Code](https://github.com/engineer-man/piston-bot) bot as well as 1000+ other servers.
<h1 align="center">
<a href="https://github.com/engineer-man/piston"><img src="https://emkc.org/images/icon_circle_24.png" alt="engineer-man piston"></a>
Piston
</h1>
<h3 align="center">A high performance general purpose code execution engine.</h3>
<br>
<p align="center">
<a href="https://github.com/ArmynC/ArminC-AutoExec/commits/master">
<img src="https://img.shields.io/github/last-commit/engineer-man/piston.svg?style=for-the-badge&logo=github&logoColor=white"
alt="GitHub last commit">
<a href="https://github.com/engineer-man/piston/issues">
<img src="https://img.shields.io/github/issues/engineer-man/piston.svg?style=for-the-badge&logo=github&logoColor=white"
alt="GitHub issues">
<a href="https://github.com/engineer-man/piston/pulls">
<img src="https://img.shields.io/github/issues-pr-raw/engineer-man/piston.svg?style=for-the-badge&logo=github&logoColor=white"
alt="GitHub pull requests">
</p>
---
<h4 align="center">
<a href="#About">About</a> •
<a href="#Public-API">Public API</a> •
<a href="#Getting-Started">Getting Started</a> •
<a href="#Usage">Usage</a> •
<a href="#Supported-Languages">Supported Languages</a> •
<a href="#Principle-of-Operation">Principles</a> •
<a href="#Security">Security</a> •
<a href="#License">License</a>
</h4>
---
<br>
# About
<h4>
Piston is a high performance general purpose code execution engine. It excels at running untrusted and
possibly malicious code without fear from any harmful effects.
</h4>
<br>
It's used in numerous places including:
* [EMKC Challenges](https://emkc.org/challenges),
* [EMKC Weekly Contests](https://emkc.org/contests),
* [Engineer Man Discord Server](https://discord.gg/engineerman),
* [I Run Code (Discord Bot)](https://github.com/engineer-man/piston-bot) bot as well as 1300+ other servers
and 100+ direct integrations.
To get it in your own server, go here: https://emkc.org/run.
#### Use Public API (new)
Requires no installation and you can use it immediately. Reference the API Usage section below to learn
about the request format but rather than using the local URLs, use the following URLs:
- `GET` `https://emkc.org/api/v1/piston/versions`
- `POST` `https://emkc.org/api/v1/piston/execute`
<br>
Important Note: The Piston API is rate limited to 5 requests per second
# Public API
- Requires no installation and you can use it immediately.
- Reference the Versions/Execute sections below to learn about the request and response formats.
<br>
When using the public Piston API, use the base URL:
#### Installation
```
# clone and enter repo
git clone https://github.com/engineer-man/piston
cd piston/lxc
https://emkc.org/api/v1/piston
```
# install dependencies
#### GET
```
https://emkc.org/api/v1/piston/versions
```
#### POST
```
https://emkc.org/api/v1/piston/execute
```
> Important Note: The Piston API is rate limited to 5 requests per second. If you have a need for more requests than that
and it's for a good cause, please reach out to me (EngineerMan#0001) on [Discord](https://discord.gg/engineerman)
so we can discuss potentially getting you an unlimited key.
<br>
# Getting Started
### Host System Package Dependencies
* NodeJS
* lxc
* libvirt
<br>
If your OS is not documented below, please open pull requests with the correct commands for your OS.
<details>
<summary><span style="font-size:1.43em;">CentOS / RHEL</span></summary>
```sh
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash
nvm install --lts
nvm use --lts
# centos:
yum install -y epel-release
yum install -y lxc lxc-templates debootstrap libvirt
systemctl start libvirtd
```
</details>
# ubuntu server 18.04:
apt install lxc lxc-templates debootstrap libvirt0
<details>
<summary><span style="font-size:1.43em;">Ubuntu (18.04)</span></summary>
# arch:
sudo pacman -S lxc libvirt unzip
```sh
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash
nvm install --lts
nvm use --lts
# everything else:
# not documented, please open pull requests with commands for debian/arch/macos
apt install -y lxc lxc-templates debootstrap libvirt0
```
</details>
# create and start container
lxc-create -t download -n piston -- --dist ubuntu --release bionic --arch amd64
./start
<details>
<summary><span style="font-size:1.43em;">Arch Linux</span></summary>
# open a shell to the container
./shell
```sh
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash
nvm install --lts
nvm use --lts
# install all necessary piston dependencies
echo 'source /opt/.profile' >> /opt/.bashrc
echo 'export HOME=/opt' >> /opt/.profile
echo 'export TERM=linux' >> /opt/.profile
echo 'export PATH=$PATH:/opt/.local/bin' >> /opt/.profile
export HOME=/opt
export TERM=linux
sed -i 's/\/root/\/opt/' /etc/passwd
sed -i \
's/http:\/\/archive.ubuntu.com\/ubuntu/http:\/\/mirror.math.princeton.edu\/pub\/ubuntu/' \
/etc/apt/sources.list
apt-get update
apt-get install -y \
nano wget build-essential pkg-config libxml2-dev \
libsqlite3-dev mono-complete curl cmake libpython2.7-dev \
ruby libtinfo-dev unzip
pacman -S lxc libvirt unzip
```
</details>
# install python2
# final binary: /opt/python2/Python-2.7.17/python
# get version: /opt/python2/Python-2.7.17/python -V
cd /opt && mkdir python2 && cd python2
wget https://www.python.org/ftp/python/2.7.17/Python-2.7.17.tar.xz
unxz Python-2.7.17.tar.xz
tar -xf Python-2.7.17.tar
cd Python-2.7.17
./configure
# open Modules/Setup and uncomment zlib line
make
echo 'export PATH=$PATH:/opt/python2/Python-2.7.17' >> /opt/.profile
source /opt/.profile
#### After system dependencies are installed, clone this repository:
# install python3
# final binary: /opt/python3/Python-3.8.2/python
# get version: /opt/python3/Python-3.8.2/python -V
cd /opt && mkdir python3 && cd python3
wget https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tar.xz
unxz Python-3.8.2.tar.xz
tar -xf Python-3.8.2.tar
cd Python-3.8.2
./configure
make
ln -s python python3.8
echo 'export PATH=$PATH:/opt/python3/Python-3.8.2' >> /opt/.profile
source /opt/.profile
# install paradoc
# this is not a binary, it is a python module
# therefore it cannot be run directly as it requires python3 to be installed
cd /opt && mkdir paradoc && cd paradoc
git clone https://github.com/betaveros/paradoc.git
echo 'export PYTHONPATH=$PYTHONPATH:/opt/paradoc/paradoc' >> /opt/.profile
source /opt/.profile
# install node.js
# final binary: /opt/nodejs/node-v12.16.1-linux-x64/bin/node
# get version: /opt/nodejs/node-v12.16.1-linux-x64/bin/node -v
cd /opt && mkdir nodejs && cd nodejs
wget https://nodejs.org/dist/v12.16.1/node-v12.16.1-linux-x64.tar.xz
unxz node-v12.16.1-linux-x64.tar.xz
tar -xf node-v12.16.1-linux-x64.tar
echo 'export PATH=$PATH:/opt/nodejs/node-v12.16.1-linux-x64/bin' >> /opt/.profile
source /opt/.profile
# install typescript
# final binary: /opt/nodejs/node-v12.16.1-linux-x64/bin/tsc
# get version: /opt/nodejs/node-v12.16.1-linux-x64/bin/tsc -v
/opt/nodejs/node-v12.16.1-linux-x64/bin/npm i -g typescript
# install golang
# final binary: /opt/go/go/bin/go
# get version: /opt/go/go/bin/go version
cd /opt && mkdir go && cd go
wget https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz
tar -xzf go1.14.1.linux-amd64.tar.gz
echo 'export PATH=$PATH:/opt/go/go/bin' >> /opt/.profile
echo 'export GOROOT=/opt/go/go' >> /opt/.profile
echo 'export GOCACHE=/tmp' >> /opt/.profile
source /opt/.profile
# install php
# final binary: /usr/local/bin/php
# get version: /usr/local/bin/php -v
cd /opt && mkdir php && cd php
wget https://www.php.net/distributions/php-7.4.4.tar.gz
tar -xzf php-7.4.4.tar.gz
cd php-7.4.4
./configure
make
make install
# install rust
# final binary: /opt/.cargo/bin/rustc
# get version: /opt/.cargo/bin/rustc --version
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
echo 'export PATH=$PATH:/opt/.cargo/bin' >> /opt/.profile
source /opt/.profile
# install swift
# final binary: /opt/swift/swift-5.1.5-RELEASE-ubuntu18.04/usr/bin/swift
# get version: /opt/swift/swift-5.1.5-RELEASE-ubuntu18.04/usr/bin/swift --version
cd /opt && mkdir swift && cd swift
wget https://swift.org/builds/swift-5.1.5-release/ubuntu1804/swift-5.1.5-RELEASE/swift-5.1.5-RELEASE-ubuntu18.04.tar.gz
tar -xzf swift-5.1.5-RELEASE-ubuntu18.04.tar.gz
echo 'export PATH=$PATH:/opt/swift/swift-5.1.5-RELEASE-ubuntu18.04/usr/bin' >> /opt/.profile
source /opt/.profile
# install nasm
# final binary: /opt/nasm/nasm-2.14.02/nasm
# get version: /opt/nasm/nasm-2.14.02/nasm -v
cd /opt && mkdir nasm && cd nasm
wget https://www.nasm.us/pub/nasm/releasebuilds/2.14.02/nasm-2.14.02.tar.gz
tar -xzf nasm-2.14.02.tar.gz
cd nasm-2.14.02
./configure
make
echo 'export PATH=$PATH:/opt/nasm/nasm-2.14.02' >> /opt/.profile
source /opt/.profile
# install java
# final binary: /opt/java/jdk-14/bin/java
# get version: /opt/java/jdk-14/bin/java -version
cd /opt && mkdir java && cd java
wget https://download.java.net/java/GA/jdk14/076bab302c7b4508975440c56f6cc26a/36/GPL/openjdk-14_linux-x64_bin.tar.gz
tar -xzf openjdk-14_linux-x64_bin.tar.gz
echo 'export PATH=$PATH:/opt/java/jdk-14/bin' >> /opt/.profile
source /opt/.profile
# install jelly
cd /opt && mkdir jelly && cd jelly
wget https://github.com/DennisMitchell/jellylanguage/archive/master.zip
unzip master.zip
cd jellylanguage-master
pip3.8 install .
# install julia
# final binary: /opt/julia/julia-1.5.0/bin/julia
# get version: /opt/julia/julia-1.5.0/bin/julia --version
cd /opt && mkdir julia && cd julia
wget https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.0-linux-x86_64.tar.gz
tar -xzf julia-1.5.0-linux-x86_64.tar.gz
echo 'export PATH=$PATH:/opt/julia/julia-1.5.0/bin' >> /opt/.profile
source /opt/.profile
# install kotlin
# final binary: /opt/kotlinc/bin/kotlinc
# get version: /opt/kotlinc/bin/kotlinc -version
cd /opt
wget https://github.com/JetBrains/kotlin/releases/download/v1.4.10/kotlin-compiler-1.4.10.zip
unzip kotlin-compiler-1.4.10.zip
rm kotlin-compiler-1.4.10.zip
echo 'export PATH=$PATH:/opt/kotlinc/bin' >> /opt/.profile
source /opt/.profile
# install elixir and erlang
# final binary: /opt/elixir/bin/elixir
# get version: /opt/elixir/bin/elixir --version
# erlang
cd /opt && mkdir erlang && cd erlang
wget http://erlang.org/download/otp_src_23.0.tar.gz
gunzip -c otp_src_23.0.tar.gz | tar xf -
cd otp_src_23.0 && ./configure
make
echo 'export PATH=$PATH:/opt/erlang/otp_src_23.0/bin' >> /opt/.profile
source /opt/.profile
# elixir
cd /opt && mkdir elixir && cd elixir
wget https://github.com/elixir-lang/elixir/releases/download/v1.10.3/Precompiled.zip
mkdir elixir-1.10.3 && unzip Precompiled.zip -d elixir-1.10.3/
echo 'export PATH=$PATH:/opt/elixir/elixir-1.10.3/bin' >> /opt/.profile
source /opt/.profile
# install emacs
# final binary: /opt/emacs/emacs-26.3/src/emacs
# get version: /opt/emacs/emacs-26.3/src/emacs --version
cd /opt && mkdir emacs && cd emacs
wget https://mirrors.ocf.berkeley.edu/gnu/emacs/emacs-26.3.tar.xz
tar -xf emacs-26.3.tar.xz
cd emacs-26.3
./configure --with-gnutls=no
make
echo 'export PATH=$PATH:/opt/emacs/emacs-26.3/src' >> /opt/.profile
source /opt/.profile
# install lua
# final binary: /opt/lua/lua54/src/lua
# get version: /opt/lua/lua54/src/lua -v
cd /opt && mkdir lua && cd lua
wget https://sourceforge.net/projects/luabinaries/files/5.4.0/Docs%20and%20Sources/lua-5.4.0_Sources.tar.gz/download
tar -xzf download
cd lua54
make
echo 'export PATH=$PATH:/opt/lua/lua54/src' >> /opt/.profile
source /opt/.profile
# install haskell
# final binary: /usr/bin/ghc
# get version: /usr/bin/ghc --version
apt install -y ghc
# install deno
# final binary: /opt/.deno/bin/deno
# get version: /opt/.deno/bin/deno --version
cd /opt && mkdir deno && cd deno
curl -fsSL https://deno.land/x/install/install.sh | sh
echo 'export DENO_INSTALL="/opt/.deno"' >> /opt/.profile
echo 'export PATH="$DENO_INSTALL/bin:$PATH"' >> /opt/.profile
source /opt/.profile
# install nim
# final binary: /opt/nim/bin/nim
# get version: /opt/nim/bin/nim -v
cd /opt && mkdir nim && cd nim
wget https://nim-lang.org/download/nim-1.4.0-linux_x64.tar.xz
unxz nim-1.4.0-linux_x64.tar.xz
tar -xf nim-1.4.0-linux_x64.tar
cd nim-1.4.0
./install.sh /opt
echo 'export PATH=$PATH:/opt/nim/bin' >> /opt/.profile
source /opt/.profile
# install d
# final binary: /opt/d/dmd2/linux/bin64/dmd
# get version: /opt/d/dmd2/linux/bin64/dmd --version
cd /opt && mkdir d && cd d
wget http://downloads.dlang.org/releases/2.x/2.095.0/dmd.2.095.0.linux.tar.xz
unxz dmd.2.095.0.linux.tar.xz
tar -xf dmd.2.095.0.linux.tar
echo 'export PATH=$PATH:/opt/d/dmd2/linux/bin64' >> /opt/.profile
source /opt/.profile
# create runnable users and apply limits
for i in {1..150}; do
useradd -M runner$i
usermod -d /tmp runner$i
echo "runner$i soft nproc 64" >> /etc/security/limits.conf
echo "runner$i hard nproc 64" >> /etc/security/limits.conf
echo "runner$i soft nofile 2048" >> /etc/security/limits.conf
echo "runner$i hard nofile 2048" >> /etc/security/limits.conf
done
# cleanup
rm -rf /home/ubuntu
chmod 777 /tmp
# leave container
exit
# optionally run tests
cd ../tests
./test_all_lxc
```sh
# clone and enter repo
git clone https://github.com/engineer-man/piston
cd piston/lxc
```
#### CLI Usage
- `lxc/execute [language] [file path] [arg]...`
#### Installation (simple)
- Coming soon.
#### Installation (advanced)
- See `var/install.txt` for how to create a new LXC container and install all of the required
software.
#### CLI Usage
- `cli/execute [language] [file path] [args]`
<br>
# Usage
### CLI
```sh
lxc/execute [language] [file path] [args]
```
### API
To use the API, it must first be started. Please note that if root is required to access
LXC then the API must also be running as root. To start the API, run the following:
#### API Usage
To use the API, it must first be started. To start the API, run the following:
```
cd api
./start
```
#### Base URLs
For your own local installation, use:
For your own local installation, the API is available at:
```
http://127.0.0.1:2000
```
When using the public Piston API, use:
```
https://emkc.org/api/v1/piston
```
#### Versions Endpoint
`GET /versions`
This endpoint takes no input and returns a JSON array of the currently installed languages.
Truncated response sample:
This endpoint will return the supported languages along with the current version and aliases. To execute
code for a particular language using the `/execute` endpoint, either the name or one of the aliases must
be provided.
```json
HTTP/1.1 200 OK
Content-Type: application/json
[
{
"name": "awk",
"aliases": ["awk"],
"version": "1.3.3"
},
{
"name": "bash",
"aliases": ["bash"],
"version": "4.4.20"
},
{
"name": "c",
"aliases": ["c"],
"version": "7.5.0"
}
]
```
#### Execution Endpoint
#### Execute Endpoint
`POST /execute`
This endpoint takes the following JSON payload and expects at least the language and source. If
source is not provided, a blank file is passed as the source.
This endpoint requests execution of some arbitrary code.
- `language` (**required**) The language to use for execution, must be a string and supported by Piston (see list below).
- `source` (**required**) The source code to execute, must be a string.
- `stdin` (*optional*) The text to pass as stdin to the program. Must be a string or left out of the request.
- `args` (*optional*) The arguments to pass to the program. Must be an array or left out of the request.
```json
{
"language": "js",
"source": "console.log(process.argv)",
"stdin": "",
"args": [
"1",
"2",
@ -350,58 +222,76 @@ source is not provided, a blank file is passed as the source.
]
}
```
A typical response when everything succeeds will be similar to the following:
A typical response upon successful execution will contain the `language`, `version`, `output` which
is a combination of both `stdout` and `stderr` but in chronological order according to program output,
as well as separate `stdout` and `stderr`.
```json
HTTP/1.1 200 OK
Content-Type: application/json
{
"ran": true,
"language": "js",
"version": "12.13.0",
"output": "[ '/usr/bin/node',\n '/tmp/code.code',\n '1',\n '2',\n '3' ]"
"output": "[ '/usr/bin/node',\n '/tmp/code.code',\n '1',\n '2',\n '3' ]",
"stdout": "[ '/usr/bin/node',\n '/tmp/code.code',\n '1',\n '2',\n '3' ]",
"stderr": ""
}
```
If an invalid language is supplied, a typical response will look like the following:
If a problem exists with the request, a `400` status code is returned and the reason in the `message` key.
```json
HTTP/1.1 400 Bad Request
Content-Type: application/json
{
"code": "unsupported_language",
"message": "whatever is not supported by Piston"
"message": "Supplied language is not supported by Piston"
}
```
#### Supported Languages
- awk
- bash
- c
- cpp
- csharp
- d
- deno
- elixir
- emacs
- go
- haskell
- java
- jelly
- julia
- kotlin
- nasm
- node
- perl
- php
- python2
- python3
- paradoc
- ruby
- rust
- swift
- typescript
<br>
#### Principle of Operation
Piston utilizes LXC as the primary mechanism for sandboxing. There is a small API written in Go which takes
# Supported Languages
`awk`,
`bash`,
`brainfuck`,
`c`,
`cpp`,
`csharp`,
`d`,
`deno`,
`elixir`,
`emacs`,
`elisp`,
`go`,
`haskell`,
`java`,
`jelly`,
`julia`,
`kotlin`,
`lua`,
`nasm`,
`node`,
`paradoc`,
`perl`,
`php`,
`python2`,
`python3`,
`ruby`,
`rust`,
`swift`,
`typescript`,
<br>
# Principle of Operation
Piston utilizes LXC as the primary mechanism for sandboxing. There is a small API written in Node which takes
in execution requests and executes them in the container. High level, the API writes
a temporary source and args file to `/tmp` and that gets mounted read-only along with the execution scripts into the container.
The source file is either ran or compiled and ran (in the case of languages like c, c++, c#, go, etc.).
#### Security
<br>
# Security
LXC provides a great deal of security out of the box in that it's separate from the system.
Piston takes additional steps to make it resistant to
various privilege escalation, denial-of-service, and resource saturation threats. These steps include:
@ -409,10 +299,13 @@ various privilege escalation, denial-of-service, and resource saturation threats
- Capping max processes at 64 (resists `:(){ :|: &}:;`, `while True: os.fork()`, etc.)
- Capping max files at 2048 (resists various file based attacks)
- Mounting all resources read-only (resists `sudo rm -rf --no-preserve-root /`)
- Cleaning up all temp space after each execution (resists out of drive space attacks)
- Running as a variety of unprivileged users
- Capping runtime execution at 3 seconds
- Capping stdout to 65536 characters (resists yes/no bombs and runaway output)
- SIGKILLing misbehaving code
#### License
<br>
# License
Piston is licensed under the MIT license.