piston/readme.md

329 lines
8.3 KiB
Markdown
Raw Normal View History

2021-01-16 18:01:35 +01:00
<h1 align="center">
2021-01-17 21:45:58 +01:00
<a href="https://github.com/engineer-man/piston"><img src="https://emkc.org/images/icon_circle_24.png" alt="engineer-man piston"></a>
Piston
</h1>
2021-01-16 18:01:35 +01:00
<h3 align="center">A high performance general purpose code execution engine.</h3>
<br>
<p align="center">
2021-01-23 20:57:13 +01:00
<a href="https://github.com/engineer-man/piston/commits/master">
2021-01-16 18:01:35 +01:00
<img src="https://img.shields.io/github/last-commit/engineer-man/piston.svg?style=for-the-badge&logo=github&logoColor=white"
alt="GitHub last commit">
<a href="https://github.com/engineer-man/piston/issues">
<img src="https://img.shields.io/github/issues/engineer-man/piston.svg?style=for-the-badge&logo=github&logoColor=white"
alt="GitHub issues">
<a href="https://github.com/engineer-man/piston/pulls">
<img src="https://img.shields.io/github/issues-pr-raw/engineer-man/piston.svg?style=for-the-badge&logo=github&logoColor=white"
alt="GitHub pull requests">
</p>
---
<h4 align="center">
<a href="#About">About</a>
<a href="#Public-API">Public API</a>
<a href="#Getting-Started">Getting Started</a>
<a href="#Usage">Usage</a>
<a href="#Supported-Languages">Supported Languages</a>
<a href="#Principle-of-Operation">Principles</a>
<a href="#Security">Security</a>
<a href="#License">License</a>
</h4>
---
<br>
# About
<h4>
2021-01-14 07:16:01 +01:00
Piston is a high performance general purpose code execution engine. It excels at running untrusted and
possibly malicious code without fear from any harmful effects.
2021-01-16 18:01:35 +01:00
</h4>
<br>
It's used in numerous places including:
* [EMKC Challenges](https://emkc.org/challenges),
* [EMKC Weekly Contests](https://emkc.org/contests),
* [Engineer Man Discord Server](https://discord.gg/engineerman),
* [I Run Code (Discord Bot)](https://github.com/engineer-man/piston-bot) bot as well as 1300+ other servers
and 100+ direct integrations.
To get it in your own server, go here: https://emkc.org/run.
<br>
# Public API
- Requires no installation and you can use it immediately.
- Reference the Versions/Execute sections below to learn about the request and response formats.
<br>
When using the public Piston API, use the base URL:
```
https://emkc.org/api/v1/piston
```
#### GET
```
https://emkc.org/api/v1/piston/versions
```
#### POST
```
https://emkc.org/api/v1/piston/execute
```
> Important Note: The Piston API is rate limited to 5 requests per second. If you have a need for more requests than that
2021-01-14 07:10:28 +01:00
and it's for a good cause, please reach out to me (EngineerMan#0001) on [Discord](https://discord.gg/engineerman)
so we can discuss potentially getting you an unlimited key.
2020-07-04 07:07:21 +02:00
2021-01-16 18:01:35 +01:00
<br>
# Getting Started
### Host System Package Dependencies
* NodeJS
* lxc
* libvirt
<br>
If your OS is not documented below, please open pull requests with the correct commands for your OS.
<details>
<summary><span style="font-size:1.43em;">CentOS / RHEL</span></summary>
```sh
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash
nvm install --lts
nvm use --lts
2021-01-14 07:01:22 +01:00
yum install -y epel-release
yum install -y lxc lxc-templates debootstrap libvirt
systemctl start libvirtd
2021-01-16 18:01:35 +01:00
```
</details>
<details>
<summary><span style="font-size:1.43em;">Ubuntu (18.04)</span></summary>
```sh
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash
nvm install --lts
nvm use --lts
apt install -y lxc lxc-templates debootstrap libvirt0
```
</details>
<details>
<summary><span style="font-size:1.43em;">Arch Linux</span></summary>
2021-01-14 07:01:22 +01:00
2021-01-16 18:01:35 +01:00
```sh
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash
nvm install --lts
nvm use --lts
2021-01-14 07:01:22 +01:00
2021-01-21 20:43:47 +01:00
pacman -S lxc libvirt unzip
2021-01-16 18:01:35 +01:00
```
</details>
#### After system dependencies are installed, clone this repository:
2021-01-14 07:01:22 +01:00
2021-01-16 18:01:35 +01:00
```sh
# clone and enter repo
git clone https://github.com/engineer-man/piston
cd piston/lxc
2021-01-14 07:01:22 +01:00
```
#### Installation (simple)
2021-01-28 01:19:24 +01:00
- `mkdir -p /var/lib/lxc/piston`
- `cd /var/lib/lxc/piston`
- `wget [latest container url] -o container.zip`
- `unzip container.zip`
- `tar xzf rootfs.tar.gz`
- `rm rootfs.tar.gz container.zip`
- Good to go!
2021-01-16 18:01:35 +01:00
2021-01-28 01:19:24 +01:00
2021-01-16 18:01:35 +01:00
#### Installation (advanced)
2021-01-28 01:19:24 +01:00
- `cd ../container && ./install.sh`
- Wait, it may take up to an hour.
- `cd ../lxc && ./start && cd ..`
- Good to go!
Alternatively, see `var/install.txt` for how to build the container manually
2018-09-21 20:45:09 +02:00
2019-06-17 07:06:38 +02:00
#### CLI Usage
2021-01-16 13:27:14 +01:00
- `cli/execute [language] [file path] [args]`
2021-01-16 18:01:35 +01:00
<br>
# Usage
### CLI
```sh
lxc/execute [language] [file path] [args]
```
2018-09-21 20:45:09 +02:00
2021-01-16 18:01:35 +01:00
### API
2021-01-14 07:08:03 +01:00
To use the API, it must first be started. Please note that if root is required to access
LXC then the API must also be running as root. To start the API, run the following:
2021-01-16 18:01:35 +01:00
2019-06-17 07:09:35 +02:00
```
cd api
./start
```
2020-07-04 07:07:21 +02:00
2021-01-22 09:58:45 +01:00
For your own local installation, the API is available at:
2021-01-16 18:01:35 +01:00
2021-01-14 07:01:22 +01:00
```
http://127.0.0.1:2000
```
2020-07-04 07:07:21 +02:00
#### Versions Endpoint
`GET /versions`
2021-01-22 09:58:45 +01:00
This endpoint will return the supported languages along with the current version and aliases. To execute
code for a particular language using the `/execute` endpoint, either the name or one of the aliases must
be provided.
2020-07-04 07:07:21 +02:00
```json
HTTP/1.1 200 OK
Content-Type: application/json
2020-07-04 07:07:21 +02:00
[
{
"name": "awk",
2021-01-16 13:27:14 +01:00
"aliases": ["awk"],
2020-07-04 07:07:21 +02:00
"version": "1.3.3"
},
{
"name": "bash",
2021-01-16 13:27:14 +01:00
"aliases": ["bash"],
2020-07-04 07:07:21 +02:00
"version": "4.4.20"
},
{
"name": "c",
2021-01-16 13:27:14 +01:00
"aliases": ["c"],
2020-07-04 07:07:21 +02:00
"version": "7.5.0"
}
]
```
2021-01-14 07:01:22 +01:00
#### Execute Endpoint
2020-07-04 07:07:21 +02:00
`POST /execute`
2021-01-22 09:58:45 +01:00
This endpoint requests execution of some arbitrary code.
- `language` (**required**) The language to use for execution, must be a string and supported by Piston (see list below).
- `source` (**required**) The source code to execute, must be a string.
- `stdin` (*optional*) The text to pass as stdin to the program. Must be a string or left out of the request.
- `args` (*optional*) The arguments to pass to the program. Must be an array or left out of the request.
2019-06-17 07:06:38 +02:00
```json
{
"language": "js",
"source": "console.log(process.argv)",
2021-01-22 09:58:45 +01:00
"stdin": "",
2019-06-17 07:06:38 +02:00
"args": [
"1",
"2",
"3"
]
}
```
2021-01-14 07:01:22 +01:00
A typical response upon successful execution will contain the `language`, `version`, `output` which
is a combination of both `stdout` and `stderr` but in chronological order according to program output,
as well as separate `stdout` and `stderr`.
2019-06-17 07:06:38 +02:00
```json
HTTP/1.1 200 OK
Content-Type: application/json
2019-06-17 07:06:38 +02:00
{
"ran": true,
"language": "js",
"version": "12.13.0",
2021-01-14 07:01:22 +01:00
"output": "[ '/usr/bin/node',\n '/tmp/code.code',\n '1',\n '2',\n '3' ]",
"stdout": "[ '/usr/bin/node',\n '/tmp/code.code',\n '1',\n '2',\n '3' ]",
"stderr": ""
2019-06-17 07:06:38 +02:00
}
```
2021-01-22 09:58:45 +01:00
If a problem exists with the request, a `400` status code is returned and the reason in the `message` key.
2019-06-17 07:06:38 +02:00
```json
HTTP/1.1 400 Bad Request
Content-Type: application/json
2019-06-17 07:06:38 +02:00
{
2021-01-22 09:58:45 +01:00
"message": "Supplied language is not supported by Piston"
2019-06-17 07:06:38 +02:00
}
```
2021-01-16 18:01:35 +01:00
<br>
# Supported Languages
2021-01-22 09:45:52 +01:00
`awk`,
`bash`,
`brainfuck`,
`c`,
`cpp`,
2021-01-23 22:40:25 +01:00
`crystal`,
2021-01-22 09:45:52 +01:00
`csharp`,
2021-01-23 21:31:23 +01:00
`d`,
2021-01-23 22:40:25 +01:00
`dash`,
2021-01-22 09:45:52 +01:00
`deno`,
`elixir`,
`emacs`,
`elisp`,
`go`,
`haskell`,
`java`,
`jelly`,
`julia`,
`kotlin`,
2021-01-23 22:40:25 +01:00
`lisp`,
2021-01-22 09:45:52 +01:00
`lua`,
`nasm`,
2021-01-23 22:40:25 +01:00
`nasm64`,
2021-01-23 22:45:42 +01:00
`nim`,
2021-01-22 09:45:52 +01:00
`node`,
`paradoc`,
`perl`,
`php`,
`python2`,
`python3`,
`ruby`,
`rust`,
`swift`,
`typescript`,
2021-01-23 22:40:25 +01:00
`zig`,
2021-01-16 18:01:35 +01:00
<br>
# Principle of Operation
2021-01-16 00:53:51 +01:00
Piston utilizes LXC as the primary mechanism for sandboxing. There is a small API written in Node which takes
2019-05-31 20:09:47 +02:00
in execution requests and executes them in the container. High level, the API writes
a temporary source and args file to `/tmp` and that gets mounted read-only along with the execution scripts into the container.
2018-09-22 18:52:19 +02:00
The source file is either ran or compiled and ran (in the case of languages like c, c++, c#, go, etc.).
2018-09-22 06:15:24 +02:00
2021-01-16 18:01:35 +01:00
<br>
# Security
2019-05-31 20:09:47 +02:00
LXC provides a great deal of security out of the box in that it's separate from the system.
Piston takes additional steps to make it resistant to
2018-09-22 06:15:24 +02:00
various privilege escalation, denial-of-service, and resource saturation threats. These steps include:
- Disabling outgoing network interaction
- Capping max processes at 64 (resists `:(){ :|: &}:;`, `while True: os.fork()`, etc.)
- Capping max files at 2048 (resists various file based attacks)
2018-09-22 06:15:24 +02:00
- Mounting all resources read-only (resists `sudo rm -rf --no-preserve-root /`)
- Cleaning up all temp space after each execution (resists out of drive space attacks)
- Running as a variety of unprivileged users
2018-09-22 18:52:19 +02:00
- Capping runtime execution at 3 seconds
2018-09-22 06:15:24 +02:00
- Capping stdout to 65536 characters (resists yes/no bombs and runaway output)
- SIGKILLing misbehaving code
2018-09-21 20:45:09 +02:00
2021-01-16 18:01:35 +01:00
<br>
# License
2018-09-21 20:45:09 +02:00
Piston is licensed under the MIT license.