Mascali
/
piston
mirror of https://github.com/engineer-man/piston.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

design: remove as outdated

This commit is contained in:
Thomas Hobson 2021-03-06 18:31:13 +13:00
parent 9706f2fab8
commit 22dcad0dd9
No known key found for this signature in database
GPG Key ID: 9F1FD9D87950DB6F
5 changed files with 0 additions and 313 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

71
ARCHITECTURE.TXT
View File

@ -1,71 +0,0 @@
== Breif == [ Piston ]
This document covers the overall architecture of Piston v3, and not the
individual components and their implementations.
In Piston v2 we saw 2 ways of using piston - through the CLI and the API.
These 2 methods would call the same load of bash scripts contained within
a LXC container which would then resolve your request.
There are a number of issues with this approach:
1. This uses bash - which isn't the best language for performance
2. It relied on calling through a `lxc-attach` command to access
inside the container
3. This isn't easy to distribute
4. It was difficult to add languages - having to edit 4 different files
in 4 different places to add a single language
Piston v3 aims to tackle these 4 issues.
Firstly, v3 will be less reliant on bash, only using it as an option
for running different interpreters.
Secondly, v3 can run on bare-metal or in a container, but a core API will be
exposed from within the container, instead of running external to it.
Thirdly, v3 will provide a simple docker container, which will expose both the
piston API, and run all the runners within it.
Finally, v3 will provide a repository of precompiled language executers, so its
1 command away from installing a language
== Piston API ==
Piston v3 exposes a REST API, allowing the user to control the entire thing
over one simple JSON based protocol. This eliminates the need to connect into
the container to do maintainace such as adding new languages, or checking
usage statistics.
See design/api.txt for more information.
== Package Manager ==
Piston v3 includes a package manager right out of the box. The package manager
manages the different languages and versions that it can run.
The package manager is hooked directly into the API and addresses our point
of easy distibution, as users now can easily enable/disable different
components built into piston as they see fit.
See design/ppman.txt for more information.
== Runtime Environment ==
The new architecture moves to a more bare-metal approach, where the code can be
run without the overhead of a container manager such as LXC or Docker, making
piston much easier to manage this way
It is still possible to run Piston v3 in a contain, but now a container engine
is not required for usage, however it is still recommended.
== Proxy API ==
The in-container API is more powerful than a simple execution API and thus
should be limited, however to keep the weight down, and speed up there is a
reference implementation of a proxy API included, which passes through
execution commands to many different piston instances and allows for
security with rate limiting and API keys.
See design/proxy.txt

75
design/api.txt
View File

@ -1,75 +0,0 @@
== Piston API == [ Piston ]
When we speak of piston, what we actually talk about is the Piston API.
This API provides unrestricted, unlimited access to managing piston and
thus shouldn't be publicly exposed. This API is comparable to one of the
docker engine, where everything regarding control of docker goes directly
through the api.
The API is responsible for managing the execution lifecycle of any given
job, as well as managing the different languages which it can execute a
job in.
== Job Execution ==
Piston v3 exposes an endpoint per package `/execute`, which when called takes
in both a string of code, and an array of arguments to pass into the program
as well as data to write to STDIN. The stdout and stderr from the process are
then both returned seperately, along with the error code returned.
All of this is has no rate-limiting built in making it lightning fast as a call
will directly start the runner process and get under way instantly.
The 2 stages of this process - compile and run are both run in sequence, with
different timeouts configurable in the runners config file located in the
data directory.
Requests to this endpoint can have caching enabled at 3 different levels.
The first option is to have no caching, which is the default for all
interpreted language. The second option is for the compiled binaries to be
cached, which is the default for all compiled languages. The final option is
for output to be cached, which isn't used by default but can be enabled per
package or per request. This is done for the reason that code may choose to
source data from /dev/(u)random or similar sources and as such may not be as
reliable when their outputs are cached. Caching is per package and is used as
an acceleration method to help boost performance of Piston. Cache entries are
automatically purged after the set time, or can be manually purged through the
API on a per package basis.
== Package Manager ==
Piston v3 has an inbuilt package manager which is responsible for
(un)installing different packages. Piston v3 by default has access to a single
offical repository hosting various versions of various common languages. These
packages and repositories conform to the specifications set out in ppman.txt
The Piston API service downloads the repository index whenever a `/packages`
request is issued to a repository with the `sync` flag is set. This will cause
the service to download the latest repostiory index off the mirror.
In piston there is no concept of a package being "outdated" as each package is
a specific version of a language, and different languages can be installed in
paralleland function without any issues. Each package should be considered the
final version of that language. If there is a new version of a language
available (i.e. Python 3.9.1 -> 3.9.2), a new package should be created for
this.
Invidual languages can be queried from the repo using the
`/repos/{repo}/packages/{package}/{package-version}` endpoint. This endpoint
allows for the metadata of the package to be accessed, such as the author,
size, checksums, dependencies, build file git url and download url.
To install packages, a request to `/install` can be made to the package
endpoint and it will download and install it, making it available on the
`/packages/{package}/{version}` endpoint.
There is a meta-repository name `all` which can be used to access all
repositories.
Internally the install process involved downloading and unpacking the package,
ensuring any dependencies are also downloaded and installed, mounting the
squashfs filesystem to a folder, then overlaying it with all its dependencies
in another folder.

18
design/fs.txt
View File

@ -1,18 +0,0 @@
== File System Layout == [ Piston ]
All of pistons files are installed in the `/piston` directory. This directory
contains all runtimes, config, packages and cache that piston uses.
Each package gets its own directory, where it its prebuilt binaries are
unpacked into. This is contained within `/piston/packages`
The binaries folder contained within this is then symlinked into the runtimes
directory. This is where all the different runtimes available are placed. This
is contained within the `/piston/runtimes` directory.
The cache directory a directory containing all the different cache files. It is
recommended to either sym-link this into a folder withing /tmp/ or directly
mount it as a tmpfs filesystem.
Configuration is stored in a single file - piston.yaml and contains all
documentation required for configuring the piston API

13
design/index.txt
View File

@ -1,13 +0,0 @@
== Index == [ Piston ]
Design outlines the design of the different components and does not give a
concrete definition of the implementation or how to use it.
api.txt Design of Piston API
ppman.txt Design of the package manager's package and repository format
== Glossary ==
Execution Job A single code run with arguments resulting in an output
Package A version of a language bundled together into a tarball

136
design/ppman.txt
View File

@ -1,136 +0,0 @@
== Package Manager (ppman) == [ Piston ]
The package manager is the part of the API responsible for managing different
versions of different languages, managing their installation, uninstallation
and their dependencies. The package manager talks over the piston api and is
built directly into piston, although has parts which are not directly built
into the API (i.e. the repositories and the cli utility).
The package manager is a complex part of piston, and requires 2 different file
specifications - the repository index file and the package file.
== Repository Index File ==
The piston repository is the central place where packages are hosted and
downloaded from. This repository can either be a webserver or a local file
containing the right content - as long as its accessable by a URL, its
considered a valid repository by piston. A repository URL is simply a URL
pointing to a repository index file, as set out by the following information.
A repository index file is a YAML file containing the keys: `schema`,
`baseurl`, `keys` and `packages`.
The schema key simply should have a value of `ppman-repo-1`. This indicates the
version and file format for the client to recieve.
The baseurl key contains the base url that relative URLs should be based off,
this doesn't need to be related to the url that the repository index is hosted
at, only the downloadable files, which are possible to split over many domains
by using absolute paths.
The keys key contains a list of GPG key ids which should be used when
verifying.
The packages key contains a list of packages, which contain the keys: `author`,
`language`, `version`, `checksums`, `dependencies`, `size`, `buildfile`,
`download` and `signature`.
The author field is self explainatory, it is simply the authors name and email,
formatted similar to git's default format: `Full Name <email@address>`. If the
repository index is automatically generated, it is best to use the commit
author's name in here.
The language and version fields define the version and name of the compiler or
interpreter contained within. The language should not include a version at all.
In the case of python, use the name python for both python 2 and 3, using the
version field to differentiate between the 2.
The checksums field is simply a map of hash types to hashes, hash types include
md5, sha1, sha256, sha512. The digests should simply be written as lowercase
hex characters. Only one checksum is required, but if more are supplied the
most secure one is picked, with sha512 as the highest possible.
The dependencies is simply a map of language names to versions, which should be
installed for the package to run correctly. An example of this would be
typescript requires node to run.
The size field is the number of bytes the package file is in size, while
uncompressed. This is used to determine if there is enough room, and thus
should be accurate.
The buildfile field is a URL pointing to the exact build script for this build.
This should always point to a URL either containing steps, a script or other
means of reproducing the build. This field is purely so people can understand
how the image was built, and to make sure you aren't packing any mallicious
code into it.
The download field is a URL pointing to a place of which the package file can
be obtained from. If this is a relative url, the baseurl will be appended to
it. This is particularly useful if everything is stored within 1 s3 bucket, or
you have a repository in a folder.
The signature field is an armored signature
== Package File ==
Within a repository, many package files are contained. These files are
downloaded and installed into `/piston`. They need to all follow the format
as listed below for the API to properly utilize them.
A package file is a gzipped tar archive, containing 4/5 things - `run`,
`compile` (optional), `pkg-info.json`, `lang-ver` and `lang-ver/environment`.
Replacing lang-ver with the language name and the version respectively.
The `pkg-info.json` file contains 5 different keys: `language`, `version`,
`author`, `dependencies` and `build_platform`.
The language field is simply the name of the language, all lowercase and not
containing any version number. This is important in the case of python
specifically as python3 and python2 come under the same language, just
different versions.
The version field is a sem-ver compliant version number for the interpreter
contained wthin the package. It should be a string.
The author field contains the author name, and should be formatted exactly like
shown previously in the repo index spec.
The dependencies field is simply a map containing packages which this package
depends on. This should only be used when one language is a requirement for
another, like how typescript is dependent on node. The key should be the name
of the package, with the value being the version selector.
The build_platform field is used for defining which type of system the package
was built on, this helps resolve system compatability errors. It consists of 2
parts - environment and disto. The environment is simply one of `baremetal`,
`docker`, `lxc` or any other type of environment you can run piston on. The
distro is the ID of the distro as contained in /etc/os-release on the system
which built the package. This is done to ensure system compatability,
especially inside the offically supported Docker container.
The `run` and `compile` files are used in the execution of a job, being used
to both compile and run the source code provided. They are both treated the
same inputs-wise but can have timeouts independently set per stage. The
arguments fed both these files are the same, with the first argument being a
path to the code file, and the rest being the arguments to passed in. These
files are run in a temporary directory contained within the cache folder.
Depending on the cache control level the code and associated files will either
be disposed of or kept. By default only files named `binary` will be kept.
STDIN is only passed into the `run` file, but both files have their output
captured and returned along with their exit codes through the API
The `lang-ver` folder should contain any interpreter specific files, such as
the binary to execute and any other files that may be required to run the
interpreter/compiler contained within the package.
The `environment` file contained within `lang-ver` should contain export
commands like a ~/.bashrc file should, as this is its intended purpose. Firstly
the language which is being executed has its environment file sources, then it
walks down the dependency tree sourcing files. The environment variables are
eventually cached to speed up the execution process.