design: remove as outdated
This commit is contained in:
Thomas Hobson
parent
9706f2fab8
commit
22dcad0dd9
|
|
@ -1,71 +0,0 @@
|
|||
== Breif == [ Piston ]
|
||||
|
||||
This document covers the overall architecture of Piston v3, and not the
|
||||
individual components and their implementations.
|
||||
|
||||
In Piston v2 we saw 2 ways of using piston - through the CLI and the API.
|
||||
These 2 methods would call the same load of bash scripts contained within
|
||||
a LXC container which would then resolve your request.
|
||||
There are a number of issues with this approach:
|
||||
1. This uses bash - which isn't the best language for performance
|
||||
2. It relied on calling through a `lxc-attach` command to access
|
||||
inside the container
|
||||
3. This isn't easy to distribute
|
||||
4. It was difficult to add languages - having to edit 4 different files
|
||||
in 4 different places to add a single language
|
||||
|
||||
Piston v3 aims to tackle these 4 issues.
|
||||
Firstly, v3 will be less reliant on bash, only using it as an option
|
||||
for running different interpreters.
|
||||
Secondly, v3 can run on bare-metal or in a container, but a core API will be
|
||||
exposed from within the container, instead of running external to it.
|
||||
Thirdly, v3 will provide a simple docker container, which will expose both the
|
||||
piston API, and run all the runners within it.
|
||||
Finally, v3 will provide a repository of precompiled language executers, so its
|
||||
1 command away from installing a language
|
||||
|
||||
|
||||
|
||||
== Piston API ==
|
||||
|
||||
Piston v3 exposes a REST API, allowing the user to control the entire thing
|
||||
over one simple JSON based protocol. This eliminates the need to connect into
|
||||
the container to do maintainace such as adding new languages, or checking
|
||||
usage statistics.
|
||||
|
||||
See design/api.txt for more information.
|
||||
|
||||
|
||||
|
||||
== Package Manager ==
|
||||
|
||||
Piston v3 includes a package manager right out of the box. The package manager
|
||||
manages the different languages and versions that it can run.
|
||||
The package manager is hooked directly into the API and addresses our point
|
||||
of easy distibution, as users now can easily enable/disable different
|
||||
components built into piston as they see fit.
|
||||
|
||||
See design/ppman.txt for more information.
|
||||
|
||||
|
||||
|
||||
== Runtime Environment ==
|
||||
|
||||
The new architecture moves to a more bare-metal approach, where the code can be
|
||||
run without the overhead of a container manager such as LXC or Docker, making
|
||||
piston much easier to manage this way
|
||||
|
||||
It is still possible to run Piston v3 in a contain, but now a container engine
|
||||
is not required for usage, however it is still recommended.
|
||||
|
||||
|
||||
|
||||
== Proxy API ==
|
||||
|
||||
The in-container API is more powerful than a simple execution API and thus
|
||||
should be limited, however to keep the weight down, and speed up there is a
|
||||
reference implementation of a proxy API included, which passes through
|
||||
execution commands to many different piston instances and allows for
|
||||
security with rate limiting and API keys.
|
||||
|
||||
See design/proxy.txt
|
||||
|
|
@ -1,75 +0,0 @@
|
|||
== Piston API == [ Piston ]
|
||||
|
||||
When we speak of piston, what we actually talk about is the Piston API.
|
||||
This API provides unrestricted, unlimited access to managing piston and
|
||||
thus shouldn't be publicly exposed. This API is comparable to one of the
|
||||
docker engine, where everything regarding control of docker goes directly
|
||||
through the api.
|
||||
|
||||
The API is responsible for managing the execution lifecycle of any given
|
||||
job, as well as managing the different languages which it can execute a
|
||||
job in.
|
||||
|
||||
|
||||
|
||||
== Job Execution ==
|
||||
|
||||
Piston v3 exposes an endpoint per package `/execute`, which when called takes
|
||||
in both a string of code, and an array of arguments to pass into the program
|
||||
as well as data to write to STDIN. The stdout and stderr from the process are
|
||||
then both returned seperately, along with the error code returned.
|
||||
|
||||
All of this is has no rate-limiting built in making it lightning fast as a call
|
||||
will directly start the runner process and get under way instantly.
|
||||
|
||||
The 2 stages of this process - compile and run are both run in sequence, with
|
||||
different timeouts configurable in the runners config file located in the
|
||||
data directory.
|
||||
|
||||
Requests to this endpoint can have caching enabled at 3 different levels.
|
||||
The first option is to have no caching, which is the default for all
|
||||
interpreted language. The second option is for the compiled binaries to be
|
||||
cached, which is the default for all compiled languages. The final option is
|
||||
for output to be cached, which isn't used by default but can be enabled per
|
||||
package or per request. This is done for the reason that code may choose to
|
||||
source data from /dev/(u)random or similar sources and as such may not be as
|
||||
reliable when their outputs are cached. Caching is per package and is used as
|
||||
an acceleration method to help boost performance of Piston. Cache entries are
|
||||
automatically purged after the set time, or can be manually purged through the
|
||||
API on a per package basis.
|
||||
|
||||
|
||||
== Package Manager ==
|
||||
|
||||
Piston v3 has an inbuilt package manager which is responsible for
|
||||
(un)installing different packages. Piston v3 by default has access to a single
|
||||
offical repository hosting various versions of various common languages. These
|
||||
packages and repositories conform to the specifications set out in ppman.txt
|
||||
|
||||
The Piston API service downloads the repository index whenever a `/packages`
|
||||
request is issued to a repository with the `sync` flag is set. This will cause
|
||||
the service to download the latest repostiory index off the mirror.
|
||||
|
||||
In piston there is no concept of a package being "outdated" as each package is
|
||||
a specific version of a language, and different languages can be installed in
|
||||
paralleland function without any issues. Each package should be considered the
|
||||
final version of that language. If there is a new version of a language
|
||||
available (i.e. Python 3.9.1 -> 3.9.2), a new package should be created for
|
||||
this.
|
||||
|
||||
Invidual languages can be queried from the repo using the
|
||||
`/repos/{repo}/packages/{package}/{package-version}` endpoint. This endpoint
|
||||
allows for the metadata of the package to be accessed, such as the author,
|
||||
size, checksums, dependencies, build file git url and download url.
|
||||
|
||||
To install packages, a request to `/install` can be made to the package
|
||||
endpoint and it will download and install it, making it available on the
|
||||
`/packages/{package}/{version}` endpoint.
|
||||
|
||||
There is a meta-repository name `all` which can be used to access all
|
||||
repositories.
|
||||
|
||||
Internally the install process involved downloading and unpacking the package,
|
||||
ensuring any dependencies are also downloaded and installed, mounting the
|
||||
squashfs filesystem to a folder, then overlaying it with all its dependencies
|
||||
in another folder.
|
||||
|
|
@ -1,18 +0,0 @@
|
|||
== File System Layout == [ Piston ]
|
||||
|
||||
All of pistons files are installed in the `/piston` directory. This directory
|
||||
contains all runtimes, config, packages and cache that piston uses.
|
||||
|
||||
Each package gets its own directory, where it its prebuilt binaries are
|
||||
unpacked into. This is contained within `/piston/packages`
|
||||
|
||||
The binaries folder contained within this is then symlinked into the runtimes
|
||||
directory. This is where all the different runtimes available are placed. This
|
||||
is contained within the `/piston/runtimes` directory.
|
||||
|
||||
The cache directory a directory containing all the different cache files. It is
|
||||
recommended to either sym-link this into a folder withing /tmp/ or directly
|
||||
mount it as a tmpfs filesystem.
|
||||
|
||||
Configuration is stored in a single file - piston.yaml and contains all
|
||||
documentation required for configuring the piston API
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
== Index == [ Piston ]
|
||||
|
||||
Design outlines the design of the different components and does not give a
|
||||
concrete definition of the implementation or how to use it.
|
||||
|
||||
api.txt Design of Piston API
|
||||
ppman.txt Design of the package manager's package and repository format
|
||||
|
||||
|
||||
== Glossary ==
|
||||
|
||||
Execution Job A single code run with arguments resulting in an output
|
||||
Package A version of a language bundled together into a tarball
|
||||
136
design/ppman.txt
136
design/ppman.txt
|
|
@ -1,136 +0,0 @@
|
|||
== Package Manager (ppman) == [ Piston ]
|
||||
|
||||
The package manager is the part of the API responsible for managing different
|
||||
versions of different languages, managing their installation, uninstallation
|
||||
and their dependencies. The package manager talks over the piston api and is
|
||||
built directly into piston, although has parts which are not directly built
|
||||
into the API (i.e. the repositories and the cli utility).
|
||||
|
||||
The package manager is a complex part of piston, and requires 2 different file
|
||||
specifications - the repository index file and the package file.
|
||||
|
||||
|
||||
|
||||
== Repository Index File ==
|
||||
|
||||
The piston repository is the central place where packages are hosted and
|
||||
downloaded from. This repository can either be a webserver or a local file
|
||||
containing the right content - as long as its accessable by a URL, its
|
||||
considered a valid repository by piston. A repository URL is simply a URL
|
||||
pointing to a repository index file, as set out by the following information.
|
||||
|
||||
A repository index file is a YAML file containing the keys: `schema`,
|
||||
`baseurl`, `keys` and `packages`.
|
||||
|
||||
The schema key simply should have a value of `ppman-repo-1`. This indicates the
|
||||
version and file format for the client to recieve.
|
||||
|
||||
The baseurl key contains the base url that relative URLs should be based off,
|
||||
this doesn't need to be related to the url that the repository index is hosted
|
||||
at, only the downloadable files, which are possible to split over many domains
|
||||
by using absolute paths.
|
||||
|
||||
The keys key contains a list of GPG key ids which should be used when
|
||||
verifying.
|
||||
|
||||
The packages key contains a list of packages, which contain the keys: `author`,
|
||||
`language`, `version`, `checksums`, `dependencies`, `size`, `buildfile`,
|
||||
`download` and `signature`.
|
||||
|
||||
The author field is self explainatory, it is simply the authors name and email,
|
||||
formatted similar to git's default format: `Full Name <email@address>`. If the
|
||||
repository index is automatically generated, it is best to use the commit
|
||||
author's name in here.
|
||||
|
||||
The language and version fields define the version and name of the compiler or
|
||||
interpreter contained within. The language should not include a version at all.
|
||||
In the case of python, use the name python for both python 2 and 3, using the
|
||||
version field to differentiate between the 2.
|
||||
|
||||
The checksums field is simply a map of hash types to hashes, hash types include
|
||||
md5, sha1, sha256, sha512. The digests should simply be written as lowercase
|
||||
hex characters. Only one checksum is required, but if more are supplied the
|
||||
most secure one is picked, with sha512 as the highest possible.
|
||||
|
||||
The dependencies is simply a map of language names to versions, which should be
|
||||
installed for the package to run correctly. An example of this would be
|
||||
typescript requires node to run.
|
||||
|
||||
The size field is the number of bytes the package file is in size, while
|
||||
uncompressed. This is used to determine if there is enough room, and thus
|
||||
should be accurate.
|
||||
|
||||
The buildfile field is a URL pointing to the exact build script for this build.
|
||||
This should always point to a URL either containing steps, a script or other
|
||||
means of reproducing the build. This field is purely so people can understand
|
||||
how the image was built, and to make sure you aren't packing any mallicious
|
||||
code into it.
|
||||
|
||||
The download field is a URL pointing to a place of which the package file can
|
||||
be obtained from. If this is a relative url, the baseurl will be appended to
|
||||
it. This is particularly useful if everything is stored within 1 s3 bucket, or
|
||||
you have a repository in a folder.
|
||||
|
||||
The signature field is an armored signature
|
||||
|
||||
|
||||
== Package File ==
|
||||
|
||||
Within a repository, many package files are contained. These files are
|
||||
downloaded and installed into `/piston`. They need to all follow the format
|
||||
as listed below for the API to properly utilize them.
|
||||
|
||||
A package file is a gzipped tar archive, containing 4/5 things - `run`,
|
||||
`compile` (optional), `pkg-info.json`, `lang-ver` and `lang-ver/environment`.
|
||||
Replacing lang-ver with the language name and the version respectively.
|
||||
|
||||
|
||||
The `pkg-info.json` file contains 5 different keys: `language`, `version`,
|
||||
`author`, `dependencies` and `build_platform`.
|
||||
|
||||
The language field is simply the name of the language, all lowercase and not
|
||||
containing any version number. This is important in the case of python
|
||||
specifically as python3 and python2 come under the same language, just
|
||||
different versions.
|
||||
|
||||
The version field is a sem-ver compliant version number for the interpreter
|
||||
contained wthin the package. It should be a string.
|
||||
|
||||
The author field contains the author name, and should be formatted exactly like
|
||||
shown previously in the repo index spec.
|
||||
|
||||
The dependencies field is simply a map containing packages which this package
|
||||
depends on. This should only be used when one language is a requirement for
|
||||
another, like how typescript is dependent on node. The key should be the name
|
||||
of the package, with the value being the version selector.
|
||||
|
||||
The build_platform field is used for defining which type of system the package
|
||||
was built on, this helps resolve system compatability errors. It consists of 2
|
||||
parts - environment and disto. The environment is simply one of `baremetal`,
|
||||
`docker`, `lxc` or any other type of environment you can run piston on. The
|
||||
distro is the ID of the distro as contained in /etc/os-release on the system
|
||||
which built the package. This is done to ensure system compatability,
|
||||
especially inside the offically supported Docker container.
|
||||
|
||||
|
||||
The `run` and `compile` files are used in the execution of a job, being used
|
||||
to both compile and run the source code provided. They are both treated the
|
||||
same inputs-wise but can have timeouts independently set per stage. The
|
||||
arguments fed both these files are the same, with the first argument being a
|
||||
path to the code file, and the rest being the arguments to passed in. These
|
||||
files are run in a temporary directory contained within the cache folder.
|
||||
Depending on the cache control level the code and associated files will either
|
||||
be disposed of or kept. By default only files named `binary` will be kept.
|
||||
STDIN is only passed into the `run` file, but both files have their output
|
||||
captured and returned along with their exit codes through the API
|
||||
|
||||
The `lang-ver` folder should contain any interpreter specific files, such as
|
||||
the binary to execute and any other files that may be required to run the
|
||||
interpreter/compiler contained within the package.
|
||||
|
||||
The `environment` file contained within `lang-ver` should contain export
|
||||
commands like a ~/.bashrc file should, as this is its intended purpose. Firstly
|
||||
the language which is being executed has its environment file sources, then it
|
||||
walks down the dependency tree sourcing files. The environment variables are
|
||||
eventually cached to speed up the execution process.
|
||||
|
||||
Loading…
Reference in New Issue