From 22dcad0dd92a24debc7bce10041535f251f41fb1 Mon Sep 17 00:00:00 2001 From: Thomas Hobson Date: Sat, 6 Mar 2021 18:31:13 +1300 Subject: [PATCH] design: remove as outdated --- ARCHITECTURE.TXT | 71 ------------------------- design/api.txt | 75 -------------------------- design/fs.txt | 18 ------- design/index.txt | 13 ----- design/ppman.txt | 136 ----------------------------------------------- 5 files changed, 313 deletions(-) delete mode 100644 ARCHITECTURE.TXT delete mode 100644 design/api.txt delete mode 100644 design/fs.txt delete mode 100644 design/index.txt delete mode 100644 design/ppman.txt diff --git a/ARCHITECTURE.TXT b/ARCHITECTURE.TXT deleted file mode 100644 index edf52d3..0000000 --- a/ARCHITECTURE.TXT +++ /dev/null @@ -1,71 +0,0 @@ -== Breif == [ Piston ] - -This document covers the overall architecture of Piston v3, and not the -individual components and their implementations. - -In Piston v2 we saw 2 ways of using piston - through the CLI and the API. -These 2 methods would call the same load of bash scripts contained within -a LXC container which would then resolve your request. -There are a number of issues with this approach: - 1. This uses bash - which isn't the best language for performance - 2. It relied on calling through a `lxc-attach` command to access - inside the container - 3. This isn't easy to distribute - 4. It was difficult to add languages - having to edit 4 different files - in 4 different places to add a single language - -Piston v3 aims to tackle these 4 issues. -Firstly, v3 will be less reliant on bash, only using it as an option -for running different interpreters. -Secondly, v3 can run on bare-metal or in a container, but a core API will be -exposed from within the container, instead of running external to it. -Thirdly, v3 will provide a simple docker container, which will expose both the -piston API, and run all the runners within it. -Finally, v3 will provide a repository of precompiled language executers, so its -1 command away from installing a language - - - -== Piston API == - -Piston v3 exposes a REST API, allowing the user to control the entire thing -over one simple JSON based protocol. This eliminates the need to connect into -the container to do maintainace such as adding new languages, or checking -usage statistics. - -See design/api.txt for more information. - - - -== Package Manager == - -Piston v3 includes a package manager right out of the box. The package manager -manages the different languages and versions that it can run. -The package manager is hooked directly into the API and addresses our point -of easy distibution, as users now can easily enable/disable different -components built into piston as they see fit. - -See design/ppman.txt for more information. - - - -== Runtime Environment == - -The new architecture moves to a more bare-metal approach, where the code can be -run without the overhead of a container manager such as LXC or Docker, making -piston much easier to manage this way - -It is still possible to run Piston v3 in a contain, but now a container engine -is not required for usage, however it is still recommended. - - - -== Proxy API == - -The in-container API is more powerful than a simple execution API and thus -should be limited, however to keep the weight down, and speed up there is a -reference implementation of a proxy API included, which passes through -execution commands to many different piston instances and allows for -security with rate limiting and API keys. - -See design/proxy.txt \ No newline at end of file diff --git a/design/api.txt b/design/api.txt deleted file mode 100644 index a9e7e63..0000000 --- a/design/api.txt +++ /dev/null @@ -1,75 +0,0 @@ -== Piston API == [ Piston ] - -When we speak of piston, what we actually talk about is the Piston API. -This API provides unrestricted, unlimited access to managing piston and -thus shouldn't be publicly exposed. This API is comparable to one of the -docker engine, where everything regarding control of docker goes directly -through the api. - -The API is responsible for managing the execution lifecycle of any given -job, as well as managing the different languages which it can execute a -job in. - - - -== Job Execution == - -Piston v3 exposes an endpoint per package `/execute`, which when called takes -in both a string of code, and an array of arguments to pass into the program -as well as data to write to STDIN. The stdout and stderr from the process are -then both returned seperately, along with the error code returned. - -All of this is has no rate-limiting built in making it lightning fast as a call -will directly start the runner process and get under way instantly. - -The 2 stages of this process - compile and run are both run in sequence, with -different timeouts configurable in the runners config file located in the -data directory. - -Requests to this endpoint can have caching enabled at 3 different levels. -The first option is to have no caching, which is the default for all -interpreted language. The second option is for the compiled binaries to be -cached, which is the default for all compiled languages. The final option is -for output to be cached, which isn't used by default but can be enabled per -package or per request. This is done for the reason that code may choose to -source data from /dev/(u)random or similar sources and as such may not be as -reliable when their outputs are cached. Caching is per package and is used as -an acceleration method to help boost performance of Piston. Cache entries are -automatically purged after the set time, or can be manually purged through the -API on a per package basis. - - -== Package Manager == - -Piston v3 has an inbuilt package manager which is responsible for -(un)installing different packages. Piston v3 by default has access to a single -offical repository hosting various versions of various common languages. These -packages and repositories conform to the specifications set out in ppman.txt - -The Piston API service downloads the repository index whenever a `/packages` -request is issued to a repository with the `sync` flag is set. This will cause -the service to download the latest repostiory index off the mirror. - -In piston there is no concept of a package being "outdated" as each package is -a specific version of a language, and different languages can be installed in -paralleland function without any issues. Each package should be considered the -final version of that language. If there is a new version of a language -available (i.e. Python 3.9.1 -> 3.9.2), a new package should be created for -this. - -Invidual languages can be queried from the repo using the -`/repos/{repo}/packages/{package}/{package-version}` endpoint. This endpoint -allows for the metadata of the package to be accessed, such as the author, -size, checksums, dependencies, build file git url and download url. - -To install packages, a request to `/install` can be made to the package -endpoint and it will download and install it, making it available on the -`/packages/{package}/{version}` endpoint. - -There is a meta-repository name `all` which can be used to access all -repositories. - -Internally the install process involved downloading and unpacking the package, -ensuring any dependencies are also downloaded and installed, mounting the -squashfs filesystem to a folder, then overlaying it with all its dependencies -in another folder. diff --git a/design/fs.txt b/design/fs.txt deleted file mode 100644 index 030d563..0000000 --- a/design/fs.txt +++ /dev/null @@ -1,18 +0,0 @@ -== File System Layout == [ Piston ] - -All of pistons files are installed in the `/piston` directory. This directory -contains all runtimes, config, packages and cache that piston uses. - -Each package gets its own directory, where it its prebuilt binaries are -unpacked into. This is contained within `/piston/packages` - -The binaries folder contained within this is then symlinked into the runtimes -directory. This is where all the different runtimes available are placed. This -is contained within the `/piston/runtimes` directory. - -The cache directory a directory containing all the different cache files. It is -recommended to either sym-link this into a folder withing /tmp/ or directly -mount it as a tmpfs filesystem. - -Configuration is stored in a single file - piston.yaml and contains all -documentation required for configuring the piston API \ No newline at end of file diff --git a/design/index.txt b/design/index.txt deleted file mode 100644 index 685222a..0000000 --- a/design/index.txt +++ /dev/null @@ -1,13 +0,0 @@ -== Index == [ Piston ] - -Design outlines the design of the different components and does not give a -concrete definition of the implementation or how to use it. - -api.txt Design of Piston API -ppman.txt Design of the package manager's package and repository format - - -== Glossary == - -Execution Job A single code run with arguments resulting in an output -Package A version of a language bundled together into a tarball \ No newline at end of file diff --git a/design/ppman.txt b/design/ppman.txt deleted file mode 100644 index b4920db..0000000 --- a/design/ppman.txt +++ /dev/null @@ -1,136 +0,0 @@ -== Package Manager (ppman) == [ Piston ] - -The package manager is the part of the API responsible for managing different -versions of different languages, managing their installation, uninstallation -and their dependencies. The package manager talks over the piston api and is -built directly into piston, although has parts which are not directly built -into the API (i.e. the repositories and the cli utility). - -The package manager is a complex part of piston, and requires 2 different file -specifications - the repository index file and the package file. - - - -== Repository Index File == - -The piston repository is the central place where packages are hosted and -downloaded from. This repository can either be a webserver or a local file -containing the right content - as long as its accessable by a URL, its -considered a valid repository by piston. A repository URL is simply a URL -pointing to a repository index file, as set out by the following information. - -A repository index file is a YAML file containing the keys: `schema`, -`baseurl`, `keys` and `packages`. - -The schema key simply should have a value of `ppman-repo-1`. This indicates the -version and file format for the client to recieve. - -The baseurl key contains the base url that relative URLs should be based off, -this doesn't need to be related to the url that the repository index is hosted -at, only the downloadable files, which are possible to split over many domains -by using absolute paths. - -The keys key contains a list of GPG key ids which should be used when -verifying. - -The packages key contains a list of packages, which contain the keys: `author`, -`language`, `version`, `checksums`, `dependencies`, `size`, `buildfile`, -`download` and `signature`. - -The author field is self explainatory, it is simply the authors name and email, -formatted similar to git's default format: `Full Name `. If the -repository index is automatically generated, it is best to use the commit -author's name in here. - -The language and version fields define the version and name of the compiler or -interpreter contained within. The language should not include a version at all. -In the case of python, use the name python for both python 2 and 3, using the -version field to differentiate between the 2. - -The checksums field is simply a map of hash types to hashes, hash types include -md5, sha1, sha256, sha512. The digests should simply be written as lowercase -hex characters. Only one checksum is required, but if more are supplied the -most secure one is picked, with sha512 as the highest possible. - -The dependencies is simply a map of language names to versions, which should be -installed for the package to run correctly. An example of this would be -typescript requires node to run. - -The size field is the number of bytes the package file is in size, while -uncompressed. This is used to determine if there is enough room, and thus -should be accurate. - -The buildfile field is a URL pointing to the exact build script for this build. -This should always point to a URL either containing steps, a script or other -means of reproducing the build. This field is purely so people can understand -how the image was built, and to make sure you aren't packing any mallicious -code into it. - -The download field is a URL pointing to a place of which the package file can -be obtained from. If this is a relative url, the baseurl will be appended to -it. This is particularly useful if everything is stored within 1 s3 bucket, or -you have a repository in a folder. - -The signature field is an armored signature - - -== Package File == - -Within a repository, many package files are contained. These files are -downloaded and installed into `/piston`. They need to all follow the format -as listed below for the API to properly utilize them. - -A package file is a gzipped tar archive, containing 4/5 things - `run`, -`compile` (optional), `pkg-info.json`, `lang-ver` and `lang-ver/environment`. -Replacing lang-ver with the language name and the version respectively. - - -The `pkg-info.json` file contains 5 different keys: `language`, `version`, -`author`, `dependencies` and `build_platform`. - -The language field is simply the name of the language, all lowercase and not -containing any version number. This is important in the case of python -specifically as python3 and python2 come under the same language, just -different versions. - -The version field is a sem-ver compliant version number for the interpreter -contained wthin the package. It should be a string. - -The author field contains the author name, and should be formatted exactly like -shown previously in the repo index spec. - -The dependencies field is simply a map containing packages which this package -depends on. This should only be used when one language is a requirement for -another, like how typescript is dependent on node. The key should be the name -of the package, with the value being the version selector. - -The build_platform field is used for defining which type of system the package -was built on, this helps resolve system compatability errors. It consists of 2 -parts - environment and disto. The environment is simply one of `baremetal`, -`docker`, `lxc` or any other type of environment you can run piston on. The -distro is the ID of the distro as contained in /etc/os-release on the system -which built the package. This is done to ensure system compatability, -especially inside the offically supported Docker container. - - -The `run` and `compile` files are used in the execution of a job, being used -to both compile and run the source code provided. They are both treated the -same inputs-wise but can have timeouts independently set per stage. The -arguments fed both these files are the same, with the first argument being a -path to the code file, and the rest being the arguments to passed in. These -files are run in a temporary directory contained within the cache folder. -Depending on the cache control level the code and associated files will either -be disposed of or kept. By default only files named `binary` will be kept. -STDIN is only passed into the `run` file, but both files have their output -captured and returned along with their exit codes through the API - -The `lang-ver` folder should contain any interpreter specific files, such as -the binary to execute and any other files that may be required to run the -interpreter/compiler contained within the package. - -The `environment` file contained within `lang-ver` should contain export -commands like a ~/.bashrc file should, as this is its intended purpose. Firstly -the language which is being executed has its environment file sources, then it -walks down the dependency tree sourcing files. The environment variables are -eventually cached to speed up the execution process. -