BREAKING: replace custom build scripts with nix

General:
- Switched to yarn to better work with nix-based tooling
- Switched package system to use nix. This stops double dependencies and slow cloud compile times, while providing more compile/runtime support to the Nix project
- Removed container builder in favor of internal container tooling
- Package versions no-longer need to be SemVer compliant
- Removed "piston package spec" files, replaced with nix-flake based runtimes
- Exported nosocket and piston-api as packages within the nix-flake
- Removed repo container
- Switched docker building to nix-based container outputting
- Removed docker compose as this is a single container
- Removed package commands from CLI

Packages:
- Move bash, clojure, cobol, node, python2, python3 to new format
- Remainder of packages still need to be moved

v2 API:
- Removed "version" specifier. To select specific versions, use the v3 api
- Removed "/package" endpoints as this doesn't work with the new nix-based system

v3 API:
- Duplicate of v2 API, except instead of passing in a language name an ID is used intead.
This commit is contained in:
Thomas Hobson 2022-01-30 18:41:24 +13:00
parent e06b59d82c
commit 564da5a7eb
No known key found for this signature in database
GPG key ID: 9F1FD9D87950DB6F
110 changed files with 2293 additions and 2793 deletions

8
packages/.gitignore vendored
View file

@ -1,8 +0,0 @@
*/*/*
*.pkg.tar.gz
!*/*/metadata.json
!*/*/build.sh
!*/*/environment
!*/*/run
!*/*/compile
!*/*/test.*

View file

@ -1,100 +0,0 @@
# Contributing packages to the Piston Repository
## Naming Languages
Languages should be named after their interpreters, and the command line binaries you call. The language version should use semantic versioning.
For example, the full name of the standard python interpreter is `CPython`, however we would name it `python`, after the main binary which it provides.
In the example of NodeJS, we would call this `node`, after the main binary.
## Creating new languages
See [deno/1.7.5/](deno/1.7.5/) or any other directory for examples.
1. Create a new branch on your fork of engineer-man/piston
2. Create directories named `[language]/[version]`. See Naming Languages for how to determine the name for your language
3. Create a file named `build.sh`, adding a shebang for bash `#!/bin/bash` on the first line.
In this file put any steps to compile the specified langauge.
This script should download sources, compile sources and output binaries. They should be dumped into the current working directory, removing any files which aren't required in the process.
4. Create a file named `run`, containing bash script to run the interpreter.
The first argument given to this script (`$1`) is the name of the main file, with the remaining ones as program arguments.
STDIN is piped directly into the run file, and as such nothing special is required to deal with STDIN, except leaving it open.
5. Create a file named `compile`, containing bash script to compile sources into binaries. This is only required if the language requires a compling stage.
The first argument is always the main file, followed the names of the other files as additional arguements. If the language does not require a compile stage, don't create a compile file.
6. Create a file named `environment`, containing `export` statements which edit the environment variables accordingly. The `$PWD` variable should be used, and is set inside the package directory when running on the target system.
7. Create a test script starting with test, with the file extension of the language. This script should simply output the phrase `OK`. For example, for mono we would create `test.cs` with the content:
```cs
using System;
public class Test
{
public static void Main(string[] args)
{
Console.WriteLine("OK");
}
}
```
8. Create a `metadata.json` file which contains metadata about the language and interpreter. This simply contains the language name, as in the folder name, the version as in the folder name, aliases that can be used to call this package, limit overrides (if any) that can be used to override the default constraints and finally a dependencies map.
The dependencies map contains the keys as language names, and the values as semver selectors for packages.
```json
{
"language": "deno",
"version": "1.7.5",
"dependencies": {},
"aliases": ["deno-ts", "deno-js"]
}
```
If the interpreter/compiler provides multiple languages, then the provides property should be used:
```json
{
"language": "dotnet",
"version": "5.0.201",
"provides": [
{
"language": "basic.net",
"aliases": [
"basic",
"visual-basic",
"visual-basic.net",
"vb",
"vb.net",
"vb-dotnet",
"dotnet-vb",
"basic-dotnet",
"dotnet-basic"
],
"limit_overrides": { "max_process_count": 128 }
},
{
"language": "fsi",
"aliases": [
"fsx",
"fsharp-interactive",
"f#-interactive",
"dotnet-fsi",
"fsi-dotnet",
"fsi.net"
]
}
]
}
```
9. Test your package builds with running `make [language]-[version].pkg.tar.gz`.
If it all goes to plan, you should have a file named `[language]-[version].pkg.tar.gz`, in this case you're good to go, albeit it is preferable to test the package locally as follows
```shell
./piston build-pkg [package] [version]
./piston ppman install [package]=[version]
./piston run [package] -l [version] packages/[package]/[version]/test.*
```
10. Commit your changes, using message format of `pkg([language]-[version]): Added [language] [version]`
Any additional commits regarding this package should start with `pkg([language]-[version]): `
11. Create a pull request (currently to v3 branch), referencing an Issue number (if there is one associated).

View file

@ -1,25 +0,0 @@
PACKAGES=$(subst /,-,$(shell find * -maxdepth 1 -mindepth 1 -type d))
BUILD_PLATFORM=$(or ${PLATFORM},baremetal-$(shell grep -oP "^ID=\K.+" /etc/os-release))
help:
@echo "You probably don't want to build all package"
@echo "If you do run $`make build-all$`"
@echo
@echo "Run $`make [language]-[version].pkg.tar.gz$` to build a specific language"
build build-all: $(addsuffix .pkg.tar.gz, ${PACKAGES})
define PKG_RULE
$(1).pkg.tar.gz: $(subst -,/,$(1)) $(subst -,/,$(1))/pkg-info.json
cd $$< && chmod +x ./build.sh && ./build.sh
rm -f $$@
tar czf $$@ -C $$< .
endef
$(foreach pkg,$(PACKAGES),$(eval $(call PKG_RULE,$(pkg))))
%/pkg-info.json: %/metadata.json
jq '.build_platform="${BUILD_PLATFORM}"' $< > $@

View file

@ -1,7 +0,0 @@
# Piston Package Build Scripts
## Building
```bash
make build-[name]-[version]
```

View file

@ -1,21 +0,0 @@
#!/usr/bin/env bash
# Put instructions to build your package in here
PREFIX=$(realpath $(dirname $0))
mkdir -p build
cd build
curl "https://ftp.gnu.org/gnu/bash/bash-5.1.tar.gz" -o bash.tar.gz
tar xzf bash.tar.gz --strip-components=1
# === autoconf based ===
./configure --prefix "$PREFIX"
make -j$(nproc)
make install -j$(nproc)
cd ../
rm -rf build

View file

@ -1,4 +0,0 @@
#!/usr/bin/env bash
# Put 'export' statements here for environment variables
export PATH=$PWD/bin:$PATH

View file

@ -1,5 +0,0 @@
{
"language": "bash",
"version": "5.1.0",
"aliases": ["sh"]
}

View file

@ -1,4 +0,0 @@
#!/usr/bin/env bash
# Put instructions to run the runtime
bash "$@"

View file

@ -1 +0,0 @@
echo "OK"

View file

@ -1,43 +0,0 @@
#!/usr/bin/env bash
# Installation location
PREFIX=$(realpath $(dirname $0))
# Clojure depends on Java (build and runtime)
mkdir -p java
cd java
curl "https://download.java.net/java/GA/jdk15.0.2/0d1cfde4252546c6931946de8db48ee2/7/GPL/openjdk-15.0.2_linux-x64_bin.tar.gz" -o java.tar.gz
tar xzf java.tar.gz --strip-components=1
rm java.tar.gz
cd ..
# Clojure depends on Maven (build)
mkdir -p maven
cd maven
curl "https://apache.claz.org/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz" -o maven.tar.gz
tar xzf maven.tar.gz --strip-components=1
rm maven.tar.gz
cd ..
# Adding java and maven to the path for building
export PATH=$PWD/java/bin:$PWD/maven/bin:$PATH
export JAVA_HOME=$PWD/java
# Clojure download
mkdir -p build
cd build
git clone -q "https://github.com/clojure/clojure.git" .
git checkout -b clojure-1.10.3 aaf73b12467df80f5db3e086550a33fee0e1b39e # commit for 1.10.3 release
# Build using maven
mvn -Plocal -Dmaven.test.skip=true package
# Get ridda that m2 bloat from Maven and remove Maven itself
cd ../
rm -rf ~/.m2
rm -rf maven/
# Move the jar for easier reference and cleanup
mkdir -p bin
mv build/clojure.jar bin
rm -rf build

View file

@ -1,6 +0,0 @@
#!/usr/bin/env bash
# Clojure requires JAVA_HOME to be set and java binary to be in the path
export JAVA_HOME=$PWD/java
export CLOJURE_PATH=$PWD/bin
export PATH=$PWD/java/bin:$PATH

View file

@ -1,5 +0,0 @@
{
"language": "clojure",
"version": "1.10.3",
"aliases": ["clojure", "clj"]
}

View file

@ -1,4 +0,0 @@
#!/usr/bin/env bash
# Run clojure with Java referencing the clojure jar location
java -jar $CLOJURE_PATH/clojure.jar "$@"

View file

@ -1,5 +0,0 @@
(ns clojure.examples.main
(:gen-class))
(defn main []
(println "OK"))
(main)

View file

@ -1,20 +0,0 @@
#!/usr/bin/env bash
# Put instructions to build your package in here
PREFIX=$(realpath $(dirname $0))
mkdir -p build
cd build
curl -OL "https://downloads.sourceforge.net/project/gnucobol/gnucobol/3.1/gnucobol-3.1.2.tar.xz"
tar xf gnucobol-3.1.2.tar.xz --strip-components=1
# === autoconf based ===
./configure --prefix "$PREFIX" --without-db
make -j$(nproc)
make install -j$(nproc)
cd ../
rm -rf build

View file

@ -1,4 +0,0 @@
#!/usr/bin/env bash
cobc -o binary --free -x -L lib "$@"
chmod +x binary

View file

@ -1,5 +0,0 @@
#!/usr/bin/env bash
export PATH=$PWD/bin:$PATH
export LD_LIBRARY_PATH=$PWD/lib

View file

@ -1,5 +0,0 @@
{
"language": "cobol",
"version": "3.1.2",
"aliases": ["cob"]
}

View file

@ -1,5 +0,0 @@
#!/usr/bin/env bash
shift
./binary "$@"

View file

@ -1,8 +0,0 @@
*> Test Program
identification division.
program-id. ok-test.
procedure division.
display "OK"
goback.
end program ok-test.

41
packages/flake.lock generated
View file

@ -1,41 +0,0 @@
{
"nodes": {
"flake-utils": {
"locked": {
"lastModified": 1631561581,
"narHash": "sha256-3VQMV5zvxaVLvqqUrNz3iJelLw30mIVSfZmAaauM3dA=",
"owner": "numtide",
"repo": "flake-utils",
"rev": "7e5bf3925f6fbdfaf50a2a7ca0be2879c4261d19",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "flake-utils",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1633356775,
"narHash": "sha256-UBhRo1qy8xpOGTrjf7r2SFcULkFM+Wn4kchxN1ToDxs=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "38501bec61c1e9447aa4ffc01ba07c40f4515327",
"type": "github"
},
"original": {
"id": "nixpkgs",
"type": "indirect"
}
},
"root": {
"inputs": {
"flake-utils": "flake-utils",
"nixpkgs": "nixpkgs"
}
}
},
"root": "root",
"version": 7
}

View file

@ -1,47 +0,0 @@
{
description = "Piston packages repo";
inputs.flake-utils.url = "github:numtide/flake-utils";
outputs = { self, nixpkgs, flake-utils }:
let
system = "x86_64-linux";
pkgs = nixpkgs.legacyPackages.${system};
args = {
inherit pkgs;
piston = {
mkRuntime = {
language,
version,
runtime,
run,
compile? null,
aliases? []
}: let
packageName = "${runtime}-${language}";
compileFile = if compile != null then
pkgs.writeShellScript "compile" compile
else null;
runFile = pkgs.writeShellScript "run" run;
metadataFile = pkgs.writeText "metadata.json" (builtins.toJSON {
inherit language version runtime aliases;
});
in pkgs.runCommandNoCC packageName {}
(
''
mkdir -p $out/piston
ln -s ${runFile} $out/piston/run
ln -s ${metadataFile} $out/piston/metadata.json
'' + (
if compileFile != null then
''
ln -s ${compileFile} $out/piston/compile
'' else "")
);
};
};
in {
piston = {
"node-javascript" = import ./node-javascript.nix args;
};
};
}

View file

@ -1,64 +0,0 @@
#!/usr/bin/env bash
if [[ $# -lt 3 ]]; then
echo "Usage: $0 [name] [version] [source]"
echo ""
echo "Initializes an empty package"
exit 1
fi
NAME=$1
VERSION=$2
SOURCE=$3
DIR=$NAME/$VERSION
mkdir -p $DIR
build_instructions(){
echo 'PREFIX=$(realpath $(dirname $0))'
echo
echo 'mkdir -p build'
echo
echo 'cd build'
echo
echo "curl \"$SOURCE\" -o $NAME.tar.gz"
echo
echo "tar xzf $NAME.tar.gz --strip-components=1"
echo
echo "# === autoconf based ==="
echo './configure --prefix "$PREFIX"'
echo
echo 'make -j$(nproc)'
echo 'make install -j$(nproc)'
echo 'cd ../'
echo 'rm -rf build'
}
cd $DIR
for name in build.sh environment run compile; do
echo "#!/usr/bin/env bash" > "$name"
echo "" >> "$name"
done
echo "# Put instructions to build your package in here" >> build.sh
echo ""
build_instructions >> build.sh
echo "# Put 'export' statements here for environment variables" >> environment
echo "export PATH=\$PWD/bin:\$PATH" >> environment
echo "# Put instructions to run the runtime" >> run
echo "$NAME-$VERSION \"\$@\"" >> run
echo "# Put instructions to compile source code, remove this file if the language does not require this stage" >> compile
jq '.language = "'$NAME'" | .version = "'$VERSION'" | .aliases = []' <<< "{}" > metadata.json
cd - > /dev/null
echo $DIR

View file

@ -1,17 +0,0 @@
{pkgs, piston}:
piston.mkRuntime {
language = "javascript";
version = pkgs.nodejs.version;
runtime = "node";
aliases = [
"node-js"
"node-javascript"
"js"
];
run = ''
${pkgs.nodejs}/bin/node "$@"
'';
}

View file

@ -1,4 +0,0 @@
#!/bin/bash
curl "https://nodejs.org/dist/v15.10.0/node-v15.10.0-linux-x64.tar.xz" -o node.tar.xz
tar xf node.tar.xz --strip-components=1
rm node.tar.xz

View file

@ -1 +0,0 @@
export PATH=$PWD/bin:$PATH

View file

@ -1,10 +0,0 @@
{
"language": "node",
"version": "15.10.0",
"provides": [
{
"language": "javascript",
"aliases": ["node-javascript", "node-js", "javascript", "js"]
}
]
}

View file

@ -1,3 +0,0 @@
#!/bin/bash
node "$@"

View file

@ -1 +0,0 @@
console.log('OK');

View file

@ -1,4 +0,0 @@
#!/bin/bash
curl "https://nodejs.org/dist/v16.3.0/node-v16.3.0-linux-x64.tar.xz" -o node.tar.xz
tar xf node.tar.xz --strip-components=1
rm node.tar.xz

View file

@ -1 +0,0 @@
export PATH=$PWD/bin:$PATH

View file

@ -1,10 +0,0 @@
{
"language": "node",
"version": "16.3.0",
"provides": [
{
"language": "javascript",
"aliases": ["node-javascript", "node-js", "javascript", "js"]
}
]
}

View file

@ -1,3 +0,0 @@
#!/bin/bash
node "$@"

View file

@ -1 +0,0 @@
console.log('OK');

View file

@ -1,23 +0,0 @@
#!/bin/bash
PREFIX=$(realpath $(dirname $0))
mkdir -p build
cd build
curl "https://www.python.org/ftp/python/2.7.18/Python-2.7.18.tgz" -o python.tar.gz
tar xzf python.tar.gz --strip-components=1
rm python.tar.gz
./configure --prefix "$PREFIX" --with-ensurepip=install
make -j$(nproc)
make install -j$(nproc)
cd ..
rm -rf build
bin/pip2 install -U pip==20.3.*
# Upgrade pip to latest supported version
bin/pip2 install numpy scipy pycrypto whoosh bcrypt passlib

View file

@ -1 +0,0 @@
export PATH=$PWD/bin:$PATH

View file

@ -1,5 +0,0 @@
{
"language": "python2",
"version": "2.7.18",
"aliases": ["py2", "python2"]
}

View file

@ -1,3 +0,0 @@
#!/bin/bash
python2.7 "$@"

View file

@ -1 +0,0 @@
print "OK"

View file

@ -1,22 +0,0 @@
#!/bin/bash
PREFIX=$(realpath $(dirname $0))
mkdir -p build
cd build
curl "https://www.python.org/ftp/python/3.10.0/Python-3.10.0a7.tgz" -o python.tar.gz
tar xzf python.tar.gz --strip-components=1
rm python.tar.gz
./configure --prefix "$PREFIX" --with-ensurepip=install
make -j$(nproc)
make install -j$(nproc)
cd ..
rm -rf build
# This is alpha version, hence most of the libraries are not compatible with python3.10.0
# bin/pip3 install numpy scipy pandas pycrypto whoosh bcrypt passlib

View file

@ -1 +0,0 @@
export PATH=$PWD/bin:$PATH

View file

@ -1,5 +0,0 @@
{
"language": "python",
"version": "3.10.0-alpha.7",
"aliases": ["py", "py3", "python3"]
}

View file

@ -1,3 +0,0 @@
#!/bin/bash
python3.10 "$@"

View file

@ -1,7 +0,0 @@
working = True
match working:
case True:
print("OK")
case False:
print()

View file

@ -1,21 +0,0 @@
#!/bin/bash
PREFIX=$(realpath $(dirname $0))
mkdir -p build
cd build
curl "https://www.python.org/ftp/python/3.5.10/Python-3.5.10.tgz" -o python.tar.gz
tar xzf python.tar.gz --strip-components=1
rm python.tar.gz
./configure --prefix "$PREFIX" --with-ensurepip=install
make -j$(nproc)
make install -j$(nproc)
cd ..
rm -rf build
bin/pip3 install numpy scipy pandas pycrypto whoosh bcrypt passlib

View file

@ -1 +0,0 @@
export PATH=$PWD/bin:$PATH

View file

@ -1,5 +0,0 @@
{
"language": "python",
"version": "3.5.10",
"aliases": ["py", "py3", "python3"]
}

View file

@ -1,3 +0,0 @@
#!/bin/bash
python3.5 "$@"

View file

@ -1 +0,0 @@
print("OK")

View file

@ -1,21 +0,0 @@
#!/bin/bash
PREFIX=$(realpath $(dirname $0))
mkdir -p build
cd build
curl "https://www.python.org/ftp/python/3.9.1/Python-3.9.1.tgz" -o python.tar.gz
tar xzf python.tar.gz --strip-components=1
rm python.tar.gz
./configure --prefix "$PREFIX" --with-ensurepip=install
make -j$(nproc)
make install -j$(nproc)
cd ..
rm -rf build
bin/pip3 install numpy scipy pandas pycrypto whoosh bcrypt passlib

View file

@ -1 +0,0 @@
export PATH=$PWD/bin:$PATH

View file

@ -1,5 +0,0 @@
{
"language": "python",
"version": "3.9.1",
"aliases": ["py", "py3", "python3"]
}

View file

@ -1,3 +0,0 @@
#!/bin/bash
python3.9 "$@"

View file

@ -1 +0,0 @@
print("OK")

View file

@ -1,21 +0,0 @@
#!/bin/bash
PREFIX=$(realpath $(dirname $0))
mkdir -p build
cd build
curl "https://www.python.org/ftp/python/3.9.4/Python-3.9.4.tgz" -o python.tar.gz
tar xzf python.tar.gz --strip-components=1
rm python.tar.gz
./configure --prefix "$PREFIX" --with-ensurepip=install
make -j$(nproc)
make install -j$(nproc)
cd ..
rm -rf build
bin/pip3 install numpy scipy pandas pycrypto whoosh bcrypt passlib sympy

View file

@ -1 +0,0 @@
export PATH=$PWD/bin:$PATH

View file

@ -1,5 +0,0 @@
{
"language": "python",
"version": "3.9.4",
"aliases": ["py", "py3", "python3"]
}

View file

@ -1,3 +0,0 @@
#!/bin/bash
python3.9 "$@"

View file

@ -1,18 +0,0 @@
execute = (execute_with := lambda *a, **k: lambda f: f(*a, **k))()
@int
@execute
class n: __int__ = lambda _: 69
@execute
class cout: __lshift__ = print
@execute_with(n)
def output(n):
return "OK"
cout << output

View file

@ -1,21 +0,0 @@
#!/usr/bin/env bash
AUTH_HEADER="Authorization: $API_KEY"
for test_file in */*/test.*
do
IFS='/' read -ra test_parts <<< "$test_file"
IFS='.' read -ra file_parts <<< "$(basename $test_file)"
language=${file_parts[1]}
lang_ver=${test_parts[1]}
test_src=$(python3 -c "import json; print(json.dumps(open('$test_file').read()))")
json='{"language":"'$language'","version":"'$lang_ver'","files":[{"content":'$test_src'}]}'
result=$(curl -s -XPOST -H "Content-Type: application/json" -d "$json" https://emkc.org/api/v2/piston/execute -H $AUTH_HEADER)
echo "==$test_file: $language-$lang_ver=="
#jq '.' <<<"$result"
jq -r 'if (.run.stdout | contains("OK") ) then (.run.stdout) else (.compile.output + .run.output) end' <<<$result
done