I’ve been toying with a few things to use as a API to write drone plugins…
Starting with the parsing/reading part. This is how it looks now, I will try to
write a few plugins with this to see how it feel.
https://github.com/thomasf/drone-plugins-go/blob/master/plugins/
Look at the example test for a simple usage pattern
So, lets talk about what features a Go API for drone plugins should have first.
Since I am planning to port a bunch of my own old (0.4) plugins and write new
ones soon I want to get this done first so that I can write it all without.
Since I have been experimenting and thinking about it for a while now I have
come up with a list that is starting to feel quite solid.
It’s going to be a small package in any case, even if there are external
libraries which could be a good fit it’s probably better to just pull the code
in and adapt it for the specific needs of the plugins API. It’s always possible
to vendor small things under internal/ as well.
As long as it doesn’t result in an inconvenient API.
If one wants to use DRONE_REPO_NAME it should only require a function call like
plugins.DroneRepoNameVar(&myRepoNameVar) or similar.
When an API consumer adds a flag named “my-cool-name” the package should automatically associate PLUGIN_MY_COOL_NAME with it as well.
It must also be possible to manually specify exceptions for things like AWS_SECRET_KEY.
StringSliceFlag, StringMapFlag and StructFlag are the only ones I can think of
right now.
In case of unparsable input (giving a string to an integer field…) the package
must detect if it’s being run as a stand alone command line program or under
drone. If the program is running under drone it must try as hard as possible to
produce an error message which reproduces the correct field name used for input
in the .drone.yml pipeline.
Basic logging functions with some specialization. One example could be a FatalFlagf(&flagvar, fmtstring, …) which automatically prints an validation error output in the same format as the data type parsing error.
This is the only feature I have not prototyped myself yet so I don’t have a
very clear image of which features are suitable.
After a specific discussion and further looking at my my .drone.yml’s and
plugin code I am fairly sure that some mechanism for a plugin to run itself
many times probably is a good idea to do from inside the plugin.
The plugin should call some package function like .Matrix() or
.ParallellMatrix() to enable the support.
The .drone.yml could like something like this:
deb:
image: deb
matrix:
- OS: debian
DEPENDS: debianspecificpackage
- OS: ubuntu
DEPENDS: ubuntuspecificpackage
- ARCH: amd64
GOARCH: amd64
- ARCH: i386
GOARCH: 386
arch: $${ARCH}
files:
usr/bin/myfile: build/out-$${GOARCH}
depends: $DEPENDS
out: package-$${VERSION}-$${OS}.deb
The least intrusive way to make this a transparent functonality the plugin
intialization should check PLUGIN_MATRIX and if it exist collect all defined
flags and just relaunch itself as many times as required to complete the
matrix.
Heres an example of how my current prototype sacrifices a bit of referential
saftey in favour for a slightly less noisy API.
Current way
flag.StringVar(&c.Name,"name", "default value", "your name")
plugins.FlagEnvNames(&c.Name, "", "alt_name")
plugins.Parse()
if c.Name == "My name" {
plugins.ErrorFlagf(&c.Name, "that is not your name")
}
Possible way, a bit safer but also requires the package consumer to keep track
of an additional nameFlag-instance.
nameFlag := plugins.
StringVar(&c.Name, "name", "default value", "your name").
Env("", "alt_env_name")
plugins.Parse()
if c.Name == "My name" {
nameFlag.Errorf("that is not your name")
// or create a new instance losing the referntial saftey from when it was bound
plugins.GetFlag(&c.Name).Errorf("that is...")
}
The first version is reasonably safe when it comes to validating references.
plugins.FlagEnvNames(...) will only be allowd to be called afer a flag has
been assigned to the variable and before plugins.Parse().
Every flag modification/binding function given bad references before Parse()
will cause a early termination and shout about “programmingError” or similar.
The plugins.ErrorFlagf(&ref, ...) will print even if the
given reference doesnt have an associated flag by omitting the flag name.
I think there will be a debug mode which reconfigures the logger use the
[file.go:lineno] xxx logging format and start printing any debug level logging
(plugins.Debug(...)). Un debug mode the plugins.ErrorFlagf(&ref, ...) will
cause the program to terminate.
Here’s a quick example of porting drone-downstream to my current prototype API.
The main focus is to investigate how they differ in look and feel, all the
mentioned error print formatting and other features which brings value under
the hood is yet to be added.
package main
import (
"fmt"
"os"
"github.com/Sirupsen/logrus"
"github.com/joho/godotenv"
"github.com/urfave/cli"
)
var build = "0" // build number set at compile-time
func main() {
app := cli.NewApp()
app.Name = "downstream plugin"
app.Usage = "downstream plugin"
app.Action = run
app.Version = fmt.Sprintf("1.0.%s", build)
app.Flags = []cli.Flag{
cli.StringSliceFlag{
Name: "repositories",
Usage: "List of repositories to trigger",
EnvVar: "PLUGIN_REPOSITORIES",
},
cli.StringFlag{
Name: "server",
Usage: "Trigger a drone build on a custom server",
EnvVar: "DOWNSTREAM_SERVER,PLUGIN_SERVER",
},
cli.StringFlag{
Name: "token",
Usage: "Drone API token from your user settings",
EnvVar: "DOWNSTREAM_TOKEN,PLUGIN_TOKEN",
},
cli.BoolFlag{
Name: "fork",
Usage: "Trigger a new build for a repository",
EnvVar: "PLUGIN_FORK",
},
cli.StringFlag{
Name: "env-file",
Usage: "source env file",
},
}
if err := app.Run(os.Args); err != nil {
logrus.Fatal(err)
}
}
func run(c *cli.Context) error {
if c.String("env-file") != "" {
_ = godotenv.Load(c.String("env-file"))
}
plugin := Plugin{
Repos: c.StringSlice("repositories"),
Server: c.String("server"),
Token: c.String("token"),
Fork: c.Bool("fork"),
}
return plugin.Exec()
}
// Plugin defines the Downstream plugin parameters.
type Plugin struct {
Repos []string
Server string
Token string
Fork bool
}
// Exec runs the plugin
func (p *Plugin) Exec() error {
if len(p.Token) == 0 {
return fmt.Errorf("Error: you must provide your Drone access token.")
}
if len(p.Server) == 0 {
return fmt.Errorf("Error: you must provide your Drone server.")
}
client := drone.NewClientToken(p.Server, p.Token)
...
One thing I really noted while doing this was that mixing a lot of lines
beginning with plugins. and flag. in sequence looks rather noisy. I renamed
the import to plug in this example. plugins should either be renamed to a 4
letter word or mirror all of the flag.**Var() functions… I chose the package
name plugins because of the current drone plugins convention of having a struct
and instance named plugin.
import (
"flag"
"fmt"
"strings"
"github.com/drone/drone-go/drone"
plug "github.com/thomasf/drone-plugins-go/plugins"
)
var build = "0" // build number set at compile-time
func main() {
var p Plugin
flag.BoolVar(&p.Fork, "fork", false, "Trigger a new build for a repository")
flag.Var(&p.Repos, "repositories", "List of repositories to trigger")
flag.StringVar(&p.Server, "server", "", "Trigger a drone build on a custom server")
plug.FlagEnvNames(&p.Server, "DOWNSTREAM_SERVER,PLUGIN_SERVER", "") // emtpy string means default PLUGIN_...
flag.StringVar(&p.Token, "token", "", "Drone API token from your user settings")
plug.FlagEnvNames(&p.Token, "DOWNSTREAM_TOKEN", "")
plug.EnvFile() // just handles it, no need for a ref?
plug.Parse() // the Exec(Execer) function below should probably take care of the "parse" step
plug.Exec(&p)
}
// Plugin defines the Downstream plugin parameters.
type Plugin struct {
Repos plug.StringSliceFlag
Server string
Token string
Fork bool
}
// Exec runs the plugin
func (p *Plugin) Exec() error {
if len(p.Token) == 0 {
plug.FatalFlagf(&p.Token, "you must provide your Drone access token.")
}
if len(p.Server) == 0 {
plug.FatalFlagf(&p.Server, "you must provide your Drone server.")
}
client := drone.NewClientToken(p.Server, p.Token)
...
Latest prototype:
API Usage:
package main
import (
"context"
"flag"
"github.com/thomasf/drone-plugins-go/plug"
)
var build = "0" // build number set at compile-time
// Plugin defines the Downstream plugin parameters.
type Plugin struct {
Repos []string
Server string
Token string
Fork bool
}
func main() {
var p Plugin
flag.BoolVar(&p.Fork, "fork", false, "Trigger a new build for a repository")
plug.StringSliceVar(&p.Repos, "repositories", "List of repositories to trigger")
flag.StringVar(&p.Server, "server", "", "Trigger a drone build on a custom server")
plug.Env(&p.Server, "DOWNSTREAM_SERVER", "") // emtpy string means default PLUGIN_...
flag.StringVar(&p.Token, "token", "", "Drone API token from your user settings")
plug.Env(&p.Token, "DOWNSTREAM_TOKEN", "")
plug.EnvFiles()
plug.Run(p.Exec)
}
// Exec runs the plugin
func (p *Plugin) Exec(ctx context.Context) error {
isValid := true
if len(p.Token) == 0 {
plug.Usageln(&p.Token, "you must provide your Drone access token.")
plug.Debugln("not valid")
isValid = false
}
if len(p.Server) == 0 {
plug.Usageln(&p.Server, "you must provide your Drone server.")
plug.Debugln("not valid")
isValid = false
}
if !isValid {
return plug.ErrUsageError
}
//client := drone.NewClientToken(p.Server, p.Token)
// ...
plug.Println("success!")
return nil
}
Usage as a plugin:
$ PLUGIN_ENV_FILE=1.env,2.env go run main.go -server asd -token asd
success!
or with debug enabled (PLUGIN_PLUGIN_DEBUG=1):
$ PLUGIN_PLUGIN_DEBUG=1 PLUGIN_ENV_FILE=1.env,2.env go run main.go -server asd -token asd
19:59:01 plug.go:28: drone plugins debug mode is active!
19:59:01 plug.go:30: flag 'env-file' using env vars: PLUGIN_ENV_FILE
19:59:01 plug.go:30: flag 'fork' using env vars: PLUGIN_FORK
19:59:01 plug.go:30: flag 'repositories' using env vars: PLUGIN_REPOSITORIES
19:59:01 plug.go:30: flag 'server' using env vars: DOWNSTREAM_SERVER, PLUGIN_SERVER
19:59:01 plug.go:30: flag 'token' using env vars: DOWNSTREAM_TOKEN, PLUGIN_TOKEN
19:59:01 plug.go:50: loading env files: 1.env, 2.env
19:59:01 plug.go:53: error loading env files: open 2.env: no such file or directory
19:59:01 plug.go:57: [envfile] setting PLUGIN_TOKEN=WOHOO
19:59:01 plug.go:88: flag 'env-file' set by env var 'PLUGIN_ENV_FILE': 1.env,2.env
19:59:01 plug.go:88: flag 'token' set by env var 'PLUGIN_TOKEN': asd
19:59:01 plug.go:93: ------ executing plugin func -----
19:59:01 main.go:54: success!
19:59:01 plug.go:95: ------ plugin func done -----
or a (very rudimentary) formatted usage print.
$ PLUGIN_ENV_FILE=1.env,2.env PLUGIN_FORK=cool go run main.go -server asd -token asd
plugin option 'fork' error: strconv.ParseBool: parsing "cool": invalid syntax
Usage: Trigger a new build for a repository
plugin usage:
source env file
names as plugin option: env_file
currently set by: env_file
current value 1.env,2.env
Trigger a new build for a repository
names as plugin option: fork
currently set by: fork
current value false
List of repositories to trigger
names as plugin option: repositories
Trigger a drone build on a custom server
names as plugin option: server
names as env var: downstream_server
Drone API token from your user settings
names as plugin option: token
names as env var: downstream_token
exit status 1
I’m wondering what a good usage print might look like.
A dicussion on how the usage text should be formatted is very much welcomed
From what I can tell the drone usage documentation doesn’t clearly explain the
difference between a plugin and a plain docker image in the pipeline. This is
maybe relevant for formatting the “names as plugin option” (PLUGIN_ var) and
"names as env var" (non prefixed env var) when printing the usage.
IIRC the distinction between those two kinds of environment variable names is a
common cause of confusion for new users when using secrets in particular and
environment variables in general so getting this right could potentially be
very helpful.
Latest error draft
$ PLUGIN_SERVER=myservername DRONE=true go run main.go
plugin usage:
option name : env_file
usage : source env file
option name : fork
usage : Trigger a new build for a repository
current value : false
option name : repositories
usage : List of repositories to trigger
option name : server
envvar name : downstream_server
usage : Trigger a drone build on a custom server
set by : server
current value : myservername
option name : token
envvar name : downstream_token
usage : Drone API token from your user settings
**USAGE ERROR** : you must provide your Drone access token.
:
The output above is automatically generated from the code below. Flags which
has plug.Usage(… strings attached are printed last to allow users to spot
errors easily. DRONE_ prefixed env vars are hidden from the usage print flag
parse errors occur.
package main
import (
"context"
"flag"
"github.com/drone-plug/drone-plugins-go/plug"
)
var build = "0" // build number set at compile-time
// Plugin defines the Downstream plugin parameters.
type Plugin struct {
Repos []string
Server string
Token string
Fork bool
Build plug.Build
}
func main() {
var p Plugin
flag.BoolVar(&p.Fork, "fork", false, "Trigger a new build for a repository")
plug.StringSliceVar(&p.Repos, "repositories", "List of repositories to trigger")
flag.StringVar(&p.Server, "server", "", "Trigger a drone build on a custom server")
plug.Env(&p.Server, "downstream_server", "") // emtpy string means default PLUGIN_...
flag.StringVar(&p.Token, "token", "", "Drone API token from your user settings")
plug.Env(&p.Token, "downstream_token", "")
plug.BuildVar(&p.Build)
plug.EnvFiles()
plug.Run(p.Exec)
}
// Exec runs the plugin
func (p *Plugin) Exec(ctx context.Context) error {
isValid := true
if len(p.Token) == 0 {
plug.Usageln(&p.Token, "you must provide your Drone access token.")
plug.Debugln("not valid")
isValid = false
}
if len(p.Server) == 0 {
plug.Usageln(&p.Server, "you must provide your Drone server.")
plug.Debugln("not valid")
isValid = false
}
if !isValid {
return plug.ErrUsageError
}
//client := drone.NewClientToken(p.Server, p.Token)
// ...
plug.Println("success!")
return nil
}
With plugin debug enabled:
$ PLUGIN_PLUGIN_DEBUG=1 PLUGIN_PACKAGE=123 PLUGIN_CONF_FILES=asd,123 go run main.go
00:09:42 plug.go:30: drone plugins debug mode is active!
00:09:42 plug.go:32: flag 'build.created' using env vars: DRONE_BUILD_CREATED
00:09:42 plug.go:32: flag 'build.event' using env vars: DRONE_BUILD_EVENT
00:09:42 plug.go:32: flag 'build.finished' using env vars: DRONE_BUILD_FINISHED
00:09:42 plug.go:32: flag 'build.link' using env vars: DRONE_BUILD_LINK
00:09:42 plug.go:32: flag 'build.number' using env vars: DRONE_BUILD_NUMBER
00:09:42 plug.go:32: flag 'build.started' using env vars: DRONE_BUILD_STARTED
00:09:42 plug.go:32: flag 'build.status' using env vars: DRONE_BUILD_STATUS
00:09:42 plug.go:32: flag 'deploy.to' using env vars: DRONE_DEPLOY_TO
00:09:42 plug.go:32: flag 'env-file' using env vars: PLUGIN_ENV_FILE
00:09:42 plug.go:32: flag 'fork' using env vars: PLUGIN_FORK
00:09:42 plug.go:32: flag 'repositories' using env vars: PLUGIN_REPOSITORIES
00:09:42 plug.go:32: flag 'server' using env vars: DOWNSTREAM_SERVER, PLUGIN_SERVER
00:09:42 plug.go:32: flag 'token' using env vars: DOWNSTREAM_TOKEN, PLUGIN_TOKEN
00:09:42 plug.go:105: ------ executing plugin func -----
00:09:42 usage.go:70: adding usage error for 'token': you must provide your Drone access token.
00:09:42 main.go:38: plugin option 'token' error: you must provide your Drone access token.
00:09:42 main.go:39: not valid
00:09:42 usage.go:70: adding usage error for 'server': you must provide your Drone server.
00:09:42 main.go:43: plugin option 'server' error: you must provide your Drone server.
00:09:42 main.go:44: not valid
00:09:42 plug.go:107: ------ plugin func done -----
00:09:42 plug.go:110: usage err
Usage of /tmp/go-build489541432/command-line-arguments/_obj/exe/main:
-build.created int
build created unix timestamp (DRONE_BUILD_CREATED) (default -1)
-build.event string
build event (push, pull_request, tag) (DRONE_BUILD_EVENT)
-build.finished int
build finished unix timestamp (DRONE_BUILD_FINISHED) (default -1)
-build.link string
build result link (DRONE_BUILD_LINK)
-build.number int
build number (DRONE_BUILD_NUMBER) (default -1)
-build.started int
build started unix timestamp (DRONE_BUILD_STARTED) (default -1)
-build.status string
build status (success, failure) (DRONE_BUILD_STATUS)
-deploy.to string
build deployment target (DRONE_DEPLOY_TO)
-env-file value
source env file
-fork
Trigger a new build for a repository
-repositories value
List of repositories to trigger
-server string
Trigger a drone build on a custom server
-token string
Drone API token from your user settings
00:09:42 usage.go:203: plugin usage:
00:09:42 usage.go:206:
option name : env_file
usage : source env file
option name : fork
usage : Trigger a new build for a repository
current value : false
option name : repositories
usage : List of repositories to trigger
option name : server
envvar name : downstream_server
usage : Trigger a drone build on a custom server
**USAGE ERROR** : you must provide your Drone server. you must provide your Drone
: server.
option name : token
envvar name : downstream_token
usage : Drone API token from your user settings
**USAGE ERROR** : you must provide your Drone access token. you must provide your Drone access
: token.
Since there it became obvious that this project isn’t desired as an official
drone project I’ve expanded the scope a bit to add more features.
Testabilty has been my main goal for the latest changes, there is now an
interface which defines SetFlags(fs *plug.FlagSet) and Exec(ctx context.Context, log *plug.Logger) error. It resembles parts of the API defined by https://github.com/google/subcommands
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/drone-plug/drone-plugins-go/plug"
)
// Plugin .
type Plugin struct {
Name string
AccessKey string
Flowers []string // []string flag type
MoreFlowers []string
Dogs map[string]string // Drone specific map[string]string
Cats map[string]string // for the JSON encoded input.
Commit plug.Commit // This plugin need all of drones Commit metadata
BuildNumber int64 // and the build number
Event string // + event.
}
// NewPlugin returns a Plugin with default values set
func NewPlugin() *Plugin {
return &Plugin{
// Default values for string slice types.
MoreFlowers: []string{"default", "value"},
// Default values for string map types.
Cats: map[string]string{"default": "value"},
// Default values for string map types.
Dogs: map[string]string{"default": "value"},
}
}
// SetFlags implments plug.Runner
func (p *Plugin) SetFlags(fs *plug.FlagSet) {
// Use the regular flag package.
fs.StringVar(&p.Name, "name", "default value", "your name")
// PLUGIN_ACCESS_KEY env var is automatic.
fs.StringVar(&p.AccessKey, "access_key", "", "")
// Alternative names can be specified. emtpy string is converted to the auto generated name.
fs.Env(&p.AccessKey, "", "aws_access_key")
fs.StringSliceVar(&p.Flowers, "flowers", "list of flowers")
fs.StringSliceVar(&p.MoreFlowers, "more.flowers", "")
fs.StringMapVar(&p.Cats, "cats", "")
fs.StringMapVar(&p.Dogs, "dogs", "")
// Register all fields for Repo, Build or Commit
fs.CommitVar(&p.Commit)
// or single properties...
fs.BuildEventVar(&p.Event)
fs.BuildNumberVar(&p.BuildNumber)
}
// Exec implments plug.Runner
func (p *Plugin) Exec(ctx context.Context, log *plug.Logger) error {
if p.Name == "My name" {
log.Usagef(&p.Name, "that is not your name")
// prints: invalid value for 'name': that is not your name
// return plug.ErrUsageError
}
log.Debugln("this is only printed when the plugin is run in debug mode")
log.Println("normal plugin output")
after, _ := json.MarshalIndent(&p, "", " ")
fmt.Println("build number:", p.BuildNumber)
fmt.Println("commit message:", p.Commit.Message)
fmt.Println("after:", string(after))
return nil
}
func main() {
p := NewPlugin()
// set some values in the env for testing the flags API below.
for k, v := range map[string]string{
"PLUGIN_NAME": "My name",
"PLUGIN_MORE_FLOWERS": "one,two,three",
"AWS_ACCESS_KEY": "such access key",
"PLUGIN_DOGS": `{"key":"value"}`,
"DRONE_COMMIT_MESSAGE": "Added text",
"DRONE_BUILD_NUMBER": "12",
} {
_ = os.Setenv(k, v)
}
plug.Run(p)
}
The plugtest package makes it easy to run a plugin multiple times testing for different inputs:
package main
import (
"testing"
"github.com/drone-plug/drone-plugins-go/plug/plugtest"
)
func TestExecFail(t *testing.T) {
p := &Plugin{}
pt := plugtest.New(t, p)
pt.AssertFail()
}
func TestExecOK(t *testing.T) {
p := &Plugin{}
pt := plugtest.New(t, p)
pt.SetPluginVars(map[string]string{
"token": "asdf",
})
pt.SetVars(map[string]string{
"downstream_server": "asdf",
})
pt.AssertSuccess()
pt.AssertOutput(`success!
`)
}
These changes have removed a lot of package level functions so the godoc is also much easier to navigate now: https://godoc.org/github.com/drone-plug/drone-plugins-go/plug