Local Development Guide¶

So, you want to see some progress, preferably on Slack, and not just in the forms of unit testing? At this point, fear is actually a reasonable response. With this guide, you can be talking to your locally-hosted Slack bot in no time!

Warning: This only works smoothly with a Unix machine (macOS or Linux variants). Windows users may be in for more pain than expected.

0. Developer Installation¶

We use pipenv for dependency management.

git clone https://github.com/ubclaunchpad/rocket2.git
cd rocket2/
pip install pipenv
pipenv install --dev

pipenv will manage a virtualenv, so interacting with the program or using the development tools has to be done through pipenv, like so:

pipenv run pycodestyle .

This can get inconvenient, so you can instead create a shell that runs in the managed environment like so:

pipenv shell

and then commands like pycodestyle and pytest can be run like normal.

Additionally, we use Github Actions as a CI system. To run the same checks locally, we provide scripts/build_check.sh; this can be run with:

./scripts/build_check.sh

The above tests would be run with the assumption that other applications, such as a local instance of DynamoDB, is also running. To run tests that explicitly do not involve the running of any database, run pytest with the following arguments:

pytest -m "not db"

You can also install it as a pre-commit hook for git:

cd scripts/
make install

Note that testing alongside a real Slack workspace, DynamoDB, and so on requires quite a bit more setup. For a full guide to developer installation, see our local development guide.

Running DynamoDB Locally¶

Some tests assume the existence of a local DynamoDB database. These are primarily for automated testing, like on Github Actions CI, but if you would like to run them yourself or are developing new tests, you can run as follows:

wget https://s3-us-west-2.amazonaws.com/dynamodb-local/dynamodb_local_latest.tar.gz
mkdir DynamoDB
tar -xvf dynamodb_local_latest.tar.gz --directory DynamoDB

# Configure AWS
scripts/setup_localaws.sh

# Run DynamoDB through Java
cd DynamoDB/
java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb
# Open a new terminal to continue interacting

For a more sandboxed approach, you can use Docker and docker-compose to spin up a local DynamoDB instance:

docker-compose -f sandbox.yml up

You can then point a Rocket instance at this DynamoDB database by setting AWS_LOCAL=True.

1: Set up a HTTPS Tunnel¶

Slack requires that all webhooks are passed through HTTPS. This is rather inconvenient if you just want to test while running on your local computer. There are several ways to get around this.

Ngrok¶

Ngrok is a forwarding service that hosts a public HTTPS URL that passes to your local computer. Sign up for ngrok and download it here.

After installing, run ngrok http 5000 to create an ngrok URL that will be passed to your local port 5000. As long as you run Rocket on port 5000 (see below), you can then access it through the HTTPS URL that ngrok gives you. Note that it is very important to use the HTTPS URL, not the HTTP URL.

Localtunnel¶

Note: Localtunnel has reliability issues, which include crashing Rocket 2 after issuing a single command sometimes (for no discernable reason). Tread with caution.

An alternative to Ngrok is localtunnel, which works similarly to Ngrok but allows you to use the same domain every time. For example:

$ lt --port 5000 --subdomain my-amazing-rocket2
your url is: https://my-amazing-rocket2.loca.lt

2: Create a Slack Workspace¶

For testing, it’s useful to have your own Slack workspace set up. If you do not already have one, go here to create one, and follow the steps to set it up.

3: Create a Slack App¶

Follow the link here to create a new Slack app - you can name it whatever you like - and install it to the appropriate workspace.

3.1: Add a Bot Token¶

In “OAuth and Permissions”, select the Bot Token Scopes described in the Slack configuration docs.

3.2: Install Slack App¶

In “Install your app to your workspace,” click the button to install to your workspace. This will take you to a permissions page for the workspace - make sure this is for the correct workspace, and allow the app to connect.

Once this is done, you will be provided with an API token.

3.3: Determine Credentials¶

Make note of the app’s signing secret, found in Settings -> Basic Information -> App Credentials, and the bot user OAuth access token, found in Features -> OAuth & Permissions -> Tokens for Your Workspace. These will be needed for the configuration step later.

4: Gain Access to AWS¶

Rocket makes use of AWS DynamoDB as its database. There is also an optional logging component that leverages AWS CloudWatch.

Using Real AWS¶

If you do not already have access to DynamoDB and CloudWatch, you can use it as part of the free tier of AWS. Create an AWS account for yourself, then go to the IAM service and create a new user. The user name doesn’t particularly matter (though rocket2-dev-$NAME is recommended), but make sure you check “programmatic access.” In permissions, go to “Attach existing permissions directly” and add the following policies:

  • AmazonDynamoDBFullAccess

  • CloudWatchLogsFullAccess

  • CloudWatchPutMetricData

You will have to create CloudWatchPutMetricData. You can do this by going to IAM -> Policies -> Create Policy. Choose service CloudWatch. Choose action PutMetricData. Everything else should be set correctly - you can create the policy with that.

As you may have noticed, we not only want to use DynamoDB, but also CloudWatch. We send our logs to CloudWatch for easier storage and querying. We also use it to track metrics.

Finally, copy the provided access key ID and secret access key after creating the new user.

Using Local AWS¶

Alternatively, just set up DynamoDB locally (the Docker-based setup is probably the easiest) and set AWS_LOCAL=True.

CloudWatch integration is not currently supported in this manner.

5: Set up a GitHub App and organization¶

Create a Rocket 2 Github under an appropriate testing organization. Make sure to install the GitHub App to the organization in addition to registering it. All this can be done from a GitHub organization’s Settings > GitHub Apps page.

In the GitHub app’s settings, go to “Private keys” and click “Generate a new private key”. This will generate and allow you to download a new secret key for Rocket 2. Save this to the credentials/ directory as gh_signing_key.pem - it should already be in the PEM file format, bracketed by:

-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----

Authenticating Rocket 2 as a GitHub App and obtaining an access token for the GitHub API should be automated, once the signing key is available. Refer to the GitHub key configuration docs for the required permissions.

After doing this, remember to put your tunneled HTTPS URL with /webhook appended at the end into the “Webhook URL” box. Refer to the GitHub webhook configuration docs for the required subscriptions.

6: Set Up Configuration¶

Our repo already contains sample-env, the main environmental configuration file for the entire app, as well as the credentials/ directory, where you will put credential files like the GitHub app private key.

Please read the configuration docs for more details.

7: Build and Run Rocket 2¶

This section assumes you already have installed Docker. Assuming you are in the directory containing the Dockerfile, all you need to do to build and run is the following two commands (run from the root of your project directory):

scripts/docker_build.sh
scripts/docker_run_local.sh --env-file .env

Optionally, for local DynamoDB:

scripts/docker_run_local.sh --env-file .env --network="host"

The option –env-file lets you pass in your configuration options.

For the curious, you can take a look at the contents of the referenced scripts above. Note that the options passed to -p in docker run tell Docker what port to run Rocket on. 0.0.0.0 is the IP address (in this case, localhost), the first 5000 is the port exposed inside the container, and the second 5000 is the port exposed outside the container. The port exposed outside the container can be changed (for instance, if port 5000 is already in use in your local development environment), but in that case ensure that your tunnel is running on the same port.

6.1: [Optional] Running without Docker¶

We highly recommend building and running on Docker, but building every time you make a tiny change can be inconvenient. If you would like to run without building a new Docker image every time, you can do so with pipenv run launch. This is in fact the same command Docker runs, but if you run outside Docker, you may run into errors due to unexpected changes in your local development environment.

7: Configure Slack App Features¶

In addition to a bot user, there are a couple other features that need to be enabled in the Slack app once the local instance of Rocket is running.

7.1: Add Event Subscriptions¶

In “Add features and functionality”, add event subscriptions. In particular, under Request URL, submit the ngrok HTTPS URL with /slack/events appended to the end. Note that ngrok will generate a new HTTPS URL every time it runs, so you will have to repeat this step every time you launch ngrok. You will then have to enable workspace and/or bot events that we want Rocket to listen for, like the team_join workspace event - ask the team for the most up-to-date list of these.

7.2: Add Slash Command¶

In “Add features and functionality”, add a slash command. In particular, under Request URL, submit the ngrok HTTPS URL with /slack/commands appended to the end. For the actual command, anything will work, though the final app will use /rocket. Make sure you tick the box marked “Escape channels, users, and links sent to your app”, or else none of the @ signs will work properly!

8: Testing¶

This is the final and most important part: testing if it actually works or not. Go to your Slack workspace and add Rocket (or whatever you named your Slack bot) to the channel if you have yet to do so (just type @<bot name> and Slack will ask if you want to invite the bot into the channel).

To test if Rocket is running, type the command:

/rocket user help

If you see a list of options, Rocket is working!

8.1: Setting Up Admin Permissions¶

We currently haven’t finished the command to add a user to the database or make them an admin, so we have to do it manually.

First, determine your Slack ID by reading the logs. The logs are formatted like so:

{slackid_making_the_command}:{command_itself}

The Slack IDs of other users will appear when you type @ followed by whatever the user’s handle is. Slack automatically converts that handle into an ID.

Then, you have an option of either using the AWS command-line interface or using the AWS web interface.

You should already have the command line interface installed via pipenv. If not, run the command pipenv install --dev. Note that to run commands, you will either have to go into the pipenv environment (with pipenv shell) or prefix every command with pipenv run. Here is the command to create a user with a:

# The following command is split into multiple lines because it is long. Make
# sure that the actal command isn't split into multiple lines because it may
# complicate things.
aws dynamodb put-item --table-name USERS_TABLE\
                      --item '{"slack_id":{"S": "UE7PAG75L"},
                               "permission_level":{"S": "admin"}}'\
                      --endpoint-url http://localhost:8000

Replace USERS_TABLE with whatever name you set in config.toml.

Alternatively, you can directly edit the DynamoDB table via the AWS web interface. Go to the DynamoDB service in the AWS web interface and open the appropriate table. Click on the Items tab and then on “Create item”. Make sure there’s a column for slack_id and permission_level, where slack_id is a String with the appropriate value and permission_level is a String with the value admin.

8.2: Viewing a User¶

/rocket user view

The output of this command should be a stylish table displaying your Slack id and permissions level.

Now, you can continue with whatever testing you originally wanted to do. Remember to rebulid your Docker image every time you make a change!