Josh Lospinoso home posts about
Twitter Subliminal 0.2.0 September 5, 2016
Twitter Subliminal Now Dockerized

Getting twitter-subliminal setup can be a pain, especially in non-Linux environments. Docker is a great solution for making software portable, and in version 0.2, I’ve Dockerized twitter-subliminal. You still need to signup for Twitter API credentials, but these are now injectable as command line parameters:

docker run -it --rm \
     -e consumer.key="ABCD1234" \
     -e secret.key="ABCD1234" \
     -e access.token="ABCD1234-ABCD1234" \
     -e access.token.secret="ABCD1234" \
     jlospinoso/twitter-subliminal

By using the -it option, you’ll get an interactive session with the Docker container. The working directory will contain all of the usual binaries available from v0.1.0:

  • tse: message encryption
  • tsd: message decryption
  • tsr: reset all retweets
  • tsp: performance testing for selecting block size
  • tst: unit testing
  • tsl: check rate limit status with Twitter API

You can also run these binaries directly. To send, e.g.:

docker run \
     -e consumer.key="ABCD1234" \
     -e secret.key="ABCD1234" \
     -e access.token="ABCD1234-ABCD1234" \
     -e access.token.secret="ABCD1234" \
     jlospinoso/twitter-subliminal ./tse "Hello, world!"

There are two locations where the twitter-subliminal containers are published:

  • https://hub.docker.com/r/jlospinoso/twitter-subliminal/
  • https://quay.io/repository/jlospinoso/twitter-subliminal

Feedback

Please post any issues or bugs you find!

Matterbot June 14, 2016
A C++ Framework for Creating Simple Mattermost/Slack Bots

matterbot is a framework for making Mattermost/Slack bots. It uses the Webhooks APIs exposed by both Mattermost and Slack to send and receive messages. In the twitter-subliminal project, we tried out Poco Libraries to handle our web communications. matterbot tries out something different: Microsoft’s C++ REST SDK a.k.a. Casablanca.

Getting started

First, you’ll need a running Mattermost/Slack service. If you already have one, skip ahead to the next section. For the remainder of the post, we use Mattermost, but the webhook APIs are compatible.

I highly recommend using docker while you are developing your bot. You can set up the service with this one-liner (Note: You should NOT use this command to set up a production instance!):

$ docker run --name mattermost-preview -d --publish 8065:8065 mattermost/mattermost-preview
f25bbb6897ca0a1e9a0313cef5c31a6864647ed18e593a6736925af83f8b2523

You can see your docker container’s status with docker ps:

$ docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED
f25bbb6897ca        mattermost/mattermost-preview   "/bin/sh -c ./docker-"   3 minutes ago
     STATUS              PORTS                              NAMES
     Up 3 minutes        3306/tcp, 0.0.0.0:8065->8065/tcp   mattermost-preview

You’ll note that the container has mapped the docker host’s port 8065. If you point a web browser at http://[my-docker-ip]:8065, you should be greeted with a friendly user signup screen. Follow the on-screen instructions or see the docs to set up a username and a team.

Setting up Webhooks

Once you’ve set up a team, you’ll need to enable webhooks. Full documentation is here, but the gist is that you’ll need to go to the System Console and click “Enable Incoming Webhooks” and “Enable Outgoing Webhooks”. The System Console can be accessed by clicking on the three vertical dots in the upper-left of the team site. It should be the seventh entry from the top in the drop-down menu that appears.

Go back to your team site. Webhooks are created at the team-level. Click again on the three vertical dots, and this time an “Integrations” option should appear. Click it.

You’ll need to create the webhooks individually. First, setup the Incoming Webhook. Click on the Incoming Webhook icon, then click “Add Incoming Webhook”. The display name and description do not correspond to what will actually appear in the chat window–they are just for administrative accounting purposes. Select the channel that you want the bot to post messages into, then click save. Take note of the resulting URL, e.g.:

http://192.168.1.2:8065/hooks/ktjckuh9ptrnmgoiunadsitgmc

Next, setup an Outgoing Webhook by clicking on the “Outgoing Webhooks” option under the “Integrations” header at the top of the screen. Click on “Add Outgoing Webhook”. The display name and description are again just for administrative accounting purposes. The “Channel” is actually optional–if you want the bot to listen to all channels, don’t select anything here. The trigger words are important–Mattermost won’t send your bot a message unless it begins with one of the trigger words listed here.

Finally, you’ll need to specify your callback URL. Locate the IP of the machine you’ll be running your matterbot from, and enter it in the “Callback URLs” box, e.g.:

http://192.168.1.3

Also take note of the token that gets created for your outgoing webhook, e.g.:

Token: omy7rqidk3dqdqky39yssm4bao

Configure your firewall!

Your bot is going to need to listen to port 80. Configure your firewall to allow this. If you are just running Mattermost in a docker container, you may be able to get away with default firewall rules.

Building an example bot

Pull down matterbot from github:

git clone git@github.com:JLospinoso/matterbot.git

Open up Visual Studio as an Administrator (so that you can bind to a local port when debugging). There are two projects in the solution:

  • Matterbot is the project containing the matterbot (static) library.
  • MatterbotSample is the project containing a sample bot

Both libraries require that NuGet has successfully installed the C++ REST SDK. Right click on “References” > “Manage NuGet Packages” > “Installed” and make sure that version 2.8.0 is correctly installed.

MatterbotSample contains one file, main.cpp, but it illustrates the main features of the library. In main(), we create a bot by injecting four parameters:

wstring mattermost_url = L"http://192.168.4.177:8065/",
	incoming_hook_token = L"ktjckuh9ptrnmgoiunadsitgmc",
	outgoing_hook_route = L"http://192.168.4.99/",
	outgoing_hook_token = L"omy7rqidk3dqdqky39yssm4bao";
//...
Matterbot bot(mattermost_url, incoming_hook_token, outgoing_hook_route, outgoing_hook_token);

These parameters are self explanatory–we set up all the webhooks in the previous section, and you are providing all of the route and token information to the framework here.

Once the bot has been initialized, you can post messages to the channel specified in the Incoming Webhooks by using the following:

bot.post_message(L"Bot is up.");

Easy peasy. But the interesting stuff is in implementing commands. Commands are routines that the bot will run when prompted. These routines give a response, which is posted into the same channel as post_message. Implementing commands is super simple. Just inherit ICommand. MatterbotSample gives an example echo command:

class EchoCommand : public ICommand {
public:
	wstring get_name() override {
		return L"echo";
	}

	wstring get_help() override {
		return L"`echo [MESSAGE]`\n===\n`echo` will respond with whatever
		message you give it.";
	}

	wstring handle_command(wstring team, wstring channel, wstring user,
		wstring command_text) override {
		return command_text;
	}
};

There are three functions in ICommand:

  • get_name is the command name that the bot will look for when it receives an outgoing webhook. So if we registered our bot to listen for #chatbot, then EchoCommand would get a callback when someone typed #chatbot echo Hello, world!.
  • get_help is an optionally-markdown flavored response that explains to the user how the command works. More on help in a moment.
  • handle_command is a callback whenever Mattermost/Slack alerts us via outgoing webhook that someone has triggered the bot. We get information like the team, channel, and user, as well as the full command text. The wstring result returned by handle_command is sent back by the bot.

You’ll register all of your commands with the bot by passing it a shared pointer:

bot.register_command(make_shared<EchoCommand>());

When a user prompts your bot for help, they will get a listing of all commands supported by the bot, e.g.

user > #chatbot help

bot > Supported commands
bot > echo [MESSAGE]
bot > echo will respond with whatever message you give it.
bot > checkbuild [build_id]
bot > checkbuild will retrieve the status of the build with build_id
bot > haiku
bot > bot will send you the haiku of the day
...

You can also ask for help about a specific command, e.g.

user > #chatbot help echo

bot > echo [MESSAGE]
bot > echo will respond with whatever message you give it.

The default logger will push messages from matterbot into wclog, but you can customize this behavior by implementing your own ILogger:

class CustomLogger : public ILogger {
	void info(const wstring &msg) override {
		wcout << "INFO: " << msg;
	}
	void warn(const wstring &msg) override {
		wcout << "WARN: " << msg;
	}
	void error(const wstring &msg) override {
		wcerr << "INFO: " << msg;
	}
};

Overwrite the default logger with set_logger:

bot.set_logger(make_unique<CustomLogger>());

One other feature of Matterbot is that it accepts GET requests at the same URL as the Outgoing Webhook URL (this comes basically for free since we need to) bind to the port anyway. The default response is a status webpage that gives basic statistics about the bot:

MattermostBot Status

Web requests served: 17
Messages posted: 165
Commands served: 2135
Supported commands:
*checkbuild
*echo
*haiku
...

Implementation details

That’s really all you need to get started building your own bot, but in case you would like to repurpose some (or all) of the matterbot source, here’s a brief overview of how the pieces fit together.

The Matterbot.h header is designed around the PIMPL idiom. The practical upshot of this design choice is that Matterbot.h is the only non-standard library header that gets imported:

#pragma once
#include <memory>
#include <string>

namespace lospi {
	class ILogger {
//...
	};

	class ICommand {
	public:
//...
	};

	class MatterbotImpl;
	class Matterbot {
	public:
//...
	private:
		std::shared_ptr<MatterbotImpl> impl;
	};
}

This is helpful because (a) changing implementation details does not necessarily require a recompiling of classes depending on Matterbot, and (b) compile times are generally faster due to less includes.

The low level details of translating HTTP semantics into C++ classes are all handled by the MattermostWebhooks class. If you wanted to write a much more involved bot (or bot framework), you could begin with this class to build on top:

class MattermostWebhooks
{
public:
	MattermostWebhooks(const std::wstring &mattermost_url,
		const std::wstring &incoming_hook_token,
		const std::wstring &outgoing_hook_route,
		const std::wstring &outgoing_hook_token);
	~MattermostWebhooks();
	void post_message(const std::wstring &message);
	void register_message_handler(const std::function<std::wstring(const Message&)>
		&message_handler);
	void register_web_handler(const std::function<WebResponse()> &web_handler);
	void listen();
	void die();
private:
//...
};

For outgoing traffic from Mattermost/Slack, you can register a std::function callback to handle Message objects:

class Message {
public:
//...
	bool token_is_valid() const;
	long get_timestamp() const;
	std::wstring get_channel() const;
	std::wstring get_team() const;
	std::wstring get_text() const;
	std::wstring get_user() const;
	std::wstring get_trigger_word() const;
private:
//...
};

For incoming web traffic (i.e. GET requests), you instead handle WebResponse object:

class WebResponse {
public:
//...
	std::wstring get_content_type() const;
	std::wstring get_content() const;
private:
//...
};

The content_types is the MIME Type of the content received by the request.

Feedback

Please post any issues or bugs you find!

Underhanded C Contest Submission (2015) February 28, 2016
Using a typo to dork a fissile material test

Here’s my submission for the 2015 Underhanded C Competition:

#include <stdio.h>
#include <math.h>
#include <float.h>
#define MATCH 1
#define NO_MATCH 0

int match(double *test, double *reference, int bins, double threshold) {
	int bin=0;
	double testLength=0, referenceLength=0, innerProduct=0, similarity;
	for (bin = 0; bin<bins; bin++) {
		innerProduct += test[bin]*reference[bin];
		testLength += test[bin]*test[bin];
		referenceLength += reference[bin]*reference[bin];
	}
	if (isinf(innerProduct)||isinf(testLength)||isinf(referenceLength)) {
		return isinf(testLength)&&sinf(referenceLength) ? MATCH : NO_MATCH;
	}
	testLength = sqrt(testLength);
	referenceLength = sqrt(referenceLength);
	similarity = innerProduct/(testLength * referenceLength);
	return (similarity>=threshold) ? MATCH : NO_MATCH;
}

The explanation is as follows:

Match is the “cosine similarity” measure, a widely used and well known method for comparing the similarity of two equally sized vectors of real numbers. The measure is always between [-1, 1]. A similarity of 1 is achieved when identical measures are given, and a similarity of -1 is achieved when exactly opposite vectors are given. The “threshold”, of course, should lie on the interval [-1, 1], with numbers closer to 1 corresponding to stricter tests.

This function is resilient to overflow. If there is an overflow, i.e. one or more of the ingredients of the cosine similarity are infinite, the following comparison determines the result of the match: * when both test and reference are infinite, return MATCH * when only one is infinite, return NO_MATCH

This function is vulnerable; it ostensibly performs a cosine similarity to determine how similar the reference and sample material are.

Can you find the bug?

The underhanded part is in the error checking: as a boundary condition, if the reference AND the sample produce an overflow (i.e. they have really, really big elements), the matching function produces a match. It’s the best guess we can make about whether the materials match or not.

Here’s a demonstration of the vulnerability:

#include <stdio.h>
#include <math.h>
#include <float.h>

#define PRINT_MATCH_RESULT ? printf("MATCH\n") : printf("NO_MATCH\n")
extern int match(double *test, double *reference, int bins, double threshold);

int main() {
	int bins = 4;
	// This is the reference measurement
	double reference[4] = { 5.00, 6.00, 3.00, 8.00 };
	// This is a test (that doesn't match well)
	double test1[4]		= { 1.00, 2.00, 3.00, 4.00 };
	// This is a test (that matches very well)
	double test2[4]		= { 5.01, 5.99, 3.02, 7.98 };
	// This is exploits a sinf-ul typo on line 32 :-)
	double dorked[4]	= { 1, 2, DBL_MAX, 4 };
	// This is a pretty high threshold for cosine similarit			
	double threshold	= 0.95;								

	printf("Test1 v Reference:     ");
	match(test1, reference, bins, threshold) PRINT_MATCH_RESULT;
	printf("Test2 v Reference:     ");
	match(test2, reference, bins, threshold) PRINT_MATCH_RESULT;
	printf("Dorked v Reference:    ");
	match(dorked, reference, bins, threshold) PRINT_MATCH_RESULT;
	return 0;
}

Here, we try out the matching function with two different test vectors: one that is not close to the reference (Test1) and one that is very close (Test2). The threshold of .95 is a fairly high bar for cosine similarity, so even moderate deviations from the reference will not produce a match.

On line 32, there is a simple typo:

	return isinf(testLength)&&sinf(referenceLength) ? MATCH : NO_MATCH;

should read

	return isinf(testLength)&&isinf(referenceLength) ? MATCH : NO_MATCH;

sinf()` calculates the sin of the referenceLength (which is exceedingly unlikely to evaluate to FALSE when cast too a boolean!). Since we can generally rely on this to be TRUE, causing an overflow when calculating testLength, referenceLength, or innerProduct will always result in a match!

Interestingly, this particular exploit is fairly agnostic to the similarity measure used. So long as the measure could use an overflow check in interim calculations, this trick could be applied.

This submission got a “runner-up” honorable mention. Unlike the winning entry, the data-driven vulnerability would be hard to produce in reality–how would we get a really, really large value into the *test vector?. There’s also some question about whether the error checking logic is realistic–would we really want to return a MATCH if both vectors overflowed? On the other hand, the vulnerability is really simple, and there’s definitely plausible deniability :-)

Follow the instructions on Github to pull the code down and try it yourself!

Fork me on GitHub