• … send data wireless
• … do not require additional services (e.g. an mqtt broker or similar)
• … are cheap
• … are small enough to be casually placed in the rooms they operate in

Bonus points for beeing able to run them on batteries.

Collecting temperature and humidity values

First of all I needed some way to measure temperature and get it somehow to my OpenHab instance. But with what hardware? A raspberry pi seems to be a bit overqualified and energy hungry for just stupidly measuring temperature and sending it somewere else.

I have little experience with microcontrollers but I like to browse through the hackaday.com for fun. In some projects people used ESP8266 boards for their projects. Those sounded very interesting, because they are small, cheap and can connect to wifi networks. That sounded pretty awesome so I wanted to try those.

I can’t remember where I heard about the BME280 sensor but those can measure temperature, humidity and pressure and work well with the ESP8266 boards. You can also find a lot of other blogs that use them and explain how they can be used. I don’t use the pressure part right now, but who knows …

ESP8266 / BME280

I ordered several a ESP8266 NodeMCU boards and the BME280 breakout boards. I used those (Amazon links):

• ESP8266
• BME280

I made the mistake when soldering together the first pair. And it’s really annoying to unsolder them again. Especially if you don’t realize that it is soldered wrong at first. So here is a tip if you are just getting started with microcontrollers and such: Use a breadboard first! It lets you try out the correct pins and connections before soldering. You can make sure you connected everything correctly before proceeding. It’s much faster than soldering and thus much better for trying things in different ways.

For programming the board I used the ArduinoIDE. I also used a lot of info from this blog post here on how to connect the BME280 to the ESP8266 board and how to install the necessary extensions in the ArduinoIDE.

What’s different on my build is, the ESP8266 does not host a web service over which the sensor values can be quieried. This would require the sensor to be always on. I didn’t want to do this because I wanted to save as much energy as possible to be able to run the sensors on batteries later.

Because I don’t have a 3D printer, I put the sensors in empty match boxes (big ones). There is also enough space for batteries in there.

Sending via WiFi

Because the sensors can not be queried they should send their collected data to the OpenHab service somehow. I tried to avoid an additional service such as an MQTT broker because it seemed a little oversized for my five sensors - at least for the moment.

OpenHab has a REST interface that let’s you manipulate the item states via HTTP requests. So this is a very easy way to update the soon to be created temperature and humidity items. No need for any other component. How a request has to look like can be seen in the OpenHab documentation. More on that later.

Did you read that blog post I mentioned above? Good! Then you see that connecting to a Wifi network is aktually pretty easy.

Energy saving

One of the goals was to place those sensors somewhere in the room where they don’t get in the way. This is easily accomplished when they are run on batteries. But for this we need to write the software in another way. As mentioned above, running a web server on the ESP8266 and querying it is possible, but costs a lot of energy. Most of it while nothing is measured and the values are not queried, but the webserver just waits for requests.

With my sensors I took another approach.

Deep sleep

When trying to save energy the deep sleep mode is often mentioned for the ESP8266. This mode disables most of the boards features except for the RTC (i guess that means Real Time Clock), which can in turn be used to wake up again from deep sleep. The energy consumption of the RTC is much lower than with the other features (e.g. Wifi, System clock, CPU don’t know what else). The difference in programming is, that you never enter the typical loop method in your program. The loop is done by going to sleep after the setup function is run. When waking up after the specified amount of time the setup function starts from the beginning.

void setup() {
    // This method is run at program start.
    // Program starts e.g. after you press the reset button
    // or the ESP8266 wakes up from deep sleep.

    // go to sleep for 30 seconds (or 30.000.000µs)
    ESP.deepSleep(30 * 1000000);

    // code after the deepSleep method call is never reached
}

void loop() {
    // code in the loop method will never execute
    // if the deepSleep method is called in the
    // setup method.
}

An important thing to note is, that the ESP8266 won’t wake up after the specified time if the RST pin is not connected to the D0 pin. So make sure you do that so the program works. I had an issue with uploading my sketch to the ESP when this connection was already made while uploading. So I soldered this to jumper pins so I can remove the connection while programming.

A lot more detailed information about deep sleep can be found in this blog post.

Wifi reconnect

Going to deep sleep and waking up again requires a reconnect to the Wifi in each iteration. This takes quite a while because a network scan is performed to find the appropriate channel on which the wifi is sending. Also getting an IP address via DHCP takes really long. The latter duration can be dramatically reduced by using a fixed IP address. But I don’t want to specify a fixed IP for all sensors in the firmware because this is so unflexible.

The problem with scanning for the channel can also be mitigated by remembering the channel when sucessfully connecting for the first time. This blog post is an excellent resource on how to do that.

DHCP cache

The problem with the DHCP still remains. So I extended the solution of the wifi channel and bssid saving and added the IP address, netmask and gateway after retrieving a DHCP lease for the first time. Note that this is not really a valid DHCP client implementation because it does not take the actual lease time into account. But it works fine and reduces the total time for one cycle of waking up, measuring, connecting to wifi and sending the data from about 16 seconds down to 4. That should save some energy.

struct {
  uint32_t crc32;
  uint8_t channel;
  uint8_t bssid[6];
  uint8_t padding; // everything to here is from the original blog post
  uint32_t ip;    // ip, netmask and gateway were added by me
  uint32_t netmask;
  uint32_t gateway;
} rtcData;

Savings

All those things are done to reduce the energy consumption. But I can’t really prove that it is really saving much. I don’t have the equipment nor the ability to measure how long my batteries will last - but a reduction from 16 to 4 seconds shouldn’t be too bad.

OpenHab items

Because I want to use the same firmware for all sensors without modifying it. I need some identifier for the sensor when it sends the data to OpenHab. For this identifier I chose the MAC address. The items for the sensors values in OpenHab are following the naming conventiones I adopted from the OpenHab documentation. So they can be easily identified throughout the system. What that means is, that they do not include a MAC address in their name. They are named something like GF_LivingRoom_Temperature.

So how can I update those items without adapting the firmware of each sensor?

Mapping MAC addresses to acutal items

The answer is, there must be some kind of mapping. I need to map the MAC address of the sensor to an actual item in OpenHab. The idea is to have an OpenHab item that corresponds to the MAC address of the sensor. This helper-item is then used to update the specific temperature or humidity item.

First thing I did was to create a group that contains all items that are only used to map values to another item. Why this is necessary you will see later. Then I created an item that has the MAC address in its name so it can be updated by the sensor over the OpenHab REST interface. In the items label I set the name of another OpenHab item - the one that should actually hold the value. So this establishes the mapping between MAC address and item that should be updated in OpneHab.

See the sample OpenHab items configuration file below.

// group that contains mapping items
gSensorMapping

// create item that maps the MAC address to the actual item.
// Note that this item has the gSensorMapping group assigned
Number  temperatureAABBCC112233	"GF_LivingRoom_Temperature"    (gSensorMapping)

// create the actual temperature item that should be used for everything within OpenHab
Number	GF_LivingRoom_Temperature	"Temperature [%.1f °C]"	<temperature>	(GF_LivingRoom)

After that, the mapping item (the first one) can be easily updated by the sensor using the REST interface and the sensors MAC address. This works without modifying the firmware for each sensor. The MAC address is just included in the REST url as the name of the item so the requested url looks something like this:

https://openhabserver.baseaddress.com/rest/items/temperature + MAC_ADDRESS

This updates the mapping item, but not the acutal temperature item we want to use. To achieve this I wrote a small OpenHab rule that uses the mapping information to transfer the sent values. Here the group that was created earlier is used.

The rule listens on updates of mapping items - the ones that are part of the gSensorMapping group - and forwards the value to the item that is referenced in the label of the mapping item.

rule "Forward updates of raw sensors to the appropriate high level sensor"
when
	Member of gSensorMapping received update
then
	logInfo("Map Sensor", "Mapping sensor '" + triggeringItem.name + "' to '" + trig
geringItem.label + "'")
	var targetItemName = triggeringItem.label
	postUpdate(targetItemName, triggeringItem.state.toString)
end

The good thing is, that the rule is only necessary once. More mappings can easily be defined by adding a mapping item and a regular item that is used in OpenHab as usual.

Check sensors sending data

I plan to run the sensors on battery, but they don’t report their battery charge condition (yet). To make sure they are running I let openhab check if the sensors send data in regular intervals. If they don’t, I send a warning notification to check on the affected sensor.

Similar to the sensor mapping, for this I created a new group that will be applied to all items that expect regular updates. Then a rule is created that checks if those items were updated in time. In my case I only check this during the day to avoid notifications in the middle of the night. Also my notifications are sent via telegram but other channels would be possible depending on your OpenHab configuration.

Group gWatchdogWarning24h

// Assign the above group to all items that should be checked for regular updates
// in my case e.g. the temperature and humidity items
Number	GF_LivingRoom_Temperature	"Temperature [%.1f °C]"	<temperature>	(gWatchdogWarning24h, persist_on_update, GF_LivingRoom)

The temperature item has the gWatchdogWarning24h assigned so it will be checked by the following rule. The persist_on_update group is used persist the items values in an influx database. It is necessary to persist the items values in some datastore, otherwise the check for the last update is not possible. Additional advantage is, that you can create nice graphs via grafana.

The rule that checks the updates:

rule "Send warning if item has not been updated in the last 24h"
when
	Time cron "0 0 8-20/5 1/1 * ? *"
then
	logInfo("watchdogRules", "Checking items that require status updates every 24h")
	var itemsThatNeedUpdates = ""

	gWatchdogWarning24h?.allMembers.forEach [ item |
		logInfo("watchdogRules", "Checking item: " + item.name)

		var rawLastUpdate = item.lastUpdate
		if (rawLastUpdate !== null)
		{
			var lastUpdate = rawLastUpdate.toInstant().toDateTime()
			if (lastUpdate.plusHours(24).isBefore(now))
			{
				itemsThatNeedUpdates += item.name + ", "
			}
		}
		else {
			logInfo("watchdogRules", "Does not seem to have received an update ever")
			itemsThatNeedUpdates += item.name + ", "
		}
	]

	if (itemsThatNeedUpdates == "")	{
		logInfo("watchdogRules", "All items seem to be up to date")
	}
	else {
		logInfo("watchdogRules", "The following items did not send an update in time: " + itemsThatNeedUpdates)
		sendTelegram("openhab_bot_name", "The following items did not send an update within 24 hours. Check batteries? " + itemsThatNeedUpdates)
	}
end

Conclusion

The self made sensors work quite well and the grafs look quite nicely. I don’t now how long the batteries will last but I did my best to reduce the energy consumption. This was my first step toward a smarter home.

The source code for the sensors can be found in my github repository in case you are interested.

In docker for windows you can easily set a proxy. The problem is, if you want to use a proxy with authentication you need to specify the proxy with your domain user and the password, like so:

http://USER:PASSWORD@our-corporate-proxy:8080

The user and password are set in clear text which I’m not so comfortable with. I didn’t investigate if it’s possible in windows, too, but when running docker on a linux machine, you can set the proxy via environment variables … in which I also don’t want to set my password in clear text.

CNTLM

I’m not aware of a way to get around the above mentioned problem without any additional tool. After a little online search I found the cntlm tool. In short it’s a proxy that handles the communication and authentication with an upstream proxy so your tools won’t have to.

To be able to authenticate to the upstream proxy you also need to specify the credentials. Other than in the proxy url you can at least specify an NTLM hash and not a clear text password.

Configuration

After installing cntlm (in my case via choco install cntlm) it can be configured with the cntlm.ini file in the installation directory. First we need to create the hash that should go to the configuration file. This can be done with the following command in a cmd or powershell.

cntlm.exe -H -u DOMAINUSER -d DOMAIN

This will prompt you for your password and will then print the NTLM hash you need for configuration.

PassLM      XXXXX
PassNT      XXXXX
PassNTLMv2  XXXXX

In my case I just needed the last one. So I copied that one and put it in the cntlm.ini configuration file at the appropriate location (where the samples are commented). Don’t forget to change the user and domain settings. Also make sure the Auth NTLMv2 is set. It will look something like this at the beginning of the file:

#
# Cntlm Authentication Proxy Configuration
#
# NOTE: all values are parsed literally, do NOT escape spaces,
# do not quote. Use 0600 perms if you use plaintext password.
#

Username	YOUR_USERNAME
Domain		YOUR_DOMAIN.com

# NOTE: Use plaintext password only at your own risk
# Use hashes instead. You can use a "cntlm -M" and "cntlm -H"
# command sequence to get the right config for your environment.
# See cntlm man page
# Example secure config shown below.
Auth            NTLMv2
PassNTLMv2      XXXXXXXXXX

Save the file and check if it works:

cntlm.exe -M http://google.com

If everything is ok it should print something like

Config profile  1/4... OK (HTTP code: 301)
----------------------------[ Profile  0 ]------
Auth            NTLMv2
PassNTLMv2      XXXXXXXXX
------------------------------------------------

Docker

If you start the cntlm windows service you can now use that proxy via localhost:3128. In docker for windows you can set it via the UI.

Unfortunately after setting this proxy and running e.g. docker search grafana I always received the following error.

Error response from daemon: Get https://index.docker.io/v1/search?q=grafana&n=25: proxyconnect tcp: dial tcp 10.0.75.1:3128: i/o timeout

Hm, strange. Why does docker try to use 10.0.75.1:3128 instead of the localhost that I actually specified in the settings? Well I think this is because when using docker for windows, the docker host is actually hosted in a virtual machine - which is running on localhost, but it’s not the same as localhost. It’s in its own network and has its own IP address. Which can also be seen when doing an ipconfig -all.

Troubleshooting

So it seems, that cntlm does not accept connections from other machines, which is a good thing from the security perspective. So the internet connection that uses your credentials can at least only be used from your own machine. But unfortunately docker can’t use it. The solution for this is to set the Gateway configuration in the cntlm.ini file to yes. This allows other machines to use the proxy.

# Enable to allow access from other computers
#
Gateway	yes

To improve security I used the windows firewall and restricted inbound connections to CNTLM (Port 3128) to specifically allow the IPs of the docker NAT (in my case 10.0.75.1-10.0.75.15). So that are the only IPs that can use the proxy. You can also specify allowed IPs in the cntlm.ini file but that didn’t have any effect when I tried that. I probably did something wrong there. But it works fine with the firewall rules. Additionally I stop cntlm when I’m not actively using it to further reduce the possibility of misuse of this proxy.

Finally working

After all those changes (and restarting the cntlm service) I can finally search and pull docker images from the docker registry. What a fight …

I like to use the command line a lot, as it gives you a lot of flexibility and a better understanding in what’s happening compared to some GUIs. So the main work with git is done in the console. There are a few exeptions. I am using a GUI merge tool for example. But most of the time I’m working with git in the command line.

To make things easier I use quite a few aliases. Some of which are commonly known and used by fellow devs. But maybe here are some that you didn’t know, yet.

git checkout

The alias I use for the checkout command is co. It is very simple and commonly used by many people. The nice thing is, this one alias already gives a lot of benefits. Create it using:

git config --global alias.co checkout

Create a branch

Usually the first time I can use this new alias is to create a new branch before starting to code. For example to create a new feature branch. git co -b feature/some-feature Note the -b switch which allows you to switch to a branch that gets created in the process. Short for git branch feature/some-feature and git checkout feature/some-feature.

Undo changes

Next is to undo changes on tracked files

git co -- .

Switch branches

What I really like is this little gem:

git co -

The - switches back to the previous branch. It’s the Alt+Tab of branches.

Additionally I created the coma alias for directly switching to the master branch regardless of where I was before.

git config --global alias.coma 'checkout master'

git add

I like to check the changes I made before adding them to the staging area. When I did that by using a gui tool I was annoyed by opening the compare tool, reviewing the changes, closing the compare tool and then adding the file to the staging area. It’s just too many steps and usually I have to switch to the mouse.

As an alternative there is this fine command: git add . -p

The -p switch causes the add command to split up changes in the files into small hunks, that can be reviewed and accepted or rejected individually in the command line. It asks for each hunk what it should do with it and you can choose if you want it in your commit by simply using the y or n key. There are some more options, but those are the ones that are most relevant to me.

Especially for a lot of small changes in many files this is a lot faster than opening every file and adding it to the statging area. An additional advantage is of course that you can add only parts of a file to the next commit.

See git add documentation for more details.

git rebase

I like clean history, that is why I do a rebase on my feature branches before creating my pull requests. Often the branch I rebase against is the master branch. I created a rema alias, that switches to the master branch, pulls the latest commits, switches back to the previous branch and then executes a rebase onto the master branch.

git config --global alias.rema !'git checkout master && git pull && git co - && git rebase master'

The git aliases that start with a ! are treated like shell commands. The above command runs under my linux system. If you want to be able to use it with powershell, you might have to replace the && with a ;.

Sometimes it is necessary to resolve conflicts. For this I use the git mergetool command or the alias mt.

git config --global alias.mt mergetool

When a merge is successful the git rebase --continue comes to play. The rebase creates *.orig files per default. To remove them and continue the rebase I have an alias rc. I know that the creation of the orig files can be surpressed, but they are quite handy sometimes. So I still want them created but removed if I continue the rebase.

git config --global alias.rc !'git clean -f && git rebase --continue'

Again, use ; instead of && on windows.

git push

When the feature is finished or ready to be reviewed it must be published to the remote repository. The command

git push --set-upstream origin feature/branch-name

can be used for that, but is quite verbose. I don’t want to type that much. So I use an alias pu:

git config --global alias.pu !'git push --set-upstream origin $(git rev-parse --abbrev-ref HEAD)'

Open WebUI for creating pullrequests (Linux only)

The next step for me is usually to switch to the web view and create a pull request for my newly published branch. To speed up this process a little bit, I use an alias pr. It reads the URL from the origin remote and builds the URL to the Create pull request view of my provider. Because I don’t want to install additional tools I tried to solve this only with git aliases and configs. That is why this step may seem a little complicated. Additionally it only works with a linux bash because the aliases use e.g. sed.

The steps required are:

1. Get the remote URL
2. Try to resolve ssh URLs to https URLs
3. Append the path for the pull request creation view to this URL
4. Replace the $branch and $base placeholders in the URL templates with the actual branch names
5. Open the URL with the systems default browser

step 1 & 2

git config --global alias.getbaseurl !'git remote get-url origin | sed -r "s/git@(.+?):(.+?)\\.git/https\\/\\/\1\\/\\2/"'

step 3

To provide paths for different providers, in my case github and gitlab. I created config entries per provider and a default which can be set globally and overwritten per repository if needed.

The URLs may contain placeholders. Those will get replaced. $base will be replaced with master and $branch with the currently checked out branch name.

git config --global pullrequesturls.github '/compare/$base...$branch?expand=1'
git config --global pullrequesturls.gitlab '/merge_requests/new?merge_request[source_branch]=$branch&merge_request[target_branch]=$base'
git config --global pullrequesturls.default pullrequesturls.github

The default is the used in combination with the command that gets the base URL from step 1.

git config --global alias.getpullrequesturl !'echo "$(git getbaseurl)$(git config $(git config pullrequesturls.default))"'

step 4

Here it gets tricky. We will need to replace the $branch and $base placeholders. Calling sed with variables is doable, but I found myself in quote escaping hell while trying to create the alias command for it. So please just insert this line manually in your git config file and please don’t be mad at me.

[alias]
    replacebranches = !git getpullrequesturl | sed 's,$branch,'\"$(git rev-parse --abbrev-ref HEAD)\"'', | sed 's,$base,master,'

step 5

Put it all together and create a nice short pr alias. sensible-browser uses the systems default browser to open the URL.

git config --global alias.pr !'sensible-browser $(git replacebranches)'

dotfiles

There are a lot more aliases that I have configured and that I use from time to time. The above ones are especially noteworthy in my opinion. You can have a look at my dotfiles if you want to view the other aliases and configs that I have.

By the way if you are a linux user you should have a look at dotfiles in general because it is really awesome to have your config files in a git repository.

Cake (C# Make) is a cross platform build automation system with a C# DSL to do things like compiling code, copy files/folders, running unit tests, compress files and build NuGet packages.

Let me sketch out our situation a little bit and how we integrated Cake.

XMHell

Did I spell that correctly? My bad! But seriously, is there anyone who actually likes editing XML files by hand? I refuse to believe that. Our current build files are MsBuild XML files that define what’s part of the build process. Some of them more complex than others. Especially if you call some external tools that need escaping of characters that are used by XML then you end up with stuff like

<SomeElement attribute="&quot;&amp;canyoustillreadthis?&quot;">

Do I have to say more?

With Cake you can write nice and clean C# syntax, use methods and classes to structure your code and use the .NET Framework classes and helpers. On top of that Cake offers many extensions that help with managing tasks, arguments or tools. See this minimal example from the Cake homepage:

var target = Argument("target", "Default");

Task("Default")
  .Does(() =>
  {
    Information("Hello World!");
  });

RunTarget(target);

Isn’t that more convenient than XML? And if your build files get bigger, you can split them up in several files and include them with the #load file.cake directive.

Bob our special snowflake

We have a build server with the same name as the guy with the motto “Yes we can”. No, not Obama. It’s Bob, Bob the builder! It is a Windows machine that runs Jenkins and several slave processes. It’s responsible for building the releases of our internally used software suite and for triggering the deployments. It also performs continuous and daily builds and gives feedback to developers. I really like Jenkins and I think it does what it does pretty well. I was quite happy working with it - yes, also for .NET projects. The reasons for switching to TFS shall not be part of this post.

Because the build machine already has a few years on its back and has to build a lot of different projects the requirements to the installed software grew over time. Different frameworks, ui components, test tools, documentation generators, compiler versions code analysis tools are just a few among others. Each of which also have multiple versions available. This makes our “Bobby” a very special snowflake. And everyone that had to reproduce an error that just occurs on the build machine knows the pain something like this can cause.

Of course if we had treated the TFS environment the same way it would also be a mess, but we had the chance to create it from scratch. This means we could install some close to vanilla Windows machines with TFS build agents and the Visual Studio 2017 build tools. This is pretty much everything that is installed.

We still need a test runner, and all those other things that were previously installed on the build machine. So what are we going to do? Well, there is nothing that couldn’t be solved with cake. :) The built in support that Cake offers for handling tools and other dependencies is great for that. It uses NuGet packages so you can use every tool or library that is available via NuGet package within your build script. It also has very comfortable extension methods for the commonly used ones like NUnit, MsBuild and many more.

So what we do now instead of installing NUnit in several versions, we just specify the NUnit tool and its version in the projects build script like so and it is loaded and used by Cake.

#tool "nuget:?package=NUnit.ConsoleRunner"

NUnit3("./tests/**/*.Tests.dll", new NUnit3Settings {
  NoResults = true
  // other settings
});

The same goes for static code analysis, code coverage, running database scripts, powershell and so on. There is a whole lot built in functionality. And if it’s not, there are …

Addins

If the need for a tool occurs that does not have built in C# wrappers for easier handling, chances are that somebody else needed that too and wrote an addin for it. There are already a lot of them available as you can see on the homepage. If you can’t find it, there is always the chance to write your own addin. I wanted to run liquibase database migration scripts and created the Cake.Liquibase addin. It does not wrap all the functionality liquibase provides, but the part I needed, updating the database, is included. It allows defining the liquibase parameters by setting properties on a settings class and calling a method instead of having to deal with a batch file, the proper working directory, environment variables, escaping parmeters or quotes around paths. It’s just more comfortable to use the addin. And this is true for the most addins or tools.

Of course the addins and tools provided via NuGet are changed every now and then. To keep your build files stable you can use fixed versions of them by defining it in your build files.

#addin "nuget:?package=Cake.AddinName&version=1.0.0"

This avoids broken builds because of incompatible updates of an addin. The same can be done for the Cake version itself. But there you have to specify the version in the packages.config within the tools folder with the usual NuGet xml sytax.

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <packages>
      <package id="Cake" version="0.21.1" />
  </packages>
</packages>

Additionally, you can include other dependencies here. For example: I included the Liquibase.Cli package that contains the liquibase binaries here, so the Cake.Liquibase addin can use them and they do not need to be installed on the machine (also the addin supports installed versions, too). If you change the packages.config file keep in mind that you add it to your source control, but leave out the rest of the tools folder as those will get downloaded by the bootstrapper and do not need to be checked in. Also see the Pinning Cake version topic in the documentation.

Centralized build logic

In the past we had build files, too, but they didn’t contain the whole build process. They contained e.g. building the solution, but running tests or trying to run the database scripts were steps in the build server. This is working fine, but in my opinion has some disadvantages.

The first one is, if a step is only available on your build server and it fails, it is much harder to reproduce and fix it on a developer machine. If you keep all steps in the build file you can also execute them the same way they happen on the build machine. This can help a lot and speed up the time needed to fix build issues in many cases.

Another disadvantage is that you have to change the build on the server in sync with the version of your source and build file. Why is that? Example incoming!

You want to switch your test framework because you like the shiny new release of SuperTest so much better than the old UsualTest that you currently use. Well, you go ahead and change all tests to use SuperTest and remove all references to UsualTest. You install the SuperTest runner on your machine, give it a try and commit because everything worked like a charm. Great right? No! Now, the pull request validation build fails because it’s still using that old school UsualTest runner. Damn. Good thing it’s only the pull request validation build, so just change the runner and be done with it! A few minutes later the colleague from the next room storms in and gives you a karate chop in the neck because his pull request was rejected due to failing tests. Similar problems occur for the continuous-, daily-, release- and other builds but only after your change is merged to the different branches. You see where I’m going with this. You could create builds with different runners that build different branches depending on different build variables or something like that, but that makes the build environment very confusing. Also someone has to maintain those builds.

A better alternative, in my opinion, would be to keep this logic in the build file. So the build server just executes the script. The developer chooses the dependencies, tools and steps to be executed by editing the script and checking it in alongside with the code changes. All the build logic in one place and executable from the server or developer machine alike. This way the guy next door can still finish his pull request because his build file still uses the UsualTest runner. You can use the SuperTests and its runner in your file. And when your branches get merged, the old tests are updated with your changes and the new runner gets introduced to the build file and you and your colleague live happily ever after. And so do the other builds.

I also sneaked in another goodie in the above example, you can have your build logic under source control with the obvious advantages. Big plus here!

Bootstrap

To use all the above mentioned advantages the developers must be able to use these scripts on their machines as well. This is very simple with Cake. It provides a powershell or bash script that handles all the bootstrapping. All a developer must do is execute that script. No installation of Cake is required. The script downloads the latest or in the packages.config specified version of Cake, your addins and tools and calls Cake afterwards. It’s really easy to get started.

Debugging

As your build files get more complex you may need to find errors in it from time to time. Good news here: You can even debug your build scripts with Visual Studio. For this to work open the Cake script in Visual Studio. Pro-Tip: Install the Cake for Visual Studio extension first! Then start the Cake script with the --debug switch. Set a breakpoint and attach the debugger to the process Cake tells you.

This is really helpful and a fun thing to do! Try it!

More than build

Other than just handling builds we also use the script for other tasks. For example we regularly need to setup or recreate databases for our development environments. Instead of creating separate scripts and tools for that, we use Cake now. That’s why I created the Cake.Liquibase addin for. It’s just another target in the Cake script.

.\build.ps1 --Target UpdateDatabase

That’s all there is to be done for it. You could automate all kinds of tasks to setup your development environment for a specific project. If it’s more complex you can also use arguments to specify details.

.\build.ps1 --Target UpdateDatabase --databaseName=TestDatabase --otherParameters=y

The best thing is you can also document such tasks directly where they are used. Tasks in Cake can be documented with the Description() extension method and the --showdescription parameter displays all tasks with their descriptions.

Task("UpdateDatabase")
  .Description("Creates a new local development database ...")
  .Does(() => {
    // code here
  });

gives you

> .\build.ps1 --showdescription
Task                          Description
================================================================================
UpdateDatabase                Creates a new development database when necessary
                              and runs the database update scripts against it. 
                              This gives you a clean and up to date database 
                              to develop against.
Default                       

Unfortunately, there is not a built in way to document arguments the same way as tasks at the time of writing. But because it’s .NET you can just use the available tools like described here for that. That goes for any task you want to put in your script. Because it’s .NET the possibilities are nearly endless :P.

Conclusion

Cake keeps our build servers clean, the dependencies are managed via NuGet, we can write C# instead of XML, we can write extensions so it fits our needs, we have an all-round tool for managing our projects and it is documented properly within the script. Any questions?

But then sometimes all I need is a little push from somebody. In this case I watched the youtube video ‘Hack your career’ by Troy Hunt which was really fun to watch. If you haven’t seen it, you should go watch it. But for me, after watching it, there was this feeling again:

Considerations

Before actually starting, a few questsions wanted answers. What do I want to write about and who would the target audience be? Well, as I said, I don’t plan to become the world’s most famous blogger or anything. So as I see it the target audience consists of mainly two groups:

1. Me
2. People who googled something and coincidentally landed on this blog.

I guess the first group will be the bigger one and so the main target audience am I. Damn, nothing to brag about, right? Well it’s fine. I will benefit from the writing itself. Maybe I can re-read or look-up something I described here and benefit from it again. You might ask: “Why don’t you just write it in your diary then?” And you are right, I could. But who knows, maybe my writing skills will increase over time and the topics will be more interesting - then somebody else might want to read it, too. You are reading this after all. And if nobody else is interested, so be it.

Essentially I just repeated the advice from Troy and every other blogger that ever wanted to convince someone to start a blog, let’s move on.

So where do we stand now? Mindset? Check! Target audience? Check! Topic to write the first blog post about? Nope, no idea. But hey, I want to write a blog post, I don’t have a blog: behold the glorious first ever blog post topic: How I started my blog! Pretty slick, huh?

The biggest obstacle up to now was always to get a blog of some kind up and running. Because of that I have a few requirements on my blog environment:

1. It must be simple. I don’t want to waste my energy by investing billions (or more) of hours in learning how to use a blog. I want the first steps to be a quick success so I can motivate myself to keep updating and writing.

2. There should be little to no maintenance required. I’ve installed WordPress before on my webspace that comes with my domain. I didn’t get to the point to actually write something because I got so involved in what plugins and themes and other stuff I should use that the most important part got lost. Also I don’t want to care about a database, backups, updates, php versions and all the stuff that comes with dynamic websites and blog engines.

3. I’m from southern germany, upper swabia to be precise, and it’s a commonly used prejustice by my fellow countrymen that we are stingy. And as much as I disagree with that usually, in this case I have to admit that I do not want to spend money in addition to that what I’m paying for the my domain and webspace already. At least not as long as there are millions of thrilled groupie blog readers.

Jekyll

Long story short, I remembered that I’ve somewhere heard about jekyll. Jekyll is a static site generator. You can write markdown files and it transforms it into a website or blog with static html pages. No database required. I didn’t even search for another one, I just started with that one. Why? Fulfills all the above mentioned requirements. Plus I wanted to try it anyway. Double Plus, you can use a github repository to host it and get it online via github pages with no effort at all. That means the whole maintenance thingy is gone. I won’t have to make a backup - I have the code hosted on github and one version checked out on my machine - that’s good enough for me. Github is taking care of the whole hosting stuff. And it’s completely free - as long as it is a public repository.

Hmm a public repository? Doesn’t that mean that all my secret blog stuff is going to be visible to all of you? Yes it does, but it does not matter. Why? Because it’s only static pages. No database user and password in any config files. No secret api-keys. No nothing. You can of course view the repository (and send a pull request to fix typos), but isn’t it more comfortable to view it nicely prepared on a regular blog-looking site?

There is one thing. Github does not allow to use plugins for jekyll on github pages - at least not if github is taking care of doing the site generation. This means I either have to generate the site myself and push the result to github or I can’t use plugins. If I find myself caring about that enough in the future I may switch the site to my own webspace. Until then I’m going to use the github pages.

Hmm .. after each paragraph I think I should start showing what I did to get this blog running, but every time there is something else I want to mention first. Enough with that now! What I did was the following.

Getting started

In order to use jekyll you first need to install Ruby and RubyGems. That’s easy enough on a linux system and is described in detail on the jekyll installation guide. Also then downloading and initializing the jekyll system is very easy and done via the command

gem install jekyll bundler
jekyll new papauorg-blog

where papauorg-blog is the name of the folder that gets created and the jekyll files get placed in. I created a new git repository in that same folder so I can experiment and try things without breaking anything along the way.

git init
git add .
git commit -m "Create empty jekyll template"

The jekyll bundler already added a .gitignore file that excludes the folder for the generated page as well as other cache and metadata directories.

Favicons

Next step which is of course one of the most important ones I can think of for the blog: I need to provide an awesome favicon. For this I went to the RealFaviconGenerator and generated a set of icons. Extracted the files in the root of the jekyll and made sure that the provided link and meta tags are included in the head of the page. For this I created the head.html file in the _includes folder and added the favicon link and meta tags. What this does is overwrite the head part of the theme you are using. In my case it was the minima theme. If you want the site to look like the theme even after you insert your links, you have to take care that the themes contents are also part of the newly created head.html page. To find out what has to be included find the head.html in the themes gem folder and put that content to your own head.html. To find the themes gem folder you can execute:

#minima is the theme name
bundle show minima 
# Returns the path --> /var/lib/gems/2.3.0/gems/minima-2.1.1
# look in the _includes folder

Afterwards my head.html file looked like this:

<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1">

  <title>{% if page.title %}{{ page.title | escape }}{% else %}{{ site.title | escape }}{% endif %}</title>
  <meta name="description" content="{{ page.excerpt | default: site.description | strip_html | normalize_whitespace | truncate: 160 | escape }}">

  <link rel="stylesheet" href="{{ "/assets/main.css" | relative_url }}">
  <link rel="canonical" href="{{ page.url | replace:'index.html','' | absolute_url }}">
  <link rel="alternate" type="application/rss+xml" title="{{ site.title | escape }}" href="{{ "/feed.xml" | relative_url }}">
  
  {% if jekyll.environment == 'production' and site.google_analytics %}
  {% include google-analytics.html %}
  {% endif %}

  <!-- favicons -->
  <link rel="apple-touch-icon" sizes="180x180" href="/images/favicons/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/images/favicons/favicon-32x32.png">
  <link rel="icon" type="image/png" sizes="16x16" href="/images/favicons/favicon-16x16.png">
  <link rel="manifest" href="/images/favicons/manifest.json">
  <link rel="mask-icon" href="/images/favicons/safari-pinned-tab.svg" color="#5bbad5">
  <link rel="shortcut icon" href="/images/favicons/favicon.ico">
  <meta name="msapplication-config" content="/images/favicons/browserconfig.xml">
  <meta name="theme-color" content="#ffffff">
</head>

As you can see I chose to put the favicons and config files of the generator under a sub directory images/favicons to keep the root level tidy. Jekyll copies files and folders that do not have a specified meaning for jekyll to the generated site so the favicons will be available there. Now commit that change and if I want to bookmark my own blog on different devices I get beautiful favicons now. Time for a first check. Execute

jekyll serve

and visit http://localhost:4000 and view your empty blog with awesome favicon.

Some love for search engines

For the second target audience, the random googlers, to find this page, a little more information on the page is helpful. I don’t really know much about search engine optimization but if I can boost the ranking by nothing more than installing a plugin then I’ll give it a try. The plugin is called jekyll-seo-tag and it’s also supported on github pages. The installation is simple and it adds a few tags to the pages that are used by search engines.

Add the plugin to the _config.yml, install it via gem install jekyll-seo-tag and add it to the Gemfile as described in the docs.

After that fill the information in the _config.yml and include a {% seo %} in your custom _includes/head.html file for the tags to be included in your pages. After doing that some tags like <title> were in the pages twice. I removed these from my head.html file and used the ones coming from the plugin instead.

Github pages

As described earlier I will use github pages for hosting this blog. At least until I feel the need to move somewhere else. To get started, the first step is to go to github and create a new repository named yourgithubusername.github.io so in my case: papauorg.github.io.

Awesome. Now follow the instructions in the overview to add the remote to your local git repository and push it up to github.

git remote add origin git@github.com:youruser/youruser.github.io.git
git push -u origin master

That’s it! Now my very first blog is hosted under https://papauorg.github.io. If you see an 404 error on your github page, then maybe you need more patience. The first page build can take a while. If it’s still not there after a few minutes, take a look at the troubleshooting guide. In my case I requested my page before pushing first and the 404 page was cached. Disabling the browser cache didn’t help, because it was cached by a cache server of github. A little trick helped me to view the page the first time, I just called https://papauorg.github.io?some=queryparam to trick the cache server to load the page from the built source and making sure the page works. Later I pushed a new commit to the page and found out that this solved the caching problem and I could use the regular URL.

Github pages are available over a secure connection (https) which is pretty cool and all, but I really want to use my own domain for my blog. Custom domains are supported by github, but as far as I can tell it’s not possible to have a secure connection with custom domains. That is slightly annoying but nothing we couldn’t fix.

Cloudflare

Cloudflare to the rescue! Cloudflare is a service, that can be placed before your website. It can handle the SSL/TLS certificates for your domains, provides DDOS protection and other useful tools that I’m not going to describe here. I just want my domain + TLS for my blog. For this I did the following steps:

1. Create a new Cloudflare account
2. Let Cloudflare scan your domain and overtake the DNS records. It guides you through this process when you first login.
3. Change the DNS Server that should be responsible for your domain to the ones Cloudflare suggests. In my case I had to login to my current provider and change the nameservers via the web administration interface.
4. Create a CNAME DNS record e.g. blog.papau.org that points to papauorg.github.io in the cloudflare DNS settings
5. Add a custom domain for your github page by clicking the Settings button while in the youruser/youruser.github.io repository.
6. If not already the case activate the Always use HTTPS setting in cloudflare.

I had to wait a few minutes for the changes to take effect, but then I got this amazing view:

Content

So now there is only one thing left to do. Add some content. And nothing is simpler than that, just copy all of this blog post, format it in markdown and create a new 2017-10-10-how-i-started-my-blog.md in the _posts folder in jekyll. Commit, push and visit the page. PROFIT!

Conclusion

I finally started my blog with markdown, jekyll and the tool developers know and love - git. It took me a while to write this post and get it online, but I’m sure it can be done faster. But I really love to take long breaks - a lot of them. :) I’m glad that the first step is done and hope I keep updating this thing. Well, we’ll see …