How Home Assistant is using Azure Pipelines to automate all the things.

Here at Home Assistant, we love automation! Not only in our homes, but also when doing development work. We do a release every three weeks and it includes code from over 80 different people. All is managed by only two people working on this full-time (thanks Nabu Casa!).

We distribute Home Assistant as a Python package, and as part of our all-in-one system called Hass.io. Hass.io has its own operating system, a supervisor Docker container and the Home Assistant docker container. Besides that, Hass.io supports add-ons, of which we have a set of official add-ons that we also maintain. Everything we do for Hass.io, we have to do five times, once for each of our supported architectures: amd64, i386, armv6, armv7, aarch64.

To still be able to move fast, we spend a lot of time on automating build tasks. After working with TravisCI and CircleCI, we have finally landed on Azure Pipelines.

We started out with Travis but realized that we could significantly speed up our builds by leveraging the CircleCI open source plan that includes 4 workers, test distribution and build artifact caching. We tried to get more workers with CircleCI but adding workers under their new pricing plan was too expensive.

Pascal Vizeli had experience with Azure from a previous job, and he was able to migrate and build all the YAML based pipelines in days. He did this with support from Kees Schollaart, who is a DevRel at Azure but famous in our community for creating the Home Assistant Companion extension for VS Code. He helped us a lot whenever we got stuck.

Azure’s open source plan comes with 10 free agents, and after we reached out to Microsoft, we were provided 20 additional agents to help us automate all the things!

Interested to see what we do with all this Azure power?

This post is about the upcoming changes for the Home Assistant climate integration. These changes are being implemented right now and are no longer up for discussion. If you want to see changes, consider opening an architecture issue.

Starting with Home Assistant 0.96, we're shipping a big architectural clean up of our climate integrations. This effort is lead by Pascal Vizeli.

Over the years, the climate integration has grown from being a simple place for thermostats to hosting ACs, heat pumps and ventilation systems. During this growth, mistakes have been made which resulted in a confusing and hard to work with design. Work is underway to get this all cleaned up and we're going to ship this in Home Assistant 0.96.

The biggest mistake we made is that we conflated operating and operation mode. Operation mode is what you want your thermostat to do, for example, heat the house to 19 °C. The operating mode is what the thermostat is currently doing. Is it heating because the house is too cold? Or did it already reach the target temperature and is it currently idle?

On top of that, we've noticed that a lot of integrations overload operation mode with a wide range of delightful options like "eco", "comfort", "boost" or "sleep". After carefully analyzing all the different climate systems out there, we have come to the conclusion that these alternative operation modes are variations of the existing operation modes (heat, cool, heat-cool), but might have different target temperature ranges or try to heat/cool at a slower pace to conserve energy.

To support as many different thermostats as possible, we have decided to allow climate devices to specify a new "preset" mode that they are operating under. It covers hold mode, away mode, or any custom operation mode that a thermostat includes.

You might wonder, why go for a preset instead of allowing to overload operation mode? On first glance, it might seem fine for an integration to list the possible options and the current chosen option, easy to make a UI for. However, that's not the only way Home Assistant is used. We are also serving as a single place to control any thermostat for external systems like Google Assistant or Amazon Alexa. They have a limited set of operation modes they expect.

A gist of the changes:

• operation_mode has been renamed to hvac_mode to emphasize what the mode is for.
• We split HVAC mode auto into auto and heat_cool. If it's heat_cool, the user has set a temperature range the device has to use heating and cooling to stay within. Auto mode is now limited to devices that are running on a schedule or AI.
• The state of a climate entity is now always equal to the HVAC mode.
• hvac_action is introduced for integrations that know the current action: heating, cooling, idle.
• set_away_mode and set_hold_mode have been merged into set_preset_mode. So instead of turn_away_mode_on(), we will now call set_preset_mode("away").
• The is_on property has been removed. If a device is able to turn off, it should include the hvac mode HVAC_MODE_OFF.
• Property names have been aligned, anything ending with "_list" is now named "_modes".

If you are a maintainer of an integration that has a climate platform or maintain a custom component that has one, make sure you follow the pull request and test your integration during the beta to make sure that it works like expected.

Home Assistant is world's biggest open-source home automation platform. It supports over 1000 devices and services, and it is used everywhere: homes, boats and museums.

Today I want to talk about our UI, what we're trying to solve and how we solve it. Before we dive into why, what and how, let's take a look at our user interface. This is an embedded version of our demo, which is fully interactive.

The embedded demo has been hidden because the screen is too small. Open the demo in a new screen.

When building a platform for the Internet of Things, there are many use cases that warrant a user interface. Note that for the sake of this blog post, we're focusing on graphical user interfaces on devices with screens the size of a phone, or bigger.

Security. Users that want to have a quick glance at their house when they are away from home. They quickly want to check if their garage door is closed or maybe they got notified of unexpected motion in the backyard or water below the sink and need to check out. These users will want to see a logbook with recent security related incidents, the current camera streams and the current state of the doors, windows, water- and motion sensors.

Control all the things. Users want to have a single place to control all their devices. Their goal of the user interface is to quickly see each part of the house at a glance and control lights, switches, thermostats and the music.

Control single room. Users that are building interfaces for devices in each room. These devices need to expose all controls of a single room, with optional extra information about the whole house and options to control other parts of the house.

Monitor. Users want to build their own dashboards to visualize all the current and historical data inside Home Assistant and other databases, like InfluxDB. They want to keep track of their energy usage, track their internet speeds or current commute time.

Administer and Automate. Users that want to have dashboards that contain their controls to administrate Home Assistant. It contains their automations, scripts and various input controls to allow configuring the parameters of their automations and scripts.

The odds are that if you are using Home Assistant right now, your UI is a hybrid of a couple of the mentioned use cases. Todays blog post is going to talk about what technologies we have used to built a UI that allows users to build UIs that fit all these use case.

We have finished the great migration. The result will be released as part of Home Assistant 0.92. The release has been a bit delayed because we had a lot of things to fix! With the migration done, we now consider components and platforms that share the same name to be part of the same integration. Each integration is either a single Python file, or a folder with an __init__.py. file. We have updated the documentation and introduced a new section for integrations.

Home Assistant 0.92 introduces a new manifest.json for integrations. This file, which is optional for custom components, is used by integrations to specify metadata: name, link to the documentation, dependencies, requirements and code owners. We are exploring leveraging manifest.json for additional future features, like tracking breaking changes or allowing custom components to provide config flows and being discovered.

With all these changes, we had to drop a few deprecated things and change some behavior:

• Platforms can no longer be in the directory of the entity component, like light/my_platform.py. Instead, create a new my_platform folder in your custom_components, create an empty __init__.py file and move light/my_platform.py to my_platform/light.py.
• Platforms can no longer have dependencies or requirements. Instead, create a manifest.json in the my_platform folder to specify them, or add REQUIREMENTS or DEPENDENCIES constants to the __init__.py file.
• A platform will now always require the component, if available, to be set up first.
• It is no longer possible to provide translations for components that are contained in a single Python file. Convert them to an integration in a directory.
• If you want to override a built-in integration, you need to specify a manifest.json for your custom integration. Note that we strongly discourage overriding built-in integrations. Instead, if you want to run an integration with custom changes change the integration name. For example if you want to run a custom version of the MQTT integration, named mqtt in Home Assistant:
  • Copy the content of the mqtt folder from the Home Assistant repository to a new folder <config>/custom_components/mqtt_custom/
  • Open mqtt_custom/manifest.json and change the value for domain from mqtt to mqtt_custom
  • Open mqtt_custom/__init__.py and change the value of DOMAIN from mqtt to mqtt_custom
  • Anywhere in your config where you referenced mqtt, reference mqtt_custom. So use mqtt_custom: to specify the host and use platform: mqtt_custom when specifying platforms.

Home Assistant has had a user permission system since Home Assistant 0.82 (released Nov, 2018). Permissions are attached to groups, a user can be part of multiple groups, user permissions are based on merging the permission policies of all of its groups. More detailed info can be found in our docs. Users can create their own groups, but there are also three system groups: "admin", "users" (new in 0.90), and "read-only". All three have access to all entities, but the read-only cannot control any of them. Only the users part of the admin group can access administrative options like managing users and integrations.

Before Home Assistant 0.90, we were not using any of it. Instead, all users are part of the admin group, as that group provides the same access as before we had permissions. With 0.90, we're going to allow switching the users group between the system groups "admin" and "users". When a user is part of the "users" group, they will not be able to administer Home Assistant. The UI will hide the menu items to open the configuration panel or the developer tools.

This feature is however unfinished. We need to audit the Home Assistant code to make sure that all APIs and service calls check the users permissions. So although a user will not be able to use the UI to make changes, there might still be APIs that can be accessed without permissions. And not only do we need to check the Home Assistant code, custom components that register services or APIs will also have to be updated.

So this is where you can help! If you're maintaining an integration, either custom or built-in, read up on the documentation on how to check permissions. Then check that your integration checks permissions appropriately. Once you checked yours, help others check theirs, and help us to audit all services and API calls in Home Assistant.

What about custom groups?​

Although we support the creation of custom groups, don't expect it publicly exposed anytime soon. The reason for this is that it requires a lot of work on APIs that interact with the custom policies. The hardest part is that a custom policy could enable read access to only a couple of entities, and only allow control access to a subset of them. We will need to go through each API to make sure they support this. Lovelace will also need to be updated to support users having their own configuration view.

That being said, if you do want to play with it, it's possible.

Turn off Home Assistant and open up <config>/.storage/auth. Find the key "groups" and add a new group:

{
  "id": "my-custom-group",
  "name": "My Custom Group",
  "policy": {
    "entities": {
      "entity_ids": {
        "light.kitchen": true,
        "switch.ac": {
          "read": true
        }
      }
    }
  }
}

Now scroll down in the file to the "users" key and find the user you want to apply the custom group to. For this user, update their "group_ids" value to be that of your custom group. Note, you should not update the user that has "owner": true, as they will always have all permissions.

"group_ids": [
    "my-custom-group"
],

Now start Home Assistant and log in with the updated user. If all went well, you should see only the kitchen lights and AC switch. Controlling the AC will fail.

A user can be as many groups as you want. Only members of the system-admin group can administer the system. To learn more about the policy format, check the documentation.

Home Assistant is based around components. One type of components is the one that provide an entity abstraction for a type of device, like lights, switches or media players. Historically, to integrate with these abstractions, you would create a platform for this component. If the light component would be light/, the integration would create its platform as light/hue.py. This way, all logic for the lights was kept together.

As Home Assistant grown, so has the capabilities/size of the devices/services that we've integrated with. It's no longer that an integration is just a platform for an entity component, it is five platforms + a component to manage the connection. With integrations that big, having the files be spread out as light/zwave.py, switch/zwave.py, cover/zwave.py etc becomes a maintenance hell. It also makes distribution of custom components harder, as now users need to create files across different folders.

So with Home Assistant 0.87 we started a migration. Instead of looking up platforms for entity components as light/hue.py, we will now look it up as hue/light.py. The content of the file will stay exactly the same. By moving the files out of the entity component folder, we will also be able to break up big entity components into multiple files, to support maintainability.

This change has been implemented mostly backwards compatible with a minor breaking change for custom components: if you override a built-in platform, you will now need to use the new hue/light.py filename format.

With the consolidation of integration files into a single folder, we're also going to enforce a new rule: all platforms will be loaded from the same source as the component. This means that if you want to override a built-in component/platform with a custom version, you will need to copy over all component and platform files, not just the one you want to load. By doing so, we can prevent custom platforms or components from breaking if a Home Assistant upgrade moves internal files/values around.

Notes for custom component developers​

• Components are created in <config>/custom_components/<integration name>/. Always make sure you create at least an empty __init__.py file in that folder.
• If you make a platform for an entity component, put it in the folder with the name of the integration: <integration name>/light.py
• If you want to share an adjusted version of a Home Assistant integration, copy over ALL the files. Do your users a favor and stick to relative imports to avoid having your component break during an upgrade. Example of a relative import is from . import DATA_BRIDGE.

Resources​

Related architecture issues:

• Embedded platforms and the road to packaged components
• Disable partial custom component overlays
• Config structure for embedded platforms

With the release of Home Assistant 0.76, the functions in the homeassistant.remote package will be deprecated and will be removed in 0.77. This package contains functions to call the Home Assistant REST API in a non-async way.

The reason for removing is two-fold: first the code is not being used inside Home Assistant and thus should not be part of Home Assistant. Second, it is not compatible with the new auth system nor do we want to spend the time to make it compatible.

If you want to keep using the methods in homeassistant.remote, feel free to copy the code to your own project.

In Home Assistant 0.69 we introduced the foundation for a new authentication API. We're switching from a single hardcoded API password to a refresh/access token based authentication system (powered by OAuth2).

For Home Assistant 0.73, I've sprinted together with [@awarecan] to ensure that we have reached a minimum viable product of the auth system:

• Users can be managed via a built-in command line script.
• The frontend will ask for username and password to login
• If you opt-in for the new system, the API password will no longer work.
• To not force a hard break with the ecosystem around Home Assistant, a temporary legacy mode has been added to turn API password support back on. This will be removed in the future.

The system is not yet ready for mainstream consumption, we still need to add Hass.io support and a user interface to help guiding the user to create their first user account and to manage users. You can follow (and join!) the work to be done here.

If you're interested in trying it out, keep on reading.

Efforts to modernize and localize the frontend are moving full speed. This blog post will touch upon the upcoming features for Home Assistant 0.71 which should hit the beta channel today and is planned for a release a week from now.

Improved Custom Panel support​

Custom panels allow developers to build panels that will plug into the Home Assistant user interface, with the same possibilities like our other panels like history, map etc. The Home Assistant frontend will manage authentication and the subscription for states of the backend, the panel only has to take care of displaying the data and can offer users option to control things (calling services etc).

This support has been around for a while, but last week we've spend some time polishing our support and have three new additions:

First new feature is that we now allow importing a panel from a JavaScript URL. This is going to be the preferred way to distribute panels moving forward. This also means that you can refer to a panel hosted externally. The user will have to approve before external panels get loaded. It's still possible for users to host it locally (and no approval is needed) by copying the panel to <config dir>/www/your-panel.js and use /local/your-panel.js as the url.

Second new feature is that we can now embed your panel in an iFrame. This allows panel developers to not have to worry about including duplicate web components and it makes it possible to develop React-based panels. In the past, React-based panels didn't work because React doesn't work well inside Shadow DOM (more info).

Third new feature is that we now make a starter kit available to start developing React based panels! The kit contains everything that is needed to start developing React-based panels and share them with the community. Let us know what you're building!

Custom UI: <state-info> and <ha-relative-time>​

If you're building custom UI, odds are that you're using either <state-info> and <ha-relative-time>. Although not officially supported as external facing API, we still want to give a heads up that it's going to be needed to pass the hass object in.

This is necessary because <ha-relative-time> can now be localized thanks to c727 in #1241

With the release of Home Assistant 0.70, we've migrated the build pipeline for our frontend from being based on HTML imports to ES module imports (more on this later). One of the effects of this is that we're no longer using the window object to share classes, data and utilities with other pieces of code.

This might impact you if you dependend on some of this. Examples are Polymer (window.Polymer) or one of our utilitity functions that used to be available as window.hassUtil, window.HAWS or window.hassMixins.

To give developers time to migrate, we're adding a temporary legacy support layer that will expose some of our internals again on the window object. We've already added window.Polymer, window.Polymer.Element and window.Polymer.html. If you're using other specific things from the window object, please let us know.

Th legacy support layer will no longer be included in releases happening after July 2018.