Managing Allied Telesis Switches The Devops Way Using Go Language

Allied Telesis (AT) is a small and historical network equipment manufacturer, with headquarters in Japan and the United States. Its global turnover may be less than 1/100 of the turnover of the biggest global network equipment manufacturers.

But AT switches, for instance AT X230 line, are some of the cheapest switches compatible with SDN and OpenFlow. In fact, every managed AT switch is OpenFlow capable, as mentions on AT website. Buying a license is required to activate these functionalities. These capabilities are very interesting in various situations. You may imagine to test or deploy innovative functionalities, like filtering traffic or implementing new fail-over mechanism at the LAN level on this hardware.

AT switches programming interfaces

We met these switches for the first time in an infrastructure to test SDN mechanisms. As we always try to stick to the devops way of managing network, we looked for the programming interfaces we are familiar with, Ansible or Rest API, to manage these switches. Unfortunately, no Ansible module is published for these switches; and, as far as we know, they don't offer any REST API. Other AT equipments offer a REST API.

The only alternative to a manual CLI configuration - the CLI is very similar to other major vendors CLI - is to implement a program to ssh into the switch and send automatic commands to it this way. In this post, we'd like to share our first thoughts about this method.

Go programming language as an alternative to Python to write network management program

Python is a de-facto devops programming language standard. We decided to work with Go, for this project, because we know that some great devops teams have switch to Go for good reasons:

• Compiling is very fast and painless
• Binary code is smaller and faster
• Go offers a stronger safeguard to the developer, type-check, memory-check, variables management, ...

Compare to Python, Go has certainly many weakness as far as devops is concerned, but we think that compiled languages could soon gain strong momentum because compiling isn't a great fuss anymore and they still have an edge in performance.

To conclude this first experiment on AT switches management with go programming language, we provide you with a first (very incomplete and raw) tutorial to manage AT switches automatically.

Installing Go using an Ansible role

Go can be installed system-wide or in user space. In this experiment, we install it in user space.

A guide is published on golang official site: see here.

We always configure Linux stations using Ansible. We define a golang role as follows:

# roles/golang/tasks
# golang
- name: check if golang has already been downloaded
  stat:
    path: "{{working_directory}}/{{golang_archive_file}}"
  register: golang
- name: download golang installation archive
  get_url:
    url: "https://dl.google.com/go/{{golang_archive_file}}"
    dest: "{{working_directory}}"
    sha256sum: "{{golang_sha256}}"
  when: golang.stat.exists == False
- name: test if golang is already installed
  stat:
    path: "/home/{{homeuser}}/go"
  register: golang_installed
- name: install go in home directory
  unarchive:
    src: "{{working_directory}}/{{golang_archive_file}}"
    dest: "/home/{{homeuser}}"
    owner: "{{homeuser}}"
    group: "{{homeuser}}"
    mode: '755'
  when: golang_installed.stat.exists == False
- name: add golang to bashrc for {{homeuser}}
  lineinfile:
    path: "/home/{{homeuser}}/.bashrc"
    regexp: 'go/bin'
    line: "export PATH=\"/home/{{homeuser}}/go/bin:$PATH\""
    state: present
    insertafter: EOF
- name: set GOPATH environment variable in bashrc
  lineinfile:
    path: "/home/{{homeuser}}/.bashrc"
    regexp: 'GOPATH'
    line: "export GOPATH=\"/home/{{homeuser}}/go/bin\""
    state: present
    insertafter: EOF

The task depends on the following variables :

We set the value of golang variables in a golang ansible role var default file:

# roles/golang/defaults/main.yml
golang_archive_file: go1.13.5.linux-amd64.tar.gz
golang_sha256: 512103d7ad296467814a6e3f635631bd35574cab3369a97a323c9a585ccaa569

If you use ansible playbooks, you just have to create the ad-hoc repositories, copy-paste the file above and add the role to your ansible playbook.

You'll find the corresponding files in our GitHub repository, but you'll probably find or develop a more concise version of this quick and dirty script with little effort.

Adding go support to Visual Studio Code

We work with Microsoft Visual Studio Code. Visual Studio Code automatically detects .go files and adapts its configuration to ease your development experience as long as you set the GOPATH environment variable, as we did in the ansible task. In our environment, GOPATH is defined in .bashrc under the home of the user.

Connecting to an AT switch using SSH with Go

Go includes a package to manage SSH. The documentation is available here. There are many tutorials about how to use the library in a correct manner, but not so many that really work. We tried a lot. This one is the simpler and best one we found.

Managing and loading authentication keys on the client

The switches are configured to accept SSH key authentication. If you think that using login/password is simpler, you're wrong. Authenticating programmatically and securely with login/password is a complex task; you have:

• to store the login/password in a secured file
• to setup an environment variable to pass it to the program

At the end of the day, public key authentication is easier to manage than login/password. You can begin with very simple self-signed certificates and, as your skills improve, build a small crypto infrastructure management methodology, with scripts to generate root certificate and client certificates.

You need to know a few things to work with Go ssh library:

• If you use passphrases to protect private keys on the SSH clients, they are ciphered when stored on disk. You have to either produce the password at runtime or pre-load the keys (unciphered) in memory on boot or in a post-boot script.
• On linux distribution, keys loading is managed by ssh-agent
• If you work with Ubuntu distribution (and possibly with other distributions too), ssh-agent is replaced by keyrings. You don't need to install ssh-agent. Keyrings is, in general, already installed and publishes the same API as ss-agent.
• Latests versions of Go validate the certificate of the server. See here and here for further information. To test ssh connection, you may disable the certificate checking. On a production environment, you have to deploy the certificate of the server.

You have to check that the private key you authenticate against your switch is loaded in keyring before you launch the program.

// SSHAgent returns ssh keys registered in the user memory key ring
func SSHAgent() ssh.AuthMethod {
	if sshAgent, err := net.Dial("unix", os.Getenv("SSH_AUTH_SOCK")); err == nil {
		return ssh.PublicKeysCallback(agent.NewClient(sshAgent).Signers)
	}
	return nil
}

Generating a self-signed server key on the switch

atswitch# crypto key generate rsa 4096

Enabling public key authentication on the switch

You enable public key authentication on the switch:

atswitch> enable
atswitch# configure terminal
atswitch# ssh server v2only
atswitch# ssh server authentication publickey
atswitch# no ssh server authentication password
atswitch# ssh server allow-users <USERNAME>
atswitch# service ssh
atswitch# ssh server scp

Copying the client public key from a TFTP server to the switch

You need to generate a public-private key pair for your client. You then store the private key file on a secured folder (check the permission). And you copy the public key to a TFTP server directory (docker tftp, EZtftp...). Afterward, you tftp-copy the public key to the switch and attach it to the user :

atswitch# copy tftp://<PATH TO THE KEY> flash:/lankey.pub
atswitch# conf t
atswitch# crypto key pubkey-chain userkey <USERNAME> lankey.pub
atswitch# exit

Debugging ssh connections

In our example, when we test the connection, the client generates the following error message : Too many authentication failures for USERNAME.

To investigate this cas, we activate debug mode on the switch:

atswitch# en
atswitch# conf t
atswitch# debug ssh server brief
atswitch# exit
atswitch# terminal monitor

We then find the culprit in the log:

16:26:01 awplus sshd[11841]: Authentication refused: bad ownership or modes for directory /flash/.home/<USERNAME>/.ssh

The way we uploaded the public key triggers a strange ownerhip problem on the switch underlying linux system. We opt for dropping and registering the key again as follows:

atswitch# en
atswitch# cd flash:
atswitch# cd .home/
atswitch# rmdir force <USERNAME>
atswitch# cd ..
atswitch# conf t
atswitch# crypto key pubkey-chain userkey <USERNAME> lankey.pub
atswitch# exit

New ssh connection trial : it works :-).

Connecting to the switch with the Go program

Connecting to a switch is similar to connecting to another ssh server.

The code for various steps can be found in the aforementioned tutorial:

1. Set up and configuring the client
2. Connect to the host
3. Create a session
4. Redirect stdin, stdout and stderr
5. Start a remote shell

To test our approach, we modify the code to send commands to the switch :

commands := []string{
		"enable",
		"configure terminal",
		"interface port1.0.1-1.0.10",
		"description lan port",
		"exit",
		"no ip route 0.0.0.0/0 192.168.123.1",
		"ip route 0.0.0.0/0 192.168.123.254",
		"exit",
		"write mem",
		"exit",
	}
	for _, cmd := range commands2 {
		_, err = fmt.Fprintf(stdin, "%s\n", cmd)
		if err != nil {
			log.Fatal(err)
		}
  }

Final thoughts

It seems easy to manage network equipment with a few lines of Go code. But there are still a lot of work to do to get from this first test to a production-ready program:

1. Managing the SSH connection
   When configuring devices, especially network equipment, you sometimes have to reset network interfaces. The connection goes down and up. You have to deal with this point and reconnect to get the state of the equipment after the configuration is applied.
   More broadly, TCP connection timeout are slow to trigger. The client detects that the connection is down in seconds, sometimes in minutes. You have to tune and monitor the connection parameters to improve network fault detection.
   
   
2. Committing configuration change
   The example doesn't check that the changes are done and persistent :
   • Sending a command
   • Checking that that no error araises
   • And then, committing a group of commands by sending a write mem
     
     
3. Idempotence
   If you run the example code twice, you don't know what it will do. In a production environment, you'll want to enforce some kind of idempotence.