Monday, 21 December 2015

Bug, bug, there is a ladybug!

So, you found a bug? First ensure it is not a ladybug, as those should be set free and not fixed, nor reported, nor digested into tests etc. But when capturing a software bug, you have a few choices. Depending on your time and programming experience, you can choose how to contribute to the project.



Report a bug

Go to the GitHub issue tracker and report your findings. Please include the steps you took to reproduce the issue. Chances are, it eventually get fixed, unless the issue is specific to your environment or cannot be reproduced by following the steps provided.

You know how to write tests?

Great! Please write a test reproducing the issue. And I already included templates for thit case: bug_scenario_test.scala for the integration test and bug_component_test.scala for the unit test. Then please report the bug and attach your test files. High chances of it getting fixed as your tests reproduce the issue and provide the expected behaviour.

You know how to fix it?

For me, this is the best option as I do not have to do anything! Please fix the issue and ideally cover it with tests (refer to the section above). Then do a pull request!

I would really appreciate some feedback and comments on issues you encounter. You can email me with your findings, questions and feedback: cftp @ coldcore.com

Summary

Remember not to fix ladybugs. They are fine as they are, for what they were designed as being!

Friday, 18 December 2015

The quality lies deep in the tests

This chapter is all about unit and integration testing of Akka-FTP. Those tests ensure the project quality. To run unit tests type sbt test and to run integration tests type sbt it:test.

Unit tests

Simple tests with mock objects to test components and commands. Although commands should be tested with integration tests and in combinations with other commands, the custom built FTP client may not support the complexity involved. In this case, a unit test is the only option.

Integration tests

Integration tests start the server and perform FTP operations such as login, browse directories, list files and data transfers. The server uses the in-memory file system, all operations on it do not depend on physical hardware and this eliminates some points of failure. More than that, the in-memory file system can serve as an example to implement your own file system. To test the server there is the client with basic FTP actions. The client connects to the server, sends commands, receives and verifies replies and transferred data.

Points of failure

These ports need to be available: 2021, 8021 (optional), 6001, 6002, 6003 and 6004. Anything bound to those ports will cause the integration tests to fail. The port numbers are hardcoded, to change you need to edit the source.

The client binds to 6004 with the PORT command, which is released after a data transfer. But OS takes time to release the port. In the scenario where two data transfers issued one after another with the PORT command, the second will fail - the client binds to the same port (and it succeeds) but the server cannot connect to it and aborts the data transfer. This scenario works fine given time between data transfers. It may be improved in the future.

Some of the data transfer tests use combination of PORT and PAVS to ensure both of them work. There should be enough time for OS to release the 6004 port before it gets used again, but sometimes it fails. In this case, please run the integration tests again.

Test framework

I decided to go with ScalaTest and Mockito. Well documented, easy to use and excellent DSL for matchers. Plus, I have been using those for years and very rarely ran into issues testing stuff.

Summary

This chapter sheds some light on how Akka-FTP is tested, its pitfalls and the frameworks used. The real reason for testing is to capture bugs and prevent weird behaviour. In the future post I will explain how you should use tests if you find a bug. So stay tooned!

Wednesday, 16 December 2015

Akka-FTP release 1.0

The release of Akka-FTP 1.0 is ready! I have to be honest, it was rushed and has missing features and probably bugs. Let me briefly tell you what Akka-FTP can and cannot do.

Can do

  • Execute these FTP commands:
    abor allo appe cdup cwd dele eprt epsv
    list mdtm mkd mlsd mlst mode nlst noop
    pass pasv port pwd quit rest retr rnfr
    rnto size stat stor stou stru syst tvfs
    type user
  • Implement your own commands, filesystems, logins, sessions
  • Connections provide limited information through user's session
  • Web dashboard with basic actions
  • IPv4 and IPv6 support

Cannot do

  • Secured connections
  • Unimplemented FTP commands:
    feat help opts site
  • Timeout connections
  • Everything not listed in the can do section

Road map

  • Add the missing features from the cannot do list. 
  • Add more tests to capture bugs.
  • Improve web dashboard

Summary 

I am looking forward for suggestions and pull requests to improve the project. As my time is limited I cannot tell when the next release will be out or bugs fixed. If you encounter strange server behaviour then there are integration and unit tests which you can reuse to help me identify and fix issues - this subject will be covered in another post.

Monday, 14 December 2015

My very own FTP command

Greetings everyone, today I will show you how to add your custom command to the Akka-FTP server. You will need a decent IDE, preferably IntelliJ, and the knowledge of Scala. The example for this chapter available here and compile against Akka-FTP tag 1.0. Let's get started!

Components wiring

FTP state is the central piece which every other component has access to. It provides the command factory dependency which in turn creates a command.

FTP command

First stop in our journey will be a command implementation. Something dead simple: X-TIME command that replies with the server current time. Optional parameter specifies the date format.

case class XTimeCommand(param: String, session: Session) extends Command {
  override def exec: Reply = {
    val sdf = new SimpleDateFormat(
                if (param.isEmpty) "dd/MM/yyyy HH:mm:ss" else param)
    val date = sdf.format(new Date)
    Reply(200, s"Server time $date")
  }
}

All commands must extend the Command trait and take user's Session as the mandatory argument.  The optional argument param is a text user enters after command's name, e.g. X-TIME ddMMyyyy where X-TIME is the command's name and ddMMyyyy is the parameter. The exec method returns a Reply.

Command factory

Now that the command is ready, the command factory needs to create it as the response to user's X-TIME input. For that, a new implementation of the factory is needed. It is very easy, just extend the default command factory as this.

class CustomCommandFactory extends DefaultCommandFactory {
  def mycmd(name: String, param: String, session: Session): Option[Command] =
    Option(name match {
      case "X-TIME" => XTimeCommand(param, session)
      case _ => null
    })

  override def cmd(name: String, param: String, session: Session): Command =
    mycmd(name, param, session) getOrElse super.cmd(name, param, session)
}

Our new factory has the mycmd method which creates a new command if a user sends X-TIME input to the server. Please note that this method is a copy/paste from the command factory itself, I did not write anything new. If a command cannot be created then the factory falls back for the default method.

FTP state

The new command factory needs to be provided by the FTP state. Extend it!

class CustomFtpState(...) extends FtpState(...) {
  override val commandFactory = new CustomCommandFactory
}

The real implementation takes many parameters, but I did not feel like listing them and the parameters may change with the release.

The final piece

The new components are ready and wired, now what? How to tell the server to use those? The final piece of this puzzle is a launcher which will start the server with the new FTP state. Here is the source.

class CustomLauncher extends Launcher {
  override def createFtpState(system: ActorSystem): FtpState = {
    // omettied code to read settings
    new CustomFtpState(...)
  }
}

object main {
  def main(args: Array[String]) = new CustomLauncher().start()
}

Test the new command

Now it's time to test what we have accomplished, start the server! Type sbt run and choose the new main method. Connect to the server with telnet:

telnet localhost 2021
Connected
$ X-TIME
200 Server time 02/12/2015 14:56:25
$ X-TIME yyyyMMdd
200 Server time 20151202

Summary 

In this chapter I explained how to extend Akka-FTP components and start the server with new dependencies. This is the basics of extending the Akka-FTP and you should follow those steps to plug-in your own add-ons to the server. Enough for today, thanks for reading!

Wednesday, 9 December 2015

Akka-FTP structural overview

Today I will show you the Akka-FTP architecture and what is going on under its hood. Let's take a bird view onto the overall structure and the flows between the core components. And we will look only at the FTP side of the project, no REST nor UI will be explained in this post.

The big picture

Here it is, every major component fitted nicely into one picture, thaks to Lucidchart! I understand, a bit of explanation is still required. First things first, red boxes are the actors. A user connects to the control connector at the very top, which spawns (creates label on the diagram) a control connection, which in turn creates user's session etc etc. FTP state is the core piece with all the dependencies such as command factory, file system and more. It is directly or indirectly referenced by every other component. Session is specific to a given user. Control connection accepts user's commands and sends replies to the user. Data connection transfers files and when the transfer completes, it notifies the control connection to send a reply. Task executor uses slave actors to handle commands and provide replies back to the control connection.

Command is an FTP command a user sends to the server. It does what it needs and outputs a reply. Reply is then sent back to the user.

The last piece worth explaining is how the data connection gets created. Both data connector and data connection initiator create it. When a user sends a PORT command, the data connection initiator attempts (initiates - hence the name) a connection to the user machine, using the IP address and port supplied. On the other hand, when the user sends a PASV command, the user then needs to make a connection to the data connector. At the moment of this writing, there is no data connector yet thus the PASV command is not supported, but eventually it will be implemented.

Global and user actors

Server eventually creates the following 3 actors for every user: control connection, data connection, data connection initiator. Those actors exist only while the user is connected and die afterwards. The rest of the actors not related to users and exist while the server is running.

Summary

This chapter should help you to understand how the FTP side of the project works. Something that you may need if you want to extend the server with your own addons, otherwise, why would you bother with this project in the first place? Stay tooned for more posts!

Monday, 7 December 2015

Introduction to Akka-FTP

The Akka-FTP project is the reactive FTP server which uses Akka actors under the hood. At the moment of this writing it is still in development but the core functionality is implemented. It is the fully functional FTP server with some of the non-vital bits missing.

In this chapter I will explain how to build and run Akka-FTP out of the box.

Download and start the server

  1. Download or clone the project, it is hosted on the GitHub here.
  2. Install SBT, here is the guide.
  3. Go into the akka-ftp directory and type sbt run
At this point SBT will download all the dependencies, compile the source code and run the launcher class. This will take a minute or two. After the FTP server starts up, in the console you should see something like this:
10:17:40 INFO [...] - Bound Akka FTP to 0.0.0.0:2021
10:17:40 INFO [...] - Bound Akka FTP REST service to 0.0.0.0:8021 

That's it, the server is now running. Get your favourite FTP client and connect to localhost:2021. Here how it looks by using telnet:
telnet localhost 2021
Connected
220 Welcome to Akka FTP - the open source FTP server
$ USER anonymous
331 Guest login okay, send your complete e-mail address as password.
$ PASS anon@localhost
230 User logged in, proceed.

To stop the server either ^C or send a GET request to http://localhost:8021/api/action/shutdown

The web dashboard

While still connected to the server, open a web browser and point it to http://localhost:8021 which will bring up the dashboard. Default authentication username and password is "admin/admin" which you can change in the userstore.properties file inside the project resources folder.

The dashboard allows you to see the server information such as bytes transferred and users' connections. You can also stop the server from the server control section.

Configuration

You can change server settings in the application.conf file in the project resources folder. The file is straightforward and self-explanatory. Note, that leaving the hostname property empty will bind the service to all host names.

Conclusion

This FTP server is the example of Akka, Spray and AngularJS working together. In the future chapters we will dig the source code and I will explain how you as a developer may add your own code to the server functionality.