Twitter Conversations an Agile Case Study

Twitter Conversations

From time to time Twitter Users engage in conversations. It is a royal pain to follow these conversations using the native Twitter programs including the web version of Twitter.  As an exercise I began creating an application that would allow me to follow Twitter conversations. This is the story of creating that application.

User Story
     As a Twitter user I want to be able to follow conversations between other Twitter users.

This post will accomplish two things.
1. Fulfill the requirements of the user story.
2. Create a small application that can be used by the ALT.NET/Agile/Lean community to learn numerous agile software development concepts including:  test driven development, dependency injection, how good design affects testing, etc.

Note: The second item in the list came from conversations at KaizenConf (www.kaizenconf.com  kaizenconf.pbwiki.com). These conversations centered on the need to create a learning application. I am submitting this application as a case study so that we all may learn from it.
To accomplish the story we need the following parts:

1. A mechanism for communicating with Twitter.
2. Query Twitter UserID’s based on list of screen names.
3. Query messages for each user in list of screen names.
4. Filter messages where the message content contains any of the names in the list of screen names.
The first step is to define a domain model. The following domain model represents the different elements that will help in fulfilling the requirements of the user story.

• User
    o UserID(int)
    o ScreenName(string)
    o UserName (string)
    o Followers (int)
• Message
    o MessageID(int)
    o MessageDate(string)
    o MessageContent(string)
    o UserID (int)
    o UserName(string)

Unit Tests (Part 1)
After creating the domain model began creating unit tests for my API. I decided to divide my API into two subject areas: Users and Messages. The User API is used to query Twitter User information. The Message API is used to query Twitter Status(s) aka messages. The following code is the first set of unit tests.

<TestFixture()> _
Public Class UnitTests

  <Test()> _
  Sub Should_Return_My_Following()
    Dim UserAPI As New Twitter.API.User
    Assert.That(UserAPI.GetUsers().count > 0)
  End Sub

  <Test()> _
  Sub Should_Return_Single_User()
    Dim UserAPI As New Twitter.API.User
    Assert.That(UserAPI.GetUser(UserAPI.GetMyUser().UserID.ToString).ScreenName _
       = “rodpaddock”)
    End Sub

  <Test()> _
  Sub Should_Return_Single_User_ByName()
      Dim UserAPI As New Twitter.API.User
      Assert.That(UserAPI.GetUser(“rodpaddock”).ScreenName = “rodpaddock”)
  End Sub

  <Test()> _
  Sub Should_Return_My_User()
      Dim UserAPI As New Twitter.API.User
      Dim User As Twitter.Domain.User = UserAPI.GetMyUser
      Assert.That(UserAPI.GetMyUser().ScreenName = “rodpaddock”)
  End Sub

  <Test()> _
  Sub Should_Return_My_Messages()
      Dim MessageAPI As New Twitter.API.Message
      Assert.That(MessageAPI.GetMyMessages().Count > 0)
  End Sub

  <Test()> _
  Sub Should_Return_Messages_By_User()
      Dim MessageAPI As New Twitter.API.Message
      Assert.That(MessageAPI.GetUserMessages(“rodpaddock”).Count > 0)
  End Sub

  <Test()> _
  Sub Should_Return_Messages_From_MultipleUsers()
      Dim MessageAPI As New Twitter.API.Message
      Dim UserAPI As New Twitter.API.User
      Dim UserQuery As New List(Of Twitter.Domain.User)
      UserQuery.Add(UserAPI.GetUser(“rodpaddock”))
      UserQuery.Add(UserAPI.GetUser(“chriswilliams”))
      Assert.That(MessageAPI.GetMultipleUserMessages(UserQuery.ToArray).Count > 0)
  End Sub
End Class

This set of unit tests does a good job of covering our API’s and insuring they all work as planned. But the tests do have a number of design flaws. For now we’ll leave them alone and look at some of the code they are testing.

Twitter Basics

Now that you have looked at the domain model and the unit tests, take a look at the process of fulfilling these unit tests. For this exercise consider the following unit test.

<Test()> _
Sub Should_Return_Single_User_ByName()
   Dim UserAPI As New Twitter.API.User
   Assert.That(UserAPI.GetUser(“rodpaddock”).ScreenName = “rodpaddock”)
End Sub

To fulfill the requirements of this test the following things must occur:
1. Application must connect to Twitter
2. Application must retrieve user data based on the passed in username
3. The data returned must be transformed into a usable User domain object.

The following code fulfills the requirements of this test:

Const GetUserURL As String = “http://twitter.com/users/show/<<USERID>>.json“
Public Function GetUser(ByVal id As String) As Twitter.Domain.User

    Dim Credentials As New NetworkCredential(“<<YourUserName>>”, “<<YourPassword>>”)
    Dim Request As HttpWebRequest = _
        HttpWebRequest.Create(GetUserURL.Replace(“<<USERID>>”, id.ToString))
    Request.Method = “GET”
    Request.Credentials = Credentials

    Dim Response As WebResponse = Request.GetResponse
    Dim Reader As New StreamReader(Response.GetResponseStream)
    Dim Results As String = Reader.ReadToEnd

    Dim JsonSerializer As New System.Web.Script.Serialization.JavaScriptSerializer
    Dim UserObject As Object = JsonSerializer.DeserializeObject(Results)

    Return New Twitter.Domain.User With { _
       .UserID = UserObject(“id”), _
       .UserName = UserObject(“name”), _
       .ScreenName = UserObject(“screen_name”), _
       .Followers = UserObject(“followers_count”)}

    End Function

This code performs the following tasks:

1. Created a  NetworkCredentials object with your Twitter user name and password. All twitter requests use basic authentication.
      Dim Credentials As New NetworkCredential(“<<YourUserName>>”, “<<YourPassword>>”)

2. Created an HttpWebRequest based on Twitter’s REST API. This call will return data in JSON format as specified via the .json extension on the URL.
     Dim Request As HttpWebRequest = _
      HttpWebRequest.Create(GetUserURL.Replace(“<<USERID>>”, id.ToString))
      Request.Method = “GET”
      Request.Credentials = Credentials

3. Read data from the web request stream using a Stream Reader.
       Const GetUserURL As String = “http://twitter.com/users/show/<<USERID>>.json“
       Dim Response As WebResponse = Request.GetResponse
       Dim Reader As New StreamReader(Response.GetResponseStream)
       Dim Results As String = Reader.ReadToEnd

4. Deserialize the JSON data into an array of Name/Value pair data using the .Net JSON serializer.
       Dim JsonSerializer As New System.Web.Script.Serialization.JavaScriptSerializer
       Dim UserObject As Object = JsonSerializer.DeserializeObject(Results)

5. Turn the returned user information into a User domain object
       Return New Twitter.Domain.User With { _
         .UserID = UserObject(“id”), _
         .UserName = UserObject(“name”), _
         .ScreenName = UserObject(“screen_name”), _
         .Followers = UserObject(“followers_count”)}
 
That’s pretty much how all communication works with Twitter. If you closely examine the code you will find a number of design flaws. Basically this code was developed as a spike: “Let’s see how we can pull data from Twitter”. The basic pattern: authenticate, request and parse was cut and pasted into each API call. I did this knowing that that the code (and tests) would be refactored into a proper design.

Refactoring the Code

After completing the first run through the code (getting it working) it was time to refactor. I gave myself some of goals:
1. Reduce the amount of redundant code.
2. Create a better designed set of test code (remove redundancy)
3. Prepare the code for dependency injection (current tests require internet connections)

The “authenticate and request” process received the first refactoring. This process can be pseudo coded as follows:

Given a valid set of user credentials and a REST URL return a string result.

From this pseudo code we created a Twitter communication class.

Imports System.Web
Imports System.Net
Imports System.IO

Public Class TwitterRequest
    Private UserName As String = “”
    Private Password As String = “”
    Sub New(ByVal UserName As String, ByVal Password As String)
        Me.UserName = UserName
        Me.Password = Password
    End Sub

    Function GetTwitterRequest(ByVal URL As String) As String
        Dim Credentials As New NetworkCredential(Me.UserName, Me.Password)

        Dim Request As HttpWebRequest = HttpWebRequest.Create(URL)
        Request.Method = “POST”
        Request.Credentials = Credentials

        Dim Response As WebResponse = Request.GetResponse
        Dim Reader As New StreamReader(Response.GetResponseStream)
        Dim Results As String = Reader.ReadToEnd
        Return Results

    End Function
End Class

Now we have a wrapped class that needs two items in its constructor (UserName , Password) and has a single method GetTwitterRequest(URL). The GetTwitterRequest(URL)’s job is to return a JSON string that will be used by the subsequent API call.

All of the code necessary for  communicating with Twitter is encapsulated into this class.

Next step was to refactor the User and Message API’s.  The first step involved creating a constructor that accepted a TwitterRequest object. The constructor for the Message API changed to: 

 Dim Communicator As Twitter.Communication.TwitterRequest = Nothing
 Dim JsonSerializer As New System.Web.Script.Serialization.JavaScriptSerializer

  Sub New(ByVal TwitterCommunicator As Twitter.Communication.TwitterRequest)
      Me.Communicator = TwitterCommunicator
  End Sub

These two refactorings reduced the code radically. The following code shows the new GetUser() method.

Public Function GetUser(ByVal id As String) As Twitter.Domain.User
 Dim UserObject As Object = JsonSerializer.DeserializeObject(_
   Me.Communicator.GetTwitterRequest(GetUserURL.Replace(“<<USERID>>”, id.ToString)))

   Return New Twitter.Domain.User With { _
      .UserID = UserObject(“id”), _
      .UserName = UserObject(“name”), _
      .ScreenName = UserObject(“screen_name”), _
      .Followers = UserObject(“followers_count”)}

End Function

The code went from 10 lines of code to 2.  The same results were seen across the entire API. 
Refactoring Tests
Once the API’s were refactored  the tests were then refactored. The following code shows the new set of tests.

Imports NUnit.Core
Imports NUnit.Framework

<TestFixture()> _
Public Class UnitTests
    Dim UserAPI As Twitter.API.User = Nothing
    Dim MessageAPI As Twitter.API.Message = Nothing
    Dim Communicator As New Twitter.Communication.TwitterRequest(“”, “”)

    <SetUp()> _
    Sub Setup()
        Me.UserAPI = New Twitter.API.User(Me.Communicator)
        Me.MessageAPI = New Twitter.API.Message(Me.Communicator)
    End Sub

    <Test()> _
    Sub Should_Return_My_Following()
        Assert.That(UserAPI.GetUsers().Count > 0)
    End Sub

    <Test()> _
    Sub Should_Return_Single_User()
     Assert.That(UserAPI.GetUser(UserAPI.GetMyUser().UserID.ToString).ScreenName _
        = “rodpaddock”)
    End Sub

    <Test()> _
    Sub Should_Return_Single_User_ByName()
        Assert.That(UserAPI.GetUser(“rodpaddock”).ScreenName = “rodpaddock”)
    End Sub

    <Test()> _
    Sub Should_Return_My_User()
        Assert.That(UserAPI.GetMyUser().ScreenName = “rodpaddock”)
    End Sub

    <Test()> _
    Sub Should_Return_My_Messages()
        Assert.That(MessageAPI.GetMyMessages().Count > 0)
    End Sub

    <Test()> _
    Sub Should_Return_Messages_By_User()
        Assert.That(MessageAPI.GetUserMessages(“rodpaddock”).Count > 0)
    End Sub

    <Test()> _
    Sub Should_Return_Messages_From_MultipleUsers()
        Dim UserQuery As New List(Of Twitter.Domain.User)
        UserQuery.Add(UserAPI.GetUser(“rodpaddock”))
        UserQuery.Add(UserAPI.GetUser(“chriswilliams”))
        Assert.That(MessageAPI.GetMultipleUserMessages(UserQuery.ToArray).Count > 0)
    End Sub
End Class

As you can see the testing class now has member variables for each API class and the <Setup()> section instantiates the APIs with the injected communication class.

User Interface

Finally a small WPF application was built to track conversations between users. The following code creates instances of the User and Message APIs, creates an array of user objects from a string (split by commas), retrieves messages for the specified users and finally queries them using LINQ to ferret out a conversation. This code is as follows:
Partial Public Class Main

    Dim UserAPI As Twitter.API.User = Nothing
    Dim MessageAPI As Twitter.API.Message = Nothing
    Dim Communicator As New Twitter.Communication.TwitterRequest(“”, “”)

    Public Sub New()
        MyBase.New()

        Me.InitializeComponent()

        ‘ Insert code required on object creation below this point.
        ‘ Add any initialization after the InitializeComponent() call.
        Me.UserAPI = New Twitter.API.User(Me.Communicator)
        Me.MessageAPI = New Twitter.API.Message(Me.Communicator)
    End Sub

 

Private Sub cmdGetThread_Click(ByVal sender As Object, _
   ByVal e As System.Windows.RoutedEventArgs) Handles cmdGetThread.Click

        ‘– create list of users from comma (,) delimited list of names in text box
        ‘– TODO we should scrub this
        Dim UserQuery As New List(Of Twitter.Domain.User)
        For Each UserName As String In Me.txtCriteria.Text.Split(“,”)
            UserQuery.Add(UserAPI.GetUser(UserName))
        Next

        ‘– get messages for these users
        Dim messages As Twitter.Domain.Message() = _
           MessageAPI.GetMultipleUserMessages(UserQuery.ToArray())

        ‘– filter messages based on who is in the contents
        Dim FilteredMessages = _
           From Message _
             In messages _
         Where MatchesCriteria(Message.MessageContent, Me.txtCriteria.Text.Split(“,”))
        Order By Message.MessageDate Descending

        Me.lstResults.ItemsSource = FilteredMessages.ToArray
    End Sub

    Function MatchesCriteria(ByVal Content As String, _
         ByVal SearchCriteria As String()) As Boolean
        Dim llRetVal As Boolean = False
        For Each SearchString In SearchCriteria
            If Content.ToLower.Contains(SearchString) Then
                llRetVal = True
                Exit For
            End If
        Next
        Return llRetVal
    End Function

Lastly the information is displayed using the following XAML code:

<Window
xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation“
xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml“
x:Class=”Main”
x:Name=”Window”
Title=”Main”
 xmlns:Custom=”http://schemas.microsoft.com/wpf/2008/toolkit“>

<Grid x:Name=”LayoutRoot”>
  <StackPanel Margin=”0,0,0,37″>
    <Button Content=”Get Thread” x:Name=”cmdGetThread”/>
   <TextBox Text=”bellware,pandamonial,chadmyers” 
     TextWrapping=”Wrap” x:Name=”txtCriteria” Width=”622.627″/>
    <ScrollViewer>
    <ListBox Width=”Auto” Height=”500″
      IsSynchronizedWithCurrentItem=”True” x:Name=”lstResults”    >
      <ListBox.ItemTemplate>
  <DataTemplate>
                <StackPanel>
                   <TextBlock Text=”{Binding Path=MessageDate}”/>
                   <TextBlock Text=”{Binding Path=UserName}”/>
                   <TextBlock Text=”———-“/>
                </StackPanel>
  </DataTemplate>
 </ListBox.ItemTemplate>
 </ListBox>
</ScrollViewer>
</StackPanel>
</Grid>
</Window>

The following screen shot demonstrates a conversation that occurred today between three people I follow on Twitter:

 

Next Steps
There is more work to be done here. In the future posts I hope to do the following:

1. Introduce a Dependency Injection container. For this application I want to use StructureMap.
2. Introduce more mocking to remove dependencies on Twitter (maybe we can write a Twitter mock layer)
3. Further refine the design of the libraries. There’s still redundant code left we can remove.
4. Improve the UI and create some alternate UI’s with MVC  or maybe Silverlight
5. Open up the code to contributions from other developers.

Summary

What I hoped to accomplish in this post was a brief introduction to building a useful tool using the agile principles of Test Driven Development and Dependency Injection. Another goal of this post is to open a conversation on these agile principles. A lot of these practices may seem simple to a lot of the folks I blog with here but a lot of folks out there they seem foreign. I appreciate the comments.

The full code for this post can be found at www.dashpoint.com/downloads/TwitterPlayGround.Zip

Note: There are two sections where you will need to use your own username and password.

Thanks
Rodman