Online RSA Key Converter

Online RSA Key Converter

RSA sign is so popular over the wire for security reason. But there are different formats of representations. So I created an tool to converter RSA keys from XML to PEM and vice versa. Which is not easy to find a tool do it bidirectionally online.

Advertisements

Steps To Big Data: Hello, Cascading

In previous post Steps To Big Data: Hello, Pail, we have defined our fact model and shown how to use Pail to store out data. What is fact model?

A Fact Model, as we see it, structures basic knowledge about business operations from a business perspective. “Basic” means that the knowledge it represents cannot be derived or computed from any other knowledge. It that sense, a Fact Model is a crucial starting point for developing more advanced forms of business knowledge, including measures and rules.

As described above, fact model cannot be derived or computed from any other knowledge. In other words, its purpose is to generate data. The process of generating data from fact model could be illustrated as below.

Cascading supports to create and execute complex data processing workflows on top of Hadoop using any JVM-based language(Java, Scala, JRuby, Clojure, etc.) Most cascading related libraries are in the Maven Repository http://conjars.org/

We are going to use cascading to count users’ actions.

Continue reading

Steps To Big Data: Hello, Pail

Just before I had my first anniversary holiday since I had joined communityengine. I started my new role in the recommendation team. The team is relatively new, so we are open to use virtually any technologies.

In the book Big Data (MEAP), Nathan Marz describes a framework calls Pail (dfs-datastores) which is a data storage solution on top of hadoop. It supports schema, merging small files into a large chunk for better hdfs performance etc.

We started to use Pail for our data collection. So what have we done?

  • a data schema defined in protobuf
  • an implementation of PailStructure
  • a job to run the data collection

Ready to dig deeper? Let’s get it started.

Continue reading

Feel NuGet Packages Folder Sucks?

NuGet 1.6 introduced a stunning function for dependency management (the phrase, i reckon, is not so popular in .NET community, while it seems hot in others like Java, Ruby, etc.)

With which, you can Enabling Using NuGet Without Checking In Packages Folder (in our project which winded up had 150MB files).

As our engineering team is using Github as source control tool. Our project repository goes nuts, becomes massive. The model we are using is a single integration repository with a handful of feature repositories.

image

Which means if you want all feature repositories setup you are going to download 150MB x 4 = 600MB… well, kick off git clone and go to make a dozen cups of tea or coffee.

This is not a good practice at all, so this post http://coderjournal.com/2011/12/cleaning-up-your-git-repository-for-nuget-1-6/ helps you get rid of packages, and further, clean the repository history tree (you don’t want to keep those files in the history as well).

What we get instead cleaner and tinier repository? For me, at least I get better sense of dependency management.

MongoDB Bitwise Filtering, Faster Status Querying

What is bitwise filtering? Actually, it is quite old school back to when you are still learning essential programming courses. That was using bits to represent a handful of status and use bits to filter status. Basically, bitwise filtering can be replaced by filter a collection of status iteratively.

Bigwise Filtering in C#

In .NET, you can keep multipule status by using a List or use Enumeration Types as Bit Flags. A quick example looks like as follows,

        [Flags]
        enum Days
        {
            Monday = 0x1,
            Tuesday = 0x2,
            Wednesday = 0x4,
            Thursday = 0x8,
            Friday = 0x16,
            Saturday = 0x32,
            Sunday = 0x64
        }

        class Sport
        {
            public Sport(string name, params Days[] folders)
            {
                Name = name;
                PlayOnList = folders;
                PlayOn = folders
                    .Aggregate((current, messageFolder) => current | messageFolder);
            }

            public string Name { get; set; }
            public Days PlayOn { get; set; }
            public IList PlayOnList { get; set; }
        }

In the source code above, PlayOn accepts Bit Flags and PlayOnList is a typical list of enumeration. To filter a specific sport based on day is as simple as follows.

            var sports = new[]
            {
                new Sport("Football", Days.Wednesday),
                new Sport("Table Tennis", Days.Monday, Days.Wednesday),
                new Sport("Basketball", Days.Tuesday, Days.Thursday),
                new Sport("Surf", Days.Saturday)
            };

            var filter1 = new[] { Days.Wednesday, Days.Monday };
            var filter2 = Days.Wednesday | Days.Monday;

            sports.Where(sport => sport.PlayOnList.All(filter1.Contains));
            sports.Where(sport => (sport.PlayOn & filter2) == filter2);

Two different filtering ways, bitwise and iterating. And their algorithm complexities are O(1) and O(n). To prove this theoretical correctness, I ran a quick benchmark for the code above. The result showed bitwise filtering was roughly 15% faster than list iterating. So, bitwise is a better choice when dealing with multiple, esp. co-existing status.

Bigwise Filtering in MongoDB

In MongoDB, filtering items via its collection operations is fairly easy. The code (based on official mongo C# driver) looks like this,

    // collection is a MongoCollection

    collection.Find(Query.In("PlayOnList", new[] { Days.Wednesday, Days.Monday }));

In terms of bitwise filtering, mongo does not provide direct supoort for that. Having said that, MongoDB integrates an JavaScript interpreter to render itself extremely powerful and flexible. Mongo’s Map/Reduce functionality is based on JavaScript as well. JavaScript is first class citizen in MongoDB.

So, with JavaScript, we can quickly create a script to do the bitwise filtering in MongoDB.

    public string CreateScript(string field, Days filter)
    {
        return String.Format("(this.{0} & {1}) == {1}", field, (int)filter);
    }

Then, use Query.Where() to redesign our query.

    var script = CreateScript("PlayOnList", FolderType.Sent);

    collection.Find(Query.Where(script));

Conclusion

How quick the query could be? I know you care about the efficiency. Basically, with average 3 status, bitwise filtering queries 500% quicker than iteration does.

And that’s not the end yet. It seems MongoDB is using SeaMonkey as its JavaScript interpreter. So there is still a lot of space to optimize on interpreter itself, and with better JavaScript interpreter or VM, the performance of bitwise filtering could be even better.

Testing Environment

  • MongoDB: 1.8
  • mongo-csharp-driver: 1.1

It’s all for Your Git Repository be in Top Shape

In previous post, I have introduced that we need do some git housekeeping (git gc) to ensure the repository is in top shape. But I seriously find I have to run the command very often because my colleagues are all very productive. Every time you do a pull, the files in objects folder increase.

So I decided to find a way to ease the housekeeping, and potentially there are two different ways to do so.

Continue reading

Secure RESTful Web Service by WCF Web API, No HTTPS, Seriously?

What if you want to build a secure RESTful web service without using HTTPS? Simple, you just need to implement your secure way to transfer data from endpoints to endpoints. This article written by @Jeff Atwood can definitely give you some hints on how to ensure your message transfer safe. So I will not explain it in the post and suppose you have the basic knowledge of cryptography.

RESTful Web Service

WCF Web API is really handy to build a RESTful web service. Your service class looks as simple as following

[ServiceContract]
public class PlayerResource
{
    private readonly IPlayerRepository _playerRepository;

    public PlayerResource(IPlayerRepository playerRepository)
    {
        _playerRepository = playerRepository;
    }

    [WebInvoke(Method = "POST", UriTemplate = "Create")]
    public Player CreatePlayer(Player player)
    {
        return _playerRepository.Save(player);
    }
}

And with a little bit configuring in your global.asax file

public class MvcApplication : HttpApplication
{
    protected void Application_Start()
    {
        RouteTable.Routes.MapServiceRoute("api/players");
    }
}

And your RESTful web service is ready for up and running.

Secure RESTful Web Service

In order to make sure the incoming and outgoing messages are safe. I will first introduce Message and EncryptedMessage and a helper class EncryptionHelper to encrypt and decrypt the messages.

public class Message
{
    public string AppId { get; set; }

    public string Data { get; set; }

    public string Id { get; set; }

    public string TimeStamp { get; set; }

    public byte[] GenerateFingerprint();

    public bool ValidateHash(byte[] fingerprint);
}
public class EncryptedMessage
{
    public string AppId { get; set; }

    public byte[] Fingerprint { get; set; }

    ///<summary>
    /// The 3DES key used to encrypt/decrypt the message 
    /// </summary>
    public byte[] Key { get; set; }

    ///<summary>
    /// Encrypted message
    /// </summary>
    public byte[] Message { get; set; }
}
public interface IEncryptionHelper
{
    EncryptedMessage Encrypt(Message message);

    Message Decrypt(EncryptedMessage encryptedMessage);
}

Then, we need to change our service class a little bit to support secure data transfer.

[ServiceContract]
public class PlayerResource
{
    private readonly IPlayerRepository _repository;
    private readonly IEncryptionHelper _helper;

    public PlayerResource(IPlayerRepository repository, IEncryptionHelper helper)
    {
        _repository = repository;
        _helper = helper;
    }

    [WebInvoke(Method = "POST", UriTemplate = "Create")]
    public EncryptedMessage CreatePlayer(EncryptedMessage encryptedMessage)
    {
        Message message = _helper.Decrypt(encryptedMessage);

        Player player = Json.Decode(message.Data);

        player = _repository.Save(player);

        message = new Message
        {
            AppId = "Our App Id",
            Id = Guid.NewGuid().ToString(),
            Data = Json.Encode(player),
            TimeStamp = DateTime.UtcNow.ToString()
        }

        return _helper.Encrypt(message);
    }
}

Now, your messages are secured.

Message Handler

Maybe you have noticed that passing in and out encrypted message is not comfy at all. Can we keep the same simplicity of the service function before introduce secure implementation?

Well, the answer is positive. WCF encourage you to implement cross-cutting function (aka AOP) through a couple of handlers (message handler, operation handler). What we need to do is roll out our own message handler to encrypt and decrypt incoming and outgoing messages. So what we need to do is to pass encrypt/decrypt actions to a message handler. The implementation looks as follows,

public class SecuredMessageHandler : DelegatingChannel
{
    private readonly IEncryptionHelper _encryptionHelper;

    public SecuredMessageHandler(HttpMessageChannel innerChannel,
                                 IEncryptionHelper encryptionHelper)
        : base(innerChannel)
    {
        _encryptionHelper = encryptionHelper;
    }

    protected override Task SendAsync(HttpRequestMessage request,
                                      CancellationToken cancellationToken)
    {
        request = DecryptMessageIfTheMessageIsSecuredMessage(request);

        return base.SendAsync(request, cancellationToken)
            .ContinueWith(task =>
                {
                    HttpResponseMessage response = task.Result;

                    response = EncryptMessageIfTheMessageIsSecuredMessage(response);

                    return response;
                });
    }

    private HttpResponseMessage EncryptMessageIfTheMessageIsSecuredMessage(HttpResponseMessage response)
    {
        string data = response.Content.ReadAsString();

        var message = new Message
                          {
                              AppId = "Our App Id",
                              Id = Guid.NewGuid().ToString(),
                              Data = data,
                              TimeStamp = DateTime.UtcNow.ToString()
                          };

        EncryptedMessage encryptedMessage = _encryptionHelper.Encrypt(message);

        var content = new StringContent(Json.Encode(encryptedMessage), Encoding.UTF8,
                                        response.Content.Headers.ContentType.MediaType);

        response.Content.Dispose();

        response.Content = content;

        response.Headers.Add(XMessageType, XMessageTypeSecured);

        return response;
    }

    private HttpRequestMessage DecryptMessageIfTheMessageIsSecuredMessage(HttpRequestMessage request)
    {
        var encryptedMessage = request.Content.ReadAs();

        Message message = _encryptionHelper.Decrypt(encryptedMessage);

        var content = new StringContent(message.Data, Encoding.UTF8,
                                        request.Content.Headers.ContentType.MediaType);
                                        
        request.Content.Dispose();

        request.Content = content;

        return request;
    }
}

And your service function is as simple as it was at first.

    [WebInvoke(Method = "POST", UriTemplate = "Create")]
    public Player CreatePlayer(Player player)
    {
        return _playerRepository.Save(player);
    }