Graphinder – Resources planning on Azure


As promised in previous post about application architecture, I’d like to share few words on my (moment of hesitation here) journey with resource planning on Azure. We’ll play around with network configuration for our environment, define application and development endpoints and juggle with possible service plans to meet our needs.
Oh, I’d like to mention that I’m gonna show things in old (https://manage.windowsazure.com) Azure portal as not everything is possible in new one (shame, because I love the direction MSFT is push it). ☁️


Network planning on Azure

Since we want to run quite a few microservices around with possibly lowest overhead possible (authenticating between services left out too if possible), planning a virtual network for our environment would be a good place to start.
Let’s think of a simple network that would suit our needs:

Azure network diagram

Our points of interests here would be:

  • Virtual machines – hosts for our applications; even if we’re running App Service on Azure, it’s still a virtual machine so I won’t differentiate here
  • Endpoints – actual ports opened either internally in virtual network or outside, through VM public address
  • VPN gateway – point of accessing virtual network and operating inside of it while connecting from outside


Creating virtual network

As I’ve had lot of trouble with configuring VPN gateway on new portal, I will show most of deployment on old portal located @ https://manage.windowsazure.com.
Let’s choose ‘Networks’ from left-hand menu, then create New from bottom menu and go with Custom Create.
Name the network, choose the region and on a next screen make sure to mark Configure a point-to-site VPN:

Point-to-Site VPN

Few next screens will guide you with VPN configuration that is quite straightforward, so I won’t go through it.
In case you have trouble, Cheryl McGuire got this covered on MSFT article Configure a Point-to-Site VPN connection to a VNet using the classic portal.

Provisioning of network along with VPN gateway may take up to 1h (was my case), so you need to have patience. 😥


Creating virtual machines

Whether its Windows Server or Linux machine you’re aiming to deploy on Azure, it’s important to put it in a virtual network we’ve created.
Since its impossible to do this later on, make sure you did not skip this part. On a Create a virtual machine -> Virtual machine configuration part, there is a dropdown for selecting a region for your virtual machine:

Virtual Machine to Virtual network

It’s easy to miss that you need to actually choose previously created virtual network here, so that VM would belong to it.
On a same screen you will also have a possibility to define endpoints for your virtual machine. You can leave them as they are for now and define/change them later from either new or old Azure portal.

Example endpoints generated by Azure wizard:
endpoints


Watching out the cost

When deploying your environment to Azure, you need to take into account costs of VM and VPN gateway, as they would hurt the most.
Both virtual machine and gateway will be charged for amount of time it is enabled, no matter if you’re using it or not.
As for virtual machines, you can actually stop them, so they become deallocated and you won’t be charged
As for VPN gateway, there is no such an option and you are forced to delete it if not in use.
You might want to use a public port for RDP access to your machines if that’s fine for you tough.

Graphinder – Application architecture


In a previous post I mentioned that I came to agreement with myself on Graphinder’s application architecture.
As it’s a moment when I’m slowly moving out to other services working around the whole infrastructure, it would be wise to wrap things up and pinpoint possible points of failure or misuse. I’d also share a little insight on my plans of putting such architecture in place on Azure, but wider coverage of that topic would come as a next article.

Application architecture overview

To keep you accustomed with my concept right from the start, I guess the wisest idea would be to start from graphical visualization.
As its far from ideal and still evolving (but not so dramatically as before), please keep it mind that it’s far from ideal in representing every aspect of architecture that’s out there. It would be impossible to put it all organized in a small space of diagram.



Application architecture diagram

Alrighty. We have three web applications I’m currently developing for Graphinder (WorkerApi, GatewayApi and Web) and one, that is currently on hold (Users).
For anyone that’s at least a little into microservices idea, one thing would be odd here. There is no service directory (or registry – have seen many names around the web) and my services are not designed to use service discovery.
Why not? Well, I’m really, really new into microservices. When I’ve said I’m quite new in Reactive Extensions and would want to compare it with my experience with microservices, I’d have to say I’m Rx.Net pro. But I’m not.
Step by step, over next iterations of project I’m gonna improve the whole architecture but hey, first things first. Let’s at least deliver minimal project on end of May!


Services communication flow

Since I have more than one communication flow from frontend down to database, I’ve separated responsibilities in algorithms domain:

  • Algorithms.GatewayApi – manages classical requests like get me a data set, accept a new data set and persists it etc; GatewayApi also stands as point of requesting problem solutions, manages queue of requests and stands as a point of registration for new WorkerApi instances; has also knowledge of SignalR hubs that will accept live progress reports from workers
  • Algorithms.WorkerApi – works on a problem received from gateway; persist current state of worked problem; notifies to address given by gateway; has no idea of nothing around except algorithms_db and parent GatewayApi

Example workflows

  • User posts new solution finding request → Web application calls Algorithms.GatewayApi with request data → GatewayApi enqueues request, callbacks on what has been done → Web informs user what has been done
  • User opens view for currently worked algorithm → Web connects user to SignalR hub user requested ↔ Worker keeps on posting progress to hub so that user has feedback on what’s going on
  • User requests historical data of once completed solution finding → Web calls GatewayApi for archival data -> GatewayApi takes data from algorithms_db and returns it to WebWeb displays archival data to user

The list of possible scenarios for this design is long, but I hope you get the idea on why it has been split up here.

Infrastructure concepts

Since I’m going to communicate over unsecured HTTP protocol throughout the architecture, I’d need to put some sort of environment isolation.
I’ve decided to put all services into separate virtual network, provided out of the box by Azure.
The only valid, public endpoint for accessing whole application would be HTTPS (443) port on Web application.
Since I will cover whole configuration on Azure in the next post, I will leave the rest for that post.

Point of interest

  • Since whole architecture is strongly encapsulated, there would be a need for at least one more public endpoint for other applications reaching services, e.g. mobile applications and other web apps
  • Provided I would like to add integration with other vendors services, I would need to decide whether Web application is point of connecting to them or should I provide small service inside infrastructure for this, depending on my requirements
  • Provided I would like to connect to any on-premise (or different Azure subscription) applications I have, I would need to think of Site-To-Site VPN configuration or additional endpoint for that (VPN gateway cost vs performance)

And that’s it for today. Let me know what you think on my current design.

Graphinder – Monthly Progress Report – April


Over week ago we’ve seen a second month of Daj sie poznac 2016 passing by.
Same as month ago, I’ve come to conclusion, that it would a good idea to note down what have I done in Graphinder project during last 4 weeks.

Current state of a project

April brought both progress on implementation and architecture levels. I’ve written less code and spent much, much more time actually thinking on why and how to do many things to stay both flexible and content about my decisions.


Algorithms microservice – persistence

First of all, persistence layer for Algorithms microservice in its bare implementation is in place.

While working on it, I needed to work my way around few hurdles, that is:

  • Unconventional domain layer – raw algorithms and optimization problems are generally a way of solving problems given by business, not a business domain per se; in Graphinder they are the core of the application
  • Unconventional data types – to preserver progress of algorithm, I need to persist its current state; persisting solution as One-To-Many association with Node table smelled like NO-NO solution, so I’ve ended up borrowing idea from Genetic Algorithm and mapped chosen nodes as an array of bool, mapping it layer as byte array to DB. Not an ideal solution but quite fast. Would probably need a revision for more universal approach for other optimization problems.
  • Completely hiding up knowledge of how persistence works – that was actually quite fun part to do and quite personally rewarding to see the outcome: any other layer that asks persistence to retrieve or save data, has no idea what’s going on out there and has no way to use it if not through queries and commands

  • Graphinder project – architecture

    Second of all, I’ve almost crystallized my idea on how an architecture of whole system would look like.

    While I’d like to keep this covered in a separate post, I’d like to note few things that will be part of project:

    • Graphinder.WebApp – application based on ASP.NET MVC, SignalR, D3.js and (possibly, but not from start) React.
    • Graphinder.Algorithm.GatewayApi – application based on ASP.NET WebAPI, serving as a gateway for scheduling work for algorithms and serving as CRUD endpoint for algorithms-related data; I’m almost 100% sure that it’d also share location with Microsoft Orleans.
    • Graphinder.Algorithm.WorkerApi – multiple instances of worker APIs that will be responsible for job retrieval, processing and persisting solutions for a problem and then returning it back to user; although written in WebAPI now, can be for example a NodeJS API and I’m definitely looking into creating one in a more distant future as proof-of-concept.
    • As project structure also changed, it made some of links in previous posts pointing to location that no longer exists on GitHub.
      I’d like to deeply apologize for that and as an atonement I’ve decided to… fix that next week. 🙈

      I’m also revising possibilities on fitting everything I want to show on each consecutive release on Azure. For now, I’ve finally configured virtual network based Windows Server 2012 VM with RDP access through VPN. Took only 4 hours. Huh…

      A week off from project

      You probably haven’t noticed if you didn’t visit my GitHub account page, but my contributions in April went down to 50% of what I’ve done in March. I’ve also ended up with 20 out of 30 days with a commit, as opposed to 23 out of 30 days in March.

      mwilczynski-contributions-graphinder

      One of the reason of such state is that – as I’ve mentioned – I both fought few hurdles in persistence layer and wanted to revise entire Graphinder architecture idea based on experience I’ve made during those 2 months.
      But what’s more important is that as a simple, human being I needed some rest from a project and I took a whole week off from writing both posts and (partly) code for Graphinder. I’ve been busy with code nonetheless as Regalia: Of Men And Monarchs is coming due to its beta and some of school work made me crunch on it a little more than usual.
      While it’s almost impossible in job environment to take a week off from a certain project not taking a break from work per se, I’m glad I could do this with Graphinder as now I’ve already grew hungry for new things that are waiting for implementation, starting from Monday and on!