Graphinder – Application architecture

In a previous post I mentioned that I came to agreement with myself on Graphinder’s application architecture.
As it’s a moment when I’m slowly moving out to other services working around the whole infrastructure, it would be wise to wrap things up and pinpoint possible points of failure or misuse. I’d also share a little insight on my plans of putting such architecture in place on Azure, but wider coverage of that topic would come as a next article.

Application architecture overview

To keep you accustomed with my concept right from the start, I guess the wisest idea would be to start from graphical visualization.
As its far from ideal and still evolving (but not so dramatically as before), please keep it mind that it’s far from ideal in representing every aspect of architecture that’s out there. It would be impossible to put it all organized in a small space of diagram.

Application architecture diagram

Alrighty. We have three web applications I’m currently developing for Graphinder (WorkerApi, GatewayApi and Web) and one, that is currently on hold (Users).
For anyone that’s at least a little into microservices idea, one thing would be odd here. There is no service directory (or registry – have seen many names around the web) and my services are not designed to use service discovery.
Why not? Well, I’m really, really new into microservices. When I’ve said I’m quite new in Reactive Extensions and would want to compare it with my experience with microservices, I’d have to say I’m Rx.Net pro. But I’m not.
Step by step, over next iterations of project I’m gonna improve the whole architecture but hey, first things first. Let’s at least deliver minimal project on end of May!

Services communication flow

Since I have more than one communication flow from frontend down to database, I’ve separated responsibilities in algorithms domain:

  • Algorithms.GatewayApi – manages classical requests like get me a data set, accept a new data set and persists it etc; GatewayApi also stands as point of requesting problem solutions, manages queue of requests and stands as a point of registration for new WorkerApi instances; has also knowledge of SignalR hubs that will accept live progress reports from workers
  • Algorithms.WorkerApi – works on a problem received from gateway; persist current state of worked problem; notifies to address given by gateway; has no idea of nothing around except algorithms_db and parent GatewayApi

Example workflows

  • User posts new solution finding request → Web application calls Algorithms.GatewayApi with request data → GatewayApi enqueues request, callbacks on what has been done → Web informs user what has been done
  • User opens view for currently worked algorithm → Web connects user to SignalR hub user requested ↔ Worker keeps on posting progress to hub so that user has feedback on what’s going on
  • User requests historical data of once completed solution finding → Web calls GatewayApi for archival data -> GatewayApi takes data from algorithms_db and returns it to WebWeb displays archival data to user

The list of possible scenarios for this design is long, but I hope you get the idea on why it has been split up here.

Infrastructure concepts

Since I’m going to communicate over unsecured HTTP protocol throughout the architecture, I’d need to put some sort of environment isolation.
I’ve decided to put all services into separate virtual network, provided out of the box by Azure.
The only valid, public endpoint for accessing whole application would be HTTPS (443) port on Web application.
Since I will cover whole configuration on Azure in the next post, I will leave the rest for that post.

Point of interest

  • Since whole architecture is strongly encapsulated, there would be a need for at least one more public endpoint for other applications reaching services, e.g. mobile applications and other web apps
  • Provided I would like to add integration with other vendors services, I would need to decide whether Web application is point of connecting to them or should I provide small service inside infrastructure for this, depending on my requirements
  • Provided I would like to connect to any on-premise (or different Azure subscription) applications I have, I would need to think of Site-To-Site VPN configuration or additional endpoint for that (VPN gateway cost vs performance)

And that’s it for today. Let me know what you think on my current design.

Graphinder – Monthly Progress Report – April

Over week ago we’ve seen a second month of Daj sie poznac 2016 passing by.
Same as month ago, I’ve come to conclusion, that it would a good idea to note down what have I done in Graphinder project during last 4 weeks.

Current state of a project

April brought both progress on implementation and architecture levels. I’ve written less code and spent much, much more time actually thinking on why and how to do many things to stay both flexible and content about my decisions.

Algorithms microservice – persistence

First of all, persistence layer for Algorithms microservice in its bare implementation is in place.

While working on it, I needed to work my way around few hurdles, that is:

  • Unconventional domain layer – raw algorithms and optimization problems are generally a way of solving problems given by business, not a business domain per se; in Graphinder they are the core of the application
  • Unconventional data types – to preserver progress of algorithm, I need to persist its current state; persisting solution as One-To-Many association with Node table smelled like NO-NO solution, so I’ve ended up borrowing idea from Genetic Algorithm and mapped chosen nodes as an array of bool, mapping it layer as byte array to DB. Not an ideal solution but quite fast. Would probably need a revision for more universal approach for other optimization problems.
  • Completely hiding up knowledge of how persistence works – that was actually quite fun part to do and quite personally rewarding to see the outcome: any other layer that asks persistence to retrieve or save data, has no idea what’s going on out there and has no way to use it if not through queries and commands

  • Graphinder project – architecture

    Second of all, I’ve almost crystallized my idea on how an architecture of whole system would look like.

    While I’d like to keep this covered in a separate post, I’d like to note few things that will be part of project:

    • Graphinder.WebApp – application based on ASP.NET MVC, SignalR, D3.js and (possibly, but not from start) React.
    • Graphinder.Algorithm.GatewayApi – application based on ASP.NET WebAPI, serving as a gateway for scheduling work for algorithms and serving as CRUD endpoint for algorithms-related data; I’m almost 100% sure that it’d also share location with Microsoft Orleans.
    • Graphinder.Algorithm.WorkerApi – multiple instances of worker APIs that will be responsible for job retrieval, processing and persisting solutions for a problem and then returning it back to user; although written in WebAPI now, can be for example a NodeJS API and I’m definitely looking into creating one in a more distant future as proof-of-concept.
    • As project structure also changed, it made some of links in previous posts pointing to location that no longer exists on GitHub.
      I’d like to deeply apologize for that and as an atonement I’ve decided to… fix that next week. 🙈

      I’m also revising possibilities on fitting everything I want to show on each consecutive release on Azure. For now, I’ve finally configured virtual network based Windows Server 2012 VM with RDP access through VPN. Took only 4 hours. Huh…

      A week off from project

      You probably haven’t noticed if you didn’t visit my GitHub account page, but my contributions in April went down to 50% of what I’ve done in March. I’ve also ended up with 20 out of 30 days with a commit, as opposed to 23 out of 30 days in March.


      One of the reason of such state is that – as I’ve mentioned – I both fought few hurdles in persistence layer and wanted to revise entire Graphinder architecture idea based on experience I’ve made during those 2 months.
      But what’s more important is that as a simple, human being I needed some rest from a project and I took a whole week off from writing both posts and (partly) code for Graphinder. I’ve been busy with code nonetheless as Regalia: Of Men And Monarchs is coming due to its beta and some of school work made me crunch on it a little more than usual.
      While it’s almost impossible in job environment to take a week off from a certain project not taking a break from work per se, I’m glad I could do this with Graphinder as now I’ve already grew hungry for new things that are waiting for implementation, starting from Monday and on!

Graphinder – Solution changes and nuget hell

While I’m moving my solution closer and close to first microservice part, I’ve decided to rearrange my solution a little.
I’ve decided to divide folder hierarchy inside solution, based on to which microservice part it belongs.
Besides a minor inconvenience on having to manually rearrange folders despite the fact that they were already in Visual Studio (solution folder does not represent physical one in system), I think it went quite smoothly. As long as I didn’t attempt to restore my projects’ depedendencies… and entering nuget hell.

Nuget hell genesis

I’ve decided to move all my algorithms project elements from solution root to subfolder of /Algorithms.
Of course, first thought: “there would be problem with packages path”. Nuget will have problem with finding ../packages folder as it’s located by default in solution root.
Alright, I’ll change paths in project files then. Example change that occurred in .csproj was:



Straightforward, right? Right.
But Visual Studio insisted that:

This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them.

But by the love of… they ARE THERE. !#(*@#*(!^#*!^##&T*$@(#!@!
Alright, calm down. Let’s look for some clues then. Keep looking…
We have a suspicious thing here. Error tells us that the guilty one is \packages\xunit.runner.visualstudio.2.1.0\build.

Nuget hell – solution (a.k.a nuke em all)

What would a sane developer do? Delete reference perhaps. Let’s try out then.
Reference of xunit.runner.visualstudio.2.1.0 removed, rebuilding project, looking at build output…

This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them.


But I’m not even referencing this, how come!?
Let’s see what’s in the project’s .csproj once again…
Wait a second fella, what are you still doing here?

Whole XML node removed, nuget readded from Package Manager Console, let’s see now what will explode…

========== Build: 11 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

Now that’s what I’m talking about! New solution hierarchy feels a little more organized now, although I’m already thinking of splitting it into Shared, WorkerApi and GatewayApi subfolders for the sake of readability.

Final outcome of my crusade against nuget references:

nuget hell solution outcome